Exception while fetching more than 25 posts restFb

Question

I am using restFb version 1.34.1 and I want to extract all posts of a page. Since there is a limit of 25 posts that can be obtained in one go, hence I am using the

hasNext

and

getNextPageUrl

methods to get more posts. Here is my code snippet for the same.

Connection<Post> pageFeed = facebookClient.fetchConnection(page.getId() + "/feed", Post.class);
        //pageFeed.hasNext()
        List<Post> data = pageFeed.getData();
        int count=0;
        for (Post post : data) {
            count++;
            System.out.println("count------------------------>"+count);
            System.out.println("story-->"+post.getStory());
            System.out.println("message--->"+post.getMessage());
            System.out.println("postid---------->"+post.getId());

        }
        while(pageFeed.hasNext())
        {
            System.out.println(pageFeed.getNextPageUrl());
            pageFeed = facebookClient.fetchConnection(pageFeed.getNextPageUrl(),Post.class);
            System.out.println(pageFeed);
             data = pageFeed.getData();
            for (Post post : data) {
                count++;
                System.out.println("count------------------------>"+count);
                System.out.println("story-->"+post.getStory());
                System.out.println("message--->"+post.getMessage());
                System.out.println("postid---------->"+post.getId());
                System.out.println("----------------------------");
            }
        }

I am getting first 25 posts successfully but there is an exception on the line

pageFeed = facebookClient.fetchConnection(pageFeed.getNextPageUrl(),Post.class);

Following is the exception

Exception in thread "main" com.restfb.json.JsonException: JsonObject["data"] not found.
    at com.restfb.json.JsonObject.get(JsonObject.java:482)
    at com.restfb.json.JsonObject.getJsonArray(JsonObject.java:550)
    at com.restfb.Connection.<init>(Connection.java:154)
    at com.restfb.DefaultFacebookClient.fetchConnection(DefaultFacebookClient.java:363)
    at com.orkash.restFb.PostExtractor.crawlPage(PostExtractor.java:76)
    at com.orkash.restFb.PostExtractor.main(PostExtractor.java:30)

I did check the docs and other stackthreads for the same and found this method only for fetching more than 25 posts. Can anyone please point out the mistake here.

I also made a get request to the nextPageUrl obtained via browser and obtained the json which has the key "data" in it. Here is the sample json obtained via the process mentioned above :- (The complete json is pretty long so I have pasted a snippet of it)

{
  "data": [
    {
      "id": "177526890164_10158068948860165",
      "from": {
        "name": "Narendra Modi",
        "category": "Politician",
        "id": "177526890164"
      },
      "message": "Paid tributes to Shri Sunder Lal Patwa in Bhopal.",
      "story": "Narendra Modi added 2 new photos.",
      "story_tags": {
        "0": [
          {
            "id": "177526890164",
            "name": "Narendra Modi",
            "type": "page",
            "offset": 0,
            "length": 13
          }
        ]
      },
      "picture": "https://scontent.xx.fbcdn.net/v/t1.0-0/s130x130/15673047_10158068946670165_3063831989784759068_n.jpg?oh=248d746b114f486e12be04058dfd0791&oe=591975EE",
      "link": "https://www.facebook.com/narendramodi/photos/a.10150164299700165.421791.177526890164/10158068946670165/?type=3",
      "name": "Photos from Narendra Modi's post",
      "icon": "https://www.facebook.com/images/icons/photo.gif",
      "actions": [
        {
          "name": "Comment",
          "link": "https://www.facebook.com/177526890164/posts/10158068948860165"
        },
        {
          "name": "Like",
          "link": "https://www.facebook.com/177526890164/posts/10158068948860165"
        }
      ],
      "privacy": {
        "value": "",
        "description": "",
        "friends": "",
        "allow": "",
        "deny": ""
      },
      "type": "photo",
      "status_type": "added_photos",
      "object_id": "10158068946670165",
      "created_time": "2016-12-28T11:59:25+0000",
      "updated_time": "2017-01-04T15:59:35+0000",
      "shares": {
        "count": 2708
      },
      "is_hidden": false,
      "is_expired": false,
      "likes": {
        "data": [
          {
            "id": "131388654027704",
            "name": "Lovely Setia"
          },
          {
            "id": "111162672721989",
            "name": "Dilip Bhil"
          },
          {
            "id": "1554142888157290",
            "name": "Raees Khan Plastick"
          },
          {
            "id": "105352369807500",
            "name": "Dhawan Sharma"
          },
          {
            "id": "205545709915488",
            "name": "Kehraram Parajapti"
          },
          {
            "id": "1815854775323914",
            "name": "Àvàý Śhàŕmà"
          },
          {
            "id": "1809366642682126",
            "name": "Posanjit Chakraborty"
          },
          {
            "id": "814626555322719",
            "name": "Pappu Ray"
          },
          {
            "id": "143505946020792",
            "name": "Mand Jaz"
          },
          {
            "id": "1671421303114441",
            "name": "Pranab Jyoti Gogoi"
          },
          {
            "id": "1705764099657485",
            "name": "Riya Singh"
          },
          {
            "id": "243742032692553",
            "name": "Nevji Bhli"
          },
          {
            "id": "716696131727496",
            "name": "Purushotham Naidu"
          },
          {
            "id": "193933561077517",
            "name": "Durgesh Sahu Sahu"
          },
          {
            "id": "231379427317074",
            "name": "Pankaj Kumar"
          },
          {
            "id": "132570197106057",
            "name": "Srinivas Battar"
          },
          {
            "id": "144547786036296",
            "name": "Jgtpj Uday Prakash"
          },
          {
            "id": "206532629811957",
            "name": "Rajabhau Raut"
          },
          {
            "id": "469875846448916",
            "name": "Drkishor Sinha"
          },
          {
            "id": "140811979746333",
            "name": "Satyam Lodhi"
          },
          {
            "id": "109671812871571",
            "name": "Daleep Gour Daleep"
          },
          {
            "id": "1577239832532749",
            "name": "Tejas Nemanwar"
          },
          {
            "id": "658207560971950",
            "name": "Uttamsingh Rajpurohit"
          },
          {
            "id": "147763812380490",
            "name": "Lovelygiri Giri"
          },
          {
            "id": "748935241836869",
            "name": "Abhijeet Joshi"
          }
        ],

Answer 1

You used the Connection the wrong way. It should look like this:

int count=0;
Connection<Post> pageFeed = facebookClient.fetchConnection(page.getId() + "/feed", Post.class);
for (List<Post> pageFeedConnectionPage : pageFeed)
    for (Post post : pageFeedConnectionPage) {
        count++;
        System.out.println("count------------------------>" + count);
        System.out.println("story-->" + post.getStory());
        System.out.println("message--->" + post.getMessage());
        System.out.println("postid---------->" + post.getId());
        System.out.println("----------------------------");
    }

Iterating over the Connection returns the "pages" and sends the next call automatically to the Graph API. On every page, there's a list of posts and you can iterate over this list, too.

BTW, there's an example in the documentation and there are several integration test that show how the connection is meant to be used.