Why am I limited to 1,000 rows on SODA API when I have an App Key

Question

I'm trying to load the Consumer Financial Protection Bureau (CFPB) into a Pandas Data Frame in Python 3.4. My code works, however it appears that I'm limited to 1,000 rows through the SODA API. I'm using an app token and referencing the URL here

df = pd.read_json('http://data.consumerfinance.gov/resource/x94z-ydhh.json?$$app_token=XXXXXXXXXXXXXXXX')

The data structure is perfect, however I'm limited to only 1,000 rows of data (should be closer to 300,000 rows of data).

When I access the same URL through a browser (with app token), it appears it is pulling in only 1,000 rows as well. I was under the impression that with an App Token you could access the entire data set, is that incorrect?

Answer 1

Check out the docs on the $limit parameter

The $limit parameter controls the total number of rows returned, and it defaults to 1,000 records per request.

Note: The maximum value for $limit is 50,000 records, and if you exceed that limit you'll get a 400 Bad Request response.

So you're just getting the default number of records back.

You will not be able to get all 300,000 records in a single API call - this will take multiple calls using $limit together with $offset

Try:

http://data.consumerfinance.gov/resource/x94z-ydhh.json?$limit=50000&$$app_token=XXXXXXXXXXXXXXXX