API Record Limit

[jpulver]

Is there a limit to 20,000 records being returned by the API? IF so, is there a workaround for this? Thanks

[jadudm]

Hi @jpulver ,

The API has a 20K record limit per query.

PostgREST (which is the software that acts as our API server) has pagination. So, you should be able to issue a query, find out you have 63K records, and then paginate through them in chunks of 20K,20K,20K,4K.

I believe it is safe to say that 100% of the PostgREST documentation applies to our system.

[jpulver]

Great thank you. It would be helpful if you could add that info to the FAC documentation page as I did not see that record limit mentioned anywhere. You help is appreciated

[jadudm]

I have draft improvements queued that make this more obvious, and will do thank you!

[jadudm]

To follow up:

I have been known, when pressed, to wrap all queries in a loop, and then loop until I get back [] (in Python), as a query where you've paginated to a point that there are no more records will come back empty.

https://github.com/GSA-TTS/fac-api-examples/blob/6c8c80ecd259ffd55465c32db3eada503e5cfbac/sum-over-alns/main.py#L51

might be a sloppy example. Googling for "PostgREST paginate query" might turn up other examples. (I'm between meetings, so apologies for a quick reply, but I thought some pointers might be better than none.)