Skip to content

Postmortem: Ingesting NSW Spatial GIS API

Jump to bottom
Angus edited this page Jan 14, 2025 · 1 revision

Challenges

Pagination Limits

I've found the further you paginate into a series of pages the more likely it has to crash on you, I addressed this by creating some predefined where_clause's which I can anticipate not having to paginate all that deep. This is the primary reason this exists.

Content Size of Requests

I've found when I've requested more fields in a request, it has increased the likelihood of the API timing out or omitting results for some reason. I'm fairly certain this is the case because:

  1. I saw this issue to start to occur once I increased the number of fields returned from the API.
  2. This issue stopped being an issue once I decreased the number of rows being returned while pagination.

When it broke the API didn't return a 400 or a 500, it just returned less items, and it flagged that the request was truncated or something (it's difficult to reproduce this. I initially mistook this for the number of results changing while I was ingesting the data, but I no longer believe that's the case.

Clone this wiki locally