Skip to content

Commit 1733a12

Browse files
aboydnwclaude
andcommitted
docs: address rustac quickstart review
- add an obstore cell listing the partition files - show glob narrowing (part-*_2017-*.parquet) to skip partitions, ~3x faster - call them partition files, not part files - clarify the SAS-token sentence; expand the acronym - note the search scans every partition with no index - trim the closing recap Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 9b74227 commit 1733a12

1 file changed

Lines changed: 66 additions & 6 deletions

File tree

quickstarts/rustac.ipynb

Lines changed: 66 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@
3434
"metadata": {},
3535
"outputs": [],
3636
"source": [
37-
"%pip install --quiet 'rustac[arrow]' pystac-client planetary-computer geopandas pyarrow"
37+
"%pip install --quiet 'rustac[arrow]' pystac-client planetary-computer geopandas pyarrow obstore"
3838
]
3939
},
4040
{
@@ -43,7 +43,7 @@
4343
"source": [
4444
"## Find the GeoParquet snapshot",
4545
"",
46-
"Every Planetary Computer collection carries a collection-level `geoparquet-items` asset that points at its STAC-items snapshot. The href resolves to a directory of monthly `part-NNNN` Parquet files on the `pcstacitems` storage account.",
46+
"Every Planetary Computer collection carries a collection-level `geoparquet-items` asset that points at its STAC-items snapshot. The href resolves to a directory of monthly `part-NNNN` partition files on the `pcstacitems` storage account.",
4747
"",
4848
"**Expected result:** `abfs://items/sentinel-2-l2a.parquet`."
4949
]
@@ -61,13 +61,47 @@
6161
"asset.href"
6262
]
6363
},
64+
{
65+
"cell_type": "markdown",
66+
"metadata": {},
67+
"source": [
68+
"## List the partition files",
69+
"",
70+
"The snapshot is a set of monthly `part-NNNN` partition files. List them with [obstore](https://developmentseed.org/obstore/), which signs Planetary Computer blobs through its own credential provider. Each filename embeds the partition's date range, which the query section below uses to skip partitions.",
71+
"",
72+
"**Expected result:** about 136 partition files, the first covering July 2015."
73+
]
74+
},
75+
{
76+
"cell_type": "code",
77+
"execution_count": null,
78+
"metadata": {},
79+
"outputs": [],
80+
"source": [
81+
"from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider\n",
82+
"from obstore.store import AzureStore\n",
83+
"\n",
84+
"store = AzureStore(\n",
85+
" credential_provider=PlanetaryComputerCredentialProvider(\n",
86+
" \"https://pcstacitems.blob.core.windows.net/items\"\n",
87+
" )\n",
88+
")\n",
89+
"partitions = [\n",
90+
" meta[\"path\"]\n",
91+
" for stream in store.list(prefix=\"sentinel-2-l2a.parquet\")\n",
92+
" for meta in stream\n",
93+
" if \"/part-\" in meta[\"path\"]\n",
94+
"]\n",
95+
"len(partitions), partitions[0]"
96+
]
97+
},
6498
{
6599
"cell_type": "markdown",
66100
"metadata": {},
67101
"source": [
68102
"## Authenticate the DuckDB engine",
69103
"",
70-
"The account name looks public, but reads still need a SAS token. The high-level `rustac.search()` coroutine cannot pass Azure credentials, so use `rustac.DuckdbClient` instead and configure its connection once: fetch a container SAS from the Planetary Computer token API, register an Azure secret, and switch the Azure transport to curl.",
104+
"Reading from the account needs a Shared Access Signature (SAS) token. The high-level `rustac.search()` coroutine cannot pass Azure credentials, so use `rustac.DuckdbClient` instead and configure its connection once: fetch a container SAS from the Planetary Computer token API, register an Azure secret, and switch the Azure transport to curl.",
71105
"",
72106
"The `azure_transport_option_type = 'curl'` line is not optional. Without it, DuckDB's default Azure transport fails with an opaque SSL CA-certificate error. The SAS expires after about an hour, so long-running jobs re-fetch it.",
73107
"",
@@ -102,9 +136,9 @@
102136
"source": [
103137
"## Query the snapshot",
104138
"",
105-
"Point `search()` at the snapshot glob and pass the filters you want pushed into the read, so only matching rows cross the network. The glob (`/*.parquet`) spans the monthly part files.",
139+
"Point `search()` at the snapshot glob and pass the filters you want pushed into the read, so only matching rows cross the network. The glob (`/*.parquet`) spans the monthly partition files.",
106140
"",
107-
"`DuckdbClient.search` is synchronous (no `await`); the bundled engine scans the part files for you, which takes a little time on the first call.",
141+
"`DuckdbClient.search` is synchronous (no `await`); this is a search over every partition file (about 136), with no index, so expect it to take a minute or two on the first call.",
108142
"",
109143
"**Expected result:** a handful of Portland scenes (5)."
110144
]
@@ -153,6 +187,32 @@
153187
"len(items)"
154188
]
155189
},
190+
{
191+
"cell_type": "markdown",
192+
"metadata": {},
193+
"source": [
194+
"## Narrow the glob to skip partitions",
195+
"",
196+
"Because each partition filename embeds its date range (`part-NNNN_<start>_<end>.parquet`), you can point the glob at only the partitions a query needs instead of scanning every partition. For a 2017 query, match the 2017 partitions:",
197+
"",
198+
"**Expected result:** the same scenes as the full-glob search, but noticeably faster (the read touches only the 2017 partitions)."
199+
]
200+
},
201+
{
202+
"cell_type": "code",
203+
"execution_count": null,
204+
"metadata": {},
205+
"outputs": [],
206+
"source": [
207+
"items = client.search(\n",
208+
" \"az://items/sentinel-2-l2a.parquet/part-*_2017-*.parquet\",\n",
209+
" collections=[\"sentinel-2-l2a\"],\n",
210+
" bbox=[-122.7, 45.5, -122.6, 45.6],\n",
211+
" datetime=\"2017-07-01/2017-08-01\",\n",
212+
")\n",
213+
"len(items)"
214+
]
215+
},
156216
{
157217
"cell_type": "markdown",
158218
"metadata": {},
@@ -235,7 +295,7 @@
235295
"source": [
236296
"## You're done",
237297
"",
238-
"If every cell above ran, you pushed a spatial, temporal, and cloud-cover filter into a remote GeoParquet snapshot, read back only the matching Sentinel-2 scenes, wrote them to Parquet, and bridged them into GeoPandas: STAC-GeoParquet \u2192 filtered Arrow table \u2192 GeoDataFrame, with no full download.",
298+
"If every cell above ran, you used a spatial, temporal, and cloud-cover filter to read matching Sentinel-2 scenes, wrote them to Parquet, and bridged them into GeoPandas: STAC-GeoParquet \u2192 filtered Arrow table \u2192 GeoDataFrame, with no full download.",
239299
"",
240300
"Swap in your own bbox, datetime window, or collection and the same pattern applies. For small or current-data queries, the live STAC API through `pystac-client` is the better tool. To ask analytical questions or join the catalog against other datasets, see the [DuckDB tutorial](../overview/duckdb.md), which uses the same authentication. For the imagery behind the metadata, hand a signed asset href to the [async-geotiff tutorial](../overview/async-geotiff.md)."
241301
]

0 commit comments

Comments
 (0)