|
34 | 34 | "metadata": {}, |
35 | 35 | "outputs": [], |
36 | 36 | "source": [ |
37 | | - "%pip install --quiet 'rustac[arrow]' pystac-client planetary-computer geopandas pyarrow" |
| 37 | + "%pip install --quiet 'rustac[arrow]' pystac-client planetary-computer geopandas pyarrow obstore" |
38 | 38 | ] |
39 | 39 | }, |
40 | 40 | { |
|
43 | 43 | "source": [ |
44 | 44 | "## Find the GeoParquet snapshot", |
45 | 45 | "", |
46 | | - "Every Planetary Computer collection carries a collection-level `geoparquet-items` asset that points at its STAC-items snapshot. The href resolves to a directory of monthly `part-NNNN` Parquet files on the `pcstacitems` storage account.", |
| 46 | + "Every Planetary Computer collection carries a collection-level `geoparquet-items` asset that points at its STAC-items snapshot. The href resolves to a directory of monthly `part-NNNN` partition files on the `pcstacitems` storage account.", |
47 | 47 | "", |
48 | 48 | "**Expected result:** `abfs://items/sentinel-2-l2a.parquet`." |
49 | 49 | ] |
|
61 | 61 | "asset.href" |
62 | 62 | ] |
63 | 63 | }, |
| 64 | + { |
| 65 | + "cell_type": "markdown", |
| 66 | + "metadata": {}, |
| 67 | + "source": [ |
| 68 | + "## List the partition files", |
| 69 | + "", |
| 70 | + "The snapshot is a set of monthly `part-NNNN` partition files. List them with [obstore](https://developmentseed.org/obstore/), which signs Planetary Computer blobs through its own credential provider. Each filename embeds the partition's date range, which the query section below uses to skip partitions.", |
| 71 | + "", |
| 72 | + "**Expected result:** about 136 partition files, the first covering July 2015." |
| 73 | + ] |
| 74 | + }, |
| 75 | + { |
| 76 | + "cell_type": "code", |
| 77 | + "execution_count": null, |
| 78 | + "metadata": {}, |
| 79 | + "outputs": [], |
| 80 | + "source": [ |
| 81 | + "from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider\n", |
| 82 | + "from obstore.store import AzureStore\n", |
| 83 | + "\n", |
| 84 | + "store = AzureStore(\n", |
| 85 | + " credential_provider=PlanetaryComputerCredentialProvider(\n", |
| 86 | + " \"https://pcstacitems.blob.core.windows.net/items\"\n", |
| 87 | + " )\n", |
| 88 | + ")\n", |
| 89 | + "partitions = [\n", |
| 90 | + " meta[\"path\"]\n", |
| 91 | + " for stream in store.list(prefix=\"sentinel-2-l2a.parquet\")\n", |
| 92 | + " for meta in stream\n", |
| 93 | + " if \"/part-\" in meta[\"path\"]\n", |
| 94 | + "]\n", |
| 95 | + "len(partitions), partitions[0]" |
| 96 | + ] |
| 97 | + }, |
64 | 98 | { |
65 | 99 | "cell_type": "markdown", |
66 | 100 | "metadata": {}, |
67 | 101 | "source": [ |
68 | 102 | "## Authenticate the DuckDB engine", |
69 | 103 | "", |
70 | | - "The account name looks public, but reads still need a SAS token. The high-level `rustac.search()` coroutine cannot pass Azure credentials, so use `rustac.DuckdbClient` instead and configure its connection once: fetch a container SAS from the Planetary Computer token API, register an Azure secret, and switch the Azure transport to curl.", |
| 104 | + "Reading from the account needs a Shared Access Signature (SAS) token. The high-level `rustac.search()` coroutine cannot pass Azure credentials, so use `rustac.DuckdbClient` instead and configure its connection once: fetch a container SAS from the Planetary Computer token API, register an Azure secret, and switch the Azure transport to curl.", |
71 | 105 | "", |
72 | 106 | "The `azure_transport_option_type = 'curl'` line is not optional. Without it, DuckDB's default Azure transport fails with an opaque SSL CA-certificate error. The SAS expires after about an hour, so long-running jobs re-fetch it.", |
73 | 107 | "", |
|
102 | 136 | "source": [ |
103 | 137 | "## Query the snapshot", |
104 | 138 | "", |
105 | | - "Point `search()` at the snapshot glob and pass the filters you want pushed into the read, so only matching rows cross the network. The glob (`/*.parquet`) spans the monthly part files.", |
| 139 | + "Point `search()` at the snapshot glob and pass the filters you want pushed into the read, so only matching rows cross the network. The glob (`/*.parquet`) spans the monthly partition files.", |
106 | 140 | "", |
107 | | - "`DuckdbClient.search` is synchronous (no `await`); the bundled engine scans the part files for you, which takes a little time on the first call.", |
| 141 | + "`DuckdbClient.search` is synchronous (no `await`); this is a search over every partition file (about 136), with no index, so expect it to take a minute or two on the first call.", |
108 | 142 | "", |
109 | 143 | "**Expected result:** a handful of Portland scenes (5)." |
110 | 144 | ] |
|
153 | 187 | "len(items)" |
154 | 188 | ] |
155 | 189 | }, |
| 190 | + { |
| 191 | + "cell_type": "markdown", |
| 192 | + "metadata": {}, |
| 193 | + "source": [ |
| 194 | + "## Narrow the glob to skip partitions", |
| 195 | + "", |
| 196 | + "Because each partition filename embeds its date range (`part-NNNN_<start>_<end>.parquet`), you can point the glob at only the partitions a query needs instead of scanning every partition. For a 2017 query, match the 2017 partitions:", |
| 197 | + "", |
| 198 | + "**Expected result:** the same scenes as the full-glob search, but noticeably faster (the read touches only the 2017 partitions)." |
| 199 | + ] |
| 200 | + }, |
| 201 | + { |
| 202 | + "cell_type": "code", |
| 203 | + "execution_count": null, |
| 204 | + "metadata": {}, |
| 205 | + "outputs": [], |
| 206 | + "source": [ |
| 207 | + "items = client.search(\n", |
| 208 | + " \"az://items/sentinel-2-l2a.parquet/part-*_2017-*.parquet\",\n", |
| 209 | + " collections=[\"sentinel-2-l2a\"],\n", |
| 210 | + " bbox=[-122.7, 45.5, -122.6, 45.6],\n", |
| 211 | + " datetime=\"2017-07-01/2017-08-01\",\n", |
| 212 | + ")\n", |
| 213 | + "len(items)" |
| 214 | + ] |
| 215 | + }, |
156 | 216 | { |
157 | 217 | "cell_type": "markdown", |
158 | 218 | "metadata": {}, |
|
235 | 295 | "source": [ |
236 | 296 | "## You're done", |
237 | 297 | "", |
238 | | - "If every cell above ran, you pushed a spatial, temporal, and cloud-cover filter into a remote GeoParquet snapshot, read back only the matching Sentinel-2 scenes, wrote them to Parquet, and bridged them into GeoPandas: STAC-GeoParquet \u2192 filtered Arrow table \u2192 GeoDataFrame, with no full download.", |
| 298 | + "If every cell above ran, you used a spatial, temporal, and cloud-cover filter to read matching Sentinel-2 scenes, wrote them to Parquet, and bridged them into GeoPandas: STAC-GeoParquet \u2192 filtered Arrow table \u2192 GeoDataFrame, with no full download.", |
239 | 299 | "", |
240 | 300 | "Swap in your own bbox, datetime window, or collection and the same pattern applies. For small or current-data queries, the live STAC API through `pystac-client` is the better tool. To ask analytical questions or join the catalog against other datasets, see the [DuckDB tutorial](../overview/duckdb.md), which uses the same authentication. For the imagery behind the metadata, hand a signed asset href to the [async-geotiff tutorial](../overview/async-geotiff.md)." |
241 | 301 | ] |
|
0 commit comments