Ocean Colour Notebook #2 #169
Comments
|
Hi @chiaravol, If you are waiting a long time to download data, the CDS is probably quite busy at the moment. Just let me know the parameters to use for caching. |
|
Hi Mattia, Thanks for the prompt reply. I remember having a similar issue with the previous NB as the data set is pretty heavy. It would be great if you can cache the data for me! :) I need to analyze all chlor_a, Rrs_443 and Rrs_560 data between September 1997 and June 2024 in two regions: 1) the Antarctic Ocean (between latitudes -47.5°S and -63.5°S and longitudes -180, 180) and 2) the Arctic Ocean (between latitudes 60°N and 90°N and longitudes -180, 180). It's a large amount of data. If you think it's too much, we can chache only Chlor_a data for now. Cheers, |
|
Please, note that the two regions are specified already in the NB I shared. |
|
I started caching the files right now, I'll be in touch next week. |
|
Thanks Mattia. Wishing you a good weekend as well! |
|
Hi @chiaravol,
import xarray as xr
from c3s_eqc_automatic_quality_control import download, utils
# Time period
start = "1997-09"
stop = "2024-06"
# Variables
variables = ["chlor_a"]
assert set(variables) <= {"chlor_a"} | {f"Rrs_{wl}" for wl in (443, 560)}
# Regions
regions = {
"SO": {
"lon_slice": slice(-180, 180),
"lat_slice": slice(-47.5, -63.5),
},
"NO": {
"lon_slice": slice(-180, 180),
"lat_slice": slice(90, 60),
},
}
for region, slices in regions.items():
# Enforce sorting as original data
for k, v in slices.items():
assert v.start >= v.stop if k == "lat_slice" else v.start <= v.stop, (region, k)
# Define data request
collection_id = "satellite-ocean-colour"
request = {
"projection": "regular_latitude_longitude_grid",
"version": "6_0",
"format": "zip",
}
def regionalising(ds, variable, lon_slice, lat_slice):
da = ds[variable]
da = utils.regionalise(da, lon_slice=lon_slice, lat_slice=lat_slice)
return da.to_dataset(name=variable)
datasets = []
for variable in variables:
for region, slices in regions.items():
print(f"{variable=} {region=}")
requests = download.update_request_date(
request
| {
"variable": "remote_sensing_reflectance"
if variable.startswith("Rrs")
else "mass_concentration_of_chlorophyll_a"
},
start=start,
stop=stop,
stringify_dates=True,
)
ds = download.download_and_transform(
collection_id,
requests,
transform_func=regionalising,
transform_func_kwargs=slices | {"variable": variable},
chunks={"year": 1, "month": 1, "variable": 1},
)
datasets.append(ds.expand_dims(region=[region]))
ds = xr.concat(datasets, "region") |
|
Thanks Mattia! Should I just copy-paste the code you shared in my notebook and run it, right? |
|
That would work, but I used the same code you shared with me (I just cleaned it up a bit). |
|
I just run your code, but when I print ds I only have chlor_a data for the NO region. Could you please check it out? |
|
My code was just a cleaned-up version of your notebook to cache the data. For example, add |
|
I tried ds = xr.merge(datasets) but the kernel restarted. I am running the code again, but the download is stuck at 1%. I guess a lot of people are using the VM at the moment. I'll keep you posted. Thanks again, |
|
Oops my bad, I meant to cache Rrs but I was rerunning the scripts to cache chlor_a. |
|
Should work now. I added the concatenation step in my snippet above. |
|
It works now. Thanks. Could you please cache the total_nobs variable instead of Rrs? I am not sure we will use Rrs in this notebook. |
|
OK |
|
What's Is it one of the variables downloaded with |
|
Yes, It's the total number of chl observations. Ok, we can wait for that if caching them won't take a lot of time |
Notebook description
Analysis of Chlorophyll and remote sensing reflectances in the polar oceans
Notebook link or upload
OC_PolarOceans.ipynb.zip
Anything else we need to know?
Hi Mattia,
I am working on the new Ocean Colour notebook (first time I am using the new VM). I simply modified the previous Notebook (that was working great) and I've changed the regions I want analyze and the spatial mean function since I want to have a complete datasets to perform both spatial and temporal means.
The code seems to be working (no errors), but even if I've started with a short period (i.e., 1 year), it is taking forever to download the data. Am I doing something wrong? Could you please take a look?
Thank you,
Chiara
Environment
<name: wp5
channels:
dependencies:
prefix: /data/common/miniforge3/envs/wp5
The text was updated successfully, but these errors were encountered: