statuscodes.TimeExtrapolationError: votemper sampled outside time domain

[cristinarumi]

Question

Question

Hi! The error kernels need to be placed in the end of the kernels, however, this error occurs in the SampleT. How can I solve this problem? Should I move SampleT and SampleS in the end of the Kernels?
I can't set allow_time_extrapolation to True because it is a time_periodic loop.
Thank you very much in advance.
Cristina

Supporting code/error messages

Definition of kernels

def KeepInOcean(particle, fieldset, time): #per les particules que s'escapen per la superficie
    if particle.state == StatusCode.ErrorThroughSurface:
        particle_ddepth = 1.0
        particle.state = StatusCode.Success
        
def DeleteErrorParticle(particle, fieldset, time):
    if (particle.state == StatusCode.ErrorOutOfBounds) or (particle.state == StatusCode.ErrorThroughSurface) or (particle.state == StatusCode.ErrorTimeExtrapolation) or (particle.state == StatusCode.ErrorInterpolation):
        particle.delete()
        
def SampleT(particle, fieldset, time):
    particle.votemper = fieldset.votemper[time, particle.depth, particle.lat, particle.lon]

def SampleS(particle, fieldset, time):
    particle.vosaline = fieldset.vosaline[time, particle.depth, particle.lat, particle.lon]

Kernels and execution

adv       = pset.Kernel(AdvectionRK4) 
sampT     = pset.Kernel(SampleT)
sampS     = pset.Kernel(SampleS)
submerge  = pset.Kernel(KeepInOcean)
deleteparticle = pset.Kernel(DeleteErrorParticle)

kernels=adv+sampT+sampS+submerge+deleteparticle

pset.execute(kernels, runtime=delta(days=365*90), dt=-delta(days=10), output_file=ParticleFile(outdir, pset, outputdt=delta(days=10),chunks=(5000000,1)), verbose_progress=True)

# Paste your error message within this block
File "/media/disk/Cristina/ParcelsRun/1993_2018/parcels/parcels/fieldset.py", line 1171, in computeTimeChunk
    raise TimeExtrapolationError(time, field=f, msg='In fset.computeTimeChunk')
parcels.tools.statuscodes.TimeExtrapolationError: votemper sampled outside time domain at time 1940-12-27T12:00:00.000000000.In fset.computeTimeChunk Try setting allow_time_extrapolation to True
 83%|████████▎ | 2353665600.0/2838240000.0 [165:00:37<33:58:20, 3962.15it/s]

[VeckoTheGecko]

Hi @cristinarumi . It looks like you are running your simulation for longer than you have provided field data for. You can either provide more field data to cover your time domain, or you can enable time extrapolation using the allow_time_extrapolation when you create the fieldset.

This parameter is used in the "Quickstart to Parcels" notebook in this section to work with a static field, but can also be used to extend your time domain at the end.

Your modified code (assuming you're using something like from_netcdf(), where filenames, variables, dimensions represents your existing variables) should look something like this

fieldset = FieldSet.from_netcdf(
    filenames,
    variables,
    dimensions,
+   allow_time_extrapolation=True,
    )

let me know if that helps!

[cristinarumi]

Hi,
Thank you for your answer.
However, this option is not applicable in my situation because I am using time_periodic.
By default, when time_periodic is used, allow_time_extrapolation is set to False.

[erikvansebille]

Dear @cristinarumi, I think this is an issue with your input files. Note that the error is in the fieldset.computeTimeChunk() function, not in the kernels. Hence, the order of the kernels doesn't matter in this case, and deleting particles won't help. I realise that the error message is a bit confusing in that sense

Can you check whether you really have all available timestamps for the temperature field? Looking at the error message (votemper sampled outside time domain at time 1940-12-27T12:00:00), it seems that the temperature field for 27 December 1940 is missing. Is that expected? Or is something wrong with your input data?

[cristinarumi]

Dear Erik,

I am currently using GLORYS12V1 daily data as the input data, covering the period from January 1993 to December 2018. Then, I loop this data to cover 90 years of simulation (365*90) backward in time.

Please see below the full script for the execution:

import matplotlib
import numpy as np
from parcels import FieldSet, Field, ParticleSet, JITParticle, Variable, ParticleFile, Variable, AdvectionRK4_3D
from MyParcels import DeleteErrorParticle, SampleT, SampleS, KeepInOcean
from datetime import datetime, date, timedelta as delta
from os import path
import math
from glob import glob
from scipy.constants import convert_temperature
import pandas as pd
import xarray as xr

particles_file_path="/media/disk/Cristina/ParcelsRun/" #Donde quiero que me guarde el output
data_file_path="/media/disk/Datos/GLORYS_2/" #Carpeta en donde hay las subcarpetas con cada una de las variables

#Asigno nombre del fichero de salida
outdir = particles_file_path+"parcels_run_warm_parallel_alldays_cristina_second_paralel_final.zarr"

####Fichero particulas
data=np.loadtxt(particles_file_path+"release_file_2018_1993.txt", dtype='str')

times=[datetime.strptime(t, '%Y-%m-%d') for t in data[:,0]]#numero de partícules que tirem
lons=data[:,1] #Longitud
lats=data[:,2] #Latitud
deps=data[:,3] #Profundidad
VTS =data[:,4] #Transporte

times=times#[0:5000]
lons=lons#[0:5000]
lats=lats#[0:5000]
deps=deps#[0:5000]
VTS=VTS#[0:5000]

#Crear lista para pillar ficheros cada 10 dias
timei = datetime(1993, 1, 1,12)
timef = datetime(2018, 12,31,12)
deltadays=10 
tdelta = delta(days=deltadays)

datesnum=np.datetime_as_string(np.arange(timei,timef,tdelta), unit='D')
dates=[datesnum[i].replace('-','') for i in range(len(datesnum))]

Ufiles=sorted([data_file_path+"U2/GLORYS12V1-gridU-"+i+".nc" for i in dates])
Vfiles=sorted([data_file_path+"V/GLORYS12V1-gridV-"+i+".nc" for i in dates])
Wfiles=sorted([data_file_path+"W/GLORYS12V1-gridW-"+i+".nc" for i in dates])
Tfiles=sorted([data_file_path+"T/GLORYS12V1-gridT-"+i+".nc" for i in dates])
Sfiles=sorted([data_file_path+"S/GLORYS12V1-gridS-"+i+".nc" for i in dates])

mesh_mask = data_file_path+"GLORYS12V1-mesh_mask.nc"

filenames = {'U': {'lon':mesh_mask, 'lat':mesh_mask, 'depth':Wfiles[0],'time':Ufiles, 'data':Ufiles}, 
             'V': {'lon':mesh_mask, 'lat':mesh_mask, 'depth':Wfiles[0],'time':Vfiles, 'data':Vfiles}, 
             'W': {'lon':mesh_mask, 'lat':mesh_mask, 'depth':Wfiles[0],'time':Wfiles, 'data':Wfiles},
             'votemper': {'lon':mesh_mask, 'lat':mesh_mask, 'depth':Wfiles[0],'time':Tfiles, 'data':Tfiles},
             'vosaline': {'lon':mesh_mask, 'lat':mesh_mask, 'depth':Wfiles[0],'time':Sfiles, 'data':Sfiles}}

variables   = {'U': 'vozocrtx',
               'V': 'vomecrty',
               'W': 'vovecrtz',
               'votemper': 'votemper',
               'vosaline': 'vosaline'}

dimensions  = {'U': {'lon':'glamf','lat':'gphif','depth':'depthw','time':'time_counter'},
               'V': {'lon':'glamf','lat':'gphif','depth':'depthw','time':'time_counter'}, 
               'W': {'lon':'glamf','lat':'gphif','depth':'depthw','time':'time_counter'},
               'votemper': {'lon':'glamt','lat':'gphit','depth':'depthw','time':'time_counter'},
               'vosaline': {'lon':'glamt','lat':'gphit','depth':'depthw','time':'time_counter'}}

cs = {"time": ("time", 1), "depth": ("depth", 1), "lat": ("latitude", 128), "lon": ("longitude", 128)}

fieldset = FieldSet.from_nemo(filenames, variables, dimensions, allow_time_extrapolation=False, **time_periodic=9496*86400**,chunksize=cs)

timest_out   =  10 
timest_calc  =  10 

class sParticle(JITParticle):
        VT = Variable('VT', initial=0, dtype=np.float32, to_write='once')
        votemper  = Variable( 'votemper', dtype=np.float32)
        vosaline  = Variable( 'vosaline', dtype=np.float32)
        
def simple_partition_function(coords, mpi_size=8):
    """A very simple partition function
    #that assigns particles to processors
    #"""
    return np.linspace(0, mpi_size, coords.shape[0], endpoint=False, dtype=np.int32)

pset    = ParticleSet(fieldset=fieldset, pclass=sParticle, lon=lons.astype(float), lat=lats.astype(float), depth=deps.astype(float), time=times, VT=VTS.astype(float),partition_function=simple_partition_function) 

#Kernels
adv       = pset.Kernel(AdvectionRK4_3D)
sampT     = pset.Kernel(SampleT)
sampS     = pset.Kernel(SampleS)
submerge  = pset.Kernel(KeepInOcean)
deleteparticle = pset.Kernel(DeleteErrorParticle)

kernels=adv+sampT+sampS+submerge+deleteparticle

pset.execute(kernels, runtime=delta(days=365*90), dt=-delta(days=10), output_file=ParticleFile(outdir, pset, outputdt=delta(days=10),chunks=(5000000,1)), verbose_progress=True)

print('done')

Thank you very much in advance for your assistance.

[erikvansebille]

Thanks for providing this code, @cristinarumi. I had a look through it and nothing immediately jumps out as wrong.

Can you calculate what's so special about the date that the run crashed, on 1940-12-27? Was that exactly at the boundary of the time domain, so when the time_periodic had to pick up?

And have you rerun the simulation? It could be(?) that the filesystem just had a hiccup and couldn't access that one specific temperature file because it was used by e.g. another process? Sometimes these kinds of errors magically disappear when you rerun

If a rerun doesn't help, you could test what happens if you don't use temperature. So not load them and not sample it. Does the run then succeed?

These kinds of steps are the best way to dig down and find out what is going wrong...

[cristinarumi]

Hi Erik, Thank you very much for your suggestions.

I've confirmed that the data file is fine. However, after rerunning the simulation, it still fails at the exact same point (1940-12-27).

What is special about December 1940 is that: The backpath simulation, starting in 2018, runs for 78 years completing 3 loops (26 years*3, from 2018 to 1941), but it breaks during the initialization of the fourth loop (starting in December 1940).

The time_periodic=9496*86400 represents the number of days (9496) between the first and last day of the data set (01-01-1993 to 31-12-2018, which is 26 years) converted into seconds (*86400). However, since my input data does not include leap years, the actual number of days of the data set is 9490. Do you think this discrepancy could be causing the issue?

[erikvansebille]

Yes, the leap year could very well be it. What if you set the time-periodic to lees than 26 years; e.g. only 25 years? Does that help?

[cristinarumi]

Hi Erik,
When using a time_periodic corresponding to 25 years, instead of 26 years, the following error occurs: ValueError: Time series provided is longer than the time_periodic parameter

[erikvansebille]

Hmm, I didn't think about that.

Sorry that this is somewhat messy; we hardly use the time_periodic function in my own team, so it hasn't been real-life-tested as much as most of the other features.

I'm not sure how I can help you. I don't know of an (easy) solution to your problem. The only alternative route I can think of is that you copy the netcdf files yourself, changing the dates so that you extend the length of the time series available. But that would take up four times more storage, so may not be feasible