Batch sampling on layer el10978 does not work on Sampling-service

[qifeng-bai]

Problem:
this link returns a correct value of a lat/lng from the layer el10978 -WorldClim 2.1: Temperature - annual mean
https://sampling.ala.org.au/sampling-service//intersect/el10978/-28.54999999920733/138.1

field: "el10978",
layername: "WorldClim 2.1: Temperature - annual mean",
units: "°C",
value: 21.833334

However, when we send a batch request:
https://sampling.ala.org.au/sampling-service/intersect/batch?fids=el10978&points=-28.54999999920733,138.1,38.05829999908396,145.325,-29.9,116.2,-37.67537,145.0815
No values were returned

Checked the Nginx settings on Sampling-service
We found:

location /sampling-service/intersect/batch {
              proxy_pass http://127.0.0.1:8080/sampling-service/intersect/batch;
}
location /sampling-service/ {
        proxy_pass https://spatial.ala.org.au/ws/;
}

The batch request runs on local and the others are forwarded to spatial service

We tried to send the same batch request to Spatial service directly
https://spatial.ala.org.au/ws/intersect/batch?fids=el10978&points=-28.54999999920733,138.1,38.05829999908396,145.325,-29.9,116.2,-37.67537,145.0815

It returned correct results:
latitude,longitude,el10978

-28.54999999920733,138.1,21.833334
38.05829999908396,145.325,
-29.9,116.2,19.554167
-37.67537,145.0815,14.6625

It is reasonable to believe the sampling-service and spatial-service are sharing the same layer information

BUGS
Reviewed the Layers-store code, we found if a layer file is not found on the server, layers-store does not log any error information and just return null value - It has some codes to check if the layer exists, but the code does not work

Checking layers on both servers
We found
There are 19 worldclim21_bio grid files - from worldclim21_bio1 to worldclim21_bio19 on Spatial-service, e.g. el10978 is worldclim21_bio1.gri , size is 3.5G

However, this is only 1 worldclim21_bio grid file on sampling-service, worldclim21_bio1.gri, size is 1.1G ?incorrect size?
(For testing purpose, we moved this worldclim21_bio1.gri to a backup folder)

After, worldclim21_bio1.grd and .gri (size~3.5) copied to sampling service, the same request returns:

latitude,longitude,el10978
-28.54999999920733,138.1,21.833334
38.05829999908396,145.325,
-29.9,116.2,19.554167
-37.67537,145.0815,14.6625

which is the same as the result from Spatial service

Questions:

The link https://spatial.ala.org.au/ws/layers.json returns 154 layers. And we can find 437 layers via `https://spatial.ala.org.au/ws/manageLayers/layers'

Checking 154 layers listed onSpatial, we have found 102 missing layers on sampling service

[qifeng-bai]

Layer filed stored at : /data/ala/data/layers/ready/layer

https://spatial.ala.org.au/ws/intersect/batch?fids=el10978&points=-65.2443,120.0708,-13.650000003590685,143.71000000000004,-29.99333,153.425

https://sampling.ala.org.au/sampling-service//intersect/el10978/-65.2443/120.0708

Those query have the same results / empty results. The results should be correct since those coordinates are in the ocean.