ERROR: Job 'elevation-plots-ms0:: Plotting elevation tracks'

[Sinah-astro]

I'm getting the following error when running my config file:
test277.txt

# 2021-10-11 10:46:17: Initial memory state:
# 2021-10-11 10:46:17:               total        used        free      shared  buff/cache   available
# 2021-10-11 10:46:17: Mem:           503G         34G        2.6G        9.3G        466G        456G
# 2021-10-11 10:46:17: Swap:          476G         11G        465G
# 2021-10-11 10:46:17: Running plot-elevation-tracks.py --output-name /stimela_mount/output/PSZ2G277-elevation-tracks.png /stimela_mount/msdir/PSZ2G277.ms
# Fields in /stimela_mount/msdir/PSZ2G277.ms:
# Traceback (most recent call last):
#   File "/usr/local/bin/plot-elevation-tracks.py", line 60, in <module>
#     field_time, field_radec, obs_xyz, scans = LSTElevationPlot.load_ms_fields(msname, verbose=1)
#   File "/usr/local/lib/python3.6/dist-packages/Owlcat/Plotting.py", line 1167, in load_ms_fields
#     time_ts = np.array([timestamps[row0] for row0 in first_row_per_ts])        # time per timestamp
#   File "/usr/local/lib/python3.6/dist-packages/Owlcat/Plotting.py", line 1167, in <listcomp>
#     time_ts = np.array([timestamps[row0] for row0 in first_row_per_ts])        # time per timestamp
# IndexError: index 2740059 is out of bounds for axis 0 with size 2740059
# 2021-10-11 10:46:21: plot-elevation-tracks.py exited with code 1
2021-10-11 10:46:21 CARACal.Stimela.elevation-plots-ms0 ERROR: cd /home/manaka/PSZ277/.stimela_workdir-16339419745481534 && singularity returns error code 1
2021-10-11 10:46:21 CARACal.Stimela.elevation-plots-ms0 ERROR: job failed at 2021-10-11 10:46:21.746308 after 0:00:07.155402
2021-10-11 10:46:21 CARACal ERROR: Job 'elevation-plots-ms0:: Plotting elevation tracks' failed: cd /home/manaka/PSZ277/.stimela_workdir-16339419745481534 && singularity returns error code 1 [PipelineException]
2021-10-11 10:46:21 CARACal INFO:   More information can be found in the logfile at output/logs-20211011-104611/log-caracal.txt
2021-10-11 10:46:21 CARACal INFO: exiting with error code 1

I thought it may have been that some fields aren't in the MS file, but running listobs on the MS file returns: PSZ2G277-obsinfo.txt

Logfile: log-caracal.txt

[o-smirnov]

Looking at the code, this seems like a logical impossibility. Where's the MS?

[Sinah-astro]

It's in the oates machine -- I'm running caracal from the hall machine

[o-smirnov]

Full path please?

[Sinah-astro]

oates: masters_277/

[Sinah-astro]

oates: masters_277/

oates: masters_277/PSZ2G277.ms

[o-smirnov]

OK, this is a stupid bug in Owlcat. ratt-ru/owlcat#83

It is triggered by an MS that is not sorted in time. So the workaround is to avoid such MSs. How did you end up with this one? Did you concatenate data out of order?

[Sinah-astro]

I may have done that. I thought the concatenation task automatically puts them in order

[Sinah-astro]

Log for the concatenation
concat.log
:

[o-smirnov]

I guess it just sticks them together in whatever order you give.

This is a bug and should be fixed, but the fix will take a bit to propagate. So the fastest way to proceed for you is to re-concatenate in time order.

[Sinah-astro]

I'll try that, thank you

I guess it just sticks them together in whatever order you give.

This is a bug and should be fixed, but the fix will take a bit to propagate. So the fastest way to proceed for you is to re-concatenate in time order.

[o-smirnov]

I'm certain there is other sloppily-written software out there that will break on out-of-order MSs. So it's a good idea anyway to avoid this problem.

[Sinah-astro]

By setting the parameter timesort to True with concat -- hoping that was a good move, it works fine now