"ValueError: malformed node or string" when reading source list

[o-smirnov]

Lucky me ended up with a non-parseable source list:

# running /home/oms/.venv/stimela/bin/crystalball --output-column DIR1_DATA --within dd.reg --num-sources 1000 --sky-model im4/im4-s08-sources.txt ../msdir/1608538564_sdp_l0-Jupiter-scan08.ms
# 2021-05-17 12:45:57 | INFO     | crystalball:_predict | Crystalball version 0.3.0
# 2021-05-17 12:45:57 | INFO     | region:load_regions | Read 1 inclusion region(s) from dd.reg
# Traceback (most recent call last):
#   File "/home/oms/.venv/stimela/bin/crystalball", line 33, in <module>
#     sys.exit(load_entry_point('crystalball', 'console_scripts', 'crystalball')())
#   File "/home/oms/projects/crystalball/crystalball/crystalball.py", line 155, in predict
#     return _predict(args)
#   File "/home/oms/projects/crystalball/crystalball/crystalball.py", line 172, in _predict
#     num=args.num_sources or None)
#   File "/home/oms/projects/crystalball/crystalball/wsclean.py", line 62, in import_from_wsclean
#     in load(wsclean_comp_list)}
#   File "/home/oms/.venv/stimela/lib/python3.6/site-packages/africanus/model/wsclean/file_model.py", line 208, in load
#     return _parse_lines(fh, line_nr, column_names, defaults, converters)
#   File "/home/oms/.venv/stimela/lib/python3.6/site-packages/africanus/model/wsclean/file_model.py", line 139, in _parse_lines
#     data_list.append(conv(value))
#   File "/usr/lib/python3.6/ast.py", line 85, in literal_eval
#     return _convert(node_or_string)
#   File "/usr/lib/python3.6/ast.py", line 61, in _convert
#     return list(map(_convert, node.elts))
#   File "/usr/lib/python3.6/ast.py", line 70, in _convert
#     operand = _convert(node.operand)
#   File "/usr/lib/python3.6/ast.py", line 84, in _convert
#     raise ValueError('malformed node or string: ' + repr(node))
# ValueError: malformed node or string: <_ast.Name object at 0x7fcd58f2e6a0>

@sjperkins you can get the source list at /net/muddy/home/oms/projects/Jove/selfcal2/im4/im4-s08-sources.txt (from any SARAO cluster node).

[o-smirnov]

Hmm actually, looks like it's because some of those sources were dumped as NANs:

s0c4162,POINT,-03:50:36.611,-20.37.04.578,-nan,[-nan,-nan,-nan],false,1337395507.8125,,,

Removing the nan-lines makes the list readable.

Maybe this is a wsclean bug (@francescaLoi and myself are currently discussing it with Andre), but it would be nice to add some robustness against it, particularly if Andre insists this sort of thing is legit...

[sjperkins]

Yep, this looks like it would need a fix in codex-africanus. I don't think I can transfer the issue over, so I'll just cut and paste your report over there.

[sjperkins]

While the change itself is not difficult, there are a number of steps that will need to take place.

codex-africanus has now moved forward to 3.7 as the minimum python version in ratt-ru/codex-africanus#248. Any fix for ratt-ru/codex-africanus#249 will require a codex release, as well as a Python 3.7 crystalball release.

None of these steps are difficult, but devs/users should be aware of the Python 3.7 upgrade requirement.