Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Multi model columns #76

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

WIP: Multi model columns #76

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
e44846d
support multi-column expression and depency on daskms master
smasoka Oct 30, 2020
5050a43
remove sothing.py
smasoka Oct 30, 2020
d18ccbe
1. update argument help messages
smasoka Nov 5, 2020
155c244
supports assignment (=) in expression
smasoka Jan 6, 2021
918a366
fix logic
smasoka Jan 7, 2021
0bce875
Fix infinite recursion when trying to determine blockwise meta
sjperkins Jan 8, 2021
f961d61
adds test for expression and strip data_column
smasoka Jan 18, 2021
5c38afc
flake8 fixup
smasoka Jan 18, 2021
6725ad1
Rework with upstream dask changes
sjperkins Feb 11, 2021
2813d09
Merge remote-tracking branch 'upstream/master' into multi-model_columns
sjperkins Feb 11, 2021
48f8e73
rework test fixtures so that download only occurs once
sjperkins Feb 11, 2021
01263b3
try the column case before the expression case
sjperkins Feb 11, 2021
86eeefb
provoke a test run
sjperkins Feb 11, 2021
de75c38
Revert "provoke a test run"
sjperkins Feb 11, 2021
e3195d6
flake8
sjperkins Feb 11, 2021
9af1b4e
Update help
sjperkins Feb 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions HISTORY.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ History
X.Y.Z (YYYY-MM-DD)
------------------

* Add multi model column expression (:pr:`76`)
* Removed .travis and .travis.yml (:pr:`71`) and (:pr:`73`)
* Github CI Actions (:pr:`71`)

Expand Down
5 changes: 4 additions & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,16 @@
readme = readme_file.read()

requirements = [
'dask-ms'
'@git+https://github.com/ska-sa/dask-ms.git'
'@master',
'dask[array] >= 2.2.0',
'donfig >= 0.4.0',
'numpy >= 1.14.0',
'numba >= 0.43.0',
'scipy >= 1.2.0',
'threadpoolctl >= 1.0.0',
'dask-ms >= 0.2.3',
# 'dask-ms >= 0.2.3',
'zarr >= 2.3.1'
]

Expand Down
35 changes: 18 additions & 17 deletions tricolour/apps/tricolour/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-

""" Main tricolour application """

import warnings
import re
import argparse
import contextlib
Expand All @@ -22,6 +22,7 @@
CacheProfiler, visualize)
import numpy as np
from daskms import xds_from_ms, xds_from_table, xds_to_table
from daskms.expressions import data_column_expr
from threadpoolctl import threadpool_limits

from tricolour.apps.tricolour.strat_executor import StrategyExecutor
Expand Down Expand Up @@ -183,7 +184,10 @@ def create_parser():
help="Number of channels to dilate as int "
"or string with units")
p.add_argument("-dc", "--data-column", type=str, default="DATA",
help="Name of visibility data column to flag")
help="Name of visibility data column to flag."
"Now supports multi model columns expressions"
"e.g 'DATA / (DIR1_DATA + DIR2_DATA + DIR3_DATA)'"
"In future will replace subtract-model-column")
smasoka marked this conversation as resolved.
Show resolved Hide resolved
p.add_argument("-fn", "--field-names", type=str, action='append',
default=[],
help="Name(s) of fields to flag. Defaults to flagging all")
Expand Down Expand Up @@ -211,8 +215,8 @@ def create_parser():
p.add_argument("-smc", "--subtract-model-column", default=None, type=str,
help="Subtracts specified column from data column "
"specified. "
"Flagging will proceed on residual "
"data.")
"Flagging will proceed on residual data."
"Depreciating argurment. See --data-column")
smasoka marked this conversation as resolved.
Show resolved Hide resolved
return p


Expand Down Expand Up @@ -268,8 +272,11 @@ def _main(args):
"Interactive Python Debugger, as per user request")
post_mortem_handler.disable_pdb_on_error()

log.info("Flagging on the {0:s} column".format(args.data_column))
data_column = args.data_column
# extract the name of the visibility column
# to support subtract_model_column
data_column = re.split(r'\+|-|/|\*|\(|\)', args.data_column)[0]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a regex designed to detect the kind of expressions that https://github.com/ska-sa/dask-ms/blob/master/daskms/expressions.py#L103 supports. I think it's probably better to just split on the equals as follows as its easer to separate into the two cases

parts = args.data_column.split("=")
log.info("Flagging on the {0:s} {0:s}".format(parts[0], "column" if len(parts) == 1 else "expression))

Additionally, I don't think the regex contained an equals sign so I think it would have failed on an assignment statement.

Could you please correct and test the two cases?

  1. The simple case where data_column just references a column name
  2. The complicated case where data_column contains an assignment.

log.info("Flagging on the {0:s} column".format(data_column))

masked_channels = [load_mask(fn, dilate=args.dilate_masks)
for fn in collect_masks()]
GD = args.config
Expand All @@ -281,22 +288,14 @@ def _main(args):
# Index datasets by these columns
index_cols = ['TIME']

# Reopen the datasets using the aggregated row ordering
columns = [data_column,
"FLAG",
"TIME",
"ANTENNA1",
"ANTENNA2"]

if args.subtract_model_column is not None:
columns.append(args.subtract_model_column)

xds = list(xds_from_ms(args.ms,
columns=tuple(columns),
group_cols=group_cols,
index_cols=index_cols,
chunks={"row": args.row_chunks}))

string = "EXPR = " + args.data_column
vis = data_column_expr(string, xds)

# Get support tables
st = support_tables(args.ms)
ddid_ds = st["DATA_DESCRIPTION"]
Expand Down Expand Up @@ -376,6 +375,8 @@ def _main(args):
# Visibilities from the dataset
vis = getattr(ds, data_column).data
if args.subtract_model_column is not None:
warnings.warn("-smc arg is depreciating."
"See -dc arg for expressions")
smasoka marked this conversation as resolved.
Show resolved Hide resolved
log.info("Forming residual data between '{0:s}' and "
"'{1:s}' for flagging.".format(
data_column, args.subtract_model_column))
Expand Down