Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dask version #252

Open
wants to merge 416 commits into
base: master
Choose a base branch
from
Open

Dask version #252

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
416 commits
Select commit Hold shift + click to select a range
b9a1cab
Make antenna2 tiling the same as antenna1 tiling
sjperkins Oct 25, 2017
7c0e87a
Create time + time_index arrays with dask
sjperkins Oct 25, 2017
a9296f5
Make beam_extents a dask array too
sjperkins Oct 25, 2017
24c9b47
Use time.time() instead of time.clock()
sjperkins Oct 25, 2017
c72ec25
Use recent tensorflow linking mechanism
sjperkins Oct 25, 2017
4da4d2c
Upgrade to tensorflow 1.4.0 (#227)
sjperkins Nov 3, 2017
9acf3d9
Support distributed scheduler file
sjperkins Oct 27, 2017
97952b1
Pin python-casacore == 2.1.2
sjperkins Dec 6, 2017
b5359e5
Default device should be CPU
sjperkins Dec 6, 2017
ac0c8d4
Strip out old cruft
sjperkins Jan 8, 2018
cb1cf67
Remove unused time_offsets array
sjperkins Jan 8, 2018
ec67b9e
Rename row dimension to vrow
sjperkins Jan 8, 2018
1b42247
Rename time_chunks to time_vrow_chunks
sjperkins Jan 8, 2018
c336d57
[WIP] Introduce antenna row 'arow' dimension
sjperkins Jan 9, 2018
0c45271
Use da.zeros instead of da.full to create flags
sjperkins Jan 9, 2018
e498ed5
WIP
sjperkins Jan 12, 2018
c1017c8
Use versioneer for versioning (#229)
sjperkins Nov 9, 2017
834df57
Fix various setup py issues (#231)
sjperkins Nov 13, 2017
42fda29
Relax hard requirement on python-casacore 2.1.2 (#235)
sjperkins Dec 15, 2017
890fdc7
Covert ebeam kernel to antenna row
sjperkins Feb 23, 2018
3e410f7
Allow configurable rime tensorflow library path
sjperkins Feb 23, 2018
7082d3d
Convert feed rotation kernels to antenna row
sjperkins Feb 23, 2018
39dda58
Convert create_antenna_jones to antenna row
sjperkins Feb 23, 2018
c6b45bc
Convert parallactic angle sincos to antenna row
sjperkins Feb 23, 2018
85bf2d7
Convert complex phase to antenna row
sjperkins Feb 23, 2018
16479e6
Fix e beam kernel block dimensions
sjperkins Feb 23, 2018
fa10e07
Rewrite test_phase.py as a unittest
sjperkins Feb 23, 2018
0394ff3
Remove unnecessary test case
sjperkins Feb 26, 2018
afb65fe
Update to latest dask and distributed
sjperkins Feb 26, 2018
2134a6b
Depend on tensorflow 1.6.0rc1
sjperkins Feb 26, 2018
cca3e69
Remove old __version__ import
sjperkins Feb 26, 2018
ff76c39
Correct library import in tensorflow_ops_ext.py
sjperkins Feb 26, 2018
dc75771
Allow missing terms in create_antenna_jones
sjperkins Feb 27, 2018
b9264dc
Handle some jones term test case permutations
sjperkins Feb 28, 2018
c639ba1
Handle matrix initialisiation in antenna jones
sjperkins Mar 14, 2018
4d274e5
Support complex phase in sum coherencies kernels
sjperkins Mar 14, 2018
02ab82d
Use nullptr to indicate data in CUDA kernels
sjperkins Mar 14, 2018
697d9be
Upgrade to distributed 1.21.3
sjperkins Mar 15, 2018
5760ab6
Fix indexing bug in create_antenna_jones
sjperkins Mar 15, 2018
593a613
Add missing all terms permutation
sjperkins Mar 15, 2018
1a611f1
Update Makefile for later tensorflow versions
sjperkins Mar 16, 2018
c3ba0c6
Depend on tensorflow 1.7.0rc0
sjperkins Mar 16, 2018
9067fae
First stab at queued dataset
sjperkins Mar 20, 2018
f6583a1
Rework QueueDataset
sjperkins Mar 22, 2018
488c5b6
Fix race conditions in QueueResource
sjperkins Mar 22, 2018
0ab6681
Fix Dataset and Iterator accessibility + reindent
sjperkins Mar 22, 2018
2be60f6
Turn on NDEBUG to handle refcount issues
sjperkins Mar 23, 2018
42ac72a
Enhance Dataset queue test case
sjperkins Mar 23, 2018
3415aa6
Upgrade to tensorflow 1.7.0rc1
sjperkins Mar 23, 2018
14d646e
Support op names
sjperkins Mar 23, 2018
dbbca6e
Wrap tensorflow library in tensorflow_ops.py
sjperkins Mar 23, 2018
7ddbcd0
Support nested structures in TensorQueue
sjperkins Mar 23, 2018
bf4538d
Handle numpy types in put operations
sjperkins Mar 23, 2018
4595391
Create Queue Dataset for fed inputs
sjperkins Mar 26, 2018
e8e320e
Documentation
sjperkins Mar 26, 2018
43ac1bd
Add a MapDataset
sjperkins Mar 27, 2018
0ca0fe0
Touch up and enable other test cases
sjperkins Mar 28, 2018
60d5704
Fix not meant to be included change
sjperkins Mar 28, 2018
97d3996
Depend on tensorflow 1.7.0 full release
sjperkins Apr 3, 2018
7f6d317
Use MapStagingArea for sources instead of Datasets
sjperkins Apr 4, 2018
182b628
Add internal schema
sjperkins Apr 4, 2018
48debe0
Add internal schema arrays to feed_multiple
sjperkins Apr 4, 2018
4e9e791
Basic loop working
sjperkins Apr 4, 2018
1bd66c5
prefetch_to_device TODO comment
sjperkins Apr 5, 2018
01bf4a6
Make internal keys look python private
sjperkins Apr 5, 2018
306936b
Revert "Convert feed rotation kernels to antenna row"
sjperkins Apr 5, 2018
4622068
Revert "Convert parallactic angle sincos to antenna row"
sjperkins Apr 5, 2018
4f17a4e
Revert "Convert complex phase to antenna row"
sjperkins Apr 5, 2018
3d83d64
Revert "Covert ebeam kernel to antenna row"
sjperkins Apr 5, 2018
683882b
Revert "Convert create_antenna_jones to antenna row"
sjperkins Apr 5, 2018
c63d040
Support row-based from of complex phase
sjperkins Apr 6, 2018
2f03818
Fixes
sjperkins Apr 6, 2018
f168870
Reintroduce complex phase tensorflow expression
sjperkins Apr 9, 2018
013c109
Update to tensorflow 1.8.0
sjperkins May 11, 2018
8f7df8b
Add C++ shape schema parsing utility function
sjperkins May 17, 2018
9e2e0f0
Generic tensorflow op export and construction
sjperkins May 17, 2018
43f79a8
Fix arg_type/dtype usage
sjperkins May 18, 2018
806fa01
Add a PlaceholderContext
sjperkins May 18, 2018
5b0cbdc
Warn if placeholder store fails
sjperkins May 21, 2018
ba8883c
Use AST to wrap tensorflow calls
sjperkins May 21, 2018
3370658
Yet another experiment...
sjperkins May 23, 2018
ff3f3c9
Include function object in op_defs
sjperkins May 31, 2018
a48fa21
Add a dataset in tensorflow while loop test
sjperkins May 31, 2018
d5aaa62
Add missing map_dataset.py file
sjperkins May 31, 2018
0bceee5
Add array schemas to the phase operator
sjperkins May 31, 2018
f95f479
tensorflow function analysis code using mock
sjperkins May 31, 2018
65ea978
Remove redundant function analysis code
sjperkins May 31, 2018
ffae5fc
Maintain a set of ops required the input
sjperkins Jun 1, 2018
7e51349
Strip dimension strings
sjperkins Jun 1, 2018
cea2d3e
Do integer conversion in shape parsing
sjperkins Jun 4, 2018
4574972
Fix schema assignment
sjperkins Jun 4, 2018
89b8368
Add shape schemas to op inputs
sjperkins Jun 4, 2018
60e082a
Assign placeholder to individual datasets
sjperkins Jun 4, 2018
71c9974
Fix incorrect schemas
sjperkins Jun 4, 2018
5aea242
Derive concrete input datasets from tensorflow function inputs
sjperkins Jun 6, 2018
ad901bb
Add size ops to Queue and Map Datasets
sjperkins Jun 6, 2018
2aa2080
Add python size methods
sjperkins Jun 6, 2018
1a2c4cb
Remove unused commented out imports
sjperkins Jun 6, 2018
c657f31
Mock tensorflow datasets and iterators
sjperkins Jun 7, 2018
2e7ae33
Supply cfg and device as function args
sjperkins Jun 7, 2018
2514229
Map dataset fixes
sjperkins Jun 7, 2018
9d9c96d
WIP
sjperkins Jun 11, 2018
3bc383f
xarray_ms package name change
sjperkins Jun 11, 2018
1079f19
Queue for the main dataset, Maps for the sources
sjperkins Jun 11, 2018
f2dca59
Add functions for extracting shapes from schemas
sjperkins Jun 12, 2018
abad1ca
Support empty input lists in create_antenna_jones
sjperkins Jun 15, 2018
3c5ec7d
Separate types for each CreateAntennaJones input
sjperkins Jun 15, 2018
18a72ec
Make dimension inference more robust
sjperkins Jun 15, 2018
fb59885
Introduce TensorflowInputFacade classes
sjperkins Jun 15, 2018
fddb388
Remove cruft and supply shape inference handles
sjperkins Jun 15, 2018
4b5e645
Upgrade to tensorflow 1.9.0rc1
sjperkins Jun 17, 2018
abd5f0e
Fix subtle integer precision bug
sjperkins Jun 18, 2018
5999897
import prefetch_to_device
sjperkins Jun 18, 2018
a323e3a
Remove unused function
sjperkins Jun 18, 2018
bbca3a5
return mock in FakeIterator.initializer
sjperkins Jun 18, 2018
be49cf0
Further input shape handling simplification
sjperkins Jun 23, 2018
fb416de
Modify CreateAntennaJones
sjperkins Jun 23, 2018
95a6589
Modify SumCoherencies kernel dims
sjperkins Jun 23, 2018
772262d
Optional inputs to SumCoherencies
sjperkins Jun 23, 2018
4afe440
Add a basic/simple expression of the RIME
sjperkins Jun 23, 2018
d14cf3c
Add a DDE version of the RIME
sjperkins Jun 23, 2018
9b0aa3d
Comment out failing expression phase test
sjperkins Jun 23, 2018
10bd1d2
OpKernel::Compute is not threadsafe
sjperkins Jun 28, 2018
33d62d5
Dataset API
sjperkins Jul 2, 2018
1d7ec14
Add antenna dimension inspection
sjperkins Jul 2, 2018
4f5b64e
import MeasurementSet in package
sjperkins Jul 3, 2018
c5ebcc3
Return placeholder + dataset info from analysis fn
sjperkins Jul 3, 2018
a906b4d
Add budgeting code
sjperkins Jul 3, 2018
772eae8
Don't use dim sizes for reductions for no fit case
sjperkins Jul 3, 2018
7731bce
Make QueueDataset more deterministic
sjperkins Jul 4, 2018
6d62ae8
Register iterator in QueueResource instead
sjperkins Jul 4, 2018
12d81ff
Map Iterator registration + stash support
sjperkins Jul 5, 2018
1fe761f
Remove dangling print statement
sjperkins Jul 5, 2018
8746099
Allow TensorMap to behave as a store
sjperkins Jul 6, 2018
704bda3
Test TensorMap key clearing mechanisms
sjperkins Jul 6, 2018
8c2ad21
Explicitly reference the log level
sjperkins Jul 6, 2018
95e2a4e
pep8
sjperkins Jul 6, 2018
ed274bd
Remove missing variables
sjperkins Jul 6, 2018
eaeff05
Remove comments, pep8
sjperkins Jul 6, 2018
5810468
Upgrade NVIDIA cub to 1.8.0
sjperkins Jul 6, 2018
32847c3
Move tests to test directory
sjperkins Jul 10, 2018
976e2da
Move tests to test directory
sjperkins Jul 10, 2018
76532ec
Add a tensorflow session wrapper
sjperkins Jul 11, 2018
70ede18
Test both basic and dde RIME expressions
sjperkins Jul 11, 2018
b49b883
Remove old session cache
sjperkins Jul 11, 2018
88a3029
Place global init under graph context and finalize
sjperkins Jul 11, 2018
38c016f
Add a __del__ to the wrapper class
sjperkins Jul 11, 2018
5057b55
Upgrade to tensorflow 1.9.0
sjperkins Jul 11, 2018
408b079
Get the appropriate device list
sjperkins Jul 11, 2018
67805ba
Don't use defaultdict for device_map
sjperkins Jul 11, 2018
5af7531
Add utility methods to session wrapper
sjperkins Jul 12, 2018
ef61bca
Add pop capabilities to the TensorMap
sjperkins Jul 13, 2018
4fd13f7
Remove cruft
sjperkins Jul 13, 2018
d0ff536
Add enqueueing + expr functions to wrapper
sjperkins Jul 17, 2018
4145df9
Fix wrapper size evaluation functions
sjperkins Jul 17, 2018
e32b69a
Add test for context
sjperkins Jul 17, 2018
c7d41de
Test data enqueueing, expr evaluation and cleanup
sjperkins Jul 17, 2018
fc7cc7c
Small fixes
sjperkins Jul 17, 2018
a4b088c
Turn configuration into a fixture
sjperkins Jul 17, 2018
2fe36f4
Close Queues and Maps
sjperkins Jul 17, 2018
916d36f
Rename test case
sjperkins Jul 17, 2018
3f54107
Place each shard in a separate name scope
sjperkins Jul 17, 2018
a11ca49
Add more dataset attributes
sjperkins Jul 17, 2018
4820dfe
Uniquely identify each chunk of input data
sjperkins Jul 17, 2018
c3dcf01
Remove crufty old imports
sjperkins Jul 17, 2018
f11d5af
Continuously evaluate expressions in a thread
sjperkins Jul 18, 2018
2a82d55
Maintain global and iterator init ops separately
sjperkins Jul 18, 2018
1a569ce
Infer output schemas
sjperkins Jul 18, 2018
9d4af4f
Convert floating point templates into actual types
sjperkins Jul 19, 2018
8d0bafa
Place compute result in output map
sjperkins Jul 19, 2018
8a0ad85
Make data enqueueing more generic
sjperkins Jul 19, 2018
0044ff8
Retrieve data and clear out maps in one go
sjperkins Jul 19, 2018
3aa74f3
pep8
sjperkins Aug 15, 2018
b96be62
Use a Daemon thread
sjperkins Aug 15, 2018
f02551e
Use contexts for Session Wrappers in tests
sjperkins Aug 15, 2018
151cac1
pep8
sjperkins Aug 16, 2018
6ed5404
Upgrade to tensorflow 1.10.0
sjperkins Aug 16, 2018
033fca4
Remove old dataset.py module
sjperkins Aug 16, 2018
8cb7f77
_depends_on_input_ds => _requires_input_ds
sjperkins Aug 16, 2018
4ed0a74
pep8
sjperkins Aug 16, 2018
4cb002c
mock.MagicMock doesn't care about devices
sjperkins Aug 17, 2018
1899bd9
pep8
sjperkins Aug 20, 2018
4bd9f6d
Fixes for cub 1.8.0
sjperkins Aug 20, 2018
55e7790
Create brightness matrix operator
sjperkins Aug 20, 2018
6fe6f8c
Support arbitrary input dims in phase op
sjperkins Aug 21, 2018
71fedf0
Correct op import in test_brightness.py
sjperkins Aug 21, 2018
1600591
Remove unused op templates
sjperkins Aug 21, 2018
c3dc182
The great 2018 SumCoherencies rewrite
sjperkins Aug 22, 2018
9e0c00b
Fix occasional deadlock
sjperkins Aug 27, 2018
e2d7e8a
Generic Jones Multiply Operation
sjperkins Aug 29, 2018
c68cbb6
Jones Multiply shape inference and function unify
sjperkins Aug 30, 2018
c2cb233
Optionally squeeze dims and custom output schemas
sjperkins Sep 4, 2018
d5c0728
Set a default schema for the brightness operator
sjperkins Sep 12, 2018
9caccac
Support single correlation dimensions.
sjperkins Sep 12, 2018
220fa60
Print inputs
sjperkins Sep 12, 2018
b913474
Simplify SumCoherencies
sjperkins Sep 13, 2018
bba5566
Handle singleton outputs during Mock Analysis
sjperkins Sep 14, 2018
d25ae03
Modify scripts -- jones_multiply + sum_coherencies
sjperkins Sep 14, 2018
e3daa3f
Better error messages
sjperkins Sep 17, 2018
044b34a
pep8
sjperkins Sep 17, 2018
2c34ee2
Remove unnecessary antenna_jones function
sjperkins Sep 17, 2018
98bf2f0
Convert outputs to a dict
sjperkins Sep 17, 2018
0f1e8d9
Parse the output shape schema
sjperkins Sep 17, 2018
af7e3cd
Test with random complex values+fix some breakage
sjperkins Sep 28, 2018
2d70fa9
Further GPU fixes for jones_multiply
sjperkins Oct 1, 2018
bd5330d
Fix scalar case
sjperkins Oct 2, 2018
82b17cd
Remove unused import
sjperkins Oct 3, 2018
32cfd0b
Setup dask inputs and top call for rime function
sjperkins Oct 3, 2018
3a6543d
Associate row and time dimensions
sjperkins Oct 3, 2018
b184238
Fill out dask input test case
sjperkins Oct 4, 2018
e80be56
Also test pooled key release
sjperkins Oct 4, 2018
4483822
Handle ndarray case for source inputs
sjperkins Oct 4, 2018
863df3e
Handle multiple outputs in the rime function
sjperkins Oct 4, 2018
50fff5b
Refactor reshape and chunk computation
sjperkins Oct 4, 2018
3e76084
Add missing output placeholders
sjperkins Oct 5, 2018
f88e372
Handle DDE rime in dask test case
sjperkins Oct 5, 2018
3c7d8e2
Fix single ndarray case
sjperkins Oct 5, 2018
29bda09
Decorate RIME source functions
sjperkins Oct 8, 2018
79a78c3
source_decorator -> source_context
sjperkins Oct 8, 2018
9017a12
Improve whitespace
sjperkins Oct 8, 2018
3261196
Zernike Direction Dependent Effects Tensorflow Operator (#259)
sjperkins Nov 6, 2018
78fc80e
Upgrade to tensorflow 1.12.0
sjperkins Nov 6, 2018
34ed6fe
Re-use tensorflow install message
sjperkins Nov 6, 2018
d3aee26
Support naming FakeMapDataset
sjperkins Nov 9, 2018
3f245b9
Create separate EvaluationThread class
sjperkins Nov 9, 2018
89308bf
Commit cuda kernel error checks
sjperkins Nov 9, 2018
7f994e1
Rework GaussShapeOp to work on row-based UVW
sjperkins Nov 12, 2018
0095cbe
Pin dataset creation to CPU
sjperkins Nov 12, 2018
f5a860c
Fix Zernike test cases
sjperkins Nov 12, 2018
9c8f4aa
Rework zernike polynomial test case tolerances
sjperkins Nov 12, 2018
ceb5986
Rework SersicShapeOp to row-based UVW
sjperkins Nov 12, 2018
04ec54f
Add CUDA check to ParallacticAngleSinCos
sjperkins Nov 12, 2018
d6f00dc
cuda checks for PhaseOp
sjperkins Nov 12, 2018
54a9692
Disable RIME prefetch
sjperkins Nov 12, 2018
4ca7aa6
Fix buffer_size in dde.py
sjperkins Nov 12, 2018
9d453d3
Fixup Zernike tests again
sjperkins Nov 13, 2018
024b0f8
Introduce tolerances to zernike random input test
sjperkins Nov 13, 2018
0a573ba
Zernike test formatting
sjperkins Nov 13, 2018
1fee29b
Mark test_zernike.py::test_random_inputs as xfail
sjperkins Nov 13, 2018
6c82cec
Dataset updates for tensorflow 1.12.0
sjperkins Nov 20, 2018
8784e86
Submit op_kernel_utils.h
sjperkins Nov 20, 2018
c9c2842
Session cache
sjperkins Nov 22, 2018
c879f52
Rework package structure
sjperkins Nov 22, 2018
711a637
Revert debugging code
sjperkins Nov 22, 2018
488889d
Remove probably unnecessary mutex_locks
sjperkins Nov 22, 2018
a4ad38d
Sersic shape schema stuff
sjperkins Nov 26, 2018
1fabda5
Correctly name sersic_shape
sjperkins Nov 26, 2018
d403eac
Allow maps to be named for easier debugging
sjperkins Nov 26, 2018
4cd1967
Allow queues to be named for easier debugging
sjperkins Nov 26, 2018
f816542
Last WIP
sjperkins Jan 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
montblanc/_version.py export-subst
12 changes: 12 additions & 0 deletions MANIFEST.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
include setup.cfg
include versioneer.py
include montblanc/_version.py
include montblanc/include/montblanc/*.cuh
include montblanc/impl/rime/tensorflow/rime_ops/*.cpp
include montblanc/impl/rime/tensorflow/rime_ops/*.cpp
include montblanc/impl/rime/tensorflow/rime_ops/*.cu
include montblanc/impl/rime/tensorflow/rime_ops/*.cuh
include montblanc/impl/rime/tensorflow/rime_ops/*.h
include montblanc/impl/rime/tensorflow/rime_ops/*.hpp
graft montblanc/include/
include montblanc/ext/*.cpp
Loading