Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Features #380

Open
wants to merge 123 commits into
base: main
Choose a base branch
from
Open

Features #380

wants to merge 123 commits into from

Conversation

kkappler
Copy link
Collaborator

@kkappler kkappler commented Mar 22, 2025 •
edited
Loading

This PR is part of a larger change involving mt_metadata, mth5, and mtpy-v2.

The primary goal of the PR is to add the capability to extract features from MTH5 data, store the features in MTH5, and show an example workflow where the features can be applied to generate weights in data processing.

Also, significant reorganization of processing tools has been done, including the migration of processing support tools to mth5 from aurora.

  • The programatically-generated processing config will be validated against a stored json
    • This will become part of the testing flow and allow for a processing_configuration_template.json to be used as examples of how to specify features and weights calculations.

TODO

  • The processing parameters template can be modified so we add some weighting by coherence
    • augment processing_configuration_template.json with the weights structure from issue Data Weightings in EMTF #61
    • The above fails to load  mt_metadata.base.metadata | set_attr_from_name | weights.ex.combination_style is not in the current standards.  To properly add the attribute use add_base_attribute.
  • FCCoherence needs to be added to mt_metadata as a feature
    • It will be an xarray with dims time, and frequency.
    • The frequency axis will come from mth5.processing.spectre.frequency_band_helpers, using ( log_spaced_frequencies and bands_of_constant_q to build broad coherence bands.
    • The metadata for specifying this is in mt_metadata/features/standards/fc_coherence.json
    • The features can be packed into the mth5 beforehand, or they can be computed on the fly
    • In either case, we need to write the parameters that define the bands, and the feature name into the processing_config_template
  • Aurora will read in the updated processing config, and compute the weights .. to do this we will still need somewhere to put the thresholding info, a suggested format for a weights block is here
  • An integrated test will be developed in aurora to run an example workflow with feature weights (test_feature_weighting.py)'.
  • That test will use a processing config test_processing_config_with_weights_block.json, currently stored in aurora/config

kkappler added 30 commits November 8, 2024 16:50
@kkappler
housekeeping
082fa73
@kkappler
remove uneeded ipynb form tests
f0c41d0 
- also, pytest was complaining about it locally, and it doesn't seem worth looking into
@kkappler
Merge minor changes from fix_mtpy_issue_67
89e000d 
- these don't hurt and make the flow slightly more robust
@kkappler
housekeeping / doc strings
e4df7e6
@kkappler
fix typehint
49b1f33
@kkappler
housekeeping & typehinting
a7406dd
@kkappler
doc and typehints
c81729c
@kkappler
Minor change
b7cb76c 
- make prewhitening correction depend only on keyword, not all decimation params
@kkappler
doc / type hints
080709c
@kkappler
Move recoloring boolean check above call to apply_recoloring
021ad30
@kkappler
tidy doc
b4af28b
@kkappler
Test new pathing for Band object in mt_metadata
f914a30 
- this requires reinstalling mt_metadata over mtpy
- I'm not sure whether or not to comment out the aurora install ..
... this may get quite complicated.  This push is a test to see how complicated
@kkappler
Reduce number of instances for test and update mt_metadata branch
cb86a49
@kkappler
try suppress mtpy-v2 reinstall
eed21ba
@kkappler
update import get_fft_harmonics to new path in mt_metadata
cb1225e
@kkappler
replace some calls to decimation attrs with access to time_series_dec…
df77152 
…imation in FCDecimation
@kkappler
change var name and add dtype (local tests pass)
5fa3fe7
@kkappler
add some dtyping and doc
4931648
@kkappler
add doc/notes
12d4cf4
@kkappler
add TODO
f0abee1
@kkappler
add typehint, tests pass locally
f7cc243
@kkappler
updates for partial compatability with mt_metadata issue 238 (local t…
65732fa 
…ests pass)
@kkappler
access recoloring via stft
ab90d8b
@kkappler
add a variation on tests
264dfc6
@kkappler
comment out notebooks for clearer error messages
c21793d
@kkappler
towards mt_metadata 235
503758b
@kkappler
Update per changes in mt_metadata
6e69bca 
- rename extra_pre_fft_detrend_type to per_window_detrend_type
- add explicit assignment of per_window_detrend_type="linear" in config creator
@kkappler
try another way to make tests pass
6a27f92
@kkappler
add test (towards issue #363)
c0a885d
@kkappler
echo some env info
41a414b
kujaku11 and others added 30 commits March 21, 2025 16:09
@kujaku11
Update tests.yml
4a06181
@kujaku11
Update tests.yml
77d043e
@kujaku11
Update operate_aurora.ipynb
44afca6
@kujaku11
Update transfer_function_kernel.py
1669ff7
@kujaku11
fixing mergeing errors
c1f96e8
@kkappler
Merge pull request #379 from simpeg/patches
154c1dc 
Patches into features
@kkappler
Prepare for mth5 issue 291
692dec8 
- several methods are about to be moved from mth5/timeseries/spectre to mth5/processing/spectre
- tests passing locally before this commit
- will make changes with mth5 on `fix_issue_291` branch, and test locally again before merging PR on mth5 and then push aurora
@kkappler
update imports for mth5 issue 291
e486d36
@kkappler
reduce jankyness in processing_summary
d17eadd
@kkappler
Factor processing summary out of TFK
ea4456c 
- it is now in helper functions, and at least a bit easier to maintain
- also config.decimation_info is now a property, not a method
@kkappler
minor bug fixes
b5c369d
@kkappler
delete commented function (now in mth5)
f91b1f6
@kkappler
Add reference processing config file for testing
414b3a0 
- also minor changes, typehinting and TODOs
@kkappler
debugging failing test - filecmp failed ... is reference file available?
9ae9019
@kkappler
speed up test diagnostics
f5a15ee
@kkappler
locate test targte file in absolute path
e8d16dd
@kkappler
debugging
e33de55
@kkappler
debugging
717628a
@kkappler
debugging
8fc4b6c
@kkappler
debugging
7de4d03
@kkappler
reset tests
7bd5c20
@kkappler
update test_config_creator to compare Processing objects
f7df4e5
@kkappler
remove commented code already tested in mt_metadata
f686e13
@kkappler
add json with example weights block in processing params
13fe35b
@kkappler
update test to check reference and onthefly jsons
f76b53d
@kkappler
add PROCESSING_TEMPLATES_PATH
420f96a
@kkappler
rm file after tests
4083428
@kkappler
add typehints
f9adb13
@kkappler
remove RunTimeWarning for calibration with zero DC response
16a41ec
@kkappler
add TODO to fix runtime warnings
9fe7849
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kkappler @kujaku11