-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
U/jrbogart/config reorg #112
Merged
Merged
Changes from 22 commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
f03cfda
Add template config files for each supported source type.
JoanneBogart d553e99
Add classes YamlPassthruIncludeLoader, ConfigWriter to config_utils.py
JoanneBogart 3f82bc0
various bug fixes. Can now write config fragment, top file for objec…
JoanneBogart feb5643
fix rebase missteps
JoanneBogart 9130d53
add routine update_yaml to unconditionally write file, even if it alr…
JoanneBogart 3d9ddad
reorganization of config essentially working; more polishing and test…
JoanneBogart 88e306b
bug fix needed for schema backward-compatibility
JoanneBogart 0ceeb69
Add/improve docstrings; eliminate unused code
JoanneBogart 52b2636
minor fixes so old test programs will run
JoanneBogart 4572326
centralize handling of (config) schema version in Config class
JoanneBogart 20549e2
Add CI test for new arrangement of config file
JoanneBogart 25c69e9
Rename test module
JoanneBogart 27c9683
make test names unique
JoanneBogart e43dc18
sso code needed a minor update for new config system
JoanneBogart bca6a51
sso updates for new config arrangement
JoanneBogart c774e56
updates for SSO. Also centralize "old style" handling
JoanneBogart 06c6f6c
address some reviewer comments
JoanneBogart 6da4f96
reorganize creation of yaml fragments for greater maintainability
JoanneBogart d553ccf
Add code to retrieve throughputs versions, write to flux file metadata
JoanneBogart 23a74ae
sso binary output should also have metadata
JoanneBogart 820aa6c
fix reference to throughputs version
JoanneBogart fc25144
address reviewer comments
JoanneBogart 01a5309
one last bit of clean-up
JoanneBogart File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,4 @@ | ||
from ._version import __version__ | ||
|
||
from .utils import * | ||
from .objects import * | ||
from .skyCatalogs import * | ||
from .catalog_creator import * |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
34 changes: 34 additions & 0 deletions
34
skycatalogs/data/cfg_templates/diffsky_galaxy_template.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
area_partition: | ||
nside: 32 | ||
ordering: ring | ||
type: healpix | ||
composite: | ||
bulge: required | ||
disk: required | ||
knots: optional | ||
data_file_type: parquet | ||
file_template: galaxy_(?P<healpix>\d+).parquet | ||
flux_file_template: galaxy_flux_(?P<healpix>\d+).parquet | ||
sed_file_template: galaxy_sed_(?P<healpix>\d+).hdf5 | ||
components: | ||
diffsky_bulge: | ||
MW_extinction: F19 | ||
internal_extinction: CCM | ||
parent: diffsky_galaxy | ||
sed_model: TBD | ||
spatial_model: sersic2D | ||
subtype: bulge | ||
diffsky_disk: | ||
MW_extinction: F19 | ||
internal_extinction: CCM | ||
parent: diffsky_galaxy | ||
sed_model: TBD | ||
spatial_model: sersic2D | ||
subtype: disk | ||
diffsky_knots: | ||
MW_extinction: F19 | ||
internal_extinction: CCM | ||
parent: diffsky_galaxy | ||
sed_model: TBD | ||
spatial_model: knots | ||
subtype: knots |
7 changes: 7 additions & 0 deletions
7
skycatalogs/data/cfg_templates/gaia_star_butler_template.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
sed_method: use_lut | ||
id_prefix: gaia_dr2_ | ||
area_partition: None | ||
butler_parameters: | ||
collections: HSC/defaults | ||
dstype: gaia_dr2_20200414 | ||
data_file_type: butler_refcat |
8 changes: 8 additions & 0 deletions
8
skycatalogs/data/cfg_templates/gaia_star_direct_template.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
sed_method: use_lut | ||
id_prefix: gaia_dr2_ | ||
area_partition: | ||
level: 7 | ||
type: htm | ||
data_file_type: fits | ||
data_dir: /sdf/group/rubin/datasets/refcats/htm/v1/gaia_dr2_20200414 | ||
basename_template: (?P<htm>\d+).fits |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
file_template: 'galaxy_(?P<healpix>\d+).parquet' | ||
flux_file_template: 'galaxy_flux_(?P<healpix>\d+).parquet' | ||
data_file_type: parquet | ||
area_partition: | ||
{ type: healpix, ordering: ring, nside: 32} | ||
composite: | ||
bulge: required | ||
disk: required | ||
knots: optional | ||
attribute_aliases: | ||
size_knots_true: size_disk_true | ||
size_minor_knots_true: size_minor_disk_true | ||
components: | ||
bulge_basic: | ||
subtype: bulge | ||
parent: galaxy | ||
sed_model: tophat | ||
internal_extinction: CCM | ||
MW_extinction: F19 | ||
spatial_model: sersic2D | ||
disk_basic: | ||
subtype: disk | ||
parent: galaxy | ||
sed_model: tophat | ||
internal_extinction: CCM | ||
MW_extinction: F19 | ||
spatial_model: sersic2D | ||
knots_basic: | ||
subtype: knots | ||
parent: galaxy | ||
sed_model: tophat | ||
internal_extinction: CCM | ||
MW_extinction: F19 | ||
spatial_model: knots | ||
tophat: | ||
bin_parameters: | ||
- start | ||
- width | ||
units: angstrom |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
area_partition: | ||
nside: 32 | ||
ordering: ring | ||
type: healpix | ||
data_file_type: parquet | ||
file_template: snana_(?P<healpix>\d+).parquet | ||
internal_extinction: None | ||
sed_model: snana |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
area_partition: | ||
nside: 32 | ||
ordering: ring | ||
type: healpix | ||
data_file_type: parquet | ||
file_template: sso_(?P<healpix>\d+).parquet | ||
flux_file_template: sso_flux_(?P<healpix>\d+).parquet | ||
sed_model: dbfile_angstrom_flambda |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
area_partition: | ||
nside: 32 | ||
ordering: ring | ||
type: healpix | ||
data_file_type: parquet | ||
file_template: pointsource_(?P<healpix>\d+).parquet | ||
flux_file_template: pointsource_flux_(?P<healpix>\d+).parquet | ||
internal_extinction: None | ||
sed_file_root_env_var: SIMS_SED_LIBRARY_DIR | ||
sed_model: file_nm | ||
file_nm: | ||
units: nm |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SnanaConfigFragment
andGaiaConfigFragment
aren't used anywhere?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, that's because skyCatalogs is not responsible for making those catalogs. Those routines are provided in case someone (or some thing) that does produce them wants to also produce a suitable yaml fragment.