Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rtsys metadata #4

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
b0e483f
added an expection when parsing the xml file for defectuous files
cparcerisas Oct 24, 2024
68b794d
added .idea to the gitignore for pycharm developerse
cparcerisas Oct 24, 2024
8417d19
Get metadata from wav file instead of xml
bram-cuyx Nov 12, 2024
b568260
debugging
bram-cuyx Nov 12, 2024
374f1b0
Fixed the start of the deployment being overridden by the start of ea…
bram-cuyx Nov 13, 2024
13bf8f5
log faulty wav files and break loop
bram-cuyx Nov 13, 2024
4cee670
Printing the amount of erroneous files
bram-cuyx Nov 13, 2024
0e29dc9
Testing with a differen wav header reader
bram-cuyx Nov 13, 2024
f4e3d3e
prior to pull request, logs faulty file but then halts execution
bram-cuyx Nov 13, 2024
ab5f8d9
Raise an error in faulty files
bram-cuyx Nov 13, 2024
6d25989
Merge branch 'wav-metadata' of https://github.com/lifewatch/pbp into …
cparcerisas Nov 19, 2024
4be240c
added support for resea instruments
cparcerisas Dec 16, 2024
ee8708a
added gen_resea file for RTSys instruments
cparcerisas Jan 15, 2025
2c99d94
added a check to start and end arguments so that it's a date or a dat…
cparcerisas Jan 20, 2025
57a4a51
Merge branch 'main' into metadata_gen_start_date
cparcerisas Jan 20, 2025
7bf2889
fixed error in datetime.datetime to only datetime
cparcerisas Jan 20, 2025
fca38f5
Merge branch 'metadata_gen_start_date' of https://github.com/lifewatc…
cparcerisas Jan 20, 2025
a618809
added to docs the info that start and end in meta_gen should be a dat…
cparcerisas Jan 20, 2025
3c9e625
Merge branch 'metadata_gen_start_date' into rtsys_metadata
cparcerisas Jan 20, 2025
654d012
changed the check of the start, end to utils, and added the conversio…
cparcerisas Jan 20, 2025
1452369
Merge branch 'metadata_gen_start_date' into rtsys_metadata
cparcerisas Jan 20, 2025
df42de7
added import utils
cparcerisas Jan 20, 2025
7c40d20
Merge branch 'metadata_gen_start_date' into rtsys_metadata
cparcerisas Jan 20, 2025
9b0b982
changed the utils import to import only the function
cparcerisas Jan 20, 2025
eb92e86
Merge branch 'metadata_gen_start_date' into rtsys_metadata
cparcerisas Jan 20, 2025
1cfb5c4
added the RESEA option to main_meta_generation
cparcerisas Jan 20, 2025
25d4292
changed the explanation in docs pbp-meta-gen to explain that the expe…
cparcerisas Jan 20, 2025
b8512f2
Merge branch 'metadata_gen_start_date' into rtsys_metadata
cparcerisas Jan 20, 2025
969c4c6
added RESEA to the pbp-meta-gen docs as supported instrument
cparcerisas Jan 20, 2025
fc247d5
added the option RESEA in main_meta_generator_args recorder arg
cparcerisas Jan 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
.env
.idea/
pbp-doc/site/
output*/
cloud_tmp*/
Expand Down
8 changes: 8 additions & 0 deletions .idea/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 14 additions & 0 deletions .idea/deployment.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

137 changes: 137 additions & 0 deletions .idea/inspectionProfiles/Project_Default.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/inspectionProfiles/profiles_settings.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions .idea/misc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/modules.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 15 additions & 0 deletions .idea/pbp.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 12 additions & 5 deletions pbp-doc/docs/pbp-meta-gen/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,12 @@ Once this is done, you can proceed to the main program [pbp-hmb-gen](../pbp-hmb-
Three types of audio recorders are supported: NRS, IcListen, and Soundtrap files. Here is the current supported matrix:

----------------
| Recorder | [Google Storage](https://cloud.google.com/storage/docs/buckets) | [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) | Local Storage |
|----------|-----------------------------------------------------------------|------------------|---------------|
| NRS | :material-check: | :material-close: | :material-check: |
| IcListen | :material-close: | :material-check: | :material-check: |
| Soundtrap| :material-close: | :material-check: | :material-check: |
| Recorder | [Google Storage](https://cloud.google.com/storage/docs/buckets) | [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) | Local Storage |
|-----------|-----------------------------------------------------------------|------------------------------------------------------------------------------|---------------|
| NRS | :material-check: | :material-close: | :material-check: |
| IcListen | :material-close: | :material-check: | :material-check: |
| Soundtrap | :material-close: | :material-check: | :material-check: |
| RESEA | :material-close: | :material-close: | :material-check: |


For audio that is stored in a cloud storage bucket, the URI that is required to access the audio files depends on the cloud storage provider.
Expand Down Expand Up @@ -48,6 +49,12 @@ The data must be stored in a public cloud storage bucket; private buckets are no
NRS11_191024T022220Z.d100.x.wav
```

!!! note start and end date
The start and end date should be a string date, not including hour, minutes nor seconds.
When using the API directly instead of running the `pbp-meta-gen` command-line program, start and end date should
be a datetime.datetime object starting at 00:00:00. A datetime.date can also be passed, and it will be
automatically converted. If these requirements are not met, pbp will raise an error.


## Generate JSONs with audio metadata from NRS flac files for a date range

Expand Down
10 changes: 10 additions & 0 deletions pbp/main_meta_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,16 @@ def main():
xml_dir=xml_dir.as_posix(),
)
generator.run()
if opts.recorder == "RESEA":
generator = RESEAMetadataGenerator(
log=log,
uri=opts.uri,
json_base_dir=json_dir.as_posix(),
prefixes=opts.prefix,
start=start,
end=end
)
generator.run()
except KeyboardInterrupt:
log.info("INTERRUPTED")

Expand Down
2 changes: 1 addition & 1 deletion pbp/main_meta_generator_args.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ def parse_arguments():

parser.add_argument(
"--recorder",
choices=[InstrumentType.NRS, InstrumentType.ICLISTEN, InstrumentType.SOUNDTRAP],
choices=[InstrumentType.NRS, InstrumentType.ICLISTEN, InstrumentType.SOUNDTRAP, InstrumentType.RESEA],
required=True,
help="Choose the audio instrument type",
)
Expand Down
4 changes: 4 additions & 0 deletions pbp/meta_gen/gen_abstract.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

import pandas as pd

from pbp.meta_gen.utils import check_start_end_args


class MetadataGeneratorAbstract(object):
def __init__(
Expand Down Expand Up @@ -36,6 +38,7 @@ def __init__(
:return:
"""
try:
start, end = check_start_end_args(start, end)
self.audio_loc = audio_loc
self.json_base_dir = json_base_dir
self.df = pd.DataFrame()
Expand Down Expand Up @@ -92,6 +95,7 @@ def __init__(
:return:
"""
try:
start, end = check_start_end_args(start, end)
self.audio_loc = audio_loc
self.json_base_dir = json_base_dir
self.df = pd.DataFrame()
Expand Down
Loading