Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(#128, #117) Improve logging and add availability field #133

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

(#128, #117) Improve logging and add availability field #133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added =0.25
View file
Open in desktop
Empty file.
Empty file added =0.8,
View file
Open in desktop
Empty file.
Empty file added =1.0.0
View file
Open in desktop
Empty file.
222 changes: 222 additions & 0 deletions =1.3.0
View file
Open in desktop

Large diffs are not rendered by default.

Empty file added =2.4.0
View file
Open in desktop
Empty file.
Empty file added =4.0
View file
Open in desktop
Empty file.
Empty file added =4.2.2
View file
Open in desktop
Empty file.
5 changes: 4 additions & 1 deletion cernopendata/jsonschemas/records/record-v1.0.0.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,10 @@
"description": "Specifies if dataset is on-demand, online in future might be also tape, disk",
"enum": [
"ondemand",
"online"
"online",
"available",
"sample available",
"needs to be requested"
],
"type": "string"
},
Expand Down
20 changes: 9 additions & 11 deletions cernopendata/modules/fixtures/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@

MODE_OPTIONS = ["insert", "replace", "insert-or-replace", "insert-or-skip"]

import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")

def get_jsons_from_dir(dir):
"""Get JSON files inside a dir."""
Expand Down Expand Up @@ -78,14 +80,10 @@ def _handle_record_files(record, data):
filename = file.get("uri").split("/")[-1:][0]
f.set_uri(file.get("uri"), file.get("size"), file.get("checksum"))
if "type" in file and file["type"] == "index.json":
print(
datetime.now(),
"This is an index file. Let's check the entries that it has:",
file.get("uri"),
)
logging.info("Processing index file: %s",file.get("uri"))
# We don't need to store the index
FileIndexMetadata.create(record, f)
print("File index created")
logging.info("File index created")
f.delete()
elif "type" in file and file["type"] == "index.txt":
# The txt indexes should be ignored. Just delete the file
Expand All @@ -102,11 +100,11 @@ def _handle_record_files(record, data):
}
file.update(file_info)
except Exception as e:
click.secho(
"Recid {0} file {1} could not be loaded due "
"to {2}.".format(data.get("recid"), filename, str(e)),
fg="red",
err=True,
logging.error(
"Recid %s file %s could not be loaded due to %s ",
data.get("recid", "Unknown Recid"),
filename,
str(e),
)
record["files"] = real_files
data["files"] = real_files
Expand Down
4 changes: 2 additions & 2 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,8 +112,8 @@
"python-markdown-math>=0.3",
"python-slugify>=1.2.4",
# Webserver
"uWSGI>=2.0.21",
"uwsgitop>=0.11",
#"uWSGI>=2.0.21",
#"uwsgitop>=0.11",
# Pin SQLAlchemy version due to sqlalchemy-utils compatibility
# <https://github.com/kvesteri/sqlalchemy-utils/issues/505>
"sqlalchemy-continuum==1.4.1",
Expand Down