📊 Flunet tests #3265
Merged
📊 Flunet tests #3265
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Quick links (staging server):
Login: chart-diff: ✅No charts for review.data-diff:= Dataset garden/cancer/2024-08-30/gco_alcohol
= Table gco_alcohol
~ Column value (changed data)
~ Changed values: 2192 / 11040 (19.86%)
country year sex cancer indicator value - value +
Brazil 2020 female larynx asr 0.05 <0.1
Burkina Faso 2020 female pharynx asr 0.05 <0.1
Colombia 2020 female oesophagus asr 0.05 <0.1
Mauritius 2020 female larynx asr 0.05 <0.1
Oman 2020 male breast asr 0.05 <0.1
2024-09-11 10:59:10 [error ] Traceback (most recent call last):
File "/home/owid/etl/.venv/lib/python3.10/site-packages/requests/models.py", line 974, in json
return complexjson.loads(self.text, **kwargs)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/__init__.py", line 514, in loads
return _default_decoder.decode(s)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/decoder.py", line 386, in decode
obj, end = self.raw_decode(s)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/decoder.py", line 416, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/owid/etl/etl/datadiff.py", line 429, in cli
lines = future.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/owid/etl/etl/datadiff.py", line 422, in func
differ.summary()
File "/home/owid/etl/etl/datadiff.py", line 260, in summary
self._diff_tables(self.ds_a, self.ds_b, table_name)
File "/home/owid/etl/etl/datadiff.py", line 122, in _diff_tables
table_a = future_a.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f
return copy(f, *args, **kw)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__
do = self.iter(retry_state=retry_state)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
result = action(retry_state)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__
result = fn(*args, **kwargs)
File "/home/owid/etl/etl/datadiff.py", line 843, in get_table_with_retry
return ds[table_name]
File "/home/owid/etl/etl/datadiff.py", line 284, in __getitem__
return tables.load()
File "/home/owid/etl/lib/catalog/owid/catalog/catalogs.py", line 312, in load
return self.iloc[0].load() # type: ignore
File "/home/owid/etl/lib/catalog/owid/catalog/catalogs.py", line 363, in load
return Table.read(uri)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 179, in read
table = cls.read_feather(path, **kwargs)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 365, in read_feather
cls._add_metadata(df, path, **kwargs)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 337, in _add_metadata
metadata = cls._read_metadata(path)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 399, in _read_metadata
return cast(Dict[str, Any], requests.get(metadata_path).json())
File "/home/owid/etl/.venv/lib/python3.10/site-packages/requests/models.py", line 978, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
~ Dataset garden/who/2024-09-09/flu_test
- - title: FLuID dataset
? ^^
+ + title: FLuNet dataset
? ^^^
- - FluID is a global platform for data sharing that links regional influenza epidemiological data into a single global database. The platform provides connections between existing databases and can also be used to directly enter data by the national focal points through a web-based interface. It complements the existing virological data collection tool FluNet.
+ + FluNet is a global web-based tool for influenza virological surveillance first launched in 1997. The virological data entered into FluNet, e.g. number of influenza viruses detected by subtype, are critical for tracking the movement of viruses globally and interpreting the epidemiological data. The data at country level are publically available and updated weekly. The results are presented in various formats including tables, maps and graphs.
- - The platform accommodates both qualitative and quantitative data which facilitates the tracking of global trends, spread, intensity, and impact of influenza. These data are made freely available to health policy makers in order to assist them in making informed decisions regarding the management of influenza.
+ + The data are provided remotely by National Influenza Centres (NICs) of the Global Influenza Surveillance and Response System (GISRS) and other national influenza reference laboratories collaborating actively with GISRS, or are uploaded from WHO regional databases.
~ Table flu_test (changed metadata)
- - - case_info
~ Column case_info (changed metadata, new data, changed data)
- - title: Type of case e.g. SARI, ILI, ARI
+ + {}
+ + New values: 37925 / 71186 (53.28%)
case_info
NaN
NaN
NaN
NaN
NaN
~ Changed values: 33261 / 71186 (46.72%)
case_info - case_info +
ILI NaN
SARI NaN
SARI NaN
SARI NaN
SARI NaN
~ Column country (new data, changed data)
+ + New values: 37925 / 71186 (53.28%)
country
Laos
Nepal
Nicaragua
Pakistan
Thailand
~ Changed values: 32839 / 71186 (46.13%)
country - country +
Belgium Australia
Belgium Australia
Cuba Cambodia
Guatemala Costa Rica
Indonesia Democratic Republic of Congo
~ Column date (new data, changed data)
+ + New values: 37925 / 71186 (53.28%)
date
2023-05-22
2017-07-10
2016-09-05
2021-01-11
2023-06-12
~ Changed values: 33238 / 71186 (46.69%)
date - date +
2023-03-27 2008-12-15
2023-01-09 2016-02-08
2021-06-28 2017-09-04
2020-08-03 2009-07-06
2021-02-08 2013-12-02
~ Column denomcombined (changed metadata, new data, changed data)
- - {}
+ + title: The number of tests
+ + origins:
+ + - producer: WHO
+ + title: FluNet, World Health Organization
+ + citation_full: FluNet, World Health Organization (2023)
+ + url_main: https://www.who.int/tools/flunet/flunet-summary
+ + date_accessed: '2024-09-09'
+ + date_published: '2024'
+ + unit: tests
+ + display:
+ + numDecimalPlaces: 0
+ + processing_level: minor
+ + presentation:
+ + grapher_config:
+ + note: |-
+ + The number of tests is based on three possible sources based on data availability. If both are available, the sum of positive and negative tests is used. If not then the number of specimens processed is used. If neither are available, then the number of specimens received by the testing facility is used.
+ + topic_tags:
+ + - Influenza
+ + New values: 37925 / 71186 (53.28%)
denomcombined
89
114
102
253
204
~ Changed values: 33261 / 71186 (46.72%)
denomcombined - denomcombined +
NaN 42
NaN 1369
NaN 1478
NaN 682
NaN 3472
~ Column inf_tested (changed metadata, new data, changed data)
+ + {}
- - title: The number of influenza tests carried out
- - origins:
- - - producer: WHO
- - title: FluID, World Health Organization
- - citation_full: FluID, World Health Organization (2023)
- - url_main: https://www.who.int/teams/global-influenza-programme/surveillance-and-monitoring/fluid
- - date_accessed: '2024-09-09'
- - date_published: '2024'
- - unit: tests
- - display:
- - numDecimalPlaces: 0
- - processing_level: minor
- - presentation:
- - topic_tags:
- - - Influenza
+ + New values: 37925 / 71186 (53.28%)
inf_tested
NaN
NaN
NaN
NaN
NaN
~ Changed values: 33261 / 71186 (46.72%)
inf_tested - inf_tested +
6 NaN
1 NaN
12 NaN
0 NaN
0 NaN
~ Column level_0 (new data)
+ + New values: 37925 / 71186 (53.28%)
level_0
34878
42207
43564
46652
62034
~ Column pcnt_poscombined (changed metadata, new data, changed data)
- - {}
+ + title: The share of positive tests
+ + origins:
+ + - producer: WHO
+ + title: FluNet, World Health Organization
+ + citation_full: FluNet, World Health Organization (2023)
+ + url_main: https://www.who.int/tools/flunet/flunet-summary
+ + date_accessed: '2024-09-09'
+ + date_published: '2024'
+ + unit: '%'
+ + display:
+ + numDecimalPlaces: 0
+ + processing_level: minor
+ + presentation:
+ + grapher_config:
+ + note: |-
+ + The number of tests is based on three possible sources based on data availability. If both are available, the sum of positive and negative tests is used. If not then the number of specimens processed is used. If neither are available, then the number of specimens received by the testing facility is used.
+ + topic_tags:
+ + - Influenza
+ + New values: 37925 / 71186 (53.28%)
pcnt_poscombined
24.719101
49.122807
1.960784
5.928854
2.450980
~ Changed values: 33261 / 71186 (46.72%)
pcnt_poscombined - pcnt_poscombined +
NaN 4.761905
NaN 0.073046
NaN 13.328823
NaN 0.586510
NaN 13.940092
⚠ Found errors, create an issue please
Legend: +New ~Modified -Removed =Identical Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippetAutomatically updated datasets matching weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included Edited: 2024-09-11 10:59:13 UTC |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.