Skip to content

Commit

Permalink
metadata fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
@veronikasamborska1994
veronikasamborska1994 committed Jul 27, 2023
1 parent dc5b7bc commit c2295ff
Show file tree
Hide file tree
Showing 2 changed files with 24 additions and 172 deletions.
154 changes: 16 additions & 138 deletions etl/steps/data/garden/oecd/2023-05-19/co2_air_transport.meta.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,78 +20,38 @@ dataset:
tables:
co2_air_transport:
variables:
africa:
title: Africa
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
asia:
title: Asia
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
europe:
title: Europe
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
north_america:
title: North America
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
oceania:
title: Oceania
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
south_america:
title: South America
description: Total CO₂ emissions from international aviation.
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
world:
title: World
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
ter_dom_a:
title: Annual CO₂ emissions from domestic aviation
description: Domestic aviation represents flights which depart and arrive within the same country.
description: Domestic aviation represents flights which depart and arrive within the same country. CO₂ emissions from commercial passenger flights only.
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
ter_int_a:
title: Annual CO₂ emissions from international aviation
description: International aviation emissions are assigned to the country of departure. CO₂ emissions from commercial passenger flights only.
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
per_capita_ter_dom_a:
title: Per capita CO₂ emissions from domestic aviation
unit: kilograms
description: Domestic aviation represents flights which depart and arrive within the same country. The term "per capita" indicates that the emissions are calculated on a per-person basis, taking into account the total population of the country. By using this measurement, we can understand the average amount of CO₂ emissions attributed to each individual as a result of domestic air travel.
description: >
Domestic aviation represents flights which depart and arrive within the same country. The term "per capita" indicates that the emissions are calculated on a per-person basis, taking into account the total population of the country. By using this measurement, we can understand the average amount of CO₂ emissions attributed to each individual as a result of domestic air travel.
CO₂ emissions from commercial passenger flights only.
short_unit: kg
display:
numDecimalPlaces: 0
per_capita_ter_int_a:
title: Per capita CO₂ emissions from international aviation.
descripton: International aviation emissions are assigned to the country of departure. The term "per capita" indicates that the emissions are calculated on a per-person basis, taking into account the total population of the country. By using this measurement, we can understand the average amount of CO₂ emissions attributed to each individual as a result of domestic air travel.
descripton: >
International aviation emissions are assigned to the country of departure. The term "per capita" indicates that the emissions are calculated on a per-person basis, taking into account the total population of the country. By using this measurement, we can understand the average amount of CO₂ emissions attributed to each individual as a result of domestic air travel.
CO₂ emissions from commercial passenger flights only.
unit: kilograms
short_unit: kg
display:
Expand Down Expand Up @@ -125,101 +85,18 @@ tables:
numDecimalPlaces: 0
ter_dom_m:
title: Monthly CO₂ emissions from domestic aviation
description: Domestic aviation represents flights which depart and arrive within the same country.
description: Domestic aviation represents flights which depart and arrive within the same country. CO₂ emissions from commercial passenger flights only.
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
zeroDay: '2019-01-01'
yearIsDay: true
january:
title: January
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
february:
title: February
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
march:
title: March
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
april:
title: April
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
may:
title: May
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
june:
title: June
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
july:
title: July
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
august:
title: August
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
september:
title: September
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
october:
title: October
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
november:
title: November
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
december:
title: December
description: Total CO₂ emissions from international aviation
unit: tonnes
short_unit: t
display:
numDecimalPlaces: 0
ter_int_m:
title: Monthly CO₂ emissions from international aviation
unit: tonnes
short_unit: t
description: International aviation emissions are assigned to the country of departure.
description: International aviation emissions are assigned to the country of departure. CO₂ emissions from commercial passenger flights only.
display:
numDecimalPlaces: 0
zeroDay: '2019-01-01'
Expand All @@ -229,9 +106,10 @@ tables:
title: Monthly CO₂ total emissions from aviation
unit: tonnes
short_unit: t
description: Includes emissions from both domestict and international passegener flights. International aviation emissions are assigned to the country of departure.
description: Includes emissions from both domestic aviation. International aviation emissions are assigned to the country of departure. CO₂ emissions from commercial passenger flights only.
display:
numDecimalPlaces: 0
zeroDay: '2019-01-01'
yearIsDay: true


42 changes: 8 additions & 34 deletions etl/steps/data/garden/oecd/2023-05-19/co2_air_transport.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,9 @@ def run(dest_dir: str) -> None:
pivot_outb = add_inbound_outbound_tour(pivot_table_ye, df_tr)

# Process monthly data
pivot_df, pivot_table_mn = process_monthly_data(df, month_names)

concatenated_df = pd.merge(pivot_outb, pivot_table_mn, on=["year", "country"], how="outer")
merge_df = pd.merge(concatenated_df, pivot_df, on=["year", "country"], how="outer")

pivot_table_mn = process_monthly_data(df, month_names)
merge_df = pd.merge(pivot_outb, pivot_table_mn, on=["year", "country"], how="outer")
print(merge_df.columns)
regions_ = ["North America", "South America", "Europe", "Africa", "Asia", "Oceania", "World"]

for region in regions_:
Expand All @@ -93,29 +91,12 @@ def run(dest_dir: str) -> None:

merge_df["TER_INT_m"] = merge_df["TER_INT_m"].replace(0, np.nan)
merge_df = merge_df[merge_df["year"] != 2023]

# Apply the function to each row using apply()
merge_df.loc[~merge_df.country.isin(regions_), month_names] = np.nan
merge_df.reset_index(inplace=True, drop=True)

# Melt the DataFrame to have months as rows
df_melted = merge_df.melt(id_vars=["country", "year"], value_vars=month_names, var_name="month", value_name="value")
filtered_df = df_melted.dropna(subset=["value"])

# Pivot the melted DataFrame to reshape it with countries as columns
df_pivoted = (
filtered_df.pivot_table(index=["month", "year"], columns="country", values="value")
.reset_index()
.rename(columns={"month": "country"})
)

merged_df = pd.merge(df_pivoted, merge_df, on=["country", "year"], how="outer")
merged_df = merged_df.drop(["population"], axis=1)
merged_df["total_monthly_emissions"] = merged_df["TER_INT_m"] + merged_df["TER_DOM_m"]
merged_df.set_index(["country", "year"], inplace=True)
merge_df = merge_df.drop(["population"], axis=1)
merge_df["total_monthly_emissions"] = merge_df["TER_INT_m"] + merge_df["TER_DOM_m"]
merge_df.set_index(["country", "year"], inplace=True)

# Create a new table with the processed data.
tb_garden = Table(merged_df, short_name="co2_air_transport")
tb_garden = Table(merge_df, short_name="co2_air_transport")
#
# Save outputs.
#
Expand Down Expand Up @@ -175,13 +156,6 @@ def process_monthly_data(df, month_names):
df_mn["date"] = date_column
df_mn["emission_source"] = df_mn["emission_source"].apply(lambda x: x + "_m")

# Split by month
pivot_df = pd.pivot_table(
df_mn[df_mn["emission_source"] == "TER_INT_m"], values="value", index=["country", "year"], columns="month"
)
pivot_df.reset_index(inplace=True)
pivot_df.columns = ["country", "year"] + month_names

# Calculate the number of days since 2019
df_mn["days_since_2019"] = (df_mn["date"] - pd.to_datetime("2019-01-01")).dt.days
df_mn.drop(["month", "year", "date"], axis=1, inplace=True)
Expand All @@ -191,7 +165,7 @@ def process_monthly_data(df, month_names):
pivot_table_mn = pd.pivot_table(df_mn, values="value", index=["country", "year"], columns=["emission_source"])
pivot_table_mn.reset_index(inplace=True)

return pivot_df, pivot_table_mn
return pivot_table_mn


def add_inbound_outbound_tour(df, df_tr):
Expand Down

0 comments on commit c2295ff

Please sign in to comment.