Skip to content

Commit

Permalink
Merge pull request #79 from dataforgoodfr/gg-documentation
Browse files Browse the repository at this point in the history
A little bit of documentation with dendron
  • Loading branch information
RonanMorgan authored Feb 21, 2024
2 parents 4e03976 + 1ec79a0 commit d3f3e01
Show file tree
Hide file tree
Showing 19 changed files with 423 additions and 46 deletions.
131 changes: 120 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,124 @@
# Bloom Project
![banner](images/banner.png)

This project is maintained by D4G in order to gather vessels data.
A cron job is launched every 15min, does calls to API and save the data in a Postgresql database.
## What is Trawl Watch

## About directory architecture
The domain directory ...
The infra directory ...
**[Trawl Watch](https://twitter.com/TrawlWatch)** is an initiative launched by the **[Bloom Association](https://www.bloomassociation.org/en/)** to track and expose the most destructive fishing vessels. Inspired by L’[Avion de Bernard](https://www.instagram.com/laviondebernard/), which monitors the movements of private jets, **Trawl Watch** aims to make visible the impact of these massive trawlers on our oceans. These vessels, often referred to as _mégachalutiers_, deploy gigantic nets that can engulf marine life from the surface down to the ocean floor. The consequences are both ecological—as they devastate crucial nursery and breeding areas for marine animals—and social, as they deprive artisanal fishermen of a healthy marine ecosystem. The solution proposed by **Bloom** is to dismantle these industrial fishing ships and redistribute their quotas to small-scale fishers. A petition has been launched, and **Bloom** continues to track these megatrawlers while awaiting action from European institutions.

**Did you know that, in Europe, the largest fishing vessels, which represent 1% of the fleet, catch half of the fish?** These factory-vessels can measure up to 144 meters in length and catch 400,000 kilos of fish per day! This is as much as 1,000 small-scale fishing vessels in one day at sea.

More information can be found there :
1. [Database initialisation and versioning](./documentation/database_init.md)
2. [Development environment](./documentation/dev_env.md)
3. [Useful SQL examples](./documentation/sql_examples.md)
4. [Data models](#todo)
**These veritable sea monsters are devastating Europe’s biodiversity and coastlines.** It is important to measure the scale of the damage: about 20 of these factory-vessels can obliterate hundreds of thousands of marine animals and biodiversity treasures in one day, including in the so-called ‘Marine Protected Areas’ of French territorial waters, which are not protected at all.

## What is Bloom Association

**BLOOM** is a non-profit organization founded in 2005 that works to preserve the marine environment and species from unnecessary destruction and to increase social benefits in the fishing sector. **BLOOM** wages awareness and advocacy campaigns in order to accelerate the adoption of concrete solutions for the ocean, humans and the climate. **BLOOM** carries out scientific research projects, independent studies and evaluations that highlight crucial and unaddressed issues such as the financing mechanisms of the fishing sector. **BLOOM**’s actions are meant for the general public as well as policy-makers and economic stakeholders.

## Installing Trawl Watch with `poetry`

### Prerequisites:

1. Python (≥ `3.10`) installed on your system.
2. Ensure [Docker](https://docs.docker.com/get-docker/) is installed.
3. Ensure you have `poetry` installed. If not, you can install them using `pip`.

```bash
pip install poetry
```

### Steps:

1. **Clone the GitHub Repository:**

Clone the GitHub repository you want to install locally using the `git clone` command.

```bash
git clone https://github.com/dataforgoodfr/12_bloom.git
```

2. **Navigate to the Repository Directory:**

Use the `cd` command to navigate into the repository directory.

```bash
cd 12_bloom/
```

3. **Configure `poetry` to create a Virtual Environment inside the project:**

Ensure that poetry will create a `.venv` directory into the project with the command:

```bash
poetry config virtualenvs.in-project true
```

4. **Install Project Dependencies using `poetry`:**

Use `poetry` to install the project dependencies.

```bash
poetry install
```

This will read the `pyproject.toml` file in the repository and install all the dependencies specified.

5. **Make sure everything is all right using `poetry env info`:**

```bash
poetry env info
```

It should looks something likes:

```bash
Virtualenv
Python: 3.11.2
Implementation: CPython
Path: /home/guillaume/12_bloom/.venv
Executable: /home/guillaume/12_bloom/.venv/bin/python
Valid: True

System
Platform: linux
OS: posix
Python: 3.11.2
Path: /usr
Executable: /usr/bin/python3.11
```

6. **Activate the Virtual Environment:**

Activate the virtual environment to work within its isolated environment.

On Unix or MacOS:

```bash
poetry shell
```

7.

### Once you're done working with the project, deactivate the virtual environment.

```bash
deactivate
```

## Documentation

[Dendron](https://marketplace.visualstudio.com/items?itemName=dendron.dendron) is a powerful Visual Studio Code, or [VSCodium](https://vscodium.com/), extension designed to streamline and enhance the documentation process. With **Dendron**, documenting projects becomes intuitive and efficient, thanks to its hierarchical note-taking system. Users can organize their documentation into a tree-like structure, making it easy to navigate and manage. The extension offers robust features such as bidirectional linking, which allows for seamless navigation between related notes, and support for Markdown formatting, enabling users to create rich and visually appealing documentation. Additionally, **Dendron** provides powerful search functionality, enabling users to quickly locate specific information within their documentation vault. Overall, **Dendron** empowers developers, writers, and teams to create comprehensive and well-organized documentation, facilitating better knowledge management and collaboration. The documentation is locaed inside the `./docs/notes` directory.

Here's some basic Dendron shortcuts:

| Shortcut | Description |
| -------------- | ------------ |
| `Cmd/Ctrl + L` | Lookup notes |

## More information can be found there

1. [Database initialisation and versioning](./docs/database.initialisation.md)
2. [Development environment](./docs/development.environment.md)
3. [Useful SQL examples](./docs/sql.examples.md)
4. [Data models](#todo)

## FAQ

#todo
Binary file added data/_info_subset_data.docx
Binary file not shown.
11 changes: 11 additions & 0 deletions data/vessels_subset_02022024.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
"id","country_iso3","cfr","IMO","registration_number","external_marking","ship_name","ircs","mmsi","loa","type","mt_activated"
621,"ESP","27678 ESP","8516043","3VI-5-1-15","3VI-5-1-15","PARADANTA PRIMERO","EAAB",224378000,"58.38",">25m",False
784,"FRA","FRA000919999","9828936","919999","CC919999","SCOMBRUS","FLVN",226347000,"81.37",">25m",False
795,"FRA","NLD198300393","8224406","716900","FC716900","PRINS BERNHARD","FKHH",227302000,"88.24",">25m",False
835,"FRA","FRA000932206","9741097","932206","CC932206","GEVRED","FIUO",228066900,"77",">25m",False
863,"FRA","FRA000914221","9352913","914221","MA914221","VENT DU NORD II","FMEU",228215800,"41.92",">25m",False
920,"GBR",NULL,"9877389","GY127",NULL,"HENDRIKA JACOBA",NULL,232031183,NULL,"Demersal seiners",False
1194,"NLD","FRA000544858","7928835","FRA000544858","SL-9","JOHANNA","PHOE",244938000,"34.3",">25m",False
1280,"NLD","NLD201001155","9599731","NLD201001155","UK-153","LUB SENIOR","PCGC",246749000,"28.5",">25m",False
1431,"POL","IRL000I13000","9204556","PL02315","GDY-151","ANNELIES ILENA","SPG4839",261084090,"144.6",">25m",False
1507,"PRT","PRT000023094","9297694","PTAVE-117345-N","PTAVE-117345-N","FRANÇA MORTE","CUFE7",263581000,"73.8",">25m",False
6 changes: 6 additions & 0 deletions docs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
node_modules
.dendron.*
build
seeds
.next
pods/service-connections
37 changes: 37 additions & 0 deletions docs/dendron.code-workspace
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"folders": [
{
"path": "notes",
"name": "docs"
}
],
"settings": {
"dendron.rootDir": ".",
"files.autoSave": "onFocusChange",
"pasteImage.path": "${currentFileDir}/assets/images",
"pasteImage.prefix": "/",
"markdown-preview-enhanced.enableWikiLinkSyntax": true,
"markdown-preview-enhanced.wikiLinkFileExtension": ".md",
"editor.snippetSuggestions": "inline",
"editor.suggest.snippetsPreventQuickSuggestions": false,
"editor.suggest.showSnippets": true,
"editor.tabCompletion": "on",
"dendron.enableSelfContainedVaultWorkspace": true
},
"extensions": {
"recommendations": [
"dendron.dendron",
"dendron.dendron-paste-image",
"dendron.dendron-markdown-shortcuts",
"redhat.vscode-yaml"
],
"unwantedRecommendations": [
"dendron.dendron-markdown-links",
"dendron.dendron-markdown-notes",
"dendron.dendron-markdown-preview-enhanced",
"shd101wyy.markdown-preview-enhanced",
"kortina.vscode-markdown-notes",
"mushan.vscode-paste-image"
]
}
}
108 changes: 108 additions & 0 deletions docs/dendron.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
version: 5
dev:
enablePreviewV2: true
enableSelfContainedVaults: true
commands:
lookup:
note:
selectionMode: extract
confirmVaultOnCreate: true
vaultSelectionModeOnCreate: smart
leaveTrace: false
bubbleUpCreateNew: true
fuzzThreshold: 0.2
randomNote: {}
insertNoteLink:
aliasMode: none
enableMultiSelect: false
insertNoteIndex:
enableMarker: false
copyNoteLink:
aliasMode: title
templateHierarchy: template
workspace:
vaults:
-
fsPath: .
selfContained: true
name: docs
journal:
dailyDomain: daily
name: journal
dateFormat: y.MM.dd
addBehavior: childOfDomain
scratch:
name: scratch
dateFormat: y.MM.dd.HHmmss
addBehavior: asOwnDomain
task:
name: task
dateFormat: y.MM.dd
addBehavior: asOwnDomain
statusSymbols:
'': ' '
wip: w
done: x
assigned: a
moved: m
blocked: b
delegated: l
dropped: d
pending: 'y'
taskCompleteStatus:
- done
- x
prioritySymbols:
H: high
M: medium
L: low
todoIntegration: false
createTaskSelectionType: selection2link
graph:
zoomSpeed: 1
createStub: false
enableAutoCreateOnDefinition: false
enableXVaultWikiLink: false
enableRemoteVaultInit: true
enableUserTags: true
enableHashTags: true
workspaceVaultSyncMode: noCommit
enableAutoFoldFrontmatter: false
enableEditorDecorations: true
maxPreviewsCached: 10
maxNoteLength: 204800
enableFullHierarchyNoteTitle: false
enablePersistentHistory: false
preview:
enableFMTitle: true
enableNoteTitleForLink: true
enableFrontmatterTags: true
enableHashesForFMTags: false
enablePrettyRefs: true
enableKatex: true
automaticallyShowPreview: false
publishing:
enableFMTitle: true
enableNoteTitleForLink: true
enablePrettyRefs: true
enableKatex: true
copyAssets: true
siteHierarchies:
- root
writeStubs: false
siteRootDir: docs
seo:
title: Dendron
description: Personal Knowledge Space
github:
enableEditLink: true
editLinkText: Edit this page on GitHub
editBranch: main
editViewMode: tree
enableSiteLastModified: true
enableFrontmatterTags: true
enableHashesForFMTags: false
enableRandomlyColoredTags: true
enableTaskNotes: true
enablePrettyLinks: true
searchMode: search
21 changes: 21 additions & 0 deletions docs/notes/ais.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
id: g1i0o9ctmib83n4mydtsqfo
title: Automatic identification system
desc: ""
updated: 1708461719224
created: 1708461557185
---

The automatic identification system (AIS) is an automatic tracking system that uses transceivers on ships and is used by vessel traffic services (VTS). When satellites are used to receive AIS signatures, the term Satellite-AIS (S-AIS) is used. AIS information supplements marine radar, which continues to be the primary method of collision avoidance for water transport. Although technically and operationally distinct, the ADS-B system is analogous to AIS and performs a similar function for aircraft.[^1]

## Limits

- out of range
- intense traffic area
- expensive

## Data source

https://spire.com/maritime/

[^1]: https://en.wikipedia.org/wiki/Automatic_identification_system
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
# Database initialisation and versioning


## Database initialisation
---
id: 8vo1vo09y37hwnrkmcy7w77
title: Database initialisation
desc: ""
updated: 1708410470701
created: 1708410417004
---

First you need to run scripts which are in alembic/init_script :

- the load_vessels_data.py script will load vessels metadata from the data/chalutier_pelagique.csv file.
- the load_geometry_data.py file will load shape data from the Nonterrestrial_WDPA_Jan2023.shp file. This file is not included in this github project but you can ask for it. It's only used for the alerting part.

The second step is to load the [distance-from-port-v20201104.tiff](https://globalfishingwatch.org/data-download/datasets/public-distance-from-port-v1) and [distance-from-shore.tif](https://globalfishingwatch.org/data-download/datasets/public-distance-from-shore-v1) files. They are only used for the alerting part.

- install psql and raster2pgsql.
- install raster type in db with postgis-raster using `create extension postgis_raster`
- adapt this command for each file : `raster2pgsql -t auto -I -C -M /PATH_TO/distance-from-shore.tif public.distance_shore | PGPASSWORD='POSTGRES_PASSWORD' psql -h POSTGRES_HOSTNAME -d POSTGRES_DB -U POSTGRES_USER -p POSTGRES_PORT`

## Database versioning
The command ` alembic upgrade head` can be used in the root of the project in order to update the database schema to the last version.
- adapt this command for each file : `raster2pgsql -t auto -I -C -M /PATH_TO/distance-from-shore.tif public.distance_shore | PGPASSWORD='POSTGRES_PASSWORD' psql -h POSTGRES_HOSTNAME -d POSTGRES_DB -U POSTGRES_USER -p POSTGRES_PORT`
9 changes: 9 additions & 0 deletions docs/notes/database.versioning.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
id: kofmnl4h6i8e9uacvtd2wkr
title: Database versioning
desc: ""
updated: 1708410507271
created: 1708410490074
---

The command ` alembic upgrade head` can be used in the root of the project in order to update the database schema to the last version.
Loading

0 comments on commit d3f3e01

Please sign in to comment.