merge-psychonautwiki-tripsit-data

Script to merge data from PsychonautWiki API + pages and TripSit factsheets into one standardized format

Installation

1. Install virtualenv

   python3 -m pip install --user virtualenv
   

2. Create virtual environment

   python3 -m venv env
   

3. Activate virtual environment

   source env/bin/activate
   

4. Install packages

   python3 -m pip install -r requirements.txt
   

Usage

python scrape.py [-h] [output]

Scrape PsychonautWiki and TripSit data into unified dataset

positional arguments:
  output      Optional output file

optional arguments:
  -h, --help  show this help message and exit

Output Schema

Property	Type	Description	Source
url	string	Substance page URL	PsychonautWiki if exists, TripSit factsheet otherwise
experiencesUrl	null | string	Erowid Experience Vault page URL	TripSit
name	string	Substance Name	shortest display name of PsychonautWiki page header and TripSit API's pretty_name
aliases	string[]	List of aliases (lowercase)	Both
aliasesStr	string	Comma-separated list of aliases (for fuzzy searching)	Both
summary	string	Substance summary	TripSit
reagents	null | string	Reagent test kit reactions	TripSit
classes	null | Dictionary	Substance classifications	PsychonautWiki
classes.chemical	string[]	Chemical classifications	PsychonautWiki
classes.psychoactive	string[]	Psychoactive classifications	PsychonautWiki
toxicity	null | string[]	List of toxicity characteristics	PsychonautWiki
addictionPotential	null | string	Description of potential for addiction	PsychonautWiki
tolerance	null | Dictionary	Tolerance information	PsychonautWiki
tolerance.full	null | string	When full tolerance activates	PsychonautWiki
tolerance.half	null | string	When tolerance is reduced by half	PsychonautWiki
tolerance.zero	null | string	When tolerance is back to zero	PsychonautWiki
crossTolerances	null | string[]	Other substances or classes with cross tolerance	PsychonautWiki
roas	Dictionary[]	Routes of administration	Both (Note: PsychonautWiki ROAs are prioritized, and TripSit factsheets fill in gaps)
roas[].name	string	Name of ROA (e.g. 'Oral', 'Smoked')	Both
roas[].bioavailability	Optional<string>	Substance's bioavailability for ROA	Both
roas[].dosage	null | Dictionary[]	Dosage information for ROA	Both
roas[].dosage[].name	string	Name of dosage level for ROA (e.g. 'Threshold', 'Strong')	Both
roas[].dosage[].value	string	Typical dosage for given level of ROA (e.g. '10 - 25 mg', '100 mg +')	Both
roas[].dosage[].note	Optional<string>	Additional note	PsychonautWiki
roas[].duration	null | Dictionary[]	Duration information for ROA	Both
roas[].duration[].name	string	Name of duration item for ROA (e.g. 'Onset', 'Peak')	Both
roas[].duration[].value	string	Typical duration for given item of ROA (e.g. '30 - 60 minutes', '2 - 4 hours')	Both
roas[].duration[].note	Optional<string>	Additional note	PsychonautWiki
interactions	null | Dictionary[]	Interactions with other substances or classes	TripSit
interactions[].name	string	Name of other substance or class	TripSit
interactions[].status	string	Interaction status from TripSit combos	TripSit
interactions[].note	Optional<string>	Additional note	TripSit