notebook_tutorials/deploy_to_prolific.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy a thresh.tools Interface to Prolific\n",
    "In this tutorial, we will walk through a deployment to the [**Prolific**](https://app.prolific.co/) crowdsourcing platform. We will create a draft study and test it out. We assume you have created a Prolific account."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Data for Crowdsourcing\n",
    "In your template, make sure you have added crowdsourcing options:\n",
    "```yaml\n",
    "crowdsource: \"prolific\"\n",
    "prolific_completion_code: XXXXXXXX\n",
    "disable:\n",
    "- upload\n",
    "- download\n",
    "```\n",
    "For this tutorial, we will be using the [**demo crowdsourcing template**](https://thresh.tools/?t=demo_crowdsource), but any thresh.tools template will work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q thresh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from thresh import load_interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's begin by downloading the demo typology and data\n",
    "!curl -so crowdsource.yml https://thresh.tools/templates/demo/crowdsource.yml\n",
    "!curl -so crowdsource.json https://thresh.tools/data/demo/crowdsource.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded interface: Crowdsourcing Demo\n"
     ]
    }
   ],
   "source": [
    "# Now we can load and customize our interface template YML file we created on thresh.tools\n",
    "Interface = load_interface(\"crowdsource.yml\")\n",
    "print(f\"Loaded interface: {Interface.template_label}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Demo(\n",
      "  crowdsource: prolific, \n",
      "  prolific_completion_code: YOUR_COMPLETION_CODE, \n",
      "  disable: ['upload', 'download'], \n",
      "  instructions: ## Crowdsourcing Demo\n",
      "  You can add instructions here. Click to the end and press \"Submit\" to be redirected back to Prolific\n",
      "  , \n",
      "  prepend_instructions: True, \n",
      "  template_name: demo, \n",
      "  template_label: Crowdsourcing Demo, \n",
      "  edits: [{'name': 'edit_name', 'label': 'Edit Name', 'color': 'red', 'icon': 'fa-magnifying-glass', 'enable_input': True, 'enable_output': True}], \n",
      "  entry_class: <class 'nlproc_tools.names.DemoEntry'>, \n",
      "  edit_classes: {'edit_name': <class 'nlproc_tools.names.EditNameEdit'>}, \n",
      "  annotation_classes: {}\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Studies use a unique completion code. We begin by pasting the code into\n",
    "# the interface template.\n",
    "completion_code = \"YOUR_COMPLETION_CODE\"\n",
    "Interface.prolific_completion_code = completion_code\n",
    "print(Interface)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DemoEntry(\n",
      "  source: Gather ye annotations on the prolific tome, that its wisdom and verity may be preserved for generations hence, like jewels adorning the literary crown., \n",
      "  target: Gather your annotations on the prolific book so that its wisdom and truth can be preserved for future generations, like precious jewels decorating the literary heritage.\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Let's use the get_entry_class() to create an example for annotation\n",
    "DemoEntry = Interface.get_entry_class()\n",
    "\n",
    "entry = DemoEntry(\n",
    "    source=\"Gather ye annotations on the prolific tome, that its wisdom and verity may be preserved for generations hence, like jewels adorning the literary crown.\",\n",
    "    target=\"Gather your annotations on the prolific book so that its wisdom and truth can be preserved for future generations, like precious jewels decorating the literary heritage.\"\n",
    ")\n",
    "\n",
    "print(entry)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save our modified interface and custom data\n",
    "# Interface.export_interface(\"crowdsource.yml\")\n",
    "# Interface.export_data([entry], \"crowdsource.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create path files (for next section)\n",
    "template_path, data_path = None, None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Optional] Deploy Your Interface using GitHub\n",
    "Since Prolific needs to link to a hosted interface, we will deploy our interface publically using GitHub for hosting. You can skip this step by going to the [**Preview Interface**](#preview-interface) section.\n",
    "\n",
    "For this section you will need to create a GitHub access token: [**github.com/settings/tokens**](https://github.com/settings/tokens). Make sure to give permissions for *Administration* and *Contents* so we can push to a repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q gitpython PyGithub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created new repository: https://github.com/davidheineman/prolific-deployment-demo.git\n",
      "Pushed changes to https://github.com/davidheineman/prolific-deployment-demo.git\n"
     ]
    }
   ],
   "source": [
    "import git, os, shutil\n",
    "from github import Github\n",
    "\n",
    "local_template_path = \"crowdsource.yml\"\n",
    "local_data_path = \"crowdsource.json\"\n",
    "repository_name = \"prolific-deployment-demo\"\n",
    "\n",
    "# Get GitHub credentials\n",
    "if os.path.exists('.git_credentials'):\n",
    "    # This will read the API token from a local file\n",
    "    github_username, github_token = open('.git_credentials', \"r\").read().split('\\n')\n",
    "else:\n",
    "    github_username = input(\"Please enter your GitHub username: \")\n",
    "    github_token = getpass(\"Please enter your GitHub access token (register a token at github.com/settings/tokens): \")\n",
    "\n",
    "# Log into GitHub\n",
    "g = Github(github_token)\n",
    "user = g.get_user()\n",
    "\n",
    "# Create a new repository for our interface\n",
    "try:\n",
    "    remote_repo = g.get_repo(f\"{github_username}/{repository_name}\").clone_url\n",
    "    print(f\"Found existing repository: {remote_repo}\")\n",
    "except Exception as e:\n",
    "    remote_repo = user.create_repo(repository_name).clone_url\n",
    "    print(f\"Created new repository: {remote_repo}\")\n",
    "\n",
    "if not os.path.exists(repository_name):\n",
    "    git.Repo.clone_from(remote_repo, repository_name)\n",
    "\n",
    "repo = git.Repo(repository_name)\n",
    "\n",
    "# Upload our template and data files\n",
    "for file_path in [local_template_path, local_data_path]:\n",
    "    template_repo_path = os.path.join(os.getcwd(), repository_name, file_path)\n",
    "    shutil.copy(file_path, template_repo_path)\n",
    "    repo.index.add([template_repo_path])\n",
    "repo.index.commit(\"Add a new interface template and data\")\n",
    "\n",
    "origin = repo.remote(\"origin\")\n",
    "origin.push()\n",
    "print(f\"Pushed changes to {remote_repo}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hosted template: davidheineman/prolific-deployment-demo/master/crowdsource.yml\n",
      "Hosted data: davidheineman/prolific-deployment-demo/master/crowdsource.json\n"
     ]
    }
   ],
   "source": [
    "template_path = f\"{github_username}/{repository_name}/master/{local_template_path}\"\n",
    "data_path = f\"{github_username}/{repository_name}/master/{local_data_path}\"\n",
    "\n",
    "print(f\"Hosted template: {template_path}\")\n",
    "print(f\"Hosted data: {data_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"preview-interface\"></a>\n",
    "\n",
    "### Preview Interface\n",
    "Now we can use `thresh.tools` to view our hosted interface!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View your hosted template here:\t\t  https://nlproc.tools/?gh=davidheineman/prolific-deployment-demo/master/crowdsource.yml&PROLIFIC_PID={%PROLIFIC_PID%}&STUDY_ID={%STUDY_ID%}&SESSION_ID={%SESSION_ID%}\n",
      "Your study (with data) will be linked to: https://nlproc.tools/?gh=davidheineman/prolific-deployment-demo/master/crowdsource.yml&PROLIFIC_PID={%PROLIFIC_PID%}&STUDY_ID={%STUDY_ID%}&SESSION_ID={%SESSION_ID%}&d=davidheineman/prolific-deployment-demo/master/crowdsource.json\n"
     ]
    }
   ],
   "source": [
    "# If we skipped the above step, we will use the demo template and data\n",
    "if not template_path and not data_path:\n",
    "    template_path = \"davidheineman/thresh.tools/main/public/templates/demo/crowdsource.yml\"\n",
    "    data_path = \"davidheineman/thresh.tools/main/public/data/demo/crowdsource.json\"\n",
    "\n",
    "study_url = f\"https://thresh.tools/?gh={template_path}\"\n",
    "study_url = study_url + \"&PROLIFIC_PID={%PROLIFIC_PID%}&STUDY_ID={%STUDY_ID%}&SESSION_ID={%SESSION_ID%}\"\n",
    "\n",
    "print(f\"View your hosted template here:\\t\\t  {study_url}\")\n",
    "\n",
    "study_url = f\"{study_url}&d={data_path}\"\n",
    "\n",
    "print(f\"Your study (with data) will be linked to: {study_url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If everything works, you should see this:\n",
    "<div align=\"center\">\n",
    "    <img src=\"../public/img/prolific-gh-hosted.png\" width=\"100%\" style=\"max-width: 400px\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy to Prolific\n",
    "We will deploy entirely through Python, which will help manage data in large-scale annotation scenarios. For details on the Prolific API, see [**docs.prolific.co/docs/api-docs/public**](https://docs.prolific.co/docs/api-docs/public)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will use Dallinger, a popular annotation service management tool\n",
    "# Dallinger also has integrations with other services, such as MTurk\n",
    "!pip install -q dallinger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dallinger.prolific import ProlificService\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To get a developer token, visit your prolific workspace -> settings -> API tokens\n",
    "YOUR_PROLIFIC_API_TOKEN = \"YOUR_API_TOKEN\" \n",
    "\n",
    "# This will read the API token from a local file\n",
    "if os.path.exists('.prolific_token'):\n",
    "    YOUR_PROLIFIC_API_TOKEN = open('.prolific_token', \"r\").read()\n",
    "\n",
    "prolific = ProlificService(\n",
    "    api_token=YOUR_PROLIFIC_API_TOKEN,\n",
    "    api_version=\"v1\",\n",
    "    referer_header=None\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-29 18:39:46,811 - WARNING - Prolific API request: {\"URL\": \"https://api.prolific.co/api/v1/workspaces/\", \"method\": \"GET\", \"args\": {}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1 workspace(s), using My Workspace with id 64c55f10b177aed40ec785a4\n"
     ]
    }
   ],
   "source": [
    "# Prolific annotations are managed by workspaces and grouped by projects.\n",
    "# We will start by finding a workspace to use for annotation.\n",
    "response = prolific._req(\n",
    "    method=\"GET\",\n",
    "    endpoint=f\"/workspaces/\",\n",
    ")\n",
    "workspace_id = response['results'][0]['id']\n",
    "print(f\"Found {response['meta']['count']} workspace(s), using {response['results'][0]['title']} with id {workspace_id}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-29 18:39:49,718 - WARNING - Prolific API request: {\"URL\": \"https://api.prolific.co/api/v1/workspaces/64c55f10b177aed40ec785a4/projects/\", \"method\": \"POST\", \"args\": {\"json\": {\"title\": \"My Demo nlproc.tools Project\"}}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created new project with id 64c5bf65d65dd689969fc570\n"
     ]
    }
   ],
   "source": [
    "# Create a new project for our demo annoations.\n",
    "response = prolific._req(\n",
    "    method=\"POST\",\n",
    "    endpoint=f\"/workspaces/{workspace_id}/projects/\",\n",
    "    json={\n",
    "        \"title\": \"My Demo thresh.tools Project\"\n",
    "    },\n",
    ")\n",
    "project_id = response['id']\n",
    "print(f\"Created new project with id {project_id}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-29 18:40:36,855 - WARNING - Prolific API request: {\"URL\": \"https://api.prolific.co/api/v1/studies/\", \"method\": \"POST\", \"args\": {\"json\": {\"name\": \"My Demo Annotation Task\", \"internal_name\": \"demo-annotation-task\", \"description\": \"In this task, you will use nlproc.tools to perform fine-grained evaluation!\", \"external_study_url\": \"https://nlproc.tools/?gh=davidheineman/prolific-deployment-demo/master/crowdsource.yml&PROLIFIC_PID={%PROLIFIC_PID%}&STUDY_ID={%STUDY_ID%}&SESSION_ID={%SESSION_ID%}&d=davidheineman/prolific-deployment-demo/master/crowdsource.json\", \"prolific_id_option\": \"url_parameters\", \"completion_code\": \"YOUR_COMPLETION_CODE\", \"completion_option\": \"url\", \"total_available_places\": 1, \"estimated_completion_time\": 3, \"maximum_allowed_time\": 10, \"reward\": 4, \"eligibility_requirements\": [], \"status\": \"UNPUBLISHED\", \"device_compatibility\": [\"desktop\"], \"peripheral_requirements\": []}}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created draft study with name My Demo Annotation Task and ID 64c5bf94cf229332e5024cf6.\n"
     ]
    }
   ],
   "source": [
    "# Create a new draft study. A seperate study should be created for every annotation task.\n",
    "\n",
    "# For more information about survey options, refer to:\n",
    "#   https://docs.prolific.co/docs/api-docs/public/#tag/Studies/paths/~1api~1v1~1studies~1/post\n",
    "response = prolific.draft_study(\n",
    "    # Information about the study\n",
    "    name=\"My Demo Annotation Task\",\n",
    "    internal_name=\"demo-annotation-task\",\n",
    "    description=\"\"\"In this task, you will use thresh.tools to perform fine-grained evaluation!\"\"\",\n",
    "    # project=project_id,\n",
    "\n",
    "    # Parameters for your data collection\n",
    "    estimated_completion_time=3,\n",
    "    maximum_allowed_time=10,\n",
    "    reward=8,\n",
    "    total_available_places=1,\n",
    "    eligibility_requirements=[],\n",
    "\n",
    "    # Configuration for our thresh.tools setup\n",
    "    external_study_url=study_url,\n",
    "    completion_code=completion_code,\n",
    "    completion_option=\"url\",\n",
    "    prolific_id_option=\"url_parameters\",\n",
    "    device_compatibility=[\"desktop\"],\n",
    "    peripheral_requirements=[],\n",
    ")\n",
    "study_id = response['id']\n",
    "\n",
    "print(f\"Created draft study with name {response['name']} and ID {study_id}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View your study\n",
    "Your draft study is now public! To view, visit your Prolific dashboard at [**app.prolific.co/researcher/workspaces**](https://app.prolific.co/researcher/workspaces).\n",
    "\n",
    "<div align=\"center\">\n",
    "    <img src=\"../public/img/prolific.png\" width=\"100%\" style=\"max-width: 800px\" />\n",
    "</div>\n",
    "\n",
    "To preview your study, scroll to the bottom and click on \"Preview\":\n",
    "\n",
    "<div align=\"center\">\n",
    "    <img src=\"../public/img/prolific-study.png\" width=\"100%\" style=\"max-width: 800px\" />\n",
    "</div>\n",
    "\n",
    "If we are successful, you will see your deployed interface!\n",
    "\n",
    "<div align=\"center\">\n",
    "    <img src=\"../public/img/prolific-deployed.png\" width=\"100%\" style=\"max-width: 500px\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View your study at: https://app.prolific.co/researcher/workspaces/studies/64c5bf94cf229332e5024cf6\n"
     ]
    }
   ],
   "source": [
    "print(f\"View your study at: https://app.prolific.co/researcher/workspaces/studies/{study_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manage Existing Studies\n",
    "Now we have a working study, let's look at some more utility commands for managing the study"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-29 18:41:16,894 - WARNING - Prolific API request: {\"URL\": \"https://api.prolific.co/api/v1/studies/64c5bf94cf229332e5024cf6/\", \"method\": \"GET\", \"args\": {}}\n",
      "2023-07-29 18:41:17,329 - WARNING - Prolific API request: {\"URL\": \"https://api.prolific.co/api/v1/studies/64c5bf94cf229332e5024cf6/\", \"method\": \"PATCH\", \"args\": {\"json\": {\"total_available_places\": 6}}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The study is now set to be open to 6 participants.\n"
     ]
    }
   ],
   "source": [
    "# We can add more slots to an existing study\n",
    "response = prolific.add_participants_to_study(\n",
    "    study_id=study_id,\n",
    "    number_to_add=5\n",
    ")\n",
    "\n",
    "print(f\"The study is now set to be open to {response['total_available_places']} participants.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When you are ready, this will make the study publically available to participants\n",
    "# prolific.publish_study(study_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-29 18:41:23,195 - WARNING - Prolific API request: {\"URL\": \"https://api.prolific.co/api/v1/studies/\", \"method\": \"GET\", \"args\": {}}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'id': '64c5bf94cf229332e5024cf6',\n",
       "  'title': 'My Demo Annotation Task',\n",
       "  'annotation': 'demo-annotation-task',\n",
       "  'status': 'UNPUBLISHED',\n",
       "  'created': datetime.datetime(2023, 7, 30, 1, 40, 36, 945000, tzinfo=tzutc()),\n",
       "  'expiration': '',\n",
       "  'description': ''}]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This gets all the studies available account-wide\n",
    "prolific.get_hits()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Integrate with Firebase\n",
    "Unfortunately, Prolific has no method of managing or saving annotations, requiring users to bring-their-own database. However, Thresh integrates nicely with a lightweight Firebase database, which is an easy way to set up saving annotations and Prolific study parameters. For more detail please refer to [**Deploy a Database for Annotations**](./deploy_database_with_firebase.md).\n",
    "\n",
    "<div align=\"center\">\n",
    "    <img src=\"../public/img/firebase.png\" width=\"500\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleanup\n",
    "This will delete the resources created in this demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-29 18:41:26,137 - WARNING - Prolific API request: {\"URL\": \"https://api.prolific.co/api/v1/studies/64c5bf94cf229332e5024cf6\", \"method\": \"DELETE\", \"args\": {}}\n",
      "2023-07-29 18:41:26,669 - WARNING - Prolific API request: {\"URL\": \"https://api.prolific.co/api/v1/projects/64c5bf65d65dd689969fc570/\", \"method\": \"PATCH\", \"args\": {\"json\": {}}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully deleted study: 64c5bf94cf229332e5024cf6!\n",
      "Successfully deleted project 64c5bf65d65dd689969fc570!\n",
      "Successfully deleted GitHub repository prolific-deployment-demo!\n"
     ]
    }
   ],
   "source": [
    "# Cleanup our resources\n",
    "response = prolific.delete_study(study_id)\n",
    "if response: print(f\"Successfully deleted study: {study_id}!\")\n",
    "\n",
    "response = prolific._req(\n",
    "    method=\"PATCH\",\n",
    "    endpoint=f\"/projects/{project_id}/\",\n",
    "    json={},\n",
    ")\n",
    "if response: print(f\"Successfully deleted project {project_id}!\")\n",
    "\n",
    "if 'g' in locals(): \n",
    "    g.get_repo(f\"{github_username}/{repository_name}\").delete()\n",
    "    print(f\"Successfully deleted GitHub repository {repository_name}!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced features\n",
    "Refer to the [**Dallinger library**](https://github.com/Dallinger/Dallinger/blob/master/dallinger/prolific.py) for details about the `ProlificService` class. Here are a few other useful commands:\n",
    "\n",
    "- `pay_session_bonus()` - Pay a worker a bonus\n",
    "- `approve_participant_session()` / `get_participant_session()` - Get / approve an annotation\n",
    "\n",
    "Additionally, you can create your own API calls to Prolific using the format:\n",
    "```python\n",
    "response = prolific._req(\n",
    "    method=\"XXX\",               # The HTTP method, e.g., GET, POST\n",
    "    endpoint=f\"/XXX/YYY/\",      # The API endpoint\n",
    "    json={},                    # The request body in JSON format\n",
    ")\n",
    "```\n",
    "For a full list of API calls and options, please refer to [**docs.prolific.co/docs/api-docs/public**](https://docs.prolific.co/docs/api-docs/public)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}