notebook_tutorials/load_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Managing Fine-Grained Data with `thresh`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q thresh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from thresh import load_interface, convert_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Loading\n",
    "When you collect data on thresh.tools, you will eventually have a large amount of `<<data>>.json` files to manage. \n",
    "\n",
    "Instead of having to create dataloaders to verify each interface (which leads to boilerplate code, and can become quite complex), our `thresh` library handles all of this for you!\n",
    "\n",
    "This notebook will demonstrate the basic functionality of `load_interface` and `convert_dataset` to manage your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's begin by downloading the SALSA typology and demo dataset\n",
    "!curl -so salsa.yml https://thresh.tools/templates/salsa.yml\n",
    "!curl -so salsa.json https://thresh.tools/data/salsa.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💃 SALSA\n",
      "Success and FAilure Linguistic Simplification Annotation\n",
      "@article{heineman2023dancing,\n",
      "  title={Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA},\n",
      "  author={Heineman, David and Dou, Yao and Maddela, Mounica and Xu, Wei},\n",
      "  journal={arXiv preprint arXiv:2305.14458},\n",
      "  year={2023}\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# We can load any interface YML file into a Python object\n",
    "SALSA = load_interface(\"salsa.yml\")\n",
    "\n",
    "# Here we can view some features from the interface\n",
    "print(SALSA.template_label)\n",
    "print(SALSA.template_description)\n",
    "print(SALSA.citation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SalsaEntry(\n",
      "  annotator: annotator_1, \n",
      "  system: new-wiki-1/Human-2-written, \n",
      "  target: An important aspect of Fungi in Art is the protection of artwork from fungal damage. || Another important aspect is to make sure mycologists, artists, and society are all on the same page., \n",
      "  source: Further important aspects of Fungi in Art relate to preservation of artworks from fungal decay and contamination, as well as initiatives fostering and supporting works able to stimulate dialogues between mycologists (fungal researchers), artists, and society (as for example from the 'Massee Art Grant' by the British Mycological Society or works encouraged and supported by the Fungi Foundation)., \n",
      "  edits: [DeletionEdit(\n",
      "    input_idx: [[259, 397]], \n",
      "    annotation: DeletionAnnotation(\n",
      "      deletion_type: GoodDeletion(\n",
      "        val: 3\n",
      "      ), \n",
      "      coreference: False, \n",
      "      grammar_error: False\n",
      "    )\n",
      "  ), DeletionEdit(\n",
      "    input_idx: [[216, 237]], \n",
      "    annotation: DeletionAnnotation(\n",
      "      deletion_type: BadDeletion(\n",
      "        val: 1\n",
      "      ), \n",
      "      coreference: False, \n",
      "      grammar_error: False\n",
      "    )\n",
      "  ), DeletionEdit(\n",
      "    input_idx: [[95, 113]], \n",
      "    annotation: DeletionAnnotation(\n",
      "      deletion_type: GoodDeletion(\n",
      "        val: 2\n",
      "      ), \n",
      "      coreference: False, \n",
      "      grammar_error: False\n",
      "    )\n",
      "  ), DeletionEdit(\n",
      "    input_idx: [[125, 172]], \n",
      "    annotation: DeletionAnnotation(\n",
      "      deletion_type: GoodDeletion(\n",
      "        val: 3\n",
      "      ), \n",
      "      coreference: False, \n",
      "      grammar_error: False\n",
      "    )\n",
      "  ), SubstitutionEdit(\n",
      "    input_idx: [[0, 25]], \n",
      "    output_idx: [[0, 19]], \n",
      "    annotation: SubstitutionAnnotation(\n",
      "      substitution_info_change: Same(\n",
      "        val: trivial\n",
      "      ), \n",
      "      grammar_error: False\n",
      "    )\n",
      "  ), SubstitutionEdit(\n",
      "    input_idx: [[42, 64]], \n",
      "    output_idx: [[36, 53]], \n",
      "    annotation: SubstitutionAnnotation(\n",
      "      substitution_info_change: Same(\n",
      "        val: 3\n",
      "      ), \n",
      "      grammar_error: False\n",
      "    )\n",
      "  ), SubstitutionEdit(\n",
      "    input_idx: [[89, 94]], \n",
      "    output_idx: [[77, 84]], \n",
      "    annotation: SubstitutionAnnotation(\n",
      "      substitution_info_change: Same(\n",
      "        val: 3\n",
      "      ), \n",
      "      grammar_error: False\n",
      "    )\n",
      "  ), SubstitutionEdit(\n",
      "    input_idx: [[68, 76]], \n",
      "    output_idx: [[57, 64]], \n",
      "    annotation: SubstitutionAnnotation(\n",
      "      substitution_info_change: Same(\n",
      "        val: trivial\n",
      "      ), \n",
      "      grammar_error: False\n",
      "    )\n",
      "  ), SubstitutionEdit(\n",
      "    input_idx: [[176, 203]], \n",
      "    output_idx: [[119, 128], [163, 188]], \n",
      "    annotation: SubstitutionAnnotation(\n",
      "      substitution_info_change: Same(\n",
      "        val: 3\n",
      "      ), \n",
      "      grammar_error: False\n",
      "    )\n",
      "  )]\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# We can load our JSON data by using our interface object and calling load_annotations()\n",
    "salsa_data = SALSA.load_annotations(\"salsa.json\")\n",
    "\n",
    "# As we can see, it creates a custom object, with custom Edit and Annotation classes according\n",
    "# to the SALSA typology\n",
    "print(salsa_data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new-wiki-1/Human-2-written\n",
      "annotator_1\n",
      "DeletionEdit(\n",
      "  input_idx: [[259, 397]], \n",
      "  annotation: DeletionAnnotation(\n",
      "    deletion_type: GoodDeletion(\n",
      "      val: 3\n",
      "    ), \n",
      "    coreference: False, \n",
      "    grammar_error: False\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# We can view attributes of an entry via direct calls\n",
    "salsa_entry = salsa_data[0]\n",
    "\n",
    "print(salsa_entry.system)\n",
    "print(salsa_entry.annotator)\n",
    "print(salsa_entry.edits[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If we use our interface to create our own dataset, we can export it back to JSON\n",
    "SALSA.export_data(salsa_data, \"salsa_export.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Internal Data Classes\n",
    "\n",
    "The `thresh` has four classes: `Interface`, `Entry`, `Edit` and `Annotation`\n",
    "\n",
    "In this section, we will show how to use the Edit and Annotation classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SalsaEntry(\n",
      "  system: custom_entry, \n",
      "  annotator: None, \n",
      "  source: This is our complex sentence., \n",
      "  target: This is our complex sentence.\n",
      ")\n",
      "This is our complex sentence.\n"
     ]
    }
   ],
   "source": [
    "# Get the Entry class defined by SALSA\n",
    "SalsaEntry = SALSA.get_entry_class()\n",
    "\n",
    "# Create a new Entry object\n",
    "sent = SalsaEntry(\n",
    "    system=\"custom_entry\",\n",
    "    annotator=None,\n",
    "    source=\"This is our complex sentence.\",\n",
    "    target=\"This is our complex sentence.\"\n",
    ")\n",
    "\n",
    "print(sent)\n",
    "\n",
    "# We can get individual entries by directly calling attributes\n",
    "print(sent.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'deletion': thresh.names.DeletionEdit,\n",
       " 'reorder': thresh.names.ReorderEdit,\n",
       " 'insertion': thresh.names.InsertionEdit,\n",
       " 'substitution': thresh.names.SubstitutionEdit,\n",
       " 'split': thresh.names.SplitEdit,\n",
       " 'structure': thresh.names.StructureEdit}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at the classes for each Edit\n",
    "SALSA.edit_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InsertionEdit(\n",
      "  output_idx: [[39, 40]]\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# We can use get_edit_class() to get a specific Edit class\n",
    "InsertionEdit = SALSA.get_edit_class('insertion')\n",
    "\n",
    "# Then, we can define a new Insertion edit\n",
    "edit = InsertionEdit(\n",
    "    output_idx=[[39, 40]]\n",
    ")\n",
    "\n",
    "print(edit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SalsaEntry(\n",
      "  system: custom_entry, \n",
      "  annotator: None, \n",
      "  source: This is our complex sentence., \n",
      "  target: This is our complex sentence., \n",
      "  edits: [SubstitutionEdit(\n",
      "    input_idx: [[1, 4]], \n",
      "    output_idx: [[8, 12]]\n",
      "  ), DeletionEdit(\n",
      "    input_idx: [[1, 5]]\n",
      "  )]\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Let's add some edits to our sentence\n",
    "sent.edits = []\n",
    "\n",
    "DeletionEdit = SALSA.get_edit_class('deletion')\n",
    "SubstitutionEdit = SALSA.get_edit_class('substitution')\n",
    "\n",
    "sent.edits.append(\n",
    "    SubstitutionEdit(\n",
    "        input_idx=[[1, 4]],\n",
    "        output_idx=[[8, 12]],\n",
    "    )\n",
    ")\n",
    "\n",
    "sent.edits.append(\n",
    "    DeletionEdit(\n",
    "        input_idx=[[1, 5]],\n",
    "    )\n",
    ")\n",
    "\n",
    "print(sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'source': 'This is our complex sentence.', 'target': 'This is our complex sentence.', 'metadata': {'annotator': None, 'system': 'custom_entry'}, 'edits': [{'input_idx': [[1, 4]], 'output_idx': [[8, 12]], 'category': 'substitution'}, {'input_idx': [[1, 5]], 'category': 'deletion'}]}]\n"
     ]
    }
   ],
   "source": [
    "# We can now use our typology to convert our data back to a raw format\n",
    "raw_salsa_data = SALSA.to_dict([sent])\n",
    "print(raw_salsa_data)\n",
    "\n",
    "# Feel free to paste this into thresh.tools/?t=salsa to see a vizualization!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can save our data file as well\n",
    "SALSA.export_data(data=[sent], output_filename=\"salsa_custom_export.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One side note: If you want to load, you can simply add the datasets:\n",
    "# SALSA.load_annotations(\"dataset_1.json\") + SALSA.load_annotations(\"dataset_2.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Conversion\n",
    "Our standardized `thresh` data format is universal across fine-grained annotation schemes. To support exisitng data formats, we created dataloaders from external datasets.\n",
    "\n",
    "We support the following datasets:\n",
    "```\n",
    "frank, scarecrow, mqm, snac, propaganda, fg-rlhf, arxivedits\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 21:50:04,909 - INFO - ============================================================\n",
      "2023-08-13 21:50:04,911 - INFO - Dataset name: snac\n",
      "2023-08-13 21:50:04,913 - INFO - Data path: original_snac_data.json\n",
      "2023-08-13 21:50:04,915 - INFO - Reverse flag: False\n",
      "2023-08-13 21:50:04,917 - INFO - Output path (Optional): snac_data_thresh_format.json\n",
      "2023-08-13 21:50:04,919 - INFO - ============================================================\n",
      "2023-08-13 21:50:04,931 - INFO - Converting dataset...\n",
      "2023-08-13 21:50:05,021 - INFO - Done!\n",
      "2023-08-13 21:50:05,023 - INFO - Saving to snac_data_thresh_format.json...\n"
     ]
    }
   ],
   "source": [
    "# First, we'll download the SNaC dataset in its original form\n",
    "!pip install -q gdown\n",
    "import gdown\n",
    "gdown.download(f'https://drive.google.com/uc?id=1ff-pV2sX9XNDMdaPxY7v22T2i0235tcE', 'original_snac_data.json', quiet=True)\n",
    "\n",
    "# Then, we will convert it to our standard format\n",
    "snac_raw = convert_dataset(\n",
    "    dataset_name=\"snac\",\n",
    "    data_path=\"original_snac_data.json\",\n",
    "    output_path='snac_data_thresh_format.json' # <- (Optional) saves data to file\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'source': '',\n",
       " 'target': 'Johnnie, a determined 19 year old girl, works at the cotton mill to support her family. She plans to pay back all the debts they have accumulated.',\n",
       " 'edits': [{'id': 0, 'category': 'CharE', 'output_idx': [[0, 7]], 'votes': 1}]}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The data conversion is now in the JSON format, let's look at an example\n",
    "snac_raw[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SnacEntry(\n",
       "  target: Johnnie, a determined 19 year old girl, works at the cotton mill to support her family. She plans to pay back all the debts they have accumulated., \n",
       "  source: , \n",
       "  edits: [ChareEdit(\n",
       "    output_idx: [[0, 7]], \n",
       "    votes: 1\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To begin using this, we'll download and use the thresh.tools SNaC typology\n",
    "!curl -so snac.yml https://thresh.tools/templates/snac.yml\n",
    "\n",
    "# Now we can load and view in the standardized thresh.tools format!\n",
    "SNaC = load_interface(\"snac.yml\")\n",
    "snac = SNaC.load_annotations(snac_raw)\n",
    "\n",
    "snac[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 21:50:08,286 - INFO - ============================================================\n",
      "2023-08-13 21:50:08,287 - INFO - Dataset name: snac\n",
      "2023-08-13 21:50:08,288 - INFO - Data path: snac_data_thresh_format.json\n",
      "2023-08-13 21:50:08,289 - INFO - Reverse flag: True\n",
      "2023-08-13 21:50:08,289 - INFO - Output path (Optional): original_snac_data.json\n",
      "2023-08-13 21:50:08,290 - INFO - ============================================================\n",
      "2023-08-13 21:50:08,291 - INFO - Converting dataset...\n",
      "2023-08-13 21:50:08,320 - INFO - Saving to original_snac_data.json...\n",
      "2023-08-13 21:50:08,430 - INFO - Done!\n"
     ]
    }
   ],
   "source": [
    "# Conversion to the Thresh platform is not one-way, you can convert data collected using\n",
    "# Thesh and convert it back to the original dataset format with the \"reverse\" flag\n",
    "convert_dataset(\n",
    "    dataset_name=\"snac\",\n",
    "    data_path='snac_data_thresh_format.json',\n",
    "    output_path='original_snac_data.json',\n",
    "    reverse=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples: Loading Demo Interface Data\n",
    "Now that we've seen our data loading and dataset conversion, let's see some examples across different tasks.\n",
    "\n",
    "These cells will load our demo data for each interface as it is seen in `thresh.tools`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultipitEntry(\n",
       "  target: Relax, take a deep breath, and enjoy the moment., \n",
       "  source: Relax, take it easy., \n",
       "  edits: [AddNewEdit(\n",
       "    annotation: None, \n",
       "    output_idx: [[14, 48]]\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -so multipit.yml https://thresh.tools/templates/multipit.yml\n",
    "!curl -so multipit.json https://thresh.tools/data/multipit.json\n",
    "\n",
    "MultiPIT = load_interface(\"multipit.yml\")\n",
    "multipit_data = MultiPIT.load_annotations(\"multipit.json\")\n",
    "\n",
    "multipit_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FrankEntry(\n",
       "  target: inverness caledonian thistle came from behind to beat inverness caledonian thistle in the scottish premiership., \n",
       "  context: Brad McKay crouched to volley in Greg Tansey's deep free-kick early in the match.And Tansey converted a penalty after Massimo Donati had fouled Ross Draper.Accies were upset Ali Crawford was not awarded a second-half spot-kick for a challenge by goalkeeper Ryan Esson but netted late on through Danny Redmond.The gap between Caley Thistle and Motherwell also stands at four points, with Well behind Hamilton on goal difference after losing to Ross County.Media playback is not supported on this deviceThe first-half performance was exactly what Inverness manager Richie Foran has been searching for and came with their backs planted firmly against the wall.They were terrific. Adversity sometimes brings out the best in people, although nerves did seem to take effect after half-time.Foran has said for some time his side just needed one win to get going. They have it and look capable of more as the pressure turns, at least temporarily, to sides above.Tansey, who has agreed a pre-contract to join Aberdeen, was at the heart of the Saturday lunchtime victory. It was his delivery that found McKay ghosting in and the defender's finish was perfect.Another Tansey delivery was rewarded when Donati wrestled Draper to the ground. Tansey took the responsibility and delivered under pressure.In truth, Inverness could have been out of sight by the break.Billy Mckay should have done better from close range twice and Alex Fisher somehow screwed a header wide from a few yards after Remi Matthews had parried Tansey's drive.And, in the second period, Scott Boden could have sealed the win when clean through but dinked over.Martin Canning's side improved significantly after the break as Crawford and Redmond were sent on in place of Donati and Rakish Bingham.And Crawford thought he had earned a lifeline. As Gary Warren tried to shepherd the ball out, Esson came flying out and inexplicably took the substitute down.Referee Andrew Dallas did not point to the spot, leaving Hamilton players in disbelief. It was a huge escape for the home side.Having offered little earlier in the match, other than a great chance for Darian MacKinnon which he prodded wide, the second-half response was firm.Dougie Imrie, Crawford and Greg Docherty all came close but ultimately they did not threaten enough and gave themselves too much to do after leaving themselves repeatedly exposed in defence.Redmond's late finish was little consolation, although the goal may yet have an important part to play in the final make-up of goal difference with so few points separating the sides.Match ends, Inverness CT 2, Hamilton Academical 1.Second Half ends, Inverness CT 2, Hamilton Academical 1.Goal!  Inverness CT 2, Hamilton Academical 1. Daniel Redmond (Hamilton Academical) left footed shot from the centre of the box to the bottom left corner. Assisted by Dougie Imrie.Substitution, Inverness CT. Billy King replaces Jake Mulraney.Corner,  Hamilton Academical. Conceded by Kevin McNaughton.Substitution, Inverness CT. Kevin McNaughton replaces Brad McKay.Attempt saved. Scott McMann (Hamilton Academical) right footed shot from outside the box is saved in the bottom left corner.Delay over. They are ready to continue.Jake Mulraney (Inverness CT) wins a free kick on the right wing.Foul by Dougie Imrie (Hamilton Academical).Foul by Greg Tansey (Inverness CT).Giannis Skondras (Hamilton Academical) wins a free kick on the left wing.Attempt blocked. Henri Anier (Inverness CT) right footed shot from the centre of the box is blocked.Substitution, Hamilton Academical. Eamonn Brophy replaces Greg Docherty.Attempt missed. Scott Boden (Inverness CT) right footed shot from the centre of the box is just a bit too high.Ross Draper (Inverness CT) wins a free kick in the attacking half.Foul by Darian MacKinnon (Hamilton Academical).Attempt missed. Greg Docherty (Hamilton Academical) left footed shot from outside the box is too high.Hand ball by Billy McKay (Inverness CT).Scott Boden (Inverness CT) is shown the yellow card for a bad foul.Foul by Scott Boden (Inverness CT).Greg Docherty (Hamilton Academical) wins a free kick in the attacking half.Attempt missed. Henri Anier (Inverness CT) left footed shot from outside the box is close, but misses to the left.Attempt saved. Ali Crawford (Hamilton Academical) right footed shot from outside the box is saved in the centre of the goal.Foul by Jake Mulraney (Inverness CT).Dougie Imrie (Hamilton Academical) wins a free kick in the defensive half.Corner,  Inverness CT. Conceded by Alejandro D'Acol.Corner,  Inverness CT. Conceded by Scott McMann.Scott Boden (Inverness CT) wins a free kick in the attacking half.Foul by Georgios Sarris (Hamilton Academical).Substitution, Inverness CT. Scott Boden replaces Alex Fisher.Corner,  Hamilton Academical. Conceded by Brad McKay.Ross Draper (Inverness CT) wins a free kick on the right wing.Foul by Dougie Imrie (Hamilton Academical).Greg Tansey (Inverness CT) wins a free kick in the defensive half.Foul by Ali Crawford (Hamilton Academical).Billy McKay (Inverness CT) wins a free kick in the attacking half.Foul by Darian MacKinnon (Hamilton Academical).Delay in match Billy McKay (Inverness CT) because of an injury.Billy McKay (Inverness CT) wins a free kick in the attacking half., \n",
       "  source: premiership bottom side inverness caledonian thistle moved to within four points of opponents hamilton academical with three games to play., \n",
       "  edits: [ReleEdit(\n",
       "    annotation: None\n",
       "  ), OuteEdit(\n",
       "    annotation: None\n",
       "  ), LinkeEdit(\n",
       "    annotation: None\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -so frank.yml https://thresh.tools/templates/frank.yml\n",
    "!curl -so frank.json https://thresh.tools/data/frank.json\n",
    "\n",
    "FRANK = load_interface(\"frank.yml\")\n",
    "frank_data = FRANK.load_annotations(\"frank.json\")\n",
    "\n",
    "frank_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CwzccEntry(\n",
       "  target: Pirmi man iyo ta sinti dwele. Nusabe yo porke pirmi ansina. Kwando iyo keda alegre? Kwando gaha pasa el diya o mes na iyo triste?, \n",
       "  source: , \n",
       "  edits: [UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: cognate_interference\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[0, 5]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[10, 13]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[17, 22]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[23, 28]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: RegularError(\n",
       "          val: Segmentation(\n",
       "            val: space_omission\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[30, 36]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[40, 45]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: cognate_interference\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[46, 51]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[60, 66]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[67, 70]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[71, 75]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[84, 90]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[104, 108]]\n",
       "  ), UnintentionalEdit(\n",
       "    annotation: UnintentionalAnnotation(\n",
       "      error_type: NonRandomErrors(\n",
       "        val: ArbitrarySpelling(\n",
       "          val: Phonogramical(\n",
       "            val: homophone_graphemes\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[118, 121]]\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -so cwzcc.yml https://thresh.tools/templates/cwzcc.yml\n",
    "!curl -so cwzcc.json https://thresh.tools/data/cwzcc.json\n",
    "\n",
    "CWZCC = load_interface(\"cwzcc.yml\")\n",
    "cwzcc_data = CWZCC.load_annotations(\"cwzcc.json\")\n",
    "\n",
    "cwzcc_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ScarecrowEntry(\n",
       "  p: 0.96, \n",
       "  model: gpt3, \n",
       "  temperature: 1.0, \n",
       "  gid: 10001, \n",
       "  frequency_penalty: 0, \n",
       "  id: 1, \n",
       "  source: In the wild, animals display tender moments of affection all the time. Macedonian photographer Goran Anastasovski has spent 15 years honing his skills in wildlife photography so that he can share these touching instances with others., \n",
       "  target: To honor the effort he put into his latest set of photographs, we've gathered some of our favorites for your viewing pleasure. A lion and its cub enjoy a tender moment together. The lion's paws rest on top of the front paws of its cub, who lays its head on its mother. A polar bear embraces her cub. Polar bears care for their cubs until they're about two years old, when the cubs venture out on their own. A kangaroo rubs her cheek on her joey's hand., \n",
       "  edits: [NeedsGoogleEdit(\n",
       "    output_idx: [[300, 366]], \n",
       "    annotation: NeedsGoogleAnnotation(\n",
       "      explanation: I'm not sure if polar bear cubs stay with their mothers until they are two years old I would need to use google to find out. , \n",
       "      needs_google_type: 1\n",
       "    )\n",
       "  ), GrammarUsageEdit(\n",
       "    output_idx: [[236, 239]], \n",
       "    annotation: GrammarUsageAnnotation(\n",
       "      explanation: This \"who\" isn't very clear, it should be something like \"while the cub\", \n",
       "      grammar_usage_type: 2\n",
       "    )\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -so scarecrow.yml https://thresh.tools/templates/scarecrow.yml\n",
    "!curl -so scarecrow.json https://thresh.tools/data/scarecrow.json\n",
    "\n",
    "SCARECROW = load_interface(\"scarecrow.yml\")\n",
    "scarecrow_data = SCARECROW.load_annotations(\"scarecrow.json\")\n",
    "\n",
    "scarecrow_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SnacEntry(\n",
       "  target: It is also said that seeing Anytus pass by, Socrates remarked: \"How proudly the great man steps; he thinks, no doubt, he has performed some great and noble deed in putting me to death, and all because, seeing him deemed worthy of the highest honours of the state, I told him it ill became him to bring up his so in a tan-yard.\" He adds that Homer has ascribed to some at the point of death a power of forecasting things, and he too is minded to utter a prophecy. Once, for a brief space, he associated with the son of Anytus, and he seemed, \n",
       "  source: , \n",
       "  edits: [SceneeEdit(\n",
       "    output_idx: [[0, 42]], \n",
       "    votes: 1\n",
       "  ), GrameEdit(\n",
       "    output_idx: [[264, 326]], \n",
       "    votes: 1\n",
       "  ), ChareEdit(\n",
       "    output_idx: [[28, 34]], \n",
       "    votes: 1\n",
       "  ), ChareEdit(\n",
       "    output_idx: [[341, 346]], \n",
       "    votes: 1\n",
       "  ), GrameEdit(\n",
       "    output_idx: [[526, 539]], \n",
       "    votes: 2\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -so snac.yml https://thresh.tools/templates/snac.yml\n",
    "!curl -so snac.json https://thresh.tools/data/snac.json\n",
    "\n",
    "SNaC = load_interface(\"snac.yml\")\n",
    "snac_data = SNaC.load_annotations(\"snac.json\")\n",
    "\n",
    "snac_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PropagandaEntry(\n",
       "  target: The US Is Blatantly Telling Lies\n",
       "  \n",
       "  It’s no secret that the Trump administration has a strong distaste for Iran.\n",
       "  Iran is one of the only issues on which the U.S. president has remained relatively consistent.\n",
       "  Trump berated the country both before and after taking office.\n",
       "  However, Trump’s anti-Iran strategy goes against the better judgment of even the most anti-Iranian advisors in his staff who don’t want to see the U.S. isolated on the world stage.\n",
       "  Fortunately for Trump, however, he is not alone in his bid to isolate and demonize Iran at all costs.\n",
       "  On December 12, Trump’s ambassador to the U.N., Nikki Haley, gave a grandiose speech demonizing Iran that echoed Colin Powell’s infamous performance before the U.N. in 2003.\n",
       "  Haley’s essential claim was that Saudi Arabia is under attack by missiles supplied to Yemen by the Iranian government and that the world should not sit idly by as this goes on.\n",
       "  “If we do nothing about the missiles fired at Saudi Arabia, we will not be able to stop the violence,” Haley warned.\n",
       "  “There is clear evidence that the missiles that landed on Saudi Arabia come from Iran,” she said.\n",
       "  “The evidence is undeniable.\n",
       "  The weapons might as well have had ‘Made in Iran’ stickers all over it.”\n",
       "  Buy Gold at Discounted Prices\n",
       "  However, even as Haley opened her mouth, many commentators could already identify a number of issues with her speech.\n",
       "  As Common Dream’s Reza Marashi explained:\n",
       "  “Haley cited a UN report in her claim regarding Iranian missile transfers to the Houthis.\n",
       "  Of course, the UN has reached no such conclusion.\n",
       "  Instead, a panel of experts concluded that fired missile fragments show components from an Iranian company, but they have ‘no evidence as to the identity of the broker or supplier.’ Asked about Haley’s claim that Iran is the culprit, Sweden’s ambassador to the UN said, ‘The info I have is less clear.’ Analysts from the U.S. Department of Defense speaking to reporters at Haley’s speech openly acknowledged that they do not know the missiles’ origin.\n",
       "  Perhaps most surreal is the very same UN report cited by Haley also says the missile included a component that was manufactured by an American company.\n",
       "  Did she disingenuously omit that inconvenient bit from her remarks, or fail to read the entire UN report?\n",
       "  The world may never know.”\n",
       "  Regardless of the fact that Haley misrepresented the U.N. report in question, it appears the entire premise of the U.N. report is almost completely incorrect, anyway, according to former inspector Scott Ritter.\n",
       "  Ritter writes:\n",
       "  “The missile debris in question actually contradicts the finding of the UN panel, which held that the missiles launched against Saudi Arabia had been transferred to Yemen in pieces and assembled there by Houthi missile engineers; it is clear that the missiles in question had been in the possession of Yemen well before the Saudi Arabian-led intervention of 2015, and that their source was either Soviet or North Korean.\n",
       "  The modification kits, on the other hand, appear to be of Iranian origin, and were transported to Yemen via Oman.\n",
       "  The UN panel claims not to have any evidence of ‘external missile specialists’ working alongside the Houthi; indeed, the simplicity of the Burkhan 2-H modification concept is such that anyone already familiar with the SCUD-B missile system would be able to implement the required processes without outside assistance.” [emphasis added]\n",
       "  So where did the missiles come from, and who made them?\n",
       "  According to Ritter:\n",
       "  “Rather than the Iranian-manufactured Qiam-1 missiles Haley and the Saudi Arabian government claimed, the debris presented by Haley were of a modified Soviet-manufactured SCUD-B missile; the airframe and engine are original Soviet-made components, and many of the smaller parts on display bear Cyrillic (i.e., Russian) markings.\n",
       "  The transformation to the Burkhan 2-H design required the Houthi engineers to increase the size of the fuel and oxidizer tanks, and lengthen the airframe accordingly.\n",
       "  This is done by cutting the airframe, and welding in place the appropriate segments (this also required that the fuel supply pipe, which passes through the oxidizer tank, be similarly lengthened.)\n",
       "  The difference in quality between the factory welds and the new welds is readily discernable.\n",
       "  The increased fuel supply permits a longer engine burn, which in turn increases the range of the missile.\n",
       "  The Burkhan 2-H uses a smaller warhead than the SCUD B; as such, the guidance and control section had been reconfigured to a smaller diameter, and an inter-stage section added to connect the warhead/guidance section with the main airframe.”\n",
       "  ﻿\n",
       "  Those who have been paying attention to this conflict have been well aware that the U.S. has had little material evidence to link Yemen’s Houthis to Iranian arms suppliers.\n",
       "  In January of this year, a panel of U.N. experts stated:\n",
       "  “The panel has not seen sufficient evidence to confirm any direct large-scale supply of arms from the Government of the Islamic Republic of Iran, although there are indicators that anti-tank guided weapons being supplied to the Houthi or Saleh forces are of Iranian manufacture.” [emphasis added]\n",
       "  Even if Iran were arming the Houthis, Haley’s hypocritical anti-Iran rhetoric doesn’t excuse the U.S. for continuing a foreign policy that essentially armed ISIS through U.S. weapons transfers, or for arming al-Qaeda’s affiliate in Syria, just to name two examples of Washington’s schizophrenic approach to the region.\n",
       "  Why is the U.S. singling out Iran, especially when the Houthi rebels are sworn enemies of al-Qaeda?\n",
       "  The issues here go much deeper than nonsensical hypocrisy.\n",
       "  According to Ritter, the entire debacle has shown that if Saudi Arabia cannot contain the Houthi’s missile capabilities, it cannot possibly hope to take on Iran, which possesses a significantly more advanced military than the Houthis do on their own.\n",
       "  Ritter explains further:\n",
       "  “If a relatively unsophisticated foe such as the Houthi, using Iranian-modified Soviet and North Korean missiles derived from 40-year-old technology, can evade an enemy force using the most modern combat aircraft backed up by the most sophisticated intelligence gathering systems available, and successfully launch ballistic missiles that threaten the political and economic infrastructure of the targeted state, what does that say about the prospects of any U.S.-led coalition taking on the far more advanced mobile missile threats that exist in North Korea and Iran today?\n",
       "  The fact of the matter is that no military anywhere has shown the ability to successfully interdict in any meaningful way a determined opponent armed with mobile ballistic missile capability.\n",
       "  If the Saudi experience in Yemen is to teach us anything, it is that any military plan designed to confront nations such as North Korea, Iran and Russia that are armed with sophisticated mobile ballistic missiles had better count on those capabilities remaining intact throughout any anticipated period of hostility.\n",
       "  No amount of chest-thumping and empty rhetoric by American political and/or military leaders can offset this harsh reality.\n",
       "  This is the critical lesson of Yemen, and the United States would do well to heed it before it tries to foment a crisis based upon trumped-up charges.” [emphasis added]\n",
       "  \n",
       "  , \n",
       "  source: , \n",
       "  edits: [AppealToFearPrejudiceEdit(\n",
       "    output_idx: [[904, 1003]]\n",
       "  ), DoubtEdit(\n",
       "    output_idx: [[2154, 2260]]\n",
       "  ), LoadedLanguageEdit(\n",
       "    output_idx: [[5170, 5182]]\n",
       "  ), DoubtEdit(\n",
       "    output_idx: [[0, 32]]\n",
       "  ), DoubtEdit(\n",
       "    output_idx: [[2380, 2444]]\n",
       "  ), AppealToFearPrejudiceEdit(\n",
       "    output_idx: [[7086, 7235]]\n",
       "  ), LoadedLanguageEdit(\n",
       "    output_idx: [[1183, 1219]]\n",
       "  ), DoubtEdit(\n",
       "    output_idx: [[5443, 5543]]\n",
       "  ), WhataboutismEdit(\n",
       "    output_idx: [[305, 337]]\n",
       "  ), AppealToFearPrejudiceEdit(\n",
       "    output_idx: [[355, 449]]\n",
       "  ), LoadedLanguageEdit(\n",
       "    output_idx: [[5405, 5427]]\n",
       "  ), LoadedLanguageEdit(\n",
       "    output_idx: [[6962, 7085]]\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -so propaganda.yml https://thresh.tools/templates/propaganda.yml\n",
    "!curl -so propaganda.json https://thresh.tools/data/propaganda.json\n",
    "\n",
    "Propaganda = load_interface(\"propaganda.yml\")\n",
    "propaganda_data = Propaganda.load_annotations(\"propaganda.json\")\n",
    "\n",
    "propaganda_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fg-rlhfEntry(\n",
       "  target: Bloom is the second studio album by Australian singer and songwriter Troye Sivan, released on 31 August 2018 through EMI Australia and Capitol Records. The album follows up his 2015 debut studio album, Blue Neighbourhood, and features guest appearances from Gordi and Ariana Grande. The lead single from the album was released on 10 January 2018, accompanied by a music video directed by Grant Singer. The title track, a song about queer desire, was released on 2 May as the third single. American singer Ariana Grande was released on 13 June as the album's fourth single., \n",
       "  context: \n",
       "  \n",
       "  ### Passage Title: Bloom (Troye Sivan album)\n",
       "  Bloom is the second studio album by Australian singer and songwriter Troye Sivan, released on 31 August 2018 through EMI Australia and Capitol Records.\n",
       "  The album follows up his 2015 debut studio album, \"Blue Neighbourhood\", and features guest appearances from Gordi and Ariana Grande.\n",
       "  It was preceded by the release of the singles \"My My My! \", \"The Good Side\", \"Bloom\", \"Dance to This\" and \"Animal\".\n",
       "  At the ARIA Music Awards of 2018, the album was nominated for three awards; Album of the Year, Best Male Artist and Best Pop Release.\n",
       "  \"Bloom\" has been called Sivan's \"sex album\", as well as \"darker\", \"more guitar-driven\" and \"more danceable\" than his previous material.\n",
       "  It has also been described as containing material about defiant gay expression; the first song, \"Seventeen\", is about a sexual experience Sivan had with a man he met on Grindr.\n",
       "  Sivan wrote most of the album with American musician Leland and Canadian musician Allie X.\n",
       "  The album's production was primarily handled by Bram Inscore, Oscar Görres, Oscar Holter and Ariel Rechtshaid.\n",
       "  \"My My My!\"\n",
       "  \n",
       "  ### Passage Title: Bloom (Troye Sivan album)\n",
       "  was released as the lead single from the album on 10 January 2018, and was accompanied by a music video directed by Grant Singer.\n",
       "  The second single \"The Good Side\", was released nine days later.\n",
       "  It is an acoustic track about a breakup, with Sivan explaining that the song is an open letter to an ex-boyfriend.\n",
       "  The title track \"Bloom\", a song about queer desire, was released on 2 May as the third single.\n",
       "  \"Dance to This\", featuring American singer Ariana Grande, was released on 13 June as the album's fourth single.\n",
       "  The fifth single, \"Animal\", was released on 9 August.\n",
       "  On 14 June, Sivan announced a Target special edition would be released featuring two new songs, \"This This\" and \"Running Shoes\".\n",
       "  The album was released on 31 August 2018 by EMI Music Australia and Capitol Records.\n",
       "  Sivan is promoting the album via the Bloom Tour, which began on 21 September in Irving, Texas.\n",
       "  At Metacritic, which assigns a normalised rating out of 100 to reviews from mainstream publications, \"Bloom\" received an average score of 85, based on 15 reviews, indicating \"universal acclaim\".\n",
       "  Giving the album a perfect score, \"The Independent\"s Douglas Greenwood wrote, \"Making perfect pop isn't easy, but Troye Sivan is a star who's done his homework.\n",
       "  \n",
       "  ### Question: When does bloom by troye sivan come out?, \n",
       "  source: Bloom is the second studio album by Australian singer and songwriter Troye Sivan. This album was released on August 31, 2018 through EMI Music Australia and Capitol Records. The album's title track Bloom, a song about anal sex, was promoted using the hashtag ‘#BopsBoutBottoming, which was trending on Twitter. The song Bloom was released on May 2, 2018 as the third single from Sivan's album Bloom. , \n",
       "  edits: [FactualEdit(\n",
       "    annotation: FactualAnnotation(\n",
       "      factual_error: Irrelevant(\n",
       "        val: irrelevant\n",
       "      ), \n",
       "      explanation: None\n",
       "    ), \n",
       "    output_idx: [[283, 401]]\n",
       "  ), FactualEdit(\n",
       "    annotation: FactualAnnotation(\n",
       "      factual_error: Irrelevant(\n",
       "        val: irrelevant\n",
       "      ), \n",
       "      explanation: None\n",
       "    ), \n",
       "    output_idx: [[489, 572]]\n",
       "  ), FactualEdit(\n",
       "    annotation: FactualAnnotation(\n",
       "      factual_error: Irrelevant(\n",
       "        val: irrelevant\n",
       "      ), \n",
       "      explanation: None\n",
       "    ), \n",
       "    output_idx: [[152, 282]]\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -so fg-rlhf.yml https://thresh.tools/templates/fg-rlhf.yml\n",
    "!curl -so fg-rlhf.json https://thresh.tools/data/fg-rlhf.json\n",
    "\n",
    "FGRLHF = load_interface(\"fg-rlhf.yml\")\n",
    "fg_rlhf_data = FGRLHF.load_annotations(\"fg-rlhf.json\")\n",
    "\n",
    "fg_rlhf_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ArxiveditsEntry(\n",
       "  sentence-2-level: 2, \n",
       "  sentence-1-level: 1, \n",
       "  easy-or-hard: easy, \n",
       "  arxiv-id: 1301.2079, \n",
       "  sentence-pair-index: 0, \n",
       "  target: The other one is panel information criteria ( [MATH] ) , corresponding to [MATH] ., \n",
       "  source: The other one is panel information criteria ( [MATH] ) , correspondence with [MATH] , they also have three styles , one of them is : [EQUATION], \n",
       "  edits: [InsertionEdit(\n",
       "    annotation: InsertionAnnotation(\n",
       "      intention: Format(\n",
       "        val: format\n",
       "      )\n",
       "    ), \n",
       "    output_idx: [[81, 83]]\n",
       "  ), DeletionEdit(\n",
       "    annotation: DeletionAnnotation(\n",
       "      intention: Content(\n",
       "        val: content\n",
       "      )\n",
       "    ), \n",
       "    input_idx: [[84, 144]]\n",
       "  ), SubstituteEdit(\n",
       "    annotation: SubstituteAnnotation(\n",
       "      intention: Improve-grammar-typo(\n",
       "        val: improve-grammar-typo\n",
       "      )\n",
       "    ), \n",
       "    input_idx: [[57, 77]], \n",
       "    output_idx: [[57, 74]]\n",
       "  )]\n",
       ")"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -so arxivedits.yml https://thresh.tools/templates/arxivedits.yml\n",
    "!curl -so arxivedits.json https://thresh.tools/data/arxivedits.json\n",
    "\n",
    "ArXivEdits = load_interface(\"arxivedits.yml\")\n",
    "arXiv_edits_data = ArXivEdits.load_annotations(\"arxivedits.json\")\n",
    "\n",
    "arXiv_edits_data[0]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}