Easy Annotation Manual V1.0

This Manual provides tips for quick in-house annotation.

Task

Task 1: Given a question and two answers A and B, decide which answer is provided by humans.

Task 2: Given a question and answers A-E, choose your preferred one for the given question.

º

Data format

Example input format:

csv file with columns: [id*, topic, question, text1*, text2*, gt*],

Example output format:

For Google App Script:

csv file with columns: [id*, topic, question, text1*, text2*, annotation1*, annotation2*,...],

For Label Studio:

csv file with columns: [id*, topic, question, text1*, text2*, anno_result*, gt*], anno_result* is emply.

Note: Columns with asterisks (*) are mandatory. They will be useful for further analysis.

Method

Annotate with Google Form

Tools/Website needed

Google Sheet: Data storage, please make sure the data is in the right format.

Google App Script: Apps Script is a rapid application development platform that makes it fast and easy to create business applications that integrate with Google Workspace

Google Form: Annotation UI.

The core idea is to use Google App Script to read data from Google Sheet and generate annotation in a Google form, then we can distribute the form to complete the annotation.

Step by step

1. Prepare the data as specified in Data format.

2. Upload the data to Google Sheet.

3. Open Google App Script and paste the example code. Please refer to binary_choice.gs file under google_form directory in the github repo,.

4. Let’s do some minior modification. Please change the form_id to the actual one. You might also want to tweak the content column number in sheet.getRange('A:F') if you have customized columns. New annotator results will be recorded in new columns.

5. Run the App Script.

6. Copy the generated form link and start annotation!

7. Annotate Task2 is very similar to Task1, please use multiple_choice.gs and change the column range accordingly if you have multiple model generations.

Annotate with Label Studio

Quick Start

1. Install Label Studio: pip install label-studio

2. Start Label Studio: label-studio start

3. Open Label Studio at http://localhost:8080.

4. Sign up with an email address and password that you create.

5. Click Create to create a project and start labeling data.

6. Name the project and optionally enter a description and select a color.

7. Click Data Import and upload the data files that you want to use.

8. Click Labeling Setup and choose a template and customize the label names for your use case. In this step, please use the setting in the following Label Inference section.

9. Click Save to save your project.

10. Start Annotation!

11. Export result, please choose the format you like, CSV is recommended.

More documentation please refer to: https://labelstud.io/guide/quick_start

Label Inference

Choose custom template and use the following code:

<View>
  <Text name="text" value="$anno_text"/>
  <Choices name="anno_results" toName="text" choice="single-radio">
    <Choice value="A"/>
    <Choice value="B"/>
  </Choices>
</View>

Please also refer to the github repo.