DramaQA dataset is built upon the TV drama Another Miss Oh
and it contains 16,191 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered visual annotations and coreference resolved scripts.
Dual Matching Multistream model effectively learns character-centered representations of video to answer questions about the video.
We use python3 (3.5.2), and python2 is not supported. We use PyTorch (1.1.0), though tensorflow-gpu is necessary to launch tensorboard.
python packages: fire for commandline api
git clone --recurse-submodules (this repo)
cd $REPO_NAME/code
(use python >= 3.5)
pip install -r requirements.txt
python -m nltk.downloader 'punkt'
Place the data folder at $REPO_NAME/data
.
Download DramaQA dataset from here and place the data folder like following structure.
code/
cli.py
train.py
evaluate.py
infer.py
...
data/
AnotherMissOh/
AnotherMissOh_images/
$IMAGE_FOLDERS
AnotherMissOh_QA/
AnotherMissOhQA_train_set.json
AnotherMissOhQA_val_set.json
AnotherMissOhQA_test_set.json
AnotherMissOh_Visual.json
AnotherMissOh_script.json
cd code
python cli.py train
The above command will preprocess textual and visual input before it starts training process. Access the prompted tensorboard port to view basic statistics.
At the end of every epoch, a checkpoint file will be saved on data/ckpt/OPTION_NAMES
For further configurations, take a look at code/config.py
and
fire.
cd code
python cli.py evaluate --ckpt_name=$CKPT_NAME
Substitute CKPT_NAME to your prefered checkpoint file.
e.g. --ckpt_name=model_name_dmm_ckpt_3/loss_0.4818_epoch_15
Challenge participants should submit their outcome with this command.
python cli.py infer --ckpt_name=$CKPT_NAME
The above command will save the outcome at the prompted location.
See the Troubleshooting page and submit a new issue or contact us if you cannot find an answer.
To contact us, send an email to [email protected]