Guide to finetune on custom dataset

[vishalk2999]

I have created a dataset in the following format:

- Dataset_folder
    - videos
        - video1,mp4
        - video2.mp4
    train.json

train.json is in the following format:

[
    {
        "video":"videos/calling.mp4",
        "QA":[{
            "i":"Go through the video and understand the all the actions performed in the video",
            "q":"Describe the video",
            "a":"The person is making phone call and talking on the phone"
        }]
    },
]

How to prepare a custom dataset and what are the changes I need to do in order to train on this custom dataset for stage3 finetuning.

I have set the train_file variable of config_7b_stage3.py to the path of this train.json and i get the following error:

2024-12-07T07:52:41 | __main__: train_file: /home/ubuntu/Custom_Data/train.json
2024-12-07T07:52:41 | __main__: Creating dataset for it
2024-12-07T07:52:41 | dataset.it_dataset: Load json file
Traceback (most recent call last):
  File "/home/ubuntu/Ask-Anything/video_chat2/tasks/train_it.py", line 221, in <module>
    main(cfg)
  File "/home/ubuntu/Ask-Anything/video_chat2/tasks/train_it.py", line 138, in main
    train_loaders, train_media_types = setup_dataloaders(
  File "/home/ubuntu/Ask-Anything/video_chat2/tasks/train_it.py", line 105, in setup_dataloaders
    train_datasets = create_dataset(f"{mode}_train", config)
  File "/home/ubuntu/Ask-Anything/video_chat2/dataset/__init__.py", line 174, in create_dataset
    datasets.append(dataset_cls(**dataset_kwargs))
  File "/home/ubuntu/Ask-Anything/video_chat2/dataset/it_dataset.py", line 37, in __init__
    with open(self.label_file, 'r') as f:
IsADirectoryError: [Errno 21] Is a directory: '/'

Could you please help in understading the steps and changes required to train on a custom dataset

[vishalk2999]

Update

I was able to resolve the issue by changing train_file from

train_file =  /home/ubuntu/Custom_Data/train.json 

to

train_file =[
        "/home/ubuntu/Custom_Data/train.json", 
        "/home/ubuntu/Custom_Data/videos",
        "video",
    ]

and modify the train.json as below:

[
    {
        "video":"calling.mp4",
        "QA":[{
            "i":"Go through the video and understand the all the actions performed in the video",
            "q":"Describe the video",
            "a":"The person is making phone call and talking on the phone"
        }]
    },
]

I get the following issue now.

Traceback (most recent call last):
  File "/home/ubuntu/Ask-Anything/video_chat2/tasks/train_it.py", line 221, in <module>
    main(cfg)
  File "/home/ubuntu/Ask-Anything/video_chat2/tasks/train_it.py", line 157, in main
    ) = setup_model(
  File "/home/ubuntu/Ask-Anything/video_chat2/tasks/shared_utils.py", line 88, in setup_model
    state_dict = checkpoint["model"]
KeyError: 'model'

The model checkpoint was downloaded from here and it does not have the optimizer,scheduler,model keys with it.

Could we just change the following 2 lines

checkpoint = torch.load(config.pretrained_path, map_location="cpu")
state_dict = checkpoint["model"]

to

#checkpoint = torch.load(config.pretrained_path, map_location="cpu")
state_dict = torch.load(config.pretrained_path, map_location="cpu")

[gauravsh0812]

Hi there @vishalk2999
Thanks for the information!
I did change the dataset and the config files accordingly. I am also using a custom dataset to train and fine-tune the model.
I am also using the video QA dataset and I arranged the dataset as per this format: https://drive.google.com/file/d/1C-7xmf42QUEi4ApXTcxBHr5nLvTWXyUi/view

Now when I am trying to run, I am getting this error:

Traceback (most recent call last):
  File "/home/gauravs/githubs/video_chat/tasks/train.py", line 221, in <module>
    main(cfg)
  File "/home/gauravs/githubs/video_chat/tasks/train.py", line 169, in main
    global_step = train(
  File "/home/gauravs/githubs/video_chat/tasks/train.py", line 63, in train
    loss_dict = model(image, text)
  File "/home/gauravs/miniconda3/envs/videochat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gauravs/miniconda3/envs/videochat/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/gauravs/miniconda3/envs/videochat/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])
  File "/home/gauravs/miniconda3/envs/videochat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gauravs/githubs/video_chat/models/videochat_pt.py", line 244, in forward
    text = [t + self.end_sym for t in text_input]
  File "/home/gauravs/githubs/video_chat/models/videochat_pt.py", line 244, in <listcomp>
    text = [t + self.end_sym for t in text_input]
TypeError: unsupported operand type(s) for +: 'dict' and 'str'

The "text_input" over here are the batched QA pairs in the following format:
[
{
'q':['...', '...', '...', ....],
'a':['...', '...', '...', ....]
}
]

AM I MISSING SOMETHING??