Skip to content

speech_to_text_finetune.py does not create checkpoints #15799

@ican24

Description

@ican24

Hello,

The finetune script does not create checkpoints in the case of nemotron-3.5-asr-streaming-0.6b.nemo model.
While it creates and keeps the checkpoints, when I use nemotron-speech-streaming-en-0.6b.nemo model with same parameters.
Please help me understand reason and fix it.
Thank you in advance!

python3 examples/asr/speech_to_text_finetune.py \
        --config-path="../asr/conf/fastconformer/cache_aware_streaming" --config-name=fastconformer_transducer_bpe_streaming_prompt.yaml \
        +init_from_nemo_model=stt/nemotron-3.5-asr-streaming-0.6b.nemo \
        ++model.train_ds.manifest_filepath="train" \
        ++model.validation_ds.manifest_filepath="test" \
        ++model.optim.sched.d_model=1024 \
        ++trainer.devices=-1 \
	++trainer.accelerator="gpu" \
        ++trainer.precision=bf16 \
	++trainer.max_epochs=100 \
        ++trainer.limit_train_batches=300000 \
        ++model.train_ds.default_prompt_mode='auto' \
	++model.train_ds.batch_size=4 \
	++model.train_ds.max_duration=20 \
	++model.validation_ds.default_prompt_mode='auto' \
        ++model.optim.name="adamw" \
        ++model.optim.lr=0.1 \
        ++model.optim.weight_decay=0.001 \
        ++model.optim.sched.warmup_steps=2000 \
	++model.encoder.att_context_size='[70, 6]' \
	++exp_manager.version=test \
        ++exp_manager.use_datetime_version=False \
        ++exp_manager.exp_dir=checks

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions