训练代码未报错，但是每个epoch中步数直接跳过

[pengpengzi]

‘
(py37_audio2txt) [www@localhost FunASR-main]$ HYDRA_FULL_ERROR=1 python3 funasr/bin/train.py +model="/home/www/FunASR-main/model_zoo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/" +model_revision="v2.0.2" +train_data_set_list="data/list/audio_datasets.jsonl" +valid_data_set_list="data/list/audio_datasets.jsonl" +dataset_conf.batch_size=6000 +dataset_conf.num_workers=4 +train_conf.max_epoch=5 +output_dir="outputs" +device="cuda"
[2024-02-05 18:53:57,923][root][INFO] - download models from model hub: ms
{'model': 'ContextualParaformer', 'model_conf': {'ctc_weight': 0.0, 'lsm_weight': 0.1, 'length_normalized_loss': True, 'predictor_weight': 1.0, 'predictor_bias': 1, 'sampling_ratio': 0.75, 'inner_dim': 512}, 'encoder': 'SANMEncoder', 'encoder_conf': {'output_size': 512, 'attention_heads': 4, 'linear_units': 2048, 'num_blocks': 50, 'dropout_rate': 0.1, 'positional_dropout_rate': 0.1, 'attention_dropout_rate': 0.1, 'input_layer': 'pe', 'pos_enc_class': 'SinusoidalPositionEncoder', 'normalize_before': True, 'kernel_size': 11, 'sanm_shfit': 0, 'selfattention_layer_type': 'sanm'}, 'decoder': 'ContextualParaformerDecoder', 'decoder_conf': {'attention_heads': 4, 'linear_units': 2048, 'num_blocks': 16, 'dropout_rate': 0.1, 'positional_dropout_rate': 0.1, 'self_attention_dropout_rate': 0.1, 'src_attention_dropout_rate': 0.1, 'att_layer_num': 16, 'kernel_size': 11, 'sanm_shfit': 0}, 'predictor': 'CifPredictorV2', 'predictor_conf': {'idim': 512, 'threshold': 1.0, 'l_order': 1, 'r_order': 1, 'tail_threshold': 0.45}, 'frontend': 'WavFrontend', 'frontend_conf': {'fs': 16000, 'window': 'hamming', 'n_mels': 80, 'frame_length': 25, 'frame_shift': 10, 'lfr_m': 7, 'lfr_n': 6, 'cmvn_file': '/home/www/FunASR-main/model_zoo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/am.mvn'}, 'specaug': 'SpecAugLFR', 'specaug_conf': {'apply_time_warp': False, 'time_warp_window': 5, 'time_warp_mode': 'bicubic', 'apply_freq_mask': True, 'freq_mask_width_range': [0, 30], 'lfr_rate': 6, 'num_freq_mask': 1, 'apply_time_mask': True, 'time_mask_width_range': [0, 12], 'num_time_mask': 1}, 'train_conf': {'accum_grad': 1, 'grad_clip': 5, 'max_epoch': 5, 'val_scheduler_criterion': ['valid', 'acc'], 'best_model_criterion': [['valid', 'acc', 'max']], 'keep_nbest_models': 10, 'log_interval': 50}, 'optim': 'adam', 'optim_conf': {'lr': 0.0005}, 'scheduler': 'warmuplr', 'scheduler_conf': {'warmup_steps': 30000}, 'dataset': 'AudioDataset', 'dataset_conf': {'index_ds': 'IndexDSJsonl', 'batch_sampler': 'DynamicBatchLocalShuffleSampler', 'batch_type': 'example', 'batch_size': 6000, 'max_token_length': 2048, 'buffer_size': 500, 'shuffle': True, 'num_workers': 4}, 'tokenizer': 'CharTokenizer', 'tokenizer_conf': {'unk_symbol': '', 'split_with_space': True, 'token_list': '/home/www/FunASR-main/model_zoo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/tokens.json'}, 'ctc_conf': {'dropout_rate': 0.0, 'ctc_type': 'builtin', 'reduce': True, 'ignore_nan_grad': True}, 'normalize': None, 'init_param': '/home/www/FunASR-main/model_zoo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/model.pt', 'config': '/home/www/FunASR-main/model_zoo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/config.yaml', 'is_training': True, 'model_revision': 'v2.0.2', 'train_data_set_list': 'data/list/audio_datasets.jsonl', 'valid_data_set_list': 'data/list/audio_datasets.jsonl', 'output_dir': 'outputs', 'device': 'cuda', 'model_path': '/home/www/FunASR-main/model_zoo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/'}

tables:

----------- ** dataset_classes ** --------------
| class name | class location |
| AudioDataset | funasr/datasets/audio_datasets/datasets.py:8 |
----------- ** index_ds_classes ** --------------
| class name | class location |
| IndexDSJsonl | funasr/datasets/audio_datasets/index_ds.py:10 |
----------- ** batch_sampler_classes ** --------------
| class name | class location |
| BatchSampler | funasr/datasets/audio_datasets/samplers.py:8 |
----------- ** frontend_classes ** --------------
| class name | class location |
| WavFrontend | funasr/frontends/wav_frontend.py:79 |
| WavFrontendOnline | funasr/frontends/wav_frontend.py:217 |
----------- ** encoder_classes ** --------------
| class name | class location |
| BranchformerEncoder | funasr/models/branchformer/encoder.py:295 |
| ConformerChunkEncoder | funasr/models/bat/conformer_chunk_encoder.py:316 |
| ConformerEncoder | funasr/models/conformer/encoder.py:287 |
| DFSMN | funasr/models/fsmn_vad_streaming/encoder.py:233 |
| EBranchformerEncoder | funasr/models/e_branchformer/encoder.py:178 |
| FSMN | funasr/models/fsmn_vad_streaming/encoder.py:162 |
| SANMEncoder | funasr/models/sanm/encoder.py:162 |
| SANMEncoderChunkOpt | funasr/models/scama/encoder.py:163 |
| SANMVadEncoder | funasr/models/ct_transformer_streaming/encoder.py:149 |
| TransformerEncoder | funasr/models/transformer/encoder.py:140 |
----------- ** predictor_classes ** --------------
| class name | class location |
| CifPredictor | funasr/models/paraformer/cif_predictor.py:16 |
| CifPredictorV2 | funasr/models/paraformer/cif_predictor.py:142 |
| CifPredictorV3 | funasr/models/bicif_paraformer/cif_predictor.py:96 |
----------- ** model_classes ** --------------
| class name | class location |
| BiCifParaformer | funasr/models/bicif_paraformer/model.py:38 |
| Branchformer | funasr/models/branchformer/model.py:7 |
| CAMPPlus | funasr/models/campplus/model.py:31 |
| CTTransformer | funasr/models/ct_transformer/model.py:31 |
| CTTransformerStreaming | funasr/models/ct_transformer_streaming/model.py:28 |
| Conformer | funasr/models/conformer/model.py:9 |
| ContextualParaformer | funasr/models/contextual_paraformer/model.py:44 |
| EBranchformer | funasr/models/e_branchformer/model.py:7 |
| Emotion2vec | funasr/models/emotion2vec/model.py:35 |
| FsmnVADStreaming | funasr/models/fsmn_vad_streaming/model.py:268 |
| MonotonicAligner | funasr/models/monotonic_aligner/model.py:25 |
| Paraformer | funasr/models/paraformer/model.py:27 |
| ParaformerStreaming | funasr/models/paraformer_streaming/model.py:38 |
| SANM | funasr/models/sanm/model.py:14 |
| SCAMA | funasr/models/scama/model.py:39 |
| SeacoParaformer | funasr/models/seaco_paraformer/model.py:46 |
| Transformer | funasr/models/transformer/model.py:21 |
| UniASR | funasr/models/uniasr/model.py:26 |
----------- ** decoder_classes ** --------------
| class name | class location |
| ContextualParaformerDecoder | funasr/models/contextual_paraformer/decoder.py:104 |
| DynamicConvolution2DTransformerDecoder | funasr/models/transformer/decoder.py:589 |
| DynamicConvolutionTransformerDecoder | funasr/models/transformer/decoder.py:528 |
| FsmnDecoder | funasr/models/sanm/decoder.py:199 |
| FsmnDecoderSCAMAOpt | funasr/models/scama/decoder.py:198 |
| LightweightConvolution2DTransformerDecoder | funasr/models/transformer/decoder.py:466 |
| LightweightConvolutionTransformerDecoder | funasr/models/transformer/decoder.py:405 |
| ParaformerSANDecoder | funasr/models/paraformer/decoder.py:530 |
| ParaformerSANMDecoder | funasr/models/paraformer/decoder.py:205 |
| TransformerDecoder | funasr/models/transformer/decoder.py:356 |
----------- ** normalize_classes ** --------------
| class name | class location |
| GlobalMVN | funasr/models/normalize/global_mvn.py:12 |
| UtteranceMVN | funasr/models/normalize/utterance_mvn.py:9 |
----------- ** specaug_classes ** --------------
| class name | class location |
| SpecAug | funasr/models/specaug/specaug.py:15 |
| SpecAugLFR | funasr/models/specaug/specaug.py:105 |
----------- ** tokenizer_classes ** --------------
| class name | class location |
| CharTokenizer | funasr/tokenizer/char_tokenizer.py:11 |
| SentencepiecesTokenizer | funasr/tokenizer/sentencepiece_tokenizer.py:12 |

[2024-02-05 18:53:57,974][root][INFO] - config.yaml is saved to: outputs/config.yaml
[2024-02-05 18:54:00,038][root][INFO] - init_param is not None: ('/home/www/FunASR-main/model_zoo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/model.pt',)
[2024-02-05 18:54:00,038][root][INFO] - Loading pretrained params from /home/www/FunASR-main/model_zoo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/model.pt
[2024-02-05 18:54:02,938][root][WARNING] - distributed is not initialized, only single shard
[2024-02-05 18:54:02,938][root][INFO] - in rank: 0, num of samplers: 2, total_num of samplers across ranks: 2
[2024-02-05 18:54:02,938][root][WARNING] - distributed is not initialized, only single shard
[2024-02-05 18:54:02,938][root][INFO] - in rank: 0, num of samplers: 2, total_num of samplers across ranks: 2
/home/www/.conda/envs/py37_audio2txt/lib/python3.7/site-packages/torch/utils/data/dataloader.py:566: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
cpuset_checked))
[2024-02-05 18:54:02,939][root][WARNING] - distributed is not initialized, only single shard
No checkpoint found at 'outputs/model.pt', starting from scratch
Training Epoch: 1: 0%| | 0/1 [00:00<?, ?it/s]
Training Epoch: 1: 0%| | 0/1 [00:00<?, ?it/s]
Checkpoint saved to outputs/model.pt.ep0
/home/www/.conda/envs/py37_audio2txt/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Training Epoch: 2: 0%| | 0/1 [00:00<?, ?it/s]
Training Epoch: 2: 0%| | 0/1 [00:00<?, ?it/s]
Checkpoint saved to outputs/model.pt.ep1
Training Epoch: 3: 0%| | 0/1 [00:00<?, ?it/s]
Training Epoch: 3: 0%| | 0/1 [00:00<?, ?it/s]
Checkpoint saved to outputs/model.pt.ep2
Training Epoch: 4: 0%| | 0/1 [00:00<?, ?it/s]
Training Epoch: 4: 0%| | 0/1 [00:00<?, ?it/s]
Checkpoint saved to outputs/model.pt.ep3
Training Epoch: 5: 0%| | 0/1 [00:00<?, ?it/s]
Training Epoch: 5: 0%| | 0/1 [00:00<?, ?it/s]
Checkpoint saved to outputs/model.pt.ep4
Training Epoch: 6: 0%| | 0/1 [00:00<?, ?it/s]
Training Epoch: 6: 0%| | 0/1 [00:00<?, ?it/s]
Checkpoint saved to outputs/model.pt.ep5

’这是我的全部日志，为什么会出现 “ 0/1 [00:00<?, ?it/s]”这种情况

[LauraGPT]

这个日志的根因可以确定：你当时使用的 2024-02-05 版本中，DynamicBatchLocalShuffleSampler 丢掉了最后一个未满 batch，所以 DataLoader 报告有 1 个 batch，但实际一个也没有产出。 这不是 CUDA 卡住，也不是模型已经完成了一步训练。

从日志可以直接串起来：

1. num of samplers: 2 表示 JSONL 读到了 2 条样本。
2. 配置里是 batch_type: example、batch_size: 6000，因此 6000 表示 6000 条样本，两条数据只会组成一个未满的尾 batch。
3. 问题发生时的 sampler 的 __len__() 会返回 1，但 __iter__() 只在下一个样本装不下时才 yield batch，循环结束后没有再 yield 尾 batch（当时的源码）。所以这里是 len(dataloader) == 1，实际迭代次数却是 0。
4. 当时 trainer 用这个 len 创建 total=1 的进度条，但训练循环没有收到 batch，optimizer.step() 从未执行；epoch 末仍调用 scheduler.step()，于是正好出现日志里的 “lr_scheduler.step() before optimizer.step()” 警告（训练循环，epoch 末 scheduler）。这些 model.pt.ep* 只是按 epoch 保存，不能证明参数更新过。

另外，你设置了 max_epoch=5，日志却跑了 Epoch 1 到 6，也是当时 range(start_epoch, max_epoch + 1) 的旧版 off-by-one 行为；当前代码已经使用不包含上界的循环（当前实现）。

我用同样的 2 条样本做了最小对照测试：

2024-02-05 sampler, example/6000: reported len = 1, actual batches = []
FunASR 1.3.14, token/6000:       reported len = 1, actual batches = [[0, 1]]
FunASR 1.3.14, example/1:        reported len = 2, actual batches = [[0], [1]]

当前动态 sampler 会显式保留最后一个非空 batch（当前源码）。

建议的恢复方式

1. 升级到当前 PyPI 版本，并使用一个全新的输出目录；不要从这些零步训练 checkpoint 恢复：

python -m pip install -U "funasr==1.3.14"

2. 以当前 contextual Paraformer 官方 finetune.sh 为准。关键参数应是：

++dataset="AudioDatasetHotword"
++dataset_conf.batch_sampler="BatchSampler"
++dataset_conf.batch_type="token"
++dataset_conf.batch_size=6000
++dataset_conf.num_workers=1
++train_conf.log_interval=1
++train_conf.resume=false
++output_dir="outputs-1358-fixed"

官方脚本也是 batch_type="token" 配合 batch_size=6000（完整脚本）。如果你坚持使用 batch_type="example"，那么 batch_size 必须写真实的样本数，例如两条 smoke-test 数据用 1 或 2，不能写 6000；两种 batch 类型的含义见训练参数说明。

进度条刚创建时短暂出现一次 0/1 是正常的；整轮结束仍停在 0/1，并且没有 loss、step 和 optimizer 日志，才表示零步训练。 修复后，两条数据在 token/6000 下应完成 1/1。日志中的 num_workers=4 超过系统建议值 1 是另一个独立风险，因此上面的 smoke test 先设为 1。

最后，两条数据并且训练集与验证集使用同一个 JSONL，只适合验证训练管线，不能用于判断 contextual Paraformer 的微调效果。确认能够稳定完成 step 后，再换成独立且足够规模的 train/valid 数据。