#33512 handle last element out of range error

[itazap]

#33552
fix to handle out of range error

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Thanks! let's add the issues' reproducer as a small test!

[itazap]

@ArthurZucker original issue used stable_whisper and can't reproduce the problem if loading from WhisperTokenizer, not sure if we should add the lib dependency for a stable_whisper test?

[ylacombe]

Hey there, would it be possible to have a snippet to reproduce the issue ? It might actually be an issue with Whisper modeling code rather than on the tokenizer side.

cc @eustlb

[itazap]

@ylacombe @eustlb yes snippet and audio file is here: #33552

or you can run the below:

EDIT: I think I've pinpointed the issue. The first sequence does not end with a token in all_special_ids. Is this a Whisper requirement?

model_outputs = [{'tokens': np.array([[50258, 50259, 50359, 50364,  1468,   380,   976,   385,   300,   286,
           312,  2633,   300,   472,   551,   300,   286,  2378,   380,  2762,
           294,  5680,    13, 20180,   281,  1446,   300,  6944,  4931,   281,
           841,   512,   777,  5503,    13,   583,  1968,   420,   406,   286,
           603,  3270,   689,   286,  2117,    13, 10865,   286,   478,   445,
          1382,   281,  5268,    13,  2432, 40128,   521,   264,   777,  1393,
          3082,   286,   478,   257,  3429,   586,    13,   316, 12232,   295,
           445,   281,  1621,   926,   309,    13, 34695,   271,    13,  5303,
           257,   274,  3019,   281,  1855,   293,  1087,   484,    13,  4055,
           266, 21065,  4570,   286,  4244,   466,    13,   759,   436,   536,
           385,  6588,    13, 30308,   264, 21065,  1626,  9019,   466,    13,
          8503,   286,   478,  2633,   760,   420,  2633,   264,   558,  2372,
            13,   286,   841,   264,   596,   346,    13,   583,   406,  1547,
           281]]), 'token_timestamps': np.array([[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.3200,  0.3400,  0.5000,
          0.6600,  0.8600,  1.0000,  1.2600,  1.5000,  1.7600,  2.0400,  2.4000,
          2.5800,  2.8000,  2.9200,  3.0600,  3.2200,  3.5800,  4.4000,  5.5000,
          5.7800,  5.9200,  6.0800,  6.3000,  6.6400,  6.8000,  6.9800,  7.1600,
          7.4000,  7.6200,  8.1200,  8.2600,  8.4600,  8.7200,  9.0600,  9.1600,
          9.5400,  9.5800,  9.6600,  9.9200, 10.3000, 10.7200, 10.8400, 11.0000,
         11.1400, 11.1600, 11.4400, 11.6400, 11.7800, 12.2800, 12.5400, 12.7400,
         12.8600, 13.0200, 13.1800, 13.4000, 13.6800, 13.9000, 14.1000, 14.2400,
         14.4200, 14.5600, 14.8200, 14.8800, 15.1000, 15.3200, 15.4600, 15.6400,
         15.8200, 15.9800, 16.2800, 16.5000, 16.5800, 16.8000, 17.1400, 17.1400,
         17.4600, 17.6000, 17.7000, 17.7400, 17.8800, 18.1200, 18.3600, 18.5400,
         18.7800, 19.0200, 19.2400, 19.3800, 19.5600, 19.8600, 20.0600, 20.2800,
         20.5800, 20.8000, 20.9400, 21.1600, 21.3200, 21.5600, 21.7400, 21.8600,
         22.1000, 22.3200, 22.5400, 22.7200, 23.1800, 23.5000, 23.7400, 23.8800,
         24.0600, 24.0600, 24.3000, 24.5400, 24.7200, 24.9800, 25.2000, 25.3600,
         25.6200, 25.6600, 25.8000, 26.0600, 26.2600, 26.3400, 26.4800, 26.5200,
         26.7200, 26.8600, 27.0800]]), 'stride': (30.0, 0.0, 5.0)}, {'tokens': np.array([[50258, 50259, 50359, 50363,  1449,   466,   498,   436,   536,   385,
          6588,  3974,   257,  2307,  1626,  9019,   466,  1968,   286,   478,
          2633,  6385,   286,   478,  2633,   264,   558,  2372,   286,   841,
           264,  4588,   457,   406,  1547,   281,   652,   385,   605,   493,
           411,   257, 22209,    13,   286,   478,  2633,   760,  3275,    13,
           492,   434,  2633, 50257, 50257, 50257, 50257, 50257, 50257, 50257,
         50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257,
         50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257,
         50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257,
         50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257,
         50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257,
         50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257,
         50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257, 50257,
         50257]]), 'token_timestamps': np.array([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.2600, 0.6800, 0.9800, 1.2400,
         1.3000, 1.5600, 1.8400, 2.1200, 2.3600, 2.5600, 2.7400, 3.1600, 3.6000,
         3.8800, 4.1000, 4.1200, 4.3200, 4.5800, 4.7600, 4.8400, 4.9000, 5.2000,
         5.3600, 5.6200, 5.8200, 6.0200, 6.2600, 6.4800, 6.7400, 6.8600, 7.0800,
         7.3200, 7.4200, 7.6600, 7.8000, 8.0000, 8.2200, 8.4200, 8.5200, 8.5200,
         8.6800, 8.8200, 9.0000, 9.4800, 9.7000, 9.7600, 9.9000, 9.9600, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800, 9.9800,
         9.9800, 9.9800, 9.9800, 9.9800, 9.9800]]), 'stride': (10.0075, 5.0, 0.0)}]
        # fmt: on

        tokenizer = WhisperTokenizer.from_pretrained("onnx-community/whisper-tiny.en_timestamped")
        result = tokenizer._decode_asr(
            model_outputs, return_timestamps="word", return_language=False, time_precision=0.02
        )

[itazap]

@ylacombe the tokenization code is quite complex here and I'm not familiar with the whisper model much, if you can please advise on what could be wrong in the model_ouputs or what to test , would be grateful! 😊

[ValentinKovalev]

FYI: I have tested whisper and reproduced this issue on the main branch and compared it with the current implementation. It now works correctly.

[felipehertzer]

I have tested this changes and it solve my problem with the issue #33552

[ylacombe]

Hey @itazap, sorry for the late response! Whisper's modeling code is expected to go through a number of modeling changes.

There's one particular change which deals with EOS tokens that were wrongly removed from doing short-form (i.e audio<=30s) Whisper transcription - see #33917 that'll likely be supersede by #34135.

In particular, we also have a pending question from @eustlb: should we keep or remove these special tokens ?

That said, I'm still struggling to understand why the issue you're trying to fix never appeared in our own usage of Whisper, but only in stable-whisper.

I'm also wondering if the issue also appears when doing long-form generation (i.e when audio > 30s) , which doesn't add EOS token at the end (if I remember correctly).

I believe that we should verify a few things, before actually merging this PR:

1. try to reproduce the issue with transformers-only code, to facilitate understanding of the issue. If we can't reproduce it, then we'll have to find out why it happens only in stable-whisper
2. check if the issue still occurs with Fix Whisper shortform EOS #33917 or [WIP] [Whisper] Fix whisper decoding #34135

Depending on the answers to these questions, this PR might not be needed !