add block error for truncated blocks

[dericed]

This is to supersede the attempted method in #136. To recreate:

ffmpeg -f lavfi -i testsrc -c:v v210 -vframes 1 test.mkv
head -c 200000 test.mkv > broken.mkv
mediaconch -fx -mt -mi broken.mkv 

broken.mkv should end with a truncated Cluster. broken.mkv passes mediaconch when it has an obvious problem.

Last Cluster of mediatrace is:

<block offset="716" name="Cluster" size="25">
    <block offset="716" name="Header" size="12">
      <data offset="716" name="Name">256095861</data>
      <data offset="720" name="Size">215057</data>
    </block>
    <block offset="728" name="CRC-32" size="6">
      <block offset="728" name="Header" size="2">
        <data offset="728" name="Name">63</data>
        <data offset="729" name="Size">4</data>
      </block>
      <data offset="730" name="Value" info="Not tested 2 2261744067">2261744067</data>
    </block>
    <block offset="734" name="Timecode" info="0" size="3">
      <block offset="734" name="Header" size="2">
        <data offset="734" name="Name">103</data>
        <data offset="735" name="Size">1</data>
      </block>
      <data offset="736" name="Data">0</data>
    </block>
  </block>
</block>

I suggest that the Cluster contain an error (also the parent elements would likely get the same error), so new output should look like:

<block offset="716" name="Cluster" error="MKV-TRUNCATED-ELEMENT:1" size="25">
    <block offset="716" name="Header" size="12">
      <data offset="716" name="Name">256095861</data>
      <data offset="720" name="Size">215057</data>
    </block>
    <block offset="728" name="CRC-32" size="6">
      <block offset="728" name="Header" size="2">
        <data offset="728" name="Name">63</data>
        <data offset="729" name="Size">4</data>
      </block>
      <data offset="730" name="Value" info="Not tested 2 2261744067">2261744067</data>
    </block>
    <block offset="734" name="Timecode" info="0" size="3">
      <block offset="734" name="Header" size="2">
        <data offset="734" name="Name">103</data>
        <data offset="735" name="Size">1</data>
      </block>
      <data offset="736" name="Data">0</data>
    </block>
  </block>
</block>

If the approached used in transferrable to other element-based formats (QuickTime, AVI), then I suggest to use a more general error such as "TRUNCATED-ELEMENT:1" rather than "MKV-TRUNCATED-ELEMENT:1"

[JeromeMartinez]

there is already a "IsTruncated" field in MediaInfo report for that, anyway it may be interesting to add it in the trace too.
I currently don't catch truncated sub elements, I'll first add it at the Segment level.

[JeromeMartinez]

was actually easier than expected to have on all elements:

<block offset="47" name="Segment" size="753">
    <block offset="47" name="Header" size="12">
        <data offset="47" name="Name">139690087</data>
        <data offset="51" name="Size" error="TRUNCATED-ELEMENT:1">742</data>
    </block>
    <block>[...]</block>
    <block offset="772" name="Cues" size="28">
        <block offset="772" name="Header" size="12">
            <data offset="772" name="Name">206814059</data>
            <data offset="776" name="Size" error="TRUNCATED-ELEMENT:1">17</data>
        </block>
        <block offset="784" name="CuePoint" size="16">
            <block offset="784" name="Header" size="2">
                <data offset="784" name="Name">59</data>
                <data offset="785" name="Size" error="TRUNCATED-ELEMENT:1">15</data>
            </block>
            <block offset="786" name="CueTime" info="0" size="3">
                <block offset="786" name="Header" size="2">
                    <data offset="786" name="Name">51</data>
                    <data offset="787" name="Size">1</data>
                </block>
                <data offset="788" name="Data">0</data>
            </block>
            <block offset="789" name="CueTrackPositions" size="11">
                <block offset="789" name="Header" size="2">
                    <data offset="789" name="Name">55</data>
                    <data offset="790" name="Size" error="TRUNCATED-ELEMENT:1">10</data>
                </block>
                <block offset="791" name="CueTrack" info="1" size="3">
                    <block offset="791" name="Header" size="2">
                        <data offset="791" name="Name">119</data>
                        <data offset="792" name="Size">1</data>
                    </block>
                    <data offset="793" name="Data">1</data>
                </block>
                <block offset="794" name="CueClusterPosition" info="680" size="4">
                    <block offset="794" name="Header" size="2">
                        <data offset="794" name="Name">113</data>
                        <data offset="795" name="Size">2</data>
                    </block>
                    <data offset="796" name="Data">680</data>
                </block>
                <block offset="798" name="CueRelativePosition" size="2">
                    <block offset="798" name="Header" size="2">
                        <data offset="798" name="Name">112</data>
                        <data offset="799" name="Size" error="TRUNCATED-ELEMENT:1">1</data>
                    </block>
                </block>
            </block>
        </block>
    </block>
</block>

Is it OK?

Note: the test file is not good for unit testing because there are more errors due to your hard truncation. We need to find a way to create a file with only this error.

[dericed]

I am curious about the position of the error. It is on the Element Data Size even when it seems like it is not the VINT of the Element Data Size which is truncated. Why not include this on the Element's block?

[JeromeMartinez]

Good argument.
I as focusing on the position I test, the impact.

But... with error at the element's block, I am afraid about when we'll have several issues related to the Element's block for a similar argument. How will we do? "e1", "e2"... attributes? ;-)

[dericed]

I prefer a delimiter rather than e1 e2. Semicolon?
e="ERRORNAME:ERRORVERSION;ERRORNAME:ERRORVERSION"

[JeromeMartinez]

More difficult on my side, but not so much. fine. but in that case XSL must be ready for that.

[dericed]

Other alternative is:

<block>
  <error/>
  <error/>
</block>

but this method doesn't work for <data>. I suggest e attribute with delimiter is the best option.

[JeromeMartinez]

let's stay with e

[stubhead]

hello,

was this improvement ever integrated? it appears that mediaconch 23.10 windows issues a false-positive for truncation on non-truncated files.

starting from a valid ffv1 video file in matroska container, a new truncated file was created using the unix tool "dd" :

dd if=KMPAC020131_bt709.mkv of=truncated.mkv count=8

MediaInfo correctly identifies the truncated file as truncated ("IsTruncated : Yes") and doesn't complain about the valid file.

MediaConch was then run on the files, trying the following rules :

<rule name="File is truncated" value="IsTruncated" operator="exists" scope="mmt" level="error"/>

or

<rule name="File is truncated" value="IsTruncated" operator="=" scope="mmt" level="error">1</rule>

in both cases MediaConch issued a warning that both files were truncated ("[fail:File is truncated]").

perhaps this is user error (incorrect way to declare the rule), or is this simply not implemented in MediaConch ?