Jerky H264 rendering on RK decoder - how to change hw decoder settings - comparison with VLC-Android

[signumnova]

Exoplayer 1.5.1.

I have a particular stream that is H.264 encapsulated in mpeg-ts that was taken off a DVB-T receiver. I saved this stream to a file using VLC without any transcoding etc, and play it in exoplayer on my target device as a file, so the problem I am seeing is clearly nothing to do with network/streaming behaviors.

This particular video does not play smoothly on RK chipset-based device; its keeps freezing every few seconds, but works fine on a Samsung Galaxy phone and on VLC desktop, so it's not an inherent fault in the video itself. Problem doesn't affect audio, so it's purely a video decoder issue. Other MP4 videos play fine though, so I'm not sure what's the exact thing in it that is triggering the poor behavior.

Anyhow, I then thought I would try it on VLC-Android for comparison, and lo and behold it does the same thing. Hence it's a hardware decoder issue of some type.

In VLC for Android there is a preference setting for Hardware Acceleration. which can can be set to "Automatic", "Disabled", "Decoding acceleration" and "Full acceleration". It turns out my problem only manifests with "Full acceleration". If I set it to "Decoding Acceleration" it works fine.

I had a trawl through the VLC-Android source, and their acceleration setting that appears to work successfully does this:

           if (hardwareAcceleration == HW_ACCELERATION_DECODING) {
                media.addOption(":no-mediacodec-dr")
                media.addOption(":no-omxil-dr")
            }

These VLC options in turn influence settings on the decoder through the OMX interface.

So my question is, it seems like I need to do something similar for my exoplayer app. How to achieve this?

Possibly pertinent to this, I noticed that in media3 there is already code that tests for particular hardware codecs and introduces workarounds, such as

private static boolean codecNeedsEosPropagationWorkaround(MediaCodecInfo codecInfo) {
    String name = codecInfo.name;
    return (Util.SDK_INT <= 25 && "OMX.rk.video_decoder.avc".equals(name))

This appears to be the only hw specific reference to this hw decoder in the whole library. Not sure if this relates to my problem, nor why it's apparently only an issue if SDK < 25. May have nothing to do with it).

PS: Just to rule anything to do with the container format I converted to MP4 container but the result was the same.

[microkatz]

Hello @signumnova,

Thank you for reporting your issue and the in-depth debugging!

To note, I looked through the code history of that method and here is the relevant ExoPlayer issue, google/ExoPlayer#6184. Did increasing the limit SDK Level fix your issue? I would be surprised as you are encountering non-smooth playback rather than playback not ending(like in 6184).

As far as the hardware acceleration code comment:

if (hardwareAcceleration == HW_ACCELERATION_DECODING) {
      media.addOption(":no-mediacodec-dr")
      media.addOption(":no-omxil-dr")
} 

I'm not sure exactly what those options for VLC-Android do. But you can setup ExoPlayer to not use that specific decoder in the meantime. Your app can override the MediaCodecVideoRenderer.getDecoderInfos method roughly as follows:

protected List<MediaCodecInfo> getDecoderInfos(
      MediaCodecSelector mediaCodecSelector, Format format, boolean requiresSecureDecoder)
      throws DecoderQueryException {
    List<MediaCodecInfo> decoderInfos = super.getDecoderInfos(mediaCodecSelector, format, requiresSecureDecoder);
    if (format.sampleMimeType.equals(MimeTypes.VIDEO_H264) 
        && want_to_disable_rk) {
        // check if decoderInfos contains that "OMX.rk.video_decoder.avc" decoder
        // if yes, return a copy of the decoderInfos list excluding that decoder
  }

[signumnova]

Hmm.

I don't think that issue you referred to @microkatz is the same one, as that was about a stream ending, and my issue is about what's happening during normal playback. (I'm extremely doubtful that changing that bit of code would help).

As for what VLC is doing there, as far as I can tell it's not deselecting the RK decoder (but if you select "no acceleration" it takes a different path and uses a software decoder). What it is doing is a "reduced acceleration feature mode", turning off a couple of options associated with the hw decoder. I think dr stands for "direct render", which means I think it is disabling some sort of direct decoder to vram pathway. But that's a guess.

Regardless I'd like to be able to pass similar settings through in Exoplayer, but it is very unclear how you would do anything at the OMX layer - I can't see any examples or documentation that discuss this, esp in relation to the RK3399 chip. And I can't even see how VLC is doing this either actually. Those "options" are command line options effectively, that are interpreted deeper down in the vlc code to do certain things a bit differently.

[microkatz]

@signumnova

Yep! I would've been surprised if the method helped as you are encountering non-smooth playback rather than "playback not ending". I was just linking the old issue here as a continuation of the rk decoder requiring workarounds for odd behaviors.

I appreciate the deep dive and investigation. If you are able to identify the exact configuration settings being adjusted by vlc for the rk decoder then we could maybe see if ExoPlayer has the capability to set that configuration down as well.

[signumnova]

@microkatz

The thing is i dont think vlc has any knowledge about the rk decoder specifically.. its just changing some kind of generic internal behavior associated with what happens post decode.

I might have to pose a question in their dev form about this.

(Is there docs somewhere that detail the omx api as any player would typically use?)

Whats odd though is this issue has all the hallmarks of a decoder problem, but it could also be something like a weird frame reordering thing going into the decoder that has a timing aspect to it.

Disabling the acceleration by one degree like vlc ofders is a workaround which may be acceptable.

Disabling the hw codec all together is probably not. Does exoplayer even have software codecs to handle h264 and mpg2 video? How would it handle upscaling? Also i would lose deinterlacing since there is no software imp for that?