Allow to render effects only on stream

[levs42]

• Feature: allow to render effects only on stream

Description & motivation

This PR follows #1530 and allows to split effects rendering to show them only on the stream and hide them on the preview. This allows things like rendering UIView as effects. The main reason I implemented the split process in the ScreenObject was to avoid having two screen objects and a separate root in the Screen. I'm looking forward for the feedback before I can polish this implementation.

Type of change

• New feature (non-breaking change which adds functionality)

Screenshots:

Preview

Stream

[shogo4405]

I understand that this is a feature you want, but I don't quite understand what kind of experience you want to provide to the end-users. Therefore, I would like to start by discussing this.

1. I think it might be confusing for users if the video in the view differs from the video in the stream. What kind of value are you trying to provide in the first place?
   • For example, I can understand if you want to automatically add a watermark for non-paying users.
   • For instance, it is possible to place text, images, and videos, but do you want to set these separately?
2. I get the impression that you want to use both the front and back cameras together for the view and stream.
   • I would like to know to what extent you want the view and stream to be used in common.

[levs42]

1. The main reason we need this feature is to render UIViews on the stream and display them with UIKit on top of the preview. Normally, it wouldn't be a problem to display them in both places, as each UIView would just overlap its rendering on the preview. However, we use aspect fill mode to display the camera preview, which causes the rendered frame on the preview to shift from the UIView hierarchy frame. Here's a demonstration:

Spoiler

2. Yes, we also want to use both cameras. This case has the same coordinate shifting issue as the first one, so we might switch to aspect fit preview when users want to use both cameras.

[shogo4405]

Thank you for your understanding so far.

Regarding the first requirement, I think the request can be met if we have data from captures of unprocessed devices. From the intention, it seems like you only want the overlay display to be applied to the Stream. I suppose it would be better if effects like sepia or monochrome could also be applied. What do you think?

As for the second requirement, aspect fit might also be a solution. It seems that by preparing two raw CMSampleBuffers and two MTHKViews, the intention could be realized. Of course, the video may look slightly different, but I don't think the end-user would feel any discomfort.

[levs42]

The idea of being able to apply effects to the unprocessed buffer sounds great. It isn't a requirement in our case, so initially, I chose a simple approach without having two effects arrays. Being able to apply effects is a powerful feature; having more effects settings should be great for HK users imo. Also, on a performance note, it looks like having two roots won't create much overhead, and having two roots will make it easier to apply separate effects to both video buffers.

I would appreciate your input on this feature so that I can add any necessary changes into the PR. Alternatively, please feel free to take ownership of this feature and implement it in the way you think is most effective.

For the second requirement, we will attempt to use the suggested approach. Rendering the second camera twice seems too expensive, even though it would look better.

[shogo4405]

I have a general understanding of what you want to achieve. Thank you.

I’d like to return to the discussion about the PR. The unprocessed CMSampleBuffer can be obtained in the following delegate:

public protocol IOStreamDelegate: AnyObject {
    /// Tells the receiver a video buffer is incoming.
    func stream(_ stream: IOStream, track: UInt8, didInput buffer: CMSampleBuffer)
}

As we are discussing in this GitHub thread, I believe we can solve this issue using the CMSampleBuffer, but do you anticipate any problems?

[levs42]

It might be not enough to use the delegate method and enqueue buffers to IOStreamView directly without attaching stream to the view. attachStream leads to self.mixer.session.startRunning() call. Is there a way to call startRunning from outside of HK?

[shogo4405]

Unfortunately, in the current version, this feature is not available.

In cases where the view is not attached, startRunning() is triggered when you start publishing, which is suitable for audio streaming.
If calling session.startRunning() is an issue, I can add an API for that. Since features like audio mixing have been introduced, I thought it would be natural to have a method to manually call session.startRunning().

I think it would be helpful to have APIs like IOStream.startCapture(), stopCapture(), and isCapturing().

[levs42]

Yes startCapture() should be sufficient. I modified the sample app and enqueued the samples to the view manually, it worked fine.

[levs42]

Will try to implement the suggested solution and reopen the PR if some changes are required. Thanks for help!