[Enhancement][iOS] Support persisting capture across app focus loss

[SamJenkinsTether]

AVPro Movie Capture Version

5.2.4

Which platform is your question for?

iOS

Your question

Our iOS builds using version 5.2.4 are pretty consistently getting a capture canceled error after backgrounding app mid-capture and returning. Can get it pretty consistently after leaving it backgrounded for around 30 seconds, but occasionally will get it even on quick backgrounds. Haven't seen it happen on android, so far.

The error:
Capture cancelled
Error: domain: 1, code: -11847, message: Operation Interrupted

Can I get an idea what this error might mean given the context? I didn't mark this as a bug as I'm assuming (and hoping) its just something in my set up, so happy to add any more relevant information if its needed.

Unity version: 2022.3.25f1

[Chris-RH]

Operation interrupted indicates that the app has lost access to the encoder hardware.

This is something that we need to investigate further.

[SamJenkinsTether]

I see. Let me know if there is any info I can provide to help with this

[SamJenkinsTether]

Hello, wanted to follow up on this. Has there been any update or do you need any info from me?

[MorrisRH]

Related to #411.

[MorrisRH]

I've not been able to reproduce this. There is a good chance that the fix for #411 has also resolved this issue. If you're still seeing this after updating to the next release (5.3.1, which should be released within the next few days) then please let us know.

[Chris-RH]

AVPro Movie Capture version 5.3.1 has been released. Please let us know if it has resolved your issue.

[SamJenkinsTether]

Thank you, will grab latest and try to reproduce. Ill update this thread either way

[Chris-RH]

Hi, have you had a chance to retest yet? :)

[SamJenkinsTether]

Hello Chris,

We just updated to version 5.3.1 and still are noticing the issue. The thing is, when I make a debug build on my local machine via Unity->Xcode, I cannot reproduce the issue. But the issue is 100% reproducible via builds that are downloaded from testflight. For the most part, the difference between my local builds and the build machine builds we build before uploading to testflight are an identical process outside of a few code signing & binary stamping things related to app store. Im trying now to test the build machine builds before they are uploaded to testflight and see if the issue is reproducible so i can narrow down whether the issue occurs only after the testflight upload process. Will report back on that specifically.

That being said, any idea if anything in the testflight process might be causing that specific issue?

[Chris-RH]

Have you been able to do any further testing?

[SamJenkinsTether]

Hi, sorry was out of town until today. Will get back to you this week on any findings re: testflight vs non-testflight.

[SamJenkinsTether]

Hi,

So for some reason, builds that we pull from our build machine before they are uploaded to Testflight are unable to be sideloaded by anyone, we keep getting errors after many different attempts to fix it. This test was just to determine if the backgrounding error occurred only after the Testflight process or rather somewhere in our build process.

Local builds are working fine and do not reproduce that issue and with there being a very small difference between what our local builds do from what our build machine builds do, im inclined to assume this issue only occurs with Testflight builds. For now, since we are unable to do the sideloading test, is there anything that Testflight might do that may cause this error to occur? Or maybe can I get some idea of what actually causes it to see where to look?

[Chris-RH]

Pre testflight. What devices are the test builds (that are working) on and what devices are they not working on when you load them?
Pre testflight: Can you reproduce this on the test devices using a new project and the AVPro Movie Capture demo scene of the same capture type?

[SamJenkinsTether]

1). Pre testflight, which is my local builds, I personally have gotten it working on an iphone 14 pro max. But that same device has the issue when testing with a post-testflight build. As far as I know, all our iphone users at our company have experienced it at some point in the post-testflight builds. Most employees do not have macs, so local builds to confirm won't be easy to test.
2). Since i can only reproduce this issue post-testflight, the best I could do is maybe try the demo scene through the testflight process and see if it occurs again. Testing pre-testflight should work since it works for our project too.

[MorrisRH]

I've reproduced this in the Post-Testflight mode... not sure why TestFlight is making a difference but I'll try and get to the bottom of what's happening.

[MorrisRH]

For released builds (via the AppStore or TestFlight) iOS does not allow asset writing in the background so any currently running captures are automatically cancelled by the system when the app is suspended. This behaviour does not occur when running a local build with Xcode attached.

I'm adding support for limited background handling. You'll get about 20 seconds after the app moves to the background before the capture is stopped. Returning the app to the foreground before this time will continue the capture.

@SamJenkinsTether If you want a slightly different behaviour then let me know and I'll see what I can do.

[SamJenkinsTether]

Hi,

I appreciate you looking into this and glad you were able to reproduce on your end. My fear was that if it was happening on testflight, this could also occur on appstore releases, which looks to be true. The idea of our capture logic in our app is that we are capturing gameplay of head to head matches and allow the users to view their game after to see how the other people played. This helps with cheat determent but also with people learning other styles of play. That being said, is 20 seconds an arbitrary time and can be extended or is that done for a particular reason? We really would like the ability to continue capturing for decent amount of times if users background

[MorrisRH]

iOS currently gives ~30s processing time to background apps. 20s is me being conservative as I have no way to determine the actual time the system is giving. As soon as I request background processing time I have to signal task completion otherwise the app is terminated, something I'm sure your users would like even less. Sadly there is no way to request extra processing time beyond the initial amount.

The choices as I see it are:

1. Keep the current behaviour and cancel the capture
2. Stop the capture as soon as the app is put into the background
3. Allow a little room and give the app 20s before stopping the capture

With option 3 it is possible the system will terminate the app before the time limit expires.

[SamJenkinsTether]

Hmm, unfortunately i don't think any of those solutions can work for our use case. Backgrounding or pausing mid-game and returning is not an uncommon thing our users would do, so we wouldn't want a ton of half captured games in this case. Option # 3 is definitely not worth the risk of app being killed and option # 1 is not great either. Curious about option # 2 though, is there potentially a workaround with this? I mean we do pause upon app background but I'm guessing there is still asset writing that occurs in this case? Is there no way to stop and resume or stop and start but the start appends to the previous one?

[SamJenkinsTether]

Following up about any potential workarounds to this issue

[MorrisRH]

We've added an option to write out the current capture (instead of it being cancelled) when the app moves into the background. This will be enabled by default and will make it into the next release.

Sadly there is no support for appending to a previous capture in Apple's framework, so in order to have the illusion of a single continuous capture you do need to capture individual files and then concatenate them together. We're currently investigating how we can add this to the plugin.

[SamJenkinsTether]

Okay I appreciate it, that would definitely be a helpful addition. If anything does change with the concatenating them together part or if you think theres a chance that could make it into an update, please let us know as well.

[Chris-RH]

AVPro Movie Capture version 5.3.3 has been released. Please let us know if it has not fixed your issue.

[MorrisRH]

The option for saving the current capture when the app loses focus is found in the iOS specific section of the Misc capture settings as shown below:

We're still working on a full solution to persist captures across focus losses.

[SamJenkinsTether]

Great, thank you all for the quick turnaround on this. I'll take a look in the next few weeks and reach out with any concerns.

When you say "you're working on a full solution to persist captures", does this mean still investigating if its possible to do or that its possible to do just need to find best way to implement it? Just wanna know if our solution to concatenating background captures can be temporary or if we should be looking into more long term solutions.

Thanks!

[MorrisRH]

Possible to do, just need to find the best way to implement it. Also need to consider how best to handle edge cases such as if the app is terminated whilst in the background, I'm assuming that the captures up to the point of termination wouldn't want to be lost to the aether.

[SamJenkinsTether]

Was able to take a look at this, this seems to be working as intended for us, thank you again for the quick turnaround. Gonna look into some way to merge the separate videos together in our project, but if you could please let me know if there are any updates around that portion of it in the SDK.