PoToken implementation to solve 403 errors #11955
Conversation
|
Now the PR builds fine based on TeamNewPipe/NewPipeExtractor#1247, you can download the APK which uses poTokens! Let us know if you notice any issues. |
| private val TAG = PoTokenWebView::class.simpleName | ||
| private const val GOOGLE_API_KEY = "AIzaSyDyT5W0Jh49F30Pqqtyfdf7pDLFKLJoAnw" | ||
| private const val REQUEST_KEY = "O43z0dpjhgX20SCx4KAo" | ||
| private const val USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.3" |
There was a problem hiding this comment.
Could this be Firefox ESR like in DownloaderImpl?
There was a problem hiding this comment.
No, for some reason it does not work with the Firefox user agent. It would work with the curl user agent though, I don't know why...
|
Would it be possible to move the po token implementation to a library? Currently this is in NewPipe (the app repo) which makes it inaccessible by other apps which also have the need for po tokens. This will lead to a lot of duplicate code because it needs to be implement over and over again for each YT client app. Would be cool if this can be maintained in just one place (and multiple apps could benefit like it is already the case with NewPipeExtractor). |
You can recreate this PR in your own application. This simply connects to the extractor to support the Potoken stream. You will need to do this separately in your application. It should have been like this. |
my point was this would be inefficient If you want to implement this over and over again for each app - sure, go ahead. Keep in mind that this will likely not be "done" after the initial implementation. You will have to update the implementation in many places again. And again. And again... If this was implemented in just one place as a library it would be easier for more developers to share efforts. |
| // an asynchronous function runs in the background and it will eventually call | ||
| // `vmFunctionsCallback`, however we need to manually tell JavaScript to pass | ||
| // control to the things running in the background by interrupting this async | ||
| // function in any way, e.g. with a delay of 1ms. The loop is most probably not | ||
| // needed but is there just because. | ||
| for (let i = 0; i < 10000 && !this.vmFunctions.asyncSnapshotFunction; ++i) { | ||
| await new Promise(f => setTimeout(f, 1)) | ||
| } |
There was a problem hiding this comment.
I … don’t think this is how async works. The timeout is just gonna be scheduled on a new task, but the code before the loop still runs on a microtask on the previous task.
There was a problem hiding this comment.
Yes, but this.vm.a seems to start a standalone task in the background or something like that, and we need to explicitly pass control back to the event loop by pausing this async execution, for the background task to finish executing.
There was a problem hiding this comment.
The loop actually executes only once as far as I know, I still put a loop because you never know
|
Can there be an architecture overview of this somewhere? From a skim of the code I don’t get any idea of what problem this solves or how the solution is structured. |
This will be tried only once, and afterwards an error will be thrown
Let me know which places are not documented enough. |
|
|
@Stypox I think it would be good to include this documentation into the source code somewhere, maybe in the interface module. |
|
So that people who want to understand the code later don’t have to find this PR and looks through lots of issues first |
What is it?
Description of the changes in your PR
General information about poTokens and about this PR structure:
Createto obtain the VM code,GenerateITto obtain the integrity token after running the VM code. See the README here for the detailed steps.PoTokenGeneratoris the base class for all poToken generators. It has a factory method that allows asynchronously obtaining a new instance of aPoTokenGenerator, and then two methods to generate a poToken given a specific identifier, and a method to check if the integrity token has expired.PoTokenWebViewis currently the only implementation ofPoTokenGenerator, but we might want to add other implementations in the future, e.g. ones that do not rely on WebView.PoTokenProviderImplimplements the extractor interface and is supposed to take care of possibly multiplePoTokenGenerators (although right now there is only one based on WebView). It takes care of retrying in case of problems, recreates a newPoTokenGeneratorif the current one expired, and finally returns aPoTokenResult. APoTokenResultcontains two poTokens: one for the specific requested video id (used to fetch the player), and another that can be generated only once as the first thing and is specific to a visitor data (used in streaming urls).TODO:
Obtaining a poToken with something like HtmlUnitnot doable unfortunatelyYou can test whether the poTokens generated work also using the latest yt-dlp commit from their git repo (older commits won't work!), this way (take PLAYER_POT, STREAMING_POT and VISITOR_DATA from logcat):
Fixes the following issue(s)
Relies on the following changes
APK testing
The APK can be found by going to the "Checks" tab below the title. On the left pane, click on "CI", scroll down to "artifacts" and click "app" to download the zip file which contains the debug APK of this PR. You can find more info and a video demonstration on this wiki page.
Due diligence