-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
[YouTube] Add support for poTokens #11955
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 15 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
690b341
Interfaces for poTokens + WebView implementation
Stypox 6010c4e
Connect poToken generation to extractor
Stypox 3bdae81
Fix checkstyle
Stypox 0066b32
Unify running on main thread
Stypox f856bd9
Recreate poToken generator if current is broken
Stypox 2b183a0
Wrap logs in BuildConfig.DEBUG
Stypox e7fe84f
Make sure downloadAndRunBotguard() is called after <script> loaded
Stypox 46d0bc1
Update NewPipeExtractor
AudricV b8e050f
Adapt YoutubeHttpDataSource to extractor changes and improve requests
AudricV 70ff47b
[YouTube] Get visitorData from the service to get valid responses
AudricV ecd3f6c
[YouTube] Clarify BotGuard API key's origin and disable related Sonar…
AudricV a60bb3e
[YouTube] Change BotGuard endpoint to youtube.com's one
AudricV 056809c
Use "this" instead of "globalThis" as global scope
Stypox 3fc4873
Use Runnable instead of () -> Unit if converted to Runnable anyway
Stypox 21df24a
Detect when WebView is broken and return null poToken
Stypox 53b599b
Make JavaScript code compatible with older WebViews
Stypox 87317c6
Reorder functions in PoTokenWebView
Stypox b62a09b
Use WebSettingsCompat.setSafeBrowsingEnabled
Stypox dbee8d8
Update NewPipeExtractor to v0.24.5
Stypox File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,211 @@ | ||
| <!DOCTYPE html> | ||
| <html lang="en"><head><title></title><script> | ||
| class BotGuardClient { | ||
| constructor(options) { | ||
| this.userInteractionElement = options.userInteractionElement; | ||
| this.vm = options.globalObj[options.globalName]; | ||
| this.program = options.program; | ||
| this.vmFunctions = {}; | ||
| this.syncSnapshotFunction = null; | ||
| } | ||
|
|
||
| /** | ||
| * Factory method to create and load a BotGuardClient instance. | ||
| * @param options - Configuration options for the BotGuardClient. | ||
| * @returns A promise that resolves to a loaded BotGuardClient instance. | ||
| */ | ||
| static async create(options) { | ||
| return await new BotGuardClient(options).load(); | ||
| } | ||
|
|
||
| async load() { | ||
| if (!this.vm) | ||
| throw new Error('[BotGuardClient]: VM not found in the global object'); | ||
|
|
||
| if (!this.vm.a) | ||
| throw new Error('[BotGuardClient]: Could not load program'); | ||
|
|
||
| const vmFunctionsCallback = ( | ||
| asyncSnapshotFunction, | ||
| shutdownFunction, | ||
| passEventFunction, | ||
| checkCameraFunction | ||
| ) => { | ||
| this.vmFunctions = { | ||
| asyncSnapshotFunction: asyncSnapshotFunction, | ||
| shutdownFunction: shutdownFunction, | ||
| passEventFunction: passEventFunction, | ||
| checkCameraFunction: checkCameraFunction | ||
| }; | ||
| }; | ||
|
|
||
| try { | ||
| this.syncSnapshotFunction = await this.vm.a(this.program, vmFunctionsCallback, true, this.userInteractionElement, () => {/** no-op */ }, [ [], [] ])[0]; | ||
| } catch (error) { | ||
| throw new Error(`[BotGuardClient]: Failed to load program (${error.message})`); | ||
| } | ||
|
|
||
| // an asynchronous function runs in the background and it will eventually call | ||
| // `vmFunctionsCallback`, however we need to manually tell JavaScript to pass | ||
| // control to the things running in the background by interrupting this async | ||
| // function in any way, e.g. with a delay of 1ms. The loop is most probably not | ||
| // needed but is there just because. | ||
| for (let i = 0; i < 10000 && !this.vmFunctions.asyncSnapshotFunction; ++i) { | ||
| await new Promise(f => setTimeout(f, 1)) | ||
| } | ||
|
|
||
| return this; | ||
| } | ||
|
|
||
| /** | ||
| * Takes a snapshot asynchronously. | ||
| * @returns The snapshot result. | ||
| * @example | ||
| * ```ts | ||
| * const result = await botguard.snapshot({ | ||
| * contentBinding: { | ||
| * c: "a=6&a2=10&b=SZWDwKVIuixOp7Y4euGTgwckbJA&c=1729143849&d=1&t=7200&c1a=1&c6a=1&c6b=1&hh=HrMb5mRWTyxGJphDr0nW2Oxonh0_wl2BDqWuLHyeKLo", | ||
| * e: "ENGAGEMENT_TYPE_VIDEO_LIKE", | ||
| * encryptedVideoId: "P-vC09ZJcnM" | ||
| * } | ||
| * }); | ||
| * | ||
| * console.log(result); | ||
| * ``` | ||
| */ | ||
| async snapshot(args) { | ||
| return new Promise((resolve, reject) => { | ||
| if (!this.vmFunctions.asyncSnapshotFunction) | ||
| return reject(new Error('[BotGuardClient]: Async snapshot function not found')); | ||
|
|
||
| this.vmFunctions.asyncSnapshotFunction((response) => resolve(response), [ | ||
| args.contentBinding, | ||
| args.signedTimestamp, | ||
| args.webPoSignalOutput, | ||
| args.skipPrivacyBuffer | ||
| ]); | ||
| }); | ||
| } | ||
| } | ||
| /** | ||
| * Parses the challenge data from the provided response data. | ||
| */ | ||
| function parseChallengeData(rawData) { | ||
| let challengeData = []; | ||
|
|
||
| if (rawData.length > 1 && typeof rawData[1] === 'string') { | ||
| const descrambled = descramble(rawData[1]); | ||
| challengeData = JSON.parse(descrambled || '[]'); | ||
| } else if (rawData.length && typeof rawData[0] === 'object') { | ||
| challengeData = rawData[0]; | ||
| } | ||
|
|
||
| const [ messageId, wrappedScript, wrappedUrl, interpreterHash, program, globalName, , clientExperimentsStateBlob ] = challengeData; | ||
|
|
||
| const privateDoNotAccessOrElseSafeScriptWrappedValue = Array.isArray(wrappedScript) ? wrappedScript.find((value) => value && typeof value === 'string') : null; | ||
| const privateDoNotAccessOrElseTrustedResourceUrlWrappedValue = Array.isArray(wrappedUrl) ? wrappedUrl.find((value) => value && typeof value === 'string') : null; | ||
|
|
||
| return { | ||
| messageId, | ||
| interpreterJavascript: { | ||
| privateDoNotAccessOrElseSafeScriptWrappedValue, | ||
| privateDoNotAccessOrElseTrustedResourceUrlWrappedValue | ||
| }, | ||
| interpreterHash, | ||
| program, | ||
| globalName, | ||
| clientExperimentsStateBlob | ||
| }; | ||
| } | ||
|
|
||
| /** | ||
| * Descrambles the given challenge data. | ||
| */ | ||
| function descramble(scrambledChallenge) { | ||
| const buffer = base64ToU8(scrambledChallenge); | ||
| if (buffer.length) | ||
| return new TextDecoder().decode(buffer.map((b) => b + 97)); | ||
| } | ||
|
|
||
| const base64urlCharRegex = /[-_.]/g; | ||
|
|
||
| const base64urlToBase64Map = { | ||
| '-': '+', | ||
| _: '/', | ||
| '.': '=' | ||
| }; | ||
|
|
||
| function base64ToU8(base64) { | ||
| let base64Mod; | ||
|
|
||
| if (base64urlCharRegex.test(base64)) { | ||
| base64Mod = base64.replace(base64urlCharRegex, function (match) { | ||
| return base64urlToBase64Map[match]; | ||
| }); | ||
| } else { | ||
| base64Mod = base64; | ||
| } | ||
|
|
||
| base64Mod = atob(base64Mod); | ||
|
|
||
| return new Uint8Array( | ||
| [ ...base64Mod ].map( | ||
| (char) => char.charCodeAt(0) | ||
| ) | ||
| ); | ||
| } | ||
|
|
||
| function u8ToBase64(u8, base64url = false) { | ||
| const result = btoa(String.fromCharCode(...u8)); | ||
|
|
||
| if (base64url) { | ||
| return result | ||
| .replace(/\+/g, '-') | ||
| .replace(/\//g, '_'); | ||
| } | ||
|
|
||
| return result; | ||
| } | ||
|
|
||
| async function runBotGuard(rawChallengeData) { | ||
| const challengeData = parseChallengeData(rawChallengeData) | ||
| const interpreterJavascript = challengeData.interpreterJavascript.privateDoNotAccessOrElseSafeScriptWrappedValue; | ||
|
|
||
| if (interpreterJavascript) { | ||
| new Function(interpreterJavascript)(); | ||
| } else throw new Error('Could not load VM'); | ||
|
|
||
| const botguard = await BotGuardClient.create({ | ||
| globalName: challengeData.globalName, | ||
| globalObj: this, | ||
| program: challengeData.program | ||
| }); | ||
|
|
||
| const webPoSignalOutput = []; | ||
| const botguardResponse = await botguard.snapshot({ webPoSignalOutput }); | ||
| return { webPoSignalOutput, botguardResponse } | ||
| } | ||
|
|
||
| async function obtainPoToken(webPoSignalOutput, integrityTokenResponse, identifier) { | ||
| const integrityToken = integrityTokenResponse[0]; | ||
| const getMinter = webPoSignalOutput[0]; | ||
|
|
||
| if (!getMinter) | ||
| throw new Error('PMD:Undefined'); | ||
|
|
||
| const mintCallback = await getMinter(base64ToU8(integrityToken)); | ||
|
|
||
| if (!(mintCallback instanceof Function)) | ||
| throw new Error('APF:Failed'); | ||
|
|
||
| const result = await mintCallback(new TextEncoder().encode(identifier)); | ||
|
|
||
| if (!result) | ||
| throw new Error('YNJ:Undefined'); | ||
|
|
||
| if (!(result instanceof Uint8Array)) | ||
| throw new Error('ODM:Invalid'); | ||
|
|
||
| return u8ToBase64(result, true); | ||
| } | ||
| </script></head><body></body></html> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
13 changes: 13 additions & 0 deletions
13
app/src/main/java/org/schabi/newpipe/util/potoken/PoTokenException.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| package org.schabi.newpipe.util.potoken | ||
|
|
||
| class PoTokenException(message: String) : Exception(message) | ||
|
|
||
| // to be thrown if the WebView provided by the system is broken | ||
| class BadWebViewException(message: String) : Exception(message) | ||
|
|
||
| fun buildExceptionForJsError(error: String): Exception { | ||
| return if (error.contains("SyntaxError")) | ||
| BadWebViewException(error) | ||
| else | ||
| PoTokenException(error) | ||
| } |
35 changes: 35 additions & 0 deletions
35
app/src/main/java/org/schabi/newpipe/util/potoken/PoTokenGenerator.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| package org.schabi.newpipe.util.potoken | ||
|
|
||
| import android.content.Context | ||
| import io.reactivex.rxjava3.core.Single | ||
| import java.io.Closeable | ||
|
|
||
| /** | ||
| * This interface was created to allow for multiple methods to generate poTokens in the future (e.g. | ||
| * via WebView and via a local DOM implementation) | ||
| */ | ||
| interface PoTokenGenerator : Closeable { | ||
| /** | ||
| * Generates a poToken for the provided identifier, using the `integrityToken` and | ||
| * `webPoSignalOutput` previously obtained in the initialization of [PoTokenWebView]. Can be | ||
| * called multiple times. | ||
| */ | ||
| fun generatePoToken(identifier: String): Single<String> | ||
|
AudricV marked this conversation as resolved.
|
||
|
|
||
| /** | ||
| * @return whether the `integrityToken` is expired, in which case all tokens generated by | ||
| * [generatePoToken] will be invalid | ||
| */ | ||
| fun isExpired(): Boolean | ||
|
|
||
| interface Factory { | ||
| /** | ||
| * Initializes a [PoTokenGenerator] by loading the BotGuard VM, running it, and obtaining | ||
| * an `integrityToken`. Can then be used multiple times to generate multiple poTokens with | ||
| * [generatePoToken]. | ||
| * | ||
| * @param context used e.g. to load the HTML asset or to instantiate a WebView | ||
| */ | ||
| fun newPoTokenGenerator(context: Context): Single<PoTokenGenerator> | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I … don’t think this is how async works. The timeout is just gonna be scheduled on a new task, but the code before the loop still runs on a microtask on the previous task.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but
this.vm.aseems to start a standalone task in the background or something like that, and we need to explicitly pass control back to the event loop by pausing this async execution, for the background task to finish executing.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The loop actually executes only once as far as I know, I still put a loop because you never know