-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
[YouTube] Add support for poTokens #11955
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
690b341
Interfaces for poTokens + WebView implementation
Stypox 6010c4e
Connect poToken generation to extractor
Stypox 3bdae81
Fix checkstyle
Stypox 0066b32
Unify running on main thread
Stypox f856bd9
Recreate poToken generator if current is broken
Stypox 2b183a0
Wrap logs in BuildConfig.DEBUG
Stypox e7fe84f
Make sure downloadAndRunBotguard() is called after <script> loaded
Stypox 46d0bc1
Update NewPipeExtractor
AudricV b8e050f
Adapt YoutubeHttpDataSource to extractor changes and improve requests
AudricV 70ff47b
[YouTube] Get visitorData from the service to get valid responses
AudricV ecd3f6c
[YouTube] Clarify BotGuard API key's origin and disable related Sonar…
AudricV a60bb3e
[YouTube] Change BotGuard endpoint to youtube.com's one
AudricV 056809c
Use "this" instead of "globalThis" as global scope
Stypox 3fc4873
Use Runnable instead of () -> Unit if converted to Runnable anyway
Stypox 21df24a
Detect when WebView is broken and return null poToken
Stypox 53b599b
Make JavaScript code compatible with older WebViews
Stypox 87317c6
Reorder functions in PoTokenWebView
Stypox b62a09b
Use WebSettingsCompat.setSafeBrowsingEnabled
Stypox dbee8d8
Update NewPipeExtractor to v0.24.5
Stypox File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| <!DOCTYPE html> | ||
| <html lang="en"><head><title></title><script> | ||
| /** | ||
| * Factory method to create and load a BotGuardClient instance. | ||
| * @param options - Configuration options for the BotGuardClient. | ||
| * @returns A promise that resolves to a loaded BotGuardClient instance. | ||
| */ | ||
| function loadBotGuard(challengeData) { | ||
| this.vm = this[challengeData.globalName]; | ||
| this.program = challengeData.program; | ||
| this.vmFunctions = {}; | ||
| this.syncSnapshotFunction = null; | ||
|
|
||
| if (!this.vm) | ||
| throw new Error('[BotGuardClient]: VM not found in the global object'); | ||
|
|
||
| if (!this.vm.a) | ||
| throw new Error('[BotGuardClient]: Could not load program'); | ||
|
|
||
| const vmFunctionsCallback = function ( | ||
| asyncSnapshotFunction, | ||
| shutdownFunction, | ||
| passEventFunction, | ||
| checkCameraFunction | ||
| ) { | ||
| this.vmFunctions = { | ||
| asyncSnapshotFunction: asyncSnapshotFunction, | ||
| shutdownFunction: shutdownFunction, | ||
| passEventFunction: passEventFunction, | ||
| checkCameraFunction: checkCameraFunction | ||
| }; | ||
| }; | ||
|
|
||
| this.syncSnapshotFunction = this.vm.a(this.program, vmFunctionsCallback, true, this.userInteractionElement, function () {/** no-op */ }, [ [], [] ])[0] | ||
|
|
||
| // an asynchronous function runs in the background and it will eventually call | ||
| // `vmFunctionsCallback`, however we need to manually tell JavaScript to pass | ||
| // control to the things running in the background by interrupting this async | ||
| // function in any way, e.g. with a delay of 1ms. The loop is most probably not | ||
| // needed but is there just because. | ||
| return new Promise(function (resolve, reject) { | ||
| i = 0 | ||
| refreshIntervalId = setInterval(function () { | ||
| if (!!this.vmFunctions.asyncSnapshotFunction) { | ||
| resolve(this) | ||
| clearInterval(refreshIntervalId); | ||
| } | ||
| if (i >= 10000) { | ||
| reject("asyncSnapshotFunction is null even after 10 seconds") | ||
| clearInterval(refreshIntervalId); | ||
| } | ||
| i += 1; | ||
| }, 1); | ||
| }) | ||
| } | ||
|
|
||
| /** | ||
| * Takes a snapshot asynchronously. | ||
| * @returns The snapshot result. | ||
| * @example | ||
| * ```ts | ||
| * const result = await botguard.snapshot({ | ||
| * contentBinding: { | ||
| * c: "a=6&a2=10&b=SZWDwKVIuixOp7Y4euGTgwckbJA&c=1729143849&d=1&t=7200&c1a=1&c6a=1&c6b=1&hh=HrMb5mRWTyxGJphDr0nW2Oxonh0_wl2BDqWuLHyeKLo", | ||
| * e: "ENGAGEMENT_TYPE_VIDEO_LIKE", | ||
| * encryptedVideoId: "P-vC09ZJcnM" | ||
| * } | ||
| * }); | ||
| * | ||
| * console.log(result); | ||
| * ``` | ||
| */ | ||
| function snapshot(args) { | ||
| return new Promise(function (resolve, reject) { | ||
| if (!this.vmFunctions.asyncSnapshotFunction) | ||
| return reject(new Error('[BotGuardClient]: Async snapshot function not found')); | ||
|
|
||
| this.vmFunctions.asyncSnapshotFunction(function (response) { resolve(response) }, [ | ||
| args.contentBinding, | ||
| args.signedTimestamp, | ||
| args.webPoSignalOutput, | ||
| args.skipPrivacyBuffer | ||
| ]); | ||
| }); | ||
| } | ||
|
|
||
| function runBotGuard(challengeData) { | ||
| const interpreterJavascript = challengeData.interpreterJavascript.privateDoNotAccessOrElseSafeScriptWrappedValue; | ||
|
|
||
| if (interpreterJavascript) { | ||
| new Function(interpreterJavascript)(); | ||
| } else throw new Error('Could not load VM'); | ||
|
|
||
| const webPoSignalOutput = []; | ||
| return loadBotGuard({ | ||
| globalName: challengeData.globalName, | ||
| globalObj: this, | ||
| program: challengeData.program | ||
| }).then(function (botguard) { | ||
| return botguard.snapshot({ webPoSignalOutput: webPoSignalOutput }) | ||
| }).then(function (botguardResponse) { | ||
| return { webPoSignalOutput: webPoSignalOutput, botguardResponse: botguardResponse } | ||
| }) | ||
| } | ||
|
|
||
| function obtainPoToken(webPoSignalOutput, integrityToken, identifier) { | ||
| const getMinter = webPoSignalOutput[0]; | ||
|
|
||
| if (!getMinter) | ||
| throw new Error('PMD:Undefined'); | ||
|
|
||
| const mintCallback = getMinter(integrityToken); | ||
|
|
||
| if (!(mintCallback instanceof Function)) | ||
| throw new Error('APF:Failed'); | ||
|
|
||
| const result = mintCallback(identifier); | ||
|
|
||
| if (!result) | ||
| throw new Error('YNJ:Undefined'); | ||
|
|
||
| if (!(result instanceof Uint8Array)) | ||
| throw new Error('ODM:Invalid'); | ||
|
|
||
| return result; | ||
| } | ||
| </script></head><body></body></html> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
113 changes: 113 additions & 0 deletions
113
app/src/main/java/org/schabi/newpipe/util/potoken/JavaScriptUtil.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| package org.schabi.newpipe.util.potoken | ||
|
|
||
| import com.grack.nanojson.JsonObject | ||
| import com.grack.nanojson.JsonParser | ||
| import com.grack.nanojson.JsonWriter | ||
| import okio.ByteString.Companion.decodeBase64 | ||
| import okio.ByteString.Companion.toByteString | ||
|
|
||
| /** | ||
| * Parses the raw challenge data obtained from the Create endpoint and returns an object that can be | ||
| * embedded in a JavaScript snippet. | ||
| */ | ||
| fun parseChallengeData(rawChallengeData: String): String { | ||
| val scrambled = JsonParser.array().from(rawChallengeData) | ||
|
|
||
| val challengeData = if (scrambled.size > 1 && scrambled.isString(1)) { | ||
| val descrambled = descramble(scrambled.getString(1)) | ||
| JsonParser.array().from(descrambled) | ||
| } else { | ||
| scrambled.getArray(1) | ||
| } | ||
|
|
||
| val messageId = challengeData.getString(0) | ||
| val interpreterHash = challengeData.getString(3) | ||
| val program = challengeData.getString(4) | ||
| val globalName = challengeData.getString(5) | ||
| val clientExperimentsStateBlob = challengeData.getString(7) | ||
|
|
||
| val privateDoNotAccessOrElseSafeScriptWrappedValue = challengeData.getArray(1, null)?.find { it is String } | ||
| val privateDoNotAccessOrElseTrustedResourceUrlWrappedValue = challengeData.getArray(2, null)?.find { it is String } | ||
|
|
||
| return JsonWriter.string( | ||
| JsonObject.builder() | ||
| .value("messageId", messageId) | ||
| .`object`("interpreterJavascript") | ||
| .value("privateDoNotAccessOrElseSafeScriptWrappedValue", privateDoNotAccessOrElseSafeScriptWrappedValue) | ||
| .value("privateDoNotAccessOrElseTrustedResourceUrlWrappedValue", privateDoNotAccessOrElseTrustedResourceUrlWrappedValue) | ||
| .end() | ||
| .value("interpreterHash", interpreterHash) | ||
| .value("program", program) | ||
| .value("globalName", globalName) | ||
| .value("clientExperimentsStateBlob", clientExperimentsStateBlob) | ||
| .done() | ||
| ) | ||
| } | ||
|
|
||
| /** | ||
| * Parses the raw integrity token data obtained from the GenerateIT endpoint to a JavaScript | ||
| * `Uint8Array` that can be embedded directly in JavaScript code, and an [Int] representing the | ||
| * duration of this token in seconds. | ||
| */ | ||
| fun parseIntegrityTokenData(rawIntegrityTokenData: String): Pair<String, Long> { | ||
| val integrityTokenData = JsonParser.array().from(rawIntegrityTokenData) | ||
| return base64ToU8(integrityTokenData.getString(0)) to integrityTokenData.getLong(1) | ||
| } | ||
|
|
||
| /** | ||
| * Converts a string (usually the identifier used as input to `obtainPoToken`) to a JavaScript | ||
| * `Uint8Array` that can be embedded directly in JavaScript code. | ||
| */ | ||
| fun stringToU8(identifier: String): String { | ||
| return newUint8Array(identifier.toByteArray()) | ||
| } | ||
|
|
||
| /** | ||
| * Takes a poToken encoded as a sequence of bytes represented as integers separated by commas | ||
| * (e.g. "97,98,99" would be "abc"), which is the output of `Uint8Array::toString()` in JavaScript, | ||
| * and converts it to the specific base64 representation for poTokens. | ||
| */ | ||
| fun u8ToBase64(poToken: String): String { | ||
| return poToken.split(",") | ||
| .map { it.toUByte().toByte() } | ||
| .toByteArray() | ||
| .toByteString() | ||
| .base64() | ||
| .replace("+", "-") | ||
| .replace("/", "_") | ||
| } | ||
|
|
||
| /** | ||
| * Takes the scrambled challenge, decodes it from base64, adds 97 to each byte. | ||
| */ | ||
| private fun descramble(scrambledChallenge: String): String { | ||
| return base64ToByteString(scrambledChallenge) | ||
| .map { (it + 97).toByte() } | ||
| .toByteArray() | ||
| .decodeToString() | ||
| } | ||
|
|
||
| /** | ||
| * Decodes a base64 string encoded in the specific base64 representation used by YouTube, and | ||
| * returns a JavaScript `Uint8Array` that can be embedded directly in JavaScript code. | ||
| */ | ||
| private fun base64ToU8(base64: String): String { | ||
| return newUint8Array(base64ToByteString(base64)) | ||
| } | ||
|
|
||
| private fun newUint8Array(contents: ByteArray): String { | ||
| return "new Uint8Array([" + contents.joinToString(separator = ",") { it.toUByte().toString() } + "])" | ||
| } | ||
|
|
||
| /** | ||
| * Decodes a base64 string encoded in the specific base64 representation used by YouTube. | ||
| */ | ||
| private fun base64ToByteString(base64: String): ByteArray { | ||
| val base64Mod = base64 | ||
| .replace('-', '+') | ||
| .replace('_', '/') | ||
| .replace('.', '=') | ||
|
|
||
| return (base64Mod.decodeBase64() ?: throw PoTokenException("Cannot base64 decode")) | ||
| .toByteArray() | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.