[FIXED] Need help parsing link from iframe using BeautifulSoup and Python3

Issue

I have this url here, and I'm trying to get the video's source link, but it's located within an iframe. The video url is https://ndisk.cizgifilmlerizle.com... inside an iframe called vjs_iframe. My code is below:

import requests
from bs4 import BeautifulSoup
url = "https://m.wcostream.com/my-hero-academia-season-4-episode-5-english-dubbed"
r = requests.Session() 
headers = {"User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0"} # Noticed that website responds better with headers
req = r.get(url, headers=headers)
soup = BeautifulSoup(req.content, 'html.parser')
iframes = soup.find_all("iframe") # Returns an empty list
vjs_iframe = soup.find_all(class_="vjs_iframe") # Also returns an empty list

I don't know how to get the url within the iframe, since not even the iframe's source is loaded upon the first request. Is getting the https://ndisk.cizgifilmlerizle.com... url even possible using BeautifulSoup or would I need to use another library like selenium or something else? Thanks in advance!

Solution

This website is quite tricky: the iframe is generated from a semi-obfuscated code just after the <meta itemprop="embedURL"> (I am formatting it):

<script>var nUk = ""; var BBX = ["RnF6Mjk4NzU0MkRXcw==", "Tnl0Mjk4NzU4N3NidA==", 
// TONS OF STRINGS IN THE ARRAY
];
BBX.forEach(function EtT(value) {
   nUk += String.fromCharCode(
      parseInt(atob(value).replace(/\D/g,'')) - 2987482); 
});
document.write(decodeURIComponent(escape(nUk)));</script>

Variable names are auto-generated, if you reload the page they will change, but the obfuscating technique is the same. Each segment of that array (value in the forEach loop) contains an obfuscated character, this is how things are going:

• it decodes the base64 string (atob)
• removes all non-digit characters (the replace thing), so you have a number
• subtracts 2987482 (another auto-generated number which changes at every request)
• converts it to a character (the fromCharCode call)
• merges all characters

If you are able to execute that code in a Firefox/Chromium console, omit the document.write() and print the variable in the console: you can see that iframe code, which is then injected into the page by the document.write() call.

You should be able to pass that javascript to an interpreter, to get the iframe stuff and capture the URL, then you can scrape that URL.

To scrape this website in python, you should have some javascript interpreter, or work really hard with regexp, e.g. soup.find(string=re.compile('.*atob\(')) and then do the same as the javascript does in the browser. It's really overkill, you should do that only for learning purposes. If your mission is to download the iframe stuff, maybe it's easier to find another website.

I recommend using the lxml parser, if you are able to install that library. Moreover, i really recommend scrapy, it's a gorgeous piece of software.

Answered By - danieli