[FIXED] Getting Selenium page source before JavaScript execution

Issue

I have a case where the raw page source code from the HTTP request contains some JSON data I need to capture - let's say <code id="data">{'some':'json'}</code>. But the JavaScript executes, processes it, and removes the data from the DOM so I can't see it in webdriver.page_source.

Any ideas how I can capture this? Or at least somehow disable/pause JavaScript, .get() the page, extract what I need from source_code and then re-enable/un-pause JavaScript?

Solution

I feel like this is a bit dirty but it works. I needed to use selenium for the first part. I then handed off to requests:

session = requests.Session()
for cookie in self.webdriver.get_cookies():
    session.cookies.set(cookie['name'], cookie['value'])

response = session.get(self.webdriver.current_url).text

and simply wrote the contents to a temp file and then opened that with selenium:

response_file = utils.save_to_temp_file(response, extension='.html')
self.webdriver.get(f'file://{response_file}')`

Answered By - eth0