Issue
I am trying to learn Selenium to scrape some Javascript heavy websites. I can locate and extract information just fine. However, I find that for some sites I need to switch my user agent. I did it the following way to test it:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
PATH ="C:/my/path/to/chromedriver.exe"
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options = Options()
options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(chrome_options=options, executable_path=PATH)
driver.get("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
The code works and my user agent is switched, however there is one bug that occurs now which did not occur before. The webdriver/browser (Chrome driver) automatically closes after displaying the website for a second without me specifying the driver.quit()
argument. When I do not switch my user agent it does not close unless I do and I want to study the page a bit before closing it. I have tried to wait using time.sleep()
but this doesn't work.
How can I make the webdriver not close until specified?
Answers are greatly appreciated, preferably with a code example of how to implement the solution.
Solution
I am not sure if it is possible that the problem is related to the webdriver version you are using or not but when I tried to add time.sleep(n)
to your code while using webdriver_manager
library to download most recent version of ChromeWebDriver
I had the chance to look at the website and the browser didn't close until the timer finished.
My code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
from webdriver_manager.chrome import ChromeDriverManager
import time
# PATH ="C:/my/path/to/chromedriver.exe"
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options = Options()
options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options)
driver.get("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
time.sleep(100)
Answered By - Mahmoud Embaby
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.