[FIXED] Use online webdriver for BeautifulSoup / Selenium Python code (vs. local reference)

Issue

I wrote local Python code in Spyder that does webscraping using BeautifulSoup and Selenium. I now want to transfer that code to run online and on a schedule (using pythonanywhere). That works fine for the pure BeautifulSoup elements. The parts that use Selenium, currently have some configuration on my local setup, including a local reference to the webdriver, see below.

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome("C:/Users/my.name/Downloads/chromedriver-win64/chromedriver-win64/chromedriver.exe", options=options)
driver.get('URL TO SCRAPE')

When I transfer the code online, I obviously can't refer to my local C drive anymore. Is there such a thing as an online version of the webdriver exe file I can reference?

When I asked ChatGPT, I was referred to services like BrowserStack and SauceLabs, but (without reading their website too much) this looks a little overkill when seeing the price points.

Any advice or pointers are appreciated - thanks!

Solution

tl;dr: If you are using Selenium 4.6.0 or greater, you do not need to specify the driver location; if you are using an older version, upgrading could be a quick fix.

With 4.6.0 Selenium introduced Selenium Manager, which handles setting up browser drivers, removing the necessity to do it yourself.

From the documentation:

Setting up a browser driver once is not that complicated, but as browser release cycles got shorter, and now we have a new Chrome/Firefox/Edge version every 4-6 weeks, the task of keeping the browser driver in sync with the browser version is not that easy anymore.

Selenium Manager is a new tool that helps to get a working environment to run Selenium out of the box. Beta 1 of Selenium Manager will configure the browser drivers for Chrome, Firefox, and Edge if they are not present on the PATH.

So, while you can ensure your pythonanywhere environment is properly set up in their console and get file paths from there, you could also simply upgrade your selenium to 4.6.0.

Answered By - samsupertaco