[FIXED] Address bar shows data:, while trying to scrape using ChromeDriver Chrome through Selenium and Python

Issue

I am making a simple scraping program.

First, the user will write the name of a footballer, then I will make a link to transfermarkt.com web search, and then I would like to enter the first link and scrape data from the footballer's profile. Unfortunately, I have a problem with selenium. How do I enter a website programmatically and scrape data from the site?

Here is my code:

from urllib.request import urlopen
import bs4
from bs4 import BeautifulSoup
from selenium import webdriver

data = input('Enter name: ')
data = data.replace(" ", "+")
print(data)
link = 'https://www.transfermarkt.pl/schnellsuche/ergebnis/schnellsuche?query='
search = link + data + '&x=0&y=0'
print(search)
driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")
driver.find_element_by_css_selector('.spielprofil_tooltip tooltipstered').click()
name_box = soup.find('h1', attrs={'class': 'dataValue'})
print(name_box)

It works only to line print(search), but then I'm lost. Browser is open, but there is only data:, in the address bar.

Solution

You only need that for headless browser:

from selenium import webdriver
#####
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu') 
driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver", options=options)

But as I said, you dont need to use selenium here. Use selenium when you can't use requests or when you just want to write your code fast.

Browser is open, but there is only data:, in the address bar.

Because you didn't get the url in browser:

browser.get(source)

Answered By - Ekrem Dinçel