[FIXED] BeautifulSoup google trend

Issue

I am new to the programming world and have just finished understanding the very basics of Python. I have just started practicing web crawling and have faced a problem already. I have written a very simple code using BeautifulSoup.

from bs4 import BeautifulSoup
from urllib.request import urlopen

response = urlopen('https://trends.google.com/trends/?geo=US/')
soup = BeautifulSoup(response, 'html.parser')
for anchor in soup.select(".list-item-title"):
    print(anchor)

I want to retrieve the names of the recently trending stories; however, the code above is not functioning as it's supposed to and returns a blank.

I would be grateful if someone could point out the error. Thank you!

Solution

Google Trends(url) is dynamic meaning data is generated by JavaScript and BeautifulSoup can't render javaSceipt.So, You need automation tool something like selenium with BeautifulSoup`. Please just run the code.

Scripts:

from bs4 import BeautifulSoup
import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

url = 'https://trends.google.com/trends/?geo=US/'

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
time.sleep(8)
driver.get(url)
time.sleep(10)

soup = BeautifulSoup(driver.page_source, 'html.parser')
#driver.close()

for anchor in soup.select(".list-item-title"):
    print(anchor.text)

Output:

Meta
Lupus
Liga Europy
Chelsea
Prova do lider
Masks in Kenya
UANL
Jussie Smollett
ישי ריבו
Winter storm warning

Answered By - F.Hoque