[FIXED] How to access elements inside svg with BeautifulSoup?

Issue

I am scraping the for weather data from google search results. Finally I want to scrape data from the svg graphs which is where I have all the issues.

---

My code:

from bs4 import BeautifulSoup as bs
import requests

def get_weather_data(region):
    # const values
    USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
    LANGUAGE = "en-US,en;q=0.5" # US english
    URL = f"https://www.google.com/search?lr=lang_en&q=weather+in+{region.strip().lower().replace(' ', '+')}"
    
    # Send request and store response
    s = requests.Session()
    s.headers['User-Agent'] = USER_AGENT
    s.headers['Accept-Language'] = LANGUAGE
    s.headers['Content-Language'] = LANGUAGE
    html = s.get(URL)
    soup = bs(html.text, "html.parser")

    hourly = soup.find("svg", attrs={'id':'wob_gsvg'})
    hourly2 = soup.find("svg", attrs={'id':'wob_gsvg'}).children
    print(hourly, hourly2)

get_weather_data("London")

Output: <svg class="wob_gsvg" data-ved="2ahUKEwiToY6r0eLzAhWOpZUCHdMQC0kQnaQEegQIGRAG" id="wob_gsvg" style="height:80px"></svg> <list_iterator object at 0x00000275054D9E20>

---

But in the chrome browser console, I can see:

Main Goal

• do web scraping - weather data from google search results.
• scrape hourly forecast available

Solution

In html.text you don not have this data.

For check it try:

with open("data.html", "w") as f:
    f.write(html.text)

Then open this file on your browser.

To resolve this try to use selenium library. https://selenium-python.readthedocs.io

Answered By - dimay