[FIXED] SERP Scraping with Beautiful Soup

Issue

I'm trying to build a simple script to scrape Google's first Search Results Page and export the results in .csv. I managed to get URLs and Titles, but I cannot retrieve Descriptions. I have been using the following code:

import urllib
import requests
from bs4 import BeautifulSoup

# desktop user-agent
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"
# mobile user-agent
MOBILE_USER_AGENT = "Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"

query = "pizza recipe"
query = query.replace(' ', '+')
URL = f"https://google.com/search?q={query}"

headers = {"user-agent": USER_AGENT}
resp = requests.get(URL, headers=headers)

if resp.status_code == 200:
    soup = BeautifulSoup(resp.content, "html.parser")
    results = []
    for g in soup.find_all('div', class_='r'):
        anchors = g.find_all('a')
        if anchors:
            link = anchors[0]['href']
            title = g.find('h3').text
            desc = g.select('span')
            description = g.find('span',{'class':'st'}).text
            item = {
                "title": title,
                "link": link,
                "description": description
            }
            results.append(item)

import pandas as pd
df = pd.DataFrame(results)
df.to_excel("Export.xlsx")

I get the following message when I run the code:

description = g.find('span',{'class':'st'}).text
AttributeError: 'NoneType' object has no attribute 'text'

Essentially, the field is empty.

Can somebody please help me this line so that I can get all the information from the snippet?

Solution

It's not within the div class="r". It's under div class="s"

So change to this for description:

description = g.find_next_sibling("div", class_='s').find('span',{'class':'st'}).text

From the current element, it'll find the next div, with class="s". Then you can pull out the <span> tag

Answered By - chitown88