[FIXED] Scrape Google Quick Answer Box in Python

Issue

I am very new to Python programming and I am trying to make a simple application.

What I'm trying to do is search for a text on Google and return the links, my program does this fine. The other thing is if Google has the quick answer like in the photo below, I want to grab it, and this is where my problem lies. I tried searching online and found very few topics in which none of the codes work.

Google Quick box answer:

By examining the code of many pages I noticed that the answer is always in a class called _XWk but in Python when I get the code of the page and search for this class it doesn't find it. I tried so many ways of scarping the page in Python, but it never gets this class and I think the code it gets is less than the code the browser shows me when I open page source code.

Class _XWk:

---

Code:

import requests, lxml
from bs4 import BeautifulSoup

url = 'https://www.google.com/search?q=when%20was%20trump%20born'
h = {"User-Agent":"Chrome/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36"}

r = requests.get(url, headers=h).text
soup = BeautifulSoup(r,'lxml')

soup.find_all("div", class_="_xwk")
print (soup)

Any help is appreciated.

Solution

The line soup.find_all("div", class_="_xwk") has no effect in your code. The find_all() function returns a list of tags that match the given parameters. So, you need to save this result in a variable.

But, as you need only one such tag, you can use find(), which returns the first tag match.

Finally, to get the text inside a tag, you've to use the .text attribute.

Also, the class name is case sensitive. In the inspection, the class name is _XWk and not _xwk. Making these changes, the code:

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36'}
r = requests.get('https://www.google.com/search?q=when%20was%20trump%20born', headers=headers)
soup = BeautifulSoup(r.text, 'lxml')

result = soup.find('div', class_='_XWk')
print(result.text)
# 14 June 1946 (age 71)

Answered By - Keyur Potdar