[FIXED] How to acces the type of a place (widget-pane-link) form Google Maps from python using Beautiful Soup

Issue

I am trying to get the tag American Restaurant under the Name El Merendero and the review of 4.1 using python and beautiful soup but I was not able to enter in . Any hint on how to find this google maps category ?

what I got is this text with some Javascript error I guess :

Enable JavaScript to see Google Maps.

My code:

import requests
from bs4 import BeautifulSoup

Query = " Restaurant el merendero "

durl= "https://www.google.com/maps/search/?api=1&query=%s" % Query
page = requests.get(durl)
soup = BeautifulSoup(page.content, 'html.parser')

unicode_str = soup.decode("utf8")
encoded_str = unicode_str.encode("ascii",'ignore')
news_soup = BeautifulSoup(encoded_str, "html.parser")
a_text = news_soup.find_all('div')
print(a_text)

Solution

This is due to the fact that google map is rendered as a single page app with Angular (I believe).

BeautifulSoup is unable to execute Javascript so you'd need execute the Javascript code (eg: with Selenium or Splash or any headless browser) before it can actually view and crawl the rendered HTML.

See this repository which holds an example of what you are trying to achieve: https://github.com/registerguard/spa_scraper

You can witness what I'm saying by disabling Javascript in your browser and going to google maps.

Answered By - Jonathan Liuti