[FIXED] How to get resolve of attribute error when web scraping google for wind speed

Issue

I am web scraping google for their wind speed and up until a few weeks ago this code worked perfectly! Now I've run it and I keep getting a "AttributeError: 'NoneType' object has no attribute 'text'" error. What can I change for it to work just as it did before? I am baffled as to why it has stopped working!

import requests
import re
from bs4 import BeautifulSoup as bs

#Function that returns windpseed in a dictionary
def current_windspeed(url):
    #Defining user agent and language to scrape google
    USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36"
    #Setting language as UK english
    LANGUAGE = "en-UK,en;q=0.5"
    session=requests.Session()
    session.headers['User-Agent']= USER_AGENT
    session.headers['Accept-Language']= LANGUAGE
    session.headers['Content-Language']=LANGUAGE
    #downloads html code for google weather london
    html=session.get(url)
    #creates a new soup
    soup=bs(html.text, "html.parser")
    #dictionary where windspeed will be stored
    current_weather={}
    print(current_weather)
    current_weather['wind']=soup.find("span",attrs={"id": "wob_ws"}).text
    return current_weather

The issue its flagging is this line of code

current_weather['wind']=soup.find("span",attrs={"id": "wob_ws"}).text

How on earth do I change it to work again and why did it stop working?

Solution

The parse is assuming you get back good data all the time from the server. I load weather data 24/7 and it can go for many days or even weeks fine, but the server will not play nice once in a while and give you a 404 or an empty file or whatever on the rare occasion.

The bit of code:

soup.find("span",attrs={"id": "wob_ws"})

Is not finding anything, so it returns None.

Your code could either be written to accept bad input gracefully continue while logging the problem. You can usually just assume it will mend itself later and use a try/except and just log the error and get it the next time around. If the server changed the web page then you might need to rework the code.

Answered By - Brian Z