[FIXED] Fixing BeautifulSoup linting errors

Issue

I am using BeautifulSoup 4 to handle some JSF-related scraping. The scraping works fine, but mypy is throwing back some errors on this code:

soup = bs4.BeautifulSoup(resp.text, 'lxml')
headers = {i['name']: i['value'] if i.has_attr('value') else i['id'] 
    for i in soup.body.form.find_all(type = 'hidden')}

btns = soup.body.form.find_all(type = 'submit')

The errors I get are:

handler.py:40: error: Item "None" of "Optional[Tag]" has no attribute "form"  [union-attr]
handler.py:40: error: Item "None" of "Union[Tag, None, Any]" has no attribute "find_all"  [union-attr]
handler.py:42: error: Item "None" of "Optional[Tag]" has no attribute "form"  [union-attr]
handler.py:42: error: Item "None" of "Union[Tag, None, Any]" has no attribute "find_all"  [union-attr]
Found 4 errors in 1 file (checked 1 source file)

It seems that mypy appears to think that body is a None type. I'm rather new to both of these tools so I'm not entirely sure how to deal with this issue. Has anyone else encountered this? How do I fix these?

Solution

If your HTML document doesn't have a form element, then soup.body.form would return None. With your current code, MyPy has no way to tell whether or not the value is None at the point you're calling soup.body.form.find_all(...).

You can prevent these errors from MyPy by explicitly checking for None in your code:

soup = bs4.BeautifulSoup(resp.text, 'lxml')
if soup.body.form is None:
    raise ValueError("body.form is None")

headers = {i['name']: i['value'] if i.has_attr('value') else i['id'] 
    for i in soup.body.form.find_all(type = 'hidden')}

Answered By - larsks