[FIXED] How do you use BeautifulSoup to fetch data in specific format?

Issue

I would create a python code to fetch the average volume of a given link stock using BeautifulSoup.

What I have done so far:

import bs4
import requests
from bs4 import BeautifulSoup

r=requests.get('https://finance.yahoo.com/quote/M/key-statistics?p=M')
soup=BeautifulSoup(r.content,"html.parser")

# p = soup.find_all(class_="Fw(500) Ta(end) Pstart(10px) Miw(60px)")[1].get_text
# p = soup.find_all('td')[2].get_text     
# p = soup.find_all('table', class_='W(100%) Bdcl(c)')[70].tr.get_text

Anyway, I was able to get that number directly from google console using this command:

Document.querySelectorAll('table tbody tr td')[71].innerText
"21.07M"

Please, help with the basic explanation, I know a few about DOM.

Solution

You can use this logic to make it easier:

• Use find_all to find all the spans in the html file
• Search the spans for the correct label (Avg Vol...)
• Use parent to go up the hierarchy to the full table row
• Use find_all again from the parent to get the last cell which contains the value

Here is the updated code:

import bs4
import requests
from bs4 import BeautifulSoup

r=requests.get('https://finance.yahoo.com/quote/M/key-statistics?p=M')
soup=BeautifulSoup(r.content,"html.parser")

p = soup.find_all('span')
for s in p:   # each span
   if s.text == 'Avg Vol (10 day)':  # starting cell
       pnt = s.parent.parent  # up 2 levels, table row
       print(pnt.find_all('td')[-1].text)  # last table cell

Output

21.76M

Answered By - Mike67