[FIXED] Extracting content from webpage using BeautifulSoup

Issue

I am working on scrapping numbers from the Powerball website with the code below. However, numbers keeps coming back empty. Why is this?

import requests

from bs4 import BeautifulSoup

url = 'https://www.powerball.com/games/home'

page = requests.get(url).text


bsPage = BeautifulSoup(page)

numbers = bsPage.find_all("div", class_="field_numbers")

numbers

Solution

Can confirm @Teprr is absolutely correct. You'll need to download chrome and add chromedriver.exe to your system path for this to work but the following code gets what you are looking for. You can use other browsers too you just need their respective driver.

from bs4 import BeautifulSoup
from selenium import webdriver
import time
url = 'https://www.powerball.com/games/home'
options = webdriver.ChromeOptions()
options.add_argument('headless')
browser = webdriver.Chrome(options=options)
browser.get(url)
time.sleep(3) # wait three seconds for all the js to happen
html = browser.page_source
soup = BeautifulSoup(html, 'html.parser')
draws = soup.findAll("div", {"class":"number-card"})
print(draws)
for d in draws:
    info = d.find("div",{"class":"field_draw_date"}).getText()
    balls = d.find("div",{"class":"field_numbers"}).findAll("div",{"class":"numbers-ball"})
    numbers = [ball.getText() for ball in balls]
    print(info)
    print(numbers)

Answered By - kpie