[FIXED] Is this website scrape-able with BeautifulSoup?

Issue

I would to scrape this website : https://www.projets-environnement.gouv.fr/pages/home/

More precisely, I would like to collect the table in the div with id = table-wrapper.

My trouble is that I can't catch it with BeautifulSoup.

Here is my code :

url = 'https://www.projets-environnement.gouv.fr/pages/home/'
html = requests.get(url).text
soup = BeautifulSoup(html, "html5lib")
div_table = soup.findAll('div', id_='table-wrapper')

But div_table is a None object. Is Selenium the solution ?

Solution

I think you should use selenium:

from bs4 import BeautifulSoup

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options

url = 'https://www.projets-environnement.gouv.fr/pages/home/'

options = Options()
options.headless = True
driver = webdriver.Firefox(firefox_options=options)
driver.get(url)

html = driver.page_source
soup = BeautifulSoup(html, "html.parser")
mytable = soup.find('div', id='table-wrapper')

and you get that table.

Answered By - sentence