[FIXED] Web Scraping - Table Without Class or ID

Issue

I am working on a project which requires me to scrape a table from the following website - https://www.cmegroup.com/markets/energy/crude-oil/brent-crude-oil.settlements.html#tradeDate=03%2F31%2F2022

The problem I am facing is that the table doesnt seem to have classes when I inspect the elements.

I am new to web-scraping/BeautifulSoup and would appreciate any help I can get.

url: https://www.cmegroup.com/markets/energy/crude-oil/brent-crude-oil.settlements.html#tradeDate=03%2F31%2F2022

Solution

Actually,the url aka table is dynamic and you can grab it with the help of selenium,bs4 and pandas as follows:

Script:

import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import pandas as pd
from bs4 import BeautifulSoup
driver = webdriver.Chrome(ChromeDriverManager().install())

url ="https://www.cmegroup.com/markets/energy/crude-oil/brent-crude-oil.settlements.html#tradeDate=03%2F31%2F2022"
    
driver.maximize_window()
driver.get(url)
time.sleep(5)

soup = BeautifulSoup(driver.page_source, 'lxml')

df = pd.read_html(str(soup))[0]
print(df)

Output:

   Month  Open  High  Low  Last  Change  Settle  Est. Volume  Prior day OI
1   MAY 22     -     -    -     -    UNCH  113.45            0             0
2   JUN 22     -     -    -     -   -6.73  104.71            0             0
3   JLY 22     -     -    -     -   -5.91  102.73            0             0
4   AUG 22     -     -    -     -   -5.09  100.98            0             0
5   SEP 22     -     -    -     -   -4.34   99.43            0             0
6   OCT 22     -     -    -     -   -3.74   98.04            0             0
7   NOV 22     -     -    -     -   -3.29   96.75            0             0
8   DEC 22     -     -    -     -   -2.93   95.51            0             0
9   JAN 23     -     -    -     -   -2.62   94.35            0             0
10  FEB 23     -     -    -     -   -2.33   93.33            0             0
11  MAR 23     -     -    -     -   -2.04   92.53            0             0
12  APR 23     -     -    -     -   -1.78   91.78            0             0

Answered By - F.Hoque