[FIXED] Why is the pandas read_html function returning an empty list when there is a table on the page?

Issue

Ive tried the read_html function using urls of wikipedia and other websites with tables and they work just fine. I belive the table on this website is the same type, but I could be wrong.

This is the code I ran that includes the link to the website I'm trying to use, but its telling me that there are no tables so dfs is empty.

import pandas as pd
dfs = pd.read_html("https://anex.us/grades/?dept=ENGR&number=102")

I was expecting dfs to contain the table that is on the website below the graph.

Ive checked to see if the table is hidden in the comments and its not there.

I also tried to get an html string using selenium

data_table = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.ID, 'dataTable')))
html = data_table.get_attribute('outerHTML')

but the get_attribute just returns an empty table.

Solution

@Carcigenicate is right, the table is created dynamically. You can use this code to load your data:

import requests

url = 'https://anex.us/grades/getData/'
payload = {'dept': 'ENGR', 'number': '102'}
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0'}

data = requests.post(url, data=payload, headers=headers).json()
df = pd.DataFrame(data['classes'])

Note: use the Developer Tools of your browser to track XHR requests.

Output:

>>> df
     dept number section   A   B   C  D  F  I  S  U   Q  X           prof  year semester               gpa
0    ENGR    102      20  18  17   8  2  3  0  0  0   1  0        AMINI N  2018     FALL            2.9375
1    ENGR    102      21  18  31  15  4  1  0  0  0   0  0        KOOLA P  2018     FALL  2.88405797101449
2    ENGR    102      22  10  28  16  2  3  0  0  0   0  0         SHAW S  2018     FALL  2.67796610169492
3    ENGR    102      26   9  24  10  4  6  0  0  0   0  0  SUBRAMANIAN R  2018     FALL  2.49056603773585
4    ENGR    102     201  21  12   1  1  0  0  0  0   0  0         IJAZ M  2018     FALL  3.51428571428571
..    ...    ...     ...  ..  ..  .. .. .. .. .. ..  .. ..            ...   ...      ...               ...
486  ENGR    102     507  27  14   7  2  5  0  0  0  13  0         IJAZ M  2023   SPRING  3.01818181818182
487  ENGR    102     508  24   7  10  1  4  0  0  0   4  0         IJAZ M  2023   SPRING                 3
488  ENGR    102     540  12   3   2  1  0  0  0  0   0  0     ALVARADO L  2023   SPRING  3.44444444444444
489  ENGR    102     550   1   1   6  1  2  0  0  0   0  0        KOOLA P  2023   SPRING  1.81818181818182
490  ENGR    102     551   1   1   7  2  3  0  0  0   3  0    VILLAREAL S  2023   SPRING  1.64285714285714

[491 rows x 17 columns]

Answered By - Corralien