Issue
I have a website that has a dynamic table, which simply does not show up when I try to use combos of Selenium and BS.
I have looked at code in several answers, but none of it works.
Do you have any bulletproof code that will extract tables, no matter how dynamic the webpage is?
Here is my (unsuccessful) attempt :
# Configure Selenium options
options = webdriver.ChromeOptions()
options.add_argument('--headless') # Run in headless mode
options.add_argument('--disable-gpu')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
# Create a new instance of the Chrome driver
driver = webdriver.Chrome(options=options)
driver.get(url)
# Get the full page source
page_source = driver.page_source
# Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(page_source, 'html.parser')
# Find all tables using class name or attribute value
tables = soup.find_all('table', attrs={'class': 'table1'})
print(tables)
# Close the browser
driver.quit()
Prior to that also tried using requests.get with headers equal to:
header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
Then I also tried adding sleep or time to wait for content to load... but nada.
The table is nested here in the html:
<div class="rightContainer" style="width:730px">
<div id="master" class="mainContent Small" style="width:730px">
<div class="greyContainerBtm">
<div class="greyContainerBody">
<div class="freeTextArea">
<img class="printObj" src="/images/logo.gif" style="display:none">
<h2 class="printObj" style="display:none">Foreign Currency Fixed Deposit Rates</h2>
<b>(As at <span id="datetime">20-JUN-2023 10:30 a.m.</span>)</b>
<br><br><span>The rates shown here are indicative only and are subject to change without prior notice.</span>
<div class="mainContentArea" style="width:700px">
<br>
<h2 class="pageTitle">Board Rates</h2><br><h2 class="printObj" style="display:none">Board Rates</h2><br><table class="table1">
<tbody><tr>
<th rowspan="2" colspan="2" id="hd1_1" class="txt_acc_type valignC" style="width: 100px;">Currency </th>
<th colspan="7" class="txt_deposit_term alignC valignC">Rate (% p.a.)</th>
</tr>
<tr>
<td class="txt_overnight highlight alignR" id="hd1_2" style="width: 70px;">Minimum (Equivalent) Personal / Corporate</td>
<td class="txt_24h highlight alignR" id="hd1_3" style="width: 60px;">1 week</td>
<td class="txt_24h highlight alignR" id="hd1_4">1 month</td>
<td class="txt_24h highlight alignR" id="hd1_5">2 months</td>
<td class="txt_24h highlight alignR" id="hd1_6">3 months</td>
<td class="txt_24h highlight alignR" id="hd1_7">6 months</td>
<td class="txt_24h highlight alignR" id="hd1_8">12 months</td>
</tr>
<tr class="data">
<td class="align"><b>Australian Dollar</b></td>
<td class="align"><b>AUD</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">3.6400</td>
<td class="alignR">3.8000</td>
<td class="alignR">4.0600</td>
<td class="alignR">4.2800</td>
<td class="alignR">4.4900</td>
<td class="alignR">4.9300</td>
</tr>
<tr class="data">
<td class="align"><b>Canadian Dollar</b></td>
<td class="align"><b>CAD</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">4.2200</td>
<td class="alignR">4.2700</td>
<td class="alignR">4.4600</td>
<td class="alignR">4.6500</td>
<td class="alignR">4.8500</td>
<td class="alignR">5.2200</td>
</tr>
<tr class="data">
<td class="align"><b>Swiss Franc</b></td>
<td class="align"><b>CHF</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">1.0700</td>
<td class="alignR">1.2300</td>
<td class="alignR">1.2300</td>
<td class="alignR">1.3200</td>
<td class="alignR">1.4200</td>
<td class="alignR">1.6600</td>
</tr>
<tr class="data">
<td class="align"><b>Renminbi</b></td>
<td class="align"><b>CNY</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">0.2500</td>
<td class="alignR">1.2000</td>
<td class="alignR">1.3000</td>
<td class="alignR">1.7000</td>
<td class="alignR">1.8000</td>
<td class="alignR">2.0000</td>
</tr>
<tr class="data">
<td class="align"><b>Euro</b></td>
<td class="align"><b>EUR</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">2.8000</td>
<td class="alignR">2.9200</td>
<td class="alignR">3.0300</td>
<td class="alignR">3.1400</td>
<td class="alignR">3.3900</td>
<td class="alignR">3.5900</td>
</tr>
<tr class="data">
<td class="align"><b>Pound Sterling</b></td>
<td class="align"><b>GBP</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">4.0200</td>
<td class="alignR">4.2700</td>
<td class="alignR">4.4000</td>
<td class="alignR">4.5300</td>
<td class="alignR">4.8600</td>
<td class="alignR">5.2400</td>
</tr>
<tr class="data">
<td class="align"><b>Hong Kong Dollar</b></td>
<td class="align"><b>HKD</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">4.6300</td>
<td class="alignR">4.6400</td>
<td class="alignR">4.6400</td>
<td class="alignR">4.6400</td>
<td class="alignR">4.6600</td>
<td class="alignR">4.7500</td>
</tr>
<tr class="data">
<td class="align"><b>Japanese Yen</b></td>
<td class="align"><b>JPY</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
</tr>
<tr class="data">
<td class="align"><b>New Zealand Dollar</b></td>
<td class="align"><b>NZD</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">4.8500</td>
<td class="alignR">4.9500</td>
<td class="alignR">5.1900</td>
<td class="alignR">5.3900</td>
<td class="alignR">5.5500</td>
<td class="alignR">5.8500</td>
</tr>
<tr class="data">
<td class="align"><b>United States Dollar</b></td>
<td class="align"><b>USD</b></td>
<td class="alignR">USD10K/20K</td>
<td class="alignR">4.1700</td>
<td class="alignR">4.2500</td>
<td class="alignR">4.7200</td>
<td class="alignR">4.9000</td>
<td class="alignR">5.0000</td>
<td class="alignR">5.0400</td>
</tr>
</tbody></table>
<br> <br>
<h2 class="pageTitle">Tier Rates </h2><br><h2 class="printObj" style="display:none">Tier Rates </h2><br><table class="table2">
<tbody><tr>
<th rowspan="2" colspan="2" class="txt_acc_type valignC" id="hd1_1" width="100">Currency </th>
<th colspan="7" class="txt_deposit_term alignC valignC">Rate (% p.a.)</th>
</tr>
<tr>
<td class="txt_overnight highlight alignR" id="hd1_2" width="70">Minimum </td>
<td class="txt_24h highlight alignR" id="hd1_3" width="60">1 week</td>
<td class="txt_24h highlight alignR" id="hd1_3" width="60">1 month</td>
<td class="txt_24h highlight alignR" id="hd1_3" width="60">2 months</td>
<td class="txt_24h highlight alignR" id="hd1_3" width="60">3 months</td>
<td class="txt_24h highlight alignR" id="hd1_3" width="60">6 months</td>
<td class="txt_24h highlight alignR" id="hd1_3" width="60">12 months</td>
</tr>
<tr class="data">
<td class="align"><b>Australian Dollar</b></td>
<td class="align"><b>AUD</b></td>
<td class="alignR">100,000</td>
<td class="alignR">3.7700</td>
<td class="alignR">3.9300</td>
<td class="alignR">4.1900</td>
<td class="alignR">4.4100</td>
<td class="alignR">4.6200</td>
<td class="alignR">5.0600</td>
</tr>
<tr class="data">
<td class="align"><b>Canadian Dollar</b></td>
<td class="align"><b>CAD</b></td>
<td class="alignR">100,000</td>
<td class="alignR">4.3500</td>
<td class="alignR">4.4000</td>
<td class="alignR">4.5900</td>
<td class="alignR">4.7800</td>
<td class="alignR">4.9800</td>
<td class="alignR">5.3500</td>
</tr>
<tr class="data">
<td class="align"><b>Swiss Franc</b></td>
<td class="align"><b>CHF</b></td>
<td class="alignR">100,000</td>
<td class="alignR">1.2000</td>
<td class="alignR">1.3600</td>
<td class="alignR">1.3600</td>
<td class="alignR">1.4500</td>
<td class="alignR">1.5500</td>
<td class="alignR">1.7900</td>
</tr>
<tr class="data">
<td class="align"><b>Renminbi</b></td>
<td class="align"><b>CNY</b></td>
<td class="alignR">500,000</td>
<td class="alignR">0.3000</td>
<td class="alignR">1.5000</td>
<td class="alignR">1.6000</td>
<td class="alignR">2.0000</td>
<td class="alignR">2.1000</td>
<td class="alignR">2.3000</td>
</tr>
<tr class="data">
<td class="align"><b>Euro</b></td>
<td class="align"><b>EUR</b></td>
<td class="alignR">100,000</td>
<td class="alignR">2.9300</td>
<td class="alignR">3.0500</td>
<td class="alignR">3.1600</td>
<td class="alignR">3.2700</td>
<td class="alignR">3.5200</td>
<td class="alignR">3.7200</td>
</tr>
<tr class="data">
<td class="align"><b>Pound Sterling</b></td>
<td class="align"><b>GBP</b></td>
<td class="alignR">50,000</td>
<td class="alignR">4.1500</td>
<td class="alignR">4.4000</td>
<td class="alignR">4.5300</td>
<td class="alignR">4.6600</td>
<td class="alignR">4.9900</td>
<td class="alignR">5.3700</td>
</tr>
<tr class="data">
<td class="align"><b>Hong Kong Dollar</b></td>
<td class="align"><b>HKD</b></td>
<td class="alignR">500,000</td>
<td class="alignR">4.7600</td>
<td class="alignR">4.7700</td>
<td class="alignR">4.7700</td>
<td class="alignR">4.7700</td>
<td class="alignR">4.7900</td>
<td class="alignR">4.8800</td>
</tr>
<tr class="data">
<td class="align"><b>Japanese Yen</b></td>
<td class="align"><b>JPY</b></td>
<td class="alignR">10,000,000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
<td class="alignR">0.0000</td>
</tr>
<tr class="data">
<td class="align"><b>New Zealand Dollar</b></td>
<td class="align"><b>NZD</b></td>
<td class="alignR">100,000</td>
<td class="alignR">4.9800</td>
<td class="alignR">5.0800</td>
<td class="alignR">5.3200</td>
<td class="alignR">5.5200</td>
<td class="alignR">5.6800</td>
<td class="alignR">5.9800</td>
</tr>
<tr class="data">
<td class="align"><b>United States Dollar</b></td>
<td class="align"><b>USD</b></td>
<td class="alignR">100,000</td>
<td class="alignR">4.3000</td>
<td class="alignR">4.3800</td>
<td class="alignR">4.8500</td>
<td class="alignR">5.0300</td>
<td class="alignR">5.1300</td>
<td class="alignR">5.1700</td>
</tr>
</tbody></table>
<br><br><br>
<div style="width:700px">
<b>Notes:</b><br>For fixed deposit amount of SGD500,000 equivalent or over, please contact us for the applicable prevailing interest rate.
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div> '''
enter code here
Solution
The data you see is loaded from different URL (<iframe>):
import pandas as pd
url = 'https://www.hkbea.com/jsp/sg/beasg_rate_fcfdr.jsp?fontSize=-1'
df1, df2 = pd.read_html(url)
print(df1)
print('-'*80)
print(df2)
Prints:
Currency Rate (% p.a.)
Currency Currency.1 Unnamed: 2_level_1 Unnamed: 3_level_1 Unnamed: 4_level_1 Unnamed: 5_level_1 Unnamed: 6_level_1 Unnamed: 7_level_1 Unnamed: 8_level_1
0 Minimum (Equivalent) Personal / Corporate 1 week 1 month 2 months 3 months 6 months 12 months NaN NaN
1 Australian Dollar AUD USD10K/20K 3.6400 3.8000 4.0600 4.2800 4.49 4.93
2 Canadian Dollar CAD USD10K/20K 4.2200 4.2700 4.4600 4.6500 4.85 5.22
3 Swiss Franc CHF USD10K/20K 1.0700 1.2300 1.2300 1.3200 1.42 1.66
4 Renminbi CNY USD10K/20K 0.2500 1.2000 1.3000 1.7000 1.80 2.00
5 Euro EUR USD10K/20K 2.8000 2.9200 3.0300 3.1400 3.39 3.59
6 Pound Sterling GBP USD10K/20K 4.0200 4.2700 4.4000 4.5300 4.86 5.24
7 Hong Kong Dollar HKD USD10K/20K 4.6300 4.6400 4.6400 4.6400 4.66 4.75
8 Japanese Yen JPY USD10K/20K 0.0000 0.0000 0.0000 0.0000 0.00 0.00
9 New Zealand Dollar NZD USD10K/20K 4.8500 4.9500 5.1900 5.3900 5.55 5.85
10 United States Dollar USD USD10K/20K 4.1700 4.2500 4.7200 4.9000 5.00 5.04
--------------------------------------------------------------------------------
Currency Rate (% p.a.)
Currency Currency.1 Unnamed: 2_level_1 Unnamed: 3_level_1 Unnamed: 4_level_1 Unnamed: 5_level_1 Unnamed: 6_level_1 Unnamed: 7_level_1 Unnamed: 8_level_1
0 Minimum 1 week 1 month 2 months 3 months 6 months 12 months NaN NaN
1 Australian Dollar AUD 100000 3.7700 3.9300 4.1900 4.4100 4.62 5.06
2 Canadian Dollar CAD 100000 4.3500 4.4000 4.5900 4.7800 4.98 5.35
3 Swiss Franc CHF 100000 1.2000 1.3600 1.3600 1.4500 1.55 1.79
4 Renminbi CNY 500000 0.3000 1.5000 1.6000 2.0000 2.10 2.30
5 Euro EUR 100000 2.9300 3.0500 3.1600 3.2700 3.52 3.72
6 Pound Sterling GBP 50000 4.1500 4.4000 4.5300 4.6600 4.99 5.37
7 Hong Kong Dollar HKD 500000 4.7600 4.7700 4.7700 4.7700 4.79 4.88
8 Japanese Yen JPY 10000000 0.0000 0.0000 0.0000 0.0000 0.00 0.00
9 New Zealand Dollar NZD 100000 4.9800 5.0800 5.3200 5.5200 5.68 5.98
10 United States Dollar USD 100000 4.3000 4.3800 4.8500 5.0300 5.13 5.17
Answered By - Andrej Kesely
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.