Issue
I am trying to scrape mortgage rates from https://www.td.com/ca/en/personal-banking/products/mortgages/mortgage-rates/
When I use find_all to get value from a cell in specific table, the returned value is "!--empty--" instead of the text within that cell.
The actual html for that cell is:
<span class="h2 ng-binding ng-isolate-scope" code="a.reslrates.MTGF036C" high-ratio="false" resl-rate="" type="S">2.54%</span>
The result which is returned is:
<span class="h2" code="a.reslrates.MTGF036C" high-ratio="false" resl-rate="" type="S"><!--empty--></span>
Instead of the 2.54% rate text, I get !--empty-- result. I get Am I missing something here? Full code below:
html_text = requests.get("https://www.td.com/ca/en/personal-banking/products/mortgages/mortgage-rates/").text
soup = BeautifulSoup(html_text, "html.parser")
# Get the table
table = soup.find("div", class_="td-rates-table rates-bg-row1").table
rows = table.tbody.find_all("tr")
for row in rows:
for rate in row.find_all("td"):
print(rate)
I appreciate all responses! Thanks a lot!
Solution
Using selenium. Please install necessay dependencies and execute the script.
from selenium import webdriver
from bs4 import BeautifulSoup
import requests
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
url = 'https://www.td.com/ca/en/personal-banking/products/mortgages/mortgage-rates/'
# html_text = requests.get("https://www.td.com/ca/en/personal-banking/products/mortgages/mortgage-rates/").text
driver.get(url)
soup = BeautifulSoup(driver.page_source, "html.parser")
# Get the table
table = soup.find("div", class_="td-rates-table rates-bg-row1").table
rows = table.tbody.find_all("tr")
for row in rows:
for rate in row.find_all("td"):
print(rate.text)
Answered By - Samsul Islam
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.