[FIXED] Scraping table with Beautifulsoup and output to compare with value

Issue

I try to scrape a webpage with hourly energy prices. I want to use the data for home-automation. if the hourly price =< baseload price, certain times should turn on via Mqtt. I managed to get the data from the baseload price and the hourly prices from its column. The output from the column seems not to be in one list but in 24 lists. correct? how to fix this so that the hourly price can be compared with the baseload price?

import datetime
import pytz
import requests  
from bs4 import BeautifulSoup as bs

today_utc = pytz.utc.localize(datetime.datetime.utcnow())
today = today_utc.astimezone(pytz.timezone("Europe/Amsterdam"))
text_today = today.strftime("%y-%m-%d")
print(today)
print(text_today)

yesterday =  datetime.datetime.now(tz=pytz.timezone("Europe/Amsterdam")) - datetime.timedelta(1)
text_yesterday = yesterday.strftime("%y-%m-%d")
print(yesterday)
print(text_yesterday)

url_part1 = 'https://www.epexspot.com/en/market-data?market_area=NL&trading_date='
url_part2 = '&delivery_date='
url_part3 = '&underlying_year=&modality=Auction&sub_modality=DayAhead&technology=&product=60&data_mode=table&period=&production_period='
url_text = url_part1+text_yesterday+url_part2+text_today+url_part3
print(url_text)

html_text = requests.get(url_text).text
#print(html_text)

soup = bs(html_text,'lxml')
#print(soup.prettify())

baseload = soup.find_all('div', class_='flex day-1')
for baseload_price in baseload:
    baseload_price = baseload_price.find('span').text.replace(' ', '')
    print(baseload_price)

table = soup.find_all('tr',{'class':"child"})
#print(table)
for columns in table:
    column3 = columns.find_all('td')[3:]
    #print(columns)
    column3_text = [td.text.strip() for td in column3]
    column3_text = column3_text
    print(column3_text)

Solution

In the for loop for columns in table, you are creating a new list column3_text. If you intend for column3 text to be a list of the next 24 hours, you can replace this for loop with this:

column3_text = [column.find_all("td")[3].text.strip() for column in table]

Additionally, if you are going to be comparing the baseload price to the hourly prices, you'll want to convert the strings to floats or Decimals. :)

Answered By - David