[FIXED] Why is Beautifulsoup's selector showing error while the Scrapy's response.css working absolutely fine?

Issue

I am trying to scrape this div tag which has an id attribute equal to today's date. I created a Beautifulsoup object and used the select method but it's showing this error

My code

res = requests.get('https://sports.ndtv.com/fifa-world-cup-2022/schedules-fixtures')
soup = BeautifulSoup(res.text,'html.parser')
date_today = date.today()
d1 = date_today.strftime("%d-%m-%Y")
cont = soup.select('div#'+d1)

This is raising the error

raise SelectorSyntaxError(msg, self.pattern, index)
soupsieve.util.SelectorSyntaxError: Malformed id selector at position 3
line 1:
div#30-11-2022

While when I use Scrapy shell -

response.css('div#'+d1)[0].css('span.location::text').get()           
'Ahmad Bin Ali Stadium, Al Rayyan'

It's working perfectly fine. Can anyone please suggest what am I doing wrong? Thanks

Solution

bs4 [or rather soupsieve] doesn't like it when id selectors [#id] have hyphens (-) in them for some reason, but you can get around it by using attribute selector instead

cont = soup.select(f'div[id="{d1}"]')

should work - give it a try.

Answered By - Driftr95