[FIXED] BeautifulSoup can't get all page source sometimes

Issue

I'm using Selenium and beautifulSoup4 for scraping. The problem is that my script sometimes 'result'is empty and sometimes no. I don't understand why it's not working sometimes. Is it a security problem in the website or RAM problem ? I have no idea

page_source = BeautifulSoup(driver.page_source, "html.parser")

result= page_source.find_all('div',{'class':'pv-profile-section-pager ember-view'})

Solution

your class name can be error somewhere, you can try:

result= page_source.find_all('div',{'class': lambda x: x and 'pv-profile-section-pager' in x})

or iframe html tag can be also a problem here Select iframe using Python + Selenium

Answered By - lam vu Nguyen