Issue
I am trying to scrape a specific website using selenium and beautifulsoup. Idea is to get the link of each page and its corresponding paragraphs in a pandas dataframe.
So the resultant dataframe would be like
Link Paras
https://www.<website>.com contents of all <p> tags
/specific_page.html
For this I am using the following code snippets:
driver = webdriver.Chrome(executable_path='D:\WebScrapp\chromedriver.exe')
driver.get(url)
elem = driver.find_elements_by_xpath("//a[@href]")
link=[]
para = []
for e in elem:
try:
page = requests.get(e.get_attribute('href'))
soup = bs(page.content,'lxml')
paras = soup.find_all('p')
for p in paras:
if '<seacrh_strng>' in p.text:
link.appned(str(e.get_attribute('href')))
para.append(p.text)
except:
print('InvalidSchema: No connection adapters')
df = pd.DataFrame(zip(link,para),columns=['Link','Para'])
With the above, I am facing the below issues:
- Most of the times (or for most the elements in
elem
) it is going toexcept
block and thus printing'InvalidSchema: No connection adapters'
. - The above technique is quite slow.
Eg. I have tried url like https://www.cognizant.com
or https://www.sas.com
or https://www.bmc.com
and have got nothing in the dataframe. It is hard to believe that these websites are not using <p>
as a tag!!
In fact I have tried out paras = soup.find_all(re.compile('^h[1-6]$'))
,but no luck!
What I am missing out?
Solution
I would suggest refactoring your code like this:
import pandas as pd
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome(
executable_path="D:\WebScrapp\chromedriver.exe"
)
driver.get("https://www.bmc.com") # for testing purposes
elements = driver.find_elements(by="xpath", value="//a[@href]")
results = {"link": [], "ptag": []}
for element in elements:
try:
url = element.get_attribute("href")
response = requests.request(method="get", url=url)
parsed_html = BeautifulSoup(response.content, "lxml", from_encoding="utf-8")
for p_tag in parsed_html.find_all("p"):
results["link"].append(url)
results["ptag"].append(p_tag.text)
except:
continue
And then:
df = pd.DataFrame(results)
print(df.sample(4))
# Output
link ptag
55 https://www.bmc.com/# Gartner names BMC a Leader in the ITSM Magic Q...
138 https://www.bmc.com/it-solutions/bmc-helix-ope... Winner of the 2021 AI Breakthrough Award
303 https://www.bmc.com/it-solutions/devops.html Speak to a rep about your business needs
43 https://www.bmc.com/# Drive innovation through agility, customer cen...
Answered By - Laurent
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.