Issue
Trying to extract a wikipedia list from: https://en.wikipedia.org/wiki/List_of_Category_5_Atlantic_hurricanes using BeautifulSoup.
this is my code:
wiki = "https://en.wikipedia.org/wiki/List_of_Category_5_Atlantic_hurricanes"
page = urllib.request.urlopen(wiki)
soup = BeautifulSoup(page)
table=soup.find('table', class_="wikitable sortable") # The class of the list in wikipedia
Data = [[] for _ in range(9)] # I intend to turn this into a DataFrame
for row in table.findAll('tr'):
cells = row.findAll('td')
if len(cells)==9: # The start and end don't include a <td> tag
for i in range(9):
Data[i].append(cells[i].find(text=True))
This works quite well apart from a single value in the names column, The hurricane "New England". This is the HTML code that contains that element:
<td><span data-sort-value="New England !"> <a href="/wiki/1938_New_England_hurricane" title="1938 New England hurricane">"New England"</a></span></td>
The entry for the name in that hurricane is ' ', I think that the space between <span>
and <a>
is causing this problem.
Is there a way to fix this in .find? Is there a smarter way to access lists in Wikipedia?
How can I avoid this in the future?
Solution
Simplest way to read a table
into a data frame is read_html()
:
import pandas as pd
pd.read_html(wiki)[1]
Output:
Name Dates as aCategory 5 Duration as aCategory 5 Sustainedwind speeds Pressure Areas affected Deaths Damage(USD) Refs
0 "Cuba" October 19, 1924 12 hours 165 mph (270 km/h) 910 hPa (26.87 inHg) Central America, Mexico, CubaFlorida, The Bahamas 90 NaN [12]
1 "San Felipe IIOkeechobee" September 13–14, 1928 12 hours 160 mph (260 km/h) 929 hPa (27.43 inHg) Lesser Antilles, The BahamasUnited States East... 4000 NaN NaN
...
To improve you example you can do the following:
import requests
from bs4 import BeautifulSoup
wiki = "https://en.wikipedia.org/wiki/List_of_Category_5_Atlantic_hurricanes"
page = requests.get(wiki).content
soup = BeautifulSoup(page,'lxml')
table=soup.find('table', class_="wikitable sortable") # The class of the list in wikipedia
data = []
for row in table.select('tr')[1:-1]:
cells = []
for cell in row.select('td'):
cells.append(cell.get_text('',strip=True))
data.append(cells)
get_text('',strip=True)
will get the text from the td
and strip the space in front/end.
Answered By - HedgeHog
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.