[FIXED] Web scraping coinmarketcap.com with Python (requests & BeautifulSoup)

Issue

I want to build a list with coins from coinmarketcap.com. Every element should be a tuple.

Something like:

coins = [('btc',8500,'+0.5%','+1.2%', '-1%'), ...]

I can't get percentage: The information is in td like this:

<td class="no-wrap percent-change   text-right positive_change" data-timespan="1h" data-percentusd="0.99" data-symbol="BTC" data-sort="0.991515">0.99%</td>

How can I access 0.99% value from above? I need in fact data-percentageusd from td but I don't know what that is.

My testing script is something like:

import requests
from bs4 import BeautifulSoup

url = 'https://coinmarketcap.com/all/views/all/'
page = requests.get(url)
soup = BeautifulSoup(page.content,'html.parser')

name = soup.find_all('a', class_='currency-name-container')
price = soup.find_all('a', class_='price')
print(name)
print(price)
#how can percentage modification for 1h, 24h, 7d?
#delta_h = soup.find_all('td', ???)

Solution

You can loop over the rows of the table to get the data for each currency and store it in a tuple, and then add it to the list.

r = requests.get('https://coinmarketcap.com/all/views/all/')
soup = BeautifulSoup(r.text, 'lxml')

data = []
table = soup.find('table', id='currencies-all')
for row in table.find_all('tr'):
    try:
        symbol = row.find('td', class_='text-left col-symbol').text
        price = row.find('a', class_='price').text
        time_1h = row.find('td', {'data-timespan': '1h'}).text
        time_24h = row.find('td', {'data-timespan': '24h'}).text
        time_7d = row.find('td', {'data-timespan': '7d'}).text
    except AttributeError:
        continue

    data.append((symbol, price, time_1h, time_24h, time_7d))

for item in data:
    print(item)

Partial Output:

('BTC', '$8805.46', '0.88%', '-12.30%', '-19.95%')
('ETH', '$677.45', '0.98%', '-11.54%', '-21.66%')
('XRP', '$0.780113', '0.62%', '-10.63%', '-14.42%')
('BCH', '$970.70', '1.01%', '-11.33%', '-23.89%')
('LTC', '$166.70', '0.74%', '-10.06%', '-19.56%')
('NEO', '$83.55', '0.24%', '-16.29%', '-33.39%')
('XLM', '$0.286741', '1.13%', '-13.23%', '-11.84%')
('ADA', '$0.200449', '0.63%', '-16.92%', '-31.43%')
('XMR', '$256.92', '0.63%', '-19.98%', '-19.46%')

Since the data is missing for some currencies in the table, the code will raise an AttributeError for .text. To skip those currencies, I've used the try-except.

Answered By - Keyur Potdar