[FIXED] Web-scraping: Empty dataset after collecting information

Issue

I would like to create a dataset that includes information scraped from a website. I explain what I have done and the expected output below. I am getting empty arrays for rows and columns, then for the whole dataset, and I do not understand the reason. I hope you can help me.

1) Create an empty dataframe with only one column: this columns should contains a list of urls to use.

data_to_use = pd.DataFrame([], columns=['URL'])

2) Select urls from a previous dataset.

select_urls=dataset.URL.tolist()

This set of urls looks like:

                             URL
0                     www.bbc.co.uk
1             www.stackoverflow.com           
2                       www.who.int
3                       www.cnn.com
4         www.cooptrasportiriolo.it
...                             ...

3) Populate the column with these urls:

data_to_use['URL']= select_urls
data_to_use['URLcleaned'] = data_to_use['URL'].str.replace('^(www\.)', '')

4) Select a random sample to test: the first 50 rows in column URL

data_to_use = data_to_use.loc[1:50, 'URL']

5) Try to scrape information

import requests
import time
from bs4 import BeautifulSoup

urls= data_to_use['URLcleaned'].tolist()

ares = []

for u in urls: # in the selection there should be an error. I am not sure that I am selecting the rig
    print(u)
    url = 'https://www.urlvoid.com/scan/'+ u
    r = requests.get(url)
    ares.append(r)   

rows = []
cols = []

for ar in ares:
    soup = BeautifulSoup(ar.content, 'lxml')
    tab = soup.select("table.table.table-custom.table-striped")   
    try:
            dat = tab[0].select('tr')
            line= []
            header=[]
            for d in dat:
                row = d.select('td')
                line.append(row[1].text)
            new_header = row[0].text
            if not new_header in cols:
                cols.append(new_header)
            rows.append(line)
    except IndexError:
        continue

print(rows) # this works fine. It prints the rows. The issue comes from the next line

data_to_use = pd.DataFrame(rows,columns=cols)  

Unfortunately there is something wrong in the steps above as I am not getting any results, but only [] or __.

Error from data_to_use = pd.DataFrame(rows,columns=cols):

ValueError: 1 columns passed, passed data had 12 columns

My expected output would be:

URL          Website Address   Last Analysis   Blacklist Status \  
bbc.co.uk          Bbc.co.uk         9 days ago       0/35
stackoverflow.com Stackoverflow.com  7 days ago      0/35

Domain Registration               IP Address       Server Location    ...
996-08-01 | 24 years ago       151.101.64.81    (US) United States    ...
2003-12-26 | 17 years ago      ...

At the end I should save the dataset created in a file csv.

Solution

Putting aside the conversion to csv, let's try it this way:

urls=['gov.ie', 'who.int', 'comune.staranzano.go.it', 'cooptrasportiriolo.it', 'laprovinciadicomo.it', 'asufc.sanita.fvg.it', 'canale7.tv', 'gradenigo.it', 'leggo.it', 'urbanpost.it', 'monitorimmobiliare.it', 'comune.villachiara.bs.it', 'ilcittadinomb.it', 'europamulticlub.com']
ares = []
for u in urls:
    url = 'https://www.urlvoid.com/scan/'+u
    r = requests.get(url)
    ares.append(r)

Note that 3 of the urls have no data, so you should have only 11 rows in the dataframe. Next:

rows = []
cols = []
for ar in ares:
    soup = bs(ar.content, 'lxml')
    tab = soup.select("table.table.table-custom.table-striped")        
    if len(tab)>0:
        dat = tab[0].select('tr')
        line= []
        header=[]
        for d in dat:
            row = d.select('td')
            line.append(row[1].text)
            new_header = row[0].text
            if not new_header in cols:
                cols.append(new_header)
        rows.append(line)

my_df = pd.DataFrame(rows,columns=cols)   
my_df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 11 entries, 0 to 10
Data columns (total 12 columns):
Website Address        11 non-null object
Last Analysis          11 non-null object
Blacklist Status       11 non-null object
Domain Registration    11 non-null object
Domain Information     11 non-null object
IP Address             11 non-null object
Reverse DNS            11 non-null object
ASN                    11 non-null object
Server Location        11 non-null object
Latitude\Longitude     11 non-null object
City                   11 non-null object
Region                 11 non-null object
dtypes: object(12)
memory usage: 1.2+ KB

Answered By - Jack Fleeting