[FIXED] Using bs4 obtain linkage information and stored to dataframe

Issue

I want to get links for below web and stored in dataframe result_total, anyone can help ? (currently, the code feedback a empty result_total)

And I ues for con in cons to retrieve contents in the linkage, is there other way can obtain them at one time ?

Thanks!

import requests
from bs4 import BeautifulStoneSoup
import pandas as pd

url='https://www.nowmsg.com/findzip/findzip.asp'
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36'}
web = requests.get(url,headers = headers)
soup = BeautifulStoneSoup(web.text)
cons = soup.findAll("div","col-md-3")

result_total= pd.DataFrame()
result_single= pd.DataFrame()
for con in cons:
    con_1=con.find('a')
    link= con_1.get('href')
    text=con_1.text
    result_single['link'] = link
    result_single['state'] = text
    result_total = result_total.append(result_single)

Solution

Try this:

import pandas as pd
import requests
from bs4 import BeautifulStoneSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36',
}
web = requests.get('https://www.nowmsg.com/findzip/findzip.asp', headers=headers)
soup = [
    [
        x.find("a").getText(),
        f'https://www.nowmsg.com/findzip/{x.find("a")["href"]}',
    ]
    for x in BeautifulStoneSoup(web.text).find_all("div", "col-md-3")
]
df = pd.DataFrame(soup, columns=["State", "Link"])
print(df)

Output:

                        State                                               Link
0                Alabama - AL  https://www.nowmsg.com/findzip/county.asp?coun...
1                 Alaska - AK  https://www.nowmsg.com/findzip/county.asp?coun...
2                Arizona - AZ  https://www.nowmsg.com/findzip/county.asp?coun...
3               Arkansas - AR  https://www.nowmsg.com/findzip/county.asp?coun...
4             California - CA  https://www.nowmsg.com/findzip/county.asp?coun...
5               Colorado - CO  https://www.nowmsg.com/findzip/county.asp?coun...
6            Connecticut - CT  https://www.nowmsg.com/findzip/county.asp?coun...
7               Delaware - DE  https://www.nowmsg.com/findzip/county.asp?coun...
8   District of Columbia - DC  https://www.nowmsg.com/findzip/county.asp?coun...
9                Florida - FL  https://www.nowmsg.com/findzip/county.asp?coun...
10               Georgia - GA  https://www.nowmsg.com/findzip/county.asp?coun...
11                Hawaii - HI  https://www.nowmsg.com/findzip/county.asp?coun...
12                 Idaho - ID  https://www.nowmsg.com/findzip/county.asp?coun...
13              Illinois - IL  https://www.nowmsg.com/findzip/county.asp?coun...
14               Indiana - IN  https://www.nowmsg.com/findzip/county.asp?coun...
15                  Iowa - IA  https://www.nowmsg.com/findzip/county.asp?coun...
16                Kansas - KS  https://www.nowmsg.com/findzip/county.asp?coun...
17              Kentucky - KY  https://www.nowmsg.com/findzip/county.asp?coun...
18             Louisiana - LA  https://www.nowmsg.com/findzip/county.asp?coun...
19                 Maine - ME  https://www.nowmsg.com/findzip/county.asp?coun...
20              Maryland - MD  https://www.nowmsg.com/findzip/county.asp?coun...
21         Massachusetts - MA  https://www.nowmsg.com/findzip/county.asp?coun...
22              Michigan - MI  https://www.nowmsg.com/findzip/county.asp?coun...
23             Minnesota - MN  https://www.nowmsg.com/findzip/county.asp?coun...
24           Mississippi - MS  https://www.nowmsg.com/findzip/county.asp?coun...
25              Missouri - MO  https://www.nowmsg.com/findzip/county.asp?coun...
26               Montana - MT  https://www.nowmsg.com/findzip/county.asp?coun...
27              Nebraska - NE  https://www.nowmsg.com/findzip/county.asp?coun...
28                Nevada - NV  https://www.nowmsg.com/findzip/county.asp?coun...
29         New Hampshire - NH  https://www.nowmsg.com/findzip/county.asp?coun...
30            New Jersey - NJ  https://www.nowmsg.com/findzip/county.asp?coun...
31            New Mexico - NM  https://www.nowmsg.com/findzip/county.asp?coun...
32              New York - NY  https://www.nowmsg.com/findzip/county.asp?coun...
33        North Carolina - NC  https://www.nowmsg.com/findzip/county.asp?coun...
34          North Dakota - ND  https://www.nowmsg.com/findzip/county.asp?coun...
35                  Ohio - OH  https://www.nowmsg.com/findzip/county.asp?coun...
36              Oklahoma - OK  https://www.nowmsg.com/findzip/county.asp?coun...
37                Oregon - OR  https://www.nowmsg.com/findzip/county.asp?coun...
38          Pennsylvania - PA  https://www.nowmsg.com/findzip/county.asp?coun...
39          Rhode Island - RI  https://www.nowmsg.com/findzip/county.asp?coun...
40        South Carolina - SC  https://www.nowmsg.com/findzip/county.asp?coun...
41          South Dakota - SD  https://www.nowmsg.com/findzip/county.asp?coun...
42             Tennessee - TN  https://www.nowmsg.com/findzip/county.asp?coun...
43                 Texas - TX  https://www.nowmsg.com/findzip/county.asp?coun...
44                  Utah - UT  https://www.nowmsg.com/findzip/county.asp?coun...
45               Vermont - VT  https://www.nowmsg.com/findzip/county.asp?coun...
46              Virginia - VA  https://www.nowmsg.com/findzip/county.asp?coun...
47            Washington - WA  https://www.nowmsg.com/findzip/county.asp?coun...
48         West Virginia - WV  https://www.nowmsg.com/findzip/county.asp?coun...
49             Wisconsin - WI  https://www.nowmsg.com/findzip/county.asp?coun...
50               Wyoming - WY  https://www.nowmsg.com/findzip/county.asp?coun...

Answered By - baduker