[FIXED] Using Python to get holdings and allocation of ETFs

Issue

Referencing this old post, does anyone know how to adjust the below script to also bring in the weight(%) column?

import requests
import re

keys = ['XLU', 'XLRE']


headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0"
}


def main(url):
    with requests.Session() as req:
        req.headers.update(headers)
        for key in keys:
            r = req.get(url.format(key))
            print(f"Extracting: {r.url}")
            goal = re.findall(r'etf\\\/(.*?)\\', r.text)
            print(goal)


main("https://www.zacks.com/funds/etf/{}/holding")

Thank you Tim for you support! I have altered your script a tad to conform it into a dataframe. Here is that alteration.

...

ticker = []
weight = []
def main(url):
...
                for holding in data: 
                    goal = re.search(r'etf/([^"]*)', holding[1])
                    if goal:
                        # print(goal.group(1), *holding[2:5])
                        ticker.append(goal.group(1))
                        weight.append(*holding[3:4])
                break

main("https://www.zacks.com/funds/etf/{}/holding")

Allocation_Summary = pd.concat([pd.DataFrame(ticker), pd.DataFrame(weight)], axis=1)
Allocation_Summary.columns = ['Ticker', 'Weight']
Allocation_Summary = Allocation_Summary[Allocation_Summary['Ticker'].notnull()]
Allocation_Summary

Solution

Hopefully, this can give you a clue. I have Python read the JSON, and then parse what it needs from the JSON:

import requests
import json
import re
from pprint import pprint

keys = ['XLU', 'XLRE']


headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0"
}


def main(url):
    with requests.Session() as req:
        req.headers.update(headers)
        for key in keys:
            r = req.get(url.format(key))
            print(f"Extracting: {r.url}")
            for line in r.text.splitlines():
                if not line.startswith('etf_holdings.formatted_data'):
                    continue
                data = json.loads(line[30:-1])
                for holding in data:
                    goal = re.search(r'etf/([^"]*)', holding[1])
                    if goal:
                        print(goal.group(1), *holding[2:5])
                break


main("https://www.zacks.com/funds/etf/{}/holding")

Output:

Extracting: https://www.zacks.com/funds/etf/XLU/holding
NEE 25,229,358 16.19 14.60
DUK 9,894,174 8.50 12.12
SO 13,616,787 7.24 6.79
D 10,374,023 6.65 -4.85
EXC 12,569,054 4.90 34.42
AEP 6,428,098 4.73 -6.21
SRE 4,052,674 4.72 1.21
XEL 6,922,787 4.00 -8.07
PEG 6,496,807 3.37 8.73
AWK 2,334,148 3.16 13.84
WEC 4,057,352 3.15 -9.72
ES 4,417,902 3.08 -4.51
ED 4,409,531 2.76 -5.94
PPL 9,896,934 2.41 5.49
DTE 2,491,854 2.41 -7.20
EIX 4,880,585 2.41 11.26
AEE 3,287,118 2.29 3.59
ETR 2,581,009 2.27 0.20
FE 6,996,004 2.26 21.32
AES 8,569,861 1.94 28.87
CMS 3,723,268 1.93 -5.66
CNP 7,466,742 1.61 22.25
EVRG 2,948,980 1.57 15.13
LNT 3,217,430 1.57 1.89
ATO 1,680,813 1.41 -0.61
NRG 3,148,151 1.13 28.14
NI 5,054,044 1.08 6.43
PNW 1,453,458 1.04 -22.09
DTM 30,000 0.01 4,796.00
Extracting: https://www.zacks.com/funds/etf/XLRE/holding
AMT 1,699,073 13.15 22.53
PLD 2,764,320 9.49 48.66
CCI 1,615,015 9.02 14.95
EQIX 334,779 7.60 10.71
PSA 568,644 4.88 43.73
SPG 1,227,275 4.51 141.24
DLR 1,051,865 4.48 5.29
WELL 1,559,663 3.76 53.85
SBAC 408,435 3.74 16.89
AVB 521,505 3.13 78.34
CBRE 1,253,880 3.06 136.98
EQR 1,285,424 2.86 86.85
WY 2,798,892 2.72 30.22
ARE 512,452 2.67 34.15
O 1,395,548 2.67 23.56
EXR 499,570 2.36 70.59
VTR 1,401,434 2.28 37.41
ESS 242,807 2.11 82.44
MAA 427,719 2.07 81.63
PEAK 2,013,476 1.91 33.60
DRE 1,400,957 1.90 51.35
BXP 530,590 1.75 59.72
UDR 1,110,070 1.57 87.60
IRM 1,078,881 1.31 73.21
HST 2,638,290 1.29 61.05
REG 591,049 1.08 99.12
KIM 1,621,908 0.96 121.14
FRT 263,936 0.88 77.22
VNO 585,554 0.78 40.37

Answered By - Tim Roberts