[FIXED] How do i declare data-test-id attribute inside web scraping script

Issue

I am trying to scrape images from the toronto star news website (https://thestar.com), I want to scrape all the images with a data-test-id attribute of "medicard-image".

Here's the code i am running to preview the images i have scraped:

ht_r = requests.get("https://www.thestar.com/")
ht_soup = BeautifulSoup(ht_r.content, 'html.parser')
ht_images = ht_soup.findAll('img', attr={"data-test-id": "medicard-image"})
ht_images = ht_images[0:14]

However whenever i try to print ht_images , i get an output of [].

What am i doing wrong?

Solution

The images you see on the page are injected via JavaScript so BeautifulSoup doesn't see them. You can use this code to print all image URLs on page:

import re
import json
import requests


ht_r = requests.get("https://www.thestar.com/")
data = re.search(r"window\.__PRELOADED_STATE__ = (.*)", ht_r.text)
data = json.loads(data.group(1))

# uncomment to print all data:
# print(json.dumps(data, indent=4))

def find_images(d):
    if isinstance(d, dict):
        for k, v in d.items():
            if k == "image":
                yield v
            else:
                yield from find_images(v)
    if isinstance(d, list):
        for v in d:
            yield from find_images(v)


for img in find_images(data):
    if img["url"]:
        print("https://www.thestar.com" + img["url"])

Prints:

https://www.thestar.com/content/dam/thestar/politics/political-opinion/2022/04/12/justin-trudeau-has-an-untested-plan-to-create-jobs-will-canadians-have-the-patience-to-see-if-it-works/chrystia_freeland.jpg
https://www.thestar.com/content/dam/thestar/opinion/star-columnists/2022/04/12/blasting-kieran-moore-for-his-vacation-well-need-him-on-our-side-for-bigger-battles/kieran_moore.jpg
https://www.thestar.com/content/dam/thestar/opinion/star-columnists/2022/04/12/as-omicron-rises-are-things-falling-apart-are-we/thief_on_camera.jpg
https://www.thestar.com/content/dam/thestar/entertainment/opinion/2022/04/12/if-britney-spears-really-is-pregnant-its-time-to-get-off-social-media/britney_and_sam.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/toronto-police-chief-set-to-announce-arrests-in-daylight-murders-of-international-student-and-35-year-old-man/victims_2_.jpg
https://www.thestar.com/content/dam/thestar/politics/provincial/2022/04/12/doug-ford-draws-fire-for-comments-during-top-doctors-caribbean-vacation/doug_ford.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/just-15-per-cent-of-homeowners-hold-nearly-one-third-of-ontarios-housing-wealth-says-report/for_lease.jpg
https://www.thestar.com/content/dam/thestar/politics/federal/2022/04/12/canada-is-fighting-russias-invasion-of-ukraine-with-sanctions-but-are-they-actually-working/vladimir_putin.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/04/12/0799a6092c33428f8541cc8a6fa64c28-0799a6092c33428f8541cc8a6fa64c2-1.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/i-beg-you-mother-pleads-for-information-in-shocking-wasaga-beach-abduction-of-elnaz-hajtamiri/_1_elnaz.jpg
https://www.thestar.com/content/dam/thestar/politics/provincial/2022/04/12/ndp-says-it-will-review-policy-of-allowing-sitting-members-to-be-challenged-for-right-to-run-in-elections-andrea-horwath-says/horwath.jpg
https://www.thestar.com/content/dam/thestar/entertainment/2022/04/12/gilbert-gottfriend-has-died-at-age-67-heres-how-his-fellow-actors-and-comics-are-reacting-to-the-news/_37c33656f726494595f3491faf2d450c_37c33656f726494595f3491faf2d450.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/toronto-police-officer-faces-eight-charges-of-theft/toronto_police_cruiser.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/73-year-old-toronto-man-charged-with-kidnapping-at-royal-york-station/cruiser.jpg
https://www.thestar.com/content/dam/thestar/business/2022/04/12/to-mask-or-not-to-mask-without-ontario-backing-businesses-face-potential-legal-landmines/gil_2.jpg
https://www.thestar.com/content/dam/thestar/business/2022/04/12/this-will-remove-buyers-from-the-market-higher-interest-rates-on-wednesday-will-help-tame-frothy-housing-prices-experts-say/housing_sales.jpg
https://www.thestar.com/content/dam/thestar/news/canada/2022/04/12/coronavirus-covid-19-updates-toronto-canada-april-12/ptw_features_april_04.jpg
https://www.thestar.com/content/dam/thestar/news/canada/2022/04/11/pierre-poilievre-did-a-campaign-video-about-this-vancouver-house-heres-the-story-of-how-its-price-climbed-from-265000-to-49-million/house.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/02/28/headerimage-0-0-0.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/13-year-old-boy-granted-bail-in-shooting-death-of-toronto-boy-15/afwyouthjail02.jpg
https://www.thestar.com/content/dam/localcommunities/brampton_guardian/news/crime/2022/04/12/brampton-father-confronts-alleged-carjackers-trying-to-steal-suv-with-baby-in-back-seat/10605582_Bramptoncarjacking.JPG
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/donated-toys-were-a-big-hit-with-kids-and-parents-at-this-toronto-playground-then-the-city-tossed-them/sr_cudmorecreekpark_01.jpg
https://www.thestar.com/content/dam/thestar/sports/leafs/2022/04/12/leafs-star-auston-matthews-is-outta-control-as-he-seeks-goal-no-60-against-sabres/matthews.jpg
https://www.thestar.com/content/dam/thestar/sports/raptors/analysis/2022/04/12/raptors-coach-nick-nurse-is-in-demand-for-a-reason-he-demands-success-and-his-bag-of-tricks-is-deep/nick_nurse.jpg
https://www.thestar.com/content/dam/thespec/news/hamilton-region/2022/04/12/dominican-republic-pivot-airlines-cocaine-bust/aatif_safdar.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/04/12/real-estate-sold-sign-13superportrait-0.jpg
https://www.thestar.com/content/dam/localcommunities/scarborough_mirror/News/gta/2022/04/12/toronto-zoo-aims-to-keep-its-new-and-expectant-moms-healthy-happy/10605993_DawnMihailovicwithHermanVancouverIslandMarmot.JPG

...and so on.

Answered By - Andrej Kesely