Issue
Consider this link : https://www.michaelkors.com/logo-tape-ribbed-stretch-viscose-sweater/_/R-US_MH86NXK5ZW
Once you scroll down you see recommendations on this page. I want to get titles of these products mentioned. I have tried using this:
response.xpath('//div[@class="product-tile-container"]/a/@src').getall()
However, it returns nothing. Reason might be that the products are lazy loaded. How should I extract their titles.
Solution
You can use selenium
to scroll to the bottom of the page. However, it still takes a while for the site to load the recommendations. As such, this solution waits by using a while
loop until the product recommendation section appears:
from selenium import webdriver
from bs4 import BeautifulSoup as soup
import time
d = webdriver.Chrome('/Users/jamespetullo/Downloads/chromedriver')
d.get('https://www.michaelkors.com/logo-tape-ribbed-stretch-viscose-sweater/_/R-US_MH86NXK5ZW')
last_height = d.execute_script("return document.body.scrollHeight")
while True:
d.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(0.5)
new_height = d.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
start = soup(d.page_source, 'html.parser')
while start.find('div', {'class':'product-tile-rfk'}) is None:
start = soup(d.page_source, 'html.parser')
products = [i.find_all('li', {'class':'product-name-container'})[0].text for i in start.find_all('div', {'class':'product-tile-rfk'})]
Output:
['Ribbed Stretch-Viscose Tank', 'Ribbed Stretch-Viscose Tank Top', 'Ribbed Stretch-Viscose Tank Top', 'Stretch-Viscose Tank', 'Striped Ribbed Sweater Tank', 'Tie-Dye Stretch-Viscose Sweater', 'Striped Stretch-Viscose Tank', 'Striped Stretch-Cotton Sweater', 'Rainbow Stretch-Viscose Short-Sleeve Sweater', 'Stretch-Viscose Cropped Tank', 'Neon Striped Stretch-Viscose Tank Top', 'Geometric Grid Stretch-Viscose Top', 'Logo Tape Stretch-Viscose Pullover', 'Logo Tape Stretch-Viscose Cropped T-Shirt', 'Logo Tape Cotton-Jersey Top', 'Logo Tape Viscose Joggers', 'Logo Tape Buttoned Track Pants', 'Contrast Stripe Joggers', 'Contrast Stripe Hooded Jacket', 'Logo Tape Stretch-Viscose Pencil Skirt', 'Logo Tape Stretch-Viscose Zip-Up Hoodie', 'Stretch-Viscose Joggers', 'Cotton Asymmetric Turtleneck', 'Logo Tape Ribbed Knit Dress']
Answered By - Ajax1234
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.