[FIXED] how to get element with xpath by attribute value

Issue

trying to get the next page arrow link from this page. however response.selector.xpath('//a[aria-label="Next page"]') yields []. help! :) I was able to solve it with response.selector.xpath('//*[contains(@class, "Pagination_arrowItem")]/a')[1].attrib['href'] but then run into an issue if it's the firts page and only one arrow is shown.

Solution

You should be able to get it with:

response.selector.xpath('//a[@data-e2e-test="pagination-next"]')

See Scrapy documentation for more.

Answered By - Barry the Platipus