[FIXED] Scrapy. find a tag by its content

Issue

How to find a tag by its content? This is how I find the necessary elements, but the structure on some pages is different and this does not always work.

yield {
            ...
            'Education': response.css('.provider-item:nth-child(3) .h2-style+ span::text').get(),
            'Training': response.css('.provider-item:nth-child(4) .h2-style+ span::text').get(),
            ...                
        }

Solution

Check out the code sample

In [4]: i = response.xpath('.//span[contains(text(),"Education")]')

In [5]: i
Out[5]: [<Selector xpath='.//span[contains(text(),"Education")]' data='<span class="listing-h2 h2-style">Edu...'>]

In [6]: i.xpath('following-sibling::span[1]/text()').extract()
Out[6]:
['A.B. in Economics with a minor in Asian Studies, ',
 'Occidental College',
 'Masters in Chinese Medicine, Tai Hsuan Foundation']

Answered By - Arun Augustine