[FIXED] How to get comment in XML tag from response use Scrapy?

Issue

I have response from server like code below.
I want to get the link in the CDATA section in between the <lyric></lyric> tags.

<lyric><![CDATA[https://lrc-nct.nixcdn.com/2018/02/21/f/b/1/1/1519207822262.lrc]]></lyric>
<bgimage><![CDATA[https://avatar-nct.nixcdn.com/singer/avatar/2018/02/25/e/b/b/b/1519558155015_600.jpg]]></bgimage>
<avatar><![CDATA[https://avatar-nct.nixcdn.com/song/2018/02/26/f/8/3/d/1519640161758.jpg]]></avatar>
<coverimage><![CDATA[https://avatar-nct.nixcdn.com/song/2018/02/26/f/8/3/d/1519640161758_500.jpg]]></coverimage>
<newtab><![CDATA[https://www.nhaccuatui.com/nghe-si-hang-bingboong.html]]></newtab>

Solution

By default lxml strips away cdata and unfortunately parsel.Selector scrapy is using does not expose that option.

So you need to create lxml tree manually and then recreate your Selector:

$ scrapy shell "https://www.nhaccuatui.com/flash/xml?html5=true&key1=59f0ae8a89cea4a0eb2c3b7e40208f26"
from lxml.etree import XMLParser
from parsel import Selector

# lets fix selector
parser = XMLParser(strip_cdata=False)
root = etree.fromstring(response.body, parser=parser, base_url=response.url)
selector = Selector(root=root)

# now finding CDATA values
selector.xpath('//lyric/text()').extract()
[OUT]: ['https://lrc-nct.nixcdn.com/2018/02/07/a/a/e/f/1517979335534.lrc']

Answered By - Granitosaurus