[FIXED] Scrapy FormRequest

Issue

I'm having trouble with scrapy FormRequest. I am trying to get all reviews from this page (infinite scrolling) : https://www.temporel-voyance.com/voyance/planning/consultations-voyants-en-privee/angele/1041

When I scroll, I can see a post request with data form : {xyz":"3"}

But when I try it with scrapy command line, I am not able to get the correct response.

from scrapy import FormRequest

fetch("https://www.temporel-voyance.com/voyance/planning/consultations-voyants-en-privee/angele/1041")

req = FormRequest(response.url, formdata={"xyz":"3"})

fetch(req)

I had a look to this page : https://blog.scrapinghub.com/2016/06/22/scrapy-tips-from-the-pros-june-2016 but I am still stuck.

Can someone help me ?

Thank you very much ! Mylha

Solution

If you take a better look at the request being made, you'll see that it's actually to a different url, https://www.temporel-voyance.com/voyants/temoignages?voyant_id=1041

If you make your form request to that url, you get the updated data:

>>> req = scrapy.FormRequest('https://www.temporel-voyance.com/voyants/temoignages?voyant_id=1041', formdata={'xyz': '3'})
>>> fetch(req)
2018-10-20 13:43:42 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.temporel-voyance.com/voyants/temoignages?voyant_id=1041> (referer: None)
>>> response.css('.name_user b::text').getall()
['Chou', 'Jérôme', 'Sonia']

Answered By - stranac