[FIXED] Crawling website by selecting java script drop down menu in scrapy using splash

Issue

I am trying to get daily prices from https://www.steelmint.com/ingot-prices-indian . I have setup a scrapy script using splash where I need to select drop down menu of different dates and scrap price as number. I just need two data from page, date and price.

I am not able to get drop down change it value and now able to find any tutorial which guides towards it. Most deal with form handling but is not working.

My lua script using Splash is:

    function main(splash, args)
  local form = splash:select('form-control')
  local values = assert(form:form_values())
  values.frmDt = "14"
  values.frmMt = "March"
  values.frmYr = "2018"
  assert(form:fill(values))
  assert(splash:go(args.url))
  assert(splash:wait(0.5))
  return {
    html = splash:html(),``
    png = splash:png(),
    har = splash:har(),
  }
end

Once page is rendered I am easily getting value. Newbie here. Thanks in advance.

Solution

I think that you should run the javascript through splash on the page, it is simpler. Look at the following working example:

function main(splash, args)
  assert(splash:go(args.url))

  assert(splash:runjs('document.getElementById("frmDt").value = "14"'))
  assert(splash:runjs('document.getElementById("frmMt").value = "March"'))
  assert(splash:runjs('document.getElementById("frmYr").value = "2018"'))

  assert(splash:wait(0.5))
  return {
    html = splash:html(),``
    png = splash:png(),
    har = splash:har(),
  }
end

Answered By - aleroot