[FIXED] How can we download multiple CSV files from a URL?

Issue

I am testing this code.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


d = webdriver.Chrome('C:\\Utility\\chromedriver.exe')
d.get('https://developers.humana.com/Resource/PCTFilesList?fileType=innetwork')

# stuck here...
#links = 
for link in links:
    d.get(link)
    # click page 2, 3, 4, etc., up to 100
    for page in range(1, 100)
    page.click
d.quit()

So, I am trying to download CSV files on page 1, then click page 2 and download those files, and then click page 3 and again download those files. The sample code that I shared here should be a start, I think, but it definitely needs some improvements to work right.

How can I do this?

Solution

You can use this solution:

import requests

length = 1
url = "https://developers.humana.com/Resource/GetData?fileType=innetwork&sEcho=1&iColumns=3&sColumns=%2C%2C\
                                                                                                &iDisplayStart=0&iDisplayLength="

r = requests.get(url+str(length))
json_data = r.json()

length = json_data['iTotalRecords']
print("files ", length)
r = requests.get(url+str(length))
json_data = r.json()

for e in json_data['aaData']:
    download_url = "https://developers.humana.com/Resource/DownloadPCTFile?fileType=innetwork&fileName=" + e['name']
    print(e['name'])
    print("download url: ", download_url)

then just download files in loop.

Answered By - Max