[FIXED] Scrapy using files.middleware downloads given file without extension

Issue

I want to automate some file-exchange. I need to download a .csv-file from a website, which is secured with an authentication before you can start to download.

First, I wanted to try downloading the file, with wget, but I did not manage, so I switched to scrapy and everything works fine, my authentication and the download, BUT the file comes without an extension -.-'

here is a snippet of my spider:

    def after_login(self, response):
        accview = response.xpath('//span[@class="user-actions welcome"]')
        if accview:
            print('Logged in')
            file_url = response.xpath('//article[@class="widget-single widget-shape-widget widget"]/p/a/@href').get()
            file_url = response.urljoin(file_url)
            items = StockfwItem()
            items['file_urls'] = [file_url]
            yield items

my settings.py:

ITEM_PIPELINES = {'scrapy.pipelines.files.FilesPipeline': 1}

items.py:

    file_urls = scrapy.Field()
    files = scrapy.Field()

The reason why I am sure, that there is a problem with my spider, is that, if I download the file regular via brower, it always comes as a regular csvfile.

When I try to open the downloaded file(filename is hashed in sha1), I get the following error_msg:

  File "/usr/lib/python3.6/csv.py", line 111, in __next__
    self.fieldnames
  File "/usr/lib/python3.6/csv.py", line 98, in fieldnames
    self._fieldnames = next(self.reader)
_csv.Error: line contains NULL byte

Also when I open the downloaded file with notepad++ and save encoding as utf-8, it works without any problems...

scrapy console output:

{'file_urls': ['https://floraworld.be/Servico.Orchard.FloraWorld/Export/Export']                                                                                                                                                             ,
 'files': [{'checksum': 'f56c6411803ec45863bc9dbea65edcb9',
            'path': 'full/cc72731cc79929b50c5afb14e0f7e26dae8f069c',
            'status': 'downloaded',
            'url': 'https://floraworld.be/Servico.Orchard.FloraWorld/Export/Expo                                                                                                                                                             rt'}]}
2021-08-02 10:00:30 [scrapy.core.engine] INFO: Closing spider (finished)
2021-08-02 10:00:30 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2553,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 2,
 'downloader/request_method_count/POST': 2,
 'downloader/response_bytes': 76289,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 3,
 'downloader/response_status_count/302': 1,
 'elapsed_time_seconds': 20.892172,
 'file_count': 1,
 'file_status_count/downloaded': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2021, 8, 2, 8, 0, 30, 704638),
 'item_scraped_count': 1,
 'log_count/DEBUG': 6,
 'log_count/INFO': 10,
 'log_count/WARNING': 1,
 'memusage/max': 55566336,
 'memusage/startup': 55566336,
 'request_depth_max': 1,
 'response_received_count': 3,
 'scheduler/dequeued': 4,
 'scheduler/dequeued/memory': 4,
 'scheduler/enqueued': 4,
 'scheduler/enqueued/memory': 4,
 'splash/render.html/request_count': 1,
 'splash/render.html/response_count/200': 1,
 'start_time': datetime.datetime(2021, 8, 2, 8, 0, 9, 812466)}
2021-08-02 10:00:30 [scrapy.core.engine] INFO: Spider closed (finished)

snippet of downloaded file and opened with vim on ubuntu server:

"^@A^@r^@t^@i^@c^@l^@e^@C^@o^@d^@e^@"^@;^@"^@D^@e^@s^@c^@r^@i^@p^@t^@i^@o^@n^@"^@;^@"^@B^@B^@"^@;^@"^@K^@T^@"^@;^@"^@S^@i^@z^@e^@"^@;^@"^@P^@r^@i^@c^@e^@"^@;^@"^@S^@t^@o^@c^@k^@"^@;^@"^@D^@e^@l^@i^@v^@e^@r^@y^@D^@a^@t^@e^@"^@^M^@
^@"^@1^@0^@0^@0^@L^@"^@;^@"^@A^@l^@o^@e^@ ^@p^@l^@a^@n^@t^@ ^@x^@ ^@2^@3^@ ^@l^@v^@s^@"^@;^@"^@4^@"^@;^@"^@4^@"^@;^@"^@6^@5^@"^@;^@"^@4^@6^@,^@7^@7^@"^@;^@"^@1^@1^@8^@,^@0^@0^@0^@0^@0^@"^@;^@"^@"^@^M^@
^@"^@1^@0^@0^@0^@M^@"^@;^@"^@A^@l^@o^@e^@ ^@p^@l^@a^@n^@t^@ ^@x^@ ^@1^@7^@ ^@l^@v^@s^@"^@;^@"^@4^@"^@;^@"^@1^@2^@"^@;^@"^@5^@0^@"^@;^@"^@3^@2^@,^@6^@1^@"^@;^@"^@2^@0^@6^@,^@0^@0^@0^@0^@0^@"^@;^@"^@"^@^M^@
^@"^@1^@0^@0^@0^@S^@"^@;^@"^@A^@l^@o^@e^@ ^@p^@l^@a^@n^@t^@ ^@x^@ ^@1^@6^@ ^@l^@v^@s^@"^@;^@"^@4^@"^@;^@"^@2^@4^@"^@;^@"^@4^@0^@"^@;^@"^@2^@2^@,^@3^@2^@"^@;^@"^@-^@6^@,^@0^@0^@0^@0^@0^@"^@;^@"^@2^@3^@/^@0^@8^@/^@2^@0^@2^@1^@"^@^M^@
^@"^@1^@0^@0^@2^@M^@"^@;^@"^@B^@A^@T^@O^@N^@ ^@P^@L^@A^@N^@T^@ ^@6^@7^@ ^@C^@M^@ ^@W^@/^@P^@O^@T^@"^@;^@"^@2^@"^@;^@"^@6^@"^@;^@"^@6^@7^@"^@;^@"^@2^@2^@,^@4^@2^@"^@;^@"^@3^@3^@,^@0^@0^@0^@0^@0^@"^@;^@"^@5^@/^@0^@9^@/^@2^@0^@2^@1^@"^@^M^@
^@"^@1^@0^@0^@2^@S^@"^@;^@"^@B^@A^@T^@O^@N^@ ^@P^@L^@A^@N^@T^@ ^@4^@2^@ ^@C^@M^@ ^@W^@/^@P^@O^@T^@"^@;^@"^@4^@"^@;^@"^@1^@2^@"^@;^@"^@4^@2^@"^@;^@"^@1^@0^@,^@5^@4^@"^@;^@"^@-^@9^@5^@,^@0^@0^@0^@0^@0^@"^@;^@"^@5^@/^@0^@9^@/^@2^@0^@2^@1^@"^@^M^@
^@"^@1^@0^@0^@4^@N^@"^@;^@"^@B^@a^@t^@o^@n^@ ^@P^@l^@a^@n^@t^@"^@;^@"^@2^@"^@;^@"^@2^@"^@;^@"^@9^@9^@"^@;^@"^@1^@2^@0^@,^@9^@5^@"^@;^@"^@5^@3^@,^@0^@0^@0^@0^@0^@"^@;^@"^@3^@0^@/^@0^@9^@/^@2^@0^@2^@1^@"^@^M^@
^@"^@1^@0^@0^@5^@N^@"^@;^@"^@N^@a^@t^@u^@r^@a^@l^@ ^@s^@t^@r^@e^@l^@i^@t^@z^@i^@a^@ ^@w^@/^@p^@o^@t^@"^@;^@"^@1^@"^@;^@"^@1^@"^@;^@"^@1^@3^@0^@"^@;^@"^@2^@0^@7^@,^@4^@4^@"^@;^@"^@1^@4^@,^@0^@0^@0^@0^@0^@"^@;^@"^@1^@/^@1^@2^@/^@2^@0^@2^@1^@"^@^M^@

what the heck is this??

When I change the filename to file.csv, downloading the file to my windoof desktop and open it with notepad++ again, it looks good:

"ArticleCode";"Description";"BB";"KT";"Size";"Price";"Stock";"DeliveryDate"
"1000L";"Aloe plant x 23 lvs";"4";"4";"65";"46,77";"118,00000";""
"1000M";"Aloe plant x 17 lvs";"4";"12";"50";"32,61";"206,00000";""
"1000S";"Aloe plant x 16 lvs";"4";"24";"40";"22,32";"-6,00000";"23/08/2021"
"1002M";"BATON PLANT 67 CM W/POT";"2";"6";"67";"22,42";"33,00000";"5/09/2021"
"1002S";"BATON PLANT 42 CM W/POT";"4";"12";"42";"10,54";"-95,00000";"5/09/2021"

Solution

for all those who suffer on the same problem:

I just hit in my terminal:

cat Inputfile | tr -d '\0' > Outputfile.csv

Answered By - y.y