[FIXED] I can't access scrapyd port 6800 from browser

Issue

I searched a lot on this, it may have a simple solution that I am missing.

I have setup scrapy + scrapyd on both my local machine and my server. They work both ok when I try as "scrapyd".

I can deploy to local without a problem, and I can access to localhost:6800 as well from the browser and I can run spiders on local.

After running scrapyd on remote, I try to deploy to http://remoteip:6800/ with the same as I did deploy locally,

I get,

Packing version 1500333306
Deploying to project "projectX" in http://remoteip:6800/addversion.json
Deploy failed: <urlopen error [Errno 111] Connection refused>

I also can't access http://remoteip:6800/ from my local PC, but I can access from ssh on remote PC (with curl)

I opened inbound and outbound connections on the remote server, what else I am missing?

Thanks

Solution

First check if its running or not, run curl localhost:6800 on the server where ScrapyD is running

Check if firewall is enabled

sudo ufw status

Ideally, just allow tcp connections to 6800instead of disabling firewall, to do so

sudo ufw allow 6800/tcp
sudo ufw reload

Check your scrapyd.conf please set

bind_address=0.0.0.0

instead of

bind_address=127.x.x.x

0.0.0.0 will make scrapyD accessible for incoming connections outside the server/instance, not only localhost.

Then stop scrapyD, I do killall scrapyd to stop scrapyd

Then restart scrapyD using command scrapyd

---

Note: If you want to keep scrapyd running even after you disconnect from server, do this

nohup scrapyd >& /dev/null &

---

Also see my answer to set ScrapyD as a System Service

Answered By - Umair Ayub