[FIXED] Handshake Failure: SSL Alert number 40

Issue

I'm trying to crawl a page without success:

>> scrapy shell "XXXXXX"
...
2018-12-28 17:23:32 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET XXXXXXXX> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]
2018-12-28 17:23:32 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET XXXXXXXXXXXXX> (failed 2 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]
2018-12-28 17:23:33 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET XXXXXXXXXXXXXXXXX> (failed 3 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]
Traceback (most recent call last):
  File "/home/joaquin/Repos/extruct/env/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/joaquin/Repos/extruct/env/lib/python3.7/site-packages/scrapy/cmdline.py", line 150, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/home/joaquin/Repos/extruct/env/lib/python3.7/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
    func(*a, **kw)
  File "/home/joaquin/Repos/extruct/env/lib/python3.7/site-packages/scrapy/cmdline.py", line 157, in _run_command
    cmd.run(args, opts)
  File "/home/joaquin/Repos/extruct/env/lib/python3.7/site-packages/scrapy/commands/shell.py", line 73, in run
    shell.start(url=url, redirect=not opts.no_redirect)
  File "/home/joaquin/Repos/extruct/env/lib/python3.7/site-packages/scrapy/shell.py", line 48, in start
    self.fetch(url, spider, redirect=redirect)
  File "/home/joaquin/Repos/extruct/env/lib/python3.7/site-packages/scrapy/shell.py", line 115, in fetch
    reactor, self._schedule, request, spider)
  File "/home/joaquin/Repos/extruct/env/lib/python3.7/site-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "/home/joaquin/Repos/extruct/env/lib/python3.7/site-packages/twisted/python/failure.py", line 467, in raiseException
    raise self.value.with_traceback(self.tb)
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]

when try a SSL connection I got:

>> openssl s_client XXXXX.XXXX.XXXX:443
CONNECTED(00000003)
140087350686208:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1528:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 323 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
--- 

Also when I try this page with curl happens the same:

curl -i XXXX.XXXX.XXXX
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

I try to specify -servername in openssl but it doesn't fix the problem. Also trying to specify -tls1_2 doesn't work. TLS Info:

UPDATE

>> openssl version
OpenSSL 1.1.1a  20 Nov 2018

Solution

Found your question while searching for the exact same problem (curl succeeds to connect while openssl fails with alert number 40).

It might be related to a server with several virtual hosts to serve, and you need to tell which host you want to connect to, to let the TLS handshake succeed.

Specify the exact host name you want with -servername parameter. E.g:

openssl s_client -connect yourserver.domain.com:443 -servername yourserver.domain.com   

Answered By - Seki