[FIXED] Python SSL certificate verify error

Issue

I'm using requests to access a RESTful API. Everything seems to work. I can authenticate, pull back a session token and even unit test the methods in my class I wrote for the API. Then I tried to run my code.

First, here is the call I'm making. The headers are static session-related items that get set in init(). The body is build dynamically from data in a file and passed in to this function. All of the data is valid.

response = requests.post(url, headers=(Requestheader), data=json.dumps((Requestbody)))

When I run the code, it updates well over 100 records with the metadata I supply. Somewhere around item 150 I get the following:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)

My first step was to call the vendor and find out if ALL of their web servers had properly signed certs figuring they were load balancing me and I found a misconfigured server. They tell me this is not the case.

Then I Googled the message and found that there is a verify kwarg, so I tried:

response = requests.post(url, headers=Requestheader, data=json.dumps(Requestbody), verify=False)

I know this isn't ideal long-term, but I wanted to test it to see if The behavior is the same. It did the same thing. It ran for a while and threw the ssl error. I thought the idea of verify=False is that it wouldn't check.

The vendor suggested I check the url I'm using, but it's fine. I would think if there were a proxy server or real man-in-the-middle attack causing problems I wouldn't see success so many times before a failure. I thought maybe it is a session timeout, but that should throw a 401 status and my activity level is too high for an inactivity timeout.

I'm a python noob and not a security professional. Suggestions appreciated.

Solution

So the issue might have three resolutions as I see it:

1. A certificate is OK and there is something wrong with the code. The issue may occur, for example, while using prepared requests as described in this solution

   But I don't really think it is your case because in the snippet you've provided no such methods are used. For the two next variants, you'll need to get the URL that causes an error and explore it's certificate (can be done via browser).

2. A certificate is OK but a certificate authority that signed it is not included in CA list that is utilized by requests library. After you'll open a troubling URL, check CA in it and see if it's dates are valid and it is included in this list. If not, add CA in the trusted list for the requests library -- as explained in the answers to this StackOverflow question.

3. A certificate is not valid or self-singed. Same solution as in 2.

The general solution is to wrap your script in the try except clause and to print out all the URLs that will result in mistakes. Then try to request them one by one via requests library and see if the issue occurs. If it does, it's (2) or (3) case. If not — try to run script on another machine with fresh installed python and requests. If the run will be successful — then there's some issue in your configuration.

Answered By - Andrii K