[FIXED] Sign with openssl utility, verify with python script

Issue

I would like to verify signature for file that has been signed with the command-line utility, from within the python.

My command line.

openssl pkeyutl -sign -in data.sha256 -inkey device.key -out data-pss.sign -pkeyopt digest:sha256 -pkeyopt rsa_padding_mode:pss -pkeyopt rsa_pss_saltlen:32 -pkeyopt rsa_mgf1_md:sha256

My python script.

from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives import serialization 

# client cert -> get public key
with open('device.pem', 'rb') as fc:
    cert = x509.load_pem_x509_certificate(fc.read(), default_backend())
    public_key = cert.public_key()

# private key
with open('device.key','rb') as fk:
    private_key = serialization.load_pem_private_key(fk.read(), password=None, backend=default_backend())

# data sha256 that has been signed
with open('data.sha256', 'rb') as fd:
    data_sha256 = fd.read()

# data signature done by command line
with open('data-pss.sign', 'rb') as fds:
    data_sig = fds.read()

public_key.verify(data_sig,
                  data_sha256, 
                  padding.PSS(
                    mgf=padding.MGF1(hashes.SHA256()),
                    salt_length=32
                  ),
                  hashes.SHA256())

I get cryptography.exceptions.InvalidSignature.

If i do signing via python - no problem - the verification is successeful.

data_sig_py = private_key.sign(data_sha256,
                               padding.PSS(
                                 mgf=padding.MGF1(hashes.SHA256()),
                                 salt_length=32
                               ),
                               hashes.SHA256()
                              )

public_key.verify(data_sig_py,
                  data_sha256, 
                  padding.PSS(
                    mgf=padding.MGF1(hashes.SHA256()),
                    salt_length=32
                  ),
                  hashes.SHA256())

Solution

In the Python code, the already hashed data is passed in the second parameter of RSAPublicKey#verify(). In this case a Prehashed instance must be passed in the fourth parameter of RSAPublicKey#verify(), i.e. specifically utils.Prehashed(hashes.SHA256()), see also this example.

Currently a HashAlgorithm instance is passed as the fourth parameter, i.e. specifically hashes.SHA256(). This is also possible, but only if instead of the already hashed data the original data (i.e. not hashed data) is passed in the second parameter. RSAPublicKey#verify() hashes implicitly in this case.

I.e. in the current Python code the data is double hashed when verifying. In the OpenSSL statement, however, the only single hashed data is signed. For this reason, the verification with the Python code fails.
If the signature is created with the Python code (as in your last example), the data is also hashed twice, thus resulting in successful verification.

Answered By - Topaco