RSA_sign

Introductory note

There is some documentation out there for the OpenSSL RSA sign and verify APIs. What is sorely missing however, is some example code to clarify things. The goal of these howto sections is to expose some example code.

As a side note, I am fully aware that the EVP APIs exist and are recommended to perform the RSA signature creation and verification tasks. In any case, since the RSA_sign() and RSA_verify() APIs exist, let us illustrate how they should be used.

RSA_sign: howto

As a first step, let's consider a buffer buf of bytes of size buf_len to RSA-sign. In order to sign this data, we have, at our disposal, an RSA private key, in PEM format, in its own pkey array of bytes, of size pkey_len. Of course, we also have as much memory as needed on hand, potentially allocatable through standard malloc() calls, and all of the relevant OpenSSL APIs.

The first step is to hash the data to sign (since, as is well-known), the signature is the hash of the data, adequately encoded and padded, then encrypted with the RSA private key.

#include <stdlib.h>
#include <openssl/sha.h>

int sign_data(
        const void *buf,    /* input data: byte array */
        size_t buf_len, 
        void *pkey,         /* input private key: byte array of the PEM representation */
        size_t pkey_len,
        void **out_sig,     /* output signature block, allocated in the function */
        size_t *out_sig_len) {
    int status = EXIT_SUCCESS;
    int rc = 1;

    SHA_CTX sha_ctx = { 0 };
    unsigned char digest[SHA_DIGEST_LENGTH];

    rc = SHA1_Init(&sha_ctx);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

    rc = SHA1_Update(&sha_ctx, buf, buf_len);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

    rc = SHA1_Final(digest, &sha_ctx);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

The next step is to extract the RSA * form of the private key as is expected by the RSA_sign() function from the PEM byte array we are taking as an input. For that, let us use the usual BIO_ and PEM_ functions:

#include <openssl/bio.h>
#include <openssl/rsa.h>
#include <openssl/pem.h>

...

    BIO *b = NULL;
    RSA *r = NULL;

...

    b = BIO_new_mem_buf(pkey, pkey_len);
    r = PEM_read_bio_RSAPrivateKey(b, NULL, NULL, NULL);

We now have all the elements we need to call into RSA_sign(): the digest digest and the private key in the adequate form r. All that's left to do is to find some room for the signature (of size RSA_size()) and call the RSA_sign() function and check that it was successful.

    unsigned char *sig = NULL;
    unsigned int sig_len = 0;

...

    sig = malloc(RSA_size(r));
    if (NULL == sig) { handle_it(); status = EXIT_FAILURE; goto end; }

    rc = RSA_sign(NID_sha1, digest, sizeof digest, sig, &sig_len, r);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

Of course, the function should handle error cases adequately. My preference goes towards doing the "test-for-error, handle-it, goto-end" approach, which avoids nested levels of if/elses. It can be looked at as asserting against errors as you go. Let's just conclude the function with the error case handling, and resource freeing:

#include <openssl/err.h>

...

    *out_sig = sig;
    *out_sig_len = sig_len;
end:
    if (NULL != r) RSA_free(r);
    if (NULL != b) BIO_free(b);
    if (EXIT_SUCCESS != status) free(sig); /* in case of failure: free allocated resource */
    if (1 != rc) fprintf(stderr, "OpenSSL error: %s\n", ERR_error_string(ERR_get_error(), NULL));

    return status;
}

RSA_verify

Now that we have signed our content, we want to verify its signature. The method for this action is (of course) RSA_verify(). The inputs to the action are the content itself as a buffer buf of bytes or size buf_len, the signature block sig of size sig_len as generated by RSA_sign(), and the X509 certificate corresponding to the private key used for the signature. We will use the DER representation of the cert, in its own buffer cert of bytes of size cert_len.

Therefore, our signature verification function will look something like this:

int verify_data(
        const void *buf,    /* input data: byte array */
        size_t buf_len,
        const void *sig,    /* signature block: byte array */
        size_t sig_len,
        const void *cert,   /* input cert: byte array of the DER representation */
        size_t cert_len) {

As for the signature case, the first step is to hash the data:

    SHA_CTX sha_ctx = { 0 };
    unsigned char digest[SHA_DIGEST_LENGTH];

    rc = SHA1_Init(&sha_ctx);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

    rc = SHA1_Update(&sha_ctx, buf, buf_len);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

    rc = SHA1_Final(digest, &sha_ctx);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

The next step is to extract the RSA * form of the public key from the X509 certificate, as expected by the RSA_verify() function. This is a little less immediate as for getting the RSA private key from its PEM representation:

#include <openssl/bio.h>
#include <openssl/x509.h>
#include <openssl/evp_pkey.h>

...

    BIO *b = NULL;
    X509 *c;
    EVP_PKEY *k = NULL;

...

    b = BIO_new_mem_buf(cert, cert_len);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

    c = d2i_X509_bio(b, NULL);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

    k = X509_get_pubkey(c);
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

    /* make sure that the public key from the cert is an RSA key */
    if (EVP_PKEY_RSA != EVP_PKEY_type(k->type)) { handle_it(); status = EXIT_FAILURE; goto end; }

We have now gathered all the elements needed for the verification of the signature: the data digest digest, the signature block sig and the RSA public key corresponding to the private key used to sign the data EVP_PKEY_get1_RSA(k). All that's left to do is to perform the signature verification with RSA_verify():

    rc = RSA_verify(NID_sha1, digest, sizeof digest, sig, sig_len, EVP_PKEY_get1_RSA(k));
    if (1 != rc) { handle_it(); status = EXIT_FAILURE; goto end; }

To finish, let's tie up the loose ends and handle the error cases:

end:
    if (NULL != k) EVP_PKEY_free(k);
    if (NULL != b) BIO_free(b);

    if (1 != rc) fprintf(stderr, "OpenSSL error: %s\n", ERR_error_string(ERR_get_error()));

    return status;
}

Concluding note

Hopefully, the examples above will clarify one (of many) approach to performing RSA signature creation and verification with the OpenSSL crypto APIs. Your feedback is most welcome.