PEM Pack

From Crypto++ Wiki
Jump to: navigation, search

The PEM Pack is a partial implementation of message encryption which allows you to read and write PEM encoded keys and parameters, including encrypted private keys. The additional files include support for RSA, DSA, EC, ECDSA keys and Diffie-Hellman parameters. The ZIP contains five additional source files, a script to create test keys using OpenSSL, a C++ program to test reading and writing the keys, and a script to verify the keys written by Crypto++ using OpenSSL.

PEM encrypted private keys use OpenSSL's key derivation algorithm EVP_BytesToKey. The function is PKCS#5 v1.5 compatible for derived material up to and including 16 bytes. If the required key is over 16 bytes (for example, AES-256 needs 32 bytes), then a non-standard extension is engaged. Regardless of the key size, Crypto++ and OpenSSL will interop as expected because both libraries implement the same derivation algorithm.

PEM encryption is an old format specified in Privacy Enhancement for Internet Electronic Mail: Part I: Message Encryption and Authentication Procedures. If given a choice, you should prefer a newer standard like PKCS #8.

There is no standard list of PEM objects or names, though a request was made for one on the IETF's PKIX mailing list. The PEM Pack tries to provide support for most of those provided by OpenSSL. You can get OpenSSL's list from the project's pem.h source file (located at <openssl src>/crypto/pem/pem.h).

Finally, the collection of files in the ZIP archive are licensed under the same terms as the Crypto++ library. If you have accepted Crypto++'s licensing terms, then the PEM Pack incurs no additional burden or penalty.

Compiling

There are five C++ source files in the ZIP to add to Crypto++. The files are:

  • pem.h - the include file for the PEM routines used by applications. Everything in the file is exported with CRYPTOPP_DLL
  • pem-com.h - the internal include file for common PEM routines not exposed to the application
  • pem-com.cpp - the source file for the common PEM routines
  • pem-rd.cpp - the source file for the internal PEM load and read routines
  • pem-wr.cpp - the source file for the internal PEM save and write routines

To compile the files, simply drop them in your cryptopp folder and then execute Crypto++'s GNUmakefile. The library's makefile will automatically pick them up.

The source files clean compile with -Wall and -Wextra, and have passed Clang's address and undefined sanitizers.

The default behavior does not validate keys and parameters after reading them. If you want the keys and parameters validated after reading them, then open pem-com.h and comment the define PEM_NO_KEY_OR_PARAMETER_VALIDATION. Once compiled, changing PEM_NO_KEY_OR_PARAMETER_VALIDATION has no effect.

Note: if you want to compile pem-test.cpp, then rename pem-test.cpp.source to pem-test.cpp. However, if you rename the file and try to build the library, then some of the Makefile targets break because duplicate main are specified. So if you don't need to build pem-test.cpp, then don't perform the rename.

Public API

The public API primarily consists of PEM_Load and PEM_Save routines. The read and write routines are overloaded to accept a BufferedTransformation and a public or private key type. The supported systems are RSA, DSA, EC (both ECP and EC2N) and Diffie-Hellman.

There are two helper routines to extract a PEM object, PEM_NextObject, and identify the PEM object, PEM_GetType.

PEM_Load

Each system has a three read functions. For example, the routines to read RSA keys:

void PEM_Load(BufferedTransformation& bt, RSA::PublicKey& rsa);
void PEM_Load(BufferedTransformation& bt, RSA::PrivateKey& rsa);
void PEM_Load(BufferedTransformation& bt, RSA::PrivateKey& rsa, const char* password, size_t length);

When reading an encrypted key, you must specify the password and its length in case there's an embedded NULL character in the string. The encapsulated header includes the encryption algorithm, so there's no need to specify it.

In general, reading is deferred to Crypto++ and its various BERDecode* routines. On occasion, the PEM Pack needs to provide a modified routine. For example, for DSA keys, Crypto++ expects {version, x} (where x is the private key) while OpenSSL provides {version, p,q,g,y,x} (where y is the public element):

void PEM_LoadPrivateKey(BufferedTransformation& bt, DSA::PrivateKey& key)
{
    BERSequenceDecoder seq(bt);
    
      word32 v;                // check version
      BERDecodeUnsigned<word32>(seq, v, INTEGER, 0, 0);
    
      Integer p,q,g,y,x;
    
      p.BERDecode(seq);
      q.BERDecode(seq);
      g.BERDecode(seq);
      y.BERDecode(seq);
      x.BERDecode(seq);
    
    seq.MessageEnd();
    
    key.Initialize(p, q, g, x);
}

When there's a problem, the PEM pack will throw a Crypto++ exception; and not a custom exception. Be prepared to catch Exception, InvalidArgument and InvalidDataFormat exceptions (in addition to anything Crypto++ might throw, like a DER decode error or bad padding exception).

PEM_Save

Each PEM_Load routine has a corresponding write routine, so there are three write functions for each system. For example, in the case of RSA:

void PEM_Save(BufferedTransformation& bt, const RSA::PublicKey& rsa);
void PEM_Save(BufferedTransformation& bt, const RSA::PrivateKey& rsa);
void PEM_Save(BufferedTransformation& bt, const RSA::PrivateKey& rsa, const string& algorithm, const char* password, size_t length);

When writing an encrypted key, you must specify the password, the password's length and the algorithm. The recognized algorithms are listed below.

  • AES-256-CBC
  • AES-192-CBC
  • AES-128-CBC
  • CAMELLIA-256-CBC
  • CAMELLIA-192-CBC
  • CAMELLIA-128-CBC
  • DES-EDE3-CBC
  • IDEA-CBC
  • DES-CBC

As with the read routines, the write routines defer to DEREncode* routines but sometimes need to provide an OpenSSL compatible key. For example OpenSSL expects {version, p,q,g,y,x} for DSA, so an override is provided:

void PEM_DEREncode(BufferedTransformation& bt, const DSA::PrivateKey& key)
{
    const DL_GroupParameters_DSA& params = key.GetGroupParameters();
    
    DSA::PublicKey pkey;
    key.MakePublicKey(pkey);
    
    DERSequenceEncoder seq(bt);

      DEREncodeUnsigned<word32>(seq, 0);
      params.GetModulus().DEREncode(seq);
      params.GetSubgroupOrder().DEREncode(seq);
      params.GetGenerator().DEREncode(seq);

      pkey.GetPublicElement().DEREncode(seq);
      key.GetPrivateExponent().DEREncode(seq);
    
    seq.MessageEnd();
}

If you need a new algorithm, then modify PEM_CipherForAlgorithm in both pem-rd.cpp and pem-wr.cpp. Be sure to test the new algorithm against OpenSSL since OpenSSL only provides the algorithms listed above in its various commands (like openssl genrsa). However, OpenSSL should recognize anything EVP_get_cipherbyname understands.

When there's a problem, the PEM pack will throw a Crypto++ exception; and not a custom exception. Be prepared to catch Exception, InvalidArgument and InvalidDataFormat exceptions (in addition to anything Crypto++ might throw).

PEM_NextObject

There's also a function that allows you to read the first key or parameter called PEM_NextObject. The function locates the first PEM object in src and places it in dest. PEM_NextObject essentially peeks into the stream, so the source buffer is unchanged if there is no PEM object present.

void PEM_NextObject(BufferedTransformation& src, BufferedTransformation& dest, bool trimTrailing=true);

PEM_NextObject will silently discard any characters that proceed the PEM Object. The destination BufferedTransformation will have one line ending if it was present in source.

If trimTrailing is true, then trailing whitespace is trimmed from the source BufferedTransformation. This is a convenience function, and its intended to help "pretty print" the source buffer by removing leading whitespace after a key or parameter is extracted (trailing whitespace for the first key becomes leading whitespace for the second key).

Internally, the various PEM_Load functions call PEM_NextObject. That means you can call PEM_NextObject and then PEM_Load; or you can simply call PEM_Load alone (and PEM_Load will call PEM_NextObject).

PEM_NextObject will parse an invalid object. For example, it will parse a key or parameter with -----BEGIN FOO----- and -----END BAR-----. The parser only looks for BEGIN and END (and the dashes). The malformed input will be caught later when a particular key or parameter is loaded.

On failure, InvalidDataFormat is thrown.

PEM_GetType

The final function attempts to classify a PEM object. The function is PEM_GetType, and its also called internally after PEM_NextObject.

PEM_Type PEM_GetType(const BufferedTransformation& bt);

The function returns one of the following enums:

  • PEM_PUBLIC_KEY
  • PEM_PRIVATE_KEY
  • PEM_RSA_PUBLIC_KEY
  • PEM_RSA_PRIVATE_KEY
  • PEM_RSA_ENC_PRIVATE_KEY
  • PEM_DSA_PUBLIC_KEY
  • PEM_DSA_PRIVATE_KEY
  • PEM_DSA_ENC_PRIVATE_KEY
  • PEM_EC_PUBLIC_KEY
  • PEM_ECDSA_PUBLIC_KEY
  • PEM_EC_PRIVATE_KEY
  • PEM_EC_ENC_PRIVATE_KEY
  • PEM_EC_PARAMETERS
  • PEM_DH_PARAMETERS
  • PEM_DSA_PARAMETERS
  • PEM_REQ_CERTIFICATE
  • PEM_X509_CERTIFICATE
  • PEM_CERTIFICATE
  • PEM_UNSUPPORTED

PEM_GetType peeks at the underlying stream. It does not consume the stream in the BufferedTransformation.

The function only looks for the header (i.e., pre-encapsulated boundary), and does not probe for the footer (i.e., the post-encapsulated boundary). Its possible that PEM_GetType will return a type but later the PEM object will be rejected.

Though PEM_REQ_CERTIFICATE, PEM_CERTIFICATE and PEM_X509_CERTIFICATE are recognized, there's no corresponding PEM_Load or PEM_Save routine.

If an unknown type or bogus PEM object is presented, like -----BEGIN FOO----- and -----END BAR-----, then PEM_UNSUPPORTED is returned.

Sample Code

Using the PEM pack is straight forward. Include pem.h, and then use either PEM_Load or PEM_Save. For example:

#include <cryptopp/pem.h>
...

// Load a RSA public key
FileSource fs1("rsa-pub.pem", true);
RSA::PublicKey k1;
PEM_Load(fs1, k1);

// Load a encrypted RSA private key
FileSource fs2("rsa-enc-priv.pem", true);
RSA::PrivateKey k2;
PEM_Load(fs2, k2, "test", 4);

// Save an EC public key
DL_PublicKey_EC<ECP> k16 = ...;
FileSink fs16("ec-pub-xxx.pem", true);
PEM_Save(fs16, k16);

// Save an encrypted EC private key
DL_PrivateKey_EC<ECP> k18 = ...;
FileSink fs18("ec-enc-priv-xxx.pem", true);
PEM_Save(fs18, k18, "AES-128-CBC", "test", 4);

Depending on PEM_NO_KEY_OR_PARAMETER_VALIDATION, you may need to validate the key. If you have to validate keys, then the code would looks similar to:

FileSource fs("rsa-pub.pem", true);
RSA::PublicKey key;
PEM_Load(fs, key);

AutoSeededRandomPool prng;
bool result = key.Validate(prng, 2);
if(!result)
    throw std::exception("Failed to validate public key");

Input and Output

Below is an example of the OpenSSL and Crypto++ keys. Keys written by OpenSSL lack "xxx", while keys written by Crypto++ include "xxx" in the filename.

OpenSSL:

$ cat rsa-pub.pem 
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCl/OKZWiBSG+kyJLdMWbK81VEt
7kMBM2FZyAH1siPEyqkjfCV3zWj7zAQtzJUlIP5KrPhezb9agYo+Xwuj3ODnS20h
FxQzPuPwzdfCUoy9khs2NY7vu8KECIVNwUi4tJCaom9otKHnRbn5ZfMicLFV/bHr
mGQqXNeMYr2i1kPGVwIDAQAB
-----END PUBLIC KEY-----

$ cat rsa-enc-priv.pem 
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,CDCA2C5DC5084410C33F2FD6439F910A

AOX63PM/YJqcCfLi3NCkInuUBgV07MK8vyKJJSvBig1iFZy0VWDZPkgYUae7iZPX
O8tBLgdunN+k/S8K/bksG8m0/iU3v0B3iTsTLmLQQg0pJYjXQyY16faouJVaovqi
hp/Zohri49giJ/49Lee6dFoZomebY6fXiI33KVqv1o91i++y/Qp7d4Djwp5aKHDd
gseB5lU/kMBAVt5Q1CllBMPD0vCbvCFCyHpLrC5RwFxFEs5Cdhhny95na1i12Khm
2kPtF6hVhGBKFzCiFexOMgLfw6VDXQJqb6MRnnxbc1igCZ2ZzBlnstXK0esd4HfY
NCBt9Z1jUUoYdQ8QXPN/cN8Xp8XMfahvmoihJNk3xWmnQFVfl+azGSaD7FG7xSmS
y9KbqQCXAJCO51nXW2m2L9u8Brf4sf1Tltc+S4gJWsGFzzNpIOZ7um7dst4ArgJT
gMY6H7nqxgCNYaZNLTT1t4ALyfCWH6eI9q7clygQMsG07ncVKkIO9dRV/ElozCwR
iV/cOYdmlt4dVXUc5uSmHHvqTRUT2T5HVw5m6Gh9uJaEt9CDdeXQg/JrUeHvQ5HT
7cs3hr6cxhSlifECUKYmunXd3bbmbWTI7yb2U0uJkoDUsCkvgJJisFjdYSwAI0yZ
zKisZrftthH8OouXjFB+VVRwpAhAMqLbe1l8EP25nGJ7e+TtVLo4fHIpxr2WdF/e
NfjqnLeReHw6W4UsObqvCu1kGZygeMRSN+6mkb3dm+u2buI2SVRvIFiHkWYD3g/l
3Lk6vfaxnTYCXkSXDax0j+ckJTTAK2i9lkcH2wqMv05FoHVi7aVnR8IIV00TnC7t
-----END RSA PRIVATE KEY-----

And Crypto++:

$ cat ec-pub-xxx.pem 
-----BEGIN PUBLIC KEY-----
MFYwEAYHKoZIzj0CAQYFK4EEAAoDQgAExcZuVIOO7UZSCaZYgP+faEVrvMWeBYtD
ul2u5lrpxoCEMxXlICGCwKaB1WgkJUK5sjGk45eCMv8B/78lhjVVVw==
-----END PUBLIC KEY-----

$ cat ec-enc-priv-xxx.pem 
-----BEGIN EC PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,483C6901DE64A75A3CF218BE0A8F3AF6

u0IGXknqwne3WOz/nkgnXcfhMdRqXxjqhOmk3X1lqLwDnBBZXAytidP7b+8Teufq
xtFs9jrPoeKwWiyQ8jF6v14dQ3wDHI5LHSAJWQvcPtHZ9h5UtoPNuJktEi+eWRvt
99DCTYbC7RAx2MQnaFAFwsvUJPoJvtS8tykxXQSHqj0=
-----END EC PRIVATE KEY-----

Testing Keys

You should test the code before you use it. To do so, two scripts and a cpp source file are provided. The scripts depend on OpenSSL and PERL to create a set of PEM encoded test keys, and to verify the keys written by Crypto++ are OK. PERL is used to chop and chomp a good key into one that should cause an exception.

To begin, run pem-create-keys.sh to create the RSA, DSA, EC, ECDSA keys and Diffie-Hellman parameters. For each cryptosystem, a public key, a private key and an encrypted private key are created. The test keys are named <system>-<type>.pem. For example, for RSA, the script will create rsa-pub.pem, rsa-priv.pem and rsa-enc-priv.pem. The encrypted keys use a password of "test" (without the quotes).

Next, compile and run pem-test.cpp to exercise reading and writing the keys. For each test key created, the test program will read the key and then write it back out. When the file is written back out, its written out with an "xxx" in its name. For example, rsa-pub-xxx.pem, rsa-priv-xxx.pem and rsa-enc-priv-xxx.pem.

Finally, after the "xxx" files are written by the test program, you should run pem-verify-keys.sh to ensure OpenSSL can read them.

Note: you may have to rename pem-test.cpp.source to pem-test.cpp. The rename ensures the Crypto++ library will build correctly if all files in the ZIP are unpacked (instead of just the PEM Pack source files).

To summarize the steps:

  1. Run pem-create-keys.sh
  2. Run pem-test.exe
  3. Run pem-verify-keys.sh

Expected Output

The expected output of pem-test.exe is:

$ ./pem-test.exe
Running 0
Running 1
Running 2
Running 3
Running 4
...
Running 26
Running 27
Running 28
Running 29
Running 30
Running 31
Parsed 153 certificates from cacert.pem
All tests passed

The expected output of pem-verify-keys.sh is:

$ ./pem-verify-keys.sh 
read RSA key
read RSA key
read RSA key
read DSA param
read DSA key
read DSA key
read DSA key
read EC param
read EC key
read EC key
read EC key
Finished testing keys written by Crypto++

Downloads

pem-pack.zip - Additional source files which allow you to read and write PEM encoded keys, including encrypted private keys. The ZIP file includes a script to build test keys with OpenSSL, a small C++ test program to test reading and writing the keys, and a script to verify the keys written by Crypto++ using OpenSSL