Access Keys:
Skip to content (Access Key - 0)

Security Considerations and Encryption with Kettle

Abstract: Kettle is used more and more in enterprises where the standard obfuscation of credentials is not sufficient enough. There are requirements to use strong encryption methods and even to store internal data encrypted (covered in PDI-6168 and PDI-6170). The above use cases inspired me to create some simple transformations to test and play around with encryption.

Note: Kettle already has job entries to encrypt and decrypt files with PGP.

Authors: Jens Bleuel

License: LGPL

Kettle versions: 4.1 and later

Update since Kettle 4.2: There are two steps in the experimental section: Secret key generator, Symmetric Cryptography that cover this use case.

Attachments

Attention: When you want to use the given examples in production please be aware that they are made as a blue print and a proof of concept and need optimization.

Background

Transformation cryptographyCreateSecretKey is used to create a secret key.
Use the transformations cryptographyEncrypt and cryptographyDecrypt for testing.

The named parameters and default values are:

  • cryptographyKeySize = 128
  • cryptographyMethod = AES

The transformations were tested with AES and 128 bit: Advanced Encryption Standard (AES) is a symmetric-key encryption standard, see also http://java.sun.com/developer/technicalArticles/Security/AES/AES_v1.html

To use AES with 192- and 256-bit key sizes, you need unlimited strength cryptography.
Due to import-control restrictions imposed by some countries, the jurisdiction policy files shipped
with the Java 2 SDK, v 1.4 only permit strong cryptography to be used. An unlimited strength version of these files
(that is, with no restrictions on cryptographic strength) is available for download, however.

Instead of storing the decrypted data to a file there are a lof of other options, e.g.:

  • use the decrypted data as credentials in subsequent steps or transformations
  • put the decrypted data into variables visible in a limited scope (e.g. parent job) and use them as credentials for databases, repository etc. (see PDI-6168)
  • and many more options

We may consider:

  • Symmetric-key algorithm vs. asymmetric key algorithms (public-key cryptography)
  • Diffie-Hellman key exchange is a specific method of exchanging keys.
  • Ensure integrity e.g. by hash-codes
  • Key file handling could be optimized in different ways.
  • Please keep in mind that unencrypted data is in RAM (see PDI-6170 for a circumvention to prevent heap dumps)
  • Beneath the binary or indexed storage type, an encrypted storage type may be possible in Kettle core.

For screen shots, some further background information and a test run, please see http://kettle.bleuel.com/2011/06/07/security-considerations-and-encryption-with-kettle/


  1. Dec 11, 2012

    John Hann says:

    I used this sample to encrypt a field that is stored in MySQL, but when I tried ...

    I used this sample to encrypt a field that is stored in MySQL, but when I tried to use MySQL's AES_DECRYPT() to get the data back out, it just returned NULLs. It turns out that MySQL uses a 16 byte key for it's AES encryption, regardless of what you pass in. If there are more than 16 bytes in your string, it will just OR in those bytes to the existing key bytes. Here's the blog post that explains this process and helped me resolve the issue:http://info.michael-simons.eu/2011/07/18/mysql-compatible-aes-encryption-decryption-in-java/

    Hope this helps!

  2. Aug 06, 2013

    Eric Adolph says:

    Hello, when I have more then 1 file in the directory that fits the regular expre...

    Hello, when I have more then 1 file in the directory that fits the regular expression to decrypt, it combined them both into one output file, and also leaves a bunch of junk lines at the bottom. Is there a way to decrypt each file into it's own output file? Thanks!

    -Eric

This documentation is maintained by the Pentaho community, and members are encouraged to create new pages in the appropriate spaces, or edit existing pages that need to be corrected or updated.

Please do not leave comments on Wiki pages asking for help. They will be deleted. Use the forums instead.

Adaptavist Theme Builder (4.2.0) Powered by Atlassian Confluence 3.3.3, the Enterprise Wiki