Encryption of each knowledge at relaxation and in transit is a non-negotiable function for many organizations. Moreover, organizations working in extremely regulated and security-sensitive environments—resembling these within the monetary sector—typically require full management over the cryptographic keys used for his or her workloads.
Amazon Managed Service for Apache Flink makes it simple to course of real-time knowledge streams with sturdy safety features, together with encryption by default to assist shield your knowledge in transit and at relaxation. The service removes the complexity of managing the important thing lifecycle and controlling entry to the cryptographic materials.
If you should retain full management over your key lifecycle and entry, Managed Service for Apache Flink now helps using buyer managed keys (CMKs) saved in AWS Key Administration Service (AWS KMS) for encrypting software knowledge.
This function helps you handle your individual encryption keys and key insurance policies, so you may meet strict compliance necessities and preserve full management over delicate knowledge. With CMK integration, you may benefit from the scalability and ease of use that Managed Service for Apache Flink affords, whereas assembly your group’s safety and compliance insurance policies.
On this submit, we discover how the CMK performance works with Managed Service for Apache Flink purposes, the use instances it unlocks, and key issues for implementation.
Information encryption in Managed Service for Apache Flink
In Managed Service for Apache Flink, there are a number of features the place knowledge ought to be encrypted:
- Information at relaxation immediately managed by the service – Sturdy software storage (checkpoints and snapshots) and working software state storage (disk volumes utilized by RocksDB state backend) are routinely encrypted
- Information in transit inner to the Flink cluster – Robotically encrypted utilizing TLS/HTTPS
- Information in transit to and at relaxation in exterior techniques that your Flink software accesses – For instance, an Amazon Managed Streaming for Apache Kafka (Amazon MSK) matter by the Kafka connector or calling an endpoint by a customized AsyncIO); encryption relies on the exterior service, consumer settings, and code
For knowledge at relaxation managed by the service, checkpoints, snapshots, and working software state storage are encrypted by default utilizing AWS owned keys. In case your safety necessities require you to immediately management the encryption keys, you need to use the CMK held in AWS KMS.
Key parts and roles
To know how CMKs work in Managed Service for Apache Flink, we first must introduce the parts and roles concerned in managing and working an software utilizing CMK encryption:
- Buyer managed key (CMK):
- Resides in AWS KMS throughout the identical AWS account as your software
- Has an hooked up key coverage that defines entry permissions and utilization rights to different parts and roles
- Encrypts each sturdy software storage (checkpoints and snapshots) and working software state storage
- Managed Service for Apache Flink software:
- The applying whose storage you wish to encrypt utilizing the CMK
- Has an hooked up AWS Id and Entry Administration (IAM) execution position that grants permissions to entry exterior providers
- The execution position doesn’t have to supply any particular permissions to make use of the CMK for encryption operations
- Key administrator:
- Manages the CMK lifecycle (creation, rotation, coverage updates, and so forth)
- Will be an IAM consumer or IAM position, and utilized by a human operator or by automation
- Requires administrative entry to the CMK
- Permissions are outlined by the hooked up IAM insurance policies and the important thing coverage
- Utility operator:
- Manages the appliance lifecycle (begin/cease, configuration updates, snapshot administration, and so forth)
- Will be an IAM Consumer or IAM position, and utilized by a human operator or by automation
- Requires permissions to handle the Flink software and use the CMK for encryption operations
- Permissions are outlined by the hooked up IAM insurance policies and the important thing coverage
The next diagram illustrates the answer structure.
Enabling CMK following the precept of least privilege
When deploying purposes in manufacturing environments or dealing with delicate knowledge, you need to observe the precept of least privilege. CMK assist in Managed Service for Apache Flink has been designed with this precept in thoughts, so every part receives solely the minimal permissions essential to operate.
For detailed details about the permissions required by the appliance operator and key coverage configurations, check with Key administration in Amazon Managed Service for Apache Flink. Though these insurance policies may seem advanced at first look, this complexity is intentional and vital. For extra particulars in regards to the necessities for implementing probably the most restrictive key administration potential whereas sustaining performance, check with Least-privilege permissions.
For this submit, we spotlight some vital factors about CMK permissions:
- Utility execution position – Requires no further permissions to make use of a CMK. You don’t want to alter the permissions of an present software; the service handles CMK operations transparently throughout runtime.
- Utility operator permissions – The operator is the consumer or position who controls the appliance lifecycle. For the permissions required to function an software that makes use of CMK encryption, check with Key administration in Amazon Managed Service for Apache Flink. Along with these permissions, an operator usually has permissions on actions with the
kinesisanalyticsprefix. It’s a greatest follow to limit these permissions to a selected software defining theUseful resource. The operator should even have theiam:PassRolepermission to go the service execution position to the appliance.
To simplify managing the permissions of the operator, we advocate creating two separate IAM insurance policies, to be hooked up to the operator’s position or consumer:
- A base operator coverage defining the fundamental permissions to function the appliance lifecycle with out a CMK
- A further CMK operator coverage that provides permissions to function the appliance with a CMK
The next IAM coverage instance illustrates the permissions that ought to be included within the base operator coverage:
Seek advice from Utility lifecycle operator (API caller) permissions for the permissions to be included with the extra CMK operator coverage.
Separating these two insurance policies has a further advantage of simplifying the method of organising an software for the CMK, because of the dependencies we illustrate within the following part.
Dependencies between the important thing coverage and CMK operator coverage
When you rigorously observe the operator’s permissions and the important thing coverage defined in Create a KMS key coverage, you’ll discover some interdependencies, illustrated by the next diagram.
Specifically, we spotlight the next:
- CMK key coverage dependencies – The CMK coverage requires references to each the appliance Amazon Useful resource Title (ARN) and the important thing administrator or operator IAM roles or customers. This coverage should be outlined at key creation time by the important thing administrator.
- IAM coverage dependencies – The operator’s IAM coverage should reference each the appliance ARN and the CMK key itself. The operator position is liable for varied duties, together with configuring the appliance to make use of the CMK.
To correctly observe the precept of least privilege, every part requires the others to exist earlier than it may be appropriately configured. This necessitates a rigorously orchestrated deployment sequence.
Within the following part, we show the exact order required to resolve these dependencies whereas sustaining safety greatest practices.
Sequence of operations to create a brand new software with a CMK
When deploying a brand new software that makes use of CMK encryption, we advocate following this sequenced strategy to resolve dependency conflicts whereas sustaining safety greatest practices:
- Create the operator IAM position or consumer with a base coverage that features software lifecycle permissions. Don’t embody CMK permissions at this stage, as a result of the important thing doesn’t exist but.
- The operator creates the appliance utilizing the default AWS owned key. Preserve the appliance in a stopped state to forestall knowledge creation—there ought to be no knowledge at relaxation to encrypt throughout this section.
- Create the important thing administrator IAM position or consumer, if not already out there, with permissions to create and handle KMS keys. Seek advice from Utilizing IAM insurance policies with AWS KMS for detailed permission necessities.
- The important thing administrator creates the CMK in AWS KMS. At this level, you’ve gotten the required parts for the important thing coverage: software ARN, operator IAM position or consumer ARN, and key administrator IAM position or consumer ARN.
- Create and fix to the operator a further IAM coverage that features the CMK-specific permissions. See Utility lifecycle operator (API caller) permissions for the whole operator coverage definition.
- The operator can now modify the appliance configuration utilizing the
UpdateApplicationmotion, to allow CMK encryption, as illustrated within the following part. - The applying is now able to run with all knowledge at relaxation encrypted utilizing your CMK.
Allow the CMK with UpdateApplication
You’ll be able to configure a Managed Service for Apache Flink software to make use of a CMK utilizing the AWS Administration Console, the AWS API, AWS Command Line Interface (AWS CLI), or infrastructure as code (IaC) instruments just like the AWS Cloud Growth Equipment (AWS CDK) or AWS CloudFormation templates.
When organising CMK encryption in a manufacturing surroundings, you’ll in all probability use an automation software reasonably than the console. These instruments ultimately use the AWS API beneath the hood, and the UpdateApplication motion of the kinesisanalyticsv2 API specifically. On this submit, we analyze the additions to the API that you need to use to manage the encryption configuration.
A further top-level block ApplicationEncryptionConfigurationUpdate has been added to the UpdateApplication request payload. With this block, you may allow and disable the CMK.
You have to add the next block to the UpdateApplication request:
The KeyIdUpdate worth may be the important thing ARN, key ID, key alias title, or key alias ARN.
Disable the CMK
Equally, the next requests disable the CMK, switching again to the default AWS owned key:
Allow the CMK with CreateApplication
Theoretically, you may allow the CMK immediately once you first create the appliance utilizing the CreateApplication motion.
A top-level block ApplicationEncryptionConfiguration has been added to the CreateApplication request payload, with a syntax just like UpdateApplication.
Nonetheless, because of the interdependencies described within the earlier part, you’ll most frequently create an software with the default AWS owned key and later use UpdateApplication to allow the CMK.
When you omit ApplicationEncryptionConfiguration once you create the appliance, the default habits is utilizing the AWS owned key, for backward compatibility.
Pattern CloudFormation templates to create IAM roles and the KMS key
The method you utilize to create the roles and key and configure the appliance to make use of the CMK will range, relying on the automation you utilize and your approval and safety processes. Any automation instance we will present will seemingly not suit your processes or tooling.
Nonetheless, the next GitHub repository supplies some instance CloudFormation templates to generate a few of the IAM insurance policies and the KMS key with the proper key coverage:
- IAM coverage for the important thing administrator – Permits managing the important thing
- Base IAM coverage for the operator – Permits managing the conventional software lifecycle operations with out the CMK
- CMK IAM coverage for the operator – Offers further permissions required to handle the appliance lifecycle when the CMK is enabled
- KMS key coverage – Permits the appliance to encrypt and decrypt the appliance state and the operator to handle the appliance operations
CMK operations
We now have described the method of making a brand new Managed Service for Apache Flink software with CMK. Let’s now study different widespread operations you may carry out.
Modifications to the encryption key develop into efficient when the appliance is restarted. When you replace the configuration of a working software, this causes the appliance to restart and the brand new key for use instantly. Conversely, should you change the important thing of a READY (not working) software, the brand new key will not be really used till the appliance is restarted.
Allow a CMK on an present software
In case you have an software working with an AWS owned key, the method is just like what we described for creating new purposes. On this case, you have already got a working software state and older snapshots which are encrypted utilizing the AWS owned key.
Additionally, when you have a working software, you in all probability have already got an operator position with an IAM coverage that you need to use to manage the operator lifecycle.
The sequence of steps to allow a CMK on an present and working software is as follows:
- When you don’t have already got one, create a key administrator IAM position or consumer with permissions to create and handle keys in AWS KMS. See Utilizing IAM insurance policies with AWS KMS for extra particulars in regards to the permissions required to handle keys.
- The important thing administrator creates the CMK. The important thing coverage references the appliance ARN, the operator’s ARN, and the important thing administrator’s position or consumer ARN.
- Create a further IAM coverage that permits using the CMK and fix this coverage to the operator. Alternatively, modify the operator’s present IAM coverage by including these permissions.
- Lastly, the operator can replace the appliance and allow the CMK.The next diagram illustrates the method that happens once you execute an
UpdateApplicationmotion on the working software to allow a CMK.
The workflow consists of the next steps:
- If you replace the appliance to arrange the CMK, the next occurs:
- The applying working state, for the time being it’s encrypted with the AWS owned key, is saved in a snapshot whereas the appliance is stopped. This snapshot is encrypted with the default AWS owned key. The working software state storage is risky and destroyed when the appliance is stopped.
- The applying is redeployed, restoring the snapshot into the working software state.
- The working software state storage is now encrypted with the CMK.
- New snapshots created from this level on are encrypted utilizing the CMK.
- You’ll in all probability wish to delete all of the previous snapshots, together with the one created routinely by the
UpdateApplicationthat enabled the CMK, as a result of they’re all encrypted utilizing the AWS owned key.
Rotate the encryption key
As with every cryptographic key, it’s a greatest follow to rotate the important thing periodically for enhanced safety. Managed Service for Apache Flink doesn’t assist AWS KMS computerized key rotation, so you’ve gotten two major choices for rotating your CMK.
Choice 1: Create a brand new CMK and replace the appliance
The primary strategy includes creating a wholly new KMS key after which updating your software configuration to make use of the brand new key. This technique supplies a clear separation between the previous and new encryption keys, making it simpler to trace which knowledge was encrypted with which key model.
Let’s assume you’ve gotten a working software utilizing CMK#1 (the present key) and wish to rotate to CMK#2 (the brand new key) for enhanced safety:
- Conditions and preparation – Earlier than initiating the important thing rotation course of, it’s essential to replace the operator’s IAM coverage to incorporate permissions for each CMK#1 and CMK#2. This dual-key entry helps uninterrupted operation throughout the transition interval. After the appliance configuration has been efficiently up to date and verified, you may safely take away all permissions to CMK#1.
- Utility replace course of – The
UpdateApplicationoperation used to configure CMK#2 routinely triggers an software restart. This restart mechanism makes positive each the appliance’s working state and any newly created snapshots are encrypted utilizing the brand new CMK#2, offering quick safety advantages from the up to date encryption key. - Vital safety issues – Current snapshots, together with the automated snapshot created throughout the CMK replace course of, stay encrypted with the unique CMK#1. For full safety hygiene and to attenuate your cryptographic footprint, contemplate deleting these older snapshots after verifying that your software is functioning appropriately with the brand new encryption key.
This strategy supplies a clear separation between previous and new encrypted knowledge whereas sustaining software availability all through the important thing rotation course of.
Choice 2: Rotate the important thing materials of the prevailing CMK
The second possibility is to rotate the cryptographic materials inside your present KMS key. For a CMK used for Managed Service for Apache Flink, we advocate utilizing on-demand key materials rotation.
The advantage of this strategy is simplicity: no change is required to the appliance configuration nor to the operator’s IAM permissions.
Vital safety issues
The brand new encryption secret’s utilized by the Managed Service for Apache Flink software solely after the subsequent software restart. To make the brand new key materials efficient, instantly after the rotation, you should cease and begin utilizing snapshots to protect the appliance state or execute an UpdateApplication, which additionally forces a stop-and-restart. After the restart, you need to contemplate deleting the previous snapshots, together with the one taken routinely within the final stop-and-restart.
Change again to the AWS owned key
At any time, you may resolve to change again to utilizing an AWS owned key. The applying state continues to be encrypted, however utilizing the AWS owned key as an alternative of your CMK.
If you’re utilizing the UpdateApplication API or AWS CLI command to change again to CMK, it’s essential to explicitly go ApplicationEncryptionConfigurationUpdate, setting the important thing kind to AWS_OWNED_KEY as proven within the following snippet:
If you execute UpdateApplication to change off the CMK, the operator should nonetheless have permissions on the CMK. After the appliance is efficiently working utilizing the AWS owned key, you may safely take away any CMK-related permissions from the operator’s IAM coverage.
Check the CMK in improvement environments
In a manufacturing surroundings—or an surroundings containing delicate knowledge—you need to observe the precept of least privilege and apply the restrictive permissions described to date.
Nonetheless, if you wish to experiment with CMKs in a improvement setting, resembling utilizing the console, strictly following the manufacturing course of may develop into cumbersome. In these environments, the roles of key administrator and operator are sometimes crammed by the identical individual.
For testing functions in improvement environments, you may wish to use a permissive key coverage like the next, so you may freely experiment with CMK encryption:
This coverage mustn’t ever be utilized in an surroundings containing delicate knowledge, and particularly not in manufacturing.
Frequent caveats and pitfalls
As mentioned earlier, this function is designed to maximise safety and promote greatest practices such because the precept of least privilege. Nonetheless, this focus can introduce some nook instances try to be conscious of.
The CMK should be enabled for the service to encrypt and decrypt snapshots and working state
With AWS KMS, you may disable one key at any time. When you disable the CMK whereas the appliance is working, it’d trigger unpredictable failures. For instance, an software will be unable to revive a snapshot if the CMK used to encrypt that snapshot has been disabled. For instance, should you try to roll again an UpdateApplication that modified the CMK, and the earlier key has since been disabled, you won’t be capable of restore from an previous snapshot. Equally, you won’t be capable of restart the appliance from an older snapshot if the corresponding CMK is disabled.
When you encounter these eventualities, the answer is to reenable the required key and retry the operation.
The operator requires permissions to all keys concerned
To carry out an motion on the appliance (resembling Begin, Cease, UpdateApplication, or CreateApplicationSnapshot), the operator should have permissions for all CMKs concerned in that operation. AWS owned keys don’t require express permission.
Some operations implicitly contain two CMKs—for instance, when switching from one CMK to a different, or when switching from a CMK to an AWS owned key by disabling the CMK. In these instances, the operator should have permissions for each keys for the operation to succeed.
The identical rule applies when rolling again an UpdateApplication motion that concerned a number of CMKs.
A brand new encryption key takes impact solely after restart
A brand new encryption secret’s solely used after the appliance is restarted. That is vital once you rotate the important thing materials for a CMK. Rotating the important thing materials in AWS KMS doesn’t require updating the Managed Flink software’s configuration. Nonetheless, it’s essential to restart the appliance as a separate step after rotating the important thing. When you don’t restart the appliance, it’ll proceed to make use of the previous encryption key for its working state and snapshots till the subsequent restart.
For that reason, it is suggested to not allow computerized key rotation for the CMK. When computerized rotation is enabled, AWS KMS may rotate the important thing materials at any time, however your software won’t begin utilizing the brand new key till it’s subsequent restarted.
CMKs are solely supported with Flink runtime 1.20 or later
CMKs are solely supported when you’re utilizing the Flink runtime 1.20 or later. In case your software is at the moment utilizing an older runtime, you need to improve to Flink 1.20 first. Managed Service for Apache Flink makes it simple to improve your present software utilizing the in-place model improve.
Conclusion
Managed Service for Apache Flink supplies sturdy safety by enabling encryption by default, defending each the working state and persistently saved state of your purposes. For organizations that require full management over their encryption keys (typically attributable to regulatory or inner coverage wants), the power to make use of a CMK built-in with AWS KMS affords a brand new degree of assurance.
By utilizing CMKs, you may tailor encryption controls to your particular compliance necessities. Nonetheless, this flexibility comes with the necessity for cautious planning: the CMK function is deliberately designed to implement the precept of least privilege and robust position separation, which might introduce complexity round permissions and operational processes.
On this submit, we reviewed the important thing steps for enabling CMKs on present purposes, creating new purposes with a CMK, and managing key rotation. Every of those processes provides you larger management over your knowledge safety but in addition requires consideration to entry administration and operational greatest practices.
To get began with CMKs and for extra complete steering, check with Key administration in Amazon Managed Service for Apache Flink.