Deep dive into the Amazon Managed Service for Apache Fink utility lifecycle – Half 2

In Half 1 of this sequence, we mentioned basic operations to manage the lifecycle of your Amazon Managed Service for Apache Flink utility. In case you are utilizing higher-level instruments resembling AWS CloudFormation or Terraform, the device will execute these operations for you. Nonetheless, understanding the elemental operations and what the service mechanically does can present some degree of Mechanical Sympathy to confidently implement a extra sturdy automation.

Within the first a part of this sequence, we centered on the glad paths. In a really perfect world, failures don’t occur, and each change you deploy works completely. Nonetheless, the true world is much less predictable. Quoting Werner Vogels, Amazon’s CTO, “Every thing fails, on a regular basis.”

On this publish, we discover failure situations that may occur throughout regular operations or once you deploy a change or scale the appliance, and methods to monitor operations to detect and recuperate when one thing goes improper.

The much less glad path

A strong automation have to be designed to deal with failure situations, particularly throughout operations. To do this, we have to perceive how Apache Flink can deviate from the glad path. Because of the nature of Flink as a stateful stream processing engine, detecting and resolving failure situations requires totally different strategies in comparison with different long-running purposes, resembling microservices or short-lived serverless capabilities (resembling AWS Lambda).

Flink’s habits on runtime errors: The fail-and-restart loop

When a Flink job encounters an sudden error at runtime (an unhandled exception), the conventional habits is to fail, cease the processing, and restart from the newest checkpoint. Checkpoints enable Flink to help information consistency and no information loss in case of failure. Additionally, as a result of Flink is designed for stream processing purposes, which run repeatedly, if the error occurs once more, the default habits is to maintain restarting, hoping the issue is transient and the appliance will finally recuperate the conventional processing.In some circumstances, the issue isn’t transient, nevertheless. For instance, once you deploy a code change that accommodates a bug, inflicting the job to fail as quickly because it begins processing information, or if the anticipated schema doesn’t match the data within the supply, inflicting deserialization or processing errors. The identical situation may also occur when you mistakenly modified a configuration that forestalls a connector to succeed in the exterior system. In these circumstances, the job is caught in a fail-and-restart loop, indefinitely, or till you actively force-stop it.

When this occurs, the Managed Service for Apache Flink utility standing may be RUNNING, however the underlying Flink job is definitely failing and restarting. The AWS Administration Console provides you a touch, pointing that the appliance would possibly want consideration (see the next screenshot).

Within the following sections, we learn to monitor the appliance and job standing, to mechanically react to this example.

When beginning or updating the appliance goes improper

To know the failure mode, let’s assessment what occurs mechanically once you begin the appliance, or when the appliance restarts after you issued UpdateApplication command, as we explored in Half 1 of this sequence. The next diagram illustrates what occurs when an utility begins.

The workflow consists of the next steps:

1. Managed Service for Apache Flink provisions a cluster devoted to your utility.
2. The code and configuration are submitted to the Job Supervisor node.
3. The code within the foremost() methodology of your utility runs, defining the dataflow of your utility.
4. Flink deploys to the Job Supervisor nodes the substasks that make up your job.
5. The job and utility standing change to RUNNING. Nonetheless, subtasks begin initializing now.
6. Subtasks restore their state, if relevant, and initialize any sources. For instance, a Kafka connector’s subtask initializes the Kafka shopper and subscribes the subject.
7. When all subtasks are efficiently initialized, they alter to RUNNING standing and the job begins processing information.

To new Flink customers, it may be complicated {that a} RUNNING standing doesn’t essentially indicate the job is wholesome and processing information.When one thing goes improper in the course of the means of beginning (or restarting) the appliance, relying on the section when the issue arises, you would possibly observe two several types of failure modes:

• (a) An issue prevents the appliance code from being deployed – Your utility would possibly encounter this failure situation if the deployment fails as quickly because the code and configuration are handed to the Job Supervisor (step 2 of the method), for instance if the appliance code package deal is malformed. A typical error is when the JAR is lacking a mainClass or if mainClass factors to a category that doesn’t exist. This failure mode may also occur if the code of your foremost() methodology throws an unhandled exception (step 3). In these circumstances, the appliance fails to alter to RUNNING, and reverts to READY after the try.
• (b) The appliance is began, the job is caught in a fail-and-restart loop – An issue would possibly happen later within the course of, after the appliance standing has modified RUNNING. For instance, after the Flink job has been deployed to the cluster (step 4 of the method), a element would possibly fail to initialize (step 6). This would possibly occur when a connector is misconfigured, or an issue prevents it from connecting to the exterior system. For instance, a Kafka connector would possibly fail to hook up with the Kafka cluster due to the connector’s misconfiguration or networking points. One other potential situation is when the Flink job efficiently initializes, however it throws an exception as quickly because it begins processing information (step 7). When this occurs, Flink reacts to a runtime error and would possibly get caught in a fail-and-restart loop.

The next diagram illustrates the sequence of utility standing, together with the 2 failure situations simply described.

Troubleshooting

We’ve got examined what can go improper throughout operations, particularly once you replace a RUNNING utility or restart an utility after altering its configuration. On this part, we discover how we will act on these failure situations.

Roll again a change

Once you deploy a change and understand one thing isn’t fairly proper, you usually wish to roll again the change and put the appliance again in working order, till you examine and repair the issue. Managed Service for Apache Flink offers a swish method to revert (roll again) a change, additionally restarting the processing from the purpose it was stopped earlier than making use of the fault change, offering consistency and no information loss.In Managed Service for Apache Flink, there are two kinds of rollbacks:

• Automated – Throughout an automated rollback (additionally referred to as system rollback), if enabled, the service mechanically detects when the appliance fails to restart after a change, or when the job begins however instantly falls right into a fail-and-restart loop. In these conditions, the rollback course of mechanically restores the appliance configuration model earlier than the final change was utilized and restarts the appliance from the snapshot taken when the change was deployed. See Enhance the resilience of Amazon Managed Service for Apache Flink utility with system-rollback function for extra particulars. This function is disabled by default. You’ll be able to allow it as a part of the appliance configuration.
• Handbook – A handbook rollback API operation is sort of a system rollback, however it’s initiated by the person. If the appliance is operating however you observe one thing not behaving as anticipated after making use of a change, you may set off the rollback operation utilizing the RollbackApplication API motion or the console. Handbook rollback is feasible when the appliance is RUNNING or UPDATING.

Each rollbacks work equally, restoring the configuration model earlier than the change and restarting with the snapshot taken earlier than the change. This prevents information loss and brings you again to a model of the appliance that was working. Additionally, this makes use of the code package deal that was saved on the time you created the earlier configuration model (the one you might be rolling again to), so there isn’t a inconsistency between code, configuration, and snapshot, even when within the meantime you’ve changed or deleted the code package deal from the Amazon Easy Storage Service (Amazon S3) bucket.

Implicit rollback: Replace with an older configuration

A 3rd method to roll again a change is to easily replace the configuration, bringing it again to what it was earlier than the final change. This creates a brand new configuration model, and requires the right model of the code package deal to be obtainable within the S3 bucket once you problem the UpdateApplication command.

Why is there a 3rd possibility when the service offers system rollback and the managed RollbackApplication motion? As a result of most high-level infrastructure-as-code (IaC) frameworks resembling Terraform use this technique, explicitly overwriting the configuration. You will need to perceive this risk regardless that you’ll most likely use the managed rollback when you implement your automation based mostly on the low-level actions.

The next are two necessary caveats to think about for this implicit rollback:

• You’ll usually wish to restart the appliance from the snapshot that was taken earlier than the defective change was deployed. If the appliance is presently RUNNING and wholesome, this isn’t the newest snapshot (RESTORE_FROM_LATEST_SNAPSHOT), however moderately the earlier one. You need to set the restart from RESTORE_FROM_CUSTOM_SNAPSHOT and choose the right snapshot.
• UpdateApplication solely works if the appliance is RUNNING and wholesome, and the job could be gracefully stopped with a snapshot. Conversely, if the appliance is caught in a fail-and-restart loop, you should force-stop it first, change the configuration whereas the appliance is READY, and later begin the appliance from the snapshot that was taken earlier than the defective change was deployed.

Pressure-stop the appliance

In regular situations, you cease the appliance gracefully, with the automated snapshot creation. Nonetheless, this may not be potential in some situations, resembling if the Flink job is caught in a fail-and-restart loop. This would possibly occur, for instance, if an exterior system the job makes use of stops working, or as a result of the AWS Id and Entry Administration (IAM) configuration was erroneously modified, eradicating permissions required by the job.

When the Flink job will get caught in a fail-and-restart loop after a defective change, your first possibility must be utilizing RollbackApplication, which mechanically restores the earlier configuration and begins from the right snapshot. Within the uncommon circumstances you may’t cease the appliance gracefully or use RollbackApplication, the final resort is force-stopping the appliance. Pressure-stop makes use of the StopApplication command with Pressure=true. It’s also possible to force-stop the appliance from the console.

Once you force-stop an utility, no snapshot is taken (if that had been potential, you’d have been in a position to gracefully cease). Once you restart the appliance, you may both skip restoring from a snapshot (SKIP_RESTORE_FROM_SNAPSHOT) or use a snapshot that was beforehand taken, scheduled utilizing Snapshot Supervisor, or manually, utilizing the console or CreateApplicationSnapshot API motion.

We strongly suggest establishing scheduled snapshots for all manufacturing purposes you can’t afford restarting with no state.

Monitoring Apache Flink utility operations

Efficient monitoring of your Apache Flink purposes throughout and after operations is essential to confirm the end result of the operation and permit lifecycle automation to boost alarms or react, in case one thing goes improper.

The primary indicators you need to use throughout operations embody the FullRestarts metric (obtainable in Amazon CloudWatch) and the appliance, job, and job standing.

Monitoring the end result of an operation

The only method to detect the end result of an operation, resembling StartApplication or UpdateApplication, is to make use of the ListApplicationOperations API command. This command returns an inventory of the latest operations of a selected utility, together with upkeep occasions that power an utility restart.

For instance, to retrieve the standing of the latest operation, you need to use the next command:

aws kinesisanalyticsv2 list-application-operations 
    --application-name MyApplication 
   | jq '.ApplicationOperationInfoList 
   | sort_by(.StartTime) | final'

The output will probably be just like the next code:

{
  "Operation": "UpdateApplication",
  "OperationId": "12abCDeGghIlM",
  "StartTime": "2025-08-06T09:24:22+01:00",
  "EndTime": "2025-08-06T09:26:56+01:00",
  "OperationStatus": "IN_PROGRESS"
}

OperationStatus will observe the identical logic as the appliance standing reported by the console and by DescribeApplication. This implies it may not detect a failure in the course of the operator initialization or whereas the job begins processing information. As we have now realized, these failures would possibly put the appliance in a fail-and-restart loop. To detect these situations utilizing your automation, you should use different strategies, which we cowl in the remainder of this part.

Detecting the fail-and-restart loop utilizing the FullRestarts metric

The only method to detect whether or not the appliance is caught in a fail-and-restart loop is utilizing the fullRestarts metric, obtainable in CloudWatch Metrics. This metric counts the variety of restarts of the Flink job after you began the appliance with a StartApplication command or restarted with UpdateApplication.

In a wholesome utility, the variety of full restarts ought to ideally be zero. A single full restart may be acceptable throughout deployment or deliberate upkeep; a number of restarts usually point out some problem. We suggest to not set off an alarm on a single restart, and even a few consecutive restarts.

The alarm ought to solely be triggered when the appliance is caught in a fail-and-restart loop. This suggests checking whether or not a number of restarts have occurred over a comparatively brief time frame. Deciding the interval isn’t trivial, as a result of the time the Flink job takes to restart from a checkpoint is determined by the dimensions of the appliance state. Nonetheless, if the state of your utility is decrease than a number of GB per KPU, you may safely assume the appliance ought to begin in lower than a minute.

The aim is making a CloudWatch alarm that triggers when fullRestarts retains rising over a time interval ample for a number of restarts. For instance, assuming your utility restarts in lower than 1 minute, you may create a CloudWatch alarm that depends on the DIFF math expression of the fullRestarts metric. The next screenshot reveals an instance of the alarm particulars.

This instance is a conservative alarm, solely triggering if the appliance retains restarting for over 5 minutes. This implies you detect the issue after no less than 5 minutes. You would possibly think about lowering the time to detect the failure earlier. Nonetheless, watch out to not set off an alarm after only one or two restarts. Occasional restarts would possibly occur, for instance throughout regular upkeep (patching) that’s managed by the service, or for a transient error of an exterior system. Flink is designed to recuperate from these circumstances with minimal downtime and no information loss.

Detecting whether or not the job is up and operating: Monitoring utility, job, and job standing

We’ve got mentioned how you’ve totally different statuses: the standing of the appliance, job, and subtask. In Managed Service for Apache Flink, the appliance and job standing change to RUNNING when the subtasks are efficiently deployed on the cluster. Nonetheless, the job isn’t actually operating and processing information till all of the subtasks are RUNNING.

Observing the appliance standing throughout operations

The appliance standing is seen on the console, as proven within the following screenshot.

In your automation, you may ballot the DescribeApplication API motion to watch the appliance standing. The next command reveals methods to use the AWS Command Line Interface (AWS CLI) and jq command to extract the standing string of an utility:

aws kinesisanalyticsv2 describe-application  
    --application-name <your-application-name> 
    | jq -r '.ApplicationDetail.ApplicationStatus'

Observing job and subtask standing

Managed Service for Apache Flink provides you entry to the Flink Dashboard, which offers helpful data for troubleshooting, together with the standing of all subtasks. The next screenshot, for instance, reveals a wholesome job the place all subtasks are RUNNING.

Within the following screenshot, we will see a job the place subtasks are failing and restarting.

In your automation, once you begin the appliance or deploy a change, you wish to be certain the job is finally up and operating and processing information. This occurs when all of the subtasks are RUNNING. Notice that ready for the job standing to develop into RUNNING after an operation isn’t utterly secure. A subtask would possibly nonetheless fail and trigger the job to restart after it was reported as RUNNING.

After you execute a lifecycle operation, your automation can ballot the substasks standing ready for considered one of two occasions:

• All subtasks report RUNNING – This means the operation was profitable and your Flink job is up and operating.
• Any subtask experiences FAILING or CANCELED – This means one thing went improper, and the appliance is probably going caught in a fail-and-restart loop. You must intervene, for instance, force-stopping the appliance after which rolling again the change.

In case you are restarting from a snapshot and the state of your utility is sort of huge, you would possibly observe subtasks will report INITIALIZING standing for longer. Through the initialization, Flink restores the state of the operator earlier than altering to RUNNING.

The Flink REST API exposes the state of the subtasks, and can be utilized in your automation. In Managed Service for Apache Flink, this requires three steps:

1. Generate a pre-signed URL to entry the Flink REST API utilizing the CreateApplicationPresignedUrl API motion.
2. Make a GET request to the /jobs endpoint of the Flink REST API to retrieve the job ID.
3. Make a GET request to the /jobs/<job-id> endpoint to retrieve the standing of the subtasks.

The next GitHub repository offers a shell script to retrieve the standing of the duties of a given Managed Service for Apache Flink utility.

Monitoring subtasks failure whereas the job is operating

The method of polling the Flink REST API can be utilized in your automation, instantly after an operation, to watch whether or not the operation was finally profitable.

We strongly suggest to not repeatedly ballot the Flink REST API whereas the job is operating to detect failures. This operation is useful resource consuming, and would possibly degrade efficiency or trigger errors.

To observe for suspicious subtask standing modifications throughout regular operations, we suggest utilizing CloudWatch Logs as a substitute. The next CloudWatch Logs Insights question extracts all subtask state transitions:

fields , message
| parse message /^(?<job>.+) switched from (?<fromStatus>[A-Z]+) to (?<toStatus>[A-Z]+)./
| filter ispresent(job) and ispresent(fromStatus) and ispresent(toStatus)
| show , job, fromStatus, toStatus
| restrict 10000

How Managed Service for Apache Flink minimizes processing downtime

We’ve got seen how Flink is designed for sturdy consistency. To ensure exactly-once state consistency, Flink quickly stops the processing to deploy any modifications, together with scaling. This downtime is required for Flink to take a constant copy of the appliance state and put it aside in a savepoint. After the change is deployed, the job is restarted from the savepoint, and there’s no information loss. In Managed Service for Apache Flink, updates are totally managed. When snapshots are enabled, UpdateApplication mechanically stops the job and makes use of snapshots (based mostly on Flink’s savepoints) to retain the state.

Flink ensures no information loss. Nonetheless, your small business necessities or Service Stage Targets (SLOs) may also impose a most delay for the information obtained by downstream programs, or end-to-end latency. This delay is affected by the processing downtime, or the time the job doesn’t course of information to permit Flink deploying the change.With Flink, some processing downtime is unavoidable. Nonetheless, Managed Service for Apache Flink is designed to reduce the processing downtime once you deploy a change.

We’ve got seen how the service runs your utility in a devoted cluster, for full isolation. Once you problem UpdateApplication on a RUNNING utility, the service prepares a brand new cluster with the required quantity of sources. This operation would possibly take a while. Nonetheless, this doesn’t have an effect on the processing downtime, as a result of the service retains the job operating and processing information on the unique cluster till the final potential second, when the brand new cluster is prepared. At this level, the service stops your job with a savepoint and restarts it on the brand new cluster.

Throughout this operation, you might be solely charged for the variety of KPU of a single cluster.

The next diagram illustrates the distinction between the length of the replace operation, or the time the appliance standing is UPDATING, and the processing downtime, observable from the job standing, seen within the Flink Dashboard.

You’ll be able to observe this course of, retaining each the appliance console and Flink Dashboard open, once you replace the configuration of a operating utility, even with no modifications. The Flink Dashboard will develop into quickly unavailable when the service switches to the brand new cluster. Moreover, you may’t use the script we offered to verify the job standing for this scope. Although the cluster retains serving the Flink Dashboard till it’s tore down, the CreateApplicationPresignedUrl motion doesn’t work whereas the appliance is UPDATING.

The processing time (the time the job isn’t operating on both clusters) is determined by the time the job takes to cease with a savepoint (snapshot) and restore the state within the new cluster. This time largely is determined by the dimensions of the appliance state. Information skew may also have an effect on the savepoint time because of the barrier alignment mechanism. For a deep dive into the Flink’s barrier alignment mechanism, consult with Optimize checkpointing in your Amazon Managed Service for Apache Flink purposes with buffer debloating and unaligned checkpoints, retaining in thoughts that savepoints are at all times aligned.

For the scope of your automation, you usually wish to wait till the job is again up and operating and processing information. You usually wish to set a timeout. If each the appliance and job don’t return to RUNNING inside this timeout, one thing most likely went improper and also you would possibly wish to increase an alarm or power a rollback. This timeout ought to think about your complete replace operation length.

Conclusion

On this publish, we mentioned potential failure situations once you deploy a change or scale your utility. We confirmed how Managed Service for Apache Flink rollback functionalities can seamlessly convey you again to a secure place after a change went improper. We additionally explored how one can automate monitoring operations to watch utility, job, and subtask standing, and methods to use the fullRestarts metric to detect when the job is in a fail-and-restart loop.

For extra data, see Run a Managed Service for Apache Flink utility, Implement fault tolerance in Managed Service for Apache Flink, and Handle utility backups utilizing Snapshots.

In regards to the authors