Finest practices for upgrading from Amazon Redshift DC2 to RA3 and Amazon Redshift Serverless

Amazon Redshift is a quick, petabyte-scale cloud knowledge warehouse that makes it easy and cost-effective to investigate your knowledge utilizing commonplace SQL and your current enterprise intelligence (BI) instruments. Tens of hundreds of shoppers depend on Amazon Redshift to investigate exabytes of information and run advanced analytical queries, delivering the very best price-performance.

With a completely managed, AI-powered, massively parallel processing (MPP) structure, Amazon Redshift drives enterprise decision-making rapidly and cost-effectively. Beforehand, Amazon Redshift provided DC2 (Dense Compute) node varieties optimized for compute-intensive workloads. Nonetheless, they lacked the pliability to scale compute and storage independently and didn’t assist lots of the trendy options now accessible. As analytical calls for develop, many shoppers are upgrading from DC2 to RA3 or Amazon Redshift Serverless, which supply impartial compute and storage scaling, together with superior capabilities similar to knowledge sharing, zero-ETL integration, and built-in synthetic intelligence and machine studying (AI/ML) assist with Amazon Redshift ML.

This submit offers a sensible information to plan your goal structure and migration technique, protecting improve choices, key concerns, and finest practices to facilitate a profitable and seamless transition.

Improve course of from DC2 nodes to RA3 and Redshift Serverless

Step one in direction of improve is to grasp how the brand new structure must be sized; for this, AWS offers a suggestion desk for provisioned clusters. When figuring out the configuration for Redshift Serverless endpoints, you possibly can assess compute capability particulars by inspecting the connection between RPUs and reminiscence. Every RPU allocates 16 GiB of RAM. To estimate the bottom RPU requirement, divide your DC2 nodes cluster’s whole RAM by 16. These suggestions present steering in sizing the preliminary goal structure however depend upon the computing necessities of your workload. To raised estimate your necessities, take into account conducting a proof of idea that makes use of Redshift Check Drive to run potential configurations. To be taught extra, see Discover the very best Amazon Redshift configuration in your workload utilizing Redshift Check Drive and Efficiently conduct a proof of idea in Amazon Redshift. After you resolve on the goal configuration and structure, you possibly can construct the technique for upgrading.

Structure patterns

Step one is to outline the goal structure in your answer. You possibly can select the principle structure sample that finest aligns together with your use case from the choices offered in Structure patterns to optimize Amazon Redshift efficiency at scale. There are two foremost eventualities, as illustrated within the following diagram.

On the time of writing, Redshift Serverless doesn’t have guide workload administration; every thing runs with computerized workload administration. Contemplate isolating your workload into a number of endpoints primarily based on use case to allow impartial scaling and higher efficiency. For extra data, discuss with Structure patterns to optimize Amazon Redshift efficiency at scale.

Improve methods

You possibly can select from two potential improve choices when upgrading from DC2 nodes to RA3 nodes or Redshift Serverless:

• Full re-architecture – Step one is to judge and assess the workloads to find out whether or not you may gain advantage from a contemporary knowledge structure, then re-architect the prevailing platform through the improve course of from DC2 nodes.
• Phased method– This can be a two-stage technique. The primary stage includes an easy migration to the goal RA3 or Serverless configuration. Within the second stage, you possibly can modernize the goal structure by profiting from cutting-edge Redshift options.

We often suggest a phased method, which permits for a smoother transition whereas enabling future optimization. The primary stage of a phased method consists of the next steps:

• Consider an equal RA3 nodes or Redshift Serverless configuration in your current DC2 cluster, utilizing the sizing pointers for provisioned clusters or the compute capability choices for serverless endpoints.
• Completely validate the chosen goal configuration in a non-production surroundings utilizing Redshift Check Drive. This automated device simplifies the method of simulating your manufacturing workloads on varied potential goal configurations, enabling a complete what-if evaluation. This step is strongly beneficial.
• Proceed to the improve course of when you’re glad with the price-performance ratio of a specific goal configuration, utilizing one of many strategies detailed within the following part.

Redshift RA3 cases and Redshift Serverless present entry to highly effective new capabilities, together with zero-ETL, Amazon Redshift Streaming Ingestion, knowledge sharing writes, and impartial compute and storage scaling. To maximise these advantages, we suggest conducting a complete evaluation of your present structure (the second stage of a phased method) to establish alternatives for modernization utilizing Amazon Redshift’s newest options. For instance:

Improve choices

You possibly can select from 3 ways to resize or improve a Redshift cluster from DC2 to RA3 or Redshift Serverless: snapshot restore, basic resize, and elastic resize.

Snapshot restore

The snapshot restore technique follows a sequential course of that begins with capturing a snapshot of your current (supply) cluster. This snapshot is then used to create a brand new goal cluster together with your desired specs. After creation, it’s important to confirm knowledge integrity by confirming that knowledge has been appropriately transferred to the goal cluster. An essential consideration is that any knowledge written to the supply cluster after the preliminary snapshot have to be manually transferred to keep up synchronization.

This technique provides the next benefits:

• Permits for the validation of the brand new RA3 or Serverless setup with out affecting the prevailing DC2 cluster
• Supplies the pliability to revive to totally different AWS Areas or Availability Zones
• Minimizes cluster downtime for write operations through the transition

Bear in mind the next concerns:

• Setup and knowledge restore would possibly take longer than elastic resize.
• You would possibly encounter knowledge synchronization challenges. Any new knowledge written to the supply cluster after snapshot creation requires guide copying to the goal. This course of would possibly want a number of iterations to attain full synchronization and require downtime earlier than cutoff.
• A brand new Redshift endpoint is generated, necessitating connection updates. Contemplate renaming each clusters to be able to preserve the unique endpoint (make certain the brand new goal cluster adopts the unique supply cluster’s title)

Basic resize

Amazon Redshift creates a goal cluster and migrates your knowledge and metadata to it from the supply cluster utilizing a backup and restore operation. All of your knowledge, together with database schemas and consumer configurations, is precisely transferred to the brand new cluster. The supply cluster restarts initially and is unavailable for a couple of minutes, inflicting minimal downtime. It rapidly resumes, permitting each learn and write operations because the resize continues within the background.

Basic resize is a two-stage course of:

• Stage 1 (vital path) – Throughout this stage, metadata migration happens between the supply and goal configurations, quickly putting the supply cluster in read-only mode. This preliminary section is usually transient. When this section is full, the cluster is made accessible for learn and write queries. Though tables initially configured with KEY distribution type are quickly saved utilizing EVEN distribution, they are going to be redistributed to their authentic KEY distribution throughout Stage 2 of the method.
• Stage 2 (background operations) – This stage focuses on restoring knowledge to its authentic distribution patterns. This operation runs within the background with low precedence with out interfering with the first migration course of. The length of this stage varies primarily based on a number of elements, together with the quantity of information being redistributed, ongoing cluster workload, and the goal configuration getting used.

The general resize length is primarily decided by the info quantity being processed. You possibly can monitor progress on the Amazon Redshift console or by utilizing the SYS_RESTORE_STATE system view, which shows the proportion accomplished for the desk being transformed (accessing this view requires superuser privileges).

The basic resize method provides the next benefits:

• All potential goal node configurations are supported
• A complete reconfiguration of the supply cluster rebalances the info slices to default per node, resulting in even knowledge distribution throughout the nodes

Nonetheless, take into accout the next:

• Stage 2 redistributes the info for optimum efficiency. Nonetheless, Stage 2 runs at a decrease precedence, and in busy clusters, it may possibly take a very long time to finish. To hurry up the method, you possibly can manually run the ALTER TABLE DISTSTYLE command in your tables having KEY DISTSTYLE. By executing this command, you possibly can prioritize the info redistribution to occur quicker, mitigating any potential efficiency degradation as a result of ongoing Stage 2 course of.
• Because of the Stage 2 background redistribution course of, queries can take longer to finish through the resize operation. Contemplate enabling concurrency scaling as a mitigation technique.
• Drop pointless and unused tables earlier than initiating a resize to hurry up knowledge distribution.
• The snapshot used for the resize operation turns into devoted to this operation solely. Due to this fact, it may possibly’t be used for a desk restore or different objective.
• The cluster should function inside a digital personal cloud (VPC).
• This method requires a brand new or a latest guide snapshot taken earlier than initiating a basic resize.
• We suggest scheduling the operation throughout off-peak hours or upkeep home windows for minimal enterprise impression.

Elastic resize

When utilizing elastic resize to alter the node kind, Amazon Redshift follows a sequential course of. It begins by making a snapshot of your current cluster, then provisions a brand new goal cluster utilizing the latest knowledge from that snapshot. Whereas knowledge transfers to the brand new cluster within the background, the system stays in read-only mode. Because the resize operation approaches completion, Amazon Redshift mechanically redirects the endpoint to the brand new cluster and stops all connections to the unique one. If any points come up throughout this course of, the system sometimes performs an computerized rollback with out requiring guide intervention, although such failures are uncommon.

Elastic resize provides a number of benefits:

• It’s a fast course of that takes 10–quarter-hour on common
• Customers preserve learn entry to their knowledge through the course of, experiencing solely minimal interruption
• The cluster endpoint stays unchanged all through and after the operation

When contemplating this method, take into accout the next:

• Elastic resize operations can solely be carried out on clusters utilizing the EC2-VPC platform. Due to this fact, it’s not accessible for Redshift Serverless.
• The goal node configuration should present adequate storage capability for current knowledge.
• Not all goal cluster configurations assist elastic resize. In such circumstances, think about using basic resize or snapshot restore.
• After the method is began, elastic resize can’t be stopped.
• Knowledge slices stay unchanged; this could probably trigger some knowledge or CPU skew.

Improve suggestions

The next flowchart visually guides the decision-making course of for selecting the suitable Amazon Redshift improve technique.

When upgrading Amazon Redshift, the strategy is determined by the goal configuration and operational constraints. For Redshift Serverless, all the time use the snapshot restore technique. If upgrading to an RA3 provisioned cluster, you possibly can select from two choices: use snapshot restore if a full upkeep window with downtime is appropriate, or select basic resize for minimal downtime, as a result of it rebalances the info slices to default per node, resulting in even knowledge distribution throughout the nodes. Though you need to use elastic resize for sure node kind adjustments (for instance, DC2 to RA3) inside particular ranges, it’s not beneficial as a result of elastic resize doesn’t change the variety of slices, probably resulting in knowledge or CPU skew, which may later impression the efficiency of the Redshift cluster. Nonetheless, elastic resize stays the first suggestion when you’ll want to add or scale back nodes in an current cluster.

Finest practices for migration

When planning your migration, take into account the next finest practices:

• Conduct a pre-migration evaluation utilizing Amazon Redshift Advisor or Amazon CloudWatch.
• Select the appropriate goal structure primarily based in your use circumstances and workloads. You should utilize Redshift Check Drive to find out the appropriate goal structure.
• Backup utilizing guide snapshots, and allow automated rollback.
• Talk timelines, downtime, and adjustments to stakeholders.
• Replace runbooks with new structure particulars and endpoints.
• Validate workloads utilizing benchmarks and knowledge checksum.
• Use upkeep home windows for ultimate syncs and cutovers.

By following these practices, you possibly can obtain a managed, low-risk migration that balances efficiency, value, and operational continuity.

Conclusion

Migrating from Redshift DC2 nodes to RA3 nodes or Redshift Serverless requires a structured method to assist efficiency, cost-efficiency, and minimal disruption. By choosing the appropriate structure in your workload, and validating knowledge and workloads post-migration, organizations can seamlessly modernize their knowledge platforms. This improve facilitates long-term success, serving to groups totally harness RA3’s scalable storage or Redshift Serverless auto scaling capabilities whereas optimizing prices and efficiency.

Concerning the authors