Streamline your knowledge governance by deploying Amazon DataZone with the AWS CDK

Managing knowledge throughout various environments is usually a advanced and daunting activity. Amazon DataZone simplifies this so you may catalog, uncover, share, and govern knowledge saved throughout AWS, on premises, and third-party sources.

Many organizations handle huge quantities of information property owned by numerous groups, creating a fancy panorama that poses challenges for scalable knowledge administration. These organizations require a sturdy infrastructure as code (IaC) strategy to deploy and handle their knowledge governance options. On this publish, we discover how one can deploy Amazon DataZone utilizing the AWS Cloud Growth Package (AWS CDK) to realize seamless, scalable, and safe knowledge governance.

Overview of answer

By utilizing IaC with the AWS CDK, organizations can effectively deploy and handle their knowledge governance options. This strategy offers scalability, safety, and seamless integration throughout all groups, permitting for constant and automatic deployments.

The AWS CDK is a framework for outlining cloud IaC and provisioning it by AWS CloudFormation. Builders can use any of the supported programming languages to outline reusable cloud parts often known as constructs. A assemble is a reusable and programmable element that represents AWS sources. The AWS CDK interprets the high-level constructs outlined by you into equal CloudFormation templates. AWS CloudFormation provisions the sources specified within the template, streamlining the utilization of IaC on AWS.

Amazon DataZone core parts are the constructing blocks to create a complete end-to-end answer for knowledge administration and knowledge governance. The next are the Amazon DataZone core parts. For extra particulars, see Amazon DataZone terminology and ideas.

• Amazon DataZone area – You need to use an Amazon DataZone area to prepare your property, customers, and their tasks. By associating extra AWS accounts together with your Amazon DataZone domains, you may carry collectively your knowledge sources.
• Knowledge portal – The knowledge portal is exterior the AWS Administration Console. It is a browser-based net software the place totally different customers can catalog, uncover, govern, share, and analyze knowledge in a self-service vogue.
• Enterprise knowledge catalog – You need to use this element to catalog knowledge throughout your group with enterprise context and allow everybody in your group to find and perceive knowledge rapidly.
• Tasks – In Amazon DataZone, tasks are enterprise use case-based groupings of individuals, property (knowledge), and instruments used to simplify entry to AWS analytics.
• Environments – Inside Amazon DataZone tasks, environments are collections of zero or extra configured sources on which a given set of AWS Id and Entry Administration (IAM) principals (for instance, customers with a contributor permissions) can function.
• Amazon DataZone knowledge supply – In Amazon DataZone, you may publish an AWS Glue Knowledge Catalog knowledge supply or Amazon Redshift knowledge supply.
• Publish and subscribe workflows – You need to use these automated workflows to safe knowledge between producers and customers in a self-service method and ensure that everybody in your group has entry to the proper knowledge for the proper objective.

We use an AWS CDK app to display how one can create and deploy core parts of Amazon DataZone in an AWS account. The next diagram illustrates the first core parts that we create.

Along with the core parts deployed with the AWS CDK, we offer a customized useful resource module to create Amazon DataZone parts comparable to glossaries, glossary phrases, and metadata types, which aren’t supported by AWS CDK constructs (on the time of writing).

Stipulations

The next native machine conditions are required earlier than beginning:

• An AWS account (with AWS IAM Id Middle enabled).
• Both Bash or ZSH terminal.
• The AWS Command Line Interface (AWS CLI) v2 put in.
• Python model 3.10 or increased.
• The AWS SDK for Python model 1.34.87 or increased.
• Node model v18.17.* or increased.
• NPM model v10.2.* or increased.
• An AWS Glue desk to be registered as a pattern knowledge supply in an Amazon DataZone undertaking.
• As a part of this publish, we need to publish AWS Glue tables from an AWS Glue database that already exists. For this, you could explicitly present Amazon DataZone with the permissions to entry tables on this current AWS Glue database. For extra info, check with Configure Lake Formation permissions for Amazon DataZone.

  

Deploy the answer

Full the next steps to deploy the answer:

1. Clone the GitHub repository and go to the basis of your downloaded repository folder:

   git clone https://github.com/aws-samples/amazon-datazone-cdk-example.git
   cd amazon-datazone-cdk-example

2. Set up native dependencies:

   $ npm ci ### this may set up the packages configured in package-lock.json

3. Check in to your AWS account utilizing the AWS CLI by configuring your credential file (change <PROFILE_NAME> with the profile identify of your deployment AWS account):

   $ export AWS_PROFILE=<PROFILE_NAME>

4. Bootstrap the AWS CDK atmosphere (it is a one-time exercise and never wanted in case your AWS account is already bootstrapped):
5. Run the script to switch the placeholders on your AWS account and AWS Area within the config recordsdata:

   $ ./scripts/put together.sh <<YOUR_AWS_ACCOUNT_ID>> <<YOUR_AWS_REGION>>

The previous command will change the AWS_ACCOUNT_ID_PLACEHOLDER and AWS_REGION_PLACEHOLDER values within the following config recordsdata:

• lib/config/project_config.json
• lib/config/project_environment_config.json
• lib/constants.ts

Subsequent, you configure your Amazon DataZone area, undertaking, enterprise glossary, metadata types, and environments together with your knowledge supply.

6. Go to the file lib/constants.ts. You may maintain the DOMAIN_NAME supplied or replace it as wanted.
7. Go to the file lib/config/project_config.json. You may maintain the instance values for projectName and projectDescription or replace them. An instance worth for projectMembers has additionally been supplied (as proven within the following code snippet). Replace the worth of the memberIdentifier parameter with an IAM position ARN of your alternative that you just wish to be the proprietor of this undertaking.

   "projectMembers": [
               {
                   "memberIdentifier": "arn:aws:iam::AWS_ACCOUNT_ID_PLACEHOLDER:role/Admin",
                   "memberIdentifierType": "UserIdentifier"
               }
           ]

8. Go to the file lib/config/project_glossary_config.json. An instance enterprise glossary and glossary phrases are supplied for the tasks; you may maintain them as is or replace them together with your undertaking identify, enterprise glossary, and glossary phrases.
9. Go to the lib/config/project_form_config.json file. You may maintain the instance metadata types supplied for the tasks or replace your undertaking identify and metadata types.
10. Go to the lib/config/project_enviornment_config.json file. Replace EXISTING_GLUE_DB_NAME_PLACEHOLDER with the prevailing AWS Glue database identify in the identical AWS account the place you might be deploying the Amazon DataZone core parts with the AWS CDK. Ensure you have a minimum of one current AWS Glue desk on this AWS Glue database to publish as a knowledge supply inside Amazon DataZone. Substitute DATA_SOURCE_NAME_PLACEHOLDER and DATA_SOURCE_DESCRIPTION_PLACEHOLDER together with your alternative of Amazon DataZone knowledge supply identify and outline. An instance of a cron schedule has been supplied (see the next code snippet). That is the schedule on your knowledge supply run; you may maintain the identical or replace it.

    "Schedule":{
       "schedule":"cron(0 7 * * ? *)"
    }

Subsequent, you replace the belief coverage of the AWS CDK deployment IAM position to deploy a customized useful resource module.

11. On the IAM console, replace the belief coverage of the IAM position on your AWS CDK deployment that begins with cdk-hnb659fds-cfn-exec-role- by including the next permissions. Substitute ${ACCOUNT_ID} and ${REGION} together with your particular AWS account and Area.

         {
             "Impact": "Permit",
             "Principal": {
                 "Service": "lambda.amazonaws.com"
             },
             "Motion": "sts:AssumeRole",
             "Situation": {
                 "ArnLike": {
                     "aws:SourceArn": [
                         
                         "arn:aws:lambda:${REGION}:{ACCOUNT_ID}:function:DataZonePreqStack-GlossaryLambda*",
                         "arn:aws:lambda:${REGION}:{ACCOUNT_ID}:function:DataZonePreqStack-GlossaryTermLambda*",
                         "arn:aws:lambda:${REGION}:{ACCOUNT_ID}:function:DataZonePreqStack-FormLambda*"
                     ]
                 }
             }
         }

Now you may configure knowledge lake directors in Lake Formation.

12. On the Lake Formation console, select Administrative roles and duties within the navigation pane.
13. Beneath Knowledge lake directors, select Add and add the IAM position for AWS CDK deployment that begins with cdk-hnb659fds-cfn-exec-role- as an administrator.

This IAM position wants permissions in Lake Formation to create sources, comparable to an AWS Glue database. With out these permissions, the AWS CDK stack deployment will fail.

14. Deploy the answer:

    $ npm run cdk deploy --all

15. Throughout deployment, enter y if you wish to deploy the modifications for some stacks if you see the immediate Do you want to deploy these modifications (y/n)?.
16. After the deployment is full, sign up to your AWS account and navigate to the AWS CloudFormation console to confirm that the infrastructure deployed.

It is best to see an inventory of the deployed CloudFormation stacks, as proven within the following screenshot.

17. Open the Amazon DataZone console in your AWS account and open your area.
18. Open the knowledge portal URL accessible within the Abstract part.
    
19. Discover your undertaking within the knowledge portal and run the knowledge supply job.

It is a one-time exercise if you wish to publish and search the info supply instantly inside Amazon DataZone. In any other case, await the info supply runs in keeping with the cron schedule talked about within the previous steps.

Troubleshooting

If you happen to get the message "Area identify already exists below this account, please use one other one (Service: DataZone, Standing Code: 409, Request ID: 2d054cb0-0 fb7-466f-ae04-c53ff3c57c9a)" (RequestToken: 85ab4aa7-9e22-c7e6-8f00-80b5871e4bf7, HandlerErrorCode: AlreadyExists), change the area identify below lib/constants.ts and attempt to deploy once more.

If you happen to get the message "Useful resource of kind 'AWS::IAM::Function' with identifier 'CustomResourceProviderRole1' already exists." (RequestToken: 17a6384e-7b0f-03b3 -1161-198fb044464d, HandlerErrorCode: AlreadyExists), this implies you’re unintentionally making an attempt to deploy every thing in the identical account however a unique Area. Be certain to make use of the Area you configured in your preliminary deployment. For the sake of simplicity, the DataZonePreReqStack is in a single Area in the identical account.

If you happen to get the message “Unmanaged asset” Warning within the knowledge asset in your datazone undertaking, you could explicitly present Amazon DataZone with Lake Formation permissions to entry tables on this exterior AWS Glue database. For directions, check with Configure Lake Formation permissions for Amazon DataZone.

Clear up

To keep away from incurring future expenses, delete the sources. If in case you have already shared the info supply utilizing Amazon DataZone, then you must take away these manually first within the Amazon DataZone knowledge portal as a result of the AWS CDK isn’t in a position to robotically try this.

1. Unpublish the info throughout the Amazon DataZone knowledge portal.
2. Delete the info asset from the Amazon DataZone knowledge portal.
3. From the basis of your repository folder, run the next command:

   $ npm run cdk destroy --all

4. Delete the Amazon DataZone created databases in AWS Glue. Confer with the tricks to troubleshoot Lake Formation permission errors in AWS Glue if wanted.
   
5. Take away the created IAM roles from Lake Formation administrative roles and duties.
   

Conclusion

Amazon DataZone affords a complete answer for implementing a knowledge mesh structure, enabling organizations to deal with superior knowledge governance challenges successfully. Utilizing the AWS CDK for IaC streamlines the deployment and administration of Amazon DataZone sources, selling consistency, reproducibility, and automation. This strategy enhances knowledge group and sharing throughout your group.

Able to streamline your knowledge governance? Dive deeper into Amazon DataZone by visiting the Amazon DataZone Consumer Information. To study extra in regards to the AWS CDK, discover the AWS CDK Developer Information.

In regards to the Authors

Bandana Das is a Senior Knowledge Architect at Amazon Internet Providers and makes a speciality of knowledge and analytics. She builds event-driven knowledge architectures to assist clients in knowledge administration and data-driven decision-making. She can be captivated with enabling clients on their knowledge administration journey to the cloud.

Gezim Musliaj is a Senior DevOps Advisor with AWS Skilled Providers. He’s eager about numerous issues CI/CD, knowledge, and their software within the subject of IoT, large knowledge ingestion, and just lately MLOps and GenAI.

Sameer Ranjha is a Software program Growth Engineer on the Amazon DataZone workforce. He works within the area of contemporary knowledge architectures and software program engineering, creating scalable and environment friendly options.

Sindi Cali is an Affiliate Advisor with AWS Skilled Providers. She helps clients in constructing data-driven functions in AWS.

Bhaskar Singh is a Software program Growth Engineer on the Amazon DataZone workforce. He has contributed to implementing AWS CloudFormation assist for Amazon DataZone. He’s captivated with distributed techniques and devoted to fixing clients’ issues.