Amazon OpenSearch Ingestion is a strong information ingestion pipeline that AWS prospects use for a lot of totally different functions, similar to observability, analytics, and zero-ETL search. Many purchasers at present push logs, traces, and metrics from their functions to OpenSearch Ingestion to retailer and analyze this information.
At the moment, we’re comfortable to announce that OpenSearch Ingestion pipelines now help cross-account ingestion for push-based sources similar to HTTP and OpenTelemetry (OTel). Organizations can now use this characteristic to effortlessly share information throughout groups. For instance, many organizations have central observability groups—now these groups can create OpenSearch Ingestion pipelines and share them with different groups of their group. You can even use this characteristic to ingest information into Amazon OpenSearch Service domains or Amazon OpenSearch Serverless collections in different accounts.
Beforehand, sharing OpenSearch Ingestion pipelines throughout accounts required groups to make use of digital non-public cloud (VPC) options to share entry. For instance, groups might use VPC peering, which isn’t all the time possible, or AWS Transit Gateway. The brand new cross-account ingestion options in OpenSearch Ingestion can simplify your deployment and scale back price for sharing pipelines.
Answer overview
Let’s take a look at the best way to share a pipeline from a central logging account with two different improvement accounts (A and B). The central logging account can create an OpenSearch Ingestion pipeline utilizing a push-based supply, for instance, HTTP. After creating the pipeline, a member of the central logging workforce can grant entry to the opposite groups. They will use a useful resource coverage that offers permissions to the 2 different workforce accounts to create pipeline endpoints. After making this alteration, the OpenSearch Ingestion pipeline is offered to be used by the opposite groups.
The next diagram illustrates this configuration.
Within the following sections, we exhibit the best way to implement this resolution.
Conditions
First, the central logging account will need to have a VPC with two choices enabled.
- enableDnsSupport should be set to true
- enableDnsHostnames should be set to true
The central logging account should additionally create a push-based OpenSearch Ingestion pipeline within the VPC. This could be a pipeline receiving logs from FluentBit or OpenTelemetry telemetry.
The event accounts which might be going to connect with the pipeline additionally will need to have VPCs in the identical area with the identical DNS choices enabled.
- enableDnsSupport should be set to true
- enableDnsHostnames should be set to true
Create useful resource coverage
Because the proprietor of the pipeline, you may create a useful resource coverage that enables the 2 improvement accounts to create pipeline endpoints in opposition to your pipeline.
The next is an instance useful resource coverage for this state of affairs:
The OpenSearch Ingestion console makes it simple to create these insurance policies, as proven within the following screenshot.
Create pipeline endpoint
Now that the central logging account has shared permissions on their pipeline, the event accounts can create pipeline endpoints. A pipeline endpoint is a connection from one VPC to an OpenSearch Ingestion pipeline.
The event accounts are answerable for creating the pipeline endpoints within the VPCs they need to join from. They create this within the subnets they want and supply a safety group. The safety group ought to have an inbound rule permitting entry port HTTPS over port 443 from any supply that the event accounts must ingest logs.
Growth workforce A can create a pipeline endpoint utilizing a command just like the next:
Growth workforce A may also use the OpenSearch Ingestion console to create the pipeline endpoint.
After performing this alteration, the VPC for improvement workforce A can have a pipeline endpoint. This pipeline endpoint now permits for ingesting information into the central logging pipeline. Now, Amazon Elastic Compute Cloud (Amazon EC2) situations, Amazon Elastic Container Service (Amazon ECS) duties, Kubernetes pods, and different compute working within the VPC can ingest their log information into the pipeline utilizing instruments similar to FluentBit.
On the identical time or at a later time, improvement workforce B can create a pipeline endpoint as properly. This workforce will create it for their very own VPC.
After this, the pipeline will now have two pipeline endpoints, so each groups can ingest their log information into the central logging VPC.
Clear up
After a pipeline endpoint is created, both account can take away it. The event groups in our state of affairs can use the DeletePipelineEndpoint API to delete it from their accounts. Moreover, if the central logging account must take away a pipeline endpoint from a pipeline, it may possibly use the RevokePipelineEndpointConnections API. Each choices can be found on the OpenSearch Ingestion console as properly.
After the pipeline endpoints are eliminated, the central logging workforce may also take away the pipeline in the event that they not want it.
Conclusion
The brand new pipeline endpoint characteristic for OpenSearch Ingestion simplifies how one can share pipelines for cross-account ingestion. This can assist groups use the highly effective options of OpenSearch Ingestion and open up new prospects for groups or organizations utilizing a number of accounts and VPCs. The brand new pipeline endpoint characteristic is offered at present in AWS Areas the place OpenSearch Ingestion is offered.
To get began with cross-account ingestion in OpenSearch Ingestion, check with OpenSearch Ingestion documentation or attempt creating your first cross-account pipeline on the OpenSearch Ingestion console.