TwelveLabs video understanding fashions at the moment are obtainable in Amazon Bedrock

Earlier this yr, we preannounced that TwelveLabs video understanding fashions had been coming to Amazon Bedrock. At the moment, we’re saying the fashions at the moment are obtainable for looking by way of movies, classifying scenes, summarizing, and extracting insights with precision and reliability.

TwelveLabs has launched Marengo, a video embedding mannequin proficient at performing duties reminiscent of search and classification, and Pegasus, a video language mannequin that may generate textual content based mostly on video knowledge. These fashions are skilled on Amazon SageMaker HyperPod to ship groundbreaking video evaluation that gives textual content summaries, metadata era, and artistic optimization.

With the TwelveLabs fashions in Amazon Bedrock, you will discover particular moments utilizing pure language video search capabilities like “present me the primary landing of the sport” or “discover the scene the place the primary characters first meet” and immediately leap to these actual moments. You may also construct purposes to grasp video content material by producing descriptive textual content reminiscent of titles, matters, hashtags, summaries, chapters, or highlights for locating insights and connections with out requiring predefined labels or classes.

For instance, you will discover recurring themes in buyer suggestions or spot product utilization patterns that weren’t apparent earlier than. Whether or not you’ve a whole bunch or 1000’s of hours of video content material, now you can rework that complete library right into a searchable information useful resource whereas sustaining enterprise-grade safety and efficiency.

Let’s check out the Marengo and Pegasus movies that TwelveLabs has printed.

You possibly can rework video workflows with these fashions throughout industries. Media producers and editors can immediately find particular scenes or dialogue, which suggests you possibly can give attention to storytelling slightly than sifting by way of hours of footage. Advertising and marketing groups are streamlining their promoting workflows by rapidly personalizing content material to resonate with numerous audiences, whereas safety groups are utilizing the know-how to proactively establish potential dangers by recognizing patterns throughout a number of video feeds.

Getting began with TwelveLabs fashions in Amazon Bedrock
Earlier than getting began, in case you’re new to utilizing TwelveLabs fashions, go to the Amazon Bedrock console and select Mannequin entry within the backside left navigation pane. To entry the newest TwelveLabs fashions, request entry for Marengo Embed 2.7 or Pegasus 1.2 in TwelveLabs by obtainable Areas.

To make use of TwelveLabs fashions in Amazon Bedrock, select Chat/Textual content Playground underneath Check within the left navigation pane. Select Choose mannequin, choose TwelveLabs because the class and Pegasus because the mannequin, after which select Apply.

For looking or producing textual content out of your video, you must add your video to an Amazon Easy Storage Service (Amazon S3) bucket or enter Base64-formatted video string.

I’ll use a pattern video generated with Amazon Nova Reel with a immediate: “A cute raccoon taking part in guitar underwater.”

Enter the S3 URI of your uploaded video and the S3 Bucket Proprietor, after which run your immediate: “Inform me concerning the video by timeline”.

You may get following mannequin response:

The video opens with an underwater scene that includes a sandy seabed and swaying seaweed, making a serene environment. At round 5 seconds, a raccoon seems, swimming in the direction of the digicam with a curious expression. By 10 seconds, the raccoon is seen swimming close to the seabed. The main focus then shifts to a bunch of koi fish swimming gracefully close to the seabed from 11 to 13 seconds.

At 14 seconds, the scene transitions to an ukulele resting on the seabed, with the digicam zooming in to focus on its particulars. The raccoon reappears at 20 seconds, swimming among the many koi fish, and by 23 seconds, it begins to play the ukulele. The raccoon continues to play the ukulele from 29 to 41 seconds, with the koi fish swimming round it, including a energetic and colourful contact to the underwater setting.

From 42 to 49 seconds, the raccoon is proven taking part in the ukulele with the koi fish swimming round it. The video concludes with a display screen displaying "Video generated with Amazon Novo Reel 1.1" and "Accessible in Amazon Bedrock" from 50 to 54 seconds, adopted by the AWS brand from 55 to 57 seconds.

The TwelveLabs fashions will be simply built-in into your purposes utilizing the Amazon Bedrock Invoke API, which gives a unified interface for conversational AI interactions.

Right here’s an instance of learn how to use the AWS SDK for Python (Boto3) with the TwelveLabs Pegasus mannequin:

import boto3
import json
import base64

AWS_REGION = "us-west-2"
MODEL_ID = "us.twelvelabs.pegasus-1-2-v1:0" # OR "eu.twelvelabs.pegasus-1-2-v1:0" whether it is doing cross area inference in europe
VIDEO_PATH = "pattern.mp4"

def read_file(file_path: str) -> str:
    """Learn a file and return as base64 encoded string."""
    strive:
        with open(file_path, 'rb') as file:
            file_content = file.learn()
            return base64.b64encode(file_content).decode('utf-8')
    besides Exception as e:
        elevate Exception(f"Error studying file {file_path}: {str(e)}")


bedrock_runtime = boto3.consumer(
    service_name="bedrock-runtime",
    region_name=AWS_REGION
)

request_body = {
    "inputPrompt": "inform me concerning the video",
    "mediaSource": {
        "base64String": read_file(VIDEO_PATH)
    }
}

response = bedrock_runtime.invoke_model(
    modelId=MODEL_ID,
    physique=json.dumps(request_body),
    contentType="software/json",
    settle for="software/json"
)

response_body = json.masses(response['body'].learn())
print(json.dumps(response_body, indent=2))

The TwelveLabs Marengo Embed 2.7 mannequin generates vector embeddings from video, textual content, audio, or picture inputs. These embeddings can be utilized for similarity search, clustering, and different machine studying (ML) duties. The mannequin helps asynchronous inference by way of the Bedrock StartAsyncInvoke API.

For video supply, you possibly can request JSON format for the TwelveLabs Marengo Embed 2.7 mannequin utilizing the StartAsyncInvoke API.

{
    "modelId": "twelvelabs.marengo-embed-2-7-v1:0",
    "modelInput": {
        "inputType": "video",
        "mediaSource": {
            "s3Location": {
                "uri": "s3://your-video-object-s3-path",
                "bucketOwner": "your-video-object-s3-bucket-owner-account"
            }
        }
    },
    "outputDataConfig": {
        "s3OutputDataConfig": {
            "s3Uri": "s3://your-bucket-name"
        }
    }
}

You may get a response delivered to the desired S3 location.

{
    "embedding": [0.345, -0.678, 0.901, ...],
    "embeddingOption": "visual-text",
    "startSec": 0.0,
    "endSec": 5.0
}

That will help you get began, take a look at a broad vary of code examples for a number of use circumstances and a wide range of programming languages. To study extra, go to TwelveLabs Pegasus 1.2 and TwelveLabs Marengo Embed 2.7 within the AWS Documentation.

Now obtainable
TwelveLabs fashions are usually obtainable as we speak in Amazon Bedrock: the Marengo mannequin within the US East (N. Virginia), Europe (Eire), and Asia Pacific (Seoul) Area, and the Pegasus mannequin in US West (Oregon), and Europe (Eire) Area accessible with cross-Area inference from US and Europe Areas. Verify the full Area listing for future updates. To study extra, go to the TwelveLabs in Amazon Bedrock product web page and the Amazon Bedrock pricing web page.

Give TwelveLabs fashions a strive on the Amazon Bedrock console as we speak, and ship suggestions to AWS re:Submit for Amazon Bedrock or by way of your traditional AWS Help contacts.

— Channy

Up to date on July 16, 2025 – Revised the screenshots and code half.