TwelveLabs Large Video Model

In this section, we present how to connect Twelve Labs API to MindsDB.

Twelve Labs provides a powerful and seamless video search infrastructure for your application.

Prerequisites

Before proceeding, ensure the following prerequisites are met:

1. Install MindsDB on your system or obtain access to cloud options.
2. To use Twelve Labs with MindsDB, install the required dependencies following this instruction.
3. Copy your Twelve Labs API Key by navigating to the Dashboard page.

AI Engine

The first step to use this handler is to create an ML Engine. The required argument to create the engine is:

• twelve_labs_api_key - The TweleveLabs api key.

Once you have the API key you can establish a connection by executing the following SQL command:

CREATE ML_ENGINE twelve_labs_engine
FROM twelve_labs
USING
  twelve_labs_api_key = '<YOUR_API_KEY>';

AI Model

Now, you can use this ML Engine to create Models for the different tasks supported by the handler.

When executing the CREATE MODEL statement, the following parameters are supported in the USING clause of the query:

• engine: The name of the ML Engine to use. This is a required parameter.
• twelve_labs_api_key: The Twelve Labs API key to use for authentication, if the ML Engine is not provided.
• base_url: The base URL of the Twelve Labs API. This is an optional parameter and defaults to https://api.twelvelabs.io/v1.2.
• task: The task to perform. This is a required parameter and must be one of search or summarization.
• engine_id: The ID of the Twelve Labs engine to use. This is an optional parameter and defaults to marengo2.5. However, certain tasks may require a different engine ID; for instance, the summarization task runs only on the Pegasus family of engines. More information about the different engines can be found here.
• index_name: The name of the index to use; if it does not exist, it will be created. This is a required parameter. More information about indexes can be found here.
• index_options: A list of the types of information within the video that will be processed by the video understanding engine. This is a required parameter and can be any combination of visual, conversation, text_in_video and logo. More information about index options can be found here. Certain engines support only a subset of these options; for instance, the Pegasus family of engines only support the visual and conversation options. More information about the about these configurations can be found here.
• video_urls: A list of URLs to the videos to be indexed. This is an optional parameter, but if not specified, one of video_files, video_urls_column or video_files_column must be specified instead.
• video_files: A list of local paths to the videos to be indexed. This is an optional parameter, but if not specified, one of video_urls, video_urls_column or video_files_column must be specified instead.
• video_urls_column: The name of the column containing the URLs to the videos to be indexed. This is an optional parameter, but if not specified, one of video_urls, video_files or video_files_column must be specified instead.
• video_files_column: The name of the column containing the local paths to the videos to be indexed. This is an optional parameter, but if not specified, one of video_urls, video_files or video_urls_column must be specified instead.
• search_options: A list of the sources of information to use when performing a search. This parameter is required if the task is search and it should be a subset of index_options. More information about search options can be found here.
• search_query_column: The name of the column containing the search queries. This parameter is required if the task is search.
• summarization_type: The type of summarization to perform. This parameter is required if the task is summarization and it should be one of summary, chapter or highlight.
• prompt - Provide context for the summarization task, such as the target audience, style, tone of voice, and purpose. This is an optional parameter.

Given below are examples of creating Models for each of the supported tasks.

Search

CREATE MODEL mindsdb.twelve_labs_search
PREDICT search_results
USING
  engine = 'twelve_labs_engine',
  twelve_labs_api_key = '<YOUR_API_KEY>',
  task = 'search',
  index_name = 'index_1',
  index_options = ['visual', 'conversation', 'text_in_video', 'logo'],
  video_urls = ['https://.../video_1.mp4', 'https://.../video_2.mp4'],
  search_options = ['visual', 'conversation', 'text_in_video', 'logo'],
  query_column = 'query'

As mentioned above, the search_options parameter is specific to the search task and should be a subset of index_options.

Summarization

CREATE MODEL mindsdb.twelve_labs_summarization
PREDICT search_results
USING
  engine = 'twelve_labs_engine',
  task = 'summarization',
  engine_id = 'pegasus1',
  index_name = 'index_1',
  index_options = ['visual', 'conversation'],
  video_urls = ['https://.../video_1.mp4', 'https://.../video_2.mp4'],
  summarization_type = 'summary';

Making Predictions

Given below are examples of using Models created for each of the supported tasks.

Search

SELECT *
FROM mindsdb.twelve_labs_search
WHERE query = 'search query';

Here, the query column is the name of the column containing the search queries as specified in the query_column parameter of the CREATE MODEL statement.

Note: At the moment, only a single query can be specified in the WHERE clause of the query. The JOIN clause for making multiple predictions will be added in a future release.

Summarization

SELECT *
FROM mindsdb.twelve_labs_summarization
WHERE video_id = 'video_1';

Here, the video IDs that were indexed by a model can be found by running a DESCRIBE statement on the it. The URL or file path of the video will be available in the video_reference column. The following is an example of how to run such a DESCRIBE statement,

DESCRIBE mindsdb.twelve_labs_summarization.indexed_videos;

The response returned will look something like this,

video_id	created_at	updated_at	duration	engine_ids	filename	fps	height	size	video_reference	width
66c8425e35db9fa680cd4195	2024-02-23T03:39:10Z	2024-02-23T03:39:12Z	43.733333	pegasus1	test.mp4	30	1280	3737394	/path/to/Videos/test.mp4	720

Note: This will display all of the indexed videos that are contained within the index specified in the index_name parameter of the CREATE MODEL statement. If the same index is used for multiple models, the indexed_videos table will contain all of the videos indexed by all of the models that use that index.