A knowledge base is a batteries included RAG system that you can create and insert data into, as well as query as if it was a table.
Copy
Ask AI
-- Create a knowledge base using the OpenAI embedding model & Chroma world news indexCREATE KNOWLEDGE_BASE world_news_kb;-- Add data into the knowledge baseINSERT INTO world_news_kb (SELECT id, text as content FROM web_news);-- You can query the knowledge base as virtual tableSELECT * FROM world_news_kb WHERE content = "Best AI Startups in SF?" LIMIT 10;
Internally, it uses a vector store and an embedding model, by default it uses chroma and it will pick a suitable embedding model for the task, however, you can specify each of these if you want.Here is a general syntax for creating a knowledge base:
In this example, there is a table called world_news_with_ids that contains columns (id, content)
Copy
Ask AI
SELECT * FROM world_news_with_ids;
Now, we can create an index in the vector store, and insert one example point:
Copy
Ask AI
CREATE TABLE chroma_dev_local.world_news_index ( SELECT content, embeddings FROM embedding_model WHERE content = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tristique sollicitudin nibh sit amet commodo nulla. Risus sed vulputate odio ut enim blandit volutpat. Suspendisse ultrices gravida dictum fusce ut placerat orci. Eget nulla facilisi etiam dignissim diam. Aenean euismod elementum nisi quis eleifend quam. Ac placerat vestibulum lectus mauris ultrices eros in. Sed turpis tincidunt id aliquet risus feugiat in ante metus. Pellentesque habitant morbi tristique senectus et netus. Imperdiet massa tincidunt nunc pulvinar sapien et ligula. Leo in vitae turpis massa sed elementum tempus egestas. Aliquam malesuada bibendum arcu vitae elementum curabitur. Sit amet tellus cras adipiscing. Enim ut tellus elementum sagittis vitae et leo. Donec pretium vulputate sapien nec.");
Now, we can connect the embedding model and the vector index into one knowledge base:
Copy
Ask AI
-- Create a knowledge base using the OpenAI embedding model & Chroma world news indexCREATE KNOWLEDGE_BASE world_news_kbUSING model = embedding_model, storage = chroma_dev_local.world_news_index;
This is how we add content into the vector store every few seconds if there is new data:
Copy
Ask AI
-- Insert world news data into Knowledge Base every second, if there is new dataCREATE JOB keep_knowledge_base_up_to_date AS ( INSERT INTO world_news_kb ( SELECT id, text as content FROM world_news_with_ids WHERE id > LAST );) every second;
Note: the query has a id > LAST. This allows the job, to only select new data every time it runs.
-- Do a similarity search over the knowledge baseSELECT * FROM world_news_kb;SELECT * FROM world_news_kb WHERE content = "YouTube" LIMIT 1;SELECT * FROM world_news_kb WHERE content = "Canada and Google" LIMIT 1;