This documentation describes the integration of MindsDB with Anyscale Endpoints, a fast and scalable API to integrate OSS LLMs into apps.
The integration allows for the deployment of Anyscale Endpoints models within MindsDB, providing the models with access to data from various data sources.
To use Anyscale Endpoints within MindsDB, install the required dependencies following this instruction.
Obtain the Anyscale Endpoints API key required to deploy and use Anyscale Endpoints models within MindsDB. Follow the instructions for obtaining the API key.
Create a model using anyscale_endpoints_engine as an engine.
Copy
Ask AI
CREATE MODEL anyscale_endpoints_model[FROM integration (SELECT * FROM table)]PREDICT target_columnUSING engine = 'anyscale_endpoints_engine', -- engine name as created via CREATE ML_ENGINE model_name = 'model-name', -- choose one of available models prompt_teplate = 'prompt-to-the-model'; -- prompt message to be completed by the model
The implementation is based on the engine for the OpenAI API, as Anyscale conforms to it. There are a few notable differences, though:
Please check both lists regularly, as they are subject to change. If you try to fine-tune a model that is not supported, you will get a warning and subsequently an error from the Anyscale endpoint.
This integration only offers chat-based text completion models, either for normal text or specialized for code.
When providing a description, this integration returns the respective HuggingFace model card.
Fine-tuning requires that your dataset complies with the chat format. That is, each row should contain a context and a role. The context is the text that is the message in the chat, and the role is who authored it (system, user, or assistant, where the last one is the model). For more information, please check the fine tuning guide in the Anyscale Endpoints docs.
The base URL for this API is https://api.endpoints.anyscale.com/v1.
The following usage examples utilize anyscale_endpoints_engine to create a model with the CREATE MODEL statement.Classify text sentiment using the Mistral 7B model.
Copy
Ask AI
CREATE MODEL anyscale_endpoints_modelPREDICT sentimentUSING engine = 'anyscale_endpoints_engine', model_name = 'mistralai/Mistral-7B-Instruct-v0.1', prompt_template = 'Classify the sentiment of the following text as one of `positive`, `neutral` or `negative`: {{text}}';
Query the model to get predictions.
Copy
Ask AI
SELECT text, sentimentFROM anyscale_endpoints_modelWHERE text = 'I love machine learning!';
Here is the output:
Copy
Ask AI
+--------------------------+-----------+| text | sentiment |+--------------------------+-----------+| I love machine learning! | positive |+--------------------------+-----------+
Next StepsFollow this tutorial to see more use case examples.