LLM Integration

To use LLM and Generative AI features, you must configure the integration with a pre-built or custom LLM or Kore.ai XO GPT Module.

Review the model and feature support matrix for the Co-Pilot features and Dynamic Conversation features.

Pre-built LLM Integration

The XO Platform offers seamless integration with leading AI services like Azure OpenAI, OpenAI, and Anthropic. Utilizing pre-configured prompt templates and APIs, you can effortlessly tap into the core capabilities of these services. Although you can customize certain elements, the overall structure adheres to a standardized format for simplicity. You can quickly realize the potential of LLM with the XO platform’s plug-and-play integration with premium AI services. Along with pre-configured or default prompts, you can craft your own custom prompt optimized for their specific purposes.

Configure Pre-built LLM Integration

Steps to configure a pre-built LLM:

1. Go to Generative AI Tools > Models Library.
   

2. Click Configure Now for Azure Open AI, Open AI, or Anthropic. If you have already configured a model, click + New Model and select the LLM.

3. On the Authorization tab:

   • For Azure OpenAI, enter details like the API Key and User Sub Domain. In the models section, toggle on the required model and enter Deployment ID.
     
   • For OpenAI and Anthropic, enter the API Key.
     

4. Read the Policy Guidelines, select the checkbox, and click Save.

5. The Connection Status pop-up is displayed.

   • If a connection is successful, click Save.
     
   • If a connection fails, hover over the warning icon to view the error. Click Cancel to go back to the previous step and update the details.
     

   Note

   You can save the configuration with an unsuccessful connection.

6. The configured model is listed in the Models Library. The next step is to enable Co-Pilot and Dynamic Conversations features.

Recently Released LLM Integration

The XO Platform enables you to integrate newly launched language models into the out-of-box integration framework (OpenAI, Azure OpenAI, and Anthropic) with the required authentication. This ensures that newly launched models are immediately available without waiting for the platform to add them. Initially, these models do not include a pre-built/system prompt template, but you must create custom prompts for immediate use. Over time, the platform adds system prompts and templates while maintaining security and ensuring seamless adoption.

Key Benefits

• Immediate Access to New Models: Quickly access and utilize newly launched language models without waiting for platform-specific prompt development.
• Custom Prompt Support: Create custom prompts for new models, ensuring immediate usability while the platform gradually develops system prompts over time.
• Security & Compliance: Maintain platform security and authentication standards while eliminating integration delays.

Configure Recently Released LLM Integration

Steps to configure a recently released LLM Integration:

1. Go to Generative AI Tools > Models Library.
2. Click Configure Now for the Azure Open AI or Open AI or Anthropic. For example, Azure OpenAI.
3. On the Authorization tab, enter the details like API Key and the User Sub Doman.

4. In the Models section, click + Add. Enter the Model ID, Model Display Name, model Description, and Deployment ID. The Model ID is assigned by the LLM provider, while the Model Display Name is user-defined and used across the platform after integration is enabled.

5. Click Add Model. The current model is saved with a custom tag in the available models list. Toggle on the model. If you want to add more models, perform step 4.
   

6. Read the Policy Guidelines, select the checkbox, and click Save.

7. The Connection Status pop-up is displayed.

   • If a connection is successful, click Save.
     
   • If a connection fails, hover over the warning icon to view the error. Click Cancel to go back to the previous step and update the details.
     

   Note

   You can save the configuration with an unsuccessful connection.

8. The configured model is listed in the Models Library. The next step is to add Prompts.

Custom LLM Integration Framework

The XO Platform now enables enterprises to power up their virtual assistants with any Large Language Model (LLM) of their preference. The bring-your-own (BYO) model framework supports integrations with externally hosted models by third parties as well as models hosted by the enterprises themselves. It allows the creation of custom prompts that are optimized for their specific purposes and models. This generic framework works seamlessly with the Auth Profiles module of the platform, enabling enterprises to use the authentication mechanism of their choice.

Note

The Kore.ai XO Platform offers generative AI features for English and non-English NLU and VA languages.

Configure Custom LLM Integration

You can integrate a custom LLM and build your Prompts and Requests Library for specific features.

Steps to Integrate a Custom LLM:

1. Go to Generative AI Tools > Models Library.
2. Click Custom LLM. If you have already configured a model, click + New Model and select the Custom LLM.

3. On the Configuration tab, enter the details like Integration Name, Model Name, Endpoint, and Headers.
   

4. On the Auth tab, select an existing authorization profile from the drop-down or create a new profile to use for this request. For more information, see the Bot Authorization Overview article.

   Note

   OAuthv2.0 and Kerberos SPNEGO auth profiles are supported for the Custom LLM integration.

5. Read the Policy Guidelines, select the checkbox, and click Next.

6. On the request prompt pop-up, enter the test payload and click Save to check the connection. You can also save the test payload using the check box.
   

7. The Connection Status pop-up is displayed.

   • If a connection is successful, click Save.
     
   • If a connection fails, hover over the warning icon to view the error. Click Cancel to go back to the previous step and update the details.
     

   Note

   You can save the configuration with an unsuccessful connection.

8. The configured model is listed in the Models Library. The next step is to add Prompts.

Amazon Bedrock LLM Integration Framework

XO Platform now offers Amazon Bedrock as an out-of-box (OOB) integration. This integration lets platform users access Amazon Bedrock's models directly from the XO Platform. The users can create custom prompts for their specific use cases and use the connected models across Co-Pilot and Dynamic Conversations features. While Amazon Bedrock is available as an OOB integration, XO Platform does not provide any system prompts or templates. Users can only use the model with the help of custom prompts.

Note

Before starting the integration process, ensure you have the necessary permissions and access to the IAM role and Amazon Bedrock resources in your AWS account. For more information, see Policies and Permissions in AWS Identity and Access Management.

Configure Amazon Bedrock LLM Integration

Steps to integrate an Amazon Bedrock:

1. Go to Generative AI Tools > Models Library.
2. Click Amazon Bedrock.

3. On the Authorization tab, provide the following details.

   • Provider Name: Enter a name to identify the provider or group of language models you want to connect.
   • Model Name: Give a unique name to the language model you’re connecting with.
   • Identity Access Management (IAM) Role ARN: The IAM Role ARN enables the platform to securely access resources on behalf of users without the need for long-term access keys, supporting cross-account access, least privilege principles, and enhanced auditing capabilities.
   • Amazon STS API:
     • To assume an IAM role and obtain temporary security credentials, provide the AWS STS (Security Token Service) API endpoint, specifically the AssumeRole action. The STS API is essential for generating the temporary access key, secret key, and session token required to securely access AWS resources based on the permissions associated with the IAM role.
     • Ensure you have the correct endpoint URL for the AssumeRole action in the AWS region where your IAM role is located.
   • Amazon Resource Name (ARN): The Amazon Bedrock ARN that grants your IAM role access to the specific Language Model in Amazon Bedrock. The ARN uniquely identifies the Language Model resource within the Amazon Bedrock service.
   • Endpoint: Enter the URL to interact with the Language Model’s API.
   • Headers: If needed, include additional headers with metadata specific to your Language Model integration. These headers provide extra information or configuration options to the Language Model API.
     

4. Read the Policy Guidelines, select the checkbox, and click Next.

5. Enter the test payload and then click Test to check the connection. If the LLM connection is successful during the test call, it displays a confirmation message. If not, it shows an error message.
6. On a successful connection, the save option is enabled.
7. Click Save. The configuration successful message is displayed.
8. The next step is to add Prompts.

LLM Integration using Dynamic Variables

The platform now enables seamless configuration of custom and pre-built LLMs, including newly launched ones, by allowing you to use content, context, and environment variables, including secure ones. Secure Variables, particularly for environment settings, function just as smoothly as the current system ensures.

Ensure all necessary variables are set up before configuring the LLM. For more details, see Bot Variables.

Note

Co-Pilot features can use content and environment variables. Dynamic Conversations features can use content, context, and environment variables.

Key Advantages

Avoid Misuse of API Keys: The system minimizes the risk of unauthorized access by utilizing secure environment variables for API keys. Properly configured permissions restrict visibility and usage to authorized applications or personnel only.

Track Team-Wise Consumption: Implementing tracking mechanisms for API key usage allows teams to monitor their consumption, enabling better resource allocation and identifying potential inefficiencies.

Rotate Your API Keys: Regularly rotating API keys enhances security by limiting the risk associated with compromised keys. A well-defined rotation schedule ensures that even if a key is leaked, it becomes useless after a predetermined period.

Configure Pre-built LLM Integration using Dynamic Variables

Note

• For Azure OpenAI, you can set the API Key, Sub-Domain, and Model Deployment IDs as dynamic variables.
• For OpenAI and Anthropic, only the API Key can be configured as a dynamic variable.

Steps to configure a pre-built LLM using Dynamic Variables:

1. Go to Generative AI Tools > Models Library.
2. Choose the LLM you want to configure. For example, Azure OpenAI.
3. On the Authorization tab, enter the variable for API Key and the User Sub Doman. Toggle the required model and enter the variable for Deployment ID.
   
4. Read the Policy Guidelines, select the checkbox, and click Next.
5. Enter the sample values for the API Key, User Sub Domain, and Deployment ID on the test connection pop-up. You can also save the sample values using the check box. Click Save.
   

6. The Connection Status pop-up appears.

   • If a connection is successful, click Save.
     

   • If a connection fails, hover over the warning icon to view the error. Click Cancel to go back to the previous step and update the details.
     

     Note

     You can save the configuration with an unsuccessful connection.

7. The configured model is listed in the Models Library. The next step is to enable Co-Pilot and Dynamic Conversations features.

Configure Recently Released LLM Integration using Dynamic Variables

Note

• For Azure OpenAI, you can set the API Key, Sub-Domain, and Model Deployment IDs as dynamic variables.
• For OpenAI and Anthropic, only the API Key can be configured as a dynamic variable.

Steps to configure a recently released LLM using Dynamic Variables:

1. Go to Generative AI Tools > Models Library.
2. Choose the LLM you want to configure. For example, Azure OpenAI.
3. On the Authorization tab, enter the variable for API Key and the User Sub Doman.
4. In the Models section, click + Add. Enter the Model ID, Model Display Name, model Description, and variable for Deployment ID. The Model ID is assigned by the LLM provider, while the Model Display Name is user-defined and used across the platform after integration is enabled.
   
5. Click Add Model. The current model is saved to the available models list with a custom tag. Toggle on the model. If you want to add more models, perform step 4.
   
6. Read the Policy Guidelines, select the checkbox, and click Next.
7. Enter the sample values for the API Key, User Sub Domain, and Deployment ID on the test connection pop-up. You can also save the sample values using the check box. Click Save.
   

8. The Connection Status pop-up appears.

   • If a connection is successful, click Save.
     

   • If a connection fails, hover over the warning icon to view the error. Click Cancel to go back to the previous step and update the details.
     

     Note

     You can save the configuration with an unsuccessful connection.

9. The configured model is listed in the Models Library. The next step is to add Prompts.

Configure Custom LLM Integration using Dynamic Variables

Note

For Custom LLM, you can configure Endpoint, Authorization, and Headers fields as dynamic variables.

Steps to configure a Custom LLM using Dynamic Variables:

1. Go to Generative AI Tools > Models Library.
2. Click Custom LLM.
3. On the Configuration tab, enter the details like Integration Name and Model Name. Enter the variable for the Endpoint and Headers.
   

4. Read the Policy Guidelines, select the checkbox, and click Next.

5. Enter the sample values for the endpoints and header values. Enter the test payload and then click Test to check the connection. You can also save the sample values and test payload using the check box.
   

6. The Testing Connection pop-up appears.

   • If a connection is successful, click Save.
     

   • If a connection fails, hover over the warning icon to view the error.Click Cancel to go back to the previous step and update the details.
     

     Note

     You can save the configuration with an unsuccessful connection.

7. The configured model is listed in the Models Library. The next step is to add Prompts.

Configure Amazon Bedrock LLM Integration using Dynamic Variables

Note

For Amazon Bedrock integration, you can configure IAM Role ARN, Amazon STS API, Amazon Resource Name (ARN), Endpoint, and Headers fields as dynamic variables.

Steps to configure an Amazon Bedrock LLM using Dynamic Variables:

1. Go to Generative AI Tools > Models Library.
2. Click Amazon Bedrock.
3. On the Authorization tab, enter details like Integration Name and Model Name. Then, enter the variable for AM Role ARN, Amazon STS API, Amazon Resource Name (ARN), Endpoint, and Headers (optional).
   
4. Read the Policy Guidelines, select the checkbox, and click Next.
5. On the Testing tab, enter the sample values for the variables. Enter the test payload and then click Test to check the connection. If the LLM connection is successful during the test call, it displays a confirmation message. If not, it shows an error message.
   
6. On a successful connection, the save option is enabled.
7. Click Save. The configuration successful message is displayed.
8. The next step is to add Prompts.

Kore.ai XO GPT Integration

The new Kore.ai XO GPT Models module provides fine-tuned large language models optimized for enterprise conversational AI applications. These models have been evaluated and fine-tuned to be accurate, safe, and efficient for production deployment. Current capabilities include Conversation Summarization, User Query Rephrasing, Vector Generation and Answer Generation.

Enable Kore.ai XO GPT Integration

Steps to enable Kore.ai XO GPT Models.

1. Go to Generative AI Tools > Models Library.

2. Click Enable Now for Kore.ai XO GPT.

   If you have already configured a model, click + New Model > Kore.ai XO GPT.

3. On the Models tab, toggle on the models as required.

4. Read the Policy Guidelines, select the checkbox, and click Save.

5. The success confirmation message is displayed on the screen. The configured model is listed in the LLM Integration section. The status changes to X/Y models configured, where X is the number of the model(s) configured, and Y is the number of model(s) supported.

6. You can now use these models in Co-Pilot and Dynamic Conversations features.

Reset/Delete the Configured LLM Integration

If not using a configured LLM, you can reset/delete the LLM-specific integration using the Reset Configuration/Delete option. You can reset the pre-built integration and delete the custom integrations.

When you reset/delete the integration, the system does the following:

• Removes all the integration details like keys, endpoints, deployment names, etc.
• The system removes the model from the selection list of supported LLM features and turns off the respective feature. You can select another configured and supported LLM for that feature.
• Deletes the related Prompts and Responses.

Note

This change impacts only the in-development copy of the VA. The changes will apply to the published version when you later publish the VA with NLP configurations.

Reset Pre-build LLM IntegrationDelete Custom LLM IntegrationReset Kore.ai XO GPT Integration

Steps to reset an integration:

1. Go to Generative AI Tools > Models Library.

2. Click three dots (more icons) and then click the Reset Configuration for the model you want to reset.

3. Click Reset in the confirmation dialog.
   

4. The success message is displayed.

Steps to delete an integration:

1. Go to Generative AI Tools > Models Library.

2. Click three dots (more icons) and then click Delete for the integration you want to delete.

3. Click Delete in the confirmation dialog.
   

4. The success message is displayed.

Steps to reset an integration:

1. Go to Generative AI Tools > Models Library.

2. Click three dots (more icons) for Kore.ai XO GPT and then click the Reset Configuration for the integration you want to reset.

3. Click Reset in the confirmation dialog.
   

4. The success message is displayed.

---