Generative AI

The Generative AI module allows for the integration of a large language model (LLM) with any data source so that the bot can independently generate responses to questions it is asked, taking into account the user’s context or the offered products.

The Generative AI module consists of three elements:

• knowledge base,
• functions,
• prompts.

Knowledge base

The knowledge base is a panel that allows adding various data sources, which are then taken into account when generating a query to the LLM.

The knowledge base can be based on:

• text documents (e.g., PDF or .docx),
• web pages.

A text file should not contain images.

Adding a data source

To add a data source to the knowledge base, click the “Create” button located right next to the header. A panel will appear that allows you to import content from a text document or a webpage.

Remember to publish changes to production after importing content so that the document is saved in the knowledge base.

Source name

The name of the data source should clearly describe it and be understandable to other platform users. It can be up to 120 characters long.

Importing content from a PDF File

Importing content from a PDF file requires uploading the appropriate document from which the content is to be extracted. The maximum file size is 10 MB.

Importing content from a webpage

Importing content from a webpage requires providing a URL and an HTML selector from which the content is to be extracted. If the entered selector is “body,” the entire content from the given URL will be extracted.

When providing the URL, ensure it starts with http:// or https://.

Imported webpage content will be divided into fragments (so-called chunks) containing up to 1,000 characters. This is intended to facilitate searching for relevant fragments within the content so that the response is generated only based on them, resulting in greater precision. If you want to change this division, use the advanced settings or the “Split Document” option located above the top left corner of the imported content.

Advanced Settings

Advanced settings allow you to choose how the document should be divided. The available options are:

• “Fetch the entire content and split” – fetches content from the webpage and splits it into fragments of a specified size.
• “Each DOM element is a separate fragment” – fetches content from the webpage and splits it into fragments according to DOM elements; such fragments have no character limit.

Chunking

The minimum allowable fragment size is 500 characters, and the maximum is 1,500.

The maximum fragment size affects the number of fragments extracted from a document. If you want to obtain fragments containing 1,000 characters and the document itself is 3,000 characters long, you’ll get 3 fragments upon splitting. If you want to split the same document into fragments containing 500 characters each, you’ll get 6 fragments. A smaller fragment size ensures:

• a narrower context of the response,
• a faster response,
• fewer tokens used,
• a smaller risk of less accurate responses.

Metadata

After importing the content, describe the file using metadata. Metadata consists of a key and an assigned value. They are then used to search for relevant content in the knowledge base – based on the assigned key, the system will recognize which documents it can consult in response to a user’s question. The more metadata, the more accurate the file selection for further analysis.

Example Use of Metadata:

• Key: “Article Category,” Value: “General Questions”
• Key: “Insurance,” Value: “PZU”
• Key: “Computer Model,” Value: “Macbook Pro”

The metadata limit per document is 40kB (approximately 40,000 characters).

Testing the Knowledge Base

You can test the functionality of the knowledge base by clicking the “Test” button located to the right of the header.

During tests, you can provide a keyword for which you want to find a relevant fragment. You can modify the number of returned results – the minimum value is 1, and the maximum is 30. Additionally, you can narrow the search by specifying conditions based on metadata.

Functions

Functions are used to define specific actions, or operations for the LLM. An example of such an operation is an exact search in the knowledge base or retrieving appropriate values from a database.

Functions currently only work for the following language models:

• OpenAI
• Azure OpenAI
• Claude

Adding a function

To add a function, click the “Add Function” button.

Function name and description

The function name should be short, simple, and understandable to other platform users. You can provide detailed information about the function’s operation in its description.

Active function and fallback function

An active function is a useful setting that allows keeping the function configuration but disabling it depending on needs or bot test results.

The fallback function is a setting responsible for invoking the model when a bot query is outside the knowledge base’s scope. Additionally, in the “LLM Query” statistical event, an out_of_scope parameter is added so that such a query can be easily found and used to train the bot or expand its knowledge base.

Parameters

To retain the context of previous conversations and user data, define appropriate parameters in the function.

The parameter name will also be the variable name, so try to use a short, specific name. More details can be provided in the parameter description.

A parameter can take one of the following types:

• text,
• number,
• yes/no,
• array.

In the “Allowed Values” field, you can specify what values the parameter can accept.

You can also specify whether a parameter is required or not.

If you decide to use required parameters, make sure to include that information in the prompt. The mentioned language models tend to execute functions regardless of whether all required parameters have been provided.

Sources

You can add data sources to the message generated by the LLM. The end user will then see the information on which the response to their question was based. Sources can operate in two modes: static and based on user variables.

In static mode, the source – header, URL, and description – is added rigidly. This means that every user will see the same content for a given function. In user variable mode, you can set all these parameters as variables, along with an additional user variable being a list of objects.

Execution in progress

Sometimes, executing some functions takes more time. To enhance the user experience, you can show them the function’s progress. Simply enable the appropriate option during function configuration and add appropriate content describing which action is currently being performed. This option is especially useful when generating a good, context-based response requires using several functions in a row.

If your bot operates in different languages, you can add appropriate language versions for each description of function execution progress.

Action after function invocation

In the “When the model invokes a function” section, you can specify what should happen after the model invokes a function. This can include adding appropriate content to the prompt with additional instructions or a JSON object generated by the language model, but it can also involve performing a specific action, such as:

• saving data for the user,
• querying the knowledge base,
• fetching external data using an API request,
• sending an email,
• recording a statistical event.

It is also possible to set redirection to a specified block in the bot’s structure.

Prompts

The Prompts module allows users to create a prompt database that can later be used in Ticket Automation without the need to write them from scratch or copy them from another rule.

Adding prompts

To create a prompt, click the “Add Prompt” button. After clicking, the prompt settings screen will appear.

Prompt Name

The prompt name should briefly describe the purpose for which the prompt was created. It should also be simple and understandable to other platform users.

Prompt

Here, enter your prompt, which is an instruction for the LLM. Here are some tips on how to prepare a prompt to achieve the best results:

• Start with a short and simple instruction. This is a good basis for further iterations.
• Clearly describe the task to be performed and the desired result.
• Avoid ambiguous descriptions and instructions.
• When creating instructions, focus on what the model should do instead of what it shouldn’t do.
• Review the result and modify the prompt as needed.

Language Model

Choose which of the large language models should be used to generate the response. The newer the version of a language model, the better the results you will get.

Available language models are:

• OpenAI – GPT,
• Google Gemini,
• Claude.

Maximum Number of Tokens

A token is the basic unit of text that the model processes. Tokens can represent various parts of text, such as:

• whole words – in languages where words are short and do not have many special characters, one token can represent a whole word,
• parts of words – in languages where words are long or complex, a single word may be divided into several tokens,
• punctuation marks – punctuation marks (e.g., commas, periods) can be represented as separate tokens,
• symbols – symbols and special characters (e.g., $, #) can also be separate tokens.

The maximum number of tokens specifies the length of the response to be generated by the LLM. It can be no more than 1,000.

Temperature

Temperature affects the diversity of responses generated in OpenAI models. A higher temperature (e.g., 0.8) makes the responses more creative and less predictable, while a lower temperature (e.g., 0.2) leads to more predictable responses.

The temperature range is 0-2.

To minimize the possibility of the language model generating hallucinations, we recommend using a temperature value of no more than 0.5.

Managing prompts

To edit a prompt, click on its name in the list of prompts.

To delete a prompt, click on the settings icon and select the appropriate option from the menu. You can only delete a prompt that is not being used in any bot configuration or ticket automation.