In previous articles, I argued that a data-centric approach should be taken when engineering training data for Natural Language Understanding (NLU). Building on this, this article will discuss the importance of creating and using intents when working with Large Language Models (LLMs).
Introduction
As with AI in general, NLU Models also demand a data-centric approach to NLU Design. Improving NLU performance demands that the focus shift from the NLU model to the training data.
NLU Design best practice needs to be adhered to, where existing conversational unstructured data is converted into structured NLU training data.
NLU Design should ideally not make use of synthetic or generated data but actual customer conversations.
The image below shows the process which can be followed for a data centric approach to intent detection, creation and use. Staring with Embeddings…
Embeddings
The first step is to use conversational or user-utterance data for creating embeddings, essentially clusters of semantically similar sentences.
These groupings each constitutes an intent, each group needs to be given a label which is the “intent name”.
Read the article below for a detailed description of step one.
Cobus Greyling
Create Classifications
Once we have the groupings/clusters of training data we can start the process of creating classifications or intents. The terms “classes” and “intents” will be used interchangeably.
For the purposes of this article I’ll be making use of the Cohere LLM.
The classification can be done via the Cohere classify post endpoint:
https://api.cohere.ai/classify
The training body of text is classified into one of several classes/intents. The endpoint only needs a few examples to create a classifier leveraging a generative model.
The Colab notebook snippet below shows how to install the Cohere SDK, and how to create a client. You will need an API key which you can get for free by creating a login on the Cohere website.
For reasons of optimisation and speed, it is best to make use of a small model.
Once you have installed the SDK and created your Client, run this code to create the intents.
Each with training examples…from cohere.classify import Example
from cohere.classify import Example
examples = [
Example("Do you offer same day shipping?", "Shipping and handling policy"),
Example("Can you ship to Italy?", "Shipping and handling policy"),
Example("How long does shipping take?", "Shipping and handling policy"),
Example("Can I buy online and pick up in store?", "Shipping and handling policy"),
Example("What are your shipping options?", "Shipping and handling policy"),
Example("My order arrived damaged, can I get a refund?", "Start return or exchange"),
Example("You sent me the wrong item", "Start return or exchange"),
Example("I want to exchange my item for another colour", "Start return or exchange"),
Example("I ordered something and it wasn't what I expected. Can I return it?", "Start return or exchange"),
Example("What's your return policy?", "Start return or exchange"),
Example("Where's my package?", "Track order"),
Example("When will my order arrive?", "Track order"),
Example("What's my shipping number?", "Track order"),
Example("Which carrier is my package with?", "Track order"),
Example("Is my package delayed?", "Track order")
]You will see the training utterances are all labelled using one of three classes or intents:
Shipping and handling policy
Start return or exchange
Track order
Extract Classification
The text below shows the queries, analogous to user utterances submitted to the conversational agent…
inputs=[" Am I still able to return my order?",
"When can I expect my package?",
"Do you ship overseas?",
]As seen in the image,
(1) The input or query utterances can be submitted
(2) The Classifications are extracted
(3) the results printed
A portion from the Colab notebook showing steps 1, 2 and 3
The results:
:[
"Classification<prediction":"Start return or exchange", "confidence":0.9900205>,
"Classification<prediction":"Track order", "confidence":0.99867964>,
"Classification<prediction":"Shipping and handling policy", "confidence":0.9982495>]Conclusion
Intents are often neglected and seen as an insignificant step in the creation of a conversational agent. Frameworks like Amelia, Oracle Digital Assistant and Yellow AI offer synthetically generated training phrases. This approach can run the danger of trivialising the intent creation process.
Synthetic training data can suffice as a bootstrap measure, but will not serve well in creating a longer term sustainable solution. NLU Design and Data Best Practice should be adhered to from the onset.
Also, these synthetic training phrases are based on often “thought up” intents and intent names which are most probably not aligned with existing user intents. And definitely not addressing the long-tail of intent distribution.
It is time we start considering intents as classifications of existing customer conversations; a process of intent driven development is required for successful digital assistant deployments.
I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI and language. Including NLU design, evaluation & optimisation. Data-centric prompt tuning & LLM observability, evaluation & fine-tuning.
