Augmentation

Almost by default now, LLMs are not used solely for their vast knowledge but rather the LLM generation is augmented with reference data acting as a contextual reference, injected at inference.

 This contextual reference data enables the in-context learning capability of LLMs.

The vast general knowledge of LLMs are almost solely used in end-user UI implementations like Chat-GPT and the like.

This begs the question, if chatbots rely on retrieval-augmentation, and a limited scope of LLM functionality, will SLMs not suffice? And by implementing a SLM, these five impediments listed below will be circumnavigated…

Companies are in the experimental phase rather than moving to production due to:

1️⃣ Inference Latency
2️⃣ Token Usage Cost
3️⃣ Model Drift
4️⃣ Data Privacy Concerns
5️⃣ LLM API Rate Limits

One can almost consider SLMs as next generation NLU engines.

Microsoft Phi-2

Phi-2 is a Small Language Model (SML) with 2.7 billion parameters. 

It was trained making use of the same data sources as Phi-1.5, augmented with a new data source that consists of various NLP synthetic texts and filtered websites for safety and educational value.

Considering common sense, language understanding and logical reasoning, Phi-2 showed close-to nearly state-of-the-art performance among models with less than 13 billion parameters.

Microsoft’s intention in crafting this open-source model is to provide the research community with a non-restricted small model to explore vital safety challenges, such as reducing toxicity, understanding societal biases, enhancing controllability, and more.

Yet, considering production conversational AI implementations, SLMs are a cost-effective alternative to Large Language Models and are also useful when they are being used for less demanding tasks, tasks which do not require the power of an LLM.

Phi-2 can be run locally or via a notebook for experimentation.

Below is the phi-2 model card at HuggingFace; you can interact directly with the model form here.

Running Phi-2 In A Notebook

Here is a simple example of running Phi-2 by make use of HuggingFace within a notebook. 

I had to change the runtime type to T4 GPU to have the SLM respond in a reasonable time.

Considering most chatbot development frameworks use LLMs to generate NLU training data, and specifically different variations of intent sentence examples, I asked Phi-2 the following question:

Give me three example sentences on how someone can ask to close their account.

Here is the complete code you can paste into a notebook…

!pip install -q -U transformers
!pip install -q -U accelerate

from transformers import pipeline

model_name = "microsoft/phi-2"

pipe = pipeline(
    "text-generation",
    model=model_name,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Give met three example sentences on how someone can ask to close their acuount."

outputs = pipe(
    prompt,
    max_new_tokens=300,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=0.95,
)
print(outputs)

And the output below…Phi-2 generated three example sentences, together with follow-up questions and answers. These follow-up questions and answers can act well to present chatbot users with quick-reply buttons in the dialog.

Or it can be used to create sub-intents or nested intents.

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.

[{'generated_text': 
'Give met three example sentences on how someone can ask to close their acuount.

Answer:
1. "Can I close my account now, please?"
2. "I would like to close my account, can you help me with that?"
3. "I need to close my account, can you guide me through the process?"

Exercise: 
What should you do if you are unsure about closing your account?
Answer: 
You should seek help from a trusted adult or contact the company directly to clarify any doubts or concerns.

Exercise: 
What are some important details to have ready when closing your account?
Answer: 
Some important details to have ready include the account number, any PIN codes, and the reason for closing the account.

Exercise: Why is it important to know how to close an account on a website?
Answer: Knowing how to close an account on a website is important because it allows you to take control of your online presence and protect your personal information. It also helps you manage your finances and account settings effectively.

Discussion:
Positive Side:
The knowledge of how to close an account on a website is extremely useful in today\'s digital age. 
With the increasing reliance on the internet for various purposes, it is important for individuals to have control over their online presence. Knowing how to close an account on a website allows individuals to take control of their personal information and protect themselves from potential risks such as identity theft or fraud. It also helps individuals manage their finances and'
}]

Below Phi-2 is presented with a dialog, which acts as a contextual reference for the conversation, and how to continue the conversation. This example dialog also serves well for in-context learning, hence allowing the SLM to continue the conversation.

prompt = """
Alice: I don't know why, I'm struggling to maintain focus while studying. 
Any suggestions?

Bob: Well, have you tried creating a study schedule and sticking to it?

Alice: Yes, I have, but it doesn't seem to help much.

Bob: Hmm, maybe you should try studying in a quiet environment, 
like the library.

Alice: 
."""

outputs = pipe(
    prompt,
    max_new_tokens=300,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=0.95,
)
print(outputs)

And again below the result, Phi-2 created a number of different conversation paths from the contextual reference. 

This illustrates how Phi-2 can be used for chit-chat/small talk, developing different dialog paths, or creating a mockup of a conversation.

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
[{'generated_text': "

Alice: I don't know why, I'm struggling to maintain focus while studying. 
Any suggestions?

Bob: Well, have you tried creating a study schedule and sticking to it?

Alice: Yes, I have, but it doesn't seem to help much.

Bob: Hmm, maybe you should try studying in a quiet environment, like the library.

Alice: 

...


Alice: I tried studying in the library, but the constant noise from other students was really distracting.

Bob: Oh, that's unfortunate. Maybe you could try using noise-canceling headphones?

Alice: I thought about that, but I don't have any with me right now.

Bob: In that case, have you considered going to a coffee shop? They usually have a quieter atmosphere.

Alice:
...

Alice: I went to a coffee shop to study, but the background music was too loud for me to concentrate.

Bob: I see. Maybe you could try finding a study group?

Alice: I thought about that, but I prefer studying alone.

Bob: Well, have you considered using a study app or website? They often have features to block out distractions.

Alice: ...

Alice: I tried using a study app, but the notifications kept popping up and interrupting my studying.\nBob: Oh, that's frustrating. Maybe you could try studying in a different location altogether?\nAlice: I thought about that, but I'm not sure where else to go.\nBob: Well, have you considered studying in a park or a quiet corner of your house?\nAlice: \n...\n\nAlice: I tried studying in a park, but the presence of other people and their activities was too distracting.\nBob: I understand."}]

In Conclusion

For now, it seems highly feasible to use NLU in conjunction with a SLM to underpin a chatbot development framework.

Running a SLM locally and using an augmented generation approach with in-context learning can solve for impediments like inference latency, token cost, model drift, data privacy, data governance and more.