Recently, Arushi Raghuvanshi, Infinitus’ machine learning lead, presented the results of an Infinitus research project at the Association for Computational Linguistics annual meeting.

The resulting research paper, “Leveraging Explicit Procedural Instructions for Data-Efficient Action Prediction,” explored leveraging explicitly stored agent guidelines, such as company policies, customer service manuals, or documents on standard operating procedures, to improve large language models (LLMs) for action prediction in low-data settings. (You can read more about the research and what it means for the future of conversational AI in our blog post here.)

Below, Raghuvanshi dives into the results and explains how they can help enable our team to help more healthcare workers and patients by saving them time and reducing staff burnout. 

In addition to our day-to-day jobs, thought leaders at Infinitus are also conducting research in the AI space. Can you tell us a little about that work?

There has been a lot of attention on large language models (LLMs) recently and how incredible they are at capturing both real-sounding conversational language as well as general knowledge. 

At Infinitus, we are working on calls in the healthcare space where it’s important to be extremely accurate, which means there is no tolerance for “hallucinations” or other mistakes that sometimes happen with LLMs. This work is one of many projects we’re working on to see if and how we can incorporate LLMs to applied settings, taking advantage of their positive attributes and adding some guidelines or guardrails to prevent errors. 

You were recently invited to present the results of this project at ACL 2023 – can you tell us a little about what that research involved and what your findings entailed?

For this paper, we explore using explicit knowledge documents, which contain guidelines, together with a neural language model for action state tracking (AST). This action prediction task is how we determine what response to give in a phone call.

We can’t release internal data because it contains sensitive protected health information (PHI). So for ACL 2023, we also evaluated this approach on public datasets. This allows us to discuss results in more detail as well as more directly compare our model to other state-of-the-art (SOTA) approaches. We found that our model improved over SOTA approaches for the AST task on these datasets.

How do these results impact the work Infinitus is doing to automate tedious, often complex healthcare calls?

These results are promising, and in many cases building on top of LLMs can reduce the time it takes to automate more use cases for our digital assistant, enabling our team to help more healthcare workers and patients. Saving time in healthcare is particularly important in this time of rising healthcare costs and burnout among healthcare workers, and as always, improving efficiencies in healthcare is critical for enabling patients to get the care they need faster.

How do your findings impact the greater machine learning/continual learning space?

Our work has shown improvement over some state-of-the-art approaches like T5 for AST, not just in the healthcare domain, but on a broader set of datasets. We believe this approach can help improve accuracy in a wide variety of spaces/domains, especially in low-data industry settings. It allows us to inject explicit knowledge and guidelines to support LLMs, which can make them more robust, controllable, and easier to update as these guidelines evolve.

Is there a link where interested parties could read the full published paper?

Yes! The paper is available online here. 

Why is it important for the team at Infinitus to work on research projects like this?

It’s important for us to continually experiment with new research approaches to see if we can improve over our current approaches that are in production, whether that’s improving the automation process for NLP engineers, the conversational experience for the people we are calling, the accuracy of our results, or other aspects of our overall product.

Is Infinitus hiring engineers for its teams?

We are – and you can see a full list of Infinitus career opportunities here.