Finding Covid-19 vaccines: automating calls for good

It all started when our CEO Ankit was tagged on a tweet:

 

At Infinitus, we automate repetitive, routine phone calls so our customers can focus on the more creative and empathetic parts of their businesses. Over the past two years, we have built a platform that collects data on behalf of tens of thousands of healthcare providers to help their patients access and afford therapies as efficiently as possible. When we learned that VaccinateCA volunteers were calling hundreds of local pharmacies daily to check for vaccine availability, we asked ourselves: how can our platform help this volunteer effort?

The VaccinateCA phone calls can be broken into two parts – first, identifying whether the vaccine is available, and second, how to get the vaccine. The majority of phone calls never make it past the first question. In fact, volunteers were only getting a 10% hit rate over the 200 calls they were able to make in a day. What if we could help the volunteers focus on the more exploratory work of figuring out who can get vaccines and how, and let us figure out where the vaccine is available?  

Here’s how we built it:

Phase 1. Adapt Infrastructure for New Use Case

The Infinitus VoiceRPA platform is designed to automate phone calls. While our system is currently tuned for a few use cases, one of our goals for 2021 is to make a platform where we can quickly configure and automate any new call type. We planned to leverage our existing system as much as possible so we could get this project out the door fast. We created a new Covid Vaccination call type and a workspace for storing the data. 

The volunteers at VaccinateCA provided our team with a list of thousands of hospitals, clinics, and pharmacies that they were calling. We limited our scope to big pharmacy chains which have a large number of locations with standardized IVR (Interactive Voice Response) systems. They were also the most likely places to have vaccine availability across the state. 

Within a day, the Covid Vaccination workspace was filled with pharmacies and phone numbers, all displayed in the same internal tools currently used by our data labeling team.

Phase 2. Refine the Conversation Model

The volunteers at VaccinateCA follow this script for their phone calls:

• Is the vaccine available for those 65 years and older?
• Where can appointments be made?
• Are walk-ins accepted? 
• What documentation is needed?

We dedicated two members of our data labeling team to prepare the data to call pharmacies. They guided our digital assistant, Eva, through calls to (a) generate training data for our NLP team to automate future calls and (b) validate if pharmacists would be willing to talk to a digital assistant. Here’s what an early call sounded like:

The results were promising – pharmacists were providing the information we were looking for, but, as expected, we encountered a few challenges:

1. Some pharmacists were not sure what to do when a digital assistant called them.

 

Between 25-30% of pharmacists who heard Eva’s automated voice, hung up without interacting with her. 

To combat these outright hangups, we followed up with those pharmacies to explain who we are calling on behalf of and why, and asking if in the future they would be willing to talk to our digital assistant. 

‍

‍

‍

2. Difficulty detecting when to start asking questions.

A person can easily differentiate between a pharmacist, an IVR system, or when they’re on hold, because humans subconsciously use signals like how ‘real’ a voice sounds and background noise. Humans also use signals such as pauses and changing intonations as cues to know when someone is ready for us to ask a question.

Our current Natural Language Understanding model processes incoming voice through a speech-to-text engine first, thereby eliminating some of the audio-based nuances a human can use to guide a conversation. For some pharmacy calls it is hard to tell from just the text if the pharmacist is ready for questions, or if they still need us to wait on hold.

‍

Consider the following examples of common phrases that can appear in the IVR, but are also commonly used by pharmacists:

“How can I help you today?”

In this case, the system expects a button press or a phrase like “speak to a pharmacist” to help direct the call. However, a pharmacist may use the exact same verbiage when picking up the phone, indicating that we should start asking our vaccine related questions.

“Please hold for a moment.”

If we interpret this as something to respond to, the IVR system can route our call in unexpected ways. However, if a pharmacist asks us to continue to hold and doesn’t hear a response right away, they often hang up. 

“Thank you for calling Rite Aid.”

If we hear this in the IVR, we want to wait and listen to the menu options and press the appropriate number on a keypad. However, pharmacists also say this when picking up the phone. It is critical to ask questions as soon as we are connected to a pharmacist, otherwise they assume no one is on the line and will hang up.

‍

3. Variation in responses from free-form questions

 

Responses to yes or no questions can be understood at a much higher confidence with a smaller set of training data than open-ended questions. For example, the answer to “Where can appointments be made?” results in harder to understand responses like “Do the County of Alameda.”, which was a mistranscription of something like “Through the County of Alameda [website]”.

We also noticed that the chances of a pharmacist hanging up on us increased with the length of the call. Combined with the insight that the majority of volunteers’ phone calls never made it past the first question in the script, we realized we could streamline volunteer efforts by automating the question, “Is the vaccine available for those 65 and older?” Eva could pre-screen a much larger pool of sites, and have the volunteers follow-up only in cases where the answer was “yes”. 

‍

‍

Phase 3. Full Automation

To automatically navigate IVRs, we collected the scripts that the IVR systems follow. As long as a location belongs to a pharmacy chain (e.g. Rite Aid) whose IVR we have automated, we know what the IVR will say, except for some minor variations like the location’s address. Using fuzzy matching, we built a state machine which responds with the appropriate key press or voice response until we are put on hold to connect to a pharmacist.

Once on hold, we get a continuous speech-to-text transcript and pass it to our trained model. If someone picks up and says something like “Hi, how can I help you today?”, the model classifies it as a greeting and Eva asks the covid vaccine availability question. We got around the confusion of phrases used in the IVR and by humans picking up by enabling the hold model only once the IVR portion of the call is complete.

Finally, for the conversation itself, we focused on one question: “Does your location have the covid-19 vaccine?” Because the volunteers use this data to determine which sites to call back, we want it to be as accurate as possible. If the system gets a response that it doesn’t understand yet (it’s still improving), it will ask a follow up question “Could you please answer yes or no. Do you have covid-19 vaccines?” If we still don’t get an understandable answer, we call back another time.

Each day, Eva calls pharmacies that haven’t been confirmed to have vaccines yet (since this information changes daily). These calls are now fully automated, from navigating each pharmacy’s automated IVR, detecting when a pharmacist is on the line, and finally extracting the response from the pharmacist and saying bye before hanging up.

We have been supplying the results of our calls to the volunteers at VaccinateCA, who then prioritize positive cases to confirm additional details (such as who’s eligible, how to make appointments, etc.). 

Beyond California

Last week, we expanded our support to volunteer groups mirroring the efforts of VaccinateCA. We are now supporting groups in Colorado, Florida, Michigan, New Jersey, New York, Pennsylvania and Virginia. 

Our Impact

Over the past two weeks, we have made over 10,000 calls to pharmacies across 8 states. After the initial implementation phase, we’ve mostly been able to sit back and let the machine do its job. The data we’ve provided back to the group has proved immensely valuable, increasing the efficacy of their volunteer callers by an order of magnitude.

Working with Infinitus has been game-changing for our effort at VaccinateCA! In under a week, their digital assistant automated daily phone calls to 2500 pharmacies across California. Using Eva means our callers, and the medical professionals they speak to, spend more of their time getting the vaccine to Californians and less hunting around for answers.
– Patrick McKenzie, CEO VaccinateCA

It’s been fun working on this small side project, adapting and pushing our technology beyond what it was capable prior, while helping our community. We have many more interesting projects like this, and we’re hiring!