It’s easy to overlook the potential business uses of voice assistant technology. Despite their frequent dismissal as a publicity stunt, voice assistants are already performing a variety of roles across the enterprise, from driving down customer wait times to managing meeting schedules. With potential productivity, customer service, and customer experience (CX) benefits on offer, it’s worth considering whether your business should develop its own voice assistant.

However, it can be difficult to know where the development process should begin. Voice assistants are the sum of several complex parts. Like many machine-learning technologies, they require a depth of knowledge and experience to build out effectively.

Despite this, you don’t need to be a software developer or data scientist to start making some meaningful progress towards a voice assistant of your own. An understanding of the inner workings of this technology could go a long way towards building a great foundation for your assistant. With this goal in mind, let’s explore some of the processes behind this new technology — and the tools you’ll need to build a great product.

How Do Voice Assistants Work?

When a voice assistant receives some audio input, there are several steps that it must go through before it’s able to respond appropriately. Let’s imagine that we’ve asked a voice assistant to schedule a team meeting. A walkthrough of this task can be summarized as follows:

  • The voice assistant records the speech, removing any background noise and splitting the speech into its component parts.
  • The speech is transcribed into what the assistant thinks is the most probable sentence, based on the patterns of these sounds.
  • The assistant determines the most important words and most likely intent of the text.
  • If it requires more clarification, such as the time or date of the meeting, the assistant might compose questions. Alternatively, it may decide that it is confident enough in its answer to proceed without further information.
  • Finally, the commands are processed in a decision engine and the meeting is scheduled.

Many of these tasks rely heavily on natural language processing (NLP). This is a field of computer science that is broadly concerned with a machine’s ability to recognize what is said to it, understand its meaning and the appropriate action, and respond in language that the user will understand.

Within NLP, there are two subfields that particularly concern voice assistants: natural language understanding (NLU) and natural language generation (NLG). Unsurprisingly, NLU focuses primarily on the ability of machines to detect, comprehend and attribute a meaning to speech or text input, whereas NLG deals with the reverse process: turning computer generated responses into text or speech that a human can understand.

All things considered, there are three major problems that every voice assistant has to overcome. To work effectively, they must be able to turn speech input into accurate text, add structure to that text, and interpret its meaning correctly. By looking more closely at these problems, it’s possible to identify the tools we need to build a working voice assistant.

Related Article: Natural Language Generation Accelerates Marketing, But Keep Your Hands on the Wheel

How to Turn Speech Into Text

Voice assistants have to be able to identify words and sentences spoken by multiple people in noisy environments. They need to clearly understand reminders set by Scots, emails dictated by South Africans, and questions asked by non-native speakers of English. In addition, at any given moment a TV could be playing in the background, or the speaker could be waiting to cross a busy intersection. It’s easy to see that both the individual components of speech and the audio as a whole can change significantly, depending on where and by whom they are uttered.

Several audio processing tasks help voice assistants deal with this problem and accurately convert speech into text. Audio source separation focuses on the isolation of one or more signals from the input — in our case, the person’s voice. This step removes all unwanted background noise that could confuse the voice assistant. Following this, audio segmentation processes can divide the speech into smaller parts according to their characteristics. This could take the form of phrases, words or even individual sounds. Segmenting the audio turns it into short, homogeneous pieces that are easier to deal with. Finally, these pieces are automatically transcribed into text. Thanks to the segmentation of the text, this can be done quickly and accurately by mapping sounds to their most likely text representations.

Many companies use third-party solutions to do this work, since they can accurately cover a broad range of general use cases. However, it’s also possible to build out this software in-house. In order to train an algorithm to perform these tasks, your developers will need customized training data, labeled and annotated for your specific use case. The labels that this data contains will become the textbook that teaches your new algorithm the intricacies of your particular task — helping you to deliver a greater ROI.

Of course, building out your own solution is a more laborious process than paying for a third-party solution. However, it’s also possible to outsource some of the more time-consuming tasks, such as data annotation and transcription, to specialist services. If you have a good grasp of the particular way you want your voice assistant to parse speech, a data provider should have no trouble creating a dataset that will optimize your model for the task at hand. For those with specific use cases, such as a target demographic with a certain regional accent, it’s particularly worth investigating whether this extra effort is worth it.

Although this process allows us to accurately identify the text that corresponds to our speech input, our work is only just beginning. The next important step is to give that text structure.

Related Article: How Voice Is Changing Customer Experience Practices for the Better

How to Add Structure to Text

Humans and machines understand text in different ways. For a machine to understand a piece of text, it’s absolutely essential that important elements within that text are identified and labeled with their appropriate meaning. The process of finding and tagging these elements is called entity extraction.

An entity is a part of the text data, such as a word or phrase, that fits within the boundaries of a certain predetermined category. Common examples of categories include names, companies or places. However, entities don’t just have to be semantic. Words and phrases can also be labeled with their grammatical role or their relationship with other parts of the text, such as parent companies and their subsidiaries.

By performing entity extraction on a piece of text, it’s possible to teach a voice assistant which words do what, and how they fit together in a sentence. Once it can do this, it has a far greater chance of understanding the way language works. It can also start to guess the most likely sentence match for certain patterns of utterances.

There are a range of options to explore when looking to build out a voice assistant’s entity extraction capabilities. Several third-party providers offer entity extraction software, who can provide a ready-made solution capable of annotating for a range of entity types. As before, it’s also possible to create or order human-annotated datasets, labeled according to your own unique classification system.

Learning Opportunities

This process gives the engine behind our voice assistant more attributes to work with, preparing it to understand the structure of our original speech input. However, understanding structure is not the same as understanding the request that the speaker is making. We also need to put a framework in place that will help the voice assistant to recognize the intent of speech.

Related Article: AI Training: A Crash Course to Improve Your Project ROI

How to Recognize a Text’s Intent

The ability to understand and execute commands is a crucial feature of any voice assistant. However, languages provide us with multiple ways of asking a question. A voice assistant has to be able to recognize as many of these as possible, otherwise it will be unable to fulfil its purpose. This is often easier said than done. Consider the following three sentences:

  • Can you tell me more about your returns policy?
  • I’d like to know what I can do if I don’t care for this product.
  • So, what’s the deal with sending stuff back? Is it free?

All of these are plausible input utterances that a voice assistant could receive. They also share the same goal: to learn more about a returns policy. However, as these examples suggest, people rarely say this so explicitly. They may not phrase their requests as questions, or even use important keywords such as “returns.” If it can’t recognize this, the voice assistant simply can’t function.

Fortunately, it’s possible to avoid relying on canonical questions and answer pairs. Instead, preparations for voice assistant development should be focused on identifying intents, before sourcing variations of those intents.

An intent is the desired aim of a piece of text. For example, the intent of the phrase “I want to add a new credit card” is “update credit card information.” A variation is a different way of phrasing that intent. For example, a further variation of our credit card intent could be “please switch out my credit card.” The more variations there are for each intent, the more likely it is that the voice assistant will recognize that intent in a sentence — and the better its performance will be.

It’s essential to identify your key intents, otherwise your voice assistant won’t be able to identify the task it’s supposed to complete. Luckily, these should align pretty closely with the goals of your project. For example, those building a voice assistant for customer service could base their intents around their company FAQs.

After this, it’s useful to build a dataset of variations for these intents. Within this dataset, there should be at least 10-20 variations for each intent. When they’re used correctly during training, a large number of variations should enable your voice assistant to recognize key intents in a diverse range of text samples. It can also enable your machine-learning model to learn from new, unseen examples — meaning your voice assistant will get stronger as it performs its task.

Once these variations have been sourced, all the big pieces in our voice assistant toolkit are in place. With this array of third-party tools, solutions and datasets, we should have all the materials we need to develop a functioning voice assistant.

Related Article: Do Chatbots Dream of Electric Sheep?

Are Voice Assistants Worth the Effort?

Voice assistants are complex. However, that doesn’t mean the process of building one has to be complicated. As the technology continues to develop and further business use cases are uncovered, the time has never been better to invest in voice assistant tech.

Start by identifying the kind of solutions that will give your developers a solid platform upon which to build. It’s also a good idea to have conversations with third-party providers to figure out the level of customization that suits your needs. This research will help you to source the right building materials — and give your voice assistant the greatest chance of success.

Learn how you can join our contributor community.