Think back to the last time you texted a chatbot. Whether it was a concierge, a customer support assistant, or an AI virtual recruiter, chances are the bot guided you through a linear flow.
There’s very little “intelligence” in a bot unless it is trained with Natural Language Processing (NLP) and is able to engage a user.
Part of improving the user experience involves anticipating how a person will interact with the bot. And that has been my main focus over the last two weeks at impress.ai — figuring out how to train a chatbot so it can respond to a dialog with spelling errors.
What’s all the fuss about typos?
Our AI chatbot is used for recruiters to actively engage and shortlist candidates during the hiring process. One way we promote engagement is by letting users (candidates) ask the chatbot questions during the interview to learn more about the company, role, and career benefits.
With NLP techniques, the chatbot is able to separate each word within the question, check for errors, and then provide suggestions for what the candidate’s question could have been. This is like when you mistype a question into Google and the first thing that shows up is a ‘Did you mean?’ statement.
Typos in candidate questions limit the ability of the chatbot to respond with answers that are actually helpful. This is why we decided we to create a way for the chatbot’s algorithm to recognise spelling errors, so dialogs wouldn’t be abruptly interrupted.
Recommended tools to create a spell correction feature
It’s important to understand that the number of questions any chatbot can answer is limited to the questions within the bot’s training dataset. Based on the use case, the information can be tailored to fit this dataset. Information that is either biasing or irrelevant can be discarded from the dataset, so the chatbot only only learns answers to questions that a candidate is likely to ask.
Since our algorithm is deployed in an interactive chatbot environment, using a third-party web based spell or grammar correction tool would lead to a significant increase in response times. It would also raise concerns around data privacy.
My team and I researched several tools to help meet our goal and these were the ones we used:
Algorithm employed for our spell correction feature
Once we had the tools in place, the next step involved brainstorming and executing the right techniques. Here is what our process looked like:
However, a little testing revealed that we couldn’t rely on the similarity scores alone to replace the sentence sensibly. So we added a frequency check to the pipeline.
Is this something you see yourself doing everyday?
If you’d like to work at the forefront of innovative technology and use data to improve systems, I’d recommend you check out these jobs with one of the top technology consulting companies in Singapore and the Asia-Pacific region: