Inspiration 🌻

After using a speech to text interface, we realized it would be a cumbersome way for visually impaired people to browse the web. We decided to make a chatbot that could execute and interface with the web on behalf of the user and could be easily implemented by developers for their own sites.

What it does 🔎

Our chrome extension-enabled bot gathers input from the user and uses machine learning and NLP to associate user input with actions that the bot can execute. The extension is also programmable for developers to customize it to each website, so they can add unique, higher level actions to improve their specific user experience. The program then executes the actions on the webpage and gives feedback to the user. This innovative solution not only aids in streamlining tasks for those with visual impairments but also opens avenues for developers to customize and augment the extension with additional actions. By empowering developers to contribute to the extension’s functionality, we aimed to create a more inclusive and adaptive tool, fostering a collaborative environment that prioritizes accessibility in digital applications.

How we built it 🏗

  1. Chrome extension with interactive chatbot built in React
  2. Natural Language Processing and Model Training of phrases using Azure’s Cognitive Science LUIS API
  3. Automatically Creating LUIS Models using Node JS and YAML schema files

What’s next for Helping-Eye 🤔

First, we would like implement text-to-speech methods to read our bot’s messages to the user. In addition, we would like to develop higher level functionality for the chatbot, with deeper and more complex capabilities on a wide variety of sites.

Because of this, we tried to design Helping-Eye in such a way that developers could add to its functionality in the future. A future for Helping-Eye or something as similar as an open source resource would be tremendously beneficial for visually impaired people, and everybody else.

Awards 🏆

  • 1st Place Microsoft Challenge (best use of Azure’s cognitive services)
  • 1st Place Gartner Challenge (best chatbot)
  • Top 6 overall

Authors ✍

  • George Thayamkery (@geooot)
  • Anikait Sharma (@anikait627)
  • Aditya Pethe (@aditya-pethe)
  • Jon Waterman (@jonwaterman)

See here for more details