A device that taps into your thoughts

In 2009, Patti Maes, a professor at MIT’s Media Lab, delivered a TED talk in which she presented a wearable human-computer interface which interpreted real world gestures in real time. It was called SixthSense, and her student Pranav Mistry, whose idea it was, became an instant star.

In March this year, another of her grad students made a splash with an HCI device. Called AlterEgo, and in its first public prototype, it was created under Dr. Maes’s guidance by Arnav Kapoor and his brother Shreyas, an undergrad student. Some of the attention was because of headlines that said it read minds.

In a video chat with The Hindu, Mr. Kapur admitted to being dismayed by some of the coverage. “Our approach from the start was to make something that the user has absolute control over what information to transmit to another person or computing device. We chose this modality, which is somewhere in the sweet spot between thinking and speaking.”

Minimal disruption

How would AlterEgo be best described? “A silent-speech device would be more accurate,” he says. “An intelligence-augmentation device is a higher metaphor.”

In an email, Dr. Maes set the background: “Today’s digital devices are very disruptive. They require our complete attention.” Like SixthSense, she says, AlterEgo experiments with a radically different form factor, with the goal being to “enable a person to make seamless use of digital information and services in a minimally disruptive way. I am very excited about AlterEgo because it offers a solution that is wearable, that is private and that is very quick, convenient and seamless.”

AlterEgo interprets ‘silent speaking’ and when necessary, sends feedback to the user via audio others can’t hear. Silent speaking, Mr. Kapur says, is what we do when we consciously say a word in our minds and try to speak it internally; while we seemingly don’t move our mouths, our brains send tiny electrical signals to our speaking muscles, which AlterEgo’s sensors on the face and jaw detect.

It sends this information to a server where a machine-learning model translates it into words. “Then we have another software environment which makes sense out of the words, understands the task, , and gives you an output, or if it warrants a reply, sending the answer to your device, which transmits it to your inner ear via text-to-speech.” The transmission is through bone conduction, vibrations to the skull no one else can hear.

Deep learning

Creating a device like AlterEgo requires more than casual knowledge of a variety of fields. Mr. Kapur describes himself as “not that good a student, bad in a classroom setting though extremely fond of learning,” someone who enjoys wrestling with problems.

“My methodology is to go deep into concepts, read a lot about concepts I haven’t encountered. I think I’ve developed the ability to sit in seemingly fragmented fields, connecting them in different ways.” The Media Lab, which brings together practitioners in wildly disparate disciplines and encourages collaboration and cross-fertilisation, is a perfect environment for him.

The fields in play so far, he says, are “Neuroscience, electrophysiology; which overlaps with biotech in a way, making the signals accessible; AI, which is computer science and machine learning; hardware design; computer-aided design, how to build hardware, miniaturise all of these components; electronics for design implementation; software application design, which is when you have the platform ready, how do you develop more applications on top of that.”

Further refinement

Dr. Maes and Mr. Kapur both stress that there is a lot of work to be done before the device can get to the market. This would include refinement of the form factor. Mr. Kapur mentions materials science research in progress, which could lead to a transparent device. Multiple languages, and within those, accents, may be problems to solve.

What uses do they foresee? Mr. Kapur says that they have experimented with a number of use cases, including controlling devices, solving math problems, assisting chess and go players, assisting memory by pulling up stored information.

Other important applications he and Dr. Maes see are the enabling of people with speech or hearing disabilities. She says, “We have yet to start testing for these use cases, but I am hopeful that we will be able to help some people, and that will be incredibly gratifying.”

Mr. Kapur permits himself one large statement amidst the disclaimers about work still to be done: “It gives you superpowers.”