At the I/O 2018, Google demonstrated a new Artificial Intelligence-based feature called Duplex, which essentially is capable of making phone calls, while pretending to be a human.

At the event, Google demonstrated Duplex’s call to a hair salon receptionist, mimicking the "ums" and "hmms" pauses of human speech. In another demo, it chatted with a restaurant employee to book a table.

The initial reaction to the feature was positive in terms of what AI is now capable of. But as people and experts continue to spend more time and thought on the technology, a debate on the downside of the technology has been stoked.

Many technology critics argue that the similarity of Duplex’s voice to a humans, and the fact that another person can not tell it from a robot’s is a concern.

Robotic voices should always sound "synthetic" rather than human, Stewart Brand, an author who advocates for long-term thinking and responsibility in the face of advancing technology and other trends told Bloomberg. "Successful spoofing of any kind destroys trust."

Google Assistant making calls pretending to be human not only without disclosing that it's a bot, but adding "ummm" and "aaah" to deceive the human on the other end with the room cheering it... horrifying. Silicon Valley is ethically lost, rudderless and has not learned a thing.

— zeynep tufekci (@zeynep) May 9, 2018

While Google did say that it plans to prompt a warning before each call made by Duplex, Google is still unsure how to actually unleash the technology into the market.

The company is also mulling how human does it want to make Duplex sound. This entire debate is larger than what we see on the surface right now. As Google is working towards eerily human-like assistant, which undeniably interesting at least aesthetically, the company has to be very careful of any mishap or consumers could lose trust in their services.

Having said that Scott Huffman, an executive on Google’s Assistant team, put forth the company’s perspective that if Duplex is given an obvious synthetic voice, it would creep people out and just hang up. On the other hand, another company employee said that the company is not trying to make the assistant pretend to be human in the first place.

There’s a thin line between Google’s aim of making its assistant like a human and not deceiving real humans with software like Duplex. Google consciously decided against giving the assistant a real human background. Apparently, when it’s asked how old it is, or where it was born, it either avoids the question or says things like "I was born in a meeting."