A new platform called LipPass is one of many new behavioral verification techniques for smartphones. Up until recently, identity verification involved simple password protection, followed by physical biometric scans such as fingerprints, voiceprints, and facial IDs. Now, behavioral biometrics is taking center stage as companies develop ways to track user behavior such as finger motions – and now mouth movements – to make sure only an owner can access their phone data. According to IEEE, the new tech relies on the existing specialized infrastructure in phones without requiring extra operations that make it more expensive or introduce lag time and interrupt the user experience. The new platform was described in the January 23rd issue of the journal IEEE/ACM Transactions on Networking by a team of researchers from Shanghai Jiao Tong University in Shanghai and Indiana University-Purdue University in Indianapolis. In the paper, the authors describe how current biometric privacy protections are vulnerable to replay attacks in which a hacker intercepts data and re-transmits it to break into a phone. While there are solutions that ensure the real-time presence of the phone’s owner (such as face and voice ID), these can be affected by ambient light and noise. The researcher’s new lip reading-based user authentication system, LipPass, relies solely on lip movements and therefore is not affected by noise, though it does require a working camera and some form of lighting. The researchers say that LipPass “extracts unique behavioral characteristics of users’ speaking mouths through acoustic sensing on smartphones for user authentication.” Using the Doppler profiles of acoustic signals that people produce when they speak, the researchers have discovered “that there are unique mouth movement patterns for different individuals.”

“To characterize the mouth movements, we propose a deep learning-based method to extract efficient features from Doppler profiles and employ softmax function, support vector machine, and support vector domain description to construct multi-class identifier, binary classifiers, and spoofer detectors for mouth state identification, user identification, and spoofer detection, respectively.”

To test the efficacy of their methods, the team trialed the technology extensively on 48 volunteers and found that it could decipher a user’s mouth movements with 90.2% accuracy and identify spoofers 93.1% of the time. While biometric technologies are constantly evolving, that doesn’t mean they’re not without controversy. Behavioral profiling is a particularly complex problem, even when it’s employed to protect a user from harm. As we know, any technology that can be deployed for good can also be misused by bad actors. Before employing new technologies, it’s important to look at the potential for abuse and calculate the risks and benefits. One issue is the data collection that will have to take place side-by-side with building and improving this technology. As data and social scientists collect facial and lip data, it’s unclear how else they might use it in future research. Behavioral profiling, a controversial way of spotting crime before it occurs, could easily rely on the data freely handed over by consumers who think it’s merely being used to protect their smartphones from thieves and hackers. We already know that banks and online merchants are tracking website and app visitors’ physical movements. But as the New York Times reported last summer:

“Privacy advocates view the biometric tools as potentially troubling, partly because few companies disclose to users when and how their taps and swipes are being tracked.”

Without knowing where our data is going, we don’t know what personal information we’re providing to companies. The way you type, scroll, swipe, and even the angle at which you hold your phone can reveal highly personal information about you. In research presented to the American Bar Association, a legal expert pointed out that the U.S., in particular, has lagged behind in scrutinizing the ways in which biometric data can reveal information about current or potential medical conditions, treatments, and even non-disclosed or unidentified disabilities. Rachel J. Minter, the author of the paper, noted that while the U.S. is behind the curve, “The European Union, as well as non-E.U. countries, have considered not only the issue of personal information revealed from biometric identifiers, but have adopted strict standards for the transmission and protection of data gathered by biometric or other monitoring technologies.” While data protection is an increasingly important issue now that malicious hacks are on the rise, experts warn consumers that protecting their data from the data collectors – app developers, tech companies, and other researchers – will also be key in safeguarding their future.