Microsoft wants to lead its customers into a bold future where interacting with computers and other devices is less reliant on keyboards and typing and more like conversing with an actual human. Or, as a recent post on the company’s Windows developer blog puts it, “Speech rules, text drools.”
To demonstrate Windows 10’s speech, facial recognition and inking capabilities, the company’s developers have built a Universal Windows Platform app called FamilyNotes. The message-board app itself—written in C# and XAML—is pretty dang simple. One might even ask why we’d need an app that basically just creates glorified Post-It notes for our friends and family to read.
But the point, of course, lies in the Windows 10 features Microsoft is showing off. For starters, the app integrates with Cortana, Microsoft’s digital assistant. Users can use voice commands to get Cortana to open the app or start drafting a new note. Notes can be dictated or jotted down using InkCanvas. (Yes, this means doodles can be added.)
The app can also make use of facial recognition to ensure that a user sees only the notes meant for him/her. FamilyNotes determines when a face is present, matches the face to an existing database and then filters the notes for that person. Microsoft addresses the obvious privacy concerns in its documentation of the app, noting that apps should always let users know before snapping an image and alert them that the image will be uploaded to the Microsoft Face API.
Microsoft has provided extensive notes on the app’s development in a series of blog posts, including speed bumps the devs ran into and how they addressed them. (At one point, the team was stumped by a bug in the note dictation until they realized the computer was hearing its own voice and adding it to the note.)
Developers who want to experiment with this technology can download the Visual Studio project for FamilyNotes from Github. You’ll need a Microsoft Cognitive Services subscription to enable the facial-recognition piece.