One of the more intriguing demos on display during TechFest 2009 is called Situated Interaction, a project by Eric Horvitz and Dan Bohus of Microsoft Research Redmond that aims to enable a new generation of interactive systems that can reason about their surroundings and provide an engaging, appropriate set of responses.

The project includes a number of different examples of the technology's potential, the most notable of which presents a virtual receptionist, or situated conversational agent, that can act as a front-desk receptionist, with a lifelike avatar interacting with visitors, making shuttle reservations or welcoming and registering guests. Horvitz, a principal researcher and research-area manager of the Adaptive Systems and Interaction group, explains:

"What we're doing is about understanding some general principles about how you integrate computation into the flow of tasks in an everyday manner," he says. "The receptionist domain was just the first one we tried out, because we had a couple of interesting issues there: many people being handled by a receptionist,:recognizing who's in the same group and requiring the same task, such as taking a shuttle. How long are people waiting? Are they getting frustrated? It's about managing people's attention."

Below is a peek at the virtual receptionist at work. At top left are visualizations of the resources being utilized by the technology. Below that, the receptionist avatar is presented, alongside options for the task at hand. At bottom left is a side view of the persons being recognized--in this case, Horvitz (left) and Bohus. On the right, the scene is being analyzed by the technology: faces, clothing, affiliation, level of engagement, group, level of attention, last speaker, active engagement, goal, time.

That's just one potential manifestation of the work. Another, Bohus explains, concerns a video trivia game.

"We created a trivia-game system," says Bohus, a researcher in the Adaptive Systems and Interaction group. "We put it on the third floor of Microsoft Research, and we're getting data because people are engaging with it after they grab their coffee, for a little break."

Horvitz elaborates: 

"The trivia game was set up mostly as a way of exploring engagement. We actually rolled out the receptionist into the hallway, and she just sat there, watching people, making eye contact, People would come over, and she'd say: 'Oh, are you interested? Come over and play.' The trivia game is just a way to explore engagement and predict when somebody is interested in engaging versus not. It'll play the game with you, but if you don't do very well, it'll basically move away from you and say to somebody else: 'You want to help this guy? He's not doing very well,' and get somebody in to help you. It's very cute."

Then there's a third proof of concept in the works.

"The other task we're working on right now," Horvitz adds, "about which we're very excited, is Personal Concierge, which we're not showing today but we showed [Monday] to [Microsoft Chief Research Officer and Strategy Officer] Craig Mundie in a private session. He's very supportive. This lady, Laura, is stationed by my door, my office, and she handles my whole schedule and the statistics of my comings and goings, People come to my door, and she says: 'He's busy right now. Hang on a minute," or "He's five minutes late, he went off the network a minute ago.' She knows my schedule and knows how to negotiate with people."

It's entertaining, interacting in human fashion with a machine, but behind the scenes, a whole host of artificial-intelligence forces are at play: speech recognition, detection and tracking of persons and groups, intention recognition, attention and engagement modeling, and natural-language processing. 

"This whole project is about weaving together lots of components into a bigger whole," Horvitz says, "from natural language to computer vision, the acoustical microphone--weaving it together to see if we can get a bigger whole than the sum of its parts.

"I think you'll see a lot more [human-computer interaction] based in dealing with a presence like this, in a very natural way, with gestures and expressions, eye contact. The technology itself could be used for everything from bank receptionists to personal secretaries to productivity assistants to teaching kids in a very hot way, not like a cold, intelligent tutoring system, but as a very hot way for engaging many students in a classroom."

He then offers one more possible implementation of his and Bohus' work: 

"Something we will see is technology going into elevators," Horvitz states. "I would say there are three generations of elevators: man on a chair, simple buttons you press, and then the elevator that understands when you put your hand in the door. It waits for you while you're talking. It has a camera on the inside and a camera on the outside, and it monitors facts and conversations to know when it should hold the elevator or come and take you somewhere. It might even be overlaid into other invisible technologies, like doors that open as you approach but don't open when you're walking past them, that understands trajectories."

All this is suitably impressive, of course, but there is that other question: Who was the person whose real-life presence was pressed into service as that receptionist avatar? We're on the case.