TechFest Live! : virtual receptionist

Avatar Revealed!

robk — Thu, 26 Feb 2009 00:21:00 GMT

To be honest, it wasn't too difficult to track down the woman who served as the model for the virtual receptionist in the Situated Interaction demo. Her name is Andrea, and she works as a group assistant in Building 99, the headquarters of Microsoft Research Redmond.

Andrea isn't actually a receptionist; she just plays one on computer screens.

But she used to be. That's how she encountered Eric Horvitz.

"They originally told me they were going to do it when I was still a receptionist," she recalls. "I think I was talking to Eric. They had a different avatar originally, and they wanted to update it because it was an older version. They were excited about the project and didn't want to revamp it, essentially. So we just started talking about it, and they said, 'Hey, how would you feel about that?' and I said, 'I think it would be fun.' So I did it."

Horvitz says Andrea, whom he calls "my favorite admin," seemed a natural.

"I thought she’d be a great person," he says. "She was willing to help. She’s friendly and smart. She’s expressive. We had to catch her expressions. We have a whole folder, a day of video, where she went through all the sequences: awe, surprise, disgust, all the phonemes. We just started rendering them, but we have her on tape, so we can go deep."

Happiness, sorrow, surprise, amusement: Andrea's range of emotions bring the virtual receptionist to life.

Andrea agreed to sit for a couple of photo shoots, which ranged from the glamorous to the peculiar.

"For the first photo shoot," she recalls, "I came in, and they did all my makeup, which was fun! They had me sit in the big room at Microsoft Studios, on this stool, and they would get really close and take pictures all the way around, and I would move a millimeter, and they would say, 'Your posture! You're slumping!' The first one, I had to do a lot of expressions, which was really hard. [laughs] I don't know how to make a sad face! Was I supposed to be overly expressive? I had to do the visemes, I would start a word and then stop, so they could catch the shape of my mouth. Then they had me do a live recording. People would come up, and I would react the way I normally would at the reception desk. They videotaped several of those interactions.

"The second one was just primarily pictures and trying to get more angles--and really close up to my teeth. I had to hold my mouth open, and they got really, really close to my teeth. I think they wanted to get the texture. That was funny."

Andrea first got a glimpse of her avatar last September, right after the second photo shoot and right before Microsoft's annual company meeting.

"It was crazy," she says, "because it looked so much like me. I was really surprised, because Eric was saying: 'Now, you know, it's really at the bottom level. We haven't really refined it yet.' But I was surprised at how good it was, actually."

Having been "discovered" at the Building 99 receptionist desk, Andrea, naturally, had a few suggestions to offer to the researchers.

"When we were shooting the live footage," she says, "and I was reacting as I would as a receptionist, yeah, there were times when I said, 'No, I probably would do it this way or I would respond this way.' Some of the scripts they had me change a little bit, but not too much. I guess that would be the little added bit. They asked some questions about certain things, but it wasn't too much."

The virtualization experience has given Andrea a taste of being a celebrity of sorts in Microsoft circles.

"Probably like once a week," she smiles, "somebody says, 'Oh, so you ...' and I'm like, 'Yeah, it's me.' [laughs] But most people don't recognize me unless they knew me before. It's usually people I know who recognize me. They'll come back and say, 'A few months ago, I saw this thing ...' 'Yeah, it was me.' "

Sounds like an enjoyable adventure.

"It's pretty good," she says. "It's fun. I like it. It allows me to bond with people in a different way. They recognize me, so it's almost like an icebreaker sometimes.That's cool."

But there was one tiny drawback, pertaining to a trivia-game research project in which her avatar was used.

"I took the trivia test, because they had it in the kitchenette to gather data," she says, "and my virtual self beat myself. My virtual self was smarter than my actual self!

"That was a humbling experience."

Dan Bohus (left), Eric Horvitz, and the two Andreas

The Virtual Receptionist

robk — Wed, 25 Feb 2009 22:07:00 GMT

One of the more intriguing demos on display during TechFest 2009 is called Situated Interaction, a project by Eric Horvitz and Dan Bohus of Microsoft Research Redmond that aims to enable a new generation of interactive systems that can reason about their surroundings and provide an engaging, appropriate set of responses.

The project includes a number of different examples of the technology's potential, the most notable of which presents a virtual receptionist, or situated conversational agent, that can act as a front-desk receptionist, with a lifelike avatar interacting with visitors, making shuttle reservations or welcoming and registering guests. Horvitz, a principal researcher and research-area manager of the Adaptive Systems and Interaction group, explains:

"What we're doing is about understanding some general principles about how you integrate computation into the flow of tasks in an everyday manner," he says. "The receptionist domain was just the first one we tried out, because we had a couple of interesting issues there: many people being handled by a receptionist,:recognizing who's in the same group and requiring the same task, such as taking a shuttle. How long are people waiting? Are they getting frustrated? It's about managing people's attention."

Below is a peek at the virtual receptionist at work. At top left are visualizations of the resources being utilized by the technology. Below that, the receptionist avatar is presented, alongside options for the task at hand. At bottom left is a side view of the persons being recognized--in this case, Horvitz (left) and Bohus. On the right, the scene is being analyzed by the technology: faces, clothing, affiliation, level of engagement, group, level of attention, last speaker, active engagement, goal, time.

That's just one potential manifestation of the work. Another, Bohus explains, concerns a video trivia game.

"We created a trivia-game system," says Bohus, a researcher in the Adaptive Systems and Interaction group. "We put it on the third floor of Microsoft Research, and we're getting data because people are engaging with it after they grab their coffee, for a little break."

Horvitz elaborates:

"The trivia game was set up mostly as a way of exploring engagement. We actually rolled out the receptionist into the hallway, and she just sat there, watching people, making eye contact, People would come over, and she'd say: 'Oh, are you interested? Come over and play.' The trivia game is just a way to explore engagement and predict when somebody is interested in engaging versus not. It'll play the game with you, but if you don't do very well, it'll basically move away from you and say to somebody else: 'You want to help this guy? He's not doing very well,' and get somebody in to help you. It's very cute."

Then there's a third proof of concept in the works.

"The other task we're working on right now," Horvitz adds, "about which we're very excited, is Personal Concierge, which we're not showing today but we showed [Monday] to [Microsoft Chief Research Officer and Strategy Officer] Craig Mundie in a private session. He's very supportive. This lady, Laura, is stationed by my door, my office, and she handles my whole schedule and the statistics of my comings and goings, People come to my door, and she says: 'He's busy right now. Hang on a minute," or "He's five minutes late, he went off the network a minute ago.' She knows my schedule and knows how to negotiate with people."

It's entertaining, interacting in human fashion with a machine, but behind the scenes, a whole host of artificial-intelligence forces are at play: speech recognition, detection and tracking of persons and groups, intention recognition, attention and engagement modeling, and natural-language processing.

"This whole project is about weaving together lots of components into a bigger whole," Horvitz says, "from natural language to computer vision, the acoustical microphone--weaving it together to see if we can get a bigger whole than the sum of its parts.

"I think you'll see a lot more [human-computer interaction] based in dealing with a presence like this, in a very natural way, with gestures and expressions, eye contact. The technology itself could be used for everything from bank receptionists to personal secretaries to productivity assistants to teaching kids in a very hot way, not like a cold, intelligent tutoring system, but as a very hot way for engaging many students in a classroom."

He then offers one more possible implementation of his and Bohus' work:

"Something we will see is technology going into elevators," Horvitz states. "I would say there are three generations of elevators: man on a chair, simple buttons you press, and then the elevator that understands when you put your hand in the door. It waits for you while you're talking. It has a camera on the inside and a camera on the outside, and it monitors facts and conversations to know when it should hold the elevator or come and take you somewhere. It might even be overlaid into other invisible technologies, like doors that open as you approach but don't open when you're walking past them, that understands trajectories."

All this is suitably impressive, of course, but there is that other question: Who was the person whose real-life presence was pressed into service as that receptionist avatar? We're on the case.