TechFest Live! : Microsoft

Digital Past to Digital Presence

robk — Fri, 27 Feb 2009 04:08:00 GMT

Of the 150-odd demos in TechFest 2009, the best-named is Digital Past to Digital Presence, a collection of concepts from the Socio-Digital Systems group at Microsoft Research Cambridge. David Kirk, a post-doctoral researcher in the group, explains what ties the technologies together.

"Both bits of the booth are connecting with something's that's absent, somebody you can't be with, bringing things closer to together," he says. "With Digital Past, it's bringing you closer to your past, in two different ways. With Digital Presence, it's connecting with people you can't physically be with. In both cases, they're connecting with bits of your life that are distant in some way and bringing them together."

The demo features four discrete attempts to bridge these physical and temporal gaps: Family Archive, Time Card, CellFrame, and Wayve.

Family Archive: a table-like device with a screen built into the table top and a standard USB connection for peripherals, used for media management in the home.

"At least that's where we started," Kirk says. "It's always on, touch-interactive, multitouch display. We've built three of them and deployed them in family homes for a month at a time. We envision a world where you walk up, put a pen drive in, and content spills out into it. Once you've got digital content in, you can create containers for it.

"One of the things we're particularly interested in doing is to allow people to combine not only already-digital stuff, but to give them interesting ways of digitizing physical stuff that otherwise might not get integrated with those digital things. It's all about creating rich collections of media."

To that end, the Family Archive includes a camera stand on one side of the table that can photograph physical objects placed on the display and capture an image that can be added to a themed digital container, from, say, a vacation. The objects are rendered on a blank background to establish them as separate entities from other things that might be on the screen when the image is taken.

"You can have this integrated, easily digitized collection of all the photos you've taken, ticket stubs, postcards you might have bought," Kirk says, "and have a much richer archive of memories."

Further explorations of the concept could extend to 3-D capture of physical objects.

Time Card: "We're basically looking at ways of creating timelines," says Richard Banks, an interaction designer at the Cambridge lab, "to either represent your own life or lives of others. We have two key scenarios. One is the creation of timelines as a form of memorial for somebody's who's passed away. In this case, this is my grandfather, who left me a suitcase full of photos. I'm going through those photos and scanning them and adding metadata in order to create a timeline so I can better understand what it was he did during his life, as well as, to some extent, to honor him. It's like making a photo album about him.

"The second scenario we're interested in is about creating timelines on the fly about your own online activities. Our long-term goal is to ask: If you record these things about what you did, how will they impact you in 20 or 30 years? Wouldn't it be interesting, for example, to record the Twitter feeds you're creating now, so that in 20 years' time you can look back on them and reflect on what interested you. One of my goals with Time Card will be to create an object that for 40 years was recording my life and the things I was doing online, and at the end of my life, it could literally be unplugged and given to my daughter as an heirloom."

Currently, the Time Card focus is on archiving images and text. But, as Banks says, "that doesn't mean we're just talking about photos. Some of the content that I have in here are physical objects, like my grandfather's ribbon from when he was a pilot in the Second World War. Here's a letter about his deployment, and here's an insignia from the squadron that he was in. Although it's about imagery, it's about different kinds of imagery that come together to tell a story."

In incorporating images of physical objects, Time Card resembles Family Archive. That, Banks says, is deliberate.

"This general theme we're interested in," he says, "we're calling technology heirlooms. It's about just looking at technology generally and saying: 'What about 30 years' time? Where will this be? Who will care about it? What will people want to do with it?' "

CellFrame: a small, standalone, wireless display and communication device to bring the benefits of social networking to those not actively participating in it.

"There are certain people who might have access to the Internet and certainly might not have a wireless network at home," says Sian Lindley, currently a vendor working with the Socio-Digital Systems group. "but whom you might want to include in social networking."

The device consists of a phone in a frame. Those not inclined to adopt the latest technological trends could put such a device in their home and receive content sent to them. The scenario shown during TechFest displays a family social network, with channels for family members. By using touch input, family members could share content by dragging it across the display and dropping it onto another channel. Additional functionality enables a user to respond to received content via a simple user interface.

"It's just a very simple way," Lindley says, "of seeing who it is, not using the Internet, but still getting some of that information you might otherwise not get."

Wayve: another device, resembling a digital picture frame but incorporating technology that lets people connect playfully and creatively.

"This is like a digital sticky note," Lindley says. "I can create a message to be on display for my family, and I might leave this device somewhere like the kitchen where everyone's going to see it at a glance. It's connected wirelessly to the Internet, so I can also send and receive messages."

Wayve can send a photo to an e-mail address or a photo message to a mobile phone--or to another Wayve.

"What we expected," Lindley says, "is that this might be useful for household messaging, like 'I'm going to be late,' or to ask what was for dinner. But what we found when we deployed these was that people used it in creative and expressive ways. You can do things like play tic-tac-toe. People would do things like take pictures of their dogs and send them to their friends, and the friends would draw hats on the dogs and send them back.

"Just kind of playful, simple, expressive things as a way of connecting families."

Specification Inference for Security

robk — Fri, 27 Feb 2009 01:04:00 GMT

Aditya Nori, a researcher in the Rigorous Software Engineering team at Microsoft Research India, just gave me a brief overview of his demo, entitled Specification Inference for Security, and as he made repeated references to the poster in his TechFest booth, I though it would be instructive to share:

A higher-resolution version is available that enables you to enlarge the poster to read through everything, but the key to the project--a new algorithm that automatically infers explicit information-flow security specifications from program code--is located under Specification on the left-hand side of the poster, which defines the classification of nodes in a data-flow graph of program code as "sources," "sinks," or "sanitizers." A source is a node that returns tainted, bad data. A sink receives that bad data, and a sanitizer cleans the data, so that even if it receives tainted data, it does not pass it along. It's a case of garbage in/no garbage out.

"This project is about improving the quality of existing static-analysis tools for security," Nori explains. "Most automated tools for security rely on specifications, because they really need to know what they're searching for. Our job here is to go through the programs and automatically infer specifications in the program."

He points to the "Information flow vulnerabilities" portion of the poster and to the node named ReadData1.

"If you look at this code fragment," Nori continues, "here's a data-flow graph of the response to this code fragment. Information flows from this method or function call into this function call [Prop1]. If I just went though this graph and asked if there was something wrong, if there's a bug over here, it's hard to say, because you need more context.

"That's exactly what a static-analysis tool is going to say: You need to provide more information, and you need to say what the problem is. On the other hand, if I consider this a source, a producer of taint, and that one a sink, a consumer of taint, then the static-analysis tool will be able to say that the first part is a bad part. There could be a malicious user sending some data that screws up your database. If this [Cleanse] was a cleanser, which actually checks that whatever is being passed on to the database is safe data, then this part is OK. What the static-analysis tool needs to know, in addition to your program, is the role of every function in your program: Here's a function of source, here's a function of sink, here's a function of sanitizer."

In the real world of software development, though, that's an improbable scenario.

"It's really unreasonable to assume that if somebody is presented with 1 million lines of code, they're actually, mindfully going to go through the code and annotate every method as a source, sink, or sanitizer," Nori stipulates. "That's unacceptable. No developer is going to do it. So what we do is go through the code and automatically analyze the code and annotate every function as a source, sink, or sanitizer."

He turns to the Architecture section of the poster.

"Here's a high-level overview," he says. "We take the program, do static analysis, and convert it into a data-flow graph. The data-flow graph includes a bunch of constraints. For technical reasons, it's helpful to look upon these constraints as probabilistic constraints We take the data-flow graph, convert it into this probabilistic model, and feed that to a constraint solver. The solution to the set of constraints precisely tells us which methods in our program correspond to sources, sinks, and sanitizers."

And in a real-life test, the technique worked wonders.

"We ran our tool on 10 critical Microsoft business applications," Nori reports, "and we discovered 67 new sources, 25 new sanitizers, and 75 new sinks. The next step for us was to assess the quality of these specifications. Did these specifications really improve the quality of an existing static-analysis tool? We took a static-analysis tool that is being developer by Microsoft for security and ran that tool on these applications. The tool discovered 89 vulnerabilities, of which 20 were false positives. Then we ran the tool again with our new specifications, and we discovered 335 vulnerabilities. We were very excited about that. Another nice thing about our specifications is that they eliminated 13 of the 20 false positives from this set."

Such numbers grab attention within Microsoft, and there is a strong possibility that the specification-inference algorithm might be included in an upcoming product release. Successful technology transfer makes researchers smile, but Nori knows there is much left to be done.

"My collaborators believe this is a new way of analyzing programs, combining program analysis with statistical analysis," he says. "We are a long ways from applying this to other domains and program analysis. That's the future of this project. We look at this project as the starting point for combining program analysis and statistical analysis."

Language-Agnostic Search

robk — Thu, 26 Feb 2009 22:56:00 GMT

Some of the demos featured in TechFest 2009 were submitted by the Cairo Microsoft Innovation Center, and i got a chance to speak with a couple of Cairo researchers, Kareem Darwish and Motaz El-Saban, about their work.

"We're trying to enable multilingual search," Darwish said, "in the space of text documents and in the space of printed documents. In the case of printed documents, this is the OCRLess."

Then El-Saban took over.

"OCRLess is about language-independent technology," he said, "that allows you to search within scanned document images without the use of OCR (optical character recognition). Traditionally, if you have a document image, you would need to convert it into text using OCR, and then you can search. What we do as an alternative approach is take the text query and transform it into an image for rendering. Then we match it against the image document. It's based on image matching and indexing, and what we're showing here is five languages--English, Arabic, Chinese, Hebrew, and hieroglyphics."

Hieroglyphics ready to be searched.

It's an ingenious approach, upon which El-Saban expanded.

"The first step is to segment it," he explained. "The segment can be a character, a part of a character, or a word. in English, it's a character. We take all these segments and cluster similar shapes together in a completely unsupervised manner, then assign an ID to each cluster. Now, every page is presented, instead of by the characters, by a set of IDs, and we index this set of IDs into a regular search engine. With a query, we do the same thing. We render it into an image using the font of the book, then we segment it into pieces, and we actually assign, for each piece, the closest cluster ID.The query is a string of IDs, basically, and we go search for the string in the book. This makes it effective, because we're using image matching, and efficient, becomes we're using an underlying basic search engine.When we actually match the image query to an image inside the book, we're using template matching, an array of pixels."

So where might this work lead?

"There are a number of possibilities," El-Saban said. "Libraries could use something like this, or whatever entities sit on large volumes of documents, possibly written in many languages, for which you don't have an OCR. Another area of potential is handwriting search. You write your own notes by hand, and then, without even having to recognize your handwriting, you can still search them. I'm trying to sit with different contacts in Microsoft product groups to see if there is an interest to take this project in a specific direction."

Back to Darwish, whose project is called Trans-Bulletization.

"We're trying to enable people to search using English queries against documents in many, many different languages," he says, "and then present the results, not in the original language of the documents, but in a bulletized list, a summary that removes superfluous words from the English translation and puts the information into bulletized form. The user then can very quickly learn what the document is about and consequently make a decision whether they want to invest more time reading the full translation or the original document."

He went on to discuss how entire documents are boiled down to bulleted lists.

"The key technology is all the documents we are going to search, we translate into English first, and then deploy the bulletized technique," Darwish said.."Actually, it's a sentence-reduction technique. To reduce the sentences, we use a dependency parser that recognizes the main verb, the subject of this verb, and the object. Given that these are the core components of the sentence, then we find all the other pieces--prepositional phrases, modifiers, and so forth--then make a judgment about the information content of these pieces. If they have low information content, then they're candidates for removal, but before we remove them, we have to make sure that they don't break anything in the sentence. We won't remove a noun phrase unless that, if we remove it, it won't break the flow as measured by language models."

Such help could provide a boon to many in this information-saturated age.

"For people who work in an organization that requires sifting through a lot of documents in many, many languages, this would be really useful," Darwish concludes. "They're getting all the information content in a shortened version, so they can scan lots of documents very, very quickly. A typical user might be a reporter who wants to see how people look at a particular issue across different countries. As the person enters a query across the countries he's interested in, he gets articles from Japan, the Middle East, from China and Europe, and so forth, and then he can see all the different views at the same time, in bulleted lists. He can do this very, very efficiently and very, very quickly."

Motaz El-Saban (left) and Kareem Darwish in their TechFest booth.

Mining Personal Relationships

robk — Thu, 26 Feb 2009 19:09:00 GMT

Zaiqing Nie of the Web Search & Mining Group at Microsoft Research Asia is demonstrating a project called EntityCube, an English-language version of a wildly popular Chinese project called Renlifang.

In Mandarin, "renlifang" means "three people." The nomenclature is intentional, because the project, and its EntityCube manifestation, is all about demarcating the relationships among a group of people.

"EntityCube automatically summarizes relevant information about people," Nie explains. "For example, here, we've summarized the search results for Bill Gates. You can see that Steve Ballmer is here, IT leaders, celebrities. [He clicks on the Ballmer icon.] You can also have a Web page just for Steve Ballmer. We find all related people, the type of related people, news about him.

"The important part is that we automatically generate this. It's not manual. You can see much of this information on the Web, but human beings do that work. Our input is Web pages. The output is this summarization page and this social graph."

The EntityCube project site offers a good explanation of the technology:

"The need for collecting and understanding Web information about a real-world entity (such as a person or a product) currently is fulfilled manually through search engines. But the information about a single entity might appear in thousands of Web pages. Even if a search engine could find all the relevant Web pages about an entity, the user would need to sift through all the pages to get a complete view of the entity. EntityCube is an entity search and summarization system that efficiently generates summaries of Web entities from billions of crawled Web pages. The summarized information is used to build an object-level search engine about people, locations, and organizations and explore their relationships."

The value in such an approach is confirmed by its embrace by the Chinese public. The project page goes on to note: "Renlifang has been well received by Chinese Internet users and mainstream media in China, with positive comments and millions of page-views during peak days."

Such a reception validates the extensive effort the project has required.

"We've been working on this technology, object-level search, for five years," Nie says. "We got very positive feedback from the Chinese version. Renlifang is very popular right now. People like this idea of mining these results and integrating the search. Renlifang and EntityCube are the first time these mining techniques have been combined with search, and as far as scale and functionality, we are the best."

And the technology is not limited to the human domain. Other entities can be searched and summarized, as well.

"You can have other queries besides name queries," Nie says. "For example, the topic 'climate change.' You can find people who are connected with climate change.

"We're showing EntityCube People, but we are going to produce other ones, like EntityCube Academic, which will be the next version of Libra, and EntityCube Products. Eventually, EntityCube.com will be the portal to other object-level search verticals. That's the long-term goal of our research vision."

The next step for Nie and colleagues is to continue collecting feedback and refining the approach. The interest spawned by the TechFest demo can only help.

"We are trying to release research surveys that invite people to use it and provide feedback," Nie says. "From our feedback today and yesterday, we've found that English speakers also like this idea a lot."

Avatar Revealed!

robk — Thu, 26 Feb 2009 00:21:00 GMT

To be honest, it wasn't too difficult to track down the woman who served as the model for the virtual receptionist in the Situated Interaction demo. Her name is Andrea, and she works as a group assistant in Building 99, the headquarters of Microsoft Research Redmond.

Andrea isn't actually a receptionist; she just plays one on computer screens.

But she used to be. That's how she encountered Eric Horvitz.

"They originally told me they were going to do it when I was still a receptionist," she recalls. "I think I was talking to Eric. They had a different avatar originally, and they wanted to update it because it was an older version. They were excited about the project and didn't want to revamp it, essentially. So we just started talking about it, and they said, 'Hey, how would you feel about that?' and I said, 'I think it would be fun.' So I did it."

Horvitz says Andrea, whom he calls "my favorite admin," seemed a natural.

"I thought she’d be a great person," he says. "She was willing to help. She’s friendly and smart. She’s expressive. We had to catch her expressions. We have a whole folder, a day of video, where she went through all the sequences: awe, surprise, disgust, all the phonemes. We just started rendering them, but we have her on tape, so we can go deep."

Happiness, sorrow, surprise, amusement: Andrea's range of emotions bring the virtual receptionist to life.

Andrea agreed to sit for a couple of photo shoots, which ranged from the glamorous to the peculiar.

"For the first photo shoot," she recalls, "I came in, and they did all my makeup, which was fun! They had me sit in the big room at Microsoft Studios, on this stool, and they would get really close and take pictures all the way around, and I would move a millimeter, and they would say, 'Your posture! You're slumping!' The first one, I had to do a lot of expressions, which was really hard. [laughs] I don't know how to make a sad face! Was I supposed to be overly expressive? I had to do the visemes, I would start a word and then stop, so they could catch the shape of my mouth. Then they had me do a live recording. People would come up, and I would react the way I normally would at the reception desk. They videotaped several of those interactions.

"The second one was just primarily pictures and trying to get more angles--and really close up to my teeth. I had to hold my mouth open, and they got really, really close to my teeth. I think they wanted to get the texture. That was funny."

Andrea first got a glimpse of her avatar last September, right after the second photo shoot and right before Microsoft's annual company meeting.

"It was crazy," she says, "because it looked so much like me. I was really surprised, because Eric was saying: 'Now, you know, it's really at the bottom level. We haven't really refined it yet.' But I was surprised at how good it was, actually."

Having been "discovered" at the Building 99 receptionist desk, Andrea, naturally, had a few suggestions to offer to the researchers.

"When we were shooting the live footage," she says, "and I was reacting as I would as a receptionist, yeah, there were times when I said, 'No, I probably would do it this way or I would respond this way.' Some of the scripts they had me change a little bit, but not too much. I guess that would be the little added bit. They asked some questions about certain things, but it wasn't too much."

The virtualization experience has given Andrea a taste of being a celebrity of sorts in Microsoft circles.

"Probably like once a week," she smiles, "somebody says, 'Oh, so you ...' and I'm like, 'Yeah, it's me.' [laughs] But most people don't recognize me unless they knew me before. It's usually people I know who recognize me. They'll come back and say, 'A few months ago, I saw this thing ...' 'Yeah, it was me.' "

Sounds like an enjoyable adventure.

"It's pretty good," she says. "It's fun. I like it. It allows me to bond with people in a different way. They recognize me, so it's almost like an icebreaker sometimes.That's cool."

But there was one tiny drawback, pertaining to a trivia-game research project in which her avatar was used.

"I took the trivia test, because they had it in the kitchenette to gather data," she says, "and my virtual self beat myself. My virtual self was smarter than my actual self!

"That was a humbling experience."

Dan Bohus (left), Eric Horvitz, and the two Andreas

The Virtual Receptionist

robk — Wed, 25 Feb 2009 22:07:00 GMT

One of the more intriguing demos on display during TechFest 2009 is called Situated Interaction, a project by Eric Horvitz and Dan Bohus of Microsoft Research Redmond that aims to enable a new generation of interactive systems that can reason about their surroundings and provide an engaging, appropriate set of responses.

The project includes a number of different examples of the technology's potential, the most notable of which presents a virtual receptionist, or situated conversational agent, that can act as a front-desk receptionist, with a lifelike avatar interacting with visitors, making shuttle reservations or welcoming and registering guests. Horvitz, a principal researcher and research-area manager of the Adaptive Systems and Interaction group, explains:

"What we're doing is about understanding some general principles about how you integrate computation into the flow of tasks in an everyday manner," he says. "The receptionist domain was just the first one we tried out, because we had a couple of interesting issues there: many people being handled by a receptionist,:recognizing who's in the same group and requiring the same task, such as taking a shuttle. How long are people waiting? Are they getting frustrated? It's about managing people's attention."

Below is a peek at the virtual receptionist at work. At top left are visualizations of the resources being utilized by the technology. Below that, the receptionist avatar is presented, alongside options for the task at hand. At bottom left is a side view of the persons being recognized--in this case, Horvitz (left) and Bohus. On the right, the scene is being analyzed by the technology: faces, clothing, affiliation, level of engagement, group, level of attention, last speaker, active engagement, goal, time.

That's just one potential manifestation of the work. Another, Bohus explains, concerns a video trivia game.

"We created a trivia-game system," says Bohus, a researcher in the Adaptive Systems and Interaction group. "We put it on the third floor of Microsoft Research, and we're getting data because people are engaging with it after they grab their coffee, for a little break."

Horvitz elaborates:

"The trivia game was set up mostly as a way of exploring engagement. We actually rolled out the receptionist into the hallway, and she just sat there, watching people, making eye contact, People would come over, and she'd say: 'Oh, are you interested? Come over and play.' The trivia game is just a way to explore engagement and predict when somebody is interested in engaging versus not. It'll play the game with you, but if you don't do very well, it'll basically move away from you and say to somebody else: 'You want to help this guy? He's not doing very well,' and get somebody in to help you. It's very cute."

Then there's a third proof of concept in the works.

"The other task we're working on right now," Horvitz adds, "about which we're very excited, is Personal Concierge, which we're not showing today but we showed [Monday] to [Microsoft Chief Research Officer and Strategy Officer] Craig Mundie in a private session. He's very supportive. This lady, Laura, is stationed by my door, my office, and she handles my whole schedule and the statistics of my comings and goings, People come to my door, and she says: 'He's busy right now. Hang on a minute," or "He's five minutes late, he went off the network a minute ago.' She knows my schedule and knows how to negotiate with people."

It's entertaining, interacting in human fashion with a machine, but behind the scenes, a whole host of artificial-intelligence forces are at play: speech recognition, detection and tracking of persons and groups, intention recognition, attention and engagement modeling, and natural-language processing.

"This whole project is about weaving together lots of components into a bigger whole," Horvitz says, "from natural language to computer vision, the acoustical microphone--weaving it together to see if we can get a bigger whole than the sum of its parts.

"I think you'll see a lot more [human-computer interaction] based in dealing with a presence like this, in a very natural way, with gestures and expressions, eye contact. The technology itself could be used for everything from bank receptionists to personal secretaries to productivity assistants to teaching kids in a very hot way, not like a cold, intelligent tutoring system, but as a very hot way for engaging many students in a classroom."

He then offers one more possible implementation of his and Bohus' work:

"Something we will see is technology going into elevators," Horvitz states. "I would say there are three generations of elevators: man on a chair, simple buttons you press, and then the elevator that understands when you put your hand in the door. It waits for you while you're talking. It has a camera on the inside and a camera on the outside, and it monitors facts and conversations to know when it should hold the elevator or come and take you somewhere. It might even be overlaid into other invisible technologies, like doors that open as you approach but don't open when you're walking past them, that understands trajectories."

All this is suitably impressive, of course, but there is that other question: Who was the person whose real-life presence was pressed into service as that receptionist avatar? We're on the case.

Managing Audio in Wheel Time

robk — Wed, 25 Feb 2009 19:13:00 GMT

Ivan Tashev (right) shows the Commute UX demo to Rick Rashid (left), senior vice president of Microsoft Research, and
Craig Mundie, Microsoft chief research and strategy officer.

Ivan Tashev, principal architect in the Speech Technology group at Microsoft Research Redmond, has one of those smiles that can light up a room, and he's flashing it repeatedly as he explains his TechFest 2009 demo Commute UX: Dialog System for In-Car Infotainment.

"In this demo," he beams, "we show natural-language-processing technologies that will allow drivers to have easy access to their songs, to their information, to their phone calls. This will increase driver satisfaction, and we believe it will lead to safer human interaction in the car."

Sounds like fun work, and it must be, because in their booth, Tashev's colleagues Mike Seltzer and Y.C. Ju are all grins, as well.

"For us, speech is the most natural choice for a human-machine interface in the car," Tashev says, "because it's an eyes-busy and hands-busy task. We have noticed that in systems deployed so far--initially on high-end cars, but lately, on most of the Ford models, built on top of the Microsoft platform--they use a very complex menu structure and they are very rigid in the commands you have to say to navigate. This is why we decided that it would be much easier to reduce the number of commands and to have a flat, 'say anything anytime' menu. We require the drivers to remember just three commands: Play, Call, Replay. That's it.

"According to our research studies, in more than 50 percent of cases, the query for music is not precise, not correct. People do not remember the exact song name. In many cases, the songs are known by completely different names than what we is written on the CD label. We have sophisticated speech-processing software that tries to match the output of the speech recognizer with the potential songs or entries in the address book and to provide the most relevant choice. For example, when people don't know the exact name, they try to add more information, usually the artist's name. Even that may not be correct. We have recorded queries like, 'Play Hey Oh by the Chili Peppers,' which is either 'play title Snow (Hey Oh)' or 'play artist the Red Hot Chili Peppers.' Our system is able to understand those two pieces of information and find the right song to play."

Tashev, Seltzer, and Ku work in close tandem with Microsoft's Automotive Business Unit.

"We get feedback from them," Tashev confirms. "They try to use our technologies and our experience whenever they find it appropriate."

And make no doubt about it: This technology is coming your way, quite possibly in your next automobile.

"We strongly believe," Tashev smiles, "that such technologies, such speech-enabled human-computer interfaces in the car, are in the process of transition from a cool gadget--something which you call your neighbor to show when you buy a new car--to an integral part of the automobile, something like air conditioning, which nobody thinks is something exceptional. To go through this transition, we have to find the proper technologies to make the spoken dialog usable for the majority of drivers, without any specific training, and able to handle human errors.

"Even with a perfect speech recognizer, the human is the major source of errors today, from non-correct queries to a desire to have faster and easier access to the information a user wants. I'd love to be able to say, 'Call Ivan at home' instead of 'Call Ivan Tashev at home,' especially if I have only one Ivan in my address book. This completely disambiguates my query."

Tashev, Ju, and Seltzer (at wheel) take their demo for a spin.

The Value of TechFest

robk — Wed, 25 Feb 2009 01:57:00 GMT

I just ran into Roy Levin, Microsoft distinguished engineer and managing director of Microsoft Research Silicon Valley, in the Rainier/St. Helens Room at the Microsoft Conference Center. We found ourselves marveling at Commute UX, one of the TechFest 2009 demos I'll be posting about soon, and I asked him about what TechFest means for him and his lab.

"The value of TechFest," Levin said, "is for people to get a feel for the breadth and the depth of the innovation that goes on in Microsoft Research. In the room we're in right now, there is a sampler, really, of the larger event, and just wandering around this room, you see stuff from so many different areas of computer science and their applications to so many different areas of real life. For me, that's where the real impact of this event is: just feeling-- in a very direct way, a very visceral way--how the work that we do can make the world better through computing."

Levin's Silicon Valley lab specializes in research on distributed computing, and that means TechFest plays a particularly valuable role.

"There are a couple of demos here, in this room, from our lab that focus on understanding and modifying the way that complicated distributed systems work, in data centers in particular," he said. "The benefit, from my point of view, from participating in this event is that people get a glimpse at the problems, the very difficult problems, that we face in trying to harness the world of parallel computing and make it useful, in the form of cloud services, trying to operate these very large services and provide things on a global scale."

Finally, Levin noted, there's a special quality to the event that makes it loom large on researchers' annual calendars.

"There's a special place in my heart for TechFest," he stated, "because, obviously, it's a hometown event, compared to going to a conference or some other meeting. It also is 'hometown' in a different way, because members of Microsoft Research come in from all over the world. We get to meet with people that we don't get to see the rest of the year. That's really great."

Roy Levin (left) and Xian-Sheng Hua of Microsoft Research Asia chat during TechFest.

Improving Data-Center Performance

robk — Tue, 24 Feb 2009 21:50:00 GMT

Moises Goldszmidt (above), principal researcher at Microsoft Research Silicon Valley, is showing a pair of demos, in conjunction with lab colleague Mihai Budiu, that examines performance in data centers.

"The challenge," Goldszmidt says, "is: How do I summarize thousands of machines and hundreds of metrics and find the key elements over that huge space that's giving us surprises, such that I can let it retrieve that fingerprint? How do I do that automatically?"

The demo is called Predicting Problems in the Data Center.

"We are using very sophisticated machine-learning techniques," Goldszmidt states, "that build automated models that are able to extract the main characteristics of each one of these crises."

The value of such work is readily apparent.

"Eighty percent of the time, we're predicting one hour in advance a set of actions we need to do to mitigate a problem," he says, resulting in "less downtime, less latency for our clients using our services. Our services are more efficient to run, because we don't have to have that many people look at the problem."

The second demo in the booth is named Profiling the Performance of Distributed Systems. It features a colorful analytics engine that enables the monitoring of vast data centers.

"Once you have a large cluster that you can run your services and applications on, it's very hard to understand, if something goes wrong, what's wrong," explains Budiu (left). "It could be a hardware problem. It could be your application has partitioned the data wrongly. It could be something in between, such as the network being down.

"This is a tool which pulls a lot of metrics from the machines in your cluster and allows you to easily visualize the data and find correlations in the data. You can assign colors to metrics and drag and drop the colors into other windows to see how the metrics correlate with other metrics.You can assign colors according to how many CPU cycles are utilized and immediately drop the color to see where the high CPU cycles are being used, in which machine."

In both demos, Goldszmidt declares, the objective is the same.

"Reduce the time to understand what the heck is going on," he says. "Build a better service. That's the final goal."

Introductory Keynote

robk — Tue, 24 Feb 2009 19:50:00 GMT

Today is Public Day, when a select group of partners, customers, and media get an opportunity to view a few dozen of the demos on display during TechFest 2009.

The day's events began promptly at 10 a.m. with introductory remarks from Rick Rashid, senior vice president of Microsoft Research, and Craig Mundie, Microsoft chief research and strategy officer.

Rashid began by noting the excitement in the air.

"TechFest for us is a really fun week," Rashid told the assemblage in the Kodiak Room of the Microsoft Conference Center. "We started it in 2001, and I was particularly surprised--because I was not really a big fan of the idea--at how much fun the TechFest event was, not just for the employees who would come in and see the technologies and the displays, but it also was a lot of fun for the researchers."

TechFest also gives researchers an achievable goal for which to strive.

"It's interesting," Rashid said, "that there's a huge competition among the researchers for actually getting into TechFest every year. It's probably reviewed more rigorously than most conferences and journals, because we try to make sure that we're going to get things that are interesting to our employees, that are interesting to the people who are outside."

In Mundie's opening remarks, he addressed the question of why Microsoft considers it imperative to pursue research, particularly in perilous economic times.

"It's oftentime companies that have doubled down on their R&D during that period that have really prospered on the way out," Mundie said, "in fact, have actually developed the products or technologies that tend to lead the economy or society forward in some new way."

And he made it clear that Microsoft looks to its research organization to help generate those innovations.

"The work that comes out of Microsoft Research is truly extraordinary," Mundie said. "There's really no company in the world today, certainly in the field of computing, that is spending what we spend on research and development. There are very, very few companies that are left that do any substantive pure research in this field at all. When you factor out the hardware world and look at software more exclusively, we clearly are way out in front in terms of our commitment to that investment."

During a Q&A session that followed the opening statements, Rashid was asked how Microsoft Research prioritizes its research.

"I don't prioritize research within the organization," he replied. "We don't try to do that. What we do is to try to prioritize people. It's a very people-intensive activity, and the goal is to hire the best people we possibly can, to give them an environment and other people that will allow them to be productive and to be able to do interesting work, and to set very high standards of performance for these people so we can continue to have the best group that we possibly can."

All with the goal of pushing forward the state of the art in computer science--and contributing to Microsoft products.

"Interestingly enough," Rashid added, "new technology transfers come out of TechFest every year."

Let's see what the candidates might be this time around.

Homecoming

robk — Tue, 24 Feb 2009 02:27:00 GMT

On occasion, TechFest, Microsoft Research’s annual technology extravaganza, seems just like Old Home Week. Researchers from the six Microsoft Research labs worldwide congregate in Redmond in the waning days of winter, renewing acquaintances, comparing notes, swapping stories.

It’s always an affable gathering. Familiar, smiling faces abound. The floor of the Microsoft Conference Center, where TechFest is held, buzzes with anticipation. For many involved in the show, this is the one time over the course of the year when they can touch base with a broad cross-section of talent from across Microsoft Research. The cross-fertilization of ideas is intense.

But for some, the event is brand-new. Such is the case this year with danah boyd.

A recent, high-profile hire for Microsoft Research New England, boyd (left) only started her new job about a month ago, having just completed her doctoral dissertation for UC Berkeley. But while she has never before attended a TechFest event, she certainly has heard tales.

“My understanding,” boyd says, “is that it’s an opportunity for all the researchers to show off what they’re working on, for a couple of different audiences. One is the company itself, and one is the public at large, all of the different people Microsoft works with and some academic audiences.

“But basically, it’s a very large science fair,” she laughs. “I love the science fair!”

Her research has gained renown for its exploration of social media, with a particular focus on how teens use social-network sites and on tensions between public and private data on such sites. While she’s sure to be discussing her work during TechFest, the event also is serving as a debut of sorts for not only boyd’s affiliation with Microsoft Research New England, but also for the lab itself, which opened in July 2008.

“My lab is a new experiment of people doing things from different kinds of fields,” boyd says. “I will be demoing with my group, and I’ll also be talking about my research agenda for where we’re going with social media, different kinds of possibilities of research, and the places of interesting intersections between ethnographically minded research and network modeling and more mathematical-theory modeling of large-scale networks.

“It’s a great opportunity to talk about where we’re going and to interface with all sorts of brilliant people around here.”

Brilliance certainly is abundant during TechFest. Research is, by its very nature, an experimental pursuit. Some research projects pay huge dividends, and some leads to a blind alley. That’s why it’s called computer science.

But, with a company with the breadth and scope of Microsoft, the successful technology Microsoft Research contributes can have monumental consequences.

During TechFest 2006, Rick Szeliski of Microsoft Research Redmond was the lead researcher on a demo entitled 3-D Photo Tourism. The project on display demonstrated how automated image-matching algorithms could be used to recover the 3-D position and orientation of the thousands of photographs of famous landmarks that had been posted on the Web. The positioned images could be combined to create a 3-D representation of the landmark.

That project, handed to colleagues at Microsoft’s Live Labs, eventually was released to the public as Photosynth. On Jan. 20, 2009, in a project called The Moment, CNN.com featured a Photosynth of the moment when new U.S. President Barack Obama raised his hand to take the oath of office. Hundreds of photos were submitted by users to contribute to the synth, and the result was, as Slate describes it, “an incredibly detailed picture of what took place that morning.”

From research project to participant in a presidential inauguration in less than three years—that’s evidence of the kind of powerful technology displayed during TechFest, and this year’s will be no different. While not all the work celebrated during the show will have such dramatic impact, some very well might. It’s all there for the discovering, and in this blog, I will strive to bring to you a representative sample of the innovation on display.

Blog posts, photos, links, interviews, random observations and occurrences: That’s what this space will offer over the next three days. Subscribe to the RSS feed. Bookmark this page. Visit the link on the TechFest 2009 home page. Monitor the excitement however you like, but please, keep checking back. I’ll do my best to make it worth your while.

And, in the meantime, if you’re interested or need a refresher, below are links to the postings on last year’s TechFest Live!

Roll the Credits

robk — Fri, 07 Mar 2008 06:35:00 GMT

Three hundred fifty computers. Two hundred twenty-five monitors, Almost a mile of computer cable. More than 6,500 attendees.

Almost seven months of preparation. Three days of technical setup. Four hours of tear-down.

About 500 researchers, most of them sporting forest-green, short-sleeved polo shirts with a white TechFest 2008 logo on the left sleeve.

"It's a huge coordination effort," says Kendall Martin, director of technology for Microsoft Research, as his team begins the process of unplugging the TechFest 2008 computers, coiling the cables, and storing the gear until next year's event.

"You're trying to get hundreds of individuals to work together, so it's pretty complicated: arranging the card readers, assigning the demos, getting the demoers the equipment they need. It's quite an orchestration. And it's not just the technical staff. The planning starts months out. It's just massive. Every single player ... the content selection, the worldwide tour ... The number of people who touch this is huge."

And, with luck, so is the payoff. James Oker should know. As director of Program Management for Microsoft Research, he heads the team whose responsibility it is to stage this annual celebration of the future of technology.

So, Jim, how'd it go?

"This is an awesome event," he responds. "I've talked to people ranging from senior executives to program managers and developers working on products I've worked with, and everyone was super-excited about what they saw here.

"I have a lot of follow-up to do with a bunch of people after the show that I think is going to lead to better Microsoft products in the next couple of years."

TechFest 2008: It's a wrap.

The Art of Theory

robk — Fri, 07 Mar 2008 04:09:00 GMT

OK, consider me dazzled. This afternoon, I got to spend a few minutes with Yuval Peres, principal researcher within Microsoft Research Redmond's Theory Group. In viewing the Theory Group's TechFest demos over the past few years, it has occurred to me that computational theory occupies a space where computer science intersects with philosophy. Maybe that's why that team always seems produce posters and animations that approach the nature of art.

Peres is an amiable, bespectacled man who speaks in the precise, clipped diction of one for whom mental meanderings would represent nothing so much as a waste of time and energy. He speaks in well-formed sentences woven into crystalline paragraphs of model clarity, This afternoon, he discussed his two TechFest 2008 demos--one entitled Algorithms and E-Commerce, the other Probability and Networks. But those mundane descriptions don't do justice to his irresistable, effortless elocutionary logic.

I warned Peres that he'd have to speak simply for me to have a chance to grasp the gist of his work, and he graciously complied. Instead of inserting my comments and interrupting the flow, I present his comments virtually verbatim. In the course of a few minutes, he managed to touch upon the techniques credit-card companies use to make money, college basketball's March Madness, and the complexity of fairness:

"The main two themes of our demonstrations this year are optimization and network formation. In optimization, we're emphasizing the difference between global optimization, when you're trying to optimize the social welfare for everyone, and local optimization, in which you have lots of individuals participating in the system and each one is maximizing their self-interest. You get very different solutions when these two mechanisms are in play, and we have some pictures that demonstrate the different solutions.

"This theme of local versus global optimization is also present in a classical problem we call the overhang problem, how far you can get a sequence of blocks to extend over a table, where the classical solution, which is obtained by recursive local optimization, is much weaker than the optimal solution you get by global optimization. We have a demonstration and even a game where people can try out the difference for themselves.

"The other thing we have is network formation, and that's very basic to a lot of what Microsoft does. It's understanding networks, from the Internet to distribution networks to social networks to telephone networks. They all have different structures. We're analyzing the basic structures of how networks form.

"If you look at a picture of a network, it often looks like a big disarray, but we now understand there are three key ingredients in the formation of a network. If you want to move away from the details and understand the basic themes, they are: underlying geometry, and that's very important in, say, a road network and less important in the Internet, though it still plays a role there. There is optimization, either by some network administrator or central authority, or optimization by individuals joining the network or taking part in the network. And the third ingredient is just randomness, or luck. These ingredients come in different doses, and they yield very different-looking networks. We have some striking examples of that in our booth.

"On the application side, besides understanding networks, we have several applications. One is monetizing social networks where, when you have a network, it's not a huge one, but it's already complicated, and it's changing very fast. The challenge is for the platform owner to monetize the network, to get the return on the investment in building the network, and to do that is more tricky than it seems at first, because if you do it the wrong way, you discourage users from joining your network. You need to understand where funding flows into the network, where it flows out of the network--and both things happen at many places--and make sure, as the platform owner, you only tax where money comes out--from people who sell stuff on the network, from developers--and not from users, not from people who bring money into the network.

"That's one basic insight that's common to many networks. A credit-card company will always put the charge to the seller and not to the buyer. They'll even sometimes reimburse the buyer. It's the same principle: You take money into the system from somebody who makes money from the system. Once you understand that principle, you still have to locate where money is going out of the system, and for complicated networks which evolve all the time, this is itself a challenge where understanding the structure of networks plays a role.

"Another application we have is a joint project with the VIBE group: fantasy-sports prediction. It's designed to help groups that participate in office pools. They make predictions on something like March Madness or other sports contests--or it can also be other competitions, even political contests, where people make predictions--and after you have their predictions in, you want to know what's the chance for each participant to win the pool.

"You could just go over all possibilities if it's a small pool, but in March Madness, with 63 games, you can't go over 2⁶³ possibilities. You have to do a Monte Carlo algorithm. There are many algorithms you could use, and the initial algorithm used took three minutes per query. After we got involved, together with the VIBE group, we have an algorithm that works in one second per query, so we had a huge improvement to make this thing practical."

Yuval Peres in front of a poster that provides a graphical depiction of the effects of global optimization. Best to let him explain it:

"This poster shows what arises from fair allocation done locally. You have a thousand points spread here, and each one is trying to obtain exactly the same area as all the rest--not more than their quota, exactly their quota. The way they do it is each one grows a [color-coded] disc around it, at unit speed, and it captures all the area it reaches first. Some points get all their area nearby, but some points that are hemmed in by others, they grow this disc, but here [he points to a place where a disc's shape is warped by the presence of a previously grown disc nearby], they don't get any area, because this area is already captured by this center. They have to wait, and they somehow resurface out here [he points to an area far from the nexus of the warped disc] and get their fill of area here. At the end of the day, they all get the same area, but some of them have to go very far.

"It's a very simple method. It doesn't require any kind of coordination by the different centers and no central authority, and it gives a fair allocation. But, as you see, the structure is very complex, and some individuals have to go very far to get their quota. What turns out is that this is the hardest to do, when you want a fair allocation. ... Our contribution is: If you consider this scheme with quotas, it yields this very complex picture. Fair allocation is great, but without central authority, it's complicated.

Tiny Web Services

robk — Fri, 07 Mar 2008 01:31:00 GMT

I mentioned the other day that I had run into Feng Zhao, principal researcher in the Networked Embedded Computing group, who was looking particularly happy at the moment. Well, Feng has been demonstrating the reason for his delight over the past three days: a set of small, wireless sensor devices, branded with the Microsoft Research logo, that he and his group plan to utilize in a variety of settings.

"This is an example of a sensor that we use for research and we plan to put into a data center," Zhao says, holding one of the devices up for inspection. "The sensors currently in data centers are a different form factor. This one just got designed. It senses temperature and humidity, the kinds of parameters one cares about in a data center."

The sensors are special, Zhao says, because of their size, their functionality, and their energy efficiency.

"We're working on making sensors easier to manage and interoperable with other devices," he says, "and to make the devices speak the kinds of language and protocols that computers on the Internet speak. For example, on the Internet, computers talk in terms of Web services, TCP/IP, HTTP. But these are designed for much bigger machines, and this one only has 4K of memory. We want to shrink these very big things onto this device with a very small memory and processor, and, furthermore, use as little energy as possible: two AA batteries for a year."

It's part of a research project called Tiny Web Services, designed to develop sensor nets and Web-service techniques that can fit onto these tiny 4K sensors. The sensors are programmed to go to sleep when not needed to send information.

"Web servers anticipate a set of requests, and they have to respond immediately," Zhao explains, "so they stay on all the time, waiting for requests. This one uses AC currency, and Web services can register events to make it turn itself on. A number of these techniques are being used to make the footprint of Web-service processing much smaller and simpler but still comply with the standards of the Internet so the device can talk to other devices."

Zhao's group is one of the first to address this area, and its devices are among the smallest--and least expensive--to talk in terms Web services can understand. Beyond their small footprint, they also contain technology that is advancing the state of the art.

"By building our own devices," Zhao says, "we actually are building a stack of software. We own all the drivers, and we can program. Our goal is to develop this into a reliable system, and we're also interested in making them available to academics, to help people building applications for the environment, for conservation, or for energy savings. If you look at what's currently available, some of those devices are not as reliable or as easy to program as we'd like."

With that, he was smiling again. These days, it seems, Feng Zhao just can't help himself.

Rick Rashid, senior vice president of Microsoft Research, shows one of Feng Zhao's new wireless sensors to an audience of invited guests during Tuesday's TechFest keynote speech.

The View from England

robk — Fri, 07 Mar 2008 00:49:00 GMT

Andrew Herbert, managing director of Microsoft Research Cambridge, was busily reading e-mail when I passed him in the hall a few minutes ago, so I doubled back and asked him to provide a brief statement about the value of TechFest for his lab and for the larger organization. His response:

"The value to us is two things. It's making contact with people here in Microsoft's Redmond product groups. Obviously, being based in Cambridge, it's not so easy for us as it is for our colleagues here in Redmond. The other great thing is to see what people in the other labs are doing with their research, seeing how we can join our ideas together to build even more fascinating technologies."

And for Microsoft Research as a whole and Microsoft as a company?

"It's great for Microsoft Research, because it really is the thing that drives our technology-transfer agenda. And I'm sure, for the company, it's great to show all the employees what Research is doing and some of the exciting things that we'll find in Microsoft products in the future."

Andrew Herbert, managing director of Microsoft Research Cambridge and a Microsoft distinguished engineer.