Research Frontiers: New Stuff Coming in Pen & Multi-Touch Interfaces

Ken Hinckley — Sat, 05 Jan 2008 09:53:00 GMT

The annual Computer-Human Interaction (CHI) conference is fast approaching (April 5) and the advance program has been published. That means AlpineInker can scan the list of accepted papers and see what looks cool - but the actual papers aren't available yet, so it's a fun guessing game to read between the lines and try to figure out what the research community has been up to.

CHI is considered by many to be the flagship conference of the broad discipline of human-computer interaction, so it's important to be aware of what developments are coming. The papers are all rigorously reviewed and it's difficult to publish stuff there - typically only about 20% of the submitted papers get accepted in any given year.

I also learned last week that I'll be the technical program chair for the CHI 2009 conference. That means I just got dramatically busier (as if 11 month old twins weren't keeping me busy enough) and I probably will be posting to this blog a little less frequently than I originally planned to.

Anyway, here are my picks for what look to be some cool devices and techniques that we'll all hope to see become more commonly available as soon as possible.

Navigation Techniques for Dual-Display E-Book Readers

Authors: Nicholas Chen, Francois Guimbretiere, Morgan Dixon, Maneesh Agrawala (University of California, Berkeley - University of Maryland, College Park. Full disclosure: Francois, Nicholas, Morgan and I have co-authored papers together; Maneesh is a former Microsoft Researcher who has also co-authored papers on pen stuff with me.

Imagine the Amazon Kindle meets the Nintendo DS - only much cooler. This is a project that Francois' lab has had going for a while, and from earlier demos they've done, I know it's a device with a pair of displays that are joined together by a hinge. So you can read stuff just like you would with a real book that has two pages facing one another. But you can also flip the two displays back-to-back so you have a two-sided UMPC type of device, or you can even separate the two screens at the hinge so you have two independent devices that you can use like separate sheets of paper, with different documents on them, and they stay synchronized using a BlueTooth connection. They've hacked up each of the displays with a motion sensor that lets you flip through pages much like you would leaf through the pages of a paper document. I would love to have a dual-screen UMPC like this that I could ink on - it would be the ultimate digital moleskine notebook. It will be very interesting to see what new developments they have come up with for this paper.

Update: Their web server has been having problems, but since the original post, I've been able to access their site briefly. They have a page about the dual-display ebook reader project and a video (WMV) showing how the motion sensing works.

I also found a few images. Here is what their original prototype looked like in the dual display configuration:

And here it is with the pages pulled apart:

Finally, here is a tantalizing shot that apparently shows the new version of their reader. It looks like they have gotten it dramatically thinner and lighter. I'm also excited to see the pencil next to it and what look to be small writing styluses (or is that styli?) attached to the bottom of each screen - maybe that implies we will be able to do inking on this thing some day!

In fact, they say that they prototyped this using a 3D printer to make parts of the case and a carbon-fiber composite bezel to really keep the weight down. It's amazing what a few clever university students can pull together these days.

Update #2: This seems to be an extremely timely bit of research given the announcement today of the e-detail Dual Display Tablet PC:

The e-detail product suggests another great scenario for the University of Maryland dual-display e-Book: set it up like a picture frame so one display faces each person in a face-to-face meeting. That would be fantastic for a highly strategic one-on-one meeting, such as playing "battleship" .

K-Sketch: A "Kinetic" Sketch Pad for Novice Animators

Authors: Richard Davis, University of California, Berkeley; James Landay, University of Washington, Intel Research Seattle. Full disclosure: I've co-authored with James Landay in the past.

Wouldn't it be cool to be able to set your ink and sketches on your tablet PC in motion? In theory it's possible to do this using ink with PowerPoint's animation features, but in practice it can really difficult to throw something together quickly. Well K-Sketch has some nifty ideas and techniques embodied in a fairly clean and easy-to-use sketching interface that makes it possible to do this super fast -three times faster than you can do similar things in PowerPoint. Now that sounds like fun.

This is one that you can soon play with yourself, according to k-sketch.org:

1/3/2008: We are very close to finishing our initial release of K-Sketch, and we expect it to be ready within a month.

K-Sketch will allow ordinary computer users to create informal animations from sketches. [...] Our design allows the most important types of motion to be defined with pen gestures, and gives visual feedback for coordination of events.

A video of of K-Sketch (WMV format) shows some of what it can do. The presentation is tailored for the academic audience, but the capabilities are very cool. Be sure to keep watching until the ski jumper double flip animation - try to imagine how tough it would be to do that in PowerPoint!

Indirect Mappings of Multi-Touch Input Using One and Two Hands

Tomer Moscovich, University of Toronto , Brown University; John Hughes, Brown University.

I don't have any inside scoop on exactly what this paper is about, but Tomer has done some very cool multi-touch interaction techniques in the past. He's also the person who came up with the "virtual scroll ring" technique that is the basis of the circle-the-pen-to-scroll gesture that has been a big hit with the tool ring in InkSeine. So I'm very keen to see what clever new things Tomer has devised in the multi-touch space.

Multi-Flick: An Evaluation of Flick-Based Scrolling Techniques for Pen Interfaces

Dzimity Aliakseyeu, Philips Research Eindhoven, Netherlands; Pourang Irani, University of Manitoba, Canada; Andres Lucero, Eindhoven University of Technology, Netherlands; Sriram Subramanian, University of Bristol, U.K

Scrolling with a pen, virtual scroll ring notwithstanding, is something that begs for cool gestures that let you easily express different granularities and speeds of how to flip through your documents. From the title I'm inferring that these researchers have come up with a few different kinds of flicking gestures that let you navigate through stuff quickly and easily. They don't have the paper posted on their web site yet, but it looks like they are implementing it using a pen on a tabletop computer (kind of like the Microsoft Surface). They have about a half-dozen other papers - one with a video - of other interesting work they've done on tabletop devices.

"It's on my other computer": Computing with Multiple Devices

David Dearman, University of Toronto; Jeff Pierce, IBM. Full disclosure: Jeff Pierce and I have published together before, and are both former students of Randy Pausch, who is waging a now-famous losing battle with pancreatic cancer. But he's hanging in there so far and still making great contributions to society.

All of you mobile enthusiasts out there have struggled with keeping your mobile devices and computers in sync, only to have some key bit of information elude you at exactly the moment you need it. Jeff Pierce had some cool projects going when he was a professor at Georgia Tech on "Personal Information Environments" where his vision was that you could walk around with your cell phone or notebook computer and use opportunistic annexing to dynamically associate your device with displays, touch-screens, keyboards, or other devices you encountered. My inference is that he's made some nice progress in this space and has come up with some techniques to maintain easy access to all your stuff across multiple devices.

PieCursor: Merging Pointing and Command Selection for Rapid In-Place Tool Switching

George Fitzmaurice, Justin Matejka, Azam Khan, Mike Glueck, Gordon Kurtenbach, Autodesk

This is the same crew of guys who came up with all the interesting user interface stuff in the Alias Sketchbook application, which I still use to this day for sketching on my Tablet PC. It looks like they've come up with a spiffy way to switch between different tools with your pen on your Tablet PC without having to move the pen to the edge of the screen and tap on some silly icon or bring up a menu. There's an intriguing thumbnail image of the technique on Azam's web site:

Thumbnail of PieCursor from Azam's web site - can you guess how it works?

My best guess is that it is some kind of a floating menu that follows your pen. Then you can activate a radial menu on it to pick tools (like a Zoom function), but you can continue the same stroke to specify how much to zoom. This is a trick known as "merging" in some other papers, hence my guess based on the title of their paper. This could be something really cool to let you pick different tools and commands much faster on your Tablet PC.

Those are the highlights from my short list of what caught my eye, but there's well over 100 papers to appear at the CHI conference and I'm sure there's many other gems in there. For example, there will be a whole session on image search interfaces, including CueFlik from Microsoft Research. I'm not involved in that work but I've seen that demo in the labs here and it's pretty cool. Image Search is one area where I think Microsoft Live Search has a clearly superior user interface to Google: for example, try a Live image search for Tablet PC. CueFlik is something that could make this even cooler and more useful.

The Inaugural AlpineInker Post

Ken Hinckley — Sat, 29 Dec 2007 10:20:00 GMT

I’m Ken Hinckley, a senior researcher at Microsoft Research in Redmond, Washington, where I conduct research on new techniques and approaches to human-computer interaction. This is a longwinded way to say that I try out cool new stuff, throw it at the wall, and see if it sticks. Usually it doesn’t, but that is what makes my job fresh and fun every day.

The AlpineInker is a new forum where I will discuss developments in pen, touch, mobile devices, and anything else in my field that strikes my fancy. I’m blessed? jaded? by a researcher’s perspective, so anything I say here is personal opinion and does not represent the Microsoft corporate perspective. My thoughts and ideas should be taken with a big grain of salt, particularly if you work in a cubicle, and buy, sell, manage, or process anything as part of your daily work. But if you want to see fun stuff, are open to new ideas, and want to see some of the thinking that goes into the kind of research that I do, then this might be the place for you.

At least to start, I’ll be posting things here somewhere between every so often and when I feel like it, so subscribe to the RSS feed if you don’t want to miss anything. But when I do get around to it I plan to make substantive posts with something worth reading, rather than reporting the banal trifles of my daily life. But be forewarned, it’s likely that I won’t be able to resist occasionally posting photos of my irresistibility cute identical twin daughters (currently 11 months old), so I recommend wearing extremely dark cute-polarized sunglasses whenever visiting the site, just to be on the safe side.

My main project right now is a prototype inking application known as InkSeine. It is designed for the Tablet PC and UMPC’s (ultra-mobile PC’s). It is not a Microsoft product. That means we have free reign to try out whacky ideas without being forced into the regression-to-the-mean compromises dictated by mareting, focus groups, and the pragmatic realities of shipping a large commercial product.

The goal of the InkSeine project is to innovate the user experience for pen-operated devices. InkSeine has fantastic search functionality built-in that makes it easy to find information that you need off of your hard drive or by searching the web. This makes it really fun to quickly throw together mixed-media notes with links to files, web pages, documents, and emails, snapshots from stuff you’ve seen, and of course your notes, sketches, and annotations mixed in with all of this. It is kind of the ultimate designer’s notebook on steroids. And it is also the winner of the prestigious Third Annual Life On The Wicked Stage Ink Blot Award for the “Best Tablet PC Software That Microsoft Is Holding Hostage and Needs To Be Released Today!”

To avoid winning this award again in the future, we will be making InkSeine available as a free download from research.microsoft.com in the first quarter of 2008. Stay tuned for news and developments on that front as we make the final push to get it out the door. You’ll hear about it here first.

Got things you want me to discuss? Nifty projects I should comment on? Ideas for the coolest new things you'd like to see in future user interfaces? Well post a comment or zip me an email and I'll take a look at it.

The AlpineInker : Multi-Touch

Research Frontiers: New Stuff Coming in Pen & Multi-Touch Interfaces

The Inaugural AlpineInker Post