The AlpineInker : Hardware

The Microsoft Research Codex: Are Dual Screens the Future of Mobile Devices?

Ken Hinckley — Wed, 01 Oct 2008 10:17:00 GMT

Never buy one of anything. That's advice you should stand by when you're buying unusual gadgets. The advice was good when Randy Pausch offered it to me some 15 years ago, and it's still good now.

Of course, with 18 month old twin girls at home, this has become second nature to me. Two boxes of diapers. Two gallons of milk. Two Elmo plush dolls.

Oh, and yes, of course. Two screens for my tablet computer.

Dual-screen devices have become the subject of increasingly elaborate internet fantasies. Now, rumors about an Apple "Brick" device have stirred up dreams of future tablets, such as the alluring One Laptop Per Child v2.0 concept photos, which now orbit the internet once again.

My interest in dual-screen devices goes back a lot further, though, and was really spurred on by the University of Maryland dual-screen e-book reader project. That effort is led by Francois Guimbretiere, who is a long time collaborator and friend. I had some ideas to build on what his team had done, but also to take things in a different direction.

First of all, I wasn't interested in an ebook reader. I wanted a device that was all about writing. Sure, reading and writing go hand in hand - you encounter cool ideas and search out reference material on the web-but what I wanted to build was a tool for thought. To me that means a tool with writing, sketching, and annotating as the core of the experience.

I'd been thinking for a long time about picking up an OQO Model 02 computer. My team has an extensive code base for pen-and-tablet functionality resulting from InkSeine, and the OQO runs it out of the box. It's got an active digitizer for high fidelity pen input, and it's the smallest slate Tablet PC that money can buy. That's a pretty good start on a small form-factor tool for thought.

So when it came down to it, just how many of those OQO Model 02's do you think I purchased?

Project "Codex" was born.

Twice the Screen at Half the Size

Here's the Codex packed up for mobility. It folds up quite nicely and has a moleskine-style knitted elastic strap to hold it securely shut. There's a loop for the pen and a mesh pocket so you won't lose small accessories, business cards, or receipts that you collect in your travels.

Sure, it's a bit of a brick at present. The whole thing weighs just over 2 pounds. The OQO's are considerably thicker than I'd like. But my goal is to prototype the future as quickly as possible and start living it. The OQO offers a handsome, jet black time-travel machine in that regard.

Don't underestimate the ability to quickly pack up with the screens protected. Folding the Codex in half makes it comfortable to carry and easier to stuff into my gadget bag. It's a self contained kit for ultra-mobility.

The Codex is Not a Container for Dead Trees

A codex is just an archaic term for a bound book. But this Codex is unlike any book that you've ever read. It's not a long linear text that you flip through. To me, there's no use in going to all the trouble to build a dual-screen tablet prototype and write elaborate software just to mimic a traditional book. This is the 21^st century and it's about time we moved past containers for dead trees.

We have lots of information sources. We have multi-tasking. We have hyperlinks. We have split attention. We have a left brain and a right brain and we rarely do one thing at a time any more.

A Separation of Concerns

The Codex has two screens, it's designed to be used that way, and you won't find any half-apologetic demos that try to mash them back together into one big screen. Instead, it's all about the intelligent partitioning of tasks and interface elements across the screens.

Above is one example where I'm working on a blog post that I've had planned for a while. On the left I have a whole bunch of cool photos that I found tagged with moleskine on Flickr. I was browsing through these as inspiration for our InkSeine digital note-taking software. On the right I'm organizing bits and pieces from these photos along various themes. So I just take a snapshot from the collection on the left screen and it appears in my notes on the right screen, where I can arrange it and mark it up as I see fit. I can scroll back and forth on the left screen to find a photo that meets my current needs, while the page that I am authoring on the right screen always remains visible. The two screens are invaluable because I always have the reference material in the context of what I am working on, instead of feverishly flipping between them on a single screen.

Can I do this on a single large screen? Well, sure, I could monkey with the window placements and get everything arranged just so. But that takes a lot of effort and the temptation to expand windows to take over the full screen is hard to resist if I have to expend effort to do so. A dual-screen device that understands the partition between the screens gives a much simpler experience where I don't have to constantly manage the set-up of the windows.

Navigate without Losing the Big Picture

Here's another example. I'm further along with authoring my blog post now, and I have a bunch of material floating around in my notes. I create a page that is a Table of Contents, with links to several themes that I've identified in my Flickr moleskine investigations. If I open a link, such as my Creative Collage page, it opens on the opposite page. I don't even need a "back" command to return to where I was - I still have my navigational structure on the left, side-by-side with my content page on the right.

Unlike a traditional dead-tree book, I have no physical restriction that forces me to view consecutive pages - The Codex lets me follow links or flip through the screens separately to view any two pages together.

Working Big and Small

We can draw some inspiration from traditional media as well. In magazines and books, sidebars are a distinct section of a page that augments the main text with auxiliary information. Well, the Codex has sidebars on steroids. I can take any chunk of my notes, make it into a sidebar, and then arrange a bunch of these on a page. Here, I've made a storyboard page consisting of six sidebars. When I tap on a sidebar in one of the storyboard cells, it expands to full size on the opposite screen. I can plot out the broad sweep of my story on one screen, while maintaining full access to the zoomed-in details on the other.

Your Mother Was Right - Posture is Important!

Perhaps the coolest property of a dual-screen device is that the compact, mobile form-factor encourages shifting the device around. I can orient the screens with respect to one another, stand the device up, look at the screens in portrait or landscape, and so forth. In fact, the Codex supports about a dozen different configurations. We call these postures. (A colleague, Michael Miller, coined this term).

Add two cups of accelerometers, a dash of flex sensors, and bake with some simple software to fuse it all together. Out of the oven pops an intelligent dual-screen display system that configures itself depending on how you arrange it.

For example, if I want to work in landscape, I can flip the device into the laptop posture, as opposed to the book posture that I've been showing so far. Now I have one screen that's angled for easy reading, plus another screen that's horizontal for easy writing.

Or I can use the laptop posture to hook up one of my screens to a projector. I put the public part of my presentation on the top screen, while the controls to drive the presentation and my private notes are confined to the bottom screen.

Now perhaps I meet a friend in a café and I want to show him what I'm up to. I can just lift up on the binding of the Codex to angle the screens so that one is facing me and one is facing my friend (below, left). We call this the battleship posture - as in "You sank my Battleship!" Each person has one screen with a private view. The Codex automatically configures the software for shared whiteboarding so the collaborators can mark up the screens, pass notes back and forth, and other such foolishness.

Next I drop the binding back flat. Now both of us can view the screens, but one screen is oriented towards each person.

Don't Get Too Attached

Something that turned out to be surprisingly useful is that the Codex allows me to pull each screen right out of the binding. A firm pull pops it out, a firm press pops it back in. This is really handy for laying out the screens to suit my work, or to review a video while I jot down some notes. I can even hand the screen to another person to show them something - without actually giving them an electronic copy of the information.

What? You Still Have a Desktop Computer?!?

The Codex is just another wireless device so of course I can connect it to my desktop computer. Now, any screen capture that I take from my desktop screen shows up in my Codex notes right where I left off. For web pages, the snapshot comes across with a hyperlink back to the source, so I can easily revisit it later when I'm reviewing my notes on my Codex. This makes it the perfect companion no matter how I'm working.

A Parting Shot at Text Entry

The Codex is designed around doing most things with a pen, but I'm no idiot. When it comes to text entry, a keyboard is a good thing.

And that, in part, is why I'm puzzled that so many of the dual-display concept designs that I've seen take up a whole screen with a virtual keyboard. I'm sure people in focus groups ask for this. But I still think it's a really bad idea. What is the point of having two screens if you are instantly going to cover one of them with a picture of a keyboard? Your hands immediately occlude it from sight anyway. At that point I'm basically using a single screen again, so I might as well just grab my laptop to get a decent keyboard.

The Codex offers a solution for text entry that I much prefer, by virtue of the OQO Model 02's mechanical keyboard. I just slide the screen up and that reveals the keyboard when I need it. Works great. No dorky touchscreen keyboard. I still enjoy the full benefits of partitioning my work between the two screens. That's the way I like it.

When Can I Buy One?

The Codex is a prototype-- and a rather flaky, cobbled-together one at that. But it uses off-the-shelf devices, and there's nothing magical about the software. So the crass answer is that you can have one now if you are willing to spend some dollars, build yourself a custom binder, and write a little bit of code. That's how I started. My first prototype was a repurposed day-planner with Velcro holding the screens in there. Install a shared clipboard utility and you can start copying and pasting between screens. That will give you just enough of the experience that you will hunger for more.

What would you want out of a dual-screen device? What capabilities would make it most useful to you? How do you see dividing your own work between two screens?

In the meantime, the trajectory of ultra low-power "e-ink" displays bears watching. Check out the recent Plastic Logic device, for example. Right now e-ink is an abomination for anything interactive, but eventually some display technology will get where we want it to go. Low-cost, low-power screens are crucial to make dual display devices a practical consumer device, rather than a research lab curiosity.

The sheer number of concept devices that have popped up in the last 6 months suggests that dual-screen devices are poised to take off in the near future. My hope is that our research on the Codex can help in some small way to unearth the full promise of such devices. I, for one, am convinced that dual-display devices have a well-motivated role to play in a future ecosystem of mobile devices.

In my office, at least, that future is already here. It's just not evenly distributed.

Acknowledgements

I would like to thank the following key contributors to this project:

Morgan Dixon
Raman Sarin
Francois Guimbretiere
Ravin Balakrishnan

Some Thoughts on Automatic Screen Rotation

Ken Hinckley — Thu, 08 May 2008 08:41:00 GMT

Automatic screen rotation has been popularized by the iPhone but is also available on the OQO Model 02 thanks to Kenrick's Automatic Screen Rotator Utility (executable download). Just hold the device the way you want to use it. The screen flips to the correct portrait or landscape orientation in one second. You don't even have to think about it. What could be simpler? Kudos to Kenrick for putting this great utility together and making it available for free!

This is a program near and dear to my heart. I cobbled together custom sensor hardware, including an accelerometer to support automatic screen rotation, for my old Cassiopeia E105 Pocket PC back in the late 1990's:

Here's our video of the Sensing Pocket PC, with screen rotation and other fun stuff too.

Video: Sensing Techniques for Mobile Interaction

Later, I built sensors for the original slate Tablet PC prototypes that were floating around Microsoft. Many devices now include accelerometers for drop detection, but I'm pretty sure my prototype was the world's first Tablet PC with an accelerometer. It came with an extensive user manual: TILT ME.

That wad of electronics on the top is my sensor module. Here, I'm using the Tilt-a-Sketch application. You could draw on the tablet like an Etch-a-sketch by tilting it back and forth. Yes, it was really hard to sign your name this way, and yes, if you flipped it upside down and shook it, it erased the screen. Accelerometers can be a lot of fun.

But what was the most useful? Like Kenrick's utility, it supported automatic portrait/landscape switching depending on how you held the device. After all that hard work I had to put into building my own sensors, firmware, and software, it's mind-blowing to see this available in a free utility that I can download from the 'net for an off-the-shelf device!

My demo had a few tweaks, some never published before, that might be useful future embellishments to Kenrick's Automatic Screen Rotation utility. In essence these tweaks reduce accidental changes to the display orientation when you're working with your device. They also help to avoid rotation of the screen when you go to set your device down on your desk. Plus there's one bonus idea I tinkered with, described at the end - let me know if you like it or not.

Dead Bands for Increased Stability

Dead bands between the screen orientations made the device tend to stick to the current display orientation. This helped to avoid accidental changes to orientation.

Plot of tilt angles versus inferred instantaneous screen orientation.

To change display orientation, the tilt angles had to pass all the way through the gray ±5° dead bands, and stay within the same display region for 0.5 seconds. No screen rotation occured in the central "Flat" area.

Rotation Preview

Feedback for impending display rotations makes automatic changes to the display orientation more predictable and controllable. My Tablet PC demo displayed a "THIS SIDE UP" arrow at the center of the screen as soon as the tablet was tilted in a different direction. The change to the display format occurred one second after the arrow appeared, but only if the device was still tilted towards the new display orientation. This allowed the user to stop tilting the device to prevent an inadvertent switch.

To foreshadow a change to the display orientation, an arrow appeared immediately when the user rotated the tablet.

Of course, the arrow should not be there all the time. The arrow vanished when:

The screen changed orientation. The arrow remained visible for a couple of seconds after the switch to provide continuing feedback.
The instantaneous screen orientation returned to the current display orientation for a couple of seconds. This case occured if a user acted on the feedback to avoid an accidental change.
The user set the Tablet down flat without changing screen orientation.

I used black color-key transparency (in a layered window) for the actual bitmap used in the code.

Motion Detection

Movement of the device serves as a secondary indicator of when to switch the display format. To avoid accidental changes to the screen orientation, my Tablet PC implementation waited for motion to stop before rotating the screen. For example, this made the device less likely to change screen orientations as you set it flat on a desk.

The OQO Model 02 supports only about a 4 Hz sampling frequency on the accelerometer, so it might not be feasible to implement good motion detection at present. Nonetheless it seems worthwhile to mention it, in the hope that an increased sampling rate becomes possible in the future.

One hack to detect motion is to calculate how much the tilt values are changing, as follows:

Δx = tiltX - prevTiltX

Δy = tiltY - prevTiltY

sampleEnergy = √(Δx² + Δy²)

signalEnergy = signalEnergy*(1-α) + sampleEnergy

In the final equation, α is a decay rate. I used 0.25, with the tilt values in degrees, and signalEnergy initialized to 1.0. Motion "begins" when the signal energy rises above an onset threshold for a few samples and "stops" when the signal energy drops below a termination threshold.

Movement helped to control switching of the display format as follows:

When movement stopped, if the physical screen orientation did not match the inferred instantaneous screen orientation, a 1 second time-out began, after which the software switched the physical display orientation.
If movement began again during this time-out, the time-out for the physical display switch was cancelled.
If the instantaneous screen orientation changed again during this time-out, the time-out was restarted at its full one-second duration.

Orientation-specific Tasks

Here's one other nutty idea I experiemented with. Maybe it's useful, maybe it's not.

I assigned specific applications to specific orientations of the screen. For example, here's a screen shot where I set up Excel to appear in the landscape format, and Windows Journal in the portrait format. Flipping my Tablet PC between the two would switch between the applications, rather than just rotating the screen.

Switching orienations can switch between sets of applications as well...

This offered a simple way to partition applications into task-specific sets for each screen orientation. Unfortunately my prototype of this feature never really worked all that well. You could check off windows as belonging to each screen orientation. The prototype would hide and show the windows as you rotated your tablet. But it had some bugs. Sometimes it would hide the windows permanently, never to be seen again. That's not terribly useful. So I never did usability testing on it, but I found something about it intuitively appealing. What do you think? Would you want this feature on your tablet or mobile devices?

Summary

My experience is that the devil is in the details with sensing techniques. Small touches here and there go a long way to keep the interaction invisible in the background, rather than becoming a focus of attention when things happen that the user didn't intend. Ultimately, the goal should be to create the best possible user interface. What is the best possible interface, you might ask?

The best possible user interface is the one that you don't even notice is there at all.

Resources:

Kenrick's blog: Teaching, Technology, and Learning
Kenricks' Automatic Screen Rotator Utility, download for the OQO Model 02
Ken Hinckley's UIST 2000 conference paper and follow-up journal article on sensing techniques.
Video of the Sensing Pocket PC

Portrait Inking on the OQO Model 02

Ken Hinckley — Mon, 21 Apr 2008 10:19:00 GMT

The OQO Model 02 is almost the same size as my Moleskine Pocket Sketchbook. I suspect this is no accident. To illustrate the point, I scanned them side-by-side. The OQO is slightly narrower, which is necessary to make it fit in my shirt pocket given its 1" girth. By the way, don't let this scan fool you - the screen on the OQO is gorgeous. It's just really hard to scan properly. The other photos below give a better sense of what the screen really looks like.

I set up a custom cover page for my OQO in InkSeine to make it feel just like a new moley fresh out of the shrink wrap. Now I feel like writing important stuff in here.

I also scanned my pocket Moleskine to use for the inside pages. I love having this page style on the OQO - it just seems right.

I prefer inking on the OQO Model 02 in the portrait orientation. I can grip the device more comfortably in this orientation, and there is more room to plant my hand on the screen. This also keeps the touch-scrollers out from underneath my hand. I've experimented some with using the "secondary portrait" orientation, to flip those touch scrollers over to my left hand. That feels great, but since the keyboard rotate function only flips between the primary landscape and primary portrait orientations, it's inconvenient to go to the options panel and hunt for the command to flip to the secondary portrait orientation.

There's one other tip I have for working in the portrait orientation on the OQO's small screen. I was thinking about why it seemed easier to draw in my pocket Moleskine, even though it has nearly identical dimensions as the OQO. It's not so much the small screen size of the OQO, as it is the thickness.

So I slide out the keyboard, and I rest the meat of my palm on that. This feels more like resting my hand on the desk while I draw in my (thinner) pocket Moleskine. The OQO keyboard keys are fairly stiff so I never trigger them by accident while I'm doing this. Typically I do this while holding the OQO in my left hand; the photo below shows me doing this on the desk because I was out of hands to hold the camera, and no tripod was handy :-)

The keyboard is also convenient for hitting the Enter key, modifier keys, or the special OQO hardware hotkeys (such as the screen rotation, brighteness, and keyboard backlight) when the occasion demands.

That closes the book on this post. I'm sure I'll have more thoughts and ideas about using the OQO as I continue to work with it.

Related Posts:

My very first impression of the OQO Model 02

Make a faux-OQO to see if the size is right for you

Dig for Tablet PC Ultra-Productivity with the INTELLIMOLE

Ken Hinckley — Fri, 18 Apr 2008 10:38:00 GMT

A conversation with the Tablet PC MVP's this week reminded me of a productivity hack I constructed for my tablet a while back. I guarantee that you will either absolutely love this hack, or think it is the stupidest thing you've ever heard. In my experience, there is little gray area when I mention this idea to people.

It's no secret that buttons are in short supply when you're working with a tablet PC, particularly in the slate mode. Most tablets offer a paucity of buttons along the bezel. But even tablets that do have decent bezel buttons infuriate me because nearly all tablets place them on the right side of the screen - the same side where some 75% of users are holding the pen. So I have to fumble with the pen to use the buttons. Why they are not on the left by default is beyond my ken.

I do use the barrel button on my pen, but only begrudgingly so. It's a bit awkward, I hit it by accident, and it often messes up my pen strokes even when I do intend to hit it.

I was digging around for alternate solutions to this dilemma. I realized that I had to get everything off of the pen and tablet.

My solution? Kick that tablet into high gear with the INTELLIMOLE.

The INTELLIMOLE peeks out Tablet PC productivity running out of gas? Then step on it!

Here's what you'll need:

A wireless mouse, and a willingness to commit bodily harm to it.
A foot switch.
A soldering iron.
Black electrical tape.

Rip open your mouse (unscrew it if you are feeling humane) and just wire up the foot switch to the contacts for the wireless mouse's right-click button. Use the black electrical tape to cover up the optical mouse eye so it won't disturb the pen's cursor position. Throw the footpedal under the desk, and just make sure that the wireless mouse and the receiver are within range of one another. This is what my completed INTELLIMOLE kit looks like:

Now you can stomp your foot to right click whenever you like, without interfering with your pen or tablet. Use the Control Panel settings for the mouse to reprogram the right-button click to some other function if you like.

The downside is that you do have to plug the wireless mouse receiver into your tablet's USB port, so it's really only useful while you're using your tablet on a desk.

For that foot switch, if you want the best experience, I strongly recommend purchasing a round one. That way it doesn't matter which way it is oriented when you go to step on it. The Linemaster GEM V3 switch is a good choice, albeit a bit pricey at $50 from Allied Electronics. They do also have an assortment of cheaper ones (that aren't round).

Perhaps Tablet PC designers will finally take pity on us one day and sprinkle a button or two along the left edge of that tablet bezel as well. I'd dearly love to have a programmable "magic wand" button there that would be available for tablet PC applications to use as they saw fit.

Until then, I'll continue to tunnel through the netherworlds of Tablet PC productivity with the INTELLIMOLE.

Other Posts in the AlpineInker's Tablet PC Ultra-Productivity Series:

Faux-OQO with origami

Ken Hinckley — Fri, 04 Apr 2008 10:17:00 GMT

It's been a very busy couple of weeks for me, so I haven't had time to attend to the blog, or much of anything else for that matter.

But one thing that has been on my mind from time to time is the OQO Model 02 Ultra-Mobile PC (UMPC). It's an intriguing device but I have wavered for some time now on whether or not it is the right device for me, and whether it will offer the right fit for the projects I have in mind for it. I don't know anyone who owns one, so taking one for a test drive has not been an option.

The active digitizer on the device is extremely appealing to me. Obviously, I want to use InkSeine on it. The passive touchscreens that I've tried on other devices are rather unsatisfactory for inking.

But yeah, I keep hearing the screen is small on the OQO. But how small is it? The entire device measures 5.6" by 3.3" and is 1" thick. It weighs just one pound. Some people like the OQO's diminutive stature because they can slip it in their pocket. Others don't much care for it, because their big meaty hands just cannot write on such a small screen.

Got it. But how big is that really? How would that feel to me? Would it be useful in my nutty research project ideas? I just can't decide!

What to do, what to do?!?

Finally I could stand the waffling no more. I had a bit of time today, so I made a scale model "OQO origami" by using InkSeine and PowerPoint together. Then I printed the origami, cut it out, and folded it up into my own little "Faux OQO" device. I taped it to some cardboard to make it a little more rugged, and I carried it around with me during the day. It looks surprisingly realistic. Maybe, just for kicks, I should try to fool my co-workers with it. Tomorrow I'll have to pretend to drop it down the stairs, or fumble it off the railing of the four-story atrium that we have here in the new Microsoft Research building.

This bitmap probably won't print out to scale, but my OQO origami PowerPoint will. I've attached the file to this post so you can join in the fun. It's attached as a .zip file because I can't directly post .ppt files on this blog. Just open up the zip and you'll see the ppt file in there. The attachment also appears as a link at the very end of this post.

Now you can make your own Faux OQO and truck it with you in your travels. See if it stirs your gadget lust as well. Better yet, have some fun and pretend you are using it in meetings, on the bus, or on the subway. Fake out some gullible rubes. Go ahead.

If they get mad when they discover your ruse, just tell ‘em the AlpineInker made you do it.

Soup Up Your Tablet PC to be Ultra Productive

Ken Hinckley — Thu, 06 Mar 2008 10:47:00 GMT

I am having some great fun this week modding up one of my slate Tablet PC's. I have a nutty little idea that I am working towards, but mums the word on that for now... Either it will take the Tablet PC world by storm, or I will prove once and for all that I am a complete kook. Perhaps even both!

While I was thinking about this top-secret project, I rediscovered a great Channel 9 video. In the video Bert Keely shows how he has modded up his slate, delivered with his typical firehose of enthusiasm. Bert is an architect on the Tablet PC team and has been there since the very start. Even though this video is a few years old now, his advice is still bang-on, and his enthusiasm gets me excited about the Tablet PC all over again.

I could find nary a mention of this video, save to ask about it, in any the forums, haunts, or other dark corners of the blogosphere frequented by Tablet PC Illuminati. So I've uploaded the video to MSN Video and now you can play it right here in your browser (or get the Channel 9 MSDN Video download).

The video player that I am able to embed in this blog is a little lame. Apologies for that. Click on the link immediately below if you'd rather view it in MSN Video instead - that offers a better experience in my humble opinion.

View Video
Format: wmv
Duration: 03:35

Video: Bert Keely on Souping Up the Tablet PC (2004)

Bert is only just scratching the surface of tablet PC mods that are possible. Cestfiu has been posting about his quest to find the ultimate combination of buttons'n'stuff for his Lenovo, while e-gadgetjunkie seeks the ultimate stealth pen button settings. FeralBoy had the stroke of genius to throw small rubber door wedges in his travel bag, to prop up his tablet as needed.

Tablet stands are another popular "mod," but I have two really big beefs with those:

(1) Most importantly, they don't involve cutting, epoxying, melting, dremel-tooling, or otherwise risk bodily harm, carcinogenic vapors, or fatal impacts to your tablet; and

(2) They typically aren't suitable for my mobile kit. But here's a recent stand that looks useful, if not mobile.

If you're not just mobile but ultra-mobile, be sure to deck out your UMPC with a cool case and stuff.

While you're at it, don't forget to mod up your pen with felt-tip nibs and spring-loaded stroke nibs. Ordering these from Wacom is only slightly more difficult than gnawing your own foot off. But at least Mobile Maestro Steve S over in the GottaBeMobile forums has kindly posted the magic link.

I hope you have great fun with these ideas. Be sure to report back on your most clever, useful, or downright extravagant tablet mods!

Maybe you'll even soup up your tablet with something hotter than what I've got cooking on the back burner here at Microsoft Research...

PREVIOUS: Post #1 in the AlpineInker's Tablet PC Ultra-Productivity Series!

NEXT: ???

IOGear Mobile Digital Scribe – Yet Another Digitizer Pen?

Ken Hinckley — Tue, 04 Mar 2008 08:33:00 GMT

I just stumbled across the IOGEAR Mobile Digital Scribe GPEN200N announced today, which strikes me as yet another digital pen capture technology. But I’d love to be proven wrong and discover that it is that perfect magical solution that everyone seems to be waiting for.

IOGEAR claims this is the “first device ever to capture natural handwriting from any surface, and store it in the receiver for future use” but I believe that about as far as I can throw my gigantic 21” Wacom Cintiq display tablet. That thing weighs a metric ton and is connected by a cable suitable for a boat anchor, so trust me, it’s not very far.

The main advantage of this gadget is that it writes on ordinary paper (once you clip the receiver to it) and the stylus uses a standard ballpoint pen refill. But there have been many products in the past that employed ordinary paper, such as the ill-fated CrossPad. The

EPOS / DaneElec Z-Pen is another recent example, but from what I've heard on the GottaBeMobile forums discussion of digital pen solutions, people have had trouble actually getting their hands on one of those. So maybe this is the "first one that you can actually buy as a consumer, since the CrossPad, which didn't actually store stuff in the pen but kind-of gave the same user experience." That doesn't sound quite a slick in the marketing brochure though, I'm sure.

The IOGEAR web site is pretty thin on any technical detail as to how this thing actually works, but according to another story the Mobile Digital Scribe is tracked via an infrared sensor. I also found no mention of the critical detail of how the pen knows that you’ve flipped to a new page of notes. Pens based on the Anoto technology employ a special paper encoded with a unique identifier so that the computer knows which page you’re writing on.

Don’t sell the page-identification feature of the Anoto technology short. It is essential for trouble-free note taking.

On the old CrossPad, for example, you had to press forward/back page buttons, and if you forgot to press the button, or made a mistake somewhere along the way, all of your notes henceforth until you sync’ed with the computer would be seriously messed up, with pages mixed together willy-nilly. I would bet good money that the IOGEAR device has this same problem. If they've figured out a way to solve that without special paper, then I will be first in line to plunk down my hard-earned cash for this thing, but I rather doubt that's the case.

To me trouble-free note taking is well worth the price of special paper. Without special paper, I just don’t see any way to solve this issue, technically. If you have to stop and tap madly on some page turning button or whatever every time you want to flip back a few pages in your notes to make a quick correction, it will quickly drive you insane. I did this in the old days on my CrossPad, and look what happened to me. You have been warned!

The Agilix Capturx device is a recent example of an Anoto-based technology that for my money appears to be a better way to go. It avoids the page-turning problem and offers tight synchronization with Microsoft OneNote.

The IOGEAR Mobile Digital Scribe does come with software to download your strokes, convert handwriting to text, and also seems to have a real-time streaming mode if you use it while it is connected to your computer. I also wouldn’t be surprised if there is some way to set it up to get the stroke data into OneNote, but it doesn’t appear to have the nice integration that the Capturx does.

A 30-second Hardware Hack to Make You Ultra-Productive on Your Tablet

Ken Hinckley — Thu, 07 Feb 2008 12:03:00 GMT

A while back I came up with a hardware invention to make me more productive with my tablet when I'm using it on a desk. I figured why not give a boost to the sagging global economy by unleashing this little gem on the world now, even though it's still at what we might call the "early prototype" stage.

Here's everything you'll need to send your productivity off the charts:

Block off 30 seconds in your day-timer for construction. I also recommend closing your door and telling your secretary to hold your calls because you'll really need to concentrate.
An optical mouse.

The Microsoft IntelliMouse Optical is a good choice because you can use it in either your right or LEFT hand (more on this soon), but use whatever you have handy.

A Post-It Note.

A real paper one. You might have to special order one of these because I hear people don't use paper anymore these days.

Scotch tape (can get by without it in a bind).
A marker and/or highlighter.

Optional, but more fun. Personalize your device and show off your craftsmanship to jealous co-workers.

Flip the mouse upside down like a turtle.
Stick the post-it over the optical mouse eye on the bottom and tear off any excess paper. Don't stare at the light!

a. Use a little scotch tape to help secure it in place during heavy use.
b. I've written "NO CURSOR!" on my Post-It with a heavy black marker. It was more fun that way and I really wanted to tell that mouse take its little pointy arrow and go stick it somewhere.
Use another Post-It to add a decorative touch to the top. This makes your INTELLISCROLLER look different from other mice. A crude scotch-tape job on the curved surface also makes it bumpy, giving your INTELLISCROLLER it a nice tactile distinction as well. Here's what mine looks like:
Now, if you're right handed, point with the pen in your right hand while you use the INTELLISCROLLER in your LEFT hand for scroll, right click, left click, and double click. Since the buttons are new separated from the pen, you'll get much less interference.

a. This is where having the optical eye covered comes into play. This "mouse" won't accidentally disturb your pen position while you use INTELLISCROLLER to click, scroll, etc.
Use the Control Panel for the mouse to customize all of its buttons.
a. I have the wheel-click on my INTELLISCROLLER mapped to CTRL. This makes it a breeze to use my pen for CTRL-tap multiple-selection or CTRL-drag to copy files.
b. At times I've also had it set up to switch to the next application when I click the wheel. This way, you won't miss ALT-TAB so badly when you're using your pen. I've decided CTRL is more useful to me though.

I'm thinking about proceeding directly from this prototype to the "INTELLISCROLLER 2.0." That one will instead use a small notebook-sized wireless 5-button mouse. That would make it easier to lug this thing around as part of my mobile kit. If truth be told, my INTELLISCROLLER has been getting left out of my travel bag more often than not. Having yet another wire in there, which inevitably ensnares everything else in the pouch into one gargantuan Gordian knot, is a major limitation of this first prototype.

On a 5 button mouse, I'm thinking I'd probably use one of the extra buttons for Ctrl (to get it off the wheel button- it's too easy to roll while clicking, which sometimes leads to errors) and the other I might use for switching to the next app window, or maybe SHIFT, or something else?

Now in all honesty, I am not really expecting this device to become an overnight internet sensation. But there is a real grain of truth to its utility.

Why is such a thing useful, even if marginally so? Well, with all the standard apps that don't really treat a digital pen with the respect it deserves, actions like right-clicking, Ctrl-clicking, or double tapping are critical to be able to get things done quickly.

And as we all know, these can be really frustrating at times when you're using a pen.

Now I'm well aware that there are standard options in the control panel to customize this stuff, set up the pen button for right click, or to use tap-and-hold to activate context menus. But these things are band-aids to the fundamental problem that a pen doesn't have good buttons in the first place. On a mouse, you can press the buttons without disturbing the cursor position. On a pen this is just not possible. No matter how you set up all those crazy options, to some extent they interfere with the pure joy of knowing that every time you set your pen to the screen, you will leave digital ink, AND NOTHING ELSE.

I hope there is a more clever solution to all of this, and maybe ultimately it means that every app there ever was really should be re-written to understand the pen better.

But for now the INTELLISCROLLER is the most salient symptom of my desperation.

Update: sumocat over on the GottaBeMobile comments thread recommends:

pick up a gaming keypad, like the Belkin Nostromo, to do this. It's already designed to do these things and can also execute commands like cut, copy, and paste.

Well I think that's a great suggestion. I hope to get myself one of these and try it out! But I'm a little bit sad to hear that the INTELLISCROLLER 2.0 may already be obsolete before it could hit the market. On the other hand, a hidden virtue of the 30-second hack is that it is really easy to rip the NO CURSOR! Post-It off of the INTELLISCROLLER if you find you need an actual mouse (in emergencies only, of course).

NEXT: Post #2 in the AlpineInker's Tablet PC Ultra-Productivity Series!

Research Frontiers: New Stuff Coming in Pen & Multi-Touch Interfaces

Ken Hinckley — Sat, 05 Jan 2008 09:53:00 GMT

The annual Computer-Human Interaction (CHI) conference is fast approaching (April 5) and the advance program has been published. That means AlpineInker can scan the list of accepted papers and see what looks cool - but the actual papers aren't available yet, so it's a fun guessing game to read between the lines and try to figure out what the research community has been up to.

CHI is considered by many to be the flagship conference of the broad discipline of human-computer interaction, so it's important to be aware of what developments are coming. The papers are all rigorously reviewed and it's difficult to publish stuff there - typically only about 20% of the submitted papers get accepted in any given year.

I also learned last week that I'll be the technical program chair for the CHI 2009 conference. That means I just got dramatically busier (as if 11 month old twins weren't keeping me busy enough) and I probably will be posting to this blog a little less frequently than I originally planned to.

Anyway, here are my picks for what look to be some cool devices and techniques that we'll all hope to see become more commonly available as soon as possible.

Navigation Techniques for Dual-Display E-Book Readers

Authors: Nicholas Chen, Francois Guimbretiere, Morgan Dixon, Maneesh Agrawala (University of California, Berkeley - University of Maryland, College Park. Full disclosure: Francois, Nicholas, Morgan and I have co-authored papers together; Maneesh is a former Microsoft Researcher who has also co-authored papers on pen stuff with me.

Imagine the Amazon Kindle meets the Nintendo DS - only much cooler. This is a project that Francois' lab has had going for a while, and from earlier demos they've done, I know it's a device with a pair of displays that are joined together by a hinge. So you can read stuff just like you would with a real book that has two pages facing one another. But you can also flip the two displays back-to-back so you have a two-sided UMPC type of device, or you can even separate the two screens at the hinge so you have two independent devices that you can use like separate sheets of paper, with different documents on them, and they stay synchronized using a BlueTooth connection. They've hacked up each of the displays with a motion sensor that lets you flip through pages much like you would leaf through the pages of a paper document. I would love to have a dual-screen UMPC like this that I could ink on - it would be the ultimate digital moleskine notebook. It will be very interesting to see what new developments they have come up with for this paper.

Update: Their web server has been having problems, but since the original post, I've been able to access their site briefly. They have a page about the dual-display ebook reader project and a video (WMV) showing how the motion sensing works.

I also found a few images. Here is what their original prototype looked like in the dual display configuration:

And here it is with the pages pulled apart:

Finally, here is a tantalizing shot that apparently shows the new version of their reader. It looks like they have gotten it dramatically thinner and lighter. I'm also excited to see the pencil next to it and what look to be small writing styluses (or is that styli?) attached to the bottom of each screen - maybe that implies we will be able to do inking on this thing some day!

In fact, they say that they prototyped this using a 3D printer to make parts of the case and a carbon-fiber composite bezel to really keep the weight down. It's amazing what a few clever university students can pull together these days.

Update #2: This seems to be an extremely timely bit of research given the announcement today of the e-detail Dual Display Tablet PC:

The e-detail product suggests another great scenario for the University of Maryland dual-display e-Book: set it up like a picture frame so one display faces each person in a face-to-face meeting. That would be fantastic for a highly strategic one-on-one meeting, such as playing "battleship" .

K-Sketch: A "Kinetic" Sketch Pad for Novice Animators

Authors: Richard Davis, University of California, Berkeley; James Landay, University of Washington, Intel Research Seattle. Full disclosure: I've co-authored with James Landay in the past.

Wouldn't it be cool to be able to set your ink and sketches on your tablet PC in motion? In theory it's possible to do this using ink with PowerPoint's animation features, but in practice it can really difficult to throw something together quickly. Well K-Sketch has some nifty ideas and techniques embodied in a fairly clean and easy-to-use sketching interface that makes it possible to do this super fast -three times faster than you can do similar things in PowerPoint. Now that sounds like fun.

This is one that you can soon play with yourself, according to k-sketch.org:

1/3/2008: We are very close to finishing our initial release of K-Sketch, and we expect it to be ready within a month.

K-Sketch will allow ordinary computer users to create informal animations from sketches. [...] Our design allows the most important types of motion to be defined with pen gestures, and gives visual feedback for coordination of events.

A video of of K-Sketch (WMV format) shows some of what it can do. The presentation is tailored for the academic audience, but the capabilities are very cool. Be sure to keep watching until the ski jumper double flip animation - try to imagine how tough it would be to do that in PowerPoint!

Indirect Mappings of Multi-Touch Input Using One and Two Hands

Tomer Moscovich, University of Toronto , Brown University; John Hughes, Brown University.

I don't have any inside scoop on exactly what this paper is about, but Tomer has done some very cool multi-touch interaction techniques in the past. He's also the person who came up with the "virtual scroll ring" technique that is the basis of the circle-the-pen-to-scroll gesture that has been a big hit with the tool ring in InkSeine. So I'm very keen to see what clever new things Tomer has devised in the multi-touch space.

Multi-Flick: An Evaluation of Flick-Based Scrolling Techniques for Pen Interfaces

Dzimity Aliakseyeu, Philips Research Eindhoven, Netherlands; Pourang Irani, University of Manitoba, Canada; Andres Lucero, Eindhoven University of Technology, Netherlands; Sriram Subramanian, University of Bristol, U.K

Scrolling with a pen, virtual scroll ring notwithstanding, is something that begs for cool gestures that let you easily express different granularities and speeds of how to flip through your documents. From the title I'm inferring that these researchers have come up with a few different kinds of flicking gestures that let you navigate through stuff quickly and easily. They don't have the paper posted on their web site yet, but it looks like they are implementing it using a pen on a tabletop computer (kind of like the Microsoft Surface). They have about a half-dozen other papers - one with a video - of other interesting work they've done on tabletop devices.

"It's on my other computer": Computing with Multiple Devices

David Dearman, University of Toronto; Jeff Pierce, IBM. Full disclosure: Jeff Pierce and I have published together before, and are both former students of Randy Pausch, who is waging a now-famous losing battle with pancreatic cancer. But he's hanging in there so far and still making great contributions to society.

All of you mobile enthusiasts out there have struggled with keeping your mobile devices and computers in sync, only to have some key bit of information elude you at exactly the moment you need it. Jeff Pierce had some cool projects going when he was a professor at Georgia Tech on "Personal Information Environments" where his vision was that you could walk around with your cell phone or notebook computer and use opportunistic annexing to dynamically associate your device with displays, touch-screens, keyboards, or other devices you encountered. My inference is that he's made some nice progress in this space and has come up with some techniques to maintain easy access to all your stuff across multiple devices.

PieCursor: Merging Pointing and Command Selection for Rapid In-Place Tool Switching

George Fitzmaurice, Justin Matejka, Azam Khan, Mike Glueck, Gordon Kurtenbach, Autodesk

This is the same crew of guys who came up with all the interesting user interface stuff in the Alias Sketchbook application, which I still use to this day for sketching on my Tablet PC. It looks like they've come up with a spiffy way to switch between different tools with your pen on your Tablet PC without having to move the pen to the edge of the screen and tap on some silly icon or bring up a menu. There's an intriguing thumbnail image of the technique on Azam's web site:

Thumbnail of PieCursor from Azam's web site - can you guess how it works?

My best guess is that it is some kind of a floating menu that follows your pen. Then you can activate a radial menu on it to pick tools (like a Zoom function), but you can continue the same stroke to specify how much to zoom. This is a trick known as "merging" in some other papers, hence my guess based on the title of their paper. This could be something really cool to let you pick different tools and commands much faster on your Tablet PC.

Those are the highlights from my short list of what caught my eye, but there's well over 100 papers to appear at the CHI conference and I'm sure there's many other gems in there. For example, there will be a whole session on image search interfaces, including CueFlik from Microsoft Research. I'm not involved in that work but I've seen that demo in the labs here and it's pretty cool. Image Search is one area where I think Microsoft Live Search has a clearly superior user interface to Google: for example, try a Live image search for Tablet PC. CueFlik is something that could make this even cooler and more useful.

The Inaugural AlpineInker Post

Ken Hinckley — Sat, 29 Dec 2007 10:20:00 GMT

I’m Ken Hinckley, a senior researcher at Microsoft Research in Redmond, Washington, where I conduct research on new techniques and approaches to human-computer interaction. This is a longwinded way to say that I try out cool new stuff, throw it at the wall, and see if it sticks. Usually it doesn’t, but that is what makes my job fresh and fun every day.

The AlpineInker is a new forum where I will discuss developments in pen, touch, mobile devices, and anything else in my field that strikes my fancy. I’m blessed? jaded? by a researcher’s perspective, so anything I say here is personal opinion and does not represent the Microsoft corporate perspective. My thoughts and ideas should be taken with a big grain of salt, particularly if you work in a cubicle, and buy, sell, manage, or process anything as part of your daily work. But if you want to see fun stuff, are open to new ideas, and want to see some of the thinking that goes into the kind of research that I do, then this might be the place for you.

At least to start, I’ll be posting things here somewhere between every so often and when I feel like it, so subscribe to the RSS feed if you don’t want to miss anything. But when I do get around to it I plan to make substantive posts with something worth reading, rather than reporting the banal trifles of my daily life. But be forewarned, it’s likely that I won’t be able to resist occasionally posting photos of my irresistibility cute identical twin daughters (currently 11 months old), so I recommend wearing extremely dark cute-polarized sunglasses whenever visiting the site, just to be on the safe side.

My main project right now is a prototype inking application known as InkSeine. It is designed for the Tablet PC and UMPC’s (ultra-mobile PC’s). It is not a Microsoft product. That means we have free reign to try out whacky ideas without being forced into the regression-to-the-mean compromises dictated by mareting, focus groups, and the pragmatic realities of shipping a large commercial product.

The goal of the InkSeine project is to innovate the user experience for pen-operated devices. InkSeine has fantastic search functionality built-in that makes it easy to find information that you need off of your hard drive or by searching the web. This makes it really fun to quickly throw together mixed-media notes with links to files, web pages, documents, and emails, snapshots from stuff you’ve seen, and of course your notes, sketches, and annotations mixed in with all of this. It is kind of the ultimate designer’s notebook on steroids. And it is also the winner of the prestigious Third Annual Life On The Wicked Stage Ink Blot Award for the “Best Tablet PC Software That Microsoft Is Holding Hostage and Needs To Be Released Today!”

To avoid winning this award again in the future, we will be making InkSeine available as a free download from research.microsoft.com in the first quarter of 2008. Stay tuned for news and developments on that front as we make the final push to get it out the door. You’ll hear about it here first.

Got things you want me to discuss? Nifty projects I should comment on? Ideas for the coolest new things you'd like to see in future user interfaces? Well post a comment or zip me an email and I'll take a look at it.