daniel bierwirth

The idea is to have the phone hanging around your body - I'm holding the phone in my hand because this was a first demo shoot to prove that it works.

Instead of a phone, you can think of any other device that is able to sense the off-screen input. Simply hang it around your body and send the tracking results to the application phone.

By tracking the application phone and the users hands, it is possible to use the users spatial memory to interact with applications.

There are several reasons to explore the possibility of interacting in off-screen space. Let me clarify some of the approaches.

1. Using this approach for back-of-device interaction

As some might guess, this approach might fit to the idea of interacting with the back of devices as explained in Literature [See ACM Library].

But why did I use two phones?

Currently, it is not possible to have direct access to the IPhone's camera stream and to do the off-screen sensing while displaying another application on the same phone.

Certainly, the question about off-screen sensing for back-of device interaction is very interesting as you can read at the ACM Library. This way of interacting with devices will have a great feature as many devices will get smaller or even diminish i.e. referring to ubiquitous computing. That topic is discussed widely and raises the question about how to interact with small screen-spaced devices.

Think of a smart watch - How would you interact? As you can see: Touching the screen will not work.

2. Tracking the user's off-screen interaction

Talking about the maps application:

Let's think of a tourist from the US that does a trip to Europe. You are in Spain, for example. Even you might not have much insights about Europe, you will still know that Germany is somehow north-east from Spain.

How do you plan your trip from Spain to Germany? Open a maps application; Your application phone is (0,0) in the coordinate system. You could pinch in off-screen space where you are right now + pinch where you want to go. That way, you are enabled to navigate in a maps application without having to scroll the map on the small screen.

There are much more scenarios you could think off...

This is a project with the Human Computer Interaction Lab at Hasso Plattner Institute.