So far I’ve only talked about multi-touch enabled computers as representative devices with NUI, but not just smartphones and tablet computers belong to this category.
Besides the possibility of using the touch screen there are approaches to the control a computer via body movements. For example the Kinect for Xbox, which can detect body movements and gestures using the integrated camera, belongs to this category of input devices. To recognize the player the Kinect uses besides a color camera a depth sensor camera, sus the person can act in three dimensions.
It is also possible to recognize the movements of a person using a normal camera . This basic technique was used in the EyeToy for Playstation 2. In this technology the current webcam picture taken is compared with the previously captured image. The differences between the images reflect the movement in front of the webcam. With a high rate of captured frames per second (FPS) a high movement detection accuracy is guaranteed.
This simple method of motion detection can be used with any webcam on the PC in order to realize a simple control via body movements.
In the following example game, this method of motion detection is used. The player has the simple task of catching the falling diamonds shownon the screen with his hands. The differences between the previous and current image, which are diyplayed in green, are used to detect a contact between the hand of the player and a diamond.
Since this technique is a purely optical method, which also depends on the quality of each camera image and the FPS, the motion detection is optimal only in (good) lit rooms.
Motion detection via Webcam
Click to play in a larger window.
Another form of control can be done via accelerometers. Accelerometers are sensors to measure the accelerations and movements, and processed to determine the location of a device in three dimensions.
The acceleration sensor converts the pressure fluctuations inside the sensor, caused by an acceleration, into electrical signals. Due to the continuous measurement of the movements it can be determined in which position the device is located at any time. Newer sensors provide acceleration values and in addition their position in relation to gravity. Due to this additional reading the position of the device can be measured even more accurate.
An acceleration sensor responds only in one dimension with a maximum sensitivity, so one is required per axis.
This form of position determination is nowadays used in almost every smartphone and tablet computers to align the screen. Since these new devices can be rotated by the user in every possible direction it is important to recognize when the device is held upright (portrait) or rotated horizontally (landscape) so that the content is best displayed on the screen.
But even in games industry this technique is used for control. Thus, in the Nintendo Wii controller is also an accelerometer integrated, so that the player, for example in car racing games, only needs to move the controller in the appropriate direction to control his character.
Also voice recognition and control, as Siri on the iPhone 4S, is a new method for natural user interfaces and how to control them, but this technique is still in its infancy, so I will not discuss it in this post.
Upcoming blog entries
Making the Game – Part 5: Multi-Touch Framework (Adobe Flash / Air)
Making the Game – Part 6: Multi-touch devices and their limits
…