Tom and Hayden have been working on grabbing Kinect data, which is basically data about human movement, and sending it out across the network. The graphics server (an Alienware laptop) reads the data from the network and uses it to render an Augmented Reality scene, which is projected up on the wall.

The Kinect is a very cool Microsoft device which was designed for the Xbox 360. It's great for mapping realistic human activity into a 3D model in realtime. It has two cameras onboard - one infrared camera for reading depth data, one webcam-style camera for getting a regular video image matrix.

Tom downloaded the Microsoft SDK (there are others, for example OpenNI/NITE), which you can use to transpose the depth data onto the video. The result is a video image with identifiable humans overlaid (see the image below).

The Microsoft SDK also contains software to take this further. By analysing data from the camera streams, and using algorithms designed to find human body parts (such as limbs and torsos), the SDK can build an overlay of 3D 'skeletal' data. This means that it works out the points where it thinks the human body parts extend to and draws lines between them to make up a stick man ('skeleton').

Tom has written a small C# program which uses the SDK to grab skeletal data points and broadcast them across the network. We are using 3 Kinects, each of which can handle 2 skeletons reliably, making a total of up to 6 actors that can interact with the installation.

Hayden has created an OSC server on the graphics server to pick up the skeletal data points from the network. He then set up a test person renderer (the stick man on the wall, above) in VVVV so that we can fine-tune the Kinect interaction before passing data on to the real 3D patches.

Data from the 3D scene are in turn formatted and broadcast across the network as more OSC messages, for the sound environment to pick up. The result will be an audio/visual environment which can be controlled solely by moving in front of the Kinects.