Augmented Reality, TUIO, Computer Vision & Kinect

May 14, 2012 · Posted in Augmented Reality · Comment 

Recent research AR, Tuio, Kinect

In the 2011-2012 academic year I was a visiting artist at he School of Visual Arts MFA Computer Arts Department. I gave a series of workshops around the themes of TUIO (Tangible User Interface Output), Augmented Reality, Computer Vision, and Using the Kinect Camera.

I used the workshops as an impetus to research these areas and formulate an understanding of how to work with them in the Art context. TUIO is the metaphor that is replacing the mouse and keyboard interface of computers.
It ranges from eye movement recognition to touch tables and screens to gesture recognition. It employs computer vision techniques to recognize the gestures.

Augmented Reality superimposes a virtual camera view on a live video stream. The placement of the virtual camera view in the video stream is located using a simple marker such as a QR code or an AR Marker. Geo-Location is also used and presents the idea of a site specific virtual art.

The Kinect Camera is a game controller for Xbox that has been hacked so that anyone can use it. It contains a infrared projector, an infrared camera, a video camera, four microphones and a servomotor. The infrared projector/camera gives depth readings. This presents many possibilities for exploring 3D scanning, live 3D video, 3D interactivity etc..

I look at the Kinect as the object creator/sculpture tool. The approach to space perception in live video is 3D. This give artists a new tool/approach to working with real space and virtual space.

A bit of history

I started working with computers in the 1990’s. I was especially interested in the idea of working with the internet as an art tool. I was one of the first artists to use the internet and the web as a communication/distribution tool. I have also been interested in creating installation art that is interactive in some measure, depending upon both the computer and physical components. I find that using computers in concert with physical interfaces enhances whatever experience you are trying to convey. At this moment in time with the widespread deployment of cellphones and ubiquitous computing the network is becoming integrated into reality. The question is what does that do to our sense of reality and the world.
I feel that this is the domain of the artist.

Theoretically, it should now be possible to utilizes 3D space in a new way that is not just, dance or sculpture or installation but is enhanced and extended into a new form using Augmented Reality. I use that term in a broader idea of reality with the addition of a networked information layer.

My understanding of digital tools is that they allow you to dissect perceptual reality into it’s components and examine each component. When applied to a creative process this produces a new form of perceptual reality and a new field for making art. The job of the artist is to use these forms and root them in the continuum of art history in a dialog. It has always been a basic discourse in art that the tools and techniques, the craft of art, is only one component. A complete art work engages art history, the personal passions and interests of the artist and is an expression of human consciousness and culture at any given moment in time.

My investigations at the moment are centered around defining this digital terrain.
Here are a few of my areas of investigation.

Geo-Location – most digital devices have a locative aspect. They give out the longitude and latitude and elevation of the device. This is accomplished through GPS or cellphone towers or router information in the case of computers on the internet.

Markers or Tagging – this allows access to the information network using various types of markers or tags. This can be a QR code or an RFID chip or proximity tag.

Video space – most PDA’s cellphones etc have video incorporated into them. This creates a McCluhanesque media sphere.

Programming environments – essentially a coding space. Approximately 50 programming languages are in use today. There are references to over 2,500 dialects. Languages and programming environments are designed to solve specific problems or are used as an organizing principle. This becomes almost a stylistic genre when applied to art. For example there are subcategories of Javascript artists or Flash artists. These are artists using specific programming environments.

Interactivity and gesture recognition – there are two parts to this. The type of physical gesture and what it might trigger or what does it mean in the network.

Screenic environment – This is the live screen of smartphones and iPads. I think of this as interim interface before the HUD (Heads Up Display) becomes ubiquitous.

HUD – the screenic environment or information layer is integrated into our body/daily reality. This is the mode for putting an information layer over reality.

3D mapping – using 3D tools to superimpose a 3D information layer onto our physical reality in the form of either a projection or an AR layer .

I’m also reading William Gibson’s Spook Country which has an artist using geo-location to tag AR image/animations that can be seen with a helmet device. Of course Google is testing a prototype AR glasses that uses Android platform. It seems to me that all the cellphone, iPad devices are just a middle step.

In any case, I find AR to be very intriguing. Starting with the idea of geo-location and geo-tagging, this roots whatever artwork presented to a place. In Gibson’s book, the artist is making a ghost tour around LA with images of deaths & suicides at their original location. You can see these “re-enactments if you have a device. Immediately I like the notion of “site-specific” disembodied artworks.
Another aspect to AR is using AR Markers. this can be QR codes or textures or custom markers. This begins to set up a new way of discussion both sculpture and painting (2d image) vis a vis networked culture.
My research thus far is trying to find the most flexible AR development platform. I’ve been looking at ARToolkit which is open source but has been sold to developers. There’s an open source (free) version that many people use for marker recognition. The ARToolkit libraries are portable to a number of platforms. There’s also several side projects that use this. For example, ARToolkit marker detection has been compiled for use with Quartz Composer.