SmartArt - AD405 - Drew Browning

Interaction Design

Inputs, Outputs, and Processes or Listening, Speaking, Thinking

http://www.evl.uic.edu/drew/courses/AD405/Interaction1.htm

---

PD's interaction paradigms:

- HID (keyboard, mouse, etc)

- networking and communication protocols (osc, midi, dmx, etc)

- physical computing (microcontollers)

- computer vision

Computer Vision

Computer Vision for Artists and Designers

http://www.flong.com/texts/essays/essay_cvad/

- Traditional fields using computer vision: medical, military, industrial

- Proliferation of computer vision techniques in recent years due to:

- improvements in software tools for novice programmers

- the rapid growth of open-source code-sharing communities

- increased computer processor speeds

- lower costs of digital video hardware

- Examples of computer vision in interactive media arts:

- pioneering works of Myron Krueger and David Rokeby

- Other artists:

Golan Levin and Zachary Lieberman - Messa di Voce. (2003); http://www.tmema.org/messa/video/messa_ica1_jaapsolo_01s.mov

Marie Sester, Access (2005); http://www.accessproject.net/archives/ZKM_Interview.html

Threatbox (2007); http://www.threatbox.us/media/laboral_01.html

Scott Snibbe, Boundary Functions (1998); http://snibbe.com/scott/bf/video.html

Camille Utterback and Romy Achituv, TextRain (1999); http://www.camilleutterback.com/movies/textrain_mov.html

Jim Campbell, Solstice (1998); http://www.jimcampbell.tv/IN/INSolstice/index.html

Danny Rozin, Wooden Mirror (1999); http://www.smoothware.com/danny/woodenmirrormov.html

Zachary Booth Simpson/Mine-Control,Pond (2006): http://www.mine-control.com/pond.html

Damian Stewart / Frey, Wind (2009): http://frey.co.nz/wind

- Other works including: video dance performances, installations, multi-touch screen designs and the Reactable musical instrument.

- CV objects in GEM:

Detecting motion (frame differencing) – looks for movement – compares current frame to last frame - fails if people are stationary.

- pix_movement - motion detection using frame differencing compares adjacent frames

Detecting brightness (brightness thresholding) – looks for brightest pixels or pixels of a particular color - fails if the people are too close in color or brightness to the environment.

- pix_blob - brightness or color detection/tracking by analyzing adjacent pixels

- pix_multiblob - for multiple blobs

Detecting presence (background subtraction) – compares current frame to stored frame (background) – not good for active background or moving camera.

- pix_background - background subtraction - like motion detection but compares frame to previously recorded background

- pix_fiducialtrack - fiducial (target) detector (and tracker)

- pd-extended example patches:

- examples -> Gem > 04.video > 03.movement-detection.pd

- manuals -> 2.Image -> 17.tracking.pd

- Tutorials:

- Floss Manuals http://en.flossmanuals.net/PureData/GEMVideoTracking

- Bewegungsmelder http://web.uni-weimar.de/medien/wiki/Bewegungsmelder

- Lighting is critical! Strive for a high-contrast, low-noise input image. Carefully designed and controlled installation environment can allow use of simple CV software techniques.

- Infrared (IR) illumination improves the signal-to-noise ratio in low-light.

- retroreflective marking materials - reflecting light back towards their source - light is placed coincident with the camera's axis

- Camera - framerate (fast) and sensitivity (low light or IR)

- More sophisticated techniques including Gesture Recognition and Face Recognition can be found in CV libraries such as OpenCV (externals currently being developed for PD) and standalone applications such as EyesWeb (needs to communicate with PD through OSC).