This is a collaboration with music professor Michael Theodore. Michael is interested in using signal processing and manipulation techniques to enhance composition and live performance (both musical and visual). Particular techniques of interest include:
Separating Musical Tone from Noise
The goal is to allow separate processing of musical tone elements and noise elements (as they are percived by listeners). A key example would be the ability to separate "bow noise" for a cello from the musical note generated. A goal is to develop techniques and implementations that can operate and be manipulated in real time as part of a live performance. Michael is working with a commercial product called MAX/MSP as well as a similar open source project pd. In addition to artist control of signal processing, Michael is also interested in using sensors to "automatically" influence signal processing. Applications go beyond artistic, though, and potentially include therapeutic. One particular approach would involve spectral analysis in which the phase differences in successive DFT frames are used to differentiate signals.
Video Processing
Real time evaluation and manipulation of image, graphic, and video content both to interact with audio content (see previous section), such as using the presence and morphology of edges to affect audio, and to change the video content. Requires real time video processing and user interface manipulation. There is a growing interest in "Painterly Rendering", or "Non-Photorealistic Rendering" (NPR) as a computer graphics research topic. The area of real-time NPR is of particular interest for its potential use in live performance settings. The Max/MSP/Jitter programming environment provides for the modular construction of object-oriented "patches" which work with real-time video and audio. Custom objects can be written in C and then plugged into the visual environment. See also Craig Reynolds' site and Brown University site.
Kinesthetic Input
Combines audio-video maniuplation with dance by incorporating measurement of the dancer's body position to influence the audio-video (ranges from a glove to full body measurement). Michael colloborates with Michele Ellsworth.
Sound Stretching
Attempts to dilate or compress sounds in time while preserving perceptual integrity. Also used by the navy, for example, in attempting to render sonar signals in a manner that optimizes detection and analysis by human operators. Michael beleives this requires clever phase processing (e.g., evaluating and aligning the phase at critical points in the original and stretched signal).