InnerAction
Interactive Music and the Inner Self
November 07, 2006

	A recent arrangement of my gear aimed at hardware sequencing.

When I am interacting with my music gear, I am continually trying to determine the tonal relationships of different sounds. It might be as easy as tuning two instruments some distance apart so that together they create the sonic character that I have in mind. Other times it might be more complicated, like trying to fill in a tiny bit of spectral emptiness, not quite knowing what to tweak, but searching for it through the adjustment of controllers. Of course, this is something I can only do with my ears; its nearly impossible to make these connections by looking at the instruments. Even the way I am working with the devices may need some adjustment, and perhaps I need to move things around physically so that I can play a number of synthesizers and modules at the same time. This is something I can figure out with eyes; by looking at the mobility of my hands, and how quickly and easily I can reach each knob in combination with some other control. Sometimes I lose track of what I am doing for a few minutes, focused on some particular sound, until something goes wrong and I am reaching for an owner's manual. Other times everything works just perfectly, and I am tweaking away in the studio for many hours, psychologically in the "zone" and completely oblivious to what time it is. There is a harmony going on, a resonance between the gear and myself abling my body to function on auto-pilot. Using modular components, I have assembled an interactive device that is quite capable of putting me in a relaxed trance or altered state of consciousness.

As a sound designer, multimedia artist, and sculptor, I am very interested in creating unique works of art that enable users to achieve entrainment states. I am particularly enthusiastic in designing interactive musical instruments that do not require the user to traverse through a period of learning. For example, a musical instrument that is played simply by moving ones hands across a pool of water. Not only does this enable new users to immediately get in the zone with the music, but it also opens the instrument up to children, and people that are not able to play an instrument, because of some physical or mental disability. At the same time, I am interested in creating the tools for users to be able to transform the terms (or state) of their consciousness.

	A water touchscreen designed by Mitsubishi Electric Research Laboratories. The triangular shapes swarm about in response to movements of your hands in the water. The shapes behave very much like a school of fish.

In this slightly unusual article, I discuss how consciousness relates to interactive devices, both on an individual scale, and in the field of interactive entertainment. Then, I will examine interactive music and how it can be used to achieve states of "entrainment." I will look at some compelling examples of interactive music devices designed by well known artists. Finally, I will pinpoint some alternative controller mechanisms that can be used to build new interactive music devices that are easy enough for non-skilled musicians to operate.

Awareness

In an effort to provide hard scientific explanations, neuropsychology describes awareness in terms of "the coordinated actions of many neurons" and "biochemical states" (Cook p.189) without considering that the answer may eclipse the limitations of physical measurement.

René Descartes was one of the most influential philosophers of the 1600's. In searching for a foundation for new philosophy, he refused to accept ideas based on physical senses and opinions. He wanted his claims to be completely certain. After peeling back every bit of conjecture and reasoning about his existence, the only thing he was left with was "I think, therefore I am." As an idealist, I believe there is no separation between imagination and being, and no division between our physical bodies and the things we dream. Our thoughts create the world we know. Belief is the lens through which our actions are projected into physical matter, tangible symbols of our own conclusions. In a tight interactive loop we create, interpret and react.

	In many religions, trees symbolize the soul, heaven, earth, rebirth and transformation.

According to conscious creation theory, outside of our perception is a hidden world containing the greater portion of our being: our inner self. Our present personality, like a leaf at the end of a branch, looks outward, unable to see the tree that it grows from, unable to remember the tree's immensity and history. We subconsciously tell the branches where we would like to grow, and the tree reaches out in response. The branches form the infrastructure of existence, the inner selves of human civilization, growing season after season with our own decisions.

We communicate to different levels of awareness through books, artwork, music and films. Imagination and creativity send our consciousness outward, a willing leap into different states of mind. If we observe a painting, one that resonates with us, we become focused within a smaller frame of awareness, a self contained world. A dream is communicated through the pigment, and in this dream we can use an additional path of perception. We can feel such forces as gravity, temperature, and wind, as if we are sending a probe forward, a vessel that contains a portion of our awareness. As you read these words, you can look around the room and see the life you have painted for yourself. And as your senses react to it you are sending motives back to your inner self, back through the branches of your awareness and into the wellspring of the collective super-conscious.

In The Video Game Theory Reader, the author comments on how close virtual reality and first person perspective games are to our "core consciousness", and that, "they allow 'the full experiential flow' by linking perceptions, cognitions, and emotions with first-person actions" and that it is like "leaving one's body behind" to live through another. Of course, this artificial perception is not a complete replacement of our awareness, but a small protraction or tendril. Being immersed in a first person perspective makes the avatar's (the player's persona) awareness become an extension of the physical senses; the player sees and hears through the character, and the present personality, or physical self, becomes the character's inner sense. We do not typically imagine that our character is aware of the player, yet the character is nothing without this entity behind the joystick. I believe that this is analogous to the way the layers of the whole self are stacked, like Russian nesting dolls, each one unable to see anything but its contents, aware of the outer faces only on a subconscious level.

	Virtua Tennis Screenshot

As awareness leaps forward and proliferates into the unknown, technology has slowly taken us further into virtual awareness. One needs only to look at the earliest video games to see how far this trend has come. Atari's first commercially-successful video game was Pong, a digital representation of table tennis, making its debut in a horizontal tabletop enclosure. This coin-operated game was introduced in the bars and clubs in 1972, allowing the two players to face each other, and many other spectators to congregate on the sidelines. Everyone was positioned in such a way that there could be discourse and eye contact. While the interaction was collaborative and biological, and the environment immersive, it was hardly convincing enough to be called a "virtual" tennis game. Sega's Virtua Tennis, for the XBox 360, features photo realistic graphics, and allows players to assume the careers of famous tennis stars such as Amelie Mauresmo and Maria Sharapova, touring the world to become champions. The environments are rich and detailed, the players movements are incredibly convincing. This represents 30 years of technological advancement over Pong and we can only wonder how much more convincing tennis video games will get.

In Telematic Embrace: Visionary Theories of Art, Technology, and Consciousness, Roy Ascott describes how, in Western civilization, tables were once the central meeting places of dwellings, serving as "the arena to construct and rehearse alternative behaviors, structures, and organizations...a device for divination, a sounding board for new relationships within the house or the universe", and furthermore that they represented, "the matrix of transformation and transfer, between change and exchange, between two poles, two worlds (of discourse), between two cultures, states of awareness, good and evil." Ascott describes how, in its horizontal alignment, the tabletop equates a mirror reflecting the cosmos, and in modern times, the table, as a meeting place, has been replaced by television screens, which are symbolic of mirrors reflecting humanity. He reminds us that a mirror requires the viewer to be in a fixed position, preventing inspection from different angles and depths, that it is an "isolated rectangle representing only ourselves and talking to us only one at a time, quite separately", and "the archetypal convention of a whole culture of isolation, competition, self-love." A study conductued by the Stanford Institute for the Quantitative Study of Society (SIQSS) in 2004 revelealed that people are now spending twice as much time on-line as watching television. The internet is swinging us from isolation back into social interaction.

A Massively-Multiplayer Online Role-Playing Game (MMORPG) is a multi-user online game in which millions of players can interact, forming a virtual society which is always running. Since the players communicate only through the interface, social interactions happen only within the context of the game world. There are no missions, so to speak, because the game environment is so rich and complex that players can choose their own objectives. Relationships form, leaders emerge and tensions arise. Players may choose to take part in a complex business operation, and assume the role of a manufacturer, resource broker or retailer, for example. These group oriented goals are just as time consuming and risky as doing so within our own economy. What is particularly interesting, is that because the players true identities are hidden, the social and entrepreneurial connections that form do not depend on age, race, sex, income, or language of the players. Interactions form between people that would not normally be able or willing to interact.

The trend seems to be that as a society, as the breadth of our experiences expand, we utilize technology as a matrix for awareness.

	MORPHEUS: "If the virtual reality apparatus, as you called it, was wired to all of your senses and controlled them completely, would you be able to tell the difference between the virtual world and the real world?"

Interactive Flow

There are some fantastic capabilities in video game technology that can be applied to interactive art, musical instrument design, education and sound therapy. Interactive music is one particularly interesting feature of video games that has been gaining popularity since it's introduction by Lucas Arts in the early 90's.

Interactive game music is usually approached by splitting up the soundtrack into layers, time segments and incidental riffs, then triggering each of these elements in synchronization with game play events. This simple solution allows for a more immersive, flowing, and cinematic gaming experience. In a recent Remix Magazine article, turntablist Mike Relm, said:

	Mike Relm

“Because my musical background is that of a DJ, I think of music in terms of how different songs and parts would mix in and out of each other. If you break down the soundtrack for a video game, adaptive music does pretty much the same thing. It's all about the smooth transitions and flow. Most people play a game for hours without realizing that they've been listening to the same pieces, mixed up and rearranged depending on how they play.”

Adaptive music engines that are designed to be transparent make the technology somewhat inappreciable to the user. Unless the game is designed in such a way that the player has conscious control over the soundtrack, the transformable nature of the music goes unnoticed. For an emotional film-like experience, where the game manipulates the player's emotions, invisibility would be the best approach. Putting the player in direct command, on the other hand, allows the player to steer the emotional character through physical expression. Players move their hands or bodies with the game music and graphics, reacting in synchronization with a rhythm, directly triggering notes and loops, or even remixing parts of a large arrangement. If you "don't got rhythm" the music sounds dissonant, if you do, the music kicks into full gear. This type of interactivity can be trance-inducing, exactly as if you are improvising your own music live.

Guitar Hero is a "rock star simulator", where the player becomes the lead guitarist in a band, performing in rock battles against other players. After choosing a classic rock song to perform, and hitting start, the song intro kicks in, a signal to get ready. In the foreground is an approaching stream of graphics, communicating the necessary note and timing information needed to play the lead melody. In the background is a 3d environment, a dimly lit club. The guitar controller has a series of colored buttons on the frets, a lever which represents the picking action, and a whammy bar for bending power chords. It is necessary to get the fingering correct on the frets, and then pick the strings at exactly the right moment. As one performs, a rock persona is seen playing guitar on stage alongside band mates. When a note is missed, or played incorrectly, the music becomes appropriately discordant. The rock hero becomes visibly mortified at his mistakes, and the audience reacts with realistic shouts of disapproval. When the notes are played right, the musical energy increases.

	Guitar hero Screenshot

It is easy to pin-point the adaptive music in this game, since it reveals itself as an interactive component. By the nature of the Guitar Hero's interface and design, the music is still very linear and there is not any way to improvise the melody without making it sound bad. However, what makes this game so remarkable is that the player's objective is to synchronize body and mind with the music.

Most people are not aware of how the game engine functions. If the player is completely absorbed in the gameplay, there is not even an awareness of the user interface, like the joystick, and the computer screen. Communication flows subconsciously from the computer to our mind and back again, in complex synchronism. On his blog, Michael John of Method Games describes the gamers meditative frame of mind:

"Hip modern psychologists call this phenomenon "flow". It's a sort of transfer forward of the subconscious into actually controlling our actions. It's been well-demonstrated that "flow" occurs frequently for game players. This was first shown, unsurprisingly, with Tetris, but is now well-understood as a frequently occurring phenomenon in game play....a superconductivity of expression"

When playing games, activity continually reverberates between the computer and the participant. The game and the participant perceive, interpret and respond to each other's stimuli, in continuous resonance, which the player is only subconsciously aware of. When flow is achieved, a complex relationship forms that is a resonance of mind, body, and interface, bringing the player into an altered state of consciousness. The computer and the player have achieved entrainment. This is just like driving home from work, and suddenly realizing that you do not remember any portion of the trip. It seems rather remarkable that the interaction between our mind and the controls can completely escape us, yet something from within gets us home safely.

Dr. William S. Condon, a psychology professor at Boston University School of Medicine did a study on human communication by "micro analyzing" conversations between two people onto film running at 48 frames per second. Every frame was tagged with events that included fragments of each word spoken and corresponding micromovements of the subjects bodies. It was discovered that during each word, clusters of tiny body movements would take place in the head, eyelids, mouth, shoulders, and fingers. At normal speed these movements appear as longer gestures, but in breaking them down he could see that there was a "precise rhythm", within the body. Also the listener's motions, although much more subtle, were happening in synchronization with the speaker.

"Listeners were observed to move in precise shared synchrony wiht the speaker's speech. This appears to be a form of entrainment since there is no discernible lag even at 1/48 second....It also appears to be a universal characteristic of human communication, and perhaps characterizes much of animal behavior in general. Communication is thus like a dance, with everyone engaged in intricate and shared movements across many subtle dimensions, yet all strangely oblivious that they are doing so." (Leonard 2006)

Looking at the relationship between playing video games and having conversations, one can see how natural it is to let the subconscious control our rhythms.

Approaches

Unlike sound art installations, which are almost always designed to reach awareness on a subliminal level, interactive music systems should grab the attention of the user. It must have high entertainment value in order to sustain human involvement, and in order to make the device accessible to a wider audience, the interface and functionality should be easy enough for novices to grasp. For kids especially, the device should be safe and able to withstand a bit of abuse. What is particularly tricky for the artist, is coming up with a device that is familiar enough to invite the users in, and mysterious enough to keep them.

	Instant City

Two German researchers named Sibylle Hauert and Daniel Reichmuth, have blurred the lines between gaming, musical performance, psychology, and sculpture with a tangible musical interface called "Instant City". When unattended, Instant City is a slick looking glass table with rows of translucent plexiglass blocks around the outer edges. In actuality the table is waiting for visitors, for if someone enters the room, it slowly illuminates and radiates sound. By carefully placing the translucent blocks on the table, architecture can be built, and as this is done, a melody forms. An array of sensors in the table watches the spotlight above and each time a block is placed on top of the sensors, a change in brightness and form is detected. A computer translates this sensory data into a musical sequence. It can sense how high buildings are, their shape, their relative positions and the order in which each block has been placed.

Apparently a social music building game called "automat" can be played with this table. The information I have found about it is so esoteric, I am not sure if something has been lost in the English translation, but it sounds intriguing nonetheless:

"Inevitably a dialogue situation develops, the action on the table transforms itself into a reaction to what the others do. Although there isn't actually a possibility to play against one another, still aesthetic differences can become visible on the game board in the same way as the different personality structures do. Suddenly there could be leaders who want to build up high – or saboteur whose structural intention is to disrupt or destroy. The music suddenly plays the role of a commentator or also a referee, who through sounds remarks upon the single acts. With instant city an "instrument" has been created that can enliven a space and the events in it on a variety of levels." (Hauert, Reichmuth 2006)

Some artists utilize existing interactive devices as the functional framework such as cell phones, iPods, and portable game consoles. Toshio Iwai, a world-renowned multimedia artist from Japan, created Electroplankton for the Nintendo DS; game, musical instrument, and touchable media art all rolled into one. The artist has used this pocket-sized game console to create a museum of ten interactive multimedia projects. Each of the ten modes, called "planktons" feature interactive floating creatures, and by tapping and rubbing them directly on the screen, animations, tones, musical sequences, and rhythms are created. The graphics are wonderfully minimal, but full of life and personality. Some planktons offer game-like objectives, but it is harder to call this software a game more than a musical instrument. The sounds beg to be recorded and mixed. This is a creative tool, free of branding and violence. Since there are not any clear goals to Electroplankton, it tends to confuse and repel many consumers who are more accustomed to interacting with traditional games. However, the fact that interactive art has been introduced to gamers by a major manufacturer is very encouraging.

	SoniColumn

Interactive music devices can also be deceptively simple in functionality, consisting of nothing more than household objects, and sensors. Jin-Yo Mok, a student of Harvestworks in New York, has been experimenting for years with interactive art based around music boxes. His interactive music installation, unambiguously named Music Box, is made from a large antique music box with a handcrank connected to a laptop. The pins on the cylinder have been replaced by white LEDs, and the teeth (the metal keys, plucked by the pins) have been replaced by photo sensors. The laptop's screen presents a rectangular grid, representing the cylinder, unwrapped into 2 dimensions. To sequence the Music Box's melody, one simply draws shapes on the grid with a mouse, thereby turning on the appropriate LEDs on the cylinder. When the crank is turned, the cylinder turns and a synthesized melody is heard. Based on this idea, Jin-Yo Mok's recently completed SoniColumn, another interactive music installation. This time he used a 7 foot tall vertical aluminum column with blue LEDs, which can be turned on and off simply by touching the lights. The crank mechanism is still included, but the personality of SoniColumn more deeply futuristic than its wooden parent.

Although Music Box is interactive, it offers direct control over the arrangement versus having an engine affecting the music in response to input stimuli. This is a different kind of experience, more akin to live performance, where the functionality of the device is exposed, and the human mind plays the role of software algorithms. Some people may feel less inclined to interact with such a system, as it clearly advertises its functionality, and does not embrace unique secrets. Having some form of interactive engine makes the device even more interesting, because we then gain the ability to speak with it and uncover hidden behaviors.

In Composing Interactive Music, the author starts his book by describing the value of discussions: "Nothing is more interactive than a good conversation: two people sharing words and thoughts, both parties engaged. Ideas seem to fly. One thought spontaneously affects the next. Participants in conversation assume much past experience and find excitement in shared experience. Conversations stay within a consistent context that creates a feeling of mutual understanding without being predictable." He goes on to compare conversations to human-computer interactivity, and says, "Interactivity comes from a feeling of participation, where the range of possible actions is known or intuited, and the results have significant and obvious effects, yet there is enough mystery maintained to spark curiosity and exploration"

Naoko Kubo, Ken Matsunaga, and Kazuhiro Jo, from the Kyushu Institute of Design have created a Teething-ring Sound Instrument (TSI) for babies. The instrument consists of a teething ring that is connected to an airline hose, which is connected to a pressure sensor. The sensor is converted to a MIDI signal and sent to MAX/MSP which runs the electronic instrument.

"Pitch corresponds to the change of the sucking pressure. When the teething ring is strongly sucked, a higher note rings. The “basic part” consists of a simple 3 note C-major chord played melodically in the form of a simple musical scale. This is something the baby can identify and enjoy. In expansion part, the note changes with every suck. When the suck is repeated, ascent, descent are repeated. The change of the notes can be enjoyed even by the reflexive sucking motion. Therefore, this is ideal for use from the baby’s initial stage of growth." (Kubo 2001)

Interactive devices are best able to encourage interaction when they strike a balance between familiarity and abstruseness. At the same time, if it is overly perplexing, curiosity is diminished. We are born with these behaviors and they form the basis for learning. Babies learn more by being exposed to a balance of new and familiar things, because they are less able to filter out events taking place around them. Too much stimulation will cause them to defocus, but too little will stifle their desire to examine things. Adults require a similar equilibrium, which is recognizable in music, film, and social connections. A friend that shares a common interest helps us feel grounded, but if conversations with them fail to bring about new and stimulating experiences then we may not desire to converse with them any longer. Music and films are designed to fit into genres because people are more apt to buy things that are familiar. However, if the work is overly stereotypical, offering no new spin on the familiar style, then it is almost guaranteed to fail.

Input Mechanisms

Sensors form the path between the mind and music. In most cases they are hidden from view, and are quite simply a way to transform some form of energy into music. The sensor, when activated, sends a control signal to the interactive engine, such as a synthesizer running in a MAX/MSP environment. The focus is on non-conventional approaches at controlling interactive engines, such that people can interact freely without needing to be musically or rhythmically disposed.

Kit-Based Systems

A project kit would be the preferred way of designing a MIDI controller. If there is a commercial part that does the job, there is no need to re-invent the wheel.

	The I-Cube "Digitizer"

I-CubeX offers an enormous range of sensors allowing one to build a custom MIDI control system using almost any force imaginable. The sensors transmit 0-5 volt signals which, using one of their MIDI to voltage converters, are transformed into MIDI data. They sell a number of starter systems which include the essential components, but the fun really starts when designing custom kits. There are over 40 types of sensors to choose from that can measure light, sound, heat, air pressure, magnetism, muscle tension, distance, acceleration, and direction to name a few. They also sell actuators, like lasers and infrared beams, which can be used in combination with the sensors to form advanced triggering mechanisms, such as the impressive laser harps featured in many of Jean Michelle Jarre's concerts.

The sensors and actuators connect directly to the "Digitizer", a pocket sized analog voltage to MIDI interface, with 32 DC inputs, 8 DC outputs, MIDI in, and MIDI out. The MicroDig is a cheaper solution, offering 8 DC inputs, one MIDI output and is about the size of a lipstick case. There is also a wireless version of the MicroDig.

Each device includes extensive documentation, including bug reports, and even some recommendations about what other modules work well with it. By combining any of these devices together it is possible to create some very powerful controllers. The individual parts are fairly expensive, but in the possibilities are impressive. For researching and developing MIDI controllers, the kit-based approach is both convenient and powerful.

IBVA (Interactive Brainwave Visual Analyzer)

"The IBVA has been acclaimed throughout the world as a superior device, from professionals to really weird people." - ibva.com

IBVA Technologies has been around since the early 80's, developing a brainwave-to-voltage device which can easily send MIDI data using their software expansion package "Brain2MIDI." By using a wireless headband, a base station, and Brainfire Launcher software, users can create low resolution data streams, via electrical activity in the brain, and with Brainfire, assign that data to any desired function, such as controlling visual effects and playing games. Apparently they've had a hard time promoting this technology. They devoted a lot of website space to explaining the difficulties they have had with being ridiculed and ignored by the press. Their image gallery shows pictures of many famous musicians using it, such as Billy Idol, YMO, Deee-Lite, and Laurie Anderson.

	An IBVA system

I can find no information on the effectiveness of IBVA as a MIDI controller, but there are several creative people out there putting it to interesting use.

A German performance group uses brainwaves in their performance called "Braindance." Hinterberger, a German Doctor, has created a "Thought Translation Device (TTD)" that allows for translation of thoughts into sound and music using an electroencephalogram. A dancer is wired to a computer, and dances in real-time to music generated from her own brainwave patterns. Hinterberger says,

"By presenting the live music of selected brain rhythms to the dancers they can empathise with those rhythms and interact with them in a closed feedback loop. A braindance performance is an artistic presentation of the hidden harmonic and chaotic spiritedness of our brain by sound and dance." (Ascott 2006)

Miya Masaoka is a sound artist and musician from San Francisco area. In her fascinating installation entitled "Pieces for Plants", she has attached IBVA electrodes to the leaves of a climbing Philodendron. A performer interacts with the plant, which stimulates it into different physiological states. The software analyzes the plants response and which is transformed into a MIDI signal and sent to a synthesizer, which allows it to play music. In an alternate version of the piece, the plant collaborates as a soloist in a string ensemble. On her website, Masaoka comments about response to this work:

"The audience members — including children — seemed to have a special attraction to the project. A common response to the piece by audience members was a desire to talk about their relationship and experiences with plants during their lifetime, and things they had noticed which gave them an inkling that plants had extraordinary capabilities and awareness beyond what was normally attributed to plants. The piece evolved into really being about the people, their personal stories with plants, and I realized that I was brushing the surface of a deeper questions — our complex role as humans in a diverse, inter-dependent biological environment, and the potential for communication with plants that has not yet been discovered" (Masaoka 2006)

Mocap Suit

	The Gypsy MIDI controller

Sonalog, a maker of motion capture rigs (commonly referred to as mocap), has recently released an upper body MIDI control suit. Animators use mocap devices to record body movements in the form of data streams, which can then be used to animate digital characters for film and video games. There are two types of mocap rigs. The most elaborate one requires the actor to wear a pattern of reflective tags which are then tracked by a frame of sensors. With this system, the actor can move about freely within the frame, the one drawback being that if any of the tags are occluded from view, then the data stream needs to be edited. The other kind of rig is mechanical, where the actor wears a suit of physical controllers that track joint rotation of the arms, torso, neck, legs, and feet. This type is a little more problematic in that the person's movement is restricted by the suit, but they are not confined to staying within an enclosed performance space. The Gypsy MIDI suit is the mechanical type, with a very futuristic, robotic look.

The MIDI suit is basically just the arm portion of their full-body Gypsy 5 system. Each arm includes 6 rotation sensors that send raw data out a MIDI cable or through an optional wireless transmitter. One can purchase either one or both arms. It comes with a software package called "Exo" , written in MAX/MSP, that can be used to calibrate the system, process the raw data into notes, and remap or reassign controllers.

A review about this controller was featured in the October 2006 issue of Sound on Sound. The reviewer felt that because of the lack of visual feedback, like on a tabletop device with knobs and LED readouts, it is hard to know exactly what data the controllers are sending out at any given time. He called upon the opinions of some professional dancers and they immediately commented on how restricted their movement felt. Other than that the suit and software functioned without any major problems. The reviewer's main conclusion is that the suit is powerful and a lot of fun, but its full potential can only be determined by what is done with it.

There is great potential in this device, even just the idea behind making sound from bending and rotation. For example, a flexible sculpture could be designed that includes a hidden arm inside which could alter the sculpture's shape, the shape of the sculpture could then form music. It would possible to create a large amount of joint controllers using cheap potentiometers. So, rather than just one series of joints, the sculpture could include a number of spider-like armatures. An interface that requires such simple interaction, which is capable of sending out complex steams of data is appealing to me. It's simple enough for a child to use, and powerful enough to be the foundation of an expressive instrument.

Emotional Controllers

	Brendan Walker, wearing the prototype equipment

An engineer named Brendan Walker has been researching devices that can measure a persons emotional state. On October 17th 2006, in his Thrill Laboratory exhibit at London's Science Museum, volunteers rode on fairground rides while monitored with an ECG to measure heart rate, a camera to monitor facial expressions, and an accelerometer that measured the G-force that their bodies were subjected to.

Steve Benford, of the mixed reality lab at the University of Nottingham, says, "We will be able to see if bio-signals reliably map onto the subjective experience of thrill. The key question is to understand the reliability and predictability of the thrill experience. The data can then be used as an analytical tool to design more-immersive rides and games, so-called "real-time adaptive spaces".

Walker said, "Consider a fairground Waltzer. The ride operator watches, and listens to his riders. He'll turn up the music and spin the chairs if he's not getting shrieks of delight. But he'll also make the ride go slower if younger children are looking scared. Thrill Measuring Devices could take the place of the ride operator, and be used to automatically tailor each ride experience to the sensibilities of each rider." - we-make-money-not-art.com

For now, emotional controllers are on the up-and-coming technologies list. If people have to wear cameras on their heads, and all sorts of wires, the technology will remain at a distance. A better place to start would simply be to incorporate personality profiling into the software, in an effort to determine reactions based on individual traits. If an interactive device functions in a way that it can learn about the behavior of the user, then it could better determine what type of experience they would like to have, just like we can cumulatively teach SPAM filters what type of email is junk.

SoundBeam

"Profoundly physically or learning impaired individuals can become expressive and communicative using music and sound. The sense of control, agency and independence which this provides can be a powerful motivator, stimulating learning and interaction in other areas." - soundbeam.co.uk

	The EMS Soundbeam system

Soundbeam is a controller that translates body movements into sound by using distance sensing technology (like bat sonar). The system uses two sensors which send very short 50KHz tone bursts into the air and then listens for variations in the detected response. Proximity, distance, and velocity are then collectively transformed into a control signal which is finally sent back out as a MIDI signal. The sensor beams are about 20 feet long.

There are a number of installations that use SoundBeam systems as an interactive interface. Among the most interesting ones is a display called Dark Matter. In 1998, at the New York Hall of Science, Paul Friedlander, a British light sculptor, designed a 27 foot tall tower of colorful spiraling light. In order to do this, a very high frequency multicolor strobe light was projected onto a swirling vertical rope. Through an optical trick, the projected light formed streams of animated colors that seemed to be suspended in mid air. Two SoundBeam sensors allowed viewers to effortlessly change the shape and color of the enormous sculpture. A deep humming sound was created by the vibration of the rope, which also changed in relation to spectator movement.

In a Sound on Sound review, the reviewer said that "It is literally possible to play a MIDI instrument just by facial activity, using your mouth and slight movements of the head." Because of their sensitivity and accuracy, these systems have been used extensively with immobilized individuals. Even the most severely disabled can perform on the Soundbeam independently.

Exit

Interactive music is in an infant stage right now, but it is on the verge of a renaissance. Already, some bands are selling their music on DVD's that include specialized software for remixing their songs. Adaptive game music is starting to become more popular with the recent influx of music games, but the real potential will be unleashed by multimedia artists such as Toshio Iwai, who breaks the rules. Some of the most interesting interactive sound devices are emerging from students and researchers. Not too long ago, if you wanted to build your own instrument you had to start from scratch and etch your own circuits. Now that there are so many projects going on in the field of interactive technology, there are commercially available kits and specialized software that expedite the process of getting the project going. Now we don't have to concentrate as much on the technology, as the application of it.

An annual conference in Whales called "Consciousness Reframed", brings together over one hundred interactive artists to present and discuss technology that models the mind. This is evidence that we are beginning to fuse more internally with computers. We should ask ourselves why we are on this journey to synthetic perception. The pattern of technological interactivity is imprinted within human awareness. We feel at peace with technology when it becomes an extension of our mind and this entwinement that humanity has with computers is going to deepen as we learn more about ourselves. It is a symbol, or reflection of our inner growth and creativity. Some people may focus on our interdependence with computers and see a path towards a technological "singularity", the theoretical point when technology advances so fast that machines self replicate and devastate the earth. Instead, we are developing technology such as video games, interactive music controllers, and artificial intelligence because it helps us understand the inner architecture of our consciousness and its outward expansion.

Cited Works:

Ascott, Roy. "Consciousness Reframed: art and consciousness in the post-biological era (Abstracts.)" (2006) pag.18. Online. Internet. Available: www.planetary-collegium.net

Ascott, Roy. Telematic Embrace: Visionary Theories of Art, Technology, and Consciousness. University of California Press: Berkeley 2003

Dahl, Lynda Madden. Ten Thousand Whispers. Woodbridge Group: New York 1995

Hauert, Sibylle. Reichmuth, Daniel. "Instant City." Aminima. (2006): n.pag. Online. Internet. Available: www.aminima.com

John, Michael. "Why are Music Games so Big?" Method Games (2006): n.pag. Online. Internet. Available: www.methodgames.com

Kubo, Naoko. "TSI (Teething ring Sound Instrument)." Kyushu Institute of Design. (2001) pag.1. Online. Internet. Available: www.delivery.acm.org

Kutay, Steve. "Get in the Game." Remix. (2006): n.pag online. Internet. Available: www.remixmag.com

Leonard, George. The Silent Pulse. Layton: Gibbs Smith, 2006

Masaoka, Miya. "Brainwaves & Plants." (2006): n.pag. Online. Internet. Available: www.miyamasaoka.com

"René Descartes." Wikipedia. (2006): n.pag. Online. Internet. Available: www.wikipedia.com

The Matrix. Dir. Andy Wachowski and Larry Wachowski. Warner Bros. 1999

"Welcome to the IBVA System!" IBVA (2004): n.pag. Online. Internet. Available: www.ibva.com

"Soundbeam for special needs and music therapy." (2006): n.pag. Online. Internet. Available: www.soundbeam.co.uk

"Thrill Laboratory." (2006) n.pag. Online. Internet. Available: www.we-make-money-not-art.com

Winkler, Todd. Composing Interactive Music. MIT: Cambridge. 1998

Wolf, Mark J.P. The Video Game Theory Reader. Routledge: Routledge 2003

[back]