Copyright

©André Knörig

Accepted to the New Interfaces for Musical Expression conference in New York, June 6th-9th, 2007. We will present a demo and a poster.

Presentation at the Ars Electronica Campus in Linz, September 5th-11th, 2007. Campus is organized by HyperWerk Basel, who proposed the motto „neo-analog“.

There's a sister project to this one, developed at the same time. Have a look at MusicPencil.

To non-musicians, the idea of playing a musical instrument is often just related to controlling the instrument in terms of selecting the right note in the right rhythm in order to translate what is written in the score. In fact, most of the practicing in the beginning years of learning an instrument will be concerned with this. Professional musicians invest enormous efforts in order to attain a level of control at which the instrument becomes almost a natural extension to their bodies. As in other art forms, learning the technique is the necessary evil on the way to artistic expression.

With our novel musical interface called Articulated Paint, we want to enable non- and beginning musicians to instantly experience basic aspects of musical expression by eliminating the need to learn how to play the right notes. By giving them a taste of the joy of interpreting music individually, we hope to generate a stronger interest in learning a real musical instrument.

Using Articulated Paint is as intuitive as drawing on a canvas. The paint-brush functions as the instrument. The stroke it leaves behind gives expression to the notes that are automatically served from the computer. The computer reads the notes from a MIDI file and sets the frequency of the notes, one after the other, while the handling of the brush determines expressive parameters such as timing, dynamics, articulation, and vibrato. Using this setup, the user can experiment with different expressions for a music piece and refine it, without being burdened by the insurmountable complexity of a classical instrument.

Conductor-like musical interfaces Articulated Paint falls into the category of conductor programs, in which the computer handles the playback of the score while the musician conducts the “manner” in which this happens. Malinowski [1] gives a detailed overview of musical interfaces in this class. There are several projects that enable a user to step into the role of a real conductor, mostly using a baton as an interface to direct an electronic orchestra. iSymphony [2] is the most recent incarnation of such a system.

Another interesting mapping of bodily gesture on musical expression comes in the form of a car driving simulation, the Expression Synthesis Project [3]. The Air Worm [4] allows a user to control loudness and tempo of a music piece by playing an digital theremin.

Painterly interfaces for media control The classical interface of brush and canvas is usually highly abstracted in computer interfaces. Rozin's Easel [5] tries to bring the physical aesthetics of real painting to animated images. The I/O Brush [6] is an interface for kids to paint with their environment as a palette. The MotoGlyph project [7] makes use of an alternative classical painting device, the spray can, to generate music out of images. All these interfaces make successful use of the affordance and intuitive expressiveness of real painting tools.

Copyright

©André Knörig

Bodily expression The physical form of the interface (brush on canvas) is chosen deliberately. Motivated by a larger research theme that is concerned with the role of the body in creativity [8], the idea is to create an interface that makes this connection more explicit. As there seems to be a conditioned link between bodily movement and music, the design of a gestural musical instrument seems particularly fruitful. A conventional graphic tablet is thus not adequate. Initial ideas based on tracking the body movement of a musician similar to Marrin's _Conductor's Jacket _ [9] were set aside in favor of the more intuitive and instantaneous brush interface which also provides haptic feedback. The importance of bodily expression is preserved: As with an instrumental musician - who has to move the whole body, including breath, in an artistic conjunction with the parts controlling the instrument in order to achieve a convincing aural expression – the artist in our interface needs to engage in the interaction with the whole body. We assume that for a non-musician trying out Articulated Paint, this is a joyful and revealing experience. We are interested in the effects of such an interface on creative exploration and learning.

Playing Modes As there is no innate way of painting musical expression, we conducted an informal user study prior to building the system, in which subjects were asked to draw the expression of the melodies they were hearing. The result was a wild mixture, ranging from rather conventional waveshape-like drawings to freestyl. From this study we condensed two alternative modes of playing the instrument. The basic setup, however, is the same. A session starts with an empty screen. The music is played to the user in advance so that she can memorize the melody. She then applies the brush strokes to play the same melody by herself. Only a little amount of instructions are given in the beginning, increasing with the learning of the user. Also the parameter mapping is the same in both modes.

Bodily Mode The first mode is more focused on the movement of the brush rather than the image it creates. In fact, the user is encouraged to play it blindly. The user is painting on the a virtual “conveyor belt” that moves in a steady tempo in the direction of an imagined z-axis. This way, she is actually painting on the same vertical position over again and can concentrate on the expressive movement of the brush. Another advantage here is that the resulting image conveys the tempo of the brush movement, which is not possible with a static canvas as in the second mode.

Notational Mode This mode is closer to the conventional notational system. The user paints line-by-line from left to right. Handling of the instrument becomes more complex, while the room for expression is a bit diminished, as movement is now restricted to left-to-right. However, this comes at the benefit of a more powerful representation. It is easier to interpret and is suited to be revised and improved on. Its synaesthetic quality is enhanced by the fact that the gestural mapping we employ offers some freedom for giving each music its characteristic look. Thus, this mode might be used to develop new forms of individual, meaningful notations.

Gesture Mapping The mapping of brush movement to acoustic parameters is very direct, since it happens in real-time. Timing is controlled by either removing and reapplying the brush to elicit the next note, or by changing the direction of movement in legato (horizontally in the bodily mode, and vertically in the notational mode). The intensity of brush movement, resulting from the amount of pressure and the velocity of movement, is mapped to the dynamics of the sound. Articulation is also related to dynamics. The style of legato depends on how quickly the direction of brush movement is changed, while the style of accentuation results from the distribution of pressure when setting the brush on the canvas. Vibrato can be achieved by moving the brush in a wiggly line.

Brush movement || Audible effect

Contact pressure & speed || Dynamics Contact / directional change || Next note (non-legato / legato) Attack of contact / directional change || Articulation of next note / legato Wiggle (amplitude and frequency) || Vibrato (amplitude and frequency) Table 1: Gesture Mappings

This mapping has so far been experienced as rather natural, but the optimal mapping is still being explored. The starting coordinates (x- and y-value) of a brush stroke currently have no additional meaning, but it would be interesting to use it to vary the timbre of the tone.