RSS feed RSS Twitter Twitter Facebook Facebook 15 Questions 15 Questions

Coding Aesthetics

img  Tobias Fischer

It seems perfectly clear that we're headed towards a future where man and machine will no longer just co-exist or complement each other, but truly merge.  On a purely rational level, this merger has obvious benefits: Microscopic sondes circulating in our bloodstream will be capable of detecting illnesses before we even notice their symptoms; thought processes, our ability to learn and remember will be sped up, made more complex and supported by implanted computer technologies and hard drives; and all of our senses will be enriched and expanded upon. At the same time, sceptics have always cautioned that these supposed achievements threaten to eliminate what makes us human. In the age-old competition between progress and its containment, the arts are the ideal playground – or, depending on your perspective, battle field - to explore these topics and provide for tentative answers on what to expect as we're tumbling towards a singularity. For the past years, Sydney-based Ben Carey from the University of Technology, Sydney, has contributed to these questions by developing algorithms such as _derivations, which creates strikingly original machinal responses to human improvisations and is thus capable of interacting with human musicians more organically and, dare we say it, creatively than ever before. To Carey, a closer connection between man and machine need not be something to be afraid of, as long, as he puts it in this interview, "artists are speaking about what technology brings to their art, rather than dressing up their multifaceted work as a purely technical contribution". In this scenario, the future will turn out to be more human than what prediction algorithms may be able to compute.

The idea of a symbiosis between man and machine in creation seems to have been important to you for a long time now. Why does this whole concept, subject of many fearful scifi flics,  not feel scary to you, but rather exciting?
It’s not scary to me, no, because I feel connected to the development of the kinds of machines people might have fears about. I think as soon as you realise that a computer program is not inhuman, but the work of probably many human minds and intellects, with their own aesthetic leanings, creative ideas and imaginations, this whole concept breaks down. There’s an infectious energy amongst artists working in this kind of way, and a feeling of experimentation and openness that is refreshing. I think what’s exciting in the area is the possibility for composers and artists to take a step back and be surprised and intrigued by their own work. They might set up some parameters or construct the general scope of an interactive or generative work, but ultimately the work is constructed in real-time. It’s quite a buzz to be surprised by something you thought you knew all the ins and outs of! At first I think from a performer’s point of view this kind of context is bit of a novelty, but what I’ve found great is the dialogue that opens up about improv after a session, and about interaction and performance in general. I think making work like this is a way of asking questions, and about starting a conversation.

You've quoted Leonel Moura with: "If art has no purpose, as all the modern and post-modern theories declare, then machines are the best creators."  Rather than discussing that statement, I would be interested in how you evaluate it – is Moura's vision of the future desirable or cause for concern? How do you see the ongoing relevance of creation, regardless of these functional aspects, just like humans are still playing chess although machines have clearly outpaced them?
I think his ideas are deliberately provocative, but they point to some interesting conceptions of human creativity. Personally, I don’t believe machines are the best creators, though I believe that by endowing them with more autonomy we give ourselves as creators permission to be surprised, and to expand our ideas about what creativity is. This is a particular view about art making that resonates with me. I like the idea of the artist being able to appreciate a work they’ve developed as a viewer/auditor as much as a creator. There’s quite a contradiction in this space that I find fascinating. On the one hand, when the artist sets the boundaries, the rules or chooses the generative grammar that the software follows they are being extremely proscriptive – almost authoritarian. On the other hand, the very act of relinquishing control to those systems is one of profound detachment and liberation. Creation is and will continue to be incredibly relevant, despite these technological experiments. I think that when we teach our machines to follow a set of rules, to iteratively develop their own grammars or to interact autonomously with other humans, it might simply suggest that creators are just as interested in experiencing something strange and unquantifiable as their audience. I also I believe that by injecting some of what we know (or assume) about creativity into machines, we’re ultimately taking a strong look at ourselves as humans and as creative individuals.

In the bigger picture, the evolution of machines in creative territories is merely part of a potential fusion between the two. What are your thoughts on this prospect?
I think there is already fusion; our tools are shaping us as much as we’re shaping them. For example, I don’t think I can separate the way in which I listen to the world when walking down my street from the kinds of experiences I’ve had in the studio. I also think this idea of delegating typically repetitive functions to computers changes the conceptual level at which we work, not to mention the speed. Certain creative practices just wouldn’t be thinkable without the capabilities of our current machines. Creators are also able to choose the level of abstraction at which they want to work from parts of their practice, and the feedback from our machines is definitely influencing our next creative moves. At the same time though, I also think that the ubiquitous presence of our machines can also teach us the value of material engagement. This is one of the reasons why my acoustic solo album The Wingello Sessions is so important to me. The relationship I have with my instrument now, as a solo player, is fundamentally changed. I don’t think I understand how just yet, but I look forward to trying to work this out by playing more away from the computer.

What were some of the main motivations behind developing _derivations? Was there a sense of dissatisfaction with existing tools?
 _derivations emerged after a year’s worth of experiments with developing personal tools for interactive computer music performance. I don’t think the software was borne out of dissatisfaction with existing tools, but more a desire to develop some of the improv work I’d been involved in previously with some of the research I was doing into machine listening and generative methods. I loved the idea of using my instrument with live electronics, and I’d experimented a lot with live sampling early on. The problem was I didn’t feel entirely free to interact with the computer because I was constantly jumping between playing and processing/manipulating sampled material. I also wanted to be surprised by the computer’s contribution, and to play off that. I’d also interpreted a number works with electronics in France, and although the music and technology were very sophisticated, as a player I’d felt a distance between my contribution and the ‘magic’ of the technology. I wanted to be involved, to affect and to be affected by the software, rather than to simply trigger responses or pre-composed elements. That’s where it started really.

For those unfamiliar with its workings, could you describe _derivations in a nutshell?
_derivations is a software system that listens to a musician improvising, samples their playing and performs with them by autonomously processing material captured during an improvisation. The performer doesn’t need to touch the computer during an interaction, they just play and the system responds in kind. Instead of simply looping or layering materials, the system dynamically builds a database of ‘phrases’ it has captured from the input, and makes decisions on what to use for its response based on how similar or different a phrase might be to the improviser’s current performance. I think of it as a machine that has a growing vocabulary, but also one that has a sense for the timbral context of an improv. It’s not listening to melodic lines or rhythmic patterns but to tone colour, gathering statistics on a player’s sound and storing them against captured phrases for later reuse. I’m using the same kind of algorithms used in speech recognition technology to make links between the player’s sound and the database. Player’s can also use it to grow the system’s vocabulary from performance to performance, so the more they use the system, the larger its pool of possibilities.

One of the things that only rarely gets mentioned in the development of software is aesthetics. What role did they play, concretely, in this case?
In the case of _derivations my aesthetics drove the project for sure – I can’t separate my aesthetic preferences from the finished product. I don’t think this is that unusual really, but it’s become an interesting question when the project started being considered for use by others as well. I have a preference for a focus upon timbre, so that drove my decision to analyse this rather than to recognise melody, harmony or anything else. The kinds of processing I’m using in the system – certain types of granulation and synthetic textures – are really tied to my interest in electroacoustic and acousmatic composition. Even though they sound different depending on the improviser, and the variables are constantly changing, I had to set their boundaries which is definitely an aesthetic decision.

One of the deciding aspects about a tool like _derivations seems to be the nature of the material provided by the software. How do you strike a balance between machinal input that, on the one hand, complements the human aspect and, on the other, challenges it? How can these two terms even be integrated into the code? 
You’re right; it is a difficult balance to strike. Where _derivations is concerned this process depends on the types of connections the software makes with both the outside world and itself. It’s also a matter of the kind of processing and manipulation applied to the material brought back. Phrases are never simply replayed as recorded; their manipulations are controlled by a number of algorithmic processes so there’s always variation. The crux of it though is that the software is only really pretending to be intelligent. There’s some tension built into the working of the system because on the one hand it’s built to find exact matches between its vocabulary and outside input (or its own output – it also listens to itself), and on the other, this is a technical impossibility. No performer plays the same thing twice, and neither would you necessarily want them to, so the idea of ‘matching’ an input to a database becomes a fuzzy process immediately. These approximations create twists and turns in the software’s output, unexpected ruptures and changes of course that affect the trajectory of the music that the player then reacts to.

In the development of _derivations, was there a moment when you could actually sense that the machinal contribution was getting really inspiring and potentially more interesting than what another human could contribute?
There was a period when I first realised I could set _derivations loose to ‘perform’ alone – without any outside input. I’d loaded it up with a large and varied database of material, set some parameters and let it roam. I think I listened to its output for a couple of hours straight – it was kind of surreal. I never intended to build a standalone generative system – interaction has been my interest – but I felt like this experience changed my view of the program’s identity, and its potential for initiating ideas in performance. Even though I knew all of the material it was referencing for its output (in fact, I’d recorded most of it myself), the complexity in which it drew connections and juxtaposed them was inspiring. Conceptually, the matching idea that drives _derivations is actually quite simplistic, and certainly not very human. However, one thing I’ve realised is that it might actually be such strange and non-human qualities that point towards new ways of interacting and engaging in performance. I wouldn’t say they make the computer’s contribution more interesting than a human’s by any stretch, but they might point to different ways of conceptualising interaction, and challenging our view of our own strategies.

You mentioned that it was highly interesting to you to regard the 'mutual influence as being at the core of an interaction, rather than striving for complete autonomy between man and machine'. Why is this?
I think mutual influence and mutual dependence are nice ways of understanding the entangled relationship between humans and computers in a performance. Mutual dependence is a way of understanding how humans and computers might need each other in a performance, rather than focusing on what might make a machine able to act autonomously – completely of its own volition. Developing performance systems requires us to think about what kind of influence each party has over the other, as well as designing the kind of autonomy required of the machine. I’ve started thinking of the environment more as a symbiotic system because I don’t think either is entirely autonomous in the true sense of the word during performance. In human-machine performance, bundled into the interaction is a complex web of assumptions and expectations about machine agency that the performer navigates during every performance. Performers might second-guess the capabilities of the machine; they might ascribe causality - rightly or wrongly - between an action they took and the computer’s response, which influences their future decisions. To me this is a type of dependence, one that’s inherent in this kind of human-computer interaction. In human-human performance, although two improvisers might meet on stage for the first time and get to know each other during performance, the performers are aware of the cognitive abilities of their interlocutors simply because they know them to be human.

How does a tool like _derivations, which decidedly contributes originality to a project, change the question of ownership, do you feel? How much is the music on the finished album yours, in how much that of the software?
I think there’s a level of abstraction from my input to the machine’s interaction with the human player, a certain ambiguity that I really like. Of course, I know _derivations’ ins and outs really well, so I can recognise the kinds of gestures it comes out with no matter who is playing with it, but I think the ownership question is an interesting one, both from my point of view and from a performer’s. I like to think that the ownership of computer’s contribution is partly mine, but it also belongs to the performer, and to itself. I think performers who have played with systems like this recognise the creative authorship of the developer, but what counts is the moment-to-moment connection in performance, and because this is decidedly not controlled by the developer, there’s a beautiful disconnection that makes a performance such a weird hybrid space. The improvs on the album all prove this for me, they’re quite different and personal to each player, but they also represent different performance circumstances and dynamics – each of which I was very much a part of.

Does The Wingello Sessions in any way feel more 'personal' than the _derivations pieces?
Funnily enough I wouldn’t say that Wingello Sessions is more personal, no. Listening back to those improvs, there’s a sense of disembodiment and detachment from the real-time act that mirrors the detachment I feel from a _derivations performance. Through repeated listens I recognise why I might have made decisions whilst recording solo, just as I recognise the causes behind the computer’s decisions when it performs with another player – I can relate them to earlier decisions of mine made whilst programming. However, during an interaction I think it’s the blurry nature of these causal relationships that push a performance around the next blind corner. This is what I love about improv in general.

What can we learn about our own understanding of music by programming music software?
This is a massive question! I’m trying to keep this at the forefront of my mind looking back at my own work and to the work of others. I believe that the developer’s aesthetic motivations, and more importantly their assumptions about improvisation and interactivity are directly encoded into software like this. I think the kind of interactivity a system provides can tell us how a developer views human interaction in performance, which is extremely personal and fundamentally philosophical. At the same time, I also think writing this kind of software is a way to interrogate one’s own assumptions; at least this has been the case for me. There’s certainly a sense of vulnerability there on the part of the developer that I don’t think has really been discussed or properly understood as yet. It’s one thing to write a piece of music that embodies your ideas about what music could be, but I think developing software like this often puts more fundamental questions about human performance on the table; questions surrounding listening, analysing, responding and co-creating – things that are mostly taken for granted when no computer is involved.

Ben Carey interview by Tobias Fischer
Ben Carey image by Volker Böhm

Ben Carey's _derivations | human-machine improvisations is available from Integrated Records.

Homepage: Ben Carey

Partner sites