Every year, it is one of the absolute highlights of the Ars Electronica Festival: The Big Concert Night. This year, for the last time, the old train hall at POSTCITY Linz will host this unique spectacle; robots, dancers, the Bruckner Orchestra and much more will transform the grey industrial halls into an impressive concert location once again. This year, the “Mahler-Unfinished” project will be part of it – an attempt to complete Mahler’s unfinished tenth symphony using an artificially intelligent system.
Ali Nikrang, key researcher and music expert at the Ars Electronica Futurelab, and Markus Poschner, chief conductor of the Bruckner Orchestra Linz, reveal the challenges such a project entails – and how artificial intelligence can actually be combined with music.
Markus Poschner. Credit: vog.photo
Let’s first talk about the original music used in this project – you are working with an unfinished symphony by Gustav Mahler…
Ali Nikrang: The symphony is indeed unfinished. Although the first movement is largely complete, Mahler’s composing process was such that he first composed and sketched, then orchestrated and reworked the piece only after the first performance or rehearsal. This was not the case with this piece, even the first movement had never been performed by himself. Later, however, it was found out that the sketch of the first movement could be played without outside influence. The rest has to be completed and orchestrated and cannot be performed just like this, it is incomplete.
What exactly does it mean to orchestrate a piece?
Ali Nikrang: Every piece has, if you like, an abstract musical structure that describes the musical process in the form of harmonic and melodic progressions. The abstract structure can then be extended by a further very powerful dimension by working on the music for an ensemble with different instruments. With a large orchestra, there are of course many possibilities, for example in terms of dynamic, i.e. differences in volume, and also a rich palette of timbres. Especially in late romantic music, orchestration plays an essential role in the dramaturgy of the music.
So you used Mahler’s unfinished piece and continued composing with Artificial Intelligence?
Ali Nikrang: We took the theme from the first movement. It’s a very unusual and dark theme… it begins almost unnoticed, unaccompanied by the violas. We took this theme, more specifically the first 10 notes, and wanted to find out what an AI system could do with such an unusual theme. We used MuseNet from OpenAI, which is currently the strongest and most powerful AI model in the field. We had the AI system compose several pieces and then chose one that was very typical of the AI system. Since the result of the AI was only playable on the piano, we had to edit it for the big orchestra.
By hand?
Ali Nikrang: Exactly, by hand. However, I kept very close to the original. I tried not to change any musically relevant content and to interpret the music in such a way that it uses the large color palette of the orchestra but does not change the content. Actually, just like other composers who have arranged other movements by Mahler – they didn’t want to change anything relevant that the master sketched, but still orchestrate the material. In our case, the master was the artificially intelligent algorithm.
Credit: vog.photo
What were the conditions for such a project to take place? What does an AI-generated piece of music have to look like in order to be “playable” as a conductor and for an entire orchestra?
Markus Poschner: In principle, everything is playable for us, of course, as long as the work can be communicated sufficiently through notation. But we must not forget that notes are always merely codes or projection surfaces that we musicians have to fill with meaning – it’s about understanding and interpreting a text. This essential question determines all expression, sound, and actually all musical parameters. And of course the situation now is completely new, since we are dealing with a composition of a machine.
Ali Nikrang: Especially on an automatically controlled CEUS piano everything is playable. Musically, too, everything the AI system output was usable. They were pieces that we would definitely define as music, some better, some worse. That’s not self-evident with AI. I have been dealing with this topic for years and would not have thought that this musical quality of an AI system would be possible in the foreseeable future, not even as late as March of this year. Also I would not have thought that a few weeks later, OpenAI would present a model that is able to compose such a qualitative music.
The interesting thing is that they actually used a technology that is used for the purpose of text generation. They trained the model, roughly speaking, with music instead of text, and the model learned how to generate music instead of text. Learning in this context means to model the statistics of data in order to generate similar data.
I would say that the output of MuseNet is really very convincing. But that doesn’t mean it’s already perfect. And it doesn’t mean that it sounds like Mahler. Of course, a Mahler expert would immediately recognize that. In terms of aesthetics, the piece is in the late romantic style, but the composition is rather unusual. New materials are added again and again, which has to do with the fact that a hierarchical “Big View Thinking” is still missing in these models, so that the individual segments not only fit together, but also join together in a superordinate structure to form a directional whole.
Has the AI system been trained with Mahler’s pieces?
Ali Nikrang: Unfortunately, there is no information about the exact composition of the training set. The model was trained with hundreds of thousands of midi-files, so it is very likely that pieces by Gustav Mahler were also included.
Mr. Poschner, what do you think of the result of the work on the AI model? Do you notice any differences to conventionally composed pieces?
Markus Poschner: The technical level is astounding of course, I wouldn’t have thought such a thing possible. But, as I said, what does it mean? What does the piece of music have to tell us? We immediately feel a great uncertainty: are we allowed to feel anything? And if so, then what? Can the work of art tell us something, communicate something?
To perform AI-generated pieces with a whole orchestra is certainly not very common (yet)…
Ali Nikrang: Since there has been computer-generated music, there have been attempts to perform it with an orchestra. However, I am convinced that there has never before been an orchestral performance of such musical quality because the technology was simply not available. In addition, the whole thing takes place within the framework of classical music and with a renowned orchestra and conductor, which greatly enhances the musical quality of the performance. Another special feature, of course, is that Mahler’s Tenth Symphony is taken as the basis here and is also performed alongside the AI piece. Therefore, the whole thing is certainly a world premiere.
And finally, how could AI models like MuseNet be used in music in the future?
Markus Poschner: I can’t really answer that. But I imagine that such systems could be excellent training partners for young composers, similar to chess computers, since they have an almost infinite statistical knowledge. If I can look at how Gustav Mahler would have made purely static decisions at this or that point, it’s very valuable.
Markus Poschner is a German conductor. He studied at the Hochschule für Musik und Theater München. From 2000 to 2006, he served as chief conductor of the Georgisches Kammerorchester Ingolstadt. There followed stints at the Komische Oper Berlin, Bremer Philharmoniker and Theater Bremen. He has been a regular guest at the Münchner Philharmoniker and Dresdner Philharmonie; his many other guest appearances include performances with the Gürzenich-Orchester Köln, Bamberger Symphoniker, Staatskapelle Halle, Bruckner Orchester Linz, and Berliner Konzerthausorchester. He is also an enthusiastic Jazz pianist. In autumn 2017, Poschner takes over as chief conductor of the Bruckner Orchester Linz.
Ali Nikrang is a senior researcher & artist at the Ars Electronica Futurelab, where he’s a member of the Virtual Environments research group. He studied computer science at Johannes Kepler University in Linz and classical music at the Mozarteum in Salzburg. Before joining Ars Electronica’s staff in 2011, he worked as a researcher at the Austrian Research Institute for Artificial Intelligence, where he gained experience in the field of serious games and simulated worlds.
The Big Concert Night will take place on Friday, September 6, 2019, starting at 8 PM in the train hall at POSTCITY Linz. You can find out more on our program website or in our interview with Gerfried Stocker, Artistic Director of Ars Electronica, and Norbert Trawöger, Artistic Director of the Bruckner Orchestra Linz.
To learn more about Ars Electronica, follow us on Facebook, Twitter, Instagram et al., subscribe to our newsletter, and check us out online at https://ars.electronica.art/news/en/.