SwarmOS: Using swarm technology as a visual medium

There are doves sitting on the roof of the Ars Electronica Futurelab. A gust of wind, a movement, a sudden noise – and the swarm is taking off, every bird at the same time. Just before they hit the nearby church tower, the doves part ways, fly around the tower, find their way back to each other, as if they were programmed to do just that.

Within the Futurelab, something quite similar is being discussed: This is where the researchers of Ars Electronica, together with Japanese telecommunications giant NTT , are working on swarm technology as a means of communication. A few weeks ago, the team showed an artistic-scientific performance at Miraikan, Tokyo (we have already written about the artistic vision behind it here), while now, it is preparing for the next big gig: The “SwarmOS Research Demos” at the 2019 Ars Electronica Festival (Sept. 5 – 9).

This is where several applications for the in-house developed software SwarmOS will be shown, including Swarm Arena, which is the biggest and most complex prototype so far. Peter Holzkorn of the Ars Electronica Futurelab and Hiroshi Chigira of NTT have told us more about the technical background of SwarmOS, the current research progress, and the future of swarm technology in this interview.

One of the biggest applications for SwarmOS is Swarm Arena – can you tell me a little bit about it?

Peter Holzkorn: We have been working on a research cooperation with the Japanese telecommunications company NTT for a few years now, under the motto “Vision Towards 2020”. We have taken the year 2020, a year of big sport events, as a starting point to think about how to augment or transmit big sport events in the future, using swarms made up of drones or robots. With Swarm Arena, we are researching how to create a framework, an artistic vision, around the event using robots and swarms, and how to communicate information or visualizations.

Swarm Arena is only the newest development. Can you give an overview of what has happened so far?

Peter Holzkorn: The first project was Sky Compass, which we have shown in 2017 at the Research Forum of NTT in Tokyo. Back then, we worked exclusively with drones. The project was a Design Study in a small Demo-room, where we used only five drones to sketch out the possibilities of how these drones could guide visitors or display information. How can we use such a swarm as a system of information? Maybe within the context of a big event in a city?

Sky Compass, 2017. Credit: Michael Mayr.

At the 2017 Ars Electronica Festival, we have reworked the concept a little and moved our focus to other aspects. Most of all, we have shifted our attention to the workings of a swarm, instead of working on the aspect of our swarm working Outdoors or in the sky. From then on, our swarm technology could also work inside a stadium or in other spaces and did not have to happen in the sky.

Swarm Arena, 2018. Credit: Tom Mesic

Ever since 2018, the main project is Swarm Arena – here, we are thinking about the event itself. How can we augment and expand sports events virtually? In February 2018, we showed a mixed swarm consisting of both drones and groundbots at the NTT Research Forum. We have created an Augmented Arena and a Virtual Arena with 12 vehicles and presented the audience with the opportunity to imagine how it would look like if Swarm Arena happened on a bigger scale.

This year, 2019, we have put the focus on our groundbots. We are working with a swarm of these robots, who are all carrying a LED display with about 100 times 100 pixels. Together, they can create mosaic images, but they can also move independently and individually on the ground. This means we have a much higher resolution than with the drones, who each represented only one pixel. With the groundbots, we can create videos or images! At the same time, the action is transferred to the ground, so it’s best to view it from a slightly elevated position, as is the case in a stadium for example.

As you have already mentioned, you have created several prototypes like Augmented Arena or Virtual Arena, all of which are working with modes of public viewing or big sporting events. This doesn’t only work directly at an event, but can happen independently, on a remote location…

Peter Holzkorn: It could be a Travelling lab for Sports, Art and Entertainment. However, this is only one of the ways in which our swarm technology can be used. The groundbots are portable and could perform at a variety of different events or in different situations. Not every kind of event is suitable for every kind of presentation. The context has to fit. We are primarily interested in the swarm as a visual media – this can happen in very different ways, depending on the situation.

One of our concepts is, for example, to do a live transmission of a sports event – you could map movements or recreate them. This only works for a very limited number of events. To give just one example: At the performance of Swarm Arena at Miraikan in Tokyo in 2018, we have focused on Swimming. It is a sport discipline which you traditionally watch from above; there are very liquid movements in two dimensions; and the aspect of aesthetics is very important. This does not mean that we are only going to do swimming-events, or only live transmissions. Swimming also works very well with an artistic vision.

Credit: Ars Electronica Futurelab / Raphael Schaumburg-Lippe

Which we could see very clearly at the performance at Miraikan…

Peter Holzkorn: This is exactly what we wanted to show at the performance in Tokyo. There were scenes, where you could see the swimmers very clearly, and then there were scenes, in which abstract patterns took over, or other visuals. We wanted to show the spectrum of what is possible. Our swarm is not only a new kind of display, but offers new aesthetic possibilities.

This is exactly what Ars Electronica is known for: The combination of art, technology, and society.

Peter Holzkorn: Yes. Since we have invited artists Akiko Nakayama and Ei Wada to join and apply their own artistic practice, we could really show a very interesting aesthetic spectrum. You could see the inspiration of the sporting event, but it wasn’t only about the performance of the sportsperson, but rather the experience, the expectations, the perception, the program around the event and the atmosphere.

Hiroshi Chigira: A good example of the mixture of swarm technology and art would be “Giant Swimmer” scene in Miraikan performance where the floor was transformed into a swimming pool with the swimmers and additional artistic effects. While testing/prototyping with several computer vision technologies to extract the athletes’ aesthetic elements, and discussing with the design team, we found out the “invisible swimmer” to make the swimmer disappear while only the water splash remains to visualize its swimming maneuver. Coming up with such unique ideas with technologies and artistic essence would be crucial part to create Swarm Arena’s performance, and also a very exciting moment.

It isn’t easy to coordinate a swarm. With which challenges were you confronted?

Hiroshi Chigira: We successfully achieved the show of 39 bots indoor in Miraikan, however if we want to deploy massive swarm in our actual world, bigger swarm performance outdoor, that would be seriously challenging. From technological perspective, one of the biggest challenges we have been tackling, is tracking/positioning. Precision and reliability, both are the crucial requirements for realizing organic swarm movement.

In order to make these kinds of performances easier, Ars Electronica has developed a software: SwarmOS. Can you tell me more about it?

Peter Holzkorn: We have been working on SwarmOS for a while now. It is, like the name suggests, an operating system for swarms. We have had to work with very specific requirements for our drones, the Spaxels, for a few years already. However, we wanted to create an interface that could work with a variety of vehicles, not just with our Spaxels. At the same time, we wanted to present third parties with the opportunity to use our tools. At the first Research Demo in Tokyo, we were already using a prototypical version of SwarmOS, back then it was for groundbots and drones at the same time. This time, at the 2019 Ars Electronica Festival, we only have groundbots, but also a more developed version of SwarmOS. We are showing applications of this software, one of which is Swarm Arena.

How exactly does SwarmOS work?

Peter Holzkorn: We are using a Ground Control software as a sort of commando center. On every vehicle, we have installed a small module called “implant”, which abstracts the entire communication and logic system of the swarm control. The ground control and the implant are the core components of SwarmOS. They generate waypoints or commands for the individual vehicles. SwarmOS is in charge of the action level of the communication and synchronization, the timing and the swarm control. This can be used for several applications – for us, Swarm Arena was a Proof of Concept and also the biggest application so far.

You can use SwarmOS with a predefined choreography or in a scenario with live interaction. It is very abstract, so that it can also support other vehicles – at least, if they support certain protocols. SwarmOS should be easily integrated into other systems, that’s a big step. This is very handy for us, but also for third parties.

Up until now, we have heard about what has happened in your research so far and what the current outcomes are. Now, of course, I would love to know: What are the next steps?

Peter Holzkorn: For Swarm Arena, we of course have ideas and plans for the future. In the past, we have worked a lot on the topic of navigation, later, we have focused on Swarm Arena. Now, we are moving more and more in the direction of artistic expression. We are interested in the feel of events, in the atmosphere.

One thing that is certain is that we will use SwarmOS for all of our future applications. We are also working on SwarmOS so that we can easily share it with others – research partners and also manufacturers. Maybe individual clients won’t be doing their own drone shows anytime soon, at least not with SwarmOS, but we aren’t very far away from being able to use it with industry and research partners.

Hiroshi Chigira: We could imagine several instances related with sports events, such as remote live sports viewing or on-site sports augmentation, and a medium for ceremony events and so on, as we demonstrated in the previous Swarm Arena presentation. One possible idea would be a movable signage system in the real stadiums or sports event places. By replacing the signage displays with bots on the football fields, for example, displaying real-time information with organic movement, or entertainment contents during half-time show, would produce far-beyond sports viewing experience. A tool for visualizing/analyzing sports game would be another possibility since it’s scalability and mobility enable us flexible display functionality to effectively analyze the sports game scenes for making better team strategy or more attractive sports programs in TV broadcasting.

You can find out more about SwarmOS, Swarm Arena and previous projects within the shared research project of NTT and the Ars Electronica Futurelab on our website and in previously published interviews on the Ars Electronica Blog. For more information on the SwarmOS Research Demos at the 2019 Ars Electronica Festival, please check out this year’s festival program.

To learn more about Ars Electronica, follow us on Facebook, Twitter, Instagram et al., subscribe to our newsletter, and check us out online at https://ars.electronica.art/news/en/.