Piloting hundreds of unmanned aerial vehicles simultaneously, putting them through complex formations and orchestrating the lighting effects along the way takes, above all, excellent planning. The minimum spacing of six meters has to be maintained, the formations aligned and coordinated, and the transitional flights correctly timed—and all of it with the utmost efficiency. “It was an iterative process in which we actually had to spend a lot of time on technical details and not too much with the content or the artistic and aesthetic qualities of the show.” That’s how Michael Treml described the way the SPAXELS® Crew worked up until a couple of months ago. And he wanted to change that.
Michael has been a part of the SPAXELS® Team for about a year now, working on ideas and algorithms that facilitate the show design process. The objective: Finding a way to automate the transitions from one drone formation to the next.
Video: The Show Design Tool was the predecessor to Swarm3D.
Learning by Observing
But before he could get started with the programming, Michael had to better familiarize himself with the respective steps. “It was very difficult for me at first to gain insights because the process is so multilayered,” he recalls. “You have to take quite a lot of things into account, and it takes a lot of experience.” Accordingly, he spent his first three months hard on the heels of Patrick Berger and Raphael Schaumburg-Lippe, the two SPAXELS® show designers who configure the drone information and program appropriate color animation.
“It wasn’t a problem to set up formations aloft because you simply made sure that all drones are flying far enough apart,” explained Patrick Berger. “But when you then flew into the next formation, you had to take care to arrange it so as to avoid collisions, and that was a major effort!” Especially when most of the work has to be done manually. If, in addition to that, according to the design, a collision occurring in the simulation of the show is shown in the ground control software, then the designer had to manually correct the flight path of the endangered drone in the 3ds Max animation program. The fact that the result of this was sometimes causing more collisions than were prevented made the work even more difficult.
Michael Treml is thinking about the behavior of swarms.Credit: Vanessa Graf
Testing, Testing, Testing
This is precisely where Michael Treml went to work. “I did a lot of reading and research about collision avoidance algorithms, and all the developments in that field in recent years.” After the research, he began to try things out. “The first attempt was a total flop,” he chuckled. It turned out that the algorithm tends to fail with larger and more complex swarms. So, he went back to the drawing board, learned from his mistakes, and continued to tinker until the first version of Swarm3D was ready to deploy.
Of course, he wasn’t working alone. So that the software really does makes the design process easier, the two show designers, Patrick and Raphael, accompanied Michael’s work every step of the way with opinions and helpful tips. “Hacks” is what Treml calls these ideas, comments and suggestions delivered from the design side and assessed in terms of practical applicability during the development process. Many of these hacks turned out to be valuable functional enhancements of the algorithm. Also the fact that Swarm3D is now available as a plug-in for 3ds Max and doesn’t work, for example, as stand-alone software is essential to the design process since it means that the normal workflow is interrupted as little as possible.
The Real Ordeal: Rock in Rio 2017
In September, only a few months after beginning to develop Swarm3D, the algorithm would make its public debut. Rock in Rio, the huge music festival in Brazil, had booked the SPAXELS® to perform. So while all the preparations for the bands and concerts were underway in South America, Michael, Patrick and Raphael were putting the finishing touches on their software.
So, as if the development itself wasn’t stressful enough, the algorithm’s premiere would be at a huge public event! “We relied on our experience from previous shows, on the minimum spacing between the drones we had always used, and on the simulation in ground control,” Patrick Berger recalled. “From my perspective, that was strong enough evidence to conclude that nothing would go wrong.”
The drones paint the words “Rock in Rio” in the air. Credit: Rock in Rio
On Your Mark, Get Set, Drone Show!
And in fact, just like in the previous, manually designed shows, everything went according to plan. Not a single drone crashed, and no collisions disrupted the show—in nine appearances over the course of seven days, an excellent track record. Another big plus: the enormous time savings thanks to the new software. “The next show design we generate will take only a third of the time that it used to take,” according to Michael Treml.
Since this application could also be of interest to others staging unmanned aerial vehicle shows, the post-Rock in Rio development went into the next phase. The graphical user interface was improved, the code on the server was hosted, and final modifications were made. That brings us up to early 2018, with the release of the Beta version of Swarm3D.
An example mesh. Credit: Ars Electronica Futurelab
Time for More Creativity
“You can now create faster transitions,” is how Patrick Berger describes the advantages of this software. “Therefore, you’re also more flexible in the design process because you can change things faster.” After a change has been made, the designed meshes—or formations—are input simply, automatically and with utmost efficiency by Swarm3D. “Optimal flight paths” says Raphael Schaumburg-Lippe. Since a show always depends on the battery life of the drone, and the transitions are faster not only to design but also to fly, more formations now fit into a single show.
Take-off. Credit: Martin Hieslmair
Future Plans
And what about the future? Michael, Patrick and Raphael have no shortage of ideas—dividing the drones into sub-swarms and thus creating asymmetrical timelines with a show; individualizing certain parameters of the drones; and working with obstacles on the airfield are just a few. “Sooner or later, it would also be interesting to store the drones’ trajectories to memory and to see whether we could also achieve collision avoidance live,” Patrick Berger stated. Michael Treml added: “We’ve also talked a lot about automating the takeoff and landing behavior. Or creating a show element generator.”
Credit: Ars Electronica Futurelab
The three drone experts receive many of the ideas for Swarm3D from their surroundings. The swarm of pigeons that often circles above the Ars Electronica Futurelab, for example, has a lot to do with the new software, and so do the Linz city bees that make their home on the Ars Electronica Center’s roof. Even the surrounding steel town, Linz, has provided help. “For our algorithms that steer the drones, we derived a lot of inspiration from concepts of statistical physics, thermodynamics and metallurgy,” Michael Treml recalled. “For instance, the mathematical model used at voestalpine to describe the cooling of molten alloys can also be used to create faster transitions between drone formations.” Even observing the flows of human beings in downtown Linz lets you see how people’s “unconscious collision avoidance algorithms” work, and these insights can then be applied to drones.
So then, ideas aplenty to further upgrade and enhance Swarm3D. Who knows what will because of this software. After all, when this work commenced, nobody had an inkling that Swarm3D would not only be used in-house at the Ars Electronica Futurelab but as a plug-in for others.
You can read more about the SPAXELS® drones in this article on Autodesk.
To learn more about Ars Electronica, follow us on Facebook, Twitter, Instagram et al., subscribe to our newsletter, and check us out online at https://ars.electronica.art/news/en/.