A contrapuntal dialogue between a pianist and her avatar in the metaverse
Honorary Mention
www.lwt3.com/project/giusycaruso
MetaPhase is the artistic output of a cross-sector encounter between avant-garde pianist and researcher Dr. Giusy Caruso (BE) and the innovative start-up LWT3 (IT). This joint research was driven to explore the creative potential of data processing, human-machine interaction, and biotechnological applications in a XR performance to enhance performers’ expressiveness and audience fruition. The core is a wearable easy-to-use prototype developed by LWT3 able to acquire bio-signals (surface electromyography sEMG). The biometric metadata collection is shared via the LWT3 data platform accessible to performers/scholars/citizens.
This first experimentation of motion tracking and VR technology on stage engages both performers and audience in a groundbreaking live music co-creation which showcases the virtual METAPHASE process: the biological cellular splitting that here refers to the pianist duplication playing with her virtual agent in a meta-ecosystem.
The pianist performs by wearing a suit with light-reflecting markers and biosensor to allow the system to track her gestures and the related muscles effort in real-time. Then, the pianist wears the Oculus Rift to enter in the meta-performance-scene developed by LWT3 with Unity platform and performs the counterpoint part of Steve Reich’s Piano Phase for two pianos by interacting with another avatar-pianist playing the first part of the piece. This second virtual human is animated by the real pianist’s expressive motion previously recorded together with the audio track on a Yamaha Disklavier piano. LWT3’s wearable interfaces acquire bio-signals such as sEMG from the real pianist to modify visuals, video effects, and audio parameters.
Spectators are involved in an immersive XR performance by selecting viewpoints, viewing data in real time or in postproduction together with the pianist. This interdisciplinary collaboration between science-technology-art wants to expand the possibilities of a futuristic and phygital techno-aesthetics.
Credits
Concept and pianist (Royal Conservatoire Antwerp, IPEM – University Ghent): Dr. Giusy Caruso
LWT3 Team:
Hardware and data engineer: Dr. Paolo Belluco
Designer and VR director: Samuele Polistina
VR developer: Andrea Randone
Fashion and product designer: Luigi Sossi
With support from: Royal Conservatory of Antwerp, IPEM – University Ghent, Yamaha Music Europe
Biographies
Dr. Giusy Caruso (BE) is a concert pianist and postdoctoral researcher focused on studying human-machine interaction (HMI) for the creation of futuristic multimedia formats and music performance analysis. Chariwoman of the CREATIE research group at the Royal Conservatoire Antwerp and affiliated researcher at IPEM-Ghent University, she is Official Music Advisor of LWT3 Society Milan.
LWT3 (IT) is an innovative company founded by Dr. Paolo Belluco (sensors, biosignals and machine learning) and designer Samuele Polistina (UX/UI, product design). With expertise in data analysis, visualization, IoT infrastructure development, and human-machine interaction solutions, LWT3 works in their Data Driven Performance Lab in Milano across three pillars: Industries workers, Sport Science, and Music and Performing Arts.
Jury Statement
What kind of new possibilities will technological advancements bring to our music experiences? MetaPhase invites us to encounter the music performance in a new world where humans and avatars harmonize in both real and virtual space. What is noteworthy about this project is that the performers’ movements and gestures are captured with high accuracy by a motion tracking system and that they are embodied on the screen in the form of digital avatar. Thus, the public can savor every detail of the relationship between gesture and sound. Along with that, the application of such capturing technologies made it possible to collect bio-metadata for use in expression and in various research directions. This is one of the fantastic projects that impressed the jury with its remarkable artistic approach and accomplished innovative exploration.