Чули? Чули.

by

Letta Shtohryn

In collaboration with: Player 1 Julie-Michèle Morin, Player 2 Jeffrog Wang

Concept

The project „Чули? Чули.“ is a video game demo shown as a live performance, featuring a human protagonist (the player) and a motion-capture dancer. They both navigate the boundaries between the virtual and physical realms, shifting roles and blending identities through embodied interaction.

The title translates to ‚Have you heard? We’ve heard,‘ which in Ukrainian also conveys ‚Have you felt (it)? We’ve felt (it).‘ The project is showcased as a large projection in Deep Space 8k and is divided into two parts. In the first part, the player explores the game world, guided by a continuously evolving AI-created audio narration. In the second part, the player encounters another avatar in the game world and realises that a human body controls this avatar. The dancer, wearing a motion-capture suit, enters the play area, dances around the player, and gradually approaches them. As they progressively interact, the dancer eventually gives parts of the mocap suit to the player and takes the Xbox controller from them. Consequently, the player becomes an NPC (non-player character), and the dancer behind the avatar assumes control of the camera.

This work delves into questions of who serves as the protagonist and antagonist in online manipulations as it dynamically shifts perspectives between the player and the dancer, both virtually and metaphorically, challenging the traditional divide between virtuality and physicality. The project employs XR as a medium for facilitating cross-realm encounters, resulting in a storyline unattainable without technological augmentation.

The project utilises a video game format to metaphorically engage with manipulative online narratives created by troll farms, AI systems, and propagated by human agents. These narratives often involve spreading false or misleading information with the intent of manipulating or deceiving the audience. However, this project adopts a unique approach, capturing the experience through an AI-assisted shapeshifting of a different narrative, narrated by a woman who once witnessed giants. A central theme is the fragility of the physical realm under the influence of its digital counterpart.

Drawing from extensive PhD research on posthuman embodiment with technology, sensory maps, and intuitive epistemologies through XR and MR, this project undertakes a science-fiction exploration of the contemporary human condition, shaped by the interplay between online and offline realms. The primary focus during development of the work encompassed interactivity, virtual-physical codependency, presence across virtual modalities, speculative storytelling, and audio narration as gameplay guidance.

The somewhat novel aspect is the combination of media, the dance performance inside a video game but also a physical interaction between a player and the dancer, as well as his avatar. This allowed us to explore video game tropes, such as platform jumps and fantasy outfits, but also facilitated more conceptual immersion into the game world, which is built according to my PhD research into sensor mapping. The exploration of the role of the player and the NPC and its reversal is probably not a novel aspect, as myself and a number of artists have engaged with it, but it is novel for the context of the work and the mix of methods it is projected through.

Process Reflection

The project is driven by several interconnected themes – a reflection on truth, belief, propaganda, and online interaction, exploring their consequences in the physical world; and an exploration of our extended sensorium, enhanced and triggered by technologies, leading to speculative experiences that blur the boundaries between fact and fiction. These concepts were addressed through a fusion of media, blending a playable video game with live motion-capture performance and with a physical performance within a mixed-reality setting.

The interdisciplinary process of the project bridged (post)human theories, game studies, journalistic enquiry, media literacy, open-source investigations, performance and dance studies, AI co-creation, as well as visual art.
The process has somewhat evolved from the original gameplay idea, which involved a more passive player interacting with a number of NPCs, and later the dancer would arrive in the performance space, altering the dynamic with their presence. The original concept involving NPCs, where the player meets them and decides their path via in-game dialogue, was removed. For the purpose of this demo, the player’s agency and physical interactivity, suggested by my collaborators, took centre stage. Working with Jeffrog Wang as a dancer and excellent improviser, and Julie-Michèle Morin as a player, but also an experienced dramaturg and theatre-versed scholar, allowed me to delve deeper into the physicality of each role and the meaningful exchange of control and camera between them during the performance.

For the performance to materialise, I used a suite of software such as Marvelous Designer to create avatar clothes, MetaHuman to create avatars, Blender and Maya for rigging and mesh fixing, and of course, Unreal Engine, where the live motion capture data was fed via Live Link app from Axis Studio motion capture software. The most unpredictable element was the motion capture suit itself, as it is prone to magnetic interference.

During the process, we learned that we need to adapt our expectations to technology. The dancer was using a motion capture suit that was gradually losing connection to the receiver at the performance space – Deep Space 8K exhibits high magnetic interference due to the metallic surfaces on its walls and floor. Realising that the magnetic interference was unavoidable, the dancer Jeffrog decided to work with the collapsing avatar body. This made the act of passing the motion capture sensor to Julie (the player) more intriguing, as the avatar appeared as a body with shared control but also like an abandoned shell. The head seemed controlled by an NPC while the damaged body retained control of the world and camera. The surprises of technological failure and their exploration shaped the work to its current form, making it more fitting to the original concept than what I had in mind at the beginning of the process.