Streams in the Veins

by

Chiao-Chi Chou, Kaito Muramatsu,
Youngjun Choi

In collaboration with YouYang Hu

Concept

“Streams in the Veins” conducts a 15-minute piano ensemble involving a human pianist and plants. During this performance, the transmission of sound from various pianos leads both agents, humans and plants, into a continuous cycle of communicative improvisation. This process examines an interactive approach to non-human agency within the framework of interspecies co-creation, specifically through the medium of music. Through our attempts at musical sessions with plants, we were able to experience the temporal differences in the umwelt of plants compared to humans, such as their distinct response times to external stimuli and the resulting differences in perceptual worlds.

Chiao Chi Chou, Youngjun Choi, Kaito Muramatsu, and our collaborator Youyang Hu each brought distinct perspectives to the project. Chi focused on developing plant installations and laser devices. Youngjun handled the development of software modules enabling plant performance and AI-driven music generation. Kaito was responsible for conceptualizing and performing the musical piece, and Youyang contributed to the hardware modules of lasers and biosensors. We found ourselves deeply immersed in conversations about the music that plants might play, their physicality in performances, and the intricate process of translating plant biosignals into musical form.

Within this realm of collaboration and communication, a contemplative mind might reflect on the subtle distinctions between exerting mutual control, engaging in reciprocal learning, and the art of influencing one another. Mutual control, marked by its structured approach, risks dampening the spontaneity of creation. On the other hand, reciprocal learning, embracing an ethos of equality, nurtures innovation but requires the patience to meld diverse perspectives into a coherent whole. Meanwhile, mutual influence delicately balances guidance with the fluidity of creative thought, orchestrating an outcome that is both harmonious and dynamically enriched.

The performance begins with optical stimuli delivered by laser modules, interacting with plant leaves to elicit an action potential. The inherent latency in the plants’ response to these stimuli necessitated a careful composition of the musical piece to align with their delayed reactions, underscoring the contrast in responsiveness to external stimuli between humans and plants. This context was then reinterpreted through transferred timbre and generated notes passed through the Differentiable Digital Signal Processing (DDSP) models, aiming to equitably honor the agency of both human and non-human entities in each musical decision.

Our speculative approach thus wove together the equal contributions of all entities, cultivating a dynamic and improvisational milieu within the piano performance. The play’s essence, mirroring the fluid and ever-evolving attributes of nature, invites a deep immersion into the experience. As plant leaves react to the caress of light, we find a parallel in the way musical notes resonate within our cochlea, creating a symphony of interaction between humanity and nature. This artistic pursuit transcends conventional boundaries and emerges as a profound metaphor, celebrating the interconnectedness and harmonious coexistence of diverse life forms and modes of expression, much like streams flowing seamlessly in the veins of all beings.

Process Reflection by Chiao Chi Chou Chou

I focused on the execution of the project’s hardware and exhibition. I needed to integrate the work of collaborators and organize a hardware-software integration system to support two main systems: (1) the plant photonic signal-sound conversion system and (2) the sound-light conversion system with a user interface. I envisioned myself as a figure bridging the virtual and real worlds, fully accessing the knowledge in my collaborators’ fields through extensive discussions.

For the plant photonic signal-sound conversion system, I selected suitable broadleaf plants to meet the requirements of plant photonic detection. I also set up the power system for the plant sensors and integrated it with the structural aspects of the plant landscape. In the case of the sound-light conversion system and user interface, I designed a device that incorporated electronic keyboards, computers, sound source integration systems, LiDAR controllers, and a suspended laser emitter structure. These designs were guided by numerous discussions with collaborators from different fields and implemented on the foundation of exhibition hardware such as the internal network structure and power system.

As I integrated the requirements from different fields and attempted to run the entire project, I consistently pondered the significance of each artifact in the performance. This was quite different from my collaborators, as they won’t participate in choosing the color of a wire.. Such choices, particularly reflected in the visual meaning of human-plant interaction, led me to think about the historical image of humans dominating other living beings since the agricultural era, especially when humans stand taller than the plants. Through discussions with collaborators, I maintained the same volume for both main systems (connecting humans and plants), ensuring a certain level of symmetry, such as the piano keys arranged in opposite directions. When the audience watches the performance, they can easily observe the alternating play of the piano keys and the interconnection in an equal state.

I am excited about seeing information flow through different objects, from the pianist’s fingertips touching the keys, the signal passing through the electronic keyboard, reaching the computer, and then being converted into light by the driver. The light passes through the leaves, transforming into biological energy and another signal, and finally returns to the piano keys. This becomes a mechanism composed of living and non-living elements. This also reflects my personal approach to observing and interacting with everything in the world.

Process Reflection by Youngjun Choi

The project entailed a collaborative endeavor in creating a piano composition, which was a joint effort between a pianist and a plant equipped with biosensors. Our collective inquiry centered on discerning the nuances between mutual influence and direct control, and the various outcomes each might engender. A pivotal aspect of our approach was a conscientious effort to refrain from reducing the plant to a mere passive element in our arrangement. Instead, we aimed to recognize it as an active participant. This shift in perspective posed a significant challenge.

For me, the project was an enlightening experience in appreciating the aesthetic dimensions of performing arts, a departure from my prior focus on the technical aspects of backstage functionalities and the development of efficient programming, with less regard for the concept of agency. To underscore the importance of considering all participants in a system, animate or inanimate, the active agents must be equally influencing the outcome. Such approach of implementation through speculation enriched my understanding of the delicate balance between the technical and the artistic, fostering a newfound appreciation for the interplay between functionality and creativity.

In this project, I implemented the Differentiable Digital Signal Processing (DDSP) model which was primarily utilized for sound design and piano note generation for Disklavier placed in the Ars Electronica Center AI x Music room. This implementation focused on the fine-tuning of the model’s decoder component to our aesthetics in piano performance. I also developed a real-time audio-reactive visualizer, aligning with our project’s philosophy of interspecies co-creation. This visualizer functions through the circulation of musical notes, wherein each note and chord acts as a catalyst, triggering subsequent notes and chords in a harmonious exchange.

Sound’s ephemeral nature, inherently tied to the relentless progression of time, both enriches and limits our understanding of its contextual essence. This paradoxical relationship sparked my interest in the DDSP model, a neural network designed to learn the timbre or tone color of specific audio samples. This model achieves its results by analyzing a smaller dataset compared to previous methods. I chose to use DDSP to capture the essence of instruments or acoustic materials selected for their artistic value. Consequently, DDSP allowed us to explore the possibilities for endless sonic variations, surpassing the constraints of temporality in understanding the given characteristics of the other.

Through our collaboration of ensuring all involved parties, including both the performer, the audience, and the plants, were genuinely integrated into the cycle of interactions and influences was crucial in designing a musical piece that treats interspecies collaboration as a partnership of equal agents in co-creation.

Process Reflection by Kaito Muramatsu

The most significant realization I had while participating in this project was that the collaboration between individuals with different cultural and professional backgrounds served as an analogy to the theme of our project. Furthermore, the fact that English, being the medium of communication, was a non-native language for all participants similarly accentuated this aspect.

In my interpretation, a distinct aspect of this project lies in my pursuit as a human of engaging in a collaborative performance with plants through music sessions. These sessions resemble conversations without words, encompassing various prerequisites and characteristics. For instance, in a performance, it is essential to ‘listen to each other’ in a way that complements each other’s output. This ‘listening’ involves adjusting one’s own sounds based on the already produced sounds, aiming for the ‘whole’ to form a harmonious relationship—music. This is true regardless of whether a predetermined score exists or not.

If we draw an analogy to this aspect, in collaborative creation, we communicate using words and expressions for ‘better output.’ However, this involves not just understanding the superficial meaning but also inferring what the other person is trying to convey and why they are focused on certain aspects, considering their cultural, personal, and professional backgrounds. In other words, it becomes increasingly important to solve the inverse problem of interpreting the received words and non-verbal cues, continually searching for a better output.

Additionally, and this is not unique to our case, even when using the same language, English, musicians and engineers had different protocols for decision-making and word usage. For instance, in this project, we created a system that uses AI to compose music reflecting the physiological state of plants. This required overcoming the distinct narratives of musicians and engineers.

This means that such challenges are anticipated to be more pronounced in performances involving humans and plants. Compared to sessions with musicians who share the same musical education, the physical composition influencing the performance and the different methods of music creation present unique conditions.

What surprised me most in maximizing co-creation through consideration of others was the effectiveness of a sensation akin to positive resignation. There were times when I felt mentally unstable by contributing to aspects I was highly interested in while confronting approaches different from my own. However, ultimately realizing that I was not the sole agent in everything, accepting that fate while striving to do my best in those circumstances—essentially not personalizing the entire team’s project (which is somewhat misleading as a participant)—led to creating under the constraints of the environment. By facing the conditions presented and creating within them, I encountered unexpected musical experiences and a deeper understanding of others.

To draw an analogy again, attempting to control everything in a music session is not co-creation. In other words, the key may lie in how one maintains the ‘leeway’ to tolerate multiple outcomes, and how adeptly and spontaneously one can continue to propose the desired direction. Moreover, this might be something that humanity, not just in collaborative creation but also in facing the external world, always needs to maintain. The question then becomes: what exactly is this ‘leeway’?

Furthermore, it reminds us of the chicken and the egg dilemma (‘Which comes first, leeway or better output?’), which may also lead to a deeper understanding of the external world. Searching for and nurturing a method to methodologically acquire the flexibility to tolerate and catalyze such external disturbances and constraints is likely a crucial element especially required in future interdisciplinary educational fields.