29.08.2020

📺 Lokal/Digital - Machine Learning enabled performance

C++ PYTHON KERAS OPENFRAMEWORKS MACHINE LEARNING PUREDATA

Lokal/Digital is a interdisciplinary performance combining contemporary dance, free improvisation and live audiovisuals. The performance was conceptualized in collaboration with dancer-choreographer Evelin Stadler.

Somē playing the piano

The performance involved a number of moving parts. For one, Evelin was performing in a separate room with various props and was telecasted live onto a screen on stage. In the main stage hall, I improvised on the piano, manipulating in realtime electronic effects of the soundfeed while a visual projection reacts to my playing.

This article is mainly about the visual aspect of the performance.

The visuals

The live visual projections consisted of two parts:

  • an OpenFrameworks application running a stylized physarum simulation
  • 8 neural networks running in parallel, listening to my playing via PureData and controlling the simulation

The idea for using a physarum simulation was inspired by artist Sage Jenson, who wrote a comprehensive article about implementing the simulation. I found the visual expression of physarum to be particularly suiting to our performance and developed my own implementation in OpenFrameworks.

An octet of neural networks

The reason there are 8 neural networks is so that each one can control a separate parameter in the physarum application. The parameters exposed for control are:

  • the number of directions a particle can look at
  • the amount of deposit it leaves behind
  • a particle’s max speed
  • the max force that can be applied on a particle
  • the decay rate of the deposit trails
  • how far ahead a particle can sense
  • how steep the difference of the trails should be
  • the size of the displacement of the trails

The networks control these parameters separately, like an ensemble, to change the visual expression of the physarum in realtime.

Training the networks

The networks are trained on recordings of my own playing. I created a PureData patch that analyzes my playing, which recorded the analyses as a CSV file. The analysis outputs the following features:

  • 37 Mel-frequency cepstral coefficients
  • onset amount
  • Average RMS level
  • Spectral Centroid
  • Spectral Irregularity
  • Spectral Speed
  • Spectral Brightness
  • Harmonicity
  • Spectral Skew

The notebook used to train the networks can be found here. Disclaimer: the repo contains almost no structure and proper filenaming. It was developed before I knew what best practices were!

Realtime Setup

A python script ran an OSC server in realtime, listening to live input from the PureData patch, passing it to the neural network for predictions and then sending the result directly to the physarum application.

see all projects →