Mission

Neuromatic Game Art: Critical Game with Neuro Interfaces/* AR 581

Project lead: Margarete Jahrmann
Research Partners: Mark Coeckelbergh (Technophilosophy)/ Stefan Glasauer (Neuroscience)/ Ruth Schnell (Media Art).

research fellows: Thomas Wagensommerer, Charlotta Ruth, Georg Luif, Zarko Aleksic, Anna Dobrosovestnova. Supported by the Austrian Research Fund FWF/PEEK, national and international research partners, Philosophy of Media and Technology University of Vienna, Game Design Zurich University of the Arts and Computational Neuroscience Brandenburg Technical University.

Introduction

An urgently needed vigilant critique of the technologies of quantifying the self drives a Ludic-explorative research approach, combining real neuroscientific experiments with professional electroencephalography (EEG) and functional consumer versions of neurointerfaces for everyday use. An integral part of such an approach is a technical-philosophical consideration of biometric aspects, artificial intelligence and neuro-interfaces used in performative installations. A ludic design enables the creation of experiences consisting of elements from behavioral and cognitive science.

The research goal is to develop a new form of experimental game art in a scientific research context with game design elements and references to game cultures. From the reprogramming and conceptual change of neuro-interfaces we hybridize a new genre, Neuromatic Game Art as an epistemic method of artistic research.

The artistic research question concerns the generation of new insights into the conceptual conditions of biometric measurement of cognitive functions, their social and societal implications, and the discursive coupling of ethical and political questions concerning the measurement of the self – especially under the auspices of playing with media systems in times of social distance, controlled social behavior and voluntary self-monitoring. How is the potential possibility of measuring the innermost self, the artistic gaze and the technophilosophical thoughts on it evaluated? What social expectations and fears are linked to biometric procedures? Ultimately, we want to transform the instruments of brain measurement into artistic means of expression and take up the increasingly existing demands for self-optimization and social scoring as a global challenge in the political and social field.

Method: Ludic Experiments

Brain-computer interfaces are increasingly available as consumer versions that use a small number of electrodes. Currently, the main use of such devices is to support meditation practices through biofeedback by means of frequency analysis of the measured EEG (and possibly other data analysis such as blink detection). The consumer brain-computer interface (BCI) promises to uncover hidden states of mind and to decipher and visualize what is considered inaccessible for conscious reflection. As such, consumer BCIs support the increasing trend towards technology and computer-based self-optimization, which is evident in the abundance of healthcare devices and applications.
BCI in scientific contexts is based on the accurate and reliable extraction of EEG signals with the aim of using them as active commands, such as movement to the left or right, or as a passive readout of the intentions, opinions or emotions of the test subject.

In contrast, our artistic research approach focuses more on the subjective experience of using the device and its critical reflection than on accurate measurements and signal decoding: for example, measurement artifacts such as noise, muscle activity or even electrode movements are not only perceived as disturbing, but become possibilities for alternative use and signal transmission via additional channels. As a neuromatic device, the BCI becomes a tool of self-expression and reflection.

Both content-related narratives about biometric interfaces are considered in the research project in their social significance. For this purpose, we have designed and publicly displayed the following experimental set-ups in the exhibition context since the project launch in May 2020:

Experiment one: Neuromatic Brainwave Broadcasts

“Neuromatic Brainwave Broadcast” is a series of Citizen Science Games in the form of online live appearances with BCIs for consumers, which we developed in the first months of the pandemic-related lockdown and which were streamed on YouTube, first daily and then weekly. The show functions as a public experiment where viewers can interact via chat with players connected by different types of EEG brain devices (Muse and Neurosky) instead of watching passively, blurring the boundaries between experimenter, performer and test subject. At the same time, mistakes and technical problems are revealed and made part of the experience instead of hiding and discarding them.

During transmission, raw EEG signals from consumer BCIs worn by two or more remotely connected participants (via a combination of Bluetooth and UDP) are sent to a central computer along with simultaneously recorded video of the participants. There, real-time visualizations of the EEG signal (implemented via specifically written Matlab code), the video images, the software text output, and written instructions for the participants are assembled into a final output that is streamed to the Neuromatic YouTube channel along with the real-time sound generated from the signals (abstract generative sound). The stream serves both as a necessarily delayed bio-feedback for the participants wearing the devices and as a communication channel for the viewers who can comment and influence the performance via chat in the free paiidic Citizen Science Game. The video streams and snippets from the EEG recordings serve as data track and documentation of the artistic experiment.

Results:

The artistic research approach applied here to brain-computer interfaces led to the development of a publicly accessible explorative method, Brainwave Broadcast. This allows to explore the space of possibilities opened up by the use of different media to transform BCIs into neuromatic devices that allow a collective critical reflection on the technology. In terms of artistic research, this individual input is important for a broader reflective documentation and the generation of new knowledge about the use of experimental devices.

Experiment 2: Neuromatic GAN Portrait GAME: The artist is measured!

Read more here