Neuromatic Game Art

AAA ThP: ARG ALT Artivism Thunder Play

Margarete Jahrmann,Thomas Wagensommerer, Louise Linsenbolz, Georg Luif, Stefan Glasauer. An Area7lab production

Multiplayer open world game environment in dual play mode: Neuromatic Devices for audiovisual representation of the 5 frequency bands of the brainwave spectrum with consumer EEG head bands, EEG generated mind sculpture, generative brain sound, brainwave object, photogrammetric environment, for live players and NPCs, embracing spatial distance and mental closeness.

Live performance premiere at CIVA Contemporary Immersive Art Festival 2021.

1. Multiplayer open world game, accessible via consumer BCIs (brain computer interfaces) with a dynamically generated collective mind sound-sculpture for 4 live players and unlimited number of virtual flaneurs/audiences with chat functions and talking Non Player Characters (NPCs). Non player characters named with 3-letter abbreviations of online life (LOL, AFK…) talk in obscure nonsense languages. Players can log in with EEGs (Muse) anytime or visit online live performances with a more extreme sound and visual progress/diffraction and enter a game in a collapsing photogrammetry world. The more brain signals get into the world, the more it erodes. Players are represented as diamonds, they elevate into Diamond net space. The environment is a meditation game for live generated movies. Its voices refer to Hua-yen Buddhism: The Jewel Net of Indra.
2. The game environment starts as version of an aesthetic view and use of glitches of “found photogrammetry” (google) of a historic social meeting place with a fountain with naked figures in the center of Vienna. We adapt this tradition and transfer it as “exhibition” space to strip our own inner brainwave spectrums as instrument for a concert at the fountain.
3. The brainwave object as metaverse fountain sculpture for live performance with EEG interfaces includes a hidden reference to the pandemic: a simulation of distance regulations for game characters expressed as bounding boxes, which secure spatial distance but allow mental closeness. Infection starts during the concert. God mode is the consequence.

Besides the performance sessions the environment always can be entered. A stored version of a brain data object serves as interactive playground/ multiuser game with the aim to foster an implosion into a de-constructivist experience around the Donner Brunnen, a historic Vienna site of social contact, performance (Valie Export), exhibition, and erotic attraction, where a new virtual fountain sculpture is generated live by 4 EEG players. Up to 100 players can meet there without EEG and watch the play in the Thunderdome. The EEG input provides real-time modulation of sound and of a brainwave object, which erodes over time into a Hua Yen Diamond net. The game is a photogrammetric world for live performances with EEG interfaces. The game also includes a simulation of an experience model of bounding boxes for NPCs and player avatars, a collective goddess avatar. It builds on a pandemic model of spatial distance and mental closeness.

In the Pink Noise Brain Jam Session a collective avatar is created from four EEG players. The game takes place in a symbolic Thunderdome – voice samples via vocoder in tribute to Tina Turner as goddess of the device with EEG Brain headband. The AVATARA, incarnation of a collective goddess, changes with the thought patterns of all players. Like Aunty we reference: Mad Max, launch part in 2021. We don´t need another hero_. We continue to vocoder in PINK noise. Neal Stephenson’s The diamond age, or, A young lady’s illustrated primer (1995) serves as our blueprint. The blueprint of a three dimensional network, in which all connection points are mirrored in every facet…. is our playground. Each player is represented as a Diamond – framed in the wireframe as a bounding box. On the one hand the DAMOND box refers to the Real Social Spacer (#Sozialer Abstandhalter, Jahrmann/ Kedl/ Glasauer 2020 – built out of wire and performed live in lockdown), and as a wireframe model to the bounding box of the avatar in the game engine. On the other hand we refer to the Jewel Net of Indra from Hua-yen Buddhism – to which meditation practices are the entrance. INHALE
EXHALE:::::::::: run!

After login we mingle with NPCs – they wear netculture acronyms of 3 letters at login, they speak a
cryptic language consisting of concrete poetry, Jandlsche Schhttzzzzngrrrbbbn of special characters,
as current coding/er version. Once played, an eternally repeating, rotating timeline starts. As soon as we are too close to the goddess avatar with other players, we are turned into God Mode after a certain incubation time. We fly – see the Thunderdome with virtual thunder fountain from above. ….the place decomposes, becomes wilder, iridescent, and itself facets of the diamond network…. everything is deeply intertwingled!

Detailed technical description: “Arg_ALT_Artivism_Thunder_Play.app”

1. The program is built using the Unity Game engine and networking plugins. On the first screen every player can enter their preferred name and connect to a room. This room lives on our server and can be visited by up to 100 concurrent users. Once you join the room you can navigate a 3D space in First Person mode. The position, rotation, name and other information about the players is sent and shared through the network.
2. The Muse headset data can be streamed via the OSC protocol to a port in the local network. This gets picked up by our App, which listens on this port. We receive the raw data from the headset – TP9 – Left ear, AF7 – Left forehead, AF8 – Right forehead, TP10 – Right ear and AUXR – Right Auxiliary. If this data is shared, all other clients receive it live and the data feeds directly into our installation where it starts interacting with the environment.
3. Sound generation is connected to the frequency band of the measured brainwave signals – and informed by the live modulation of the signals. Alpha, Beta. Gamma, Delta, Theta spectrum generates base lines and accelerates frequencies, density acceleration.
4. Structure of the networked game world: The first player that joins becomes the Master Client and creates a room for everybody. The 3D space gets loaded and all the NPCs initially are spawned via this first player. Any player that joins afterwards will be notified that the game scene has already been loaded and automatically load the scene too. This way everybody will share the same space with the same NPCs.
5. The NPCs regularly look for players to accompany during the installation. When they have picked a player, they will try to move in front of them, so they are visible from the players’ point of view. They will keep a (safe) distance and after a while will lose interest and look for another player.
6. The game mechanic implements a stochastic predator-prey model in order to simulate the pandemic situation with which we were confronted with from 2020 – till 2021, at the time of the premiere. After a certain infection time and if there are too many people to close to the Donnerbrunnen, players are randomly set to God mode, which means no more physical weight but flying over the distorted photogrammetry and live concert as well as top view on the
   central Avatara goddess made out of brain signals.

How to use your own consumer EEG device to transform the sculpture:

You need a Muse EEG device (https://choosemuse.com) compatible with the MindMonitor software (https://mind-monitor.com), which runs on iOS or Android devices. Connect your Muse headband with MindMonitor, stream your EEG data over your local Wifi to the Arg_ALT_Artivism_Thunder_Play. In the MindMonitor settings, enter the IP address depicted in the Arg_ALT_Artivism_Thunder_Play app as ‘OSC Stream IP’ and 7000 as ‘OSC Stream Port’. Start streaming and observe how your brainwaves transform the sculpture!

addendum on EEG interfaces used in the performance

The sculpture used in the environment is based on a three-dimensional representation of EEG data. This WIREBRAIN was generated through the use of two consumer EEG devices (muse) and a homebrew Matlab App, developed by Stefan Glasauer, during an online connected remote session with a third EEG user. Watch the connected ongoing weekly Friday performance series of Margarete Jahrmann and Stefan Glasauer since march 2020 on the Neuromatic youtube channel!

Brain-computer interfaces are increasingly available as consumer versions using a low number of electrodes, sometimes together with other sensors such as inertial measurement units. BCI in scientific contexts relies on accurate and reliable extraction of EEG signals with the goal of using these as active commands, such as move left or right, or as passive readout of the subject’s intentions, opinions, or emotions.

In contrast, our artistic research approach focuses on the subjective experience of using the device and its critical reflection rather than accurate measurement and signal decoding: What do I experience when using such a device? Our alternative approach opens up the use of BCI as “Neuromatic Devices”, as tools that can be used beyond current utilization as instrument for scientific measurement or, in its consumer version, for self-optimizing. For example, measurement artifacts such as noise, muscle activity, or even electrode movements are not just considered as nuisance but become possibilities for alternative use and to convey signals over additional channels. As neuromatic device, the BCI becomes a tool of self-expression and reflection rather than self-optimization. Here we present live performances using BCIs as public experiment in which spectators can interact rather than watch passively, which blurs the borders between experimenter, performer, and test subject, at the same time revealing failures and technical problems and making them part of the experience rather hiding and discarding them.

During the broadcast, raw EEG signals from the consumer BCIs worn by two or more remotely connected participants are sent (via a combination of Bluetooth and UDP) together with simultaneously captured video of the participants to a central computer. There, real-time visualizations of the EEG signal (implemented via custom-written Matlab code), the video images, text output of the software, and written instructions for the participants are composed to a final output that is streamed together with real-time sound generated from the signals to the Neuromatic YouTube channel. The
stream serves as necessarily delayed bio-feedback for participants wearing devices as well as communication channel for the spectators who can comment and influence the performance via chat. The video streams and snippets from the EEG recordings serve as data trace and documentation of the artistic experiment.

The artistic research approach applied here to brain-computer interfaces resulted in the development of a publicly open explorative method, the Brainwave Broadcast, which allowed investigating the space of possibilities spanned by the use of different media for transforming BCIs into Neuromatic devices that allow a collective critical reflection of the technology. It has become a tool of artistic research, where we introduce an individual approach, the very subjective of a “contact improvisation” via chat with participants — plus the connection with other “players”, who connect via Virtual Private Network tunneling, send their data via a constant stream, and form an environment of points, clouds
and lines, similar to a morse code, direct from the neural acitivity, that shaped a communication
without words. In terms of artistic research (Jahrmann 2020) this individual input is extremely
important for a broader reflexive documentation and generation of new knowledge around the use of
experimental devices.

//project funding: FWF PEEK project Neuromatic Game Art (PI Margarete Jahrmann).

//references:
CIVA Contemporary Immersive Arts Festival, Online 2021. Live performance premiere at CIVA Contemporary Immersive Art festival 2021. https://www.twitch.tv/videos/925429063?filter=all&sort=time

Neuromatic Brainwave Broadcast https://www.youtube.com/channel/UCRuETt9HjkgXfn17ao8pCRg

Jahrmann M (2020). Expanded Game Art and Neurointerfaces as Means of Produsage. In: Abend P,
Beil B, Ossa V (eds.) Playful Participatory Practices – Theoretical and Methodological Reflections,
Springer Nature, Wiesbaden, pp 131-147. In series: Perspektiven der Game Studies. DOI
10.1007/978-3-658-28619-4

Hua-yen Buddhism: The Jewel Net of Indra. By Francis H. Cook. University Park and London: The
Pennsylvania State University Press (The Institute of Advanced Studies of World Religion Series), 1977. xiv, 141 pp.

Neal Stephenson: The diamond age, or, A young lady’s illustrated primer. Viking, London 1995, ISBN
0-670-86414-5