Neural Beatbox

ABSTRACT

Rhythm is one of the most ancient means of communication. Neural Beatbox enables anyone to collectively create beats and rhythms using their own sounds. The AI segments and classifies sounds into drum categories and continuously generates new rhythms. By combining the contributions of viewers, it creates an evolving musical dialogue between people. The AI’s slight imperfections enrich the creative expression by generating unique musical experiences.

BACKGROUND

Through interactive design principles, this audiovisual installation allows for collaboration between a user and the AI system; while the AI guides the creative process and makes decisions, the content itself only comes from humans.
This resonates with the practice of beatboxing, where the instruments are removed from the equation to put the emphasis on the creative potential of the individual.
Here, the AI becomes a tool and enabler for natural expression, trying to make the best out of any human-produced content. Despite the computational process involved, it remains imperfect and produces results that might meet or defy expectations, adding a creative element of surprise and novelty.

TECHNOLOGY

One critical aspect of the installation was to propose a setup intuitive enough to allow for a fully interactive experience, while still showcasing the potential of machine learning in the context of music production.In order to do so, the system was designed from server to client-side.

The AI system is divided into two parts:

1. SOUND CLASSIFICATION

The first step is to proceed sound files (recorded by users), divide them into meaningful segments, and use a neural network classifier to classify them into one of eight possible drum categories (e.g. “kick”, “snare”, “clap”, etc).

2. RHYTHM SEQUENCE GENERATION

The second step is to generate beats — sequences of drum patterns — that will be played.This generative part is also the result of a trained neural network – a VariationalAutoencoder (VAE) – but it involves extra processes that give us more control over the choice of sequence: for instance, each drum can be weighted in order to pull a beat with more relevant items within the current context (e.g. if a user records a kick, we will want to update the beat with a sequence containing a kick). The client-side runs as a web app, and makes uses of the moderns features provided by browsers: media recording API, advanced graphics, etc.
This allows the application to be accessible from any modern machine with minimal configuration.

Design

In the context of the event at the Barbican Centre in 2019, the system behind Neural Beatbox was first designed to run as an exhibition app, and thus features reduced interactions to fit the simple interface provided on-site.
In 2020, we redesigned it as a proper browser-friendly application, to allow anyone to access it from a browser and experience it with multiple users.

Neural Beatbox for Multi Session (2020)
A vibrant, pop design aesthetic matches the fun experience of the application. Every generated beat is displayed using a dynamic layout that changes based on the number of drums used within that particular sequence.
Out of the three color themes available, one is chosen every time you enter a session.
In addition, each user is highlighted using a different color in an attempt to create a visual sense of the interactions within the current beat.

Neural Beatbox for Barbican Centre (2019)
In order to convey the intents of the experience and interaction with it intuitive, the visual setup had to go through a proper design process. We collaborated with Alvaro Arregui (Nuevo.Studio) for that purpose, and developed a set of animations to reflect the dynamics of the music, and give a rhythm to the piece, visually as well as audibly.

INSTALLATIONS
ARCHIVES

Date	Title	Place
2019/05 – 2019/08	AI: More than Human	Barbican Centre, London UK

---

---

ARTICLES

---

CREDITS

Neural Beatbox for Multi Session (2020)

• Development (Front-end)

  Robin Jungers (Qosmo, Inc.)

• Development (Back-end)

  Bogdan Teleaga (Qosmo, Inc.)

• Machine Learning / Direction

  Nao Tokui (Qosmo, Inc.)

• Machine Learning

  Christopher Mitcheltree (Qosmo, Inc.)

• Design

  Naoki Ise (Qosmo, Inc.)

• Illustration

  Chisaki Murakami

• Project Management

  Sakiko Yasue (Qosmo, Inc.), Yumi Takahashi (Qosmo, Inc.)

• Music

  JEMAPUR

Neural Beatbox for Barbican Centre (2019)

• Concept / Direction

  Nao Tokui (Qosmo, Inc.)

• Research / Management

  Max Frenzel (Qosmo, Inc.)

• Development

  Robin Jungers (Qosmo, Inc.)

• Design

  Alvaro Arregui (Nuevo.Studio)