Unsettled Music

OVERVIEW

“Unsettled Music”, is a series of web experiments dedicated to showcasing some of our creative research on music, graphics, and machine learning. We have released a new iteration every few weeks. With this project, we’re hoping to collaborate with researchers, artists and designers from outside of the team, open a dialog, and give them a framework to try out their ideas. Like a gallery, each work can have its own space, and explore themes around music and AI with a certain freedom.

TECHNOLOGY

As the team behind Qosmo works on various projects, some focused on music making, others closer to data visualization, many small ideas and pieces of software are often left unused. Works shared as part of the series aren’t meant to explore their subject thoroughly — they are more like snapshots of an idea, with humble ambitions. This is because we focus on more the development of a method and process than the outcome itself. Exploring the outskirts of AI techniques often leads to unusual outputs, and that weirdness is a rich source of curiosity for us.

Web browser

In the past few years, the web browser has become a platform of choice for media art : not only does it provide a number of features that make modern technologies easy to access, it also, by nature, makes even the earliest prototypes available for everyone to try online. The first technical component of this project is the visual rendering engine. Web browsers have been supporting hardware-accelerated graphics for many years now with WebGL, Three.js being the most famous library built on top of it, and the one we’re using for this project.

Audio framework

The second component is the audio framework — here again, on top of the well-stocked web audio API, libraries like Tone.js allow developers to design sound performances using natural musical approaches.Other libraries like Essentia also make use of WebAssembly to implement advanced analysis algorithms with great efficiency.

Machine learning

The final and most challenging component is machine learning. With the help of hardware acceleration and the TensorFlow.js library in particular, Javascript has become a perfectly viable option for developing real-time AI systems running entirely in the browser. Some compatibility and performance issues remain though : while possible on most modern devices, running a model isn’t consistently fast on every machine, which makes it hard to have any reliable expectation on the timing of events. The size of the models involved is often problematic too. A few megabytes may not be much on a hard drive, but a poor internet connection can lead to several minutes of loading time.

For this reason, the machine learning-based experiments of this project are bound to be optimized, with some data being pre-generated, in order to remain accessible to most visitors. Working with Javascript actually allows for things to run offline seamlessly, as part of a Node.js
environment — and thus conveniently keep everything in one place. 

As we work on future experiments, we will try to explore aspects of machine learning in different directions. Some of those topics may barely touch AI in fact, but are meant to question some marginal aspects of technology and software, and give them an audiovisual transcription. We are hoping to collaborate with researchers, artists and designers from outside of the team, open a dialog, and give them a framework to try out their ideas. 

Interactivity

While some aspects of the experiments were optimized to avoid performance issues, the audio and visuals were arranged so that they can be dynamically explored in the browser. The interactivity happens through a single parameter, available on all experiments, which we called Unsettledness. In the context of AI, this translates to how far from the norm we choose to sample our new audio. Staying close to the familiar range leads to more expected output, but as we drift away from it, the model starts to produce unexpected results. Practically speaking, this unique Unsettledness slider allows you to scan a wide range of variations, from familiar and clean, to experimental and noisy.

EXPERIMENTS

---

---

ARTICLES

---

CREDITS

Nested Cycles

• AI Research & Development

  Bogdan Teleaga, Christopher Mitcheltree

• Sound Design

  Nao Tokui

• Web Development

  Robin Jungers

• Web Design

  Naoki Ise

• Project Management

  Yumi Takahashi

Broken Sample

• AI Research & Development

  Andrew Fyfe, Bogdan Teleaga

• Web Development

  Robin Jungers

• Web Design

  Naoki Ise

• Project Management

  Yumi Takahashi

Negative Space

• AI Research & Development

  Andrew Fyfe, Bogdan Teleaga, Christopher Mitcheltree

• Web Development

  Robin Jungers

• Web Design

  Naoki Ise

• Project Management

  Yumi Takahashi