AI Music Generation Based on User Data
At Dentsu Craft Tokyo, we created a new web service “TOKUCHA MUSIC” for Suntory Iyemon Tokucha, one of the pioneers of green tea functional beverages with specific medicinal benefits. “TOKUCHA MUSIC” is that uses AI to generate music based on user specified daily meals. The tempo, melody, and tone of the music change depending on the nutritional balance and type of food. The goal was to give people a new perspective on the balance of their daily diet through music. Qosmo has provided our original AI music generation system for this project.
A survey by Suntory in the fall of 2020 showed that in the new lifestyle triggered by the COVID-19, health behaviors related to diet, such as “taking breakfast” and “eating vegetables first,” have increased compared to 2019, indicating that people are becoming more conscious of their diet. Therefore, Suntory Iyemon Tokucha brand decided to develop “TOKUCHA MUSIC”, a service that easily converts your daily meals into music. Behind this project, we try to give users awareness of the opportunity to be healthy in a fun way through music.
This system is based on our original deep learning model (Variational Autoencoder using Transformer as encoder and decoder) and can generate multi-track MIDI files with drums, melody, bass line, and harmony based on conditions such as pitch height, note density, chord progression, and balance of consonance/dissonance
In this system, the nutritional balance of the diet entered by the user is calculated in terms of point ranges of food groups, called the four-gun point method. It is then used as an input for the deep learning model that creates music whose atmosphere reflects this nutritional balance.
If the user has a nutritional imbalance or too much fat or carbohydrates, the system will generate dissonant music with minor chords. On the contrary, if the user has a healthy meal with appropriate fruits and vegetables it will have light, up-tempo music. In addition, by using musical instruments that match the genre of the food, the atmosphere of the meal is represented in the music (Koto or Shakuhachi for Japanese food, flamenco guitar for Spanish food, etc.). Since the users can choose from about 2,000 different meals, they can repeatedly enjoy the countless variations of music generated by the AI. We used Web Audio technology to synthesize the instrument sounds and built the system to run on a web browser, including mobile devices.
In this way, this project has succeeded in creating an opportunity for people to review their daily dietary habits in a fun way through music by using AI music generation technology.
This project is also an interesting example of AI-based sonification and musicification of data. Moreover, it is impossible for human composers to generate countless variations of music based on specific data input, which can be seen as a unique form of music made possible through AI. Qosmo will continue to explore new applications of music and new types of music by using AI.
Takashi Kawashima（Studio Kawashima）
Yuki Sakurai（Metamosphere inc.）
Ryosuke Sone（Dentsu Craft Tokyo）
Maho Tada（Dentsu Craft Tokyo）,Jun Kato（Dentsu Craft Tokyo）,Kikuko Ando（Dentsu Craft Tokyo）
Tatsuya Maniwa（Dentsu Craft Tokyo）,Sakiko Yasue（Qosmo）, Yumi Takahashi（Qosmo）
Hiroyoshi Murata（Dentsu Craft Tokyo）
Eiki Kurokawa（Dentsu Craft Tokyo）,Atsushi Asakura（Dentsu Craft Tokyo）,Yuki Tanabe（Dentsu Craft Tokyo）
Kei Sato（Dentsu Craft Tokyo）
Hajime Sasaki（MountPosition Inc. ）
Masayuki Noda（invisible design lab）
Ikuko Komamoto（Dentsu Creative X）
Keiko Taguchi（Dentsu Creative X）,Shiho Sasaki（Dentsu Creative X）,Yui Mamiya（Dentsu Creative X）