AI Music Generation Based on User Data



At Dentsu Craft Tokyo, we created a new web service “TOKUCHA MUSIC” for Suntory Iyemon Tokucha, one of the pioneers of green tea functional beverages with specific medicinal benefits. “TOKUCHA MUSIC” is that uses AI to generate music based on user specified daily meals. The tempo, melody, and tone of the music change depending on the nutritional balance and type of food. The goal was to give people a new perspective on the balance of their daily diet through music. Qosmo has provided our original AI music generation system for this project.

Good balance of nutrition

Bad balance of nutrition


A survey by Suntory in the fall of 2020 showed that in the new lifestyle triggered by the COVID-19, health behaviors related to diet, such as “taking breakfast” and “eating vegetables first,” have increased compared to 2019, indicating that people are becoming more conscious of their diet. Therefore, Suntory Iyemon Tokucha brand decided to develop “TOKUCHA MUSIC”, a service that easily converts your daily meals into music. Behind this project, we try to give users awareness of the opportunity to be healthy in a fun way through music.


This system is based on our original deep learning model (Variational Autoencoder using Transformer as encoder and decoder) and can generate multi-track MIDI files with drums, melody, bass line, and harmony based on conditions such as pitch height, note density, chord progression, and balance of consonance/dissonance

In this system, the nutritional balance of the diet entered by the user is calculated in terms of point ranges of food groups, called the four-gun point method. It is then used as an input for the deep learning model that creates music whose atmosphere reflects this nutritional balance.

If the user has a nutritional imbalance or too much fat or carbohydrates, the system will generate dissonant music with minor chords. On the contrary, if the user has a healthy meal with appropriate fruits and vegetables it will have light, up-tempo music. In addition, by using musical instruments that match the genre of the food, the atmosphere of the meal is represented in the music (Koto or Shakuhachi for Japanese food, flamenco guitar for Spanish food, etc.). Since the users can choose from about 2,000 different meals, they can repeatedly enjoy the countless variations of music generated by the AI. We used Web Audio technology to synthesize the instrument sounds and built the system to run on a web browser, including mobile devices.

In this way, this project has succeeded in creating an opportunity for people to review their daily dietary habits in a fun way through music by using AI music generation technology.

This project is also an interesting example of AI-based sonification and musicification of data. Moreover, it is impossible for human composers to generate countless variations of music based on specific data input, which can be seen as a unique form of music made possible through AI. Qosmo will continue to explore new applications of music and new types of music by using AI.





  • Creative Direction

    Takashi Kawashima(Studio Kawashima)

  • Technical Direction/Machine Learning

    Nao Tokui(Qosmo)

  • Art Director

    Yuki Sakurai(Metamosphere inc.)

  • Planning

    Ryosuke Sone(Dentsu Craft Tokyo)

  • Produce

    Maho Tada(Dentsu Craft Tokyo),Jun Kato(Dentsu Craft Tokyo),Kikuko Ando(Dentsu Craft Tokyo)

  • Project Management

    Tatsuya Maniwa(Dentsu Craft Tokyo),Sakiko Yasue(Qosmo), Yumi Takahashi(Qosmo)

  • Audio Development

    Robin Jungers(Qosmo)

  • Machine Learning

    Christopher Mitcheltree(Qosmo)

  • Technical Direction

    Hiroyoshi Murata(Dentsu Craft Tokyo)

  • Front Engineering

    Eiki Kurokawa(Dentsu Craft Tokyo),Atsushi Asakura(Dentsu Craft Tokyo),Yuki Tanabe(Dentsu Craft Tokyo)

  • Direction Support

    Kei Sato(Dentsu Craft Tokyo)

  • Backend Engineering

    Hajime Sasaki(MountPosition Inc. )

  • Sound Design/Sound Produce

    Masayuki Noda(invisible design lab)

  • Design

    Ikuko Komamoto(Dentsu Creative X)

  • Copy Writing

    Keiko Taguchi(Dentsu Creative X),Shiho Sasaki(Dentsu Creative X),Yui Mamiya(Dentsu Creative X)


This project uses Qosmo Music & Sound AI.

Get in touch with us here!