Adnan, Martha Janicki, Gabe Warshaw

For our final ICM project we wanted to create a music effect controller that would use posenet and Ableton.

Design

We knew we wanted a physical controller, a webcam stream that was running Posenet and some way to visualize the audio.

The physical build was a box that allowed control for the following parameters: Start, Stop, Reset, EQ, Auto Pan, Transpose, and Beat Repeat, as well as 8 different music loops.

The visualizer was created in p5 using an fft analysis and then received the audio input from ableton.

Posenet data was incorporated into the interactive component of the visualizer by taking the X and Y positions of the user’s wrists and sending them into Arduino. The data was collected and smoothed in p5.js then sent to Arduino. Once in Arduino, the positions were mapped to a range between 0 and 127, so they could be read by Ableton.

The visuals

We wanted to have a video capture element that would feature a few different ways to change the stream in like a pixelated effect as well as a color tint.

After experimenting with this idea for a little bit, it didn't quite line up with the final vision we had and also felt that it took away from the actual audio of the

The final visual output we got on screen was a combination of the video stream and the audio visualizer. When the user presses one of the FX buttons, 2 blue spheres would appear over the users wrists and a timer would count down from 20 seconds to allow for the user to continually change the amount of the effect being applied to the current song.

Here is the user flow with the tool: