arkade
conduct your sound
every bedroom producer knows the frustration of having a perfect melody in their head, but when
they get to mixing, knobs and sliders kill the creativity. making music should feel magical, not like homework.
after watching friends give up on their songs because mixing felt impossible, i built arkade to let you shape sound
with your hands, making it as natural as conducting an orchestra.
experience the limited demo here.
what sparked the project
i’ve been playing piano for ~14 years and recently got into music production. however, things took an unexpected turn in 2021 with a personal loss in the family which prevented me from getting back to making music, despite the deep passion. i owed myself a favour to get back into what i loved doing the most and this opportunity seemed perfect for that.
tangible or not tangible?
as you shall see in my other projects, i love making tangible things. so, i put this constraint
on myself for this project. as a designer, i’ve always admired products that deliver delight not just through
function, but also through form.
photo credits: jetbu, telepathic instruments, axe and you shall receive.
hypothesis + research
with plenty of exciting ideas in the field of music, i was instantly overwhelmed. to help me get
out of this situation, i came up with a hypothesis: music producers struggle to come up with a melody for their next
project because they’re spoiled by choices around them.
to test this hypothesis, i quickly reached out to a bunch of music producers of all types — hobbyists, professionals,
and even those who’ve designed musical devices like the nsynth by google.
photos censored for privacy reasons.
outcomes of the research
through the interviews, not only was my hypothesis proved wrong, but i also made some interesting discoveries.
out of all these, i found #3 and #4 to be the most interesting.
quotes by participants
it was not just my own interest that i picked #3 and #4. they were backed by what some of the participants had seriously said about the aforementioned discoveries during the interview.
but... what is mixing?
imagine you’re listening to a new release by your favourite artist and the drums are
overpowering the vocals. might as well find a new favourite artist at that point. here’s where mixing comes in. it’s
one of the final steps in music making where the various instruments (or tracks) are given an additional flair
through different effects.
here’s an example of a mixed track and an unmixed one (audio credits: mub recording studio on youtube).
the underlying question
you might wonder “if they can learn how to make music, why can’t they learn how to mix their
songs?” and that’s a fair question. it’s because when it comes to mixing, the interfaces look this.
a bunch of knobs, sliders, and confusing audio effects. it’s built for engineers, not artists.
the design intent
this isn’t about teaching artists more knobs. it’s about designing tools that speak their
language. we need interfaces that listen to intention, not just control output. the tools meant to finish music are
making people quit music.
i wanted to design for clarity and discoverability while maintaining the essence of experimentation.
how might we
after lots of iterations on the how might we statements, this is what i finally decided to go with:
how might we help emerging music producers understand mixing without needing deep technical knowledge so that they can
focus on creative expression and experimentation?
collaborative ideation
one of my favourite things about this project was the two collaborative ideation sessions we did for each other in the cohort. after a quick demonstration of my problem statement, a couple of my friends came up with lots of amazing out-of-the-box ideas. thanks alec, jill aneri, rohan, dishank, leon, ali, wayne, and hun!
the first prototype
after going through all the ideas and voting on the most interesting ones, and since i already had an affinity towards building something tangible, i naturally jumped right into putting pieces together so i could quickly address the technical concerns like midi connection, compatibility, etc. and move forward.
the realisation
soon enough, i realised i was wrong. in my head, it felt like i was building yet another midi
controller, which was the complete opposite of what i had in mind initially. i felt directionless.
photo credits: eduardo cano photo co.
taking things literally
then, to break away from this approach, i wondered "what if mixing didn't have to feel
technical? what if i could turn technical jargon into something intuitive, something that feels like play?".
some of the basic controls/effects that are commonly used in mixing are volume, reverb, equaliser, panning, and
compressor. but, for the sake of the project demo, i decided to swap compressor with phaser for a more pronounced
effect.
more concepts...
with the same constraint of building something physical and tangible in mind, i started to explore some concepts and prototypes for the above mentioned effects.
'mixed' feelings
soon after testing these prototypes, i realised that they felt gimmicky, not musical. i was designing toys, not instruments. musicians don’t need more devices, they need better interactions.
a pivotal moment
the volume interaction with the gloves, however, proved to make the most sense. keeping that in
mind, i stepped back and watched musicians work. what if the interface disappeared entirely? what if your hands
became the instrument? every musician already speaks with their hands. I just needed to translate that language to
mixing.
photo credits: alex cooper, divyadarshi acharya, michael benz
shortlisting the tech
i began to list down all the possibilities where hand gestures could be used as a mode of
interaction: virtual reality, mixed reality, and computer vision. virtual and mixed reality need another device that
my target group might... not have? we’re not there just yet, let’s be honest. the only other option was computer
vision, and that’s what i went with!
so, my hands became the interface, and my webcam — the sensor.
prototyping the prototypes
i started prototyping interactions and gestures, using volume as my base. i tested multiple
approaches like raising or lowering hands, spreading fingers on one hand, spreading both hands, and more. none felt
intentional, so i settled on pinch gestures for both hands.
at one point, i wondered: what if these effects could be represented in 3d? audio is typically shown in 2d waveforms,
rarely three-dimensionally. so i developed shapes that would represent these effects spatially.
small gestures matter
gestures are the most important part of this experience. they needed to be understandable, accessible, and easy to remember. i also had to take into account that i had limited choices of gestures because the webcam cannot track your hands in 3 dimensions.
for menu
let go to select
control effects
control audio direction
putting it all in the mix
the prototypes were only melodies, and now for the final mix, presenting arkade 1.0! with each iteration and testing, arkade slowly evolved to morph it into what it is right now.
what people think
what i think
arkade is just getting started. in the next version, i’m hoping to add a 3d tracking camera for
accurate hand tracking, opening up new gestures and interactions. i’m interested in creating an invisible interface.
what if you could sense changes in value or select effects without a ui? what if you could customise them? what if
this worked live on stage? these are some questions i want to explore going ahead.
that being said, thank you for sticking around till the end!