Gesture Synth Tutorial: How to Play Music With Hand Gestures

The first session is easier if you make one small sound before trying to use every control. Gesture Synth turns hand movement in front of a webcam into notes, chords, pitch, volume, drum hits, and loop tracks. You do not need a MIDI keyboard or a download. A laptop or desktop with a webcam, speakers or headphones, and a current browser such as Chrome or Edge is enough.

If you already know which instrument you want, go straight to the Gesture Synth chords guide or the Gesture Synth piano guide. The hand gesture music library has all four instruments in one place.

Before you start

Open the Gesture Synth instrument. Sit far enough from the camera that both hands fit in the frame. Face a light source so your hands stand out from the background. A plain wall helps, but you do not need a special room.

When the browser asks for camera access, choose Allow. Gesture Synth reads hand landmarks locally in the browser. It does not upload the live camera feed. The first load may take a moment while the hand-tracking model is cached, so leave the page open until the setup message changes.

Click once inside the instrument after setup finishes. Browsers often keep Web Audio silent until the first click. If there is still no sound, check the tab volume, system volume, and headphone connection before touching the gesture settings.

Try chords first

Gesture Chords is a comfortable starting point because you can make a progression without aiming at individual keys. Choose the Easy C-major chords preset and hold both hands in view.

The left hand chooses the scale degree. One raised finger selects I, four selects IV, and five selects V in the starter setup. The right hand shapes the sound. Raise it for volume and tilt it to change the tone. The full settings add more chord and octave choices, but the preset keeps those details fixed while you learn.

Play this progression one chord per bar:

1. Raise one finger on the left hand and lift the right hand until you hear I.
2. Hold it for four beats.
3. Change to four fingers for IV.
4. Change to five fingers for V.
5. Return to four fingers for IV.

Do not rush the changes. A clear pose held for a moment works better than a fast pose that the camera has to guess. If the chord label jumps, separate your fingers and try again more slowly. The chords guide has a printable starter chart and more progressions.

Play single notes with Gesture Piano

Gesture Piano is a better fit for a melody. Face your palm toward the camera, move a fingertip over the key you want, and curl it down to tap. Lift the fingertip before the next note. All five fingertips on either hand can play, so one hand can handle a melody or both hands can play separate notes.

The down-and-up movement is the part to watch. Holding a fingertip above a key does not repeat the note. Start with neighboring keys and a slow pattern. Once the taps land consistently, use the other hand for a bass note or a simple harmony. The piano guide covers camera position and missed-note fixes in more detail.

Try the theremin and drums

Air Theremin uses continuous movement instead of fixed keys. One hand controls pitch and the other controls volume. Move through the camera space to sweep the pitch, then raise the volume hand to bring the sound in. It works well over a chord loop because the line does not need to land on exact notes.

Air Drums uses your index fingers as drumsticks. Point at a drum or cymbal, tap down, and lift before the next hit. The kit has a hi-hat, snare, three toms, and a crash. Start with hi-hat on eighth notes and snare on beats two and four. The Air Drums guide walks through that first beat.

Make a four-track loop

When one part feels steady, use the loop controls at the bottom of the instrument. For a first attempt, set 96 BPM, 4/4, and four bars.

The Gesture Synth Weld guide shows the chord-loop process in detail. In brief, select Track 1, press the red record button or R, wait for the count-in, and play I, IV, V, IV, one chord per bar. After the fourth bar, the track starts looping. Select Track 2 to add a held theremin note, a piano phrase, or a simple drum pattern.

You can mute, solo, clear, and adjust the volume of each track. The detailed guide also covers cell editing and keyboard shortcuts, so you do not need to learn those controls before making your first loop.

Save the result

There are two ways to save a performance. Export in the loop controls saves the loop mix as audio WebM. The top Record control captures the camera canvas and synthesized sound together as video. Current browsers may save MP4, while browsers without MP4 support save WebM instead.

Use Export when you only need the music. Use Record when the hand movement and instrument visuals matter. The browser creates both files locally, so you can make another take without uploading the first one.

When the instrument misbehaves

• No camera: allow camera access in the browser's site settings and reload.
• No sound: click inside the instrument, then check the tab and system volume.
• Missed piano notes: slow down and lift the fingertip between taps.
• Unstable chords: use the Easy C-major preset and hold each pose longer.
• Missed drum hits: point at the center of the target and make the downward tap clearer.
• Slow tracking: improve the light on your hands and remove a busy background.

Start with one four-bar loop, add one layer, and record it. If that works, change one thing on the next take: the tempo, chord quality, or a short theremin line. The instrument becomes much easier to learn when each take answers one small question.

Open Gesture Synth and start playing