Hand-Tracking & Generative Art
These are my attempts to make computers that respond to the way humans actually move and gesture.
The other day I had this idea to try and control a modular synth using computer vision. I loaded YOLOv4 into a python script and had it detect objects in an episode of Atlanta I had lying around, sending the X and Y positions of every object as control voltage to the oscillators. It makes this weird, haunting, sort of robotic sound — but I kind of really like it. Then I had a fun idea to run it through the webcam, and I couldn't figure out why it wasn't working — I kept seeing the chair as an object. Then I figured it out: the chair is the instrument. Why did I make a chair an instrument? I don't know. I ended up running dance videos from Sherrie Silver, the This Is America choreographer, through it instead — not just the X and Y positions but the distances between the objects — and it comes up with some really cool, interesting generative effects. I don't really know how I use this right now, or what it's for exactly. That's kind of how I know it's art.
That session: Using Computer Vision As A MIDI Controller.
The other end of the spectrum: no camera, no screen at all. I've been using Talon to control my computer, so I tried to do the dishes and code at the same time — talking to Claude while my hands were in the sink, having it answer everything out loud. There are some sharp edges — a couple times it didn't pick up everything I said — but the fact that you can detach from looking at a screen and still engage in the mechanical task of writing code and making things with a computer is completely radical. It's unlike anything I've ever experienced in my life.
That one's on video too: Coding While I Do The Dishes.
See also Flipper Zero Tools and Terminal Tools.
Hand-tracking & gesture
Hand MIDI Controller
Hand MIDI Controller is a real-time hand-tracking MIDI controller built on MediaPipe and OpenCV, turning hand movements into MIDI control signals for music production, VJing, and performance. It includes TouchDesigner integration.
Handtrack WebSockets
Handtrack WebSockets does real-time 3D hand tracking with MediaPipe, OpenCV, and FastAPI, streaming hand position and rotation over WebSockets. It includes a cyberpunk visualization layer that renders the tracked hand.
OpenCV MIDI Controller
OpenCV MIDI Controller is an experiment in sending MIDI signals based on OpenCV object detection.
Speech Echoes
Speech Echoes is a Vue app that captures live speech and renders it as decaying typographic "echoes," with a font and weight captured per utterance. It's built with p5, MediaPipe, and d3-force.
Glasses HUD
Glasses HUD is a tool for quickly prototyping AR lens visualizations and HUDs. Drag data widgets onto the left and right lenses to compose a heads-up display from live bio, motion, location, environment, and device signals.