Control PC games with body movements using webcam andPython+ MediaPipe
I’ve been experimenting with building a universal gesture-based game controller — the idea is to let you play PC games using your body instead of a traditional controller or keyboard.
It uses:
MediaPipe for body pose detection
OpenCV for image processing
A custom Python pipeline I wrote to map gestures → keypresses
A standard webcam (placed above the screen) — no VR hardware or special equipment needed
Here’s a short demo of it running with Ryse: Son of Rome: https://www.youtube.com/watch?v=0k5jMM-aK1M
I’ve also tested it on other 3rd-person action games (Assassin’s Creed, The Witcher, Star Wars Jedi Fallen Order). The system sends configurable keypresses based on detected body movements — for example:
Stepping to move the character
Arm movements to attack or block
Leaning to dodge or strafe
It works surprisingly well for games that emphasize movement and timing over precision aiming.
I’m sharing this because:
I’d love feedback on the technical approach
I’m curious what other applications people can think of (beyond gaming?)
I’d like to find a few testers to help refine the system and test across more games
Eventually I’d like to turn this into an easy-to-configure tool — but for now it’s a fun experimental project. If anyone here is interested in the code, the design tradeoffs, or trying it out, I’d be happy to share more details.
You've basically re-created the Kinect with a regular RGB webcam, right? It's a cool project, but if it's limited to emulating a controller (instead of, say, being able to control individual limb and weapon directions like with VR or Wiimote controllers), what do you think the target audience will be?
It seems like between the Kinect itself, Intel RealSense, the PlayStation Camera and other knockoffs, this has been tried many times without much market uptake. Maybe gamers are just lazy and prefer using only our thumbs instead of whole bodies, lol.
It might be fun for fitness games, though. Or any chance it could emulate VR controllers instead to let people without headsets play VR games with just their bodies?
There's a great parody video when the Nintendo Wii first came out where it showed two very different gamers playing Wii Tennis. Player one was completely throwing their entire body into it Kramer style, and the second player was sitting on the couch with a faintly bored expression on their face barely twitching their wrist to activate the motion controls.