The other day I saw the launch of Apple Vision Pro, the whole thing was very interesting, but one thing that caught my attention was the finger input. It seems very intuitive, by using the finger pinching as sort of like a cursor or mouse input. I figured I want to try it out, so I took it upon myself to create it.

Hacker News is a community-driven platform for sharing and discussing technology news, startups, and programming-related topics. Through user submissions and comments, Hacker News offers insights into emerging technology trends, industry developments, and entrepreneurial ventures. Readers can participate in discussions, share their insights, and stay informed about the latest advancements in technology and innovation.

Hacker News

The post describes a project that implements hand tracking to simulate mouse input using finger pinching, inspired by Apple Vision Pro. Initially, the author faced performance issues with the Python version of MediaPipe but found success using the web version. They managed real-time communication between the web frontend and Python backend via WebSocket. Later, they transitioned to using Tauri to build a more efficient desktop app with a Rust backend. The project also explored a mode inspired by Meta Quest for front-facing camera input. Various challenges like jitter and latency were tackled through techniques like the One Euro Filter.

Hand Tracking for Mouse Input

seems interesting i will chek it out and try it myself thank you for that :)

Really interesting read and very much out of the box problem solving! I loved the idea that moving averages could be the solution for a jittery mouse problem. This is pushing me to try this myself.
I wonder if we could implement and ship a solution so that anyone could do this out of the box.

awesome, weekends well spent i suppose :cool
i was wondering though how much time did took you to complete this project?
happy coding :)

javascript had better perf than python
(was the surprise to no one)
if only this held true for inter-process real-time audio 😞