Hand tracking with Zappar!


#1

Just experimenting with Zappar Three.js and hand tracking ml models, what do you guys think?:raised_hand_with_fingers_splayed:


#2

NICE!!!
Now just need it in unity :slight_smile:

Steve


#3

Nice work @mihir!

Is this the hand tracking model from mediapipe, or something else?

Just wondering what benefits you’re getting with using the Zappar Universal AR SDK in this example - it doesn’t seem to be making use of any of our in-built tracking types? Our SDK also doesn’t really expose an API to allow plumbing through user-provided computer vision code with the rest of our computer vision pipeline to ensure everything stays in sync. That is something we may expose in future if there’s sufficient demand, so I’d be interested to hear more about your use case.

Hand tracking in general is something we’re looking at, initially targeting use in ZapBox although we’re also keen to see what we can do on the web for mobile devices, and whether we can achieve smoother frame-rates on par with our other tracking types. The ML models tend to be more complex than face tracking ones as hands don’t have as many distinctive features, and have more potential configurations, which usually results in pretty slow frame-rates on mobile hardware as in your example.

Nice to see it working though, and an interesting experiment for sure :+1:


#4

Hello Simon,
Thank you for your response. You guessed it correctly it is mediapipe and as you stated the frame rates drops alot due to the tracking methods.
As for Zappar I am not fidgeting anything with the in-built tracking types but using zappar for just the AR camera access (setting the scene background) while the tracking is handled by mediapipe and Three.js is used for rendering the 3D model. I could achieve the same results by accessing the camera directly and rendering the camera feed on the canvas but I wanted to try handtracking with Zappar and as Zappar has easy declaration of elements like in case of Three.js renderer, scene and camera the implementation is pretty similar to the Three.js implementation for mediapipe.


#5

Wow this is amazing! Mind to share the tips how to develop? :smiley: I tried before but i kept failed:sweat:


#6

If you can dm me I can guide you through all the resources and the way I choose to go through it.