Only track image in cropped camera video

Universal AR JavaScript
#1

I am currently trying to build a curved image tracking application using the Universal AR SDK. The goal is to track the label of a bottle to then display 3D content on top of it. Since the actual bottle is pretty small it was necessary to be very close to the bottle with the camera for it to start tracking the label. In order to prevent the need to be so close I had the idea to only track for the label in a cropped center part of the camera video.

Screenshot_20220331-112615__01
Screenshot_20220331-112615__011070×1886 121 KB

(not the actual bottle used)
The bordered center part would be the actual area for the image tracking. This way a user wouldn’t have to be as close to the bottle since the target would appear much bigger in the actual tracking area.

To try a few things I switched from the React Three.js SDK to pure JavaScript and WebGL as it allowed more manual control. I already tried two things:

  1. Instead of the default camera frame drawing I used the manual way by creating a fullscreen WebGL Quad but scaled the actual image texture by changing the UV coordinates. But sadly this only affected the displayed video but not the actual tracking, since the whole camera video was still being used for the image tracking.
  2. I switched from the regular Zappar CameraSource to an HTMLElementSource so I can use a video element as the basis. But to get a video of the camera only showing the center part I had to feed the camera video into a canvas, only painting the cropped center part and then streaming this canvas back into another video element. This way I was able to create a cropped video that I could feed into the Zappar image tracking but it also introduced other problems. The video always appeared skewed in some way inside the Zappar canvas (streched horizontally or vertically and parts of the video cut off) and it also had a noticeable performance impact. The Video inside the Zappar canvas lagged behind the actual camera video and had a lot of stutters, probably because of the multiple converting stages in between.

My question now is, if there is another easy way for something like this to work reliably and with good performance?

I appreciate any help or info on how I could achieve this :blush:

#2

Hi @guzy,

I believe we’ve spoken about this approach in support, however as this is an interesting theory, I thought the below dialogue may be helpful for others.

As it stands, this approach wouldn’t really work. This is because this is actually more of a target image optimisation query than a tracking technology question.

Due to the expected size of the bottle (meaning that the image will be relatively small), it’s likely a lack of image detail or resolution which would cause trouble. Since cropping the area where the tracking looks for targets won’t really make a difference (as there’s no ability for the tracking to ‘zoom in’/ add resolution), it will still look broadly for recognisable points (that it understands your trained image to have) and if it’s recognisable enough, will ‘understand’ that this is where content should track to.

The best solution for this (unless anybody wants to experiment!) is to make sure that target images follow our best practice guide to make it as easy as possible for the tracker to find those distinguishable points.

You may notice that one of the suggestions details placing content on a flat, matte surface - this is due to curved tracking currently being in beta. Feel free to continue playing with curved tracking! I would however recommend that your label is as matte as possible, as reflective surfaces can sort of ‘trick’ tracking and cause instability.

All the best,
Francesca :blush:

1 Like