First Look - New and Improved World Tracking for WebAR

Hi everyone,

We’ve been hard at work on our new and improved World Tracking for WebAR implementation for a while now, and I’m really pleased to be able to share a first look now with our great forum community, just in time to see in the New Year!

This new implementation removes a number of the big limitations of our previous Instant World Tracking approach - you can now look away from the placement position and the content will continue to track, and we have also removed the requirement for a textured horizontal surface to place content.

Here’s a video of this first look build in action on a typically cloudy and wet winter’s day at the glourious British seaside :slight_smile:

You can try it out yourself by scanning the QR Code, or visiting this link on your smartphone.
eoPGozEaEVb6n

The source for this demo is available in the zip below, which also contains the standalone build of the Universal AR SDK for Three.js with the updated World Tracking implementation. This is exposed via the same InstantWorldTracker API as previously - simply set the pose of the anchor with a camera-relative position (and orientation option), and the library will aim to keep the content anchored at that position in the world as the user moves. In the demo, we reset this every frame during “Placement” mode, and then just stop calling the setAnchorPoseFromCameraOffset function once the user has placed the content. Other UX choices are possible with the same API.

RobotWorldTrackingDemo.zip (3.8 MB)

We’re sharing this now as it is in a state where we believe it is useful and it has a number of clear advantages over the previous implementation. There are still some caveats: it works best in static, well-textured environments, with smooth camera motion - moving the device position without rotating is best at the start (much like the general side-to-side motion that works best for initializing native ARKit and ARCore experiences). This implementation does also make portal-type experiences possible but as they often encourage fast rotational motion they are likely to suffer more from scale drift than outside-in content where the user is primarily focussing on content in front of them.

Zappar is officially closed this week for the holidays, so we’ll have much more to say about this early in 2022. One of the first tasks will be making this implementation available across more of our Universal AR SDKs and distribution types (such as making beta libraries available via npm), and looking at an update to the ZapWorks Studio WebAR sites too.

I’d like to thank the awesome Zappar team for all their work to make this first release possible. This is just the first milestone on our roadmap to a more complete World Tracking for WebAR implementation. We’ve got performance and quality improvements to come early in 2022, we’ll be working on specific improvements for portal-type experiences, and we’ll also be looking at API additions now we have the required groundwork in place for supporting other use cases such as multiple anchors and extended tracking (combining World Tracking and Image Tracking, for example).

Wishing everyone all the best for the New Year - we’re really excited by what we’ve got in store for 2022!

10 Likes

@simon this looks awesome! I have 2 questions:

  1. Is this already available on Studio? If not, when will it be?
  2. Are you guys using WebXR for this, or is it proprietary technology? What’s the minimum required iOS version?

Thanks guys. I hope you had a merry christmas and I wish you a happy new year! This is a great gift!

How about a particle system for Studio now? Hahaha.

1 Like

Hi @marks, thanks!

This will be used as a new implementation for the Instant Tracking API in Studio, so the plan is we’ll just update beta.zappar.app (and the lite-branded arweb.app) to include this implementation. We’ll likely do that in January, and then any Studio Instant Tracking content will benefit from the improvements mentioned above.

The biggest current reason not to switch immediately is the binary size impact - the web assembly is now 1.3MB compressed (vs 800KB) and 3.7MB uncompressed (vs 2.2 MB), so it will have an impact on download time and startup time. Later in the beta process we’ll look in more detail at opportunities for reducing the binary size impact.

With Studio WebAR sites the runtime is shared by all content, whereas in Universal AR each project can choose which version of the runtime libraries to use. We expect to keep iterating pretty rapidly on this implementation, so perhaps in the early days we can make it opt-in with a query string parameter or something on the Studio WebAR sites too.

This isn’t WebXR - it’s our proprietary computer vision code (in WebAssembly) using data from the standard web APIs for accessing cameras and motion sensors. So the required versions are the same as for the current libraries; effectively iOS 11.3 and up for Safari and 14.3 and up for other WKWebView browsers - full details here: https://docs.zap.works/universal-ar/javascript/compatibility/

3 Likes

@simon I sent an email called “new instant tracking urgent question” could you take a look? Thanks!

1 Like

Happy New Year everyone!

@marks - I don’t usually see the emails to support, but the learning and support team have forwarded it on to me. I’ll share the points arising here so everyone can see them. The quotes below aren’t straight from your email but I think I cover all the main points.

When can we start building Studio projects using this?

We have now discussed internally, and our rollout plan for the Studio WebAR builds is to update the bleeding-edge staging.zappar.app site (rather than the beta / lite-branded sites for now). That will make it usable for testing without the downsides mentioned above whilst we’re at this “early beta” stage. For more “permanent” deployments we will be able to deploy the same version to a custom Studio WebAR site.

There’s still some work needed to get the Studio WebAR site build updated to use the new implementation. It’s not something we can commit to a specific timeframe for at this point. As I mentioned, we’d hope to make something available this month but we’re not able to guarantee that.

Can we opt-in a project [I assume you mean a Studio project] to use this implementation before a wider roll-out?

Not on a per-project level. The runtime is part of the WebAR site for Studio content - so we have a different version on beta.zappar.app vs web.zappar.com, for example. So the level at which we can deploy this is at the site level, which means that a custom WebAR site could also be updated as mentioned above.

Yay, Portals!

As I mentioned in the first post, portal-type experiences aren’t brilliant at the moment. They’re technically possible, but will get better with future updates to the implementation. At the moment I’d definitely recommend testing to see if it meets your expectations before going that route.

In general I’m not a massive fan of portals from a UX perspective - you need a lot of floor area to make the “walk through” mechanic work, and space tends to be pretty limited in many indoor spaces (especially in homes). That said, we recognise that portal experiences are still in-demand, they are cool in the right spaces (outdoor or very large floorspace) and they do also make for good videos. For all those reasons we are committed to improving how well they work with this new implementation.

Finally, the Learning and Support Team are an awesome resource for all our “official” APIs and tools, but whilst we’re in this early beta phase we’re still heavily iterating on the implementation and so it doesn’t come with our usual high level of support. Though I can’t provide urgent responses I will keep an eye on replies to this thread and attempt to answer things on a best-effort basis here.

5 Likes

How’s the update for studio coming along?

1 Like

Hi there,
Just jumping in on this thread to see when the tracking will be available in a WebAR Environment, are we any closer? We’ve got some awesome projects to come.

Thanks,
Sean

1 Like

Hi @simon ,
Is there any update on this feature being available as a predefined template in Studio?

Regards,
Sean

1 Like

Hi sean1, there’s a webinar on that on the 8th of Feb…
Luca

Cool, where can I register for the webinar.

https://zap.works/webinars/introducing-new-and-improved-world-tracking-for-we/?utm_source=cta&utm_medium=banner&utm_campaign=webinar_planningforARsuccess

Hi @sean1 and thanks @lucalilli!

The new and improved World Tracking for WebAR will be coming to Studio in the coming weeks. Currently it’s on beta branches for our Universal AR SDKs but our dev team are working hard to get this implementation into Studio. As you say @lucalilli, it’ll be an easy to use template, setup in a similar way to the full world tracking subsymbol that exists in Studio at the moment.

I’d definitely recommend joining the webinar. If you can’t make the date, sign up and the recording will be sent after the session.

Hope this helps and look forward to seeing you in the webinar!

George

4 Likes

How’s the update for studio coming along?

Hi @marks,

No Studio update needed - the updated runtime is now available at https://wtbeta.arweb.app and should work with any Studio Instant Tracking project.

Here’s a video showing the demo content I posted on the forum back when we first introduced Instant Tracking:

1 Like

It’s not working with an instant tracking project I have. I scanned the zapcode using https://wtbeta.arweb.app and it still works as before.

My lite branded splash pages work using arweb.app?zid=z/XXXXX, however the lite branded splash pages reverts back to Zappar branding when I use the new link. Is this something that will change? If so, what’s the timeline?

p.s. It works great!! Thx for the new and improved WT

You should definitely be able to see the difference, although the new one is not without limitations - there’s a blog post going live shortly that will go into more detail.

In short for best results with the new implementation you’ll need to place the anchor near some visually textured feature, and move the device a bit with the anchor in view at the start (focusing mainly on translation rather than rotation). That will allow estimating the 3D position of all of those initial features in view, and attaching the content at the same depth as the feature closest to it in the image. From then on you should be able to move relatively freely, but try to avoid spinning around on the spot - that brings in a whole new set of features with unknown depth, and so no way to tie them into the estimated previous scale of the world map.

That’s one of the reasons why I suggest this implementation is still best suited for mainly outside-in experiences rather than things like portals encouraging lots of fast rotation. Global ground anchoring (on our roadmap) will help with scale consistency for those larger scale experiences in future.

Yes, right now the “test site” is Zappar branded, and we wouldn’t really recommend pointing live projects there, as we’re deploying changes to it quite regularly. For something more permanent without the Zappar branding we’re happy to deploy a fixed version to custom WebAR sites.

The lite-branded arweb.app splash is used for a lot of live projects, so we are a bit more cautious about rolling out large updates there. Once we have sufficient confidence in the quality and stability of this branch we will bring it to the lite-branded splash page too. It’s not something we’re able to put a timeline on yet I’m afraid as it depends on the real-world experience with the current beta one over the next few weeks. We’re likely to at least wait for one more milestone in terms of reducing wasm binary size and improving performance.

Thanks! Plenty more to work on, but it feels like a good step forward to us too.