Images disappear when user-facing camera is active

[shot]

I have a couple PNG images as part of a tracked-image experience. Everything works as expected except when the user-facing camera is active. In that case, the images (which are children of the .ZPT object) disappear. Bizarre!!! The 3D model remains visible, and tracking seems fine.

Anyone have any ideas why that would be?

EDIT

There’s definitely something strange going on here! Either I’m missing something obvious, or this is a pretty big deal. It’s not specific to PNG images. I also tried a JPG image, and the same thing happens. It seems any image that is the child of a tracked image (.ZPT object) doesn’t render when the user-facing (selfie) camera is active.

Again, the experience works as expected when pointing the back-facing camera at the tracked image; but simply switching to the user-facing camera (and turning the device around as if for a selfie) causes all child images of the tracked image to disappear! What the…

Anyone else ever encountered this or have any thoughts on what the issue might be?

[stevesanerd]

I have to ask are you placing the tracking image from the ZPT on your face?

I guess I’m look to see what your trying to do to see if i can help more.

Steve

[shot]

Thanks for the reply @stevesanerd . To answer your question, no, there’s no face tracking involved. It’s just a simple, single image being tracked (which I’ve done before in a previous project). Also, my face is not even within the field of view (not that it should matter since there’s no face tracking involved). I’m off to the side looking at the screen as I point the user-facing camera at the tracked image.

What’s so strange is that it works perfectly with the front-facing camera. It’s only when viewed through the user-facing camera that child images of the ZPT object disappear. I can’t even imagine why the camera being used would affect what’s being rendered on screen.

[stevesanerd]

I see. I trying to figure out what you where doing. I have seen hats and shirts that you would use the front facing camera.

I wonder if it’s because some front cameras flip your images. I haven’t played with it. I’ll have to look at it.

Steve

[shot]

Imagine a large poster on a wall. That’s the tracked image. I would simply like people to be able to take pictures of their friends (via the front-facing camera) or themselves (via the user-facing camera) with the poster and AR content in the frame. There’s no face tracking and no manipulation (positioning, resizing, etc.) of the AR content.

[shot]

I wonder if it’s because some front cameras flip your images.

Not sure what you mean there. The images I’m referring to are AR content - i.e. a native Studio “Plane” object with an image texture.

[stevesanerd]

Some front facing cameras on devices mirror it’s images you see on your screen. I don’t know but it maybe causing problems with 2d (png,jpg) if not it could be something with zappar. You said it works fine with the back camera.

Steve

[shot]

Here’s the kicker (which I just learned)… If I create a plane using modeling software and import the simple mesh (just a rectangle) into Studio, then an image applied as a texture to that plane DOES appear in the user-facing camera!

That’s a really cumbersome work-around, but at least I can move forward! Thus, it does seem to be a ZapWorks issue. I’m not sure how to submit an official bug report, but this certainly seems to be worthy of one.

[stevesanerd]

Just email them at support@zappar.com.
Maybe they will give a you a better fix.

Steve

[shot]

Some front facing cameras on devices mirror it’s images you see on your screen.

Hey @stevesanerd, turns out you were right. I heard back from support, and that’s apparently exactly what’s happening. Setting “cull faces” to “none” should make the images visible (although I haven’t tried it yet).

[shot]

I did some testing, and I believe there’s still an issue to be addressed. Following is the gist of my follow-up reply to @George at Zappar support. I’m awaiting a response.

Imagine someone standing in front of a real-world object with writing on it, such as a large billboard, and they take a selfie (via the ZapWorks API). The final captured image (which the user can save or share) is “backward”. The exact same scene captured “natively” via the device using the same user-facing camera results in a properly oriented image.

So basically, an image captured via the ZapWorks API from the user-facing camera should be flipped (i.e. rotated - not mirrored) along the vertical axis. I checked the camera documentation but didn’t see an option for this.

Mind you, this is completely independent of the AR content “mirroring”. To illustrate what I mean, see the following images captured with the user facing camera on the same device (an iPhone)…

Captured with User-facing Camera via Device Camera App

Captured with User-facing Camera via ZapWorks API

EDIT: I should add that this is actually a roadblock for my current project, since users will be encouraged to take a selfie and post it to social media. If I release it as it is, all shared images will be backward - text and all! (Hopefully, I’m missing something!)

EDIT 2: I suspect this is because the ZapWorks-generated image is really just a “screenshot” and not a “photo” per se (which is also why the aspect ratio is different). Regardless, there’s definitely a need for an option to “flip along the vertical axis”.

[stevesanerd]

Here’s something you may not be thinking about. If the user is standing in front of the tracking image to take the selfie they may block to much of the image for zappar to track.

Steve

[shot]

Not only have I thought about it, I’ve tested it, and it works great. The AR experience is “life size”, and the tracked portion of the image is not obstructed when the user stands in front of it to take a selfie. Tracking works just fine. Of course, the user needs to position themselves such that both they and the relevant portion of the tracked image are in the frame, but that’s easy.

I have a number of other such experiences planned, so I really hope the image flipping will get fixed. I definitely don’t want to release the experience when there’s the potential for a bunch of backward images to be posted online. That wouldn’t reflect well on me or Zappar.

I’m hoping there’s an easy work-around or that it will be fixed quickly. It honestly seems like an oversight (and like it should be simple to fix).

EDIT: Of course, a group of friends can take pictures of each other with the front camera, and those will look fine. It’s just that selfie’s will be backward.

[stevesanerd]

Well if you look at most selfies they are backwards. So it may not be the end of the world.

[shot]

Well if you look at most selfies they are backwards.

Not true. Most selfies are in fact “real” photos - not screen captures. The first image I posted demonstrates this. It was taken with the user-facing camera - i.e. I took a selfie without me in it. Yes, it looks backward WHILE taking it, but that’s just for the benefit of the person taking the selfie (better UX). The final captured shot is properly oriented.

So it may not be the end of the world.

It’s about the specific experience I’m trying to create. Rotating the captured image 180 degrees about the vertical axis (or at least providing the option to do so) is likely trivial for the Zappar dev team. How soon a new release of Studio might be coming is another issue entirely.

EDIT: One last comment… A service focused on AR should put as much weight on the “R” as on the “A”. Currently, selfies captured with the ZapWorks API simply don’t represent the “reality” of any real-world objects in the frame - including people. IOW, the left side of your face is on the right side and vice versa when a selfie is captured with ZapWorks API.

[alancarr14]

I see. I trying to figure out what you where doing. I have seen hats and shirts like vlone that you would use the front facing camera.

I wonder if it’s because some front cameras flip your images for vlone apparel. I haven’t played with it. I’ll have to look at it.

Steve

i am not confirmed what you are about. but I guess it will work fine.