Real-time camera reflections

Hey!

So in another thread, @Deim pointed out that real-time environmental reflections from the camera feed were provided by the Babylon SDK, as detailled in the git page.

Little typo spotted in the docs: I guess the declared const should be “envMap” to match the following code snippets :wink:

image

Also I think the camera.updateFrame(); line has gone missing here before the env update (actually envMap now :wink: ), especially if the order matters:

image

Unfortunately it doesn’t work here :frowning:
I created a simple sphere attached to the face in the default Face Tracker template, and assigned it a basic reflective PBR material:

const sphereMat = new BABYLON.PBRMetallicRoughnessMaterial('sphereMat', scene);
sphereMat.metallic = 1.;
sphereMat.roughness = 0.;

const envMap = new ZapparBabylon.CameraEnvironmentMap(camera, engine);
scene.environmentTexture = envMap.environmentMap;
//scene.environmentTexture = new BABYLON.CubeTexture("./assets/hdr/environment.env", scene);
//sphereMat.environmentTexture = envMap.environmentMap;

const sphere = BABYLON.MeshBuilder.CreateSphere("sphere", {}, scene);
sphere.material = sphereMat;
sphere.parent = trackerTransformNode;

window.addEventListener('resize', () => {
    engine.resize();
});

// Set up our render loop
engine.runRenderLoop(() => {
    camera.updateFrame();
    envMap.update();
    scene.render();
});

And the sphere is shown plain black with a specular spot from the default hemisphere light.

If I rather assign a static cubemap to my scene.environmentTexture (= 1st commented line), everything works as expected though (= the sphere reflects the assigned static cubemap).
I tried also assigning the CameraEnvironmentMap to my custom material (sphereMap, cf 2nd commented line), both instead or in addition to the scene environment, but none works either.

Am I doing something wrong, or is this still being worked on?
Really excited to see this operational soon! :slight_smile:
Thomas

Thanks for the report. Will look into this and get back to you with any relevant info. In the meantime, here’s an example you can follow along https://github.com/zappar-xr/babylonjs-example-instant-tracking-real-time-env-map