About audio with auto-playing video

Hello,
I’m currently using UniversalAR (A-Frame) to verify video playback with image tracking.
The video can now be played automatically at the recognized timing of image tracking. However, you need to mute the audio due to browser restrictions.
I would like to know if you have any ideas on how to play this audio.
The video that ZapWorks Studio created played the audio along with the auto-play.
If ZapWorks Studio can do it, I think UniversalAR (A-Frame) can do it too.
If you have any ideas or sample code, it would be greatly appreciated if you could share it with us.

Code

Best regards.

Sadao Tokuyama
tokuyama@1planet.co.jp
https://1planet.co.jp/

You could add the audio file along with the video file and as you call the video.play() you can also add audio.play() and audio.pause() with video.pause() and it would work smoothly. Just make sure the audio file is external or you can try the aframe sound element for the audio. below is the implementation:

function onload()
      {
         myImageGroup = document.getElementById("image-group");
         eVideo = document.getElementById("e-video");
         video = document.querySelector("#video");
         audio = document.querySelector("#audio-file");

         myImageGroup.addEventListener("zappar-visible", () => {
            video.play();
            audio.play();
            eVideo.setAttribute("visible", true);
         });
         myImageGroup.addEventListener("zappar-notvisible", () => {
            video.pause();
            audio.pause();
            eVideo.setAttribute("visible", false);
         });
      }

HTML:
<audio id="audio-file"> <source src="audioSource.mp3" type="audio/mpeg" /> </audio>

@mihir Thanks for the advice. I followed your advice, but the audio does not play. The audio plays on my computer, but not on iOS and Android. audio is also not playing without user action due to browser restrictions, right?

yes you are correct you take a look at howler.js as that is what most websites tend to use but as of now there needs some user interactions to play the audio.

As mentioned in the first post, WebAR created with ZapWorks Studio plays the audio of the video at the time of tracking, and I would like to know how ZapWorks Studio achieves this.

I had a similar issue before when working with A-Frame and multiple videos / makers. Unfortunately, iOS requires and interaction per element to produce sound and my A-Frame project required an A-Entity for each video.

I solved my autoplay sound issues by switching to Three.js. I have a single video element, that receives permission to play when the user taps a “ready” button and I swap out the video src each time a new marker is found. Swapping out the src doesn’t revoke the permission previously granted to the element, so new videos auto play with sound. Hope this helps in some way.

@kevinmerinsky Thanks for the detailed information. There is a difference in behavior between A-Frame and ThreeJS, isn’t there? Since both are JavaScript frameworks, I thought there was no difference in sound control. By the way, you mentioned iOS, but did you have to deal with Android as well?

I also find it really strange that it is possible to achieve this with WebAR in ZapWorks Studio and not with UniversalAR.

I just hope that Zappar employees can really tell us about this.

Aframe is built on top of Three.js so anything built in Three.js should also work with A-frame.js given that @tokuyama have you checked howler.js?

I think your basic problem is that you need to have an click interaction for the browser to play a video with audio. iOS is harder as it requires an interaction per element to play a video with audio, not per page like other vendors. I solved this by quickly toggling the play state of the video when the user hits the grant permissions button at the start of the experience.

If you are only playing one video you should be able to use A-Frame. A-Frame is built on Three.js but A-Frame steers hard into [entity component architecture] (https://aframe.io/docs/1.2.0/introduction/entity-component-system.html) making it harder to do multiple videos.

Hi @tokuyama,

Thanks everyone for getting involved - this is really exciting to see and hopefully we will see more and more developers collaborate in the future! This is definitely more of a mobile browser/A-Frame issue - that said, I have some thoughts on the topic which may be helpful!

As you may know, some modern mobile browsers (especially iOS) require a user interaction before certain things can be done, such as the playing of media. In the case of projects created by ZapWorks Studio, it appears as if this is not the case. However, the Launch button on the splash screen is our way of getting that user interaction, as they must consent to launching the experience. :white_check_mark:

1

We’ve actually got a video example for Universal AR for A-Frame on the way. In fact, it just needs to go through some final checks before it goes on the Example Project Library; but as I developed it, I thought it wouldn’t hurt to provide you with one of the approaches I went with, which is pretty much pure Javascript. :desktop_computer:

I previously mentioned the Launch button - with Universal AR, we need to implement our own approach, as we do not get these splash screens by default, and if you’d like a splash screen, you will need to make your own within your project itself. We do have a handy package for that though if you’re curious.

In the case of the example that is (hopefully!) coming out soon, I simply include my video as an asset as usual, making sure to make autoplay false (otherwise it may not play at all):

<!-- ...assets... -->
<!-- Video -->
<video id="bigBuckBunnyVideo" preload="auto" autoplay="false" loop="false" src="https://cdn.aframe.io/videos/bunny.mp4" crossorigin="anonymous" muted></video>
<!-- ...assets... -->

Add it to my scene:

<a-video id="bigBuckBunny" src="#bigBuckBunnyVideo" scale="1.35 1 1"></a-video>

And then use buttons to play or pause the video. In the example, I use a-frame entities with a raycaster class to make them clickable and a custom component, but because the functionality is done with Javascript (we turn muted to false):

// Initialise our video in a variable
    const bigBuckBunnyVideo = document.getElementById('bigBuckBunnyVideo');

    // Register our play component
    // This component will play the video when an entity holding it is clicked
    AFRAME.registerComponent('play', {
        init: function () {
            this.el.addEventListener('click', () => {
                bigBuckBunnyVideo.muted = false;
                bigBuckBunnyVideo.play();
            });
        },
    });

It wouldn’t surprise me if you could do this with HTML elements as well, as I am targeting the video DOM element, rather than an a-entity. :slight_smile:

Hopefully this shows an example of how we can play videos in our A-Frame experiences. Unfortunately with the browser restrictions put on us, it’s up to developers to find their own workarounds.

I think the best solution here would be for your experience to have a welcome or splash screen to your experience if possible, as this can be a quick way of unmuting your video without your users suspecting anything.

Hopefully that was helpful!

Have a nice day!
Francesca :blush:

1 Like

Hi @Francesca
Thank you for the detailed information.
I was testing a UniversalAR + A-Frame implementation to validate that the video plane would automatically play when recognized by image tracking.

The advice recommended using a splash screen, so I tried the sample provided by Zappar, but I am having trouble implementing it because there are no examples of its use.
https://docs.zap.works/universal-ar/useful-packages/splash-screen/

If you have some kind of sample application that uses Zappar’s Splash Screen, it would be very helpful if you could provide it.

Also, this is a separate issue, but I would be happy to provide a sample application of webgl-snapshot.
Splash Screen and webgl-snapshot also have sample code for parts of them, but I want to understand the whole process flow.

You need to create your own splash screen as per @Francesca’s advice and get click events from it for the autoplay.

@mihir It is my hope that if you guys have any sample code that uses splash screen, you will share it with me.

Because the sample code for the key parts of the splash screen does not allow us to understand the entire splash screen process.

you could start here with an image and a simple button: https://www.w3schools.com/howto/tryit.asp?filename=tryhow_css_button_on_image
and create a simple js to hide the whole container using display:none style property on click of that button as well as add the code @Francesca has provided in this thread as a component on this button.

2 Likes

Hello everyone!

On the topic of including the splash screen package into your project - we actually have a physics example with the splash screen using Three.js here. I’d love to be able to put it on more examples in the future, but since our current example library focuses on specific functionalities, it may not make it’s way on to every one!

As you can see, it should be pretty easy to customise!

//...not shown: adding the package...

// On load, find our splash screen and display it
    ZapparSplashScreen.showUI(
      {
        onClick: (e) => {
          e.destroy();
          // Request the necessary permission from the user
          ZapparThree.permissionRequestUI().then((granted) => {
            if (granted) {
              camera.start();
              // Hide and show correct parts
              zapparPlacementUi!.style.display = 'block';
            } else {
              ZapparThree.permissionDeniedUI();
            }
          });
        },
        title: 'Universal AR for Three.js',
        subtitle: 'Image Tracking Cannon Physics',
        buttonText: 'Tap here to start',
        background,
        logo,
      },
    );

Have a great day!
Francesca :blush:

1 Like

Hi @mihir and @Francesca,
Thank you for all your advice.

I have created a sample based on the information I have received so far.
https://webxr.run/dZzzw4kX5Q9a9

marker

source

The sound was fine on my iPhone 12 Pro and Galaxy S8. However, on the iPhone 6S(iOS14.4, Safari 14.0.3) the sound sometimes works and sometimes does not.

The first time, if the image marker is not visible after the camera is activated, the audio of the video will not be played when image tracking is performed.

The second time, when the image marker is visible after the camera is activated, the audio of the video will be played when image tracking is performed.

Is there any way to solve this problem?

Best regards.

Sadao Tokuyama
tokuyama@1planet.co.jp
https://1planet.co.jp/

Apple annoyingly rejects many HTML5 web standards on their iOS devices, for a variety of business reasons. Ultimately, you can’t pre-load or auto-play sound files before a touch event, it only plays one sound at a time, and it doesn’t support Ogg/Vorbis.
One work around would be an actual click event on the video itself as you want to only play the video/audio when the target is visible

@mihir Even if you don’t set up click events for the video in ZapWorks Studio, the audio will still play on the iPhone 6S.

Hi @tokuyama,

Unfortunately, as @mihir kindly states, this is a limitation of Safari which we do not have control of. Since you’re technically building directly for the web (and not software like ZapWorks Studio), it will be up to you to decide how you workaround this issue.

According to Apple’s documentation - A element can use the play() method to automatically play without user gestures only when it contains no audio tracks or has its muted property set to true. (Source)

This would explain the complications. Additionally, if you have more than one video in your experience, Safari typically expects a user-initiated event for each video.

It will also help to make sure that your video is web optimised according to your target browser’s guidelines - you can do this with tools such as Handbrake, though there are many more resources for this online.

Have a great day!
Francesca :blush: