The viz_webrtc plug-in allows Viz Engine to use peer-to-peer communication to be able to stream Viz Engine output to multiple clients. It opens a websocket connection between the extension and the client (browser) and then waits for the SDP response from the client to establish the peer connection. When the connection is complete, the video track is sent to the client as soon as a scene is loaded in Viz Engine. This extension also takes incoming input events from the client to be handled by Viz Engine allowing the user to trigger events, perform transformation on objects or trigger script events.

WebRTC in Software I/O Only Mode

Licensing

To operate viz_webrtc in Software I/O mode Parallel Outputs, NDI Out Channel and DVI Out HD or DVI Out Max Resolution (depending on what resolutions we want to support) needs to be licensed.

Configuration

To configure Viz Engine using WebRTC without Matrox support:

  • Uncheck User Defined in Check Video Card.

  • Set the Software I/O Mode to NDI.

WebRTC with Matrox Topology

Installation

The plug-in can be used in combination with a Matrox topology board, which means that DSX.core client must be installed if no Matrox board is installed.

Important: This procedure needs to be followed after installing DSX.core:

  1. Unregister mvfDsxCore.dll.

    1. Click Start > Run (or use the Windows command line: Search > CMD > (Right click) Run as Administrator).

    2. Type REGSVR32 /U "C:\Program Files\Matrox DSX-TopologyUtils\System64\mvfDsxCore.dll" and press ENTER.

  2. Shut down X.info in the task manager.

  3. Delete mvfDsxCore.dll from the folder C:\Program Files\Matrox DSX-TopologyUtils\System64\.

  4. Start X.info.

Licensing

To operate viz_webrtc Parallel Outputs needs to be licensed. If DSX.core is used, an additional licenses for either DSX.core HD License or DSX.core UHD License is necessary.

Common Configuration for Both Modes

The Viz Engine output enables WebRTC stream if a valid license is available and configured and the output is enabled in the Viz Configuration File. 

Information: To use WebRTC output, it has to be explicitly enabled by setting WebrtcOut1.Enable = 1 in section MATROX_CONFIG in your Viz Configuration File.

It is also possible to enable WebRTC output or check if it is enabled using the following commands:

CONFIGURATION*CHANNELS*WEBRTCOUT_0*ENABLE SET 1
CONFIGURATION*CHANNELS*WEBRTCOUT_0*ENABLE GET

Regarding to websocket port configuration, it's possible to change the default port (9092) in Viz Configuration File.

Information: Settings for WebRTC websocket port can be found in section MATROX_CONFIG in your Viz Configuration File: WebrtcOut1.WebsocketPort. 

It is also possible to change it or check the current port using the following commands:

CONFIGURATION*CHANNELS*WEBRTCOUT_0*PORT SET <port_number>
CONFIGURATION*CHANNELS*WEBRTCOUT_0*PORT GET

Note: This configuration must be done up front. It does not take effect during runtime.

WebRTC Versions

The following table lists the WebRTC version Viz Engine is built with:

Viz Engine

WebRTC

5.2

m85

5.1

m85

WebRTC Client Support

The HTTP server is not implemented by viz_webrtc plug-in. The following information is to help every user to build their webrtc client to be able to interact with webrtc server.

Configuration

To configure webrtc client to establish a peer connection with webrtc server built into the viz_webrtc plug-in, it is necessary to create a websocket client that is responsible for handling the peer connection and also notifying the plug-in via a data channel for input handling. 

WebSocket Settings

Regarding to websocket configuration, the default port is 9092 in Viz Configuration File. Make sure that you keep the same websocket port on client side when you update it on Viz Configuration File.

this.connection = new WebSocket("ws:://127.0.0.1:9092/ws")

Peer Connection Settings

To establish a peer connection we need to set the ice server with the same configuration that was defined in viz_webrtc plug-in.

const peerConnectionConfig = {
iceServers: [
{
urls: "stun:stun.l.google.com:19302",
},
],
};

Once the RTCPeerConnection is created it is necessary to create an SDP answer on client. As the SDP offer was already created in the remote peer (server side) the websocket connection can be used to listen for incoming offers. Once that is done, the two peers have set both the local and remote session descriptions so they know the capabilities of the remote peer. This doesn't mean that the connection between the peers is ready. For this to work, we need to collect the ICE candidates at each peer and transfer to the other peer.

To receive the remote tracks that were added by the server, we register a listener on the local RTCPeerConnection listening for the track event. The RTCTrackEvent contains an array of MediaStream objects that have the same MediaStream.id values as the peer's corresponding local streams.

this.peerConnection.ontrack = (e) => {
        console.log("Remote stream added");
        this.playerElement = document.getElementById('remote-video');
        this.playerElement.srcObject = e.streams[0];
      };

More information can be found here https://webrtc.org/getting-started/peer-connections on how to create a webrtc peer connection on client side.

Events Handling Support

To be able to notify the webrtc server when a new input event is triggered, it is necessary to follow the defined message standard so that the server can recognize the content of the received message. 

Supported Actions Label

const Action = {
      Move: 0,
      Rotate: 1,
      Zoom: 2,
      KeyUp: 3,
      KeyDown: 4
  };

Supported Input Events Label

const InputEvent = {
    Keyboard: 0,
    MouseMove: 1,
    MouseLeft: 2,
    MouseRight: 3,
    MouseWheel: 4,
    Touch: 5,
    ButtonClick: 6
  };

Key up

let key = {
            "key": e.keyCode,
            "inputEvent": {
              "event": InputEvent.Keyboard,
              "action": Action.KeyUp,
            },
            "type":"key"
          };
          console.log(key)
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(key));

Key down

let key = {
            "key": e.keyCode,
            "inputEvent": {
              "event": InputEvent.Keyboard,
              "action": Action.KeyDown,
            },
            "type":"key"
          };
          console.log(key)
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(key));

Mouse down

// Mouse left down
 
if(e.which === 1) {
          this.leftClicked = true;
          let coordinates = {
            "coordinates":{
                "x":this.x,
                "y":this.y,
                "delta":this.mousedelta
            },
            "inputEvent": {
              "event": InputEvent.MouseLeft,
              "action": Action.KeyDown,
            },
            "type":"coordinates"
          };
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(coordinates));
        }
 
// Mouse right down
        if(e.which===3){
          this.rightClicked = true;
          let coordinates = {
            "coordinates":{
                "x":this.x,
                "y":this.y,
                "delta":this.mousedelta
            },
            "inputEvent": {
              "event": InputEvent.MouseRight,
              "action": Action.KeyDown,
            },
            "type":"coordinates"
          };
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(coordinates));
        }
 
// Mouse wheel down
        if(e.which===2){
          let coordinates = {
            "coordinates":{
                "x":this.x,
                "y":this.y,
                "delta":this.mousedelta
            },
            "inputEvent": {
              "event": InputEvent.MouseWheel,
              "action": Action.KeyDown,
            },
            "type":"coordinates"
          };
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(coordinates));
        }

Mouse up

// Mouse left up
 
if(e.which === 1) {
          this.leftClicked = true;
          let coordinates = {
            "coordinates":{
                "x":this.x,
                "y":this.y,
                "delta":this.mousedelta
            },
            "inputEvent": {
              "event": InputEvent.MouseLeft,
              "action": Action.KeyUp,
            },
            "type":"coordinates"
          };
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(coordinates));
        }
 
// Mouse right up
        if(e.which===3){
          this.rightClicked = true;
          let coordinates = {
            "coordinates":{
                "x":this.x,
                "y":this.y,
                "delta":this.mousedelta
            },
            "inputEvent": {
              "event": InputEvent.MouseRight,
              "action": Action.KeyUp,
            },
            "type":"coordinates"
          };
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(coordinates));
        }
 
// Mouse wheel up
        if(e.which===2){
          let coordinates = {
            "coordinates":{
                "x":this.x,
                "y":this.y,
                "delta":this.mousedelta
            },
            "inputEvent": {
              "event": InputEvent.MouseWheel,
              "action": Action.KeyUp,
            },
            "type":"coordinates"
          };
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(coordinates));
        }

Mouse Wheel zoom

let coordinates = {
          "coordinates":{
              "x":this.x,
              "y":this.y,
              "delta":this.mousedelta
          },
          "inputEvent": {
            "event": InputEvent.MouseWheel,
            "action": Action.Zoom,
          },
          "type":"coordinates"
        };
        this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(coordinates));

Mouse Move

let coordinates = {
          "coordinates":{
              "x":this.x,
              "y":this.y,
              "delta":this.mousedelta
          },
          "inputEvent": {
            "event": InputEvent.MouseMove,
            "action": Action.Rotate,
          },
          "type":"coordinates"
        };
        this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(coordinates));

Client Resize

const resolution = {
            "resolution":{
                "width":this.innerWidth,
                "height":this.innerHeight
            },
            "type":"resolution"
          };
          this.resizeVideo();
          this.connection && this.connection.readyState == 1 && this.connection.send(JSON.stringify(resolution));

Note: It's important that webrtc server is aware of client size changes. The client resolution affects directly the coordinates that are sent to the server, so you must notify the server each time the resolution changes. Furthermore, the coordinates sent to the server must refer to the video element that is tracking the webrtc stream that comes from the server. For this reason, each time the browser is resized, the video element must also be resized accordingly.