Viz Plug-ins User Guide
Version 5.1 | Published December 12, 2023 ©
viz_webrtc
The viz_webrtc plug-in allows Viz Engine to use peer-to-peer communication to be able to stream Viz Engine output to multiple clients. It opens a websocket connection between the extension and the client (browser) and then waits for the SDP response from the client to establish the peer connection. When the connection is complete, the video track is sent to the client as soon as a scene is loaded in Viz Engine. This extension also takes incoming input events from the client to be handled by Viz Engine allowing the user to trigger events, perform transformation on objects or trigger script events.
WebRTC in Software I/O Only Mode
Licensing
To operate viz_webrtc in Software I/O mode Parallel Outputs, NDI Out Channel and DVI Out HD or DVI Out Max Resolution (depending on what resolutions we want to support) needs to be licensed.
Configuration
To configure Viz Engine using WebRTC without Matrox support:
-
Uncheck User Defined in Check Video Card.
-
Set the Software I/O Mode to NDI.
WebRTC with Matrox Topology
Installation
The plug-in can be used in combination with a Matrox topology board, which means that DSX.core client must be installed if no Matrox board is installed.
Important: This procedure needs to be followed after installing DSX.core:
-
Unregister mvfDsxCore.dll.
-
Click Start > Run (or use the Windows command line: Search > CMD > (Right click) Run as Administrator).
-
Type REGSVR32 /U "C:\Program Files\Matrox DSX-TopologyUtils\System64\mvfDsxCore.dll" and press ENTER.
-
-
Shut down X.info in the task manager.
-
Delete mvfDsxCore.dll from the folder C:\Program Files\Matrox DSX-TopologyUtils\System64\.
-
Start X.info .
Licensing
To operate viz_webrtc Parallel Outputs needs to be licensed. If DSX.core is used, an additional licenses for either DSX.core HD License or DSX.core UHD License is necessary.
Common Configuration for Both Modes
The Viz Engine output enables WebRTC stream if a valid license is available and configured and the output is enabled in the Viz Configuration File.
Information: To use WebRTC output, it has to be explicitly enabled by setting WebrtcOut1.Enable = 1 in section MATROX_CONFIG in your Viz Configuration File.
It is also possible to enable WebRTC output or check if it is enabled using the following commands:
CONFIGURATION*CHANNELS*WEBRTCOUT_0*ENABLE SET
1
CONFIGURATION*CHANNELS*WEBRTCOUT_0*ENABLE GET
Regarding to websocket port configuration, it's possible to change the default port (9092) in Viz Configuration File.
Information: Settings for WebRTC websocket port can be found in section MATROX_CONFIG in your Viz Configuration File: WebrtcOut1.WebsocketPort.
It is also possible to change it or check the current port using the following commands:
CONFIGURATION*CHANNELS*WEBRTCOUT_0*PORT SET <port_number>
CONFIGURATION*CHANNELS*WEBRTCOUT_0*PORT GET
Note: This configuration must be done up front. It does not take effect during runtime.
WebRTC Versions
The following table lists the WebRTC version Viz Engine is built with:
Viz Engine |
WebRTC |
5.2 |
m85 |
5.1 |
m85 |
WebRTC Client Support
The HTTP server is not implemented by viz_webrtc plug-in. The following information is to help every user to build their webrtc client to be able to interact with webrtc server.
Configuration
To configure webrtc client to establish a peer connection with webrtc server built into the viz_webrtc plug-in, it is necessary to create a websocket client that is responsible for handling the peer connection and also notifying the plug-in via a data channel for input handling.
WebSocket Settings
Regarding to websocket configuration, the default port is 9092 in Viz Configuration File. Make sure that you keep the same websocket port on client side when you update it on Viz Configuration File.
this
.connection =
new
WebSocket(
"ws:://127.0.0.1:9092/ws"
)
Peer Connection Settings
To establish a peer connection we need to set the ice server with the same configuration that was defined in viz_webrtc plug-in.
const
peerConnectionConfig = {
iceServers: [
{
urls:
"stun:stun.l.google.com:19302"
,
},
],
};
Once the RTCPeerConnection is created it is necessary to create an SDP answer on client. As the SDP offer was already created in the remote peer (server side) the websocket connection can be used to listen for incoming offers. Once that is done, the two peers have set both the local and remote session descriptions so they know the capabilities of the remote peer. This doesn't mean that the connection between the peers is ready. For this to work, we need to collect the ICE candidates at each peer and transfer to the other peer.
To receive the remote tracks that were added by the server, we register a listener on the local RTCPeerConnection listening for the track event. The RTCTrackEvent contains an array of MediaStream objects that have the same MediaStream.id values as the peer's corresponding local streams.
this
.peerConnection.ontrack = (e) => {
console.log(
"Remote stream added"
);
this
.playerElement = document.getElementById(
'remote-video'
);
this
.playerElement.srcObject = e.streams[
0
];
};
More information can be found here https://webrtc.org/getting-started/peer-connections on how to create a webrtc peer connection on client side.
Events Handling Support
To be able to notify the webrtc server when a new input event is triggered, it is necessary to follow the defined message standard so that the server can recognize the content of the received message.
Supported Actions Label
const
Action = {
Move:
0
,
Rotate:
1
,
Zoom:
2
,
KeyUp:
3
,
KeyDown:
4
};
Supported Input Events Label
const
InputEvent = {
Keyboard:
0
,
MouseMove:
1
,
MouseLeft:
2
,
MouseRight:
3
,
MouseWheel:
4
,
Touch:
5
,
ButtonClick:
6
};
Key up
let key = {
"key"
: e.keyCode,
"inputEvent"
: {
"event"
: InputEvent.Keyboard,
"action"
: Action.KeyUp,
},
"type"
:
"key"
};
console.log(key)
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(key));
Key down
let key = {
"key"
: e.keyCode,
"inputEvent"
: {
"event"
: InputEvent.Keyboard,
"action"
: Action.KeyDown,
},
"type"
:
"key"
};
console.log(key)
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(key));
Mouse down
// Mouse left down
if
(e.which ===
1
) {
this
.leftClicked =
true
;
let coordinates = {
"coordinates"
:{
"x"
:
this
.x,
"y"
:
this
.y,
"delta"
:
this
.mousedelta
},
"inputEvent"
: {
"event"
: InputEvent.MouseLeft,
"action"
: Action.KeyDown,
},
"type"
:
"coordinates"
};
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(coordinates));
}
// Mouse right down
if
(e.which===
3
){
this
.rightClicked =
true
;
let coordinates = {
"coordinates"
:{
"x"
:
this
.x,
"y"
:
this
.y,
"delta"
:
this
.mousedelta
},
"inputEvent"
: {
"event"
: InputEvent.MouseRight,
"action"
: Action.KeyDown,
},
"type"
:
"coordinates"
};
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(coordinates));
}
// Mouse wheel down
if
(e.which===
2
){
let coordinates = {
"coordinates"
:{
"x"
:
this
.x,
"y"
:
this
.y,
"delta"
:
this
.mousedelta
},
"inputEvent"
: {
"event"
: InputEvent.MouseWheel,
"action"
: Action.KeyDown,
},
"type"
:
"coordinates"
};
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(coordinates));
}
Mouse up
// Mouse left up
if
(e.which ===
1
) {
this
.leftClicked =
true
;
let coordinates = {
"coordinates"
:{
"x"
:
this
.x,
"y"
:
this
.y,
"delta"
:
this
.mousedelta
},
"inputEvent"
: {
"event"
: InputEvent.MouseLeft,
"action"
: Action.KeyUp,
},
"type"
:
"coordinates"
};
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(coordinates));
}
// Mouse right up
if
(e.which===
3
){
this
.rightClicked =
true
;
let coordinates = {
"coordinates"
:{
"x"
:
this
.x,
"y"
:
this
.y,
"delta"
:
this
.mousedelta
},
"inputEvent"
: {
"event"
: InputEvent.MouseRight,
"action"
: Action.KeyUp,
},
"type"
:
"coordinates"
};
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(coordinates));
}
// Mouse wheel up
if
(e.which===
2
){
let coordinates = {
"coordinates"
:{
"x"
:
this
.x,
"y"
:
this
.y,
"delta"
:
this
.mousedelta
},
"inputEvent"
: {
"event"
: InputEvent.MouseWheel,
"action"
: Action.KeyUp,
},
"type"
:
"coordinates"
};
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(coordinates));
}
Mouse Wheel zoom
let coordinates = {
"coordinates"
:{
"x"
:
this
.x,
"y"
:
this
.y,
"delta"
:
this
.mousedelta
},
"inputEvent"
: {
"event"
: InputEvent.MouseWheel,
"action"
: Action.Zoom,
},
"type"
:
"coordinates"
};
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(coordinates));
Mouse Move
let coordinates = {
"coordinates"
:{
"x"
:
this
.x,
"y"
:
this
.y,
"delta"
:
this
.mousedelta
},
"inputEvent"
: {
"event"
: InputEvent.MouseMove,
"action"
: Action.Rotate,
},
"type"
:
"coordinates"
};
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(coordinates));
Client Resize
const
resolution = {
"resolution"
:{
"width"
:
this
.innerWidth,
"height"
:
this
.innerHeight
},
"type"
:
"resolution"
};
this
.resizeVideo();
this
.connection &&
this
.connection.readyState ==
1
&&
this
.connection.send(JSON.stringify(resolution));
Note: It's important that webrtc server is aware of client size changes. The client resolution affects directly the coordinates that are sent to the server, so you must notify the server each time the resolution changes. Furthermore, the coordinates sent to the server must refer to the video element that is tracking the webrtc stream that comes from the server. For this reason, each time the browser is resized, the video element must also be resized accordingly.