Documentation Center

Viz Engine Administrator Guide

Version 5.1 | Published November 09, 2023 ©

Logfiles Configuration

Location of Log Configuration

Viz Engine Log Configuration is stored in .json files. Viz Engine is looking in its ProgramData folder for log configuration files. By default this folder is located at %ProgramData%\vizrt\VizEngine if not changed by command line argument. Viz Engine looks for a file called viz_engine_%N_log_config.json, where %N is replaced by the instance number, starting at 0 for the first instance. If that file is not found it tries viz_engine_log_config.json next. In case that also fails the last resort is the built-in log configuration.

This approach allows to have distinct log configuration for each Viz Engine instance or a common one for all instances, depending on the use case, one or the other might be preferred.

Order of Log Configuration Search:

  • ${ProgramData}\viz_engine_%N_log_config.json

  • ${ProgramData}\viz_engine_log_config.json

  • built-in as fallback

Concept of Log Configuration

The Log Configuration consists of loggers who contain zero, one, or more sinks. A sink uses a formatter. The application knows loggers by its ID. A sink used by a logger determines the destination where a log record is sent to and a formatter is used to process the log record to bring it to the desired format expected by the sink.

Formatters: [ formatter1 , formatter2, ...]

formatter1 : {id, concrete_type}, ...

Loggers: [ logger1 , logger2, ...]

logger1 : {id, sinks: [ sink1 , sink2, ...]}, ...

sink1 : {id, formatter1 , concrete_type}, ...

Warning: Only the formatters and sinks listed below are supported. Using other formatters or sinks result in undefined behavior!

Known Sinks

At the moment there are four sinks available. Two file sinks and two console sinks.

  • daily_file_sink: Creates a log file that is daily rotated.

  • rotating_file_sink: Creates a log file that is rotated when it exceeds a certain size.

  • stdout_color_sink: Creates a colored console sink writing on stdout.

  • stderr_color_sink: Creates a colored console sink writing on stderr.

Parameters Common To All Sinks

Some parameters are common to all concrete sinks and are described below.

"id" : "default_file_sink" ,
"formatter" : "engine_file_formatter" ,
"log_level" : "debug" ,
"deferred" : false ,

  • id: Determines identifier for this sink. This ID is used by the application.

  • formatter: References an id of a formatter.

  • log_level: Applies a log level to this sink. Possible values are:

    • trace

    • debug

    • info

    • warning

    • error

    • critical

    • off

  • deferred: Does not create this sink during configuration. It may later be instantiated by the application.

daily_file_sink

Creates a log file that is rotated daily. The concrete parameters are below.

"daily_file_sink" : {
"base_filename" : "logs/viz_engine_%N_default" ,
"rotation_hour" : 9 ,
"rotation_minute" : 3 ,
"truncate" : false, "max_files" : 10 }

  • base_filename: Determines the path and name of your log file. It may include a placeholder for instance as %N. Path can be absolute or relative. If path is relative, then it is always relative to the current directory. Creates filename in the form basename.YYYY-MM-DD.

  • rotation_hour: Rotates at this hour. Range 0-23. Do not include a leading zero.

  • rotation_minute: Rotates at this minute. Range 0-59. Do not include a leading zero.

  • truncate: Empties the log file at start or reconfiguration of this sink, if set to true.

  • max_files: Determines the maximum number of files to keep after rotating.

rotating_file_sink

Creates a log file that is rotated when it exceeds a certain size. The concrete parameters are below.

"rotating_file_sink" : {
"base_filename" : "logs/viz_engine_%N_shaders" ,
"max_size_MB" : 1024 ,
"max_files" : 8 ,
"rotate_on_open" : false
}

  • base_filename: Determines path and name of your log file. It may include a placeholder for instance as %N. Path can be absolute or relative. If path is relative, then it is always relative to the current directory.

  • max_size_MB: Sets the maximum size in MB after which file gets rotated.

  • max_files: Determines the maximum number files to keep after rotating.

  • rotate_on_open: Rotates the log file at start or reconfiguration of this sink, if set to true.

stdout_color_sink and stderr_color_sink

Creates a colored console sink writing on stdout or stderr. Since they are identical they are gathered together here in the documentation. The concrete parameters are below.

"stdout_color_sink" : {
"color_mode" : "automatic"
}

"stderr_color_sink": { "color_mode": "automatic" }

  • color_mode: Sets the color mode used by sinks with color support. Possible values are:

    • automatic

    • always

    • never

Known Formatters

At the moment there is one formatter known.

  • pattern_formatter: A highly versatile text formatter.

Parameters Common To All Formatters

"id" : "engine_file_formatter" ,

  • id: Sets the identifier for this formatter. This ID is used as a reference by a sink.

pattern_formatter

A highly versatile text formatter.

"pattern_formatter" : {
"pattern_string" : "[%Y-%m-%dT%H:%M:%S.%fZ] [%l][pid=%P][tid=%t]: %a %v" ,
"pattern_time_type" : "utc" ,
"eol" : "default" ,
"enterprise_id" : "27566.3.1.%N" ,
"sd_id_fallback" : "renderer" ,
"escape_nl" : true ,
"attr_flag_formatter" : "a", "attr_prefix" : ""
}

  • pattern_string: Describes how to format the log message. Placeholders in the string expand to.

    • "%n" logger name.

    • "%l" log level.

    • "%L" short log level.

    • "%a" weekday. THIS FLAG HAS BEEN RE_PURPOSED TO

    • "%a" viz::log::attributes

    • "%A" short weekday.

    • "%b" or "%h" month.

    • "%B" short month.

    • "%c" datetime.

    • "%C" year 2digits.

    • "%Y" year 4 digits.

    • "%D" or "%x" datetime MM/DD/YY.

    • "%m" month 1-12.

    • "%d" day of month 1-31.

    • "%H" hours 24.

    • "%I" hours 12.

    • "%M" minutes.

    • "%S" seconds.

    • "%e" milliseconds.

    • "%f" microseconds.

    • "%F" nanoseconds.

    • "%E" seconds since the epoch.

    • "%p" am/pm.

    • "%r" 12 hour clock 02:55:02 pm.

    • "%R" 24-hour HH:MM time.

    • "%T" or "%X" ISO 8601 time format (HH:MM:SS).

    • "%z" timezone.

    • "%t" thread id.

    • "%P" pid. see

    • "%v" message text.

    • "%^" color range start.

    • "%$" color range end

    • "%@" source location (filename:linenumber).

    • "%s" short source filename - without directory name.

    • "%g" full source filename.

    • "%#" source line number.

    • "%!" source funcname.

    • "%%" "%" char.

    • "%u" elapsed time since last log message in nanoseconds.

    • "%i" elapsed time since last log message in microseconds.

    • "%o" elapsed time since last log message in milliseconds.

    • "%O" elapsed time since last log message in seconds.

Some placeholders may not be available and depend on the log component itself, such as source location, funcname, etc. This is not a defect, but intentional.

  • pattern_time_type: Determines how the time is presented. Possible values are:

    • local

    • utc

  • eol: string to append at the end of the log message. Possible values are:

    • default ... the default value for end of line on your OS

    • LF ... line feed

    • CR ... carriage return

    • CRLF ... carriage return followed by line feed

    • "none" ... nothing will be appended

  • enterprise_id: Sets the IANA enterprise ID to use together with the sd_id attribute. It is good practice to leave it to the configured value to be able to identify log messages later on.

  • sd_id_fallback: Sets a fallback for the sd_id attribute if the attribute needs to be emitted but is not explicitly provided in the log message.

  • escape_nl: Escapes CR and LF in the log message, if true.

  • attr_flag_formatter: Determines the character in the pattern_string that selects the attributes formatter.

  • attr_prefix: Sets a string that is printed before each attributes formatter.

Logger

logger: parameter
"logger": {
"id": "engine_sync_console",
"log_level": "info",
"flush_level": "error",
"async_mode": true,
"register": true,
"deferred": false,
"dup_filter_max_skip_duration": 0,
"sinks": [ ... ]
}

  • id: Determines identifier for this logger. This ID is used by the application.

  • log_level: Applies a log level to this logger. Possible values are:

    • trace

    • debug

    • info

    • warning

    • error

    • critical

    • off

  • flush_level: Flushes this logger on all sinks when a log message with this level or higher is logged. Possible values are:

    • trace

    • debug

    • info

    • warning

    • error

    • critical

    • off

  • async_mode: Operates the logger asynchronously.

  • register: Registers the logger in the logger registry if true.

  • deferred: Does not create this logger by default . It is instantiated later by the application.

  • dup_filter_max_skip_duration: Determines the time interval (in seconds) within which duplicate payloads are filtered out when greater than zero.

  • sinks: Lists sinks for this logger. See below in the Built-In Log Config for details.

Known Loggers in the Viz Engine

The Viz Engine uses the following loggers:

  • engine_default

  • engine_sync_console

  • engine_shaders

  • engine_sync_console: Answers to input read from the console is using the synchronous console logger.

  • engine_shaders: Emits shader construction to this logger.

  • engine_default: All other log messages end up here.

Built-In Log Config

Viz Engine built-in log configuration
{
 "version": "1.0",
  "async_params": {
    "queue_size": 32768,
    "overflow_policy": "overrun_oldest",
    "number_of_workers": 1,
    "flush_interval": 10
  },
  "formatters": [
    {
      "id": "engine_file_formatter",
      "concrete": {
        "pattern_formatter": {
          "pattern_string": "[%Y-%m-%d %H:%M:%S.%fZ] [%l][pid=%P][tid=%t]: %a %v",
          "pattern_time_type": "utc",
          "eol": "default",
          "enterprise_id": "27566.3.1.1.%N",
          "sd_id_fallback": "renderer",
          "escape_nl": true,
          "attr_flag_formatter": "a"
        }
      }
    },
    {
      "id": "engine_console_formatter",
      "concrete": {
        "pattern_formatter": {
          "pattern_string": "%^[%Y-%m-%d %H:%M:%S.%e] [%L]:%$ %v%a",
          "pattern_time_type": "local",
          "eol": "default",
          "enterprise_id": "",
          "sd_id_fallback": "",
          "escape_nl": false,
          "attr_flag_formatter": "a",
          "attr_prefix": "\\n\\t"
        }
      }
    },
    {
      "id": "plain_console_formatter",
      "concrete": {
        "pattern_formatter": {
          "pattern_string": "%v",
          "pattern_time_type": "local",
          "eol": "default",
          "enterprise_id": "",
          "sd_id_fallback": "",
          "escape_nl": false
        }
      }
    }
  ],
  "loggers": [
    {
      "logger": {
        "id": "engine_default",
        "log_level": "info",
        "flush_level": "error",
        "async_mode": true,
        "register": true,
        "deferred": false,
"dup_filter_max_skip_duration": 0,
        "sinks": [
          {
            "id": "default_file_sink",
            "formatter": "engine_file_formatter",
            "log_level": "debug",
            "deferred": false,
            "concrete": {
              "rotating_file_sink": {
                "base_filename": "logs/viz_engine_%N_default",
                "max_size_MB": 1024,
                "max_files": 10,
                "rotate_on_open": false
              }
            }
          },
          {
            "id": "console_sink",
            "formatter": "engine_console_formatter",
            "log_level": "debug",
            "deferred": true,
            "concrete": {
              "stdout_color_sink": {
                "color_mode": "automatic"
              }
            }
          }
        ]
      }
    },
    {
      "logger": {
        "id": "engine_sync_console",
        "log_level": "info",
        "flush_level": "error",
        "async_mode": false,
        "register": false,
        "deferred": false,
"dup_filter_max_skip_duration": 0,
      "sinks": [
          {
            "id": "sync_console_sink",
            "formatter": "plain_console_formatter",
            "log_level": "debug",
            "deferred": true,
            "concrete": {
              "stdout_color_sink": {
                "color_mode": "automatic"
              }
            }
          }
        ]
      }
    },
    {
      "logger": {
        "id": "engine_shaders",
        "log_level": "info",
        "flush_level": "info",
        "async_mode": true,
        "register": true,
        "deferred": true,
"dup_filter_max_skip_duration": 0,
      "sinks": [
          {
            "id": "shader_file_sink",
            "formatter": "engine_file_formatter",
            "log_level": "info",
            "deferred": false,
            "concrete": {
              "rotating_file_sink": {
                "base_filename": "logs/viz_engine_%N_shaders",
                "max_size_MB": 1024,
                "max_files": 10,
                "rotate_on_open": false
              }
            }
          }
        ]
      }
    }
  ]
}

The first object contains the parameters for the default asynchronous factory. Followed by three formatters, engine_file_formatter, engine_console_formatter, and plain_console_formatter. The last one just emits the log message as it was received. No other decoration, such as timestamps are added. The former two add timestamps and the engine_console_formatter does colorization where possible.

On the loggers side we have engine_default, which has a log_level of info, is using the asynchronous factory and is created during the initial configuration. It uses two sinks, a rotating_file_sink logging to logs/viz_engine_%N_default, and stdout_color_sink to emit logs messages to the console. Next is engine_shaders, it uses a rotating_file_sink as well and the same formatter as engine_default. It is not created at configuration time but on demand. Next is engine_sync_console, it is created at configuration time, does not register itself and the sink is created deferred. Meaning the logger can be found but the sink is created on demand.

Examples

Log to File only

If one does not want any messages emitted onto the console window the console sink can just be omitted from the engine_default logger. Such as:

{
"logger" : {
"id" : "engine_default" ,
"log_level" : "info" ,
"flush_level" : "error" ,
"async_mode" : true ,
"register" : true ,
"deferred" : false , "dup_filter_max_skip_duration": 0, "sinks" : [
{
"id" : "default_file_sink" ,
"formatter" : "engine_file_formatter" ,
"log_level" : "debug" ,
"deferred" : false ,
"concrete" : {
"rotating_file_sink" : {
"base_filename" : "logs/viz_engine_%N_default" ,
"max_size_MB" : 1024 ,
"max_files" : 10 ,
"rotate_on_open" : false
}
}
},
{
"id" : "console_sink" ,
"formatter" : "engine_console_formatter" ,
"log_level" : "debug" ,
"deferred" : true ,
"concrete" : {
"stdout_color_sink" : {
"color_mode" : "automatic"
}
}
}
]
}
}

Only Warning and Errors on Console

Sometimes it may be beneficial if only warning and error appear in the console. For this just set the log_level in the console sink to warning.

{
"logger" : {
"id" : "engine_default" ,
"log_level" : "info" ,
"flush_level" : "error" ,
"async_mode" : true ,
"register" : true ,
"deferred" : false , "dup_filter_max_skip_duration": 0, "sinks" : [
{
"id" : "default_file_sink" ,
"formatter" : "engine_file_formatter" ,
"log_level" : "debug" ,
"deferred" : false ,
"concrete" : {
"rotating_file_sink" : {
"base_filename" : "logs/viz_engine_%N_default" ,
"max_size_MB" : 1024 ,
"max_files" : 10 ,
"rotate_on_open" : true
}
}
},
{
"id" : "console_sink" ,
"formatter" : "engine_console_formatter" ,
"log_level" : "warning" ,
"deferred" : true ,
"concrete" : {
"stdout_color_sink" : {
"color_mode" : "automatic"
}
}
}
]
}
}