In high-load environments, particularly when using Pilot Data Server (PDS) and numerous Pilot clients (ActiveX, Pilot Edge, Director, Template Wizard) connected via GH REST, the GH REST service can become a bottleneck, leading to unresponsiveness, timeouts, and session instability. This is often caused by resource exhaustion (thread pools, sockets) due to a very high volume of requests, especially during PDS indexing or if clients create many short-lived connections. Network instability can worsen these issues.

It is recommended to use GH REST 2.9.3 or newer which includes improvements for thread pool management, socket handling, connection limits, robustness against very high load of requests and additional logging.

It is recommended to use PDS 9.3.1 or newer as many optimizations were applied to reduce load on GH REST by reducing the amount of requests and updates.

Consider adjusting the following configuration parameters (restart GH REST service after changes):

GH REST Configuration (config.xml, httpconfig.xml):

• RequestTimeout (in config.xml): May need increasing from default 120000 (ms) if complex requests frequently time out under load (for example, try 180000).

• httpThreadPoolSize (httpconfig.xml): Increase from default 80 based on server cores and load (for example, 100 or higher) to handle more concurrent HTTP requests.

• MaxConnections (httpconfig.xml, new in GH REST 2.9.3): Set a global limit on simultaneous open HTTP connections (for example, 1000) to prevent socket exhaustion. Default is 1000. Connections above this are rejected.

• MaxSessionsUser (config.xml, new in GH REST 2.9.3): Define higher session pool limits for specific users handling many clients (like the user PDS/Pilot connects as). The global MaxSessions (default 10 ) still applies overall, but this allows better distribution. Increase stepwise (for example, <MaxSessionsUser name="pilot_user">30</MaxSessionsUser>).

• useKeepAlive (httpconfig.xml): Default is yes. While disabling (no) can sometimes mitigate issues with sockets stuck in CLOSE_WAIT caused by problematic client connection handling, it increases overhead. GH REST 2.9.3 or newer handle keep-alive more robustly. Prefer keeping it enabled if possible.

  • socketTimeout (in httpconfig.xml): This applies only when useKeepAlive is enabled. It defines how long (in milliseconds) the web server will keep an idle HTTP connection open waiting for subsequent requests from the same client before automatically closing it. The default is 15000ms. Reducing this might help free up sockets faster if many clients connect briefly and then go idle, but increases overhead for clients making frequent requests. Increasing it can reduce connection setup overhead but might hold sockets open longer than necessary.

• NoParallelImageUploads: Default is the number of CPU cores. May need reduction (for example, to six from a previously higher value) if concurrent large uploads contribute to load.

• Logging: Ensure DEBUG and TRACE levels are disabled in production as they add large overhead.

Graphic Hub Configuration (GH Terminal):

• Search instances: Reduce from default (CPU cores) if searches contribute significantly to load during peaks (for example, 8).

Media Sequencer Configuration (default.xml):

• Ensure that the http max_connections setting is set to a value of 40-50 (from the default of 1000) to avoid exhaustion on the back-ends by doing too many requests in parallel.

  <handler name="my_vcp_service" type="vcp_service">

  <service_document>http://vcpserver:8177/service</service_document>

  <http>

  <max_connections>50</max_connections>

  </http>

  </handler>

  • max_connections: Maximum number of simultaneous HTTP connection to the Pilot Data Server. Must be at least 1. Default: 1000.