Install the training environment

The headless installer installs RS with Docker and without human interaction. Instead of user prompts, the installer parameters are taken from the properties file.

The installer prompts are defined in the file install-rs-headless.properties:

Start the installation:

./install-rs-headless.sh

The properties file is in the same directory as the script.

Use the --dry-run flag to validate the properties file, print the settings, and exit without changing anything.

Before installing, you must download the install script (install_rosette.sh) and the license file (rosette-license.xml). The remaining files will be downloaded into the same directory as the install script.  

To start the install, execute the install script from the directory in which it resides.

bash install_rosette.sh

The install will guide you through downloading the files and configuring your system.  Each question has a default; you can accept the default or enter a different value.

Both the headless and interactive installers will update Rosette Server as required. If you have an existing installation of Rosette Server, or you didn't choose to update it on install, you can manually run the scripts to update Rosette Server.

ras_server_1 | raise RosetteException(code, message, url)
ras_server_1 | rosette.api.RosetteException: unauthorizedProfile: Access to profile 
'ad-suggestions' not granted:
ras_server_1 | http://ras_proxy:42/rs/rest/v1/entities

There is not a single one size fits all number here. The best value for max heap size depends on a number of factors:

Please note that it’s not recommended setting the max heap to the amount of physical RAM in the system. More heap doesn’t always translate to better performance, especially depending on your garbage collection settings.

Rosette Server’s data files are loaded into virtual memory. Some endpoints, such as /entities, involve a large amount of data. In order for Rosette to operate at its peak performance, we recommend that you reserve enough free memory to allow memory mapping of all our data files so that page misses are minimized at runtime.

To modify the JVM heap for standalone, edit the file server/conf/wrapper.conf and modify the value of wrapper.java.maxmemory.

# Maximum JVM heap in GB
ROSETTE_JVM_MAX_HEAP=32
# Minimum JVM heap in GB
ROSETTE_JVM_MIN_HEAP=32

We also recommend increasing the worker threads to 4, as described in Configuring worker threads for HTTP transport.

The number of models that can be simultaneously trained depends on the size of the models and the memory available.

Once the model is written to disk, it consumes relatively little memory (~2 GB) for runtime requirements. The training and writing phases are much more memory intensive, each consuming approximately three times more memory. Typically, a model actively training will require approximately 10 GB of RAM.

Total memory consumption depends on the number of models being trained simultaneously, as well as the size of the models. The training server is a Java application and all operations use the JVM heap. To allow more simultaneous annotations on more projects, increase the RAM allocated to the JVM in REX Training Server.

To modify the JVM heap:

The application.yml file controls the configuration of the ETS application. The values in this file rarely change and are relative to the container, meaning the values are only used within the container and have no relevance to the machine running the container.

Server

This section contains the basic server setup. context-path is the part of the URL prepended to all ETS URLs for example /ets/info. In the container, ETS is listening on port 9999.

server:
  servlet:
    context-path: /ets
  port: 9999

Logging

This section contains the log setup. The default log level is info and can be changed by setting the ETS_LOGGING_LEVEL value in the .env file. The com.netflix.eureka.cluster is set to ERROR because by default it fills the log with unneeded log messages. The same is true for com.netflix.eureka.registry. If you would like to log everything, the two lines referencing com.netflix.* can be commented out with a #.

logging:
  level:
    ROOT: ${ETS_LOGGING_LEVEL:info}
    com.netflix.eureka.cluster: ERROR
    com.netflix.eureka.registry: WARN

Management

This section controls the management services, including health and metrics. This service can be run on a different port so the management services are not on the same interface as the ETS API. Note: enabling this management port will require changes to the docker-compose.yml file to expose the port.

The health endpoint was customized to disable the display of disk space reporting, ping reporting and refresh information as it cluttered the response. In addition, the health endpoint is configured to always show details of the dependent services (P-ETS and in training mode, RS). To change the behavior and get a simple UP/DOWN response set show-details to never.

The following management endpoints are enabled: info, health, metrics, and prometheus. There are approximately 20 additional management endpoints that can be enabled.

Metrics is enabled to expose runtime metric information about the ETS process, memory consumption, threads and CPU usage.

Prometheus is enabled so that ETS can be used as a data source for monitoring applications such as Graphana.

Endpoint timing information is enabled and available using the /ets/prometheus endpoint

management:
# Management can be on a separate port
#  server:
#    port: 9888
  health:
    diskspace:
      enabled: false
    ping:
      enabled: false
    refresh:
      enabled: false
  endpoint:
    health:
      show-details: always
  endpoints:
    web:
      base-path: /
      exposure:
        include: "prometheus, metrics, health, info"
  metrics:
    web:
      server:
        auto-time-requests: true

Eureka

ETS_HOST is only used when ETS is running remotely to PETS

eureka:
  dashboard:
    path: /eurekadashboard
    enabled: false
  instance:
    appname: JETS
    hostname: ${ETS_HOST:ets-server}
    leaseRenewalIntervalInSeconds: 30
    leaseExpirationDurationInSeconds: 120
    status-page-url: /ets/info
    health-check-url: /ets/health
server:
    enableSelfPreservation: false
  client:
    healthcheck:
      enabled: false
#  As the server we don't want to register with ourselves
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone:
 http://${eureka.instance.hostname}:${server.port}/ets/eureka/

Info

This section determines the ETS operating mode (training or extraction). The ETS_TRUSTSTORE_FILENAME and ETS_KEYSTORE_FILENAME are only defined when running outside a container.

info:
  app:
    name: "events-training-server"
    description: "Rosette Event Extraction and Training Server"
    version: "1.0.0"
    build: "071221145647"
ets:
    operating-mode: "${ETS_MODE:training}"
rs:
  # rsUrl is only required in training configuration
  rsUrl: ${RS_URL:}
  rsConnectTimeoutMS:30000
    ssl:
    enable-outgoing-ssl: ${ENABLE_OUTGOING_SSL:false}
        key-store: ${ETS_KEYSTORE_FILENAME:/certs/keystore.jks}
    key-store-password: ${ETS_KEYSTORE_PW:}
    trust-store: ${ETS_TRUSTSTORE_FILENAME:/certs/truststore.jks}
    trust-store-password: ${ETS_TRUSTSTORE_PW:}
pets:
    minimumVersion: v1.0.0
    connectTimeoutMS: 60000
    readTimeoutMS: 60000
    writeBufferSizeKB: 1000

Springdoc

springdoc:
  show-actuator: true
  # Enable/disable swagger documentation
  api-docs:
    enabled: true
spring:
  banner:
    location: classpath:ets-banner.txt
  resources:
    add-mappings: false
  cloud:
    discovery:
      client:
        composite-indicator:
          enabled: false
        health-indicator:
          enabled: false

This process describes how to capture the logs for the ETS frontend process, (the J-ETS server). The backend worker (P-ETS) processes logs through the docker subsystem.

Configuring the Log Files

Log File Naming

The default log file name is spring.log. If you prefer a different name, you can change the log file name.

Log Rotation

By default, once logs reach 10 MB they are archived. That is, they are compressed with a date stamp and sequence number such as ets-server.log.2022-03-04.0.gz.The file size at which this occurs can be changed by setting the max-size in the file /basis/ets/config/application.yml.

logging: 
  file:  
    name: /logs/ets-server.log 
    max-size: 20 MB

Supported values for the max-size include MB and KB.