Manage the Training Servers

To run on the console:

cd /basis/rosette/server/bin
./launch.sh console

To stop in the console:

CTRL-C

To run as a daemon:

./launch.sh start

To stop the daemon:

./launch.sh stop

Rosette Server and REX Training Server must be running.

Rosette Adaptation Studio has scripts on each server to monitor the health and status of the system. Run them at startup to verify the system is ready to go.

The scripts are:

where /basis is the default installation directory. If you've installed in a different directory, replace /basis with your install directory.

Each script verifies that the Docker containers have loaded and all components are running.

To check the status of ETS, open a browser and proceed to http:/{host}:{port}/ets/health. The default port is 9999.

The workers indicate that ETS is available and Rosette can communicate with it:

{"status": "UP",
"components": {
    "PETS-Workers": {
       "status": "UP",
       "details": {
           "P-ETS Workers": "1 Worker(s) Available"
       }
     },
    "RosetteServer": {
       "status": "UP",
       "details": {
           "Rosette Server": "Available at http://memento:8181/rest/v1"
       }
     }
   }
}

Use the following links to verify the status of each of the servers:

Model Training Suite is shipped with a sample events model that can be used to verify the installation for events. Use the import_project.sh script to import the project.

The superuser can create users and perform all tasks. At install, the superuser is set to admin.

The first time you log in as admin, you will be forced to change the password. The initial login information is:

The superuser password can be reset by running the reset_admin.sh in /basis/ras/scripts. After reset, you will be prompted to change the password when you log in the first time.

Each component generates logs which may be useful if a problem occurs.

The location of the logs is determined during installation. The files in the default locations are:

The events training server (ETS) leverages Docker log files, allowing you to customize log file location, rotation, log format (e.g. JSON), and if the logs are stored off the device. To display log information, from the ETS install directory:

docker compose logs 

or to follow the logs:

docker compose logs -f 

See the Docker documentation for configuration and usage information.

Log files are also created during installation. Refer to these files for details on how each component was installed, as well as SSL configuration.

/rest/metrics/health/worker

The metrics/health/worker endpoint provides basic healthy information indicating whether the application is up or down.

curl localhost:8181/rest/metrics/health/worker

{"healthy":true,"message":null,"error":null,"details":null,"timestamp":"2020-09-15T16:30:49.235Z"}

{"message":"Rosette at your service","time":1467912784915}

{"name":"Rosette","version":"<current-version>","buildNumber":"","buildTime":""}

The /rts/info/server endpoint

$ curl "http://localhost:9080/rts/info/server"

returns the configuration properties along with the version:

{"assetRoot": "/basis/rts",
"concurrentSerializationThreads": 1,
"concurrentTrainThreads": 2,
"concurrentWordclassTrainingThreads": 2,
"memoryCacheTimeout": 180,
"serializationDelay": 300,
"version": "1.0.1",
"workspaceRoot": "/basis/rts/workspaces"}

DAL Connection Test

The DAL connection test verifies the connection between RTS and the mongo instance on RAS by connecting to RTS directly. If RAS is not reachable, this test could hang for 1-2 minutes waiting for the connection.

curl -v -X POST "http://localhost:9080/rts/rex/test-dal-connector" --header 'Content-Type: application/json' -d '{
        "connectorId": "AS_MONGO",
        "corpusType": "ANNOTATED_TRAIN_CORPUS",
        "healthCheck" : "true",
  "config": {
    "projectId": "5f1470b6412ff29b8e4982f3",
    "sampleIds": "5f158b0e412ff29b8e4983b8,5f158b0e412ff29b8e4983b5,5f158b0e412ff29b8e4983b7"
  }
}'

If it is working, it will return:

{"tests":[{"connectorId":"AS_MONGO","message":"Health check passed","success":true}]}

This message indicates that the port is open and mongo is responding to requests.

The DAL connector is configured using the file /basis/rts/config/mongodal_config.yaml. The file contains 3 lines:

connectionString: 192.168.0.145:27017
maxSamples: 100000
useSsl: false 

The connectionString points to the mongo instance in RAS which is needed by the DAL connector in RTS. If the test is not successful, use cURL or a mongo client to test the connection to the mongo server without going through RTS.

curl 192.168.0.145:27017 

When using cURL, a successful response will be a message from mongo similar to:

"It looks like you are trying to access MongoDB over HTTP on the native driver port." 

This indicates the port is open and mongo is responding to requests.

Ping the server to test that RTS is running and confirm that you can connect to it:

curl localhost:9080/rts/info/server

Verify the settings in the .env file in the rs-docker directory

$ cat .env 

returns:

# Connector information
RTS_URL=http://192.168.1.234:9080
RTS_SOURCE_STRING=statistical

Verify that the hostname/port are correct and reachable from the host. In the Rosette Server container, the following command should return 200.

curl $RTS_URL

Ping the server to test that ETS is running and confirm that you can connect to it:

curl -v http://<host>:<port>/rest/v1/events/info

If Rosette Server can't connect to ETS, check the event-extractor-factory-config.yaml file in the rs/config/rosapi directory. Verify that the value of eventTrainingServerUrl is correct and not commented out.

Rosette Adaptation Studio uses mongo as the backend database to store projects. The mongo client is required to perform a health check of the backend.

For performance, db.enable.FreeMonitoring() can be enabled and will provide an external API with statistics. This is not recommended for use in production.

The Manage Project page in the Studio will display the status of Rosette Server and REX Training Server.

To run the provided scripts you need a certificate and a key generated for each host that is running a component (RTS, RS, RAS, ETS). Each of the certificates must be signed by the root certificate authority (CA) that will be shared by all hosts running components. All certificates, keys, and root certificate authority files must be stored in PEM format.

RTS, ETS, and RS are Java processes, so they require a keystore for the host and truststore for the root CA in a password protected Java Key Store (JKS) format in addition to PEM-format files. There is a script provided, generate-keystores.sh, in the RTS docker directory that will convert the certificate and key into a password protected JKS file. Additionally, this script will generate a password protected truststore JKS file. For ETS, the generate-keystores.sh script is located in the scripts directory.

As long as the root CA is included, any truststore can be used. The truststore created by the provided script does not have to be used.

Rosette Adaptation Studio includes scripts to enable and disable SSL support between the servers. The scripts are found in the following directories:

Enable SSL

Install and test the complete Rosette Adaptation Studio installation before enabling SSL. This will verify that everything is properly installed.

For each component:

sudo systemctl restart docker

Disable SSL

Rosette Server is based on the Java-based OSGI with Jetty web server. To enable incoming SSL connections the server configuration must be updated to include the path to the truststore, keystore, and their respective passwords. Additionally, to enable outgoing SSL connections, e.g. RTS Decoder, custom code etc., the Java virtual machine settings must be updated to include the path to the truststore, keystore, and their respective passwords as well.

The enable-rs-ssl.sh script performs the following actions:

The disable-rs-ssl.sh script reverses the actions above:

A script to create a trust and/or keystore for Rosette Server can be found in both the RTS and ETS directories. The scripts are identical.

Generating a keystore will require a certificate and key in PEM format.

Generating a truststore will require a Root Certificate Authority (Root CA) certificate in PEM format.

REX Training Server is based on the Java-based Open Liberty web container. To enable incoming and outgoing SSL the server configuration must be updated to include the path to the truststore, keystore, and their respective passwords. Additionally the mongodal_config.yaml file (in the config directory) must be updated to set the useSsl flag to true, enabling SSL between RTS and the mongo instance running on Rosette Adaptation Studio.

The enable-rts-ssl.sh script performs the following actions:

The disable-rts-ssl.sh script reverses the actions above:

The optimal system configuration for the production server depends on the size of the input provided for event extraction. Benchmarks for different server configurations are provided to help you select the proper hardware for the production environment.

Overall combined throughput across 20 concurrent users (requests/second)

Rosette Adaptation Studio is based on multiple technologies: python server, nginx reverse proxy, mongoDB server and React. As such, the configuration for RAS will be different from a Java process. All components internal to RAS use nginx for outgoing communication. Mongo is exposed to the host machine and can be protected by SSL. One of the primary differences between the RAS scripts and RS/RTS scripts is that the RAS scripts use PEM files rather than JKS files. Also, since RAS is made of one incoming connection (mongo) and three outgoing connections (RTS,RS, ETS) it is possible to selectively enable SSL for mongodb, RTS and RS.

The enable-ras-ssl.sh script performs the following actions:

The disable-ras-ssl.sh script reverses the actions above:

You can enable and disable the SSL connection between the browser and Rosette Adaptation Studio.

ETS requires a certificate and key in PEM file format, in addition to a root certificate in PEM format.

The enable-ets-ssl.sh script performs the following actions:

The disable-ets-ssl.sh script performs the following actions: