Once installed, start the servers in the following order:
Run the Events Training Server (ETS)
-
Navigate to the installation directory containing the Docker Compose files. The default is:
/basis/ets/ets-docker
-
Run the service in detached mode.
docker compose up -d
-
To check that the service has started, check the logs.
docker compose logs -f
-
To check the status of ETS, open a browser and proceed to http:/{host}:{port}/ets/health
. The default port is 9999.
The workers indicate that ETS is available and Rosette can communicate with it:
{"status": "UP",
"components": {
"PETS-Workers": {
"status": "UP",
"details": {
"P-ETS Workers": "1 Worker(s) Available"
}
},
"RosetteServer": {
"status": "UP",
"details": {
"Rosette Server": "Available at http://memento:8181/rest/v1"
}
}
}
}
Run the REX Training Server (RTS)
-
Navigate to the installation directory containing the Docker Compose files. The default is:
/basis/rts/rts-docker
-
Run the service in detached mode.
docker compose up -d
-
To check that the service has started, check the logs. This will display the last few lines of the logs from the service.
tail /basis/rts/logs/messages.log
-
The service can be tested by using a browser and navigating to http://<host>:<port>
. This will list the endpoints to be used for health checks, API documentation, and metrics.
Example:
http://localhost:9080
To run on the console:
cd /basis/rosette/server/bin
./launch.sh console
To stop in the console:
CTRL-C
To run as a daemon:
./launch.sh start
To stop the daemon:
./launch.sh stop
Note
Check the logs in /basis/rosette/server/logs/
to see any errors from startup or during the server run.
Note
If your installation includes the entity extraction component (rex-root
), you may see failed to open ...
warning messages for data files in languages not installed in your system. These can safely be ignored.
Run Rosette Adaptation Studio (RAS)
Rosette Server and REX Training Server must be running.
-
Navigate to the installation directory (default /basis/ras)
.
-
Execute the script ./scripts/start-standalone.sh
. This script starts the services in detached mode and prints the logs. The message "Rosette Active Learning Studio Server <version> now up" will be displayed when the server is ready. You can then press CTRL-C to disconnect from the logs.
-
Using a browser, navigate to http(s)://<host>:<port>/
to display the RAS landing page.
http://localhost:80
Rosette Adaptation Studio has scripts on each server to monitor the health and status of the system. Run them at startup to verify the system is ready to go.
The scripts are:
-
/basis/rs/scripts/rs-healthcheck.sh
-
/basis/rts/scripts/rts-healthcheck.sh
-
/basis/ras/scripts/ras-healthcheck.sh
where /basis
is the default installation directory. If you've installed in a different directory, replace /basis
with your install directory.
Each script verifies that the Docker containers have loaded and all components are running.
To check the status of ETS, open a browser and proceed to http:/{host}:{port}/ets/health
. The default port is 9999.
The workers indicate that ETS is available and Rosette can communicate with it:
{"status": "UP",
"components": {
"PETS-Workers": {
"status": "UP",
"details": {
"P-ETS Workers": "1 Worker(s) Available"
}
},
"RosetteServer": {
"status": "UP",
"details": {
"Rosette Server": "Available at http://memento:8181/rest/v1"
}
}
}
}
Use the following links to verify the status of each of the servers:
Model Training Suite is shipped with a sample events model that can be used to verify the installation for events. Use the import_project.sh
script to import the project.
Important
The initial superuser password for Rosette Adaptation Studio is set on install and must be changed on first login.
The superuser can create users and perform all tasks. At install, the superuser is set to admin
.
The first time you log in as admin
, you will be forced to change the password. The initial login information is:
-
Name: admin
-
Password: admin
The superuser password can be reset by running the reset_admin.sh
in /basis/ras/scripts
. After reset, you will be prompted to change the password when you log in the first time.
These commands are executed in the directory with the docker compose files, e.g. /basis/rts/rts-docker
.
Table 13. Basic Troubleshooting Commands
Command
|
Purpose
|
docker compose config
|
Applies the variables from the .env file to the docker-compose.yml file so you can see how the containers will be started.
|
docker compose up
|
If you omit the -d , the command will dump the logs to the console. CTRL-C will bring the stack down.
|
docker compose -p <name> up -d
|
The -p flag gives the container a project name. This is useful if you are running multiple instances in a single VM. For example, if you are upgrading versions and you want to run them side by side. docker ps will show the names. Example: 09_rex-training_server_1 , 08_rex_training_server .
|
docker compose logs
|
Attaches to the logs
|
printenv (from inside a container)
|
Allows you to see the ENV variables the container is using.
|
Example 1. docker compose config
docker compose config
services:
rex-training-server:
environment:
AS_MONGO_DAL_CONNECTOR_CONFIG_DIR: /basis/rts/config
image: rex-training-server-tom-0.4.1:latest
ports:
- 9080:9080/tcp
volumes:
- /basis/0.8.final/rts/workspaces:/basis/rts/workspaces:rw
- /basis/0.8.final/rts/assets:/basis/rts:rw
- /basis/0.8.final/rts/config:/basis/rts/config:rw
- /basis/0.8.final/rts/logs:/logs:rw
- /basis/0.8.final/rts/rts-docker/jvm.options:/config/jvm.options:rw
version: '3.0'
Each component generates logs which may be useful if a problem occurs.
The location of the logs is determined during installation. The files in the default locations are:
-
/basis/rs/logs/wrapper.logs
-
/basis/rts/logs/messages.logs
-
/basis/ras/logs/error.log
(Nginx error log)
-
/basis/ras/logs/access.log
(Nginx access log)
-
/basis/ras/logs/server.log
The events training server (ETS) leverages Docker log files, allowing you to customize log file location, rotation, log format (e.g. JSON), and if the logs are stored off the device. To display log information, from the ETS install directory:
docker compose logs
or to follow the logs:
docker compose logs -f
See the Docker documentation for configuration and usage information.
Log files are also created during installation. Refer to these files for details on how each component was installed, as well as SSL configuration.
/rest/metrics/health/worker
The metrics/health/worker
endpoint provides basic healthy information indicating whether the application is up or down.
curl localhost:8181/rest/metrics/health/worker
{"healthy":true,"message":null,"error":null,"details":null,"timestamp":"2020-09-15T16:30:49.235Z"}
Ping the server to test that Rosette is running and you can connect to it.
-
bash:
curl http://localhost:8181/rest/v1/ping
-
Windows Powershell:
Invoke-WebRequest -Uri http://localhost:8181/rest/v1/ping
-
Windows Command Prompt:
start "" http://localhost:8181/rest/v1/ping
This should return:
{"message":"Rosette at your service","time":1467912784915}
This should return:
{"name":"Rosette","version":"<current-version>","buildNumber":"","buildTime":""}
RTS → RAS Troubleshooting/Monitoring
The /rts/info/server
endpoint
$ curl "http://localhost:9080/rts/info/server"
returns the configuration properties along with the version:
{"assetRoot": "/basis/rts",
"concurrentSerializationThreads": 1,
"concurrentTrainThreads": 2,
"concurrentWordclassTrainingThreads": 2,
"memoryCacheTimeout": 180,
"serializationDelay": 300,
"version": "1.0.1",
"workspaceRoot": "/basis/rts/workspaces"}
DAL Connection Test
The DAL connection test verifies the connection between RTS and the mongo instance on RAS by connecting to RTS directly. If RAS is not reachable, this test could hang for 1-2 minutes waiting for the connection.
curl -v -X POST "http://localhost:9080/rts/rex/test-dal-connector" --header 'Content-Type: application/json' -d '{
"connectorId": "AS_MONGO",
"corpusType": "ANNOTATED_TRAIN_CORPUS",
"healthCheck" : "true",
"config": {
"projectId": "5f1470b6412ff29b8e4982f3",
"sampleIds": "5f158b0e412ff29b8e4983b8,5f158b0e412ff29b8e4983b5,5f158b0e412ff29b8e4983b7"
}
}'
If it is working, it will return:
{"tests":[{"connectorId":"AS_MONGO","message":"Health check passed","success":true}]}
This message indicates that the port is open and mongo is responding to requests.
The DAL connector is configured using the file /basis/rts/config/mongodal_config.yaml
. The file contains 3 lines:
connectionString: 192.168.0.145:27017
maxSamples: 100000
useSsl: false
The connectionString
points to the mongo instance in RAS which is needed by the DAL connector in RTS. If the test is not successful, use cURL or a mongo client to test the connection to the mongo server without going through RTS.
curl 192.168.0.145:27017
When using cURL, a successful response will be a message from mongo similar to:
"It looks like you are trying to access MongoDB over HTTP on the native driver port."
This indicates the port is open and mongo is responding to requests.
RS → RTS Troubleshooting/Monitoring
Ping the server to test that RTS is running and confirm that you can connect to it:
curl localhost:9080/rts/info/server
Verify the settings in the .env
file in the rs-docker directory
$ cat .env
returns:
# Connector information
RTS_URL=http://192.168.1.234:9080
RTS_SOURCE_STRING=statistical
Verify that the hostname/port are correct and reachable from the host. In the Rosette Server container, the following command should return 200.
curl $RTS_URL
RS → ETS Troubleshooting Monitoring
Ping the server to test that ETS is running and confirm that you can connect to it:
curl -v http://<host>:<port>/rest/v1/events/info
If Rosette Server can't connect to ETS, check the event-extractor-factory-config.yaml
file in the rs/config/rosapi
directory. Verify that the value of eventTrainingServerUrl
is correct and not commented out.
Rosette Adaptation Studio
Rosette Adaptation Studio uses mongo as the backend database to store projects. The mongo client is required to perform a health check of the backend.
For performance, db.enable.FreeMonitoring()
can be enabled and will provide an external API with statistics. This is not recommended for use in production.
The Manage Project page in the Studio will display the status of Rosette Server and REX Training Server.
Note
SSL for the front end browser, the connection from the web client to the Rosette Adaptation Studio server, can be enabled when RAS is installed. After installation of all servers is complete, you can enable SSL between the servers.
This section describes how to enable and disable SSL support between the servers.
Enabling front end SSL support is independent of enabling SSL between the servers.
To run the provided scripts you need a certificate and a key generated for each host that is running a component (RTS, RS, RAS, ETS). Each of the certificates must be signed by the root certificate authority (CA) that will be shared by all hosts running components. All certificates, keys, and root certificate authority files must be stored in PEM format.
Note
If all components are running on a single host, the same certificate and key PEM files can be shared by RAS, RTS, ETS, and RS.
RTS, ETS, and RS are Java processes, so they require a keystore for the host and truststore for the root CA in a password protected Java Key Store (JKS) format in addition to PEM-format files. There is a script provided, generate-keystores.sh
, in the RTS docker
directory that will convert the certificate and key into a password protected JKS file. Additionally, this script will generate a password protected truststore JKS file. For ETS, the generate-keystores.sh
script is located in the scripts directory.
As long as the root CA is included, any truststore can be used. The truststore created by the provided script does not have to be used.
Enable and Disable SSL Support
Note
These instructions assume you have certificates for each of the servers. You will need the location and passwords for the keystore and the truststore for each server.
Rosette Adaptation Studio includes scripts to enable and disable SSL support between the servers. The scripts are found in the following directories:
-
/basis/rs/rs-docker
-
/basis/rts/rts-docker
-
/basis/ras/scripts
-
/basis/ets/scripts
Enable SSL
Install and test the complete Rosette Adaptation Studio installation before enabling SSL. This will verify that everything is properly installed.
For each component:
Warning
You must shut down the all services before enabling SSL between them.
If you receive an error when restarting the services: "Cannot start service servicename: error while creating mount source path", the services were not shut down before enabling SSL.
To continue, restart Docker:
sudo systemctl restart docker
then restart the services.
-
Once the installation has been verified, shut down the services.
-
Execute the script from the directory containing scripts. For each product, you will be prompted for the location and passwords for the keystore and truststore.
-
/basis/rs/rs-docker/enable-rs-ssl.sh
-
/basis/rts/rts-docker/enable-rts-ssl.sh
-
/basis/ras/scripts/enable-ras-ssl.sh
-
/basis/ets/scripts/enable-ets-ssl.sh
-
Start the services.
Disable SSL
-
Shut down the services.
-
Execute the script from the directory containing the docker-compose files.
-
/basis/rs/rs-docker/disable-rs-ssl.sh
-
/basis/rts/rts-docker/disable-rts-ssl.sh
-
/basis/ras/scripts/disable-ras-ssl.sh
-
/basis/ets/scripts/disable-ets-ssl.sh
-
Start the services.
Rosette Server SSL Scripts
Rosette Server is based on the Java-based OSGI with Jetty web server. To enable incoming SSL connections the server configuration must be updated to include the path to the truststore, keystore, and their respective passwords. Additionally, to enable outgoing SSL connections, e.g. RTS Decoder, custom code etc., the Java virtual machine settings must be updated to include the path to the truststore, keystore, and their respective passwords as well.
The enable-rs-ssl.sh
script performs the following actions:
-
Prompts for the location and passwords of the truststore and keystore files.
-
Copies the truststore and keystore to the configuration directory so that the container has access to them.
Note
if you are re-enabling SSL and using truststores and keystores already present in the configuration directory you will see a warning message about the files being identical and not being copied. This message can safely be ignored.
-
Copies ssl-conf-template.conf
to the configuration directory and renames it to ssl-conf.conf
.
-
ssl-conf.conf
will contain parameters to the JVM settings for RS and will hold the names and passwords of the truststore and keystore.
-
On startup, if the RS container detects the presence of the ssl-conf.conf
file in the config directory and the environment variable ENABLE_SSL is set to 1 (set in the .env
file), then the configuration file will be included in the wrapper.conf
file for RS. The wrapper.conf
file is responsible for setting up the runtime environment for RS.
-
Sets ENABLE_SSL to 1 in .env
.
-
Adds/uncomments the keystore and truststore file names and passwords in the file org.apache.cxf.http.jetty-main.cfg
in the config directory.
-
Sets the URL scheme to https in the file com.basistech.ws.cxf.cfg
in the config directory.
-
Sets the scheme of the RTS_URL to https in the .env
file.
The disable-rs-ssl.sh
script reverses the actions above:
Note
This script does not delete the truststore or keystore from the configuration directory.
-
Sets ENABLE_SSL to 0 in .env
.
-
Comments out the keystore and trustore file names and passwords in the file org.apache.cxf.http.jetty-main.cfg
in the config directory.
-
Sets the URL scheme to http in the file com.basistech.ws.cxf.cfg
in the config directory.
-
Sets the scheme of the RTS_URL to http in the .env
file.
Enabling SSL in Rosette Server
A script to create a trust and/or keystore for Rosette Server can be found in both the RTS and ETS directories. The scripts are identical.
Generating a keystore will require a certificate and key in PEM format.
Generating a truststore will require a Root Certificate Authority (Root CA) certificate in PEM format.
Note
The Root Certificate Authority can also be added to the truststore used system-wide by Java. If this option is used, then the trust store does not need to be explicitly set in the steps below. Typically, the global certificate authority certificate truststore is in <JAVA_INSTALL>/lib/security/cacerts
with the default password of changeit
.
SSL Over Inbound Rosette Server Connections
-
Edit <ROSETTE_SERVER_INSTALL>/server/launcher/config/org.apache.cxf.http.jetty-main.cfg
. Add the following.
For a keystore:
tlsServerParameters.keyManagers.keyPassword=<key store password>
tlsServerParameters.keyManagers.keyStore.file=<path to the keystore file>
tlsServerParameters.keyManagers.keyStore.type=either JKS or PKCS12 depending on the type of store created
tlsServerParameters.keyManagers.keyStore.password=<key store password>
For a truststore:
tlsServerParameters.trustManagers.keyStore.file=<path to the truststore file>
tlsServerParameters.trustManagers.keyStore.type=either JKS or PKCS12 depending on the type of store created
tlsServerParameters.trustManagers.keyStore.password=<trust store password>
tlsServerParameters.clientAuthentication.required=true
-
Edit <ROSETTE_SERVER_INSTALL>/server/launcher/config/com.basistech.ws.cxf.cfg
and change the scheme from http
to https
.
SSL Over Outbound Rosette Server Connections
-
Create a file named ssl-conf.conf
. Edit the file, adding the following contents:
#encoding=UTF-8
#Uncomment the line below to enable SSL debugging
#-Djavax.net.debug=ssl
-Djavax.net.ssl.keyStore=<full path to the Java keystore file (jks|pkcs12)>
-Djavax.net.ssl.keyStorePassword=<KEY_STORE_PASSWORD>
-Djavax.net.ssl.trustStore=<full path to the Java truststore file (jks|pkcs12)>
-Djavax.net.ssl.trustStorePassword=<TRUST_STORE_PASSWORD>
-
Edit <ROSETTE_SERVER_INSTALL>/server/conf/wrapper.conf
. Add the following to the end of the file:
wrapper.java.additional_file=<path to the ssl-conf.conf file>
for example:
wrapper.java.additional_file=/rosette/server/launcher/config/ssl-conf.conf
-
If Rosette Server was previously configured to communicate with RTS, then the RTS_URL
in the <ROSETTE_SERVER_INSTALL>/server/conf/wrapper.conf
must be changed to use https.
-
If Rosette Server was previously configured to communicate with ETS, then the eventTrainingServerUrl
in the <ROSETTE_SERVER_INSTALL>/server/launcher/config/rosapi/event-extractor-factory-config.yaml
must be changed to use https.
REX Training Server SSL Scripts
REX Training Server is based on the Java-based Open Liberty web container. To enable incoming and outgoing SSL the server configuration must be updated to include the path to the truststore, keystore, and their respective passwords. Additionally the mongodal_config.yaml
file (in the config directory) must be updated to set the useSsl
flag to true
, enabling SSL between RTS and the mongo instance running on Rosette Adaptation Studio.
The enable-rts-ssl.sh
script performs the following actions:
-
Prompts for the location and passwords of the truststore and keystore files.
-
Copies the truststore and keystore to the configuration directory so that the container has access to them.
Note
if you are re-enabling SSL and using truststores and keystores already present in the configuration directory you will see a warning message about the files being identical and not being copied. This message can safely be ignored.
-
Copies a server.template.xml
file to the configuration directory and renames it to server.ssl.xml
.
-
Replaces the values for the truststore and keystore file names and passwords in the server.ssl.xml
file.
-
Updates the docker-compose.yml
file and enables mounting the server.ssl.xml
file as server.xml.
This updates the server’s configuration.
-
Updates the .env
file with the name of the SERVER_XML file (server.ssl.xml
).
-
Enables ssl in the mongodal_config.yaml
file.
The disable-rts-ssl.sh
script reverses the actions above:
Note
This script does not delete the truststore or keystore from the configuration directory.
-
Updates the docker-compose.yml
file and comments out the mounting of the SERVER_XML file.
-
Updates the .env
file and comments out the SERVER_XML filename.
-
Deletes the server.ssl.xml
file from the configuration directory.
Event Extraction Server Requirements
The optimal system configuration for the production server depends on the size of the input provided for event extraction. Benchmarks for different server configurations are provided to help you select the proper hardware for the production environment.
Table 14. Server Configurations
System Size
|
CPU Cores
|
CPU Threads
|
Total RAM
|
RAM allocated to Java Heap
|
Small
|
4
|
8
|
32Gb
|
20Gb
|
Medium
|
8
|
26
|
64Gb
|
24Gb
|
Large
|
16
|
32
|
64Gb
|
32Gb
|
Overall combined throughput across 20 concurrent users (requests/second)
Table 15. Throughput Measurements (requests/second)
System Size
|
SMS (50 characters)
|
Tweet (200 characters)
|
Email (1000 characters)
|
Book Chapter (16000 characters)
|
Small
|
49.3
|
26.3
|
8.43
|
0.6
|
Medium
|
107.9
|
58.7
|
18.2
|
1.2
|
Large
|
154.3
|
91.6
|
28.8
|
2.1
|
Rosette Adaptation Studio SSL Scripts
Rosette Adaptation Studio is based on multiple technologies: python server, nginx reverse proxy, mongoDB server and React. As such, the configuration for RAS will be different from a Java process. All components internal to RAS use nginx for outgoing communication. Mongo is exposed to the host machine and can be protected by SSL. One of the primary differences between the RAS scripts and RS/RTS scripts is that the RAS scripts use PEM files rather than JKS files. Also, since RAS is made of one incoming connection (mongo) and three outgoing connections (RTS,RS, ETS) it is possible to selectively enable SSL for mongodb, RTS and RS.
The enable-ras-ssl.sh
script performs the following actions:
-
Prompts for the locations of the certificate, key, and root CA PEM files.
-
Copies the PEM files to WEBSITE_HTTPS_CERT_DIR defined in the .env
file.
-
If incoming mongodb traffic is to use SSL:
-
The certificate and key are concatenated into a single file for use by mongo and stored in the WEBSITE_HTTPS_CERT_DIR. This file is named <certificate name without file extension>-ras-cert-key.pem
.
-
The file proxy-ssl-template.conf
is copied and renamed proxy-mongo-ssl.conf
. This file will contain nginx SSL settings for the certificate and ca certificate to use when internal components communicate with mongodb.
-
The file mongo-ssl-template-docker-compose.override.yml
is copied and renamed docker-compose.override.yml
. This file contains the alternative startup command for mongodb that includes parameters to only accept SSL traffic. Additionally it mounts the proxy-mongo-ssl.conf
file so that the ras-proxy service can gain access to the SSL configuration file.
-
If the outgoing traffic to RS is to use SSL:
-
The file proxy-ssl-template.conf
is copied and renamed proxy-rs-ssl.conf
. This file will contain nginx SSL settings for the certificate and ca certificate to use when internal components communicate with RS.
-
The nginx-template.conf
file is updated to uncomment the inclusion of the proxy-rs-ssl.conf
file.
-
The docker-compose.yml
file is updated to mount the proxy-rs-ssl.conf
file so that nginx can load it.
-
The scheme for ROSETTE_URL is changed to https in .env
-
If the outgoing traffic to RTS is to use SSL:
-
The file proxy-ssl-template.conf
is copied and renamed proxy-rts-ssl.conf
. This file will contain nginx SSL settings for the certificate and ca certificate to use when internal components communicate with RS.
-
The nginx-template.conf
file is updated to uncomment the inclusion of the proxy-rts-ssl.conf
file.
-
The docker-compose.yml
file is updated to mount the proxy-rts-ssl.conf
file so that nginx can load it.
-
Changes the scheme of RTS_URL to https in the .env
file.
The disable-ras-ssl.sh
script reverses the actions above:
Note
This script does not delete the PEM files from the WEBSITE_HTTPS_CERT_DIR directory.
-
If SSL is to be disabled for incoming mongodb traffic:
-
Comments out the inclusion of the proxy-mongo-ssl.conf
from the nginx-template.conf
file.
-
Deletes the proxy-mongo-ssl.conf
file.
-
Backs up then deletes the proxy-mongo-ssl.conf
file.
-
If SSL is to be disabled for outgoing RS traffic:
-
Deletes the proxy-rs-ssl.conf
file.
-
Comments out the inclusion of the proxy-rs-ssl.conf
from the nginx-template.conf
file.
-
Comments out the mount of the proxy-rs-ssl.conf
from the docker-compose.yml
file.
-
If SSl is to be disabled for outgoing RTS traffic:
-
Deletes the proxy-rts-ssl.conf
file.
-
Comments out the inclusion of the proxy-rts-ssl.conf
from the nginx-template.conf
file.
-
Comments out the mount of the proxy-rts-ssl.conf
from the docker-compose.yml
file.
-
Changes the scheme of RTS_URL to http in the .env
file.
You can enable and disable the SSL connection between the browser and Rosette Adaptation Studio.
-
To disable SSL for incoming browser connections to the Studio, use the disable-browser-ras-ssl.sh
script.
-
To enable SSL for incoming browser connections to the Studio, use the enable-browser-ras-ssl.sh
script. You will need a certificate and a key in PEM format.
Event Training Server SSL Scripts
ETS requires a certificate and key in PEM file format, in addition to a root certificate in PEM format.
The enable-ets-ssl.sh
script performs the following actions:
-
Prompts for the location and passwords of the truststore and keystore files and attempts to validate them.
-
Prompts whether the files should be copied to the ./certs
directory (recommended but not required).
-
Updates ./ets-docker/.env
, setting the following:
-
Sets NGINX_CONF_FILE
to nginx-ssl.conf
.
-
Sets NGINX_CERT_PEM_FILE
, NGINX_KEY_PEM_FILE
and NGINX_TRUSTED_PEM_FILE
to the respective PEM files.
-
Sets ETS_KEYSTORE_PW
, ETS_TRUSTSTORE_PW
, ETS_KEYSTORE_FILE
and ETS_TRUSTSTORE_FILE
(if in training mode).
-
Sets ENABLE_OUTGOING_SSL
to true (if in training mode).
The disable-ets-ssl.sh
script performs the following actions: