To main content

Running Glowroot Central on OpenLiberty

Published by Benjamin Marwell on

Glowroot is a lightweighted but feature-rich java VM agent, which will allow you to collect and evaluate metrics from almost any VM. For larger setups, you might want to install a central collection instance. This guide covers how to set up Glowroot Central on OpenLiberty, a lightweigt and extremely fast Java Application Server (aka Servlet Container).

Downloading and installing OpenLiberty

An overview of the Liberty family

The first step is to download and install WebSphere Liberty Profile or OpenLiberty. The distributions "OpenLiberty" and "IBM WebSphere Liberty Profile" differ in what they provide: Liberty Profile is available with enterprise support and advances featuers. For this showcase and most applications, OpenLiberty is more than suficient.

Both distributions are available in multiple flavours / editions:

  • kernel
    Just contains the internal OSGi Framework. Best if you want to create your own, tightly feature-restricted container.
  • microProfile
    Contains all the microProfile features plus some common web features, like jaxrs, CDI, concurrency, monitoring, JWT (Json Web Tokens), etc.
    This distribution is enough to run Glowroot Central on Openliberty.
    It is just 51 MiB for v20.0.0.8!
  • webProfile
    The most common distribution, if you came from the Java EE (Enterprise Edition) standards. Contains only some of the more advanced features, light ejbLight, JPA, webSockets, and others.
    Still just 86 MiB!
  • Jakarta EE 8
    As the name suggests, contains all features of the Enterprise Edition which are needed to get a JavaEE certification. Contains the EJB features.

Downloading and installing OpenLiberty MicroProfile Edition

Now, this step couldn't be easier, especially for a classic installation. Just head over to the runtime release page, download the latest MicroProfile archive and extract it to a directoy of your choice. Note that I like to rename the resulting "wlp" directory to something like "openliberty-20.0.0.8mp" to keep track of multiple installations.

Configuring and installing glowroot-central on Open Liberty

The actual app installation of glowroot-central is pretty easy, as well. It is done in four steps:

  1. Creating a liberty service instance
  2. Copying the glowroot-central.war artifact to the apps folder
  3. Configuring the OpenLiberty server instance
  4. Configuring Glowroot

Creating a liberty server instance

The first step will be to create a self-contained instance for glowroot. Unlike you might be used from tomcat, you start a JVM (process) for every (web) application from their own directory with their own directory without duplicating the Liberty installation. Go into the liberty installation directory and type:

user@aeon:~/.local/apps/openliberty-20.0.0.8mp$ ./bin/server create glowroot-central

Server glowroot-central created.

That's it! The basic configuration was created in the subdirectory ./usr/servers/glowroot-central. This directory is called a server directory, in this case this is the "glowroot-central" server.

Pro tip: If you want to use the same server instance with multiple installations of OpenLiberty, set the environment variable WLP_USER_DIR to something like $HOME/.local/share/wlp-usr prior to running the create command. This way you do not need to copy your server over to the new installation and maintain multiple instances.

Copying the artefact to its place

Go to the glowroot release page and download the latest glowroot-central-*-dist.war. If the latest version is still 0.13.6, you should rather use 0.13.5 because of an infinispan upgrade bug which is not yet released.

Then copy the artefact into the apps/ folder of your liberty server directory.

Configuring the OpenLiberty server instance

Now open the file server.xml of your glowroot-central server. In the features section, you will see the "microProfile-3.3" feature activated. This is a "meta-feature", because it pulls in other features (such as mpConfig, mpHealth, etc.). We can reduce the number of features by specifying them one by one like so:

<server>
    <!-- Enable features -->
    <featureManager>
        <!-- glowroot needs at least servlet-3.1, but only 4.0 is included in recent mp and web editions. -->
        <feature>servlet-4.0</feature>
    </featureManager>
    <!-- other tags omitted for brevity -->
<server>

You will also need tell Liberty where to find the app. We could have placed the war file into the dropins/ folder instead for automatic discovery, but by specifying the war application manually we have more control (like specifying a custom context root i.e. root path).

<server>
   <!-- … features … -->
  <application id="glowroot"
               location="glowroot-central-0.13.5-dist.war"
               name="glowroot"
               type="war"
               context-root="/"
  />
</server>

While we could configure many more things (more on that later), this is sufficient for now.

Pro tip: If you want to specify a specific JDK/JRE which OpenLiberty should use for running the server, open the file server.env and set the JAVA_HOME variable. This will be specific for this very server and will not affect anything else on your PC.

Configuring Glowroot Central

Glowroot will pick up it's configuration from a system variable called glowroot.central.dir. Let’s create this directory and point glowroot to it.

user@aeon:~/.local/share/wlp-usr/servers/glowroot-central$ mkdir glowroot-central-properties
user@aeon:~/.local/share/wlp-usr/servers/glowroot-central$ touch glowroot-central-properties/glowroot-central.properties
user@aeon:~/.local/share/wlp-usr/servers/glowroot-central$ echo "-Dglowroot.central.dir=$PWD/glowroot-central-properties" >> jvm.options

As you can see, you can configure Liberty’s jvm options (including system properties) using a newline-delimited file called jvm.options. For the time being, we will not bother about any other JVM options and create the glowroot config.

# default is cassandra.contactPoints=127.0.0.1
cassandra.contactPoints=127.0.0.1

# default is to connect without credentials
cassandra.username=

# default is to connect without credentials
cassandra.password=

# default is cassandra.keyspace=glowroot
cassandra.keyspace=

# default is cassandra.consistencyLevel=QUORUM
cassandra.consistencyLevel=

# default is grpc.bindAddress=0.0.0.0
grpc.bindAddress=0.0.0.0

# default is grpc.httpPort=8181
# set this to "none" in order to not serve gRPC over HTTP
grpc.httpPort=8181

# default is grpc.httpsPort=none
# set this to a port number in order to serve gRPC over HTTPS on that port
# the certificate and private key to be used must be placed in the same directory as this properties
# file, with filenames "grpc-cert.pem" and "grpc-key.pem", where grpc-cert.pem is a PEM encoded
# X.509 certificate chain, and grpc-key.pem is a PEM encoded PKCS#8 format private key without a
# passphrase (for example, a self signed certificate can be generated at the command line meeting
# the above requirements using OpenSSL 1.0.0 or later:
# "openssl req -new -x509 -nodes -days 365 -out grpc-cert.pem -keyout grpc-key.pem")
grpc.httpsPort=none

jgroups.configurationFile=jgroups-tcp.xml
jgroups.localAddress=aeon.fritz.box
jgroups.localPort=7800
jgroups.initialNodes=localhost:7800
jgroups.password=my_glowroot_top_secret_password

The above config will go to the glowroot-central.properties file we touched earlier. We are assuming a cassandra installation on the same machine, and we have two more ports, which will give us three ports in total:

  1. TCP Port 9080 where liberty will serve Glowroot’s web interface
  2. TCP Port 7800 for jgroup (glowroot node discovery)
  3. TCP Port 8181 where the glowroot agents will connect to.

Running Glowroot Central on OpenLiberty

Now that you have everything configured (leaving out cassandra for yourself), you can fire up OpenLiberty with the glowroot-central server. Therefore, head back to the OpenLiberty directory and run

user@aeon:~/.local/apps/openliberty-20.0.0.8web$ ./bin/server run glowroot-central

Launching glowroot-central (Open Liberty 20.0.0.8/wlp-1.0.43.cl200820200721-1900) on Eclipse OpenJ9 VM, version 11.0.6+10 (en_GB)
[AUDIT   ] CWWKE0001I: The server glowroot-central has been launched.
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/
[AUDIT   ] CWWKZ0001I: Application glowroot started in 4.355 seconds.
[AUDIT   ] CWWKF0012I: The server installed the following features: [servlet-4.0].
[AUDIT   ] CWWKF0011I: The glowroot-central server is ready to run a smarter planet. The glowroot-central server started in 6.083 seconds.
2020-08-18 09:47:49.390 WARN  org.glowroot.common.util.Version - could not find Build-Time attribute in META-INF/MANIFEST.MF file
2020-08-18 09:47:49.392 INFO  org.glowroot - Glowroot version: 0.13.5
2020-08-18 09:47:49.394 INFO  org.glowroot - Java version: 11.0.6
2020-08-18 09:47:49.395 INFO  org.glowroot - Glowroot home: $USER/.local/share/wlp-usr/servers/glowroot-central/glowroot-central-properties
2020-08-18 09:47:50.531 INFO  o.i.f.GlobalComponentRegistry - ISPN000128: Infinispan version: Infinispan 'Infinity Minus ONE +2' 9.4.5.Final
2020-08-18 09:47:51.009 INFO  o.i.r.t.jgroups.JGroupsTransport - ISPN000078: Starting JGroups channel ISPN
2020-08-18 09:47:53.202 INFO  org.infinispan.CLUSTER - ISPN000094: Received new cluster view for channel ISPN: [aeon-58615|0] (1) [aeon-58615]
2020-08-18 09:47:53.298 INFO  o.i.r.t.jgroups.JGroupsTransport - ISPN000079: Channel ISPN local address is aeon-58615, physical addresses are [127.0.0.1:7800]
2020-08-18 09:47:54.199 INFO  c.datastax.driver.core.ClockFactory - Using native clock to generate timestamps.
2020-08-18 09:47:54.214 INFO  com.datastax.driver.core - DataStax Java driver 3.6.0 for Apache Cassandra
2020-08-18 09:47:54.218 INFO  c.d.driver.core.GuavaCompatibility - Detected Guava >= 19 in the classpath, using modern compatibility layer
2020-08-18 09:47:54.366 INFO  com.datastax.driver.core.NettyUtil - Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
2020-08-18 09:47:54.642 WARN  org.glowroot.central.CentralModule - waiting for Cassandra (127.0.0.1)...

Well, that’s it! The run command will actually start the server as a foreground process, suitable for a quick test or running in containers. You can also use the start command instead for forking the serer into the background. You will find the logs in the server's logs/ directory.

Browse to http://localhost:9080/ to see Glowroot’s web interface once cassandra is set up.

Pro tip: Boot up a quick cassandra instance from docker by running this command beforehand:

docker run  \
    -p 127.0.0.1:9042:9042 \
    -p 127.0.0.1:9160:9160 \
    --name cassandra \
    cassandra:3

Additional configuration options for OpenLiberty

Disabling dropin scanning and early servlet loading

To disable update triggers when changing the configuration or active file system scanning, you can add an applicationMonitor tag with the specific attributes.

You might also want to add two webContainer options which will allow liberty to load and initialize the servlets early and skip some (usually not needed) processing of resources in the META-INF folder of your web applications:

<server>
  <!-- previous options omitted for brevity -->

  <!-- do not defer servlet loading and skip resource processing. -->
  <webContainer deferServletLoad="false" skipMetaInfResourcesProcessing="true" />
  <!-- do not scan the dropin folder and do not reload configuration. -->
  <applicationMonitor updateTrigger="disabled" dropinsEnabled="false"/>
</server>

Activating more logging

Furthermore, you can activate some additional logging options. For example, http access logging (similar to Apache httpd’s access_log) and change the messages.log-timestamps to the ISO format (yyyy-mm-dd).

<server>
  <!-- previous options omitted for brevity -->

  <!-- use the ISO date/time format and do not rotate logs early. -->
  <logging isoDateFormat="true" maxFileSize="0" />

  <httpAccessLogging enabled="true" />
</server>

Set JVM options (like max heap) for OpenLiberty

Another popular configuration is controlling how much heap the web application can consume by modifying the jvm.options file. For instance, you might want to add the following entries to your configuration file.

# set min and max heap
-Xms512m
-Xmx2048m

# set the region, language and timezone of this server
-Duser.language=en
-Duser.region=GB
-Duser.timezone=Europe/London

# force servlet parameter and body encoding to UTF-8
-Dfile.encoding=UTF-8
-Dclient.encoding.override=UTF-8

Use a different java runtime for Glowroot Central Collector

OpenLiberty was developed with the Eclipse OpenJ9 JVM in mind. The OpenJ9 JVM is another OpenJDK variant which was derived from the former proprietary IBM J9 VM and is an alternative to hotspot. It also includes a different heap layout and different garbage collectors, as well as more JVM debug options.

Download Java 11 OpenJ9 from adoptopenjdk.net or using jabba install adopt-openj9@1.11.0-7 and set the path via your server.env file:

jabba install adopt-openj9@1.11.0-7
echo "JAVA_HOME=$HOME/.jabba/jdk/adopt-openj9@1.11.0-6" >> server.env

Conclusion

In conclusion, you now know how to set up Glowroot Central Collector on OpenLiberty and have learned about the most important configuration options for both the actual application and OpenLiberty.

Do you have more questions? Put them into the comments!

Web links