Monitor Your Applications With JConsole - Part 2

[Original Page]

  Using JConsole to monitor and manage applications running on a remote machine involves some extra configuration, especially if security is an issue or if the application is running behind a firewall. In part 2 we'll look at how to enable secure access to remote applications. In part 3we'll deal with firewall issues.

In Part 1 we showed how to use JConsole to monitor and manage applications running on the local machine. As an examplewe used JConsole to monitor the memory usage of an application running in JBoss. Because JConsole itself uses quite a bit of resources it'snot recommended to use JConsole locally in a production environment, so in part 2 we will use JConsole to monitor JBoss on a remote machinethat's not behind a firewall. Finally in Part 3 we'll do the same thing for a remote machine that is behind a firewall.

When dealing with remote access to an application there are two things that we have to ask ourselves.

• Should access to the remote application be secured or should everybody on the local network be allowed access?
• Is the remote machine behind a firewall?

Normally for production environments access should be secured and the remote machine will be behind a firewall, while for example access to a shared development machine may have a less strict security policy. The key point is that whatever the environment you're planning on using JConsole for, make sure that you're doing it in a way that's in line with the security policies of your company.

Connecting to remote applications

Let's start with the simplest scenario of a remote machine that's not behind a firewall and can be accessed by everyone on the local network.First of all we need to configure the port that JConsole is going to use for connecting to JBoss. We do this by adding the following JVM optionin the JBoss startup script:

 -Dcom.sun.management.jmxremote.port=11099
	

This enables us to connect to the JVM running JBoss on port 11099 using the Remote Process connection as shown below. We can choosewhatever port we like, just as long as it's not used by another process.  

  We also need to disable authentication and SSL, since both are enabled by default if we enable remote access. This is done by adding the followingJVM options:

 -Dcom.sun.management.jmxremote.authenticate=false
 -Dcom.sun.management.jmxremote.ssl=false
	

If we forget these system properties, two things can happen depending on the version of the JVM we are using. For more recent versions of JDK 5and JDK 6 starting JBoss will fail telling us that the password file could not be found. Earlier versions of JDK 5 (e.g. 1.5.0_04) will startJBoss without any message indicating a problem. In this situation the problem shows itself when we try to connect from JConsole with the following error message.  

  Because we can expect the same message for many other problems during connection, let's first add some detailed logging to JConsole to make iteasier to find the cause of the problem. To do this we need a simple logging.properties file that looks something like this:

 handlers = java.util.logging.ConsoleHandler
 .level = INFO

 java.util.logging.ConsoleHandler.level = FINEST
 java.util.logging.ConsoleHandler.formatter = \
 		java.util.logging.SimpleFormatter

 // Use FINER or FINEST for javax.management.remote.level - FINEST is
 // very verbose...
 javax.management.level = FINEST
 javax.management.remote.level = FINER
	

Now we can start JConsole with the following command to enable detailed logging of the javax.management classes:

jconsole -J-Djava.util.logging.config.file=logging.properties

Connecting to the remote JVM will now result in the opening of a separate output window containing the detailed logging. If you're using JConsole with JDK 5 the logging will go to standard output. Now if there's a problem connecting to JBoss we can use the logging to diagnosethe problem. For example the logging shown below clearly indicates a problem in the SSL configuration.  

  Note that if JConsole reports that the connection has been refused that no extra logging will be generated since this means no connection has been made to the MBeanServer yet. Most common reasons for refused connections are: wrong port number, JBoss has not been startedor the firewall is blocking the connection. You may want to configure extra detailed logging in your logging.properties file to solvethis problem. To rule out network related issues, you could also first test the JBoss configuration locally by using the Remote Processconnection to connect to a local process using localhost:11099 as the connection string.

Enabling password authentication

Now that we've got the most basic scenario running and enabled detailed logging, let's try enabling username/password authentication. First thing we need is a password file. A template for this file called jmxremote.password.template can be found in the JRE_HOME/lib/management directory. We can either copy this file to jmxremote.password in the same directory and use the usernamesand passwords for all Java applications running on the same machine or copy it to any other name and directory and use it for JBoss only. We will use the second option, so we'll copy the template file to the JBOSS_HOME/bin directory as jmxjboss.password.

Open the password file and scroll to the end of the file and uncomment the monitorRole and controlRole entries.Of course it's also wise to choose better passwords, but for this example we'll just use the passwords from the template.

 # Following are two commented-out entries.  The "monitorRole" role has
 # password "QED".  The "controlRole" role has password "R&D".
 #
 monitorRole  QED
 controlRole   R&D
	

Because the passwords are stored in cleartext the password file must only be readable by the owner as is documented in the password fileitself. This also means that the owner of the password file must be the same user that's used to start JBoss. Incorrect file permissions willresult in one of the following error messages during startup:

Error: Password file read access must be restricted: jmxjboss.password
Error: Password file not readable: jmxjboss.password
	

Since we're not using the default location and name for the password file, we need to add the following JVM option to the JBoss startup script:

 -Dcom.sun.management.jmxremote.password.file=JBOSS_HOME/bin/jmxjboss.password
	

Now we need to enter a username and password from the password file when we try to connect to a remote process in JConsole. Note that it'sstill possible to connect locally without a username and password; authentication is only for remote access. If we use the monitorRolewe only have read access in JConsole. To be able to perform operations on the MBeans we need to use the controlRole. The access rightsfor these roles are defined in the file JRE_HOME/lib/management/jmxremote.access.

If we want to use other roles besides monitorRole and controlRole we need to add these roles in the access file first. Just aswith the password file we can do this either in the default access file or create our own access file and use the following JVM option inthe JBoss startup script:

 -Dcom.sun.management.jmxremote.access.file=JBOSS_HOME/bin/jmxjboss.access
	

For example if we want to add a developRole, our access file would look something like this:

 # Custom access control entries:
 # o The "monitorRole" role has readonly access.  
 # o The "controlRole" role has readwrite access.
 # o The "developRole" role has readwrite access.

 monitorRole   readonly
 controlRole   readwrite
 developRole   readwrite	
	

Enabling SSL

The next step in securing access to JBoss with JConsole is adding SSL. This may not be necessary for a shared development machine whereour main concern probably is to keep developers outside our team from managing our JBoss server via JConsole. For production environmenthowever using SSL may be required by the security policies.

Enabling SSL allows us to do three things:

1. Encrypt the communication between JConsole and the MBeanServer.
2. Use client certificates for authentication.
3. Enable authentication for the RMI Registry.

The first thing we need to do is generate a certificate that will be used by the JBoss server to encrypt the communication. For this examplewe're going to use a self-signed certificate. To create a certificate we use the keytool application that's part of the JDK.

keytool -genkey -alias jconsole -keystore JBOSS_HOME/bin/.jbossKeyStore

The -genkey or -genkeypair (JDK 6) command will generate a public and private key and create a certificate signed by the private key, hence the term self-signed certificate. The -alias option enables us to choose the alias used to store the certificate, while the -keystore options let's us create a keystore in the JBoss bin directory instead of using the default keystore location. We need to enter some extra information like a password and our name after which the certificate will be created.

To list the created certificate in our keystore we can now use the -list command.

keytool -list -keystore JBOSS_HOME/bin/.jbossKeyStore

The output should look something like this (JDK 5 uses the term keyEntry instead of PrivateKeyEntry):

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

jconsole, 21-apr-2007, PrivateKeyEntry,
Certificate fingerprint (MD5): AF:3E:79:18:E7:4E:D1:42:A5:E2:7C:7F:1F:3D:5A:6D
	

Because we're using a self-signed certificate we need to export the certificate and import the certificate in the truststore used by JConsole,otherwise JConsole will not be able to validate the certificate. We use the -export command to export the certificate on the remotemachine.

keytool -export -alias jconsole -keystore JBOSS_HOME/bin/.jbossKeyStore -file jconsole.cert

This will create the file jconsole.cert which we will copy to the machine used to run JConsole. On this machine we import the certificateinto a new truststore for JConsole using the -import command.

keytool -import -alias jboss -keystore JAVA_HOME/bin/.jconsoleKeyStore -file jconsole.cert

If we now use the -list command for this new truststore the output should look something like this:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

jboss, 21-apr-2007, trustedCertEntry,
Certificate fingerprint (MD5): AF:3E:79:18:E7:4E:D1:42:A5:E2:7C:7F:1F:3D:5A:6D	
	

As we can see we can use a different alias for our truststore and the entry is now listed as a trustedCertEntry indicating thatit does not contain the private key that is stored in the keystore on the remote machine.

Note that exporting and importing of the certificate is only necessary for self-signed certificates. For certificates signed by a CA the neededcertificates most likely are already in the default truststore located in the file cacerts in the JRE_HOME/lib/security directory.For information on how to request and import CA certificates see the Resources.

With the keystore and truststore created we can now enable SSL and configure the keystore options in the JBoss startup script using the following JVM options:

 -Dcom.sun.management.jmxremote.ssl=true
 -Djavax.net.ssl.keyStore=JBOSS_HOME/bin/.jbossKeyStore
 -Djavax.net.ssl.keyStorePassword=secret
	

Because we now added the password for the keystore to the startup script it's a good idea to restrict access to the file in the same way aswe've done for the password file.

When starting JConsole we now also need to add the location of the truststore containing the certificate used by JBoss.

jconsole -J-Djavax.net.ssl.trustStore=JAVA_HOME/bin/.jconsoleKeyStore -J-Djava.util.logging.config.file=logging.properties

When we now connect to JBoss with JConsole we should see some logging indicating that we're now using a SSL SocketFactory to make the connection.

Enabling SSL client authentication

Now that we've got SSL enabled, using SSL client authentication is very simple. All we need to do is generate another certificate for theJConsole client using the following command.

keytool -genkey -alias jconsole_client -keystore JAVA_HOME/bin/.jconsoleKeyStore

This will generate a public and private key and a self-signed certificate in the same keystore that we've created earlier as our truststore.Of course we could also create a separate keystore, but it's no problem to use one keystore for both our own keys and certificates as well asfor trusted certificates that we've imported. Listing the keystore should now look something like this:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

jboss, 21-apr-2007, trustedCertEntry,
Certificate fingerprint (MD5): AF:3E:79:18:E7:4E:D1:42:A5:E2:7C:7F:1F:3D:5A:6D
jconsole_client, 21-apr-2007, PrivateKeyEntry,
Certificate fingerprint (MD5): 15:3F:7D:AD:A5:65:8C:E8:CB:D2:4D:39:4E:68:13:01
	

Next we export the certificate and copy it to the remote machine running JBoss (remember that for CA certificates we can skip the export and import steps).

keytool -export -alias jconsole_client -keystore JAVA_HOME/bin/.jconsoleKeyStore -file jconsole_client.cert

And we import it again on the remote machine.

keytool -import -alias jconsole_client -keystore JBOSS_HOME/bin/.jbossKeyStore -file jconsole_client.cert

The JBoss keystore should now look something like this:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

jconsole_client, 29-apr-2007, trustedCertEntry,
Certificate fingerprint (MD5): 15:3F:7D:AD:A5:65:8C:E8:CB:D2:4D:39:4E:68:13:01
jconsole, 27-apr-2007, PrivateKeyEntry,
Certificate fingerprint (MD5): AF:3E:79:18:E7:4E:D1:42:A5:E2:7C:7F:1F:3D:5A:6D
	

To enable SSL client authentication we add the following JVM option to the JBoss startup script:

 -Dcom.sun.management.jmxremote.ssl.need.client.auth=true
 -Djavax.net.ssl.trustStore=JBOSS_HOME/bin/.jbossKeyStore
	

When starting JConsole we now need to pass the location of the keystore and the keystore password.

jconsole -J-Djavax.net.ssl.keyStore=JAVA_HOME/bin/.jconsoleKeyStore -J-Djavax.net.ssl.keyStorePassword=secret -J-Djavax.net.ssl.trustStore=JAVA_HOME/bin/.jconsoleKeyStore -J-Djava.util.logging.config.file=logging.properties

Now that we're using SSL client authentication we may decide that we no longer need the password authentication as well. To disable passwordauthentication set authentication to false in the JBoss startup script.

 -Dcom.sun.management.jmxremote.authenticate=false
	

The disadvantage of disabling password authentication is that we now no longer have the ability to assign readonly access to users, since accesscontrol can only be configured for password authentication. Because of this it may be a good idea to use password authentication in combinationwith SSL client authentication.

Enabling RMI registry authentication

With SSL client authentication enabled, the last step to securing access to remote applications is protecting the RMI registry by using SSL client authentication for the RMI registry as well. This feature has been added in JDK 6. If you're using JDK 5 for the remote applicationit's not possible to secure access to the RMI registry which has been identified as a potential security issue (see Resources)
To enable this feature all we need to do is add the following JVM option to the JBoss startup script:

 -Dcom.sun.management.jmxremote.registry.ssl=true
	

All the security options that can be configured for JConsole are now enabled, but there's one last hurdle we have to take for JConsole tobe truly useful in production environments. We have to be able to access remote applications that are located behind a firewall. BecauseJConsole uses RMI to communicate with remote applications this is not really straightforward as we shall see in Part 3.

Resources

• For a complete description of the system properties involved in remote monitoring with JConsole read theRemote Monitoring and Managementsection of the Java SE Monitoring and Management Guide.This section also explains the identified potential security issue related to not securing access to the RMI registry.
• Requesting and importing of CA certificates is described in thekeytool documentation.
• To learn more about certificates and keystores read the Code and Document Security sectionof the Security Features in Java SE tutorial.