13. Remote Testing

In the event that your JMeter client machine is unable, performance-wise, to simulate enough users to stress your server or is limited at network level, an option exists to control multiple, remote JMeter engines from a single JMeter client. By running JMeter remotely, you can replicate a test across many low-end computers and thus simulate a larger load on the server. One instance of the JMeter client can control any number of remote JMeter instances, and collect all the data from them. This offers the following features:

Saving of test samples to the local machine
Management of multiple JMeterEngines from a single machine
No need to copy the test plan to each server - the client sends it to all the servers

Note: The same test plan is run by all the servers. JMeter does not distribute the load between servers, each runs the full test plan. So if you set 1000 Threads and have 6 JMeter server, you end up injecting 6000 Threads.

However, remote mode does use more resources than running the same number of CLI mode tests independently. If many server instances are used, the client JMeter can become overloaded, as can the client network connection. This has been improved by switching to Stripped modes (see below) but you should always check that your client is not overloaded.

Note that while you can execute the JMeterEngine on your application server, you need to be mindful of the fact that this will be adding processing overhead on the application server and thus your testing results will be somewhat tainted. The recommended approach is to have one or more machines on the same Ethernet segment as your application server that you configure to run the JMeter Engine. This will minimize the impact of the network on the test results without impacting the performance of the application server itself.

Step 0: Configure the nodes

Make sure that all the nodes (client and servers) :

are running exactly the same version of JMeter.
are using the same version of Java on all systems. Using different versions of Java may work but is discouraged.
have a valid keystore for RMI over SSL, or you have disabled the use of SSL.

If the test uses any data files, note that these are not sent across by the client so make sure that these are available in the appropriate directory on each server. If necessary you can define different values for properties by editing the user.properties or system.properties files on each server. These properties will be picked up when the server is started and may be used in the test plan to affect its behaviour (e.g. connecting to a different remote server). Alternatively use different content in any datafiles used by the test (e.g. if each server must use unique ids, divide these between the data files)

Step 1: Start the servers

To run JMeter in remote node, start the JMeter server component on all machines you wish to run on by running the JMETER_HOME/bin/jmeter-server (unix) or JMETER_HOME/bin/jmeter-server.bat (windows) script.

Note that there can only be one JMeter server on each node unless different RMI ports are used.

The JMeter server application starts the RMI registry itself; there is no need to start RMI registry separately.

By default, RMI uses a dynamic port for the JMeter server engine. This can cause problems for firewalls, so you can define the JMeter property server.rmi.localport to control this port number. it will be used as the local port number for the server engine.

Step 2: Add the server IP to your client's Properties File

Edit the properties file on the controlling JMeter machine. In JMETER_HOME/bin/jmeter.properties, find the property named "remote_hosts" and add the value of your running JMeter server's IP address. Multiple such servers can be added, comma-delimited.

Note that you can use the -R command line option instead to specify the remote host(s) to use. This has the same effect as using -r and -Jremote_hosts={serverlist}. E.g.

jmeter -Rhost1,127.0.0.1,host2

If you define the JMeter property server.exitaftertest=true, then the server will exit after it runs a single test. See also the -X flag (described below)

Step 3a: Start the JMeter Client from a GUI client to check configuration

Now you are ready to start the controlling JMeter client. For MS-Windows, start the client with the script "bin/jmeter.bat". For UNIX, use the script "bin/jmeter". You will notice that the Run menu contains two new sub-menus: "Remote Start" and "Remote Stop" (see figure 1). These menus contain the client that you set in the properties file. Use the remote start and stop instead of the normal JMeter start and stop menu items.

Step 3b: Start the JMeter from a CLI mode Client

GUI mode should only be used for debugging, as a better alternative, you should start the test on remote server(s) from a CLI mode (command-line) client. The command to do this is:

jmeter -n -t script.jmx -r

jmeter -n -t script.jmx -R server1,server2,…

Other flags that may be useful:

-Gproperty=value: define a property in all the servers (may appear more than once)
-X: Exit remote servers at the end of the test.

The first example will start the test on whatever servers are defined in the JMeter property remote_hosts;
The second example will define remote_hosts from the list of servers and then start the test on the remote servers.
The command-line client will exit when all the remote servers have stopped.

13.1 Setting up SSL¶

Since JMeter 4.0 the default transport mechanism for RMI will use SSL. SSL needs keys and certificates to work. You will have to create those keys yourself.

The simplest setup is to use one key/cert pair for all JMeter servers and clients you want to connect. JMeter comes with a script to generate a keystore that contains one key (and its corresponding certificate) named rmi. The script is located in the bin directory and is available for Windows systems (called bin/create-rmi-keystore.bat) and Unix like systems (called bin/create-rmi-keystore.sh). It will generate a key-pair, that is valid for seven days, with a default passphrase of value 'changeit'. It is advised to call it from inside the bin directory.

When you run the script, it will ask you some questions about some names it will embed in the certificate. You can type in whatever you want, as long the keystore tool accepts it. That value has to match the property server.rmi.ssl.keystore.alias, which defaults to rmi. A sample session to create the keystore is shown below.

$ cd jmeter/bin
$ ./create-rmi-keystore.sh
What is your first and last name?
  [Unknown]:  rmi
What is the name of your organizational unit?
  [Unknown]:  My unit name
What is the name of your organization?
  [Unknown]:  My organisation name
What is the name of your City or Locality?
  [Unknown]:  Your City
What is the name of your State or Province?
  [Unknown]:  Your State
What is the two-letter country code for this unit?
  [Unknown]:  XY
Is CN=rmi, OU=My unit name, O=My organisation name, L=Your City, ST=Your State, C=XY correct?
  [no]:  yes

Copy the generated rmi_keystore.jks to jmeter/bin folder or reference it in property 'server.rmi.ssl.keystore.file'

The defaults settings for RMI should work with this setup. Copy the file bin/rmi_keystore.jks to every JMeter server and client you want to use for your distributed testing setup.

13.2 Doing it Manually¶

In some cases, the jmeter-server script may not work for you (if you are using an OS platform not anticipated by the JMeter developers). Here is how to start the JMeter servers (step 1 above) with a more manual process:

Step 1a: Start the RMI Registry

Since JMeter 2.3.1, the RMI registry is started by the JMeter server, so this section does not apply in the normal case. To revert to the previous behaviour, define the JMeter property server.rmi.create=false on the server host systems and follow the instructions below.

JMeter uses Remote Method Invocation (RMI) as the remote communication mechanism. Therefore, you need to run the RMI Registry application (which is named "rmiregistry") that comes with the JDK and is located in the "bin" directory. Before running rmiregistry, make sure that the following jars are in your system classpath:

JMETER_HOME/lib/ext/ApacheJMeter_core.jar
JMETER_HOME/lib/jorphan.jar
JMETER_HOME/lib/logkit-2.0.jar

The rmiregistry application needs access to certain JMeter classes. Run rmiregistry with no parameters. By default the application listens to port 1099.

Step 1b: Start the JMeter Server

Once the RMI Registry application is running, start the JMeter Server. Use the "-s" option with the jmeter startup script ("jmeter -s").

Steps 2 and 3 remain the same.

13.3 Tips¶

JMeter/RMI requires a connection from the client to the server. This will use the port you chose, default 1099.
JMeter/RMI also requires a reverse connection in order to return sample results from the server to the client.
These will use high-numbered ports.
These ports can be controlled by jmeter property called client.rmi.localport in jmeter.properties.
If this is non-zero, it will be used as the base for local port numbers for the client engine. At the moment JMeter will open up to three ports beginning with the port defined in client.rmi.localport. If there are any firewalls or other network filters between JMeter client and server, you will need to make sure that they are set up to allow the connections through. If necessary, use monitoring software to show what traffic is being generated.

If you're running Suse Linux, these tips may help. The default installation may enable the firewall. In that case, remote testing will not work properly. The following tips were contributed by Sergey Ten.

If you see connections refused, turn on debugging by passing the following options.

rmiregistry -J-Dsun.rmi.log.debug=true \
     -J-Dsun.rmi.server.exceptionTrace=true \
     -J-Dsun.rmi.loader.logLevel=verbose \
     -J-Dsun.rmi.dgc.logLevel=verbose \
     -J-Dsun.rmi.transport.logLevel=verbose \
     -J-Dsun.rmi.transport.tcp.logLevel=verbose \

Since JMeter 2.3.1, the RMI registry is started by the server; however the options can still be passed in from the JMeter command line. For example: "jmeter -s -Dsun.rmi.loader.logLevel=verbose" (i.e. omit the -J prefixes). Alternatively the properties can be defined in the system.properties file.

The solution to the problem is to remove the loopbacks 127.0.0.1 and 127.0.0.2 from /etc/hosts. What happens is jmeter-server can't connect to rmiregistry if 127.0.0.2 loopback is not available. Use the following settings to fix the problem.

Replace

`dirname $0`/jmeter  -s "$@"

With

HOST="-Djava.rmi.server.hostname=[computer_name][computer_domain] \
      -Djava.security.policy=`dirname $0`/[policy_file]" \
`dirname $0`/jmeter $HOST -s "$@"

Also create a policy file and add [computer_name][computer_domain] line to /etc/hosts.

In order to better support SSH-tunneling of the RMI communication channels used in remote testing, since JMeter 2.6:

a new property "client.rmi.localport" can be set to control the RMI port used by the RemoteSampleListenerImpl
To support tunneling RMI traffic over an SSH tunnel as the remote endpoint using a port on the local machine, loopback interface is now allowed to be used if it has been specified directly using the Java System Property "java.rmi.server.hostname" parameter.

13.4 Using a different port¶

By default, JMeter uses the standard RMI port 1099. It is possible to change this. For this to work successfully, all the following need to agree:

On the server, start rmiregistry using the new port number
On the server, start JMeter with the property server_port defined
On the client, update the remote_hosts property to include the new remote host:port settings

Since JMeter 2.1.1, the jmeter-server scripts provide support for changing the port. For example, assume you want to use port 1664 (perhaps 1099 is already used).

On Windows (in a DOS box)

C:\JMETER> SET SERVER_PORT=1664
C:\JMETER> JMETER-SERVER [other options]

On Unix:

$ SERVER_PORT=1664 jmeter-server [other options]

[N.B. use upper case for the environment variable]

In both cases, the script starts rmiregistry on the specified port, and then starts JMeter in server mode, having defined the "server_port" property.

The chosen port will be logged in the server jmeter.log file (rmiregistry does not create a log file).

13.5 Using a different sample sender¶

Listeners in the test plan send their results back to the client JMeter which writes the results to the specified files By default, samples are sent back synchronously as they are generated. This can affect the maximum throughput of the server test; the sample result has to be sent back before the thread can continue. There are some JMeter properties that can be set to alter this behaviour.

mode

sample sending mode - default is StrippedBatch since 2.9. This should be set on the client node.

Standard

send samples synchronously as soon as they are generated

Hold

hold samples in an array until the end of a run. This may use a lot of memory on the server and is discouraged.

DiskStore

store samples in a disk file (under java.io.temp) until the end of a run. The serialised data file is deleted on JVM exit.

StrippedDiskStore

remove responseData from successful samples, and use DiskStore sender to send them.

Batch

send saved samples when either the count (num_sample_threshold) or time (time_threshold) exceeds a threshold, at which point the samples are sent synchronously. The thresholds can be configured on the server using the following properties:

num_sample_threshold: number of samples to accumulate, default 100
time_threshold: time threshold, default 60000 ms = 60 seconds

See also the Asynch mode, described below.

Statistical

send a summary sample when either the count or time exceeds a threshold. The samples are summarised by thread group name and sample label. The following fields are accumulated:

elapsed time
latency
bytes
sample count
error count

Other fields that vary between samples are lost.

Stripped

remove responseData from successful samples

StrippedBatch

remove responseData from successful samples, and use Batch sender to send them.

Asynch

samples are temporarily stored in a local queue. A separate worker thread sends the samples. This allows the test thread to continue without waiting for the result to be sent back to the client. However, if samples are being created faster than they can be sent, the queue will eventually fill up, and the sampler thread will block until some samples can be drained from the queue. This mode is useful for smoothing out peaks in sample generation. The queue size can be adjusted by setting the JMeter property asynch.batch.queue.size (default 100) on the server node.

StrippedAsynch

remove responseData from successful samples, and use Async sender to send them.

Custom implementation

set the mode parameter to your custom sample sender class name. This must implement the interface SampleSender and have a constructor which takes a single parameter of type RemoteSampleListener.

Stripped mode family strips responseData so this means that some Elements that rely on the previous responseData being available will not work.
This is not really a problem as there is always a more efficient way to implement this feature.

The following properties apply to the Batch and Statistical modes:

num_sample_threshold: number of samples in a batch (default 100)
time_threshold: number of milliseconds to wait (default 60 seconds)

13.6 Dealing with nodes that failed starting¶

For large-scale tests there is a chance that some part of remote servers will be unavailable or down. For example, when you use automation script to allocate many cloud machines and use them as generators, some of requested machines might fail booting because of cloud's issues. Since JMeter 2.13 there are new properties to control this behaviour.

First what you might want is to retry initialization attempts in hope that failed nodes just slightly delayed their boot. To enable retries, you should set client.tries property to total number of connection attempts. By default it does only one attempt. To control retry delay, set the client.retries_delay property to number of milliseconds to sleep between attempts.

Finally, you might still want to run the test with those generators that succeeded initialization and skipping failed nodes. To enable that, set the client.continue_on_fail=true property.

13.7 Using a security-manager¶

When running JMeter in a distributed environment you have to be aware, that JMeter is basically a remote execution agent on both the server and client side. This could be used by a malicious party to gain further access, once it has compromised one of the JMeter clients or servers. To mitigate this Java has the concept of a security manager that gets asked by the JVM before potential dangerous actions are executed. Those actions could be resolving host names, creating or reading files or executing commands in the OS.

The security manager can be enabled by setting the Java system properties java.security.manager and java.security.policy. Be sure to have a look at the Quick Tour of Controlling Applications.

Using the new mechansism of setenv.sh (or setenv.bat under Windows) you can enable the security manager by adding the following code snippet to ${JMETER_HOME}/bin/setenv.sh:

JVM_ARGS=" \
   -Djava.security.manager \
   -Djava.security.policy=${JMETER_HOME}/bin/java.policy \
   -Djmeter.home=${JMETER_HOME} \
"

The JVM will now add the policies defined in the file ${JMETER_HOME}/bin/java.policy to the possibly globally defined policies. If you want your definition to be the only source for policies, use two equal signs instead of one when setting the property java.security.policy.

The policies will be dependent upon your use case and it might take a while to find the correct restricted and allowed actions. Java can help you find the needed policies with the property java.security.debug. Set it to access and it will log all permissions, that it gets asked to allow. Simply add the following line to your setenv.sh:

JVM_ARGS="${JVM_ARGS} -Djava.security.debug=access"

It might look a bit strange, that we define a Java system property jmeter.home with the value of ${JMETER_HOME}. This variable will be used in the example java.policy to limit the file system access and allow it only to read JMeters configuration and libraries and restrict write access to specific locations, only.

The following policy definition file has been used for a simple remote test. You will probably have to tweak the policies, when you run more complex scenarios. The test plans are somewhere placed inside the users home directory under a directory called jmeter-testplans. The sample java.policy looks like:

grant codeBase "file:${jmeter.home}/bin/*" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/jorphan.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/log4j-api-2.11.1.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/log4j-slf4j-impl-2.11.1.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/slf4j-api-1.7.25.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/log4j-core-2.11.1.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/ext/*" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/httpclient-4.5.6.jar" {
  permission java.net.SocketPermission "*", "connect,resolve";
};

grant codeBase "file:${jmeter.home}/lib/darcula.jar" {
  permission java.lang.RuntimePermission "modifyThreadGroup";
};

grant codeBase "file:${jmeter.home}/lib/xercesImpl-2.12.0.jar" {
  permission java.io.FilePermission "${java.home}/lib/xerces.properties", "read";
};

grant codeBase "file:${jmeter.home}/lib/groovy-all-2.4.15.jar" {
  permission groovy.security.GroovyCodeSourcePermission "/groovy/script";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.reflect";
  permission java.lang.RuntimePermission "getProtectionDomain";
};

grant {
  permission java.io.FilePermission "${jmeter.home}/backups", "read,write";
  permission java.io.FilePermission "${jmeter.home}/backups/*", "read,write,delete";
  permission java.io.FilePermission "${jmeter.home}/bin/upgrade.properties", "read";
  permission java.io.FilePermission "${jmeter.home}/lib/ext/-", "read";
  permission java.io.FilePermission "${jmeter.home}/lib/ext", "read";
  permission java.io.FilePermission "${jmeter.home}/lib/-", "read";
  permission java.io.FilePermission "${user.home}/jmeter-testplans/-", "read,write";
  permission java.io.SerializablePermission "enableSubclassImplementation";
  permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
  permission java.lang.RuntimePermission "accessClassInPackage.jdk.internal.dynalink.support";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.awt";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.misc";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.swing";
  permission java.lang.RuntimePermission "accessDeclaredMembers";
  permission java.lang.RuntimePermission "createClassLoader";
  permission java.lang.RuntimePermission "createSecurityManager";
  permission java.lang.RuntimePermission "getClassLoader";
  permission java.lang.RuntimePermission "getenv.*";
  permission java.lang.RuntimePermission "nashorn.createGlobal";
  permission java.util.PropertyPermission "*", "read";
};

The usage of java.security.AllPermission is an easy way to make your test plans work, but it might be a dangerous shortcut on your path to security.

Go to top