XLT – User Manual

Introduction

About this Document

This document describes all the components and features of the regression and load testing tool Xceptance Load Test (XLT).

To get the most out of this document, and to be able to work effectively with XLT, basic knowledge about web technologies, the Java programming language, and the JUnit concept is recommended. You definitely don’t need to be an expert, but being familiar with these topics will help you better understand the next sections.

What is XLT?

XLT stands for Xceptance LoadTest. XLT offers an easy way to write and run regression and load tests for web applications. Nearly every software that provides access via HTTP/HTML can be tested using XLT. There is also extensive JavaScript support for testing applications that use Web 2.0 technologies. XLT supports not only web testing but also SQL tests, RCP-based application tests or any other test that should run as a load or performance test in a distributed manner. Since XLT is written in Java it can be run on any platform that supports Java.

XLT Script Developer

The easiest way to start is with the XLT Script Developer. It operates as a Firefox add-on and provides an easy-to-use user interface for developing and running test cases and entire test suites. XLT Script Developer can record the page flow while navigating through a website and features a wide range of validations on the web page.

XLT Framework

Scripting test cases using the Script Developer is easy, but keep in mind that you are limited by the strictly linear approach and the set of available commands.

The XLT Framework provides different approaches for writing test cases in Java. Starting with Java code automatically generated from the recorded Script Developer test cases, you can extend your test suite by using the XLT API. This features a higher level command scripting API with a very intuitive syntax as well as the WebDriver API and also the lower level XLT Action API based on HtmlUnit.

XLT API provides a programming paradigm to translate all test scenarios into JUnit4 tests. The principles of JUnit4 and its annotations are used to implement and tag test cases. This way, each XLT test is a JUnit test, so XLT tests can be executed just like any other unit test within an existing build process.

Installation Instructions

It is recommended that you install Xceptance LoadTest before continuing with this manual.

System Requirements

Hardware

Typically, XLT requires a CPU running at 1.5 GHz at least and about 1 GB of RAM. XLT can run on slower hardware but this depends on the tests themselves. The default installation requires about 85 MB of hard disk space, but load test results might require more free space, so at least 1 GB of free disk space is recommended.

Software

XLT runs on any operating system for which a Java 6 (or higher) virtual machine is available, such as Microsoft Windows, Linux, Oracle Solaris, HP-UX, or Mac OS X. We recommend using Oracle’s JDK 6 if it is available for your platform. XLT will also run in JVMs provided by other vendors (e.g. BEA, HP, IBM), but this has not been tested extensively. In order to view the HTML load test reports generated by XLT, a modern web browser is required, such as Firefox 3, Internet Explorer 7, Opera 9 or Safari 3. Please note that JavaScript needs to be enabled to fully utilize all functionality. If you plan to use the XLT Script Developer, you will need Firefox 3.5 or above.

Installing XLT

The installation of XLT is simple. Just unzip the XLT archive to a file system location of your choice. The root directory is part of the archive, so you do not need to create it separately. Please note that although spaces in the path are supported by XLT, it is easier to code tests when the path is free of them.

On Unix-like systems, you need to grant execution rights to all sh-files in the bin directory.

chmod u+x <XLT>/bin/*.sh

If you received a license file, copy it to the directory <XLT>/config. A Basic License for XLT does not require the installation of a license file. In this case, XLT is not limited in time or functionality, but the number of virtual users is restricted to five. Also notice that in this case the Basic License Terms apply. See the file <XLT>/doc/license.html for more information.

Finally, make sure that the executable directory of your Java installation is listed in your PATH environment variable so that the XLT start scripts can find the virtual machine runtime.

In order to install the XLT Script Developer, an extension to the Firefox browser, you need to carry out the following steps:

  1. Start Firefox.
  2. Click File > Open File....
  3. Navigate to the <XLT>/tools directory and select the .xpi file. The Add-On installation dialog should appear.
  4. Click Install to finish the installation of the Script Developer extension.

Alternatively, drag the .xpi file and drop it onto the Firefox window to install it.

Updating XLT

Before you update to a newer XLT version it is highly recommend that you back up all modified files and project-specific or customized settings. In particular this includes:

After creating a backup of all these files and directories you can download the latest version from the Xceptance website and install XLT as described in the previous section of this chapter. You can have two XLT installations in parallel as the name of the unpacked installation folder includes the version and revision number by default.

Now you can copy all the backed up files and directories mentioned above to the corresponding place in the new XLT installation directory.

New available test suite settings are provided in the default.properties file of the demo test suite testsuite-pebble that is part of the XLT installation. Copy this default.properties file from <LatestXLTversion>/samples/testsuite-pebble/config to the config directory of your test suites <YourTestSuite>/config.

In order to install the latest version of XLT Script Developer just open the latest .xpi file with the Firefox File | Open file menu. The file is located at <LatestXLTversion>/tools.

If you work with Java-based test cases, you have to add the updated XLT libraries to the Java build path of the Eclipse project. See the section Importing the Pebble Test Suite into Eclipse for a detailed description of the necessary steps.

Uninstalling XLT

Before uninstalling, make sure that you have backed up all the test results and test reports that you want to keep. To uninstall XLT simply delete its installation directory. If you installed the Script Developer Firefox extension, use the Firefox Add-On dialog to remove the extension.

Script Developer

Introduction

XLT Script Developer is used to create test-case scripts. Script test cases are based on a simple syntax and a reduced set of operations, which makes them a perfect fit for non-programmers. No other tool is necessary to create, edit, and manage basic script test cases beside the Script Developer, which is a Firefox extension.

Script Developer is used to record test cases. That means the tester simply uses the application under test. All interactions with the application are recorded in the background and stored to an XML script file. While recording, you can also perform validations as well as use appropriate assert commands. Any recorded value can later be extracted out of the script into a test data file to separate test data from script code. Scripts may also be exported as ordinary Java code.

Via the Script Developer, script files can be replayed in Firefox at any time to quickly check whether the test case still runs successfully.

Settings

Before you start, make sure the Script Developer Firefox extension is installed correctly. Open the Script Developer window, either via the Firefox Tools menu or via clicking the XLT icon in the status bar.

Next you need to change some basic settings. In the Script Developer window, open the configuration dialog by clicking on XLT Script Developer|Settings in the upper left, and configure the settings appropriately.

Script Developer Settings Script Developer Settings

General Settings
Java Code Settings
Recording Settings
Replay settings
Editor settings

User Interface Elements

The main window of XLT Script Developer features the following sections and elements.

Script Developer Script Developer

Toolbar and Record/Replay Section

Script Library

The Script Library lists all available test cases and modules structured in packages. Script developer loads all test cases and modules from <test-suite>/scripts and its sub-folders. The structure of the sub-folders reflects the package structure of the script library.

The package structure in the script library also reflects the package structure of the generated JUnit wrapper classes.

Below the script library you can find an input field to filter the displayed list of test cases or modules. By clicking on the small arrow icon on the left, you can filter the list either by name, description, tag, or any combination of these. Typed characters are case insensitive. By clicking the x icon on the right side of the input field or by deleting all characters, you can reset the search result.

It is possible to change the alphabetic sorting (A-Z or Z-A) of the packages by clicking on the header of the Name column. The second column shows a description for each test case or module. You can add or edit the description in the Edit Details dialog.

Editor Tabs

By double clicking on a script (or via context menu options Edit Test Case or Edit Module), you can open a test case or module for editing. The script’s commands will be listed inside the work area on the right-hand side of the developer window. It is possible to open more than one script at a time. You can select the active test case or module by clicking on the related tab above the command list. The script editor tab displays the name, target and value attributes of each command in three separate columns, whose width can be adjusted.

Recording and Replaying Test Cases

Recording

The easiest way to create test scripts is to record the steps a user takes to interact with a web application. To do this:

  1. Open the web page you want to start with in a Firefox tab and make sure this tab remains the active tab, i.e. the foreground tab.
  2. Switch to the Script Developer and create a new test case via the context menu in the Script Library. Provide a meaningful name. As a result, an empty script editor tab opens.
  3. Click the Start Recording icon in the tool bar to start recording.
  4. Switch back to your web page and start using it. All your interactions with the page will be recorded.
  5. Once you are done with the test scenario, switch back to the Script Developer and stop recording by clicking the Stop icon.
  6. Do not forget to save the new script by clicking the Save icon or using the Ctrl+S shortcut. (A list of all available shortcuts is provided in the Appendix.)

Note that beside recording normal web page flow you can also validate the correctness of pages by using assert commands. When interacting with a web page, typically a new or changed page is displayed, which should be checked. To record assertions:

  1. Open the Firefox context menu and choose XLT Script Developer (available only while recording). A sub-menu opens.
  2. Choose an appropriate assertion from the sub-menu. (See the Appendix for a complete list of all available assertions.)
  3. Continue interacting with the web page as defined by your test scenario.

For every assertion there is also a variant that checks whether the respective condition is not true. You can record as many assertions as you like.

Replaying a Test Case

Once you’ve recorded a test case, it can be executed inside the browser.

  1. Open the respective test-case script in a test-case editor tab or activate the tab if it is already open.
  2. Click the Play icon in the tool bar.

The script will be replayed in the current Firefox browser tab.

While the script runs, you can watch the results of the script execution. You will see how links are clicked, input fields are filled, buttons and checkboxes are clicked, etc. The element on the page the script is currently dealing with is highlighted (yellow for actions and orange for validations) to make it easier for you to follow the script execution. In the script tab, the commands are marked with a special status icon. As long as the command (or module) is being executed, the status icon is displayed in yellow. Once the command is finished, the icon turns either green in case of success, or red in case of errors. Note that script execution stops immediately if a command fails.

Sometimes it is difficult to follow the script execution. In this case, use the speed slider in the tool bar to reduce the replay speed. On the other hand, if you do not want to follow the interactions and need the result as quickly as possible, increase the replay speed. Note that the actual commands are not executed slower or faster. The slider influences only the time elements are highlighted.

You can stop or pause script execution at any time. To stop a script, just click the Stop icon in the tool bar. As a result, the script is terminated. The situation is different when you click the Pause icon. In this case, script execution is suspended (after finishing the currently running command). To continue script execution, click the Continue Replay icon. Alternatively, you may click the Single Step Forward icon which causes only the next command to be executed.

You can set a start point and one or more breakpoints for a test script. See the sections Start point for replaying test cases and Break points for more details.

Note that clicks on alert, confirmation or prompt boxes will not be recorded at the moment. When replaying test cases no alert, prompt or confirmation box will pop up. XLT Script Developer always goes on with replaying the test case, simulating a return value of the alert() or confirm() function which is equivalent to clicking “OK” or “Yes”. For prompt() an empty string will be simulated as the user’s input value.

Running Batch Tests

It is possible to run a sequence of test cases or an entire test suite as a batch test. For this you have to select the desired test cases from the script library by clicking them while holding the CTRL key. Alternatively you can select one or more packages to run. Now choose Run as Batch Test from the script library context menu to start the batch test.

All selected test cases will be executed consecutively and the result shown in a new tab in the work area. Each test case has a colored bullet showing the current state of the test execution. Grey is for not tested, yellow for currently running test cases, green for passed and red for failed test cases. The total number of selected test cases, the number of already executed test cases, an error count and the elapsed time is visible in the header of the batch test tab.

Batch Test Batch Test

After the batch test is completed the Rerun failed tests button may become active if at least one test failed. Clicking this button starts a new batch test in a new tab with all the test cases that failed in the previous batch test. The result of a batch test can also be exported to HTML. All you have to do is to press the Export as HTML button and provide a name and target location for the exported file. You can then view the batch execution report with any web browser.

Batch Test HTML Report Batch Test HTML Report

Editing Script Library, Test Cases and Modules

The script test cases and modules can be modified at any time. Often enough, it is necessary to fix mistakes made during the recording of the test scenario, for example if you click the wrong link or forget to add an assertion. In the first case, just go back and continue with your test scenario. The unwanted or incorrect steps can be deleted later. In the second case, manually add the appropriate assertion commands after recording has finished. You can even modify a test case while recording. Switch to the Script Developer window, make the necessary changes, and return to the web page and continue recording.

Editing Script Library

Edit test case properties Edit test case properties

The following options for editing the script library and scripts (test cases or modules) are accessible via the context menu in the script library:

Editing Test Case Details

When selecting Edit Test Case Details in the context menu, a dialog opens that allows the user to edit details of the selected test case.

Edit Test Case Details Edit Test Case Details

Editing Module Details

Select Edit Details from the context menu while a module is selected in order to edit the details of the currently selected module.

Edit Module Details Edit Module Details

Editing Scripts

After opening a script (test case or module) as a tab in a work area you can edit this script by selecting one of the options from the context menu (right click). A script may consist of actions, commands, and module calls.

The following options for editing a script are available (item = action, command or module call):

Edit test case Edit test case

Editing Commands

To edit a command select Edit from the context menu and the XLT – Edit Command dialog will appear.

Edit command Edit command

Start Point for Replaying Test Cases

By default, the execution of a script will start at the beginning. However, sometimes it might be handy to start the script execution from a specific command.. To start a script at a specific position, select the respective command and press S to set a start point or choose Set Start Point from the context menu. A start-point marker that looks like the replay icon appears next to the command.

When running the script with the Play icon in the toolbar, the execution will now start from the marked command.

Make sure that the right page is displayed in the active Firefox tab, so that the script can be started at the selected command. Otherwise the script is likely to fail.

Break Points

In order to automatically pause the script execution at a certain command, set a breakpoint before running the script by selecting the respective command and press B afterwards. A breakpoint marker appears next to the command. When the script is replayed, execution is paused automatically when the command has been reached. You can go on with replaying the script by clicking Continue Replay or Single Step Forward in the tool bar. To clear the breakpoint, mark the command again and press B. A breakpoint may also be set or deleted via context menu or by double-clicking the breakpoint column (the leftmost column). When setting a breakpoint for a command inside a module that is used multiple times, the respective command in each module call will be marked with a breakpoint and therefore the replay is paused each time one of these module calls is reached.

Breakpoints exist in memory only. Closing the appropriate script editor tab or reloading the script will clear all of its breakpoints.

Actions

As you may have noticed, the Script Developer inserts Actions automatically while recording. An action is a sequence of steps that belong together. For example, filling in the inputs of a form, submitting the form by clicking the submit button, and checking the resulting page with assertions is typically one action. Actions are primarily used to break the page flow down into atomic steps and to give those steps a name. Action execution times are measured and reported while running load tests.

The Script Developer gives actions generic names, but you can rename them later on to facilitate script maintenance. You can also insert a new action manually at any position of your script (Insert|Action). The start of a new action will automatically end the previous action, also if the following action is part of a module call.

Please note that actions are not to be used to structure your scripts visually since they are not comments. In other words, you should not use actions to structure a long list of validations without loading a new page. Using actions in this context conflicts with the basic concept of an action, namely that it should always load one new page. The problem is that misuse of actions in script test cases ruins the load-test reports and makes it difficult to analyze the test results when the actions do not represent the page flow. Furthermore, remember to give your actions a meaningful name (e.g., Search , Browse or AddToCart).

Script Modules

Extracting a Module

Like test cases, script modules are sequences of commands, actions and optional calls to other modules. They can be written from scratch, but it is much easier to extract them from an existing test case. That means you record your test script first, and afterwards you identify parts of a script which are reusable. These parts are then factored out into separate modules. The following steps are involved in extracting a module:

  1. Open the original test case.
  2. Select the groups of commands you want to extract by clicking them while holding CTRL key.
  3. Choose Extract Module from the context menu.
  4. Provide a meaningful module name and the script package.
  5. Optionally provide tags, a description, and module parameters.

After confirming with the OK button you will find the new module in the script library. Additionally, the original test case will have been modified so that the selected items are replaced by a call to the new module.

Most test-case editing options are available for modules as well, such as edit module properties and edit commands of a module. Therefore the section Edit script library and Editing Commands of this document applies to modules, too. You can open a module in a separate tab and edit the commands of the module there. But it is also possible to edit a modules’s command from within a test case that is using the module. When you do this keep in mind that editing module’s command may also affect other test cases that use the same module.

The script library context menu has two options for creating new modules. New Script Module can be used to create a new script module from scratch as an alternative to extracting it from a test case. New Java Module creates a script interface to integrate Java code into a script test case for the purpose of running the test case outside Script Developer later but will be skipped when creating the test case using Script Developer. For more information, see the section Creating Java modules.

Defining Module Parameters

In order to increase the re-usability of modules in other contexts and scenarios, a module can be parameterized. For example, a module that logs in a user should be parameterized with the user name and password. Script module parameters are defined as part of the module’s meta data. To use these parameters later in the module’s script code as target or value, you refer to them by using the special @{} placeholder notation (for example: @{userName} and @{password}) in the target and/or value of a command.

In order to parameterize a module execute the following steps:

  1. Select the module from the library and open the Edit Module Details dialog.
  2. Declare the module parameters. You can add parameters with the + button and delete parameters with - (see figure below “Edit Module Details”).
  3. After confirming the changes in module details, open the module in an editor tab in the work area.
  4. Edit the commands that shall use any of the defined parameters. Where needed, replace literal values in the target or value of the command with @{name} placeholders (see figure below “Refer to module parameters in a command”).
  5. Save the module.

Edit Module Details Edit Module Details

Refer to module parameters in a command Refer to module parameters in a command

For all test cases using this module, you have to provide a value for each placeholder.

  1. Open the test case in an editor tab.
  2. Open the Edit module dialog by selecting Edit from the context menu of the module call inside the test case.
  3. Provide a valid value for each placeholder.
  4. Close the Edit module dialog and save the test case.

Provide Values for Module Parameters Provide Values for Module Parameters

Modules and Actions

It is possible to extract a module from any part of the script, but there are some basic rules you should follow when extracting modules, especially when it comes to actions. Actions are important when you plan to run load tests because load-test reports are based on actions. Always keep in mind the basic concepts of XLT test cases.

In XLT Script Developer an action starts with an action line in the script and ends with the beginning of the next action, regardless if the next action is part of a module or not.

In most cases, modules are used more than once in a test suite. Be careful of actions that are contained in modules. If a module starts with a sequence of commands followed by an action, then the commands of the module might be part of an action of a completely different module. This will not cause your test scripts to break, but it can cause confusion when analyzing load-test reports.

Bad style: Module call in script test case with hidden action Bad style: Module call in script test case with hidden action

Bad style: Hidden action in a module call Bad style: Hidden action in a module call

You can follow some simple rules when extracting test cases to achieve a better style and test reports that are much easier to analyze.

In other words, for each module you should decide if you:

Good style : Module calls in script test case Good style : Module calls in script test case

Good style: Modules with no action at all or completely encapsulated actions Good style: Modules with no action at all or completely encapsulated actions

Managing Test Data

By default, test data is hard-coded into the test script. In order to simplify changing test data, the data values can be extracted and saved to a data file. This data file has the same name as the test script suffixed with _data. For example, “TAuthor_data.xml” is the data file for the script “TAuthor.xml”. The data file contains simple name-value pairs.

To extract test data from a script test case or module:

  1. Choose Manage Test Data from the script library context menu. A dialog will come up that lists all values together with a generic parameter name.
  2. Check the values you want to extract and provide a parameter name for each value, if the suggested one is not a good fit.
  3. Close the dialog.
  4. Save the script.

Manage Test Data Manage Test Data

First the dialog shows only those values which can be exported to a data file. This includes typical test data such as text you have entered, but no element IDs or XPath expressions. Filtering out certain values is primarily done to increase clarity. If you want to export those values as well, click Show Optionals to display the corresponding rows.

Note that you can include previously extracted test data values back into the test case. Just open the test dialog and uncheck the values you do not want to be externalized any longer.

The defined placeholders can be used several times in the scripts as follows: ${placeholder_name}. They work as global variables. If you use those placeholders multiple times at different locations in the script then the same placeholder also appears multiple times in Manage Test Data. Changes in the value column only need to be made once and not in each duplicated test data entry. They will be synchronized automatically.

Extracting test data works not only for test-case scripts but also for modules. If a module has a test-data file and this module is called by a test case that also has a test-data file, both test-data files are used to access the required values. If both test-data files contain an entry with the same name but different values, then the test-case data file has priority.

Test data variables are scoped. That means, that a script cannot access test variables defined by a called script. Furthermore, the calling script may overwrite test data variables defined by the called script. This forms a scope chain which is defined by the call hierarchy of your scripts.

Exporting Test Cases to Java

Scripting test cases using Script Developer is easy, but limited. The scripts are strictly linear and the set of available commands cannot always replace a programming language. In certain cases, it is useful to switch to other programming languages such as Java. Java is a powerful tool that provides the structure to reuse code, as well as a wide range of available libraries. Typically, you will need to switch to Java if your tests need to act randomly, for example if you want the TBrowseCatalog test case to not always open the same catalog, but an available randomly chosen catalog instead. This behavior is especially useful for load tests which shall simulate realistic traffic.

The setting Generate JUnit wrapper class for test cases only allows for the generation of Java wrapper classes to execute the XML scripts from outside Script Developer. This is not a real export.

In contrast, Export translates the test-script file into Java syntax and generates one or more classes which represent the test case and modules. When exporting to Java you have the choice to select the API used for the resulting code. It is possible to generate code based on the XLT Scripting API or code based on the XLT Action API.

To export a script test case or module to Java follow these steps:

  1. Select the script(s) you want to export from the script library.
  2. Select Export from the content menu. The XLT – Script Export dialog will open.
  3. Define the source directory, package prefix, and API (or accept the suggested values).
  4. Click OK.

As a result, you will find the generated Java code in the specified packages.

Export Script to Java Export Script to Java

XLT Action API does not support Java modules. When exporting to XLT Action API possible calls to Java modules are omitted in the resulting code and just a comment is inserted instead.

Export to XLT Scripting API

This example shows the TAuthor test case of the demo test suite exported to XLT Scripting API code: 


/**
 * Tests the creation of new blog articles.
 */
public class TAuthor extends AbstractWebDriverScriptTestCase
{

    /**
     * Constructor.
     */
    public TAuthor()
    {
        super(new XltDriver(true), "http://localhost:8080/");
    }


    /**
     * Executes the test.
     */
    @Test
    public void test() throws Throwable
    {

        //
        // ~~~ OpenStartPage ~~~
        //
        startAction("OpenStartPage");
        deleteCookie("JSESSIONID");
        open("/pebble/");
        assertTitle("My blog");
        assertText("id=blogName", "My blog");
        final Login _login = new Login(this);
        _login.execute("username", "password");

        final CreateNewArticle _createNewArticle = new CreateNewArticle(this);
        _createNewArticle.execute(resolve("${title}"), resolve("${subtitle}"), resolve("${excerpt}"), resolve("${body}"));

        _createNewArticle.execute("Another title text", "Another subtitle text", "Another excerpt text.", "Another body text.");

        final Logout _logout = new Logout(this);
        _logout.execute();


    }
}

This is the code for the module CreateNewArticle, which is called by TAuthor: 

/**
 * A complete flow to create a new blog article.
 */
public class CreateNewArticle extends AbstractWebDriverScriptModule
{

    /**
     * Constructor.
     *
     * @param caller
     *      the calling script test case
     */
    public CreateNewArticle(final AbstractWebDriverScriptTestCase caller)
    {
        super(caller);
    }


    /**
     * Constructor.
     *
     * @param caller
     *      the calling module
     */
    public CreateNewArticle(final AbstractWebDriverScriptModule caller)
    {
        super(caller);
    }


    /**
     * Executes this module.
     * 
     * @param title
     * @param subtitle
     * @param excerpt
     * @param body
     */
    public void execute(final String title, final String subtitle, final String excerpt, final String body)
    {

        //
        // ~~~ NewBlogEntry ~~~
        //
        startAction("NewBlogEntry");
        clickAndWait("link=New blog entry");

        //
        // ~~~ WriteArticle ~~~
        //
        startAction("WriteArticle");
        type("name=title", title);
        type("name=subtitle", subtitle);
        type("name=excerpt", excerpt);
        type("name=body", body);
        clickAndWait("name=submit value=Preview");

        //
        // ~~~ SaveArticle ~~~
        //
        startAction("SaveArticle");
        clickAndWait("name=submit value=Save");

        //
        // ~~~ PublishArticle ~~~
        //
        startAction("PublishArticle");
        clickAndWait("name=submit value=Publish");

        //
        // ~~~ ConfirmPublishArticle ~~~
        //
        startAction("ConfirmPublishArticle");
        clickAndWait("name=submit");
        assertText("xpath=id('content')/div[@class='contentItem published']/h1/a", title);
        assertText("xpath=id('content')/div[@class='contentItem published']/h2", subtitle);
        assertText("xpath=id('content')/div[@class='contentItem published']/div[@class='contentItemBody']", body + "*");

    }
}

As you can see, the XLT Scripting API Java code is very similar to the script code. You can refactor the code as desired and use lower-level APIs directly, such as the WebDriver API, to implement more advanced functionality. You can also add code to randomize your tests or to retrieve test data from other sources, such as the GeneralDataProvider. See XLT Scripting API for details.

Export to XLT Action API

When exporting to XLT Action API the resulting code is based on separate classes for each action of a test case. Also, the script modules will be translated into separate classes. If a script module contains more then one action it will be represented by a flow.

To make this approach work without any problems it is very important that users follow the rules for defining actions in combination with modules. If there is an action in a module and it is not the first item of the module, then a automatically generated action will be inserted at the beginning of the exported module code.

Executing Tests Outside Script Developer

Once your test suite is complete, you can run the test cases outside Script Developer, in headless mode. This is especially useful for functional tests or load tests. In this case, the XLT framework is responsible for interpreting the script test cases and sending the respective requests to the system under test.

Note that it is always recommended to have a JUnit wrapper class for each script test case. This makes working with test scripts in your preferred IDE much easier, because these tools typically know how to work with JUnit classes.

Inside an IDE

To check whether your scripts are running successfully in headless mode, open the test project in your favorite IDE, navigate to the JUnit test classes, choose one of them, and run it as JUnit test.

Within Build Scripts

To run all or selected script test cases as part of your functional tests you need to make the corresponding JUnit classes available to your test framework, such as a junit Ant task.

Typically, it is sufficient to add the script test case wrapper classes to the list of classes to be run by JUnit. If you prefer to not have a wrapper class for each test script, you need to add the class com.xceptance.xlt.api.engine.scripting.ScriptTestCaseSuite to the list of JUnit classes. This class is a generic representative for a set of test scripts. To tell the suite class which scripts are to be executed, use the property com.xceptance.xlt.api.engine.scripting.ScriptTestCaseSuite.testCases and list the script names:

com.xceptance.xlt.api.engine.scripting.ScriptTestCaseSuite.testCases = TAuthor TVisitor

In any case, each test script will be executed once and the results will be part of the JUnit test report.

See the demo test suite in directory <xlt>/samples/testsuite-pebble for an example of how to configure Ant’s junit task to run script test cases.

As a Load Test

Performing load tests with your script test cases is quite simple. Register the JUnit classes with the XLT load-test framework and use them in your load tests as you would do with regular Java-based tests. Note that in this case, you need a JUnit wrapper around your test script. See the section Load Testing for more details.

Advanced Topics

File Generation

The figure below shows the different types of files generated by Script Developer when wrapper class generation is enabled. The files generated during export to Java are not shown.

XLT Script Developer File Generation XLT Script Developer File Generation

We will use the sample test case TAuthor (available in the demo application testsuite-pebble) as a reference to describe all of these types in detail.

Test Script (.xml)

TAuthor.xml

All the script files recorded by Script Developer are saved in an XML format following the naming convention Testcasename.xml. With reference to our sample test case, TAuthor.xml is the test-case script file which consists of all the commands and modules calls being used. The syntax is very simple and easy to understand. See below for an example:

<?xml version="1.0" encoding="UTF-8"?>
<test-case baseURL="http://localhost:8080/" xmlns="http://xlt.xceptance.com/xlt-script">
  <description>Tests the creation of new blog articles.</description>
  <action name="OpenStartPage"/>
  <command name="deleteCookie" target="JSESSIONID"/>
  <command name="open" value="/pebble/"/>
  <command name="assertTitle" value="My blog"/>
  <command name="assertText" target="id=blogName" value="My blog"/>
  <module name="Login">
    <parameter name="userName" value="username"/>
    <parameter name="password" value="password"/>
  </module>
  <module name="CreateNewArticle">
    <parameter name="title" value="${title}"/>
    <parameter name="subtitle" value="${subtitle}"/>
    <parameter name="excerpt" value="${excerpt}"/>
    <parameter name="body" value="${body}"/>
  </module>
  <module name="CreateNewArticle">
    <parameter name="title" value="Another title text"/>
    <parameter name="subtitle" value="Another subtitle text"/>
    <parameter name="excerpt" value="Another excerpt text."/>
    <parameter name="body" value="Another body text."/>
  </module>
  <module name="Logout"/>
</test-case>
Test Data (.xml)

TAuthor_Data.xml

Separating test data from script code by extracting the recorded values into data files is a very useful feature if you want to run your test more than once with multiple data sets. This concept is also known as a data-driven test. The naming convention being used is Testcasename _data.xml, so TAuthor_data.xml represents the data file of the TAuthor test case. A data file with two sets of user data could look like this:

<?xml version="1.0" encoding="utf-8"?>
<data-sets>
	<data-set>
 		<userName>fred</userName>
	 	<password>topsecret</password>
	 </data-set>
	 <data-set>
	 	<userName>wilma</userName>
	    <password>cantremember</password>
	 </data-set>
</data-sets>
</li>
JUnit Test Case Wrapper (.java)

TAuthor.java

This file is generated when you enable wrapper class generation for your test cases. This feature allows you to run your script files outside the browser via the XLT framework, which simulates a headless browser. This mode is suitable for unattended test-case execution, such as functional or load tests.

Custom Module (.java)

ComplexUserLoggedInCheck.java

If some special constructs cannot be expressed due to the basic script syntax, you can overcome that by creating your own custom modules and using them inside your scripts. Custom modules are implemented in Java. See Using Java Modules for an example.

Macros

When creating and running regression tests using the Script Developer, it is sometimes necessary to have unique data available to be able to run a test over and over again. For example an user account creation always needs a new email address, because the target system accepts each address only once.

For the purpose of generating time stamps and random strings the following macros are available:

Macros can be used in any command using the ${variable} notation. If you want to input a random or unique email address, you could use one of the following code lines as a value in the type command: 

${RANDOM.String(5)}@varmail.com

or

${NOW}@varmail.com

The resulting email addresses might look like: zghfu@varmail.com or 1295519733483@varmail.com

If you use one of these macros as a value for a module parameter, the random string or time stamp is created once for a test run and then can be used in several commands of the module with an identical value. For example, you can fill in a form with a random name, submit the form, and then validate the name on the confirmation page. This would only work if filling in the form, submitting it and the validation is all part of the same module. The name has to be defined as a module parameter and referenced in the relevant commands with ${name}, where ${RANDOM.String(8)} is the provided value for the parameter name.

Creating Java Modules

The commands supported by the Script Developer are sufficient to create working test scripts. However, some special constructs cannot be expressed due to the restricted script syntax, for example:

To overcome such limitations, custom modules can be created for test cases that are meant to run outside Script Developer (as a load test, as JUnit test in Eclipse, or integrated in a build process). The Java modules will be skipped when running the test case in the Script Developer.

Custom modules are written in Java by implementing the WebDriverCustomModule interface. This interface forces you to implement the necessary execute method and its method parameters. See below for an example:

public class ComplexUserLoggedInCheck implements WebDriverCustomModule
{
    public void execute(final WebDriver webDriver, final String... parameters)
    {
        final WebElement webElement = webDriver.findElement(By.xpath("id('sidebar')/div[1]/div[1]/span"));
        final String userName = parameters[0];

        Assert.assertTrue("Expected user name not found: " + userName, webElement.getText().contains(userName));
    }
}

Make sure the custom module class is compiled and made available on your test suite’s class path.

In order to be able to integrate Java custom modules in test scripts, you first need to register them with Script Developer. You do so by creating a new Java module script:

  1. Choose New | Java Module from the script library context menu. The Edit Module Details dialog will come up (see the figure below).
  2. Fill out the necessary fields as described below.
  3. Close the dialog.

The new Java module is now available in the module library and can be used in test scripts (test cases or modules) the same as any other script module but cannot be edited and will be skipped when running the test case inside Script Developer.

New Java Module New Java Module

If you defined module parameters you have to provide a value for each parameter when using the Java module in a test case. You access the first of these parameters in your Java code by reading the value from parameters[0], the second one by reading the value from parameters[1], and so on. The name of the parameters defined in the Script Developer Java Module script is not visible in the Java code; only the order of the parameters is relevant.

Note that Java modules should be used only if absolutely necessary, as their execution will be skipped when running the test case in Script Developer. If Script Developer comes across a Java module while replaying a script, it simply ignores it (the status icon turns gray) and continues with the next command. The next command might or might not succeed depending on what the module does with the page. If the module leaves the page unchanged (i.e. it performs advanced validations only), the rest of the script will run successfully. However, if the module changes the page or loads a new page, the commands following the module are likely to fail. This is why Java modules are mainly used for the purpose of complex validations.

Please tell us more about extensions you create, as we might include them in a future release.

XLT Framework

Basic Concepts

When performing web-based application tests the possible paths from page to page define the web page flow. The web page flow can be represented as a directed graph where the vertices are the web pages and the transitions represent the actions to get from one page to another. Typically, a test scenario covers a certain part of the page flow only, such as a specific path through the application.

XLT provides a programming paradigm that uses a three-level architecture (transactions, actions, and requests). These levels will be described in the following subsections.

Transaction

A transaction is an execution of exactly one test case or test scenario. In order to perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case, which itself executes a sequence of one or more actions.

Action

An action can be defined as one irreducible step within a test case. So an action interacts with the current page and – as a result – loads the next page. That page is associated with this action and becomes the current page for the next action in the test scenario. Generally, an action triggers one or more requests.

Request

This level is equivalent to the HTTP request level used in web browsers or in any other application that relies on HTTP communication. You do not have to deal with requests directly because they are automatically generated by the underlying HtmlUnit framework when performing actions on HTML elements.

Validation

Because we are testing the functionality of applications or pieces of software, we have to check the correctness of all responses. We strongly recommend that you handle all potential situations and use validations as often as possible. It is better to have too many checks rather than too few! They can’t do any harm, and will increase your confidence that your application works correctly. You should insert as much validation as necessary to detect any abnormal application behavior of the software being tested.

Pre-validation

Each action should have a pre-validation section that checks whether or not all of the required data is available to interact with that page and possible advance to the next one. From the end user’s point of view, we simply look for the information on the page that we need to continue our web experience, such as a form to fill in or a link to click. In case the required information cannot be found, an exception is thrown. If we run a load test, XLT will catch this exception and log all relevant information. This allows you to evaluate the results after running the test and narrow down error conditions.

Post-validation

Post-validations work similarly to pre-validations. They are used to validate the result of the interaction and ensure that the data matches the expectations.

Example

The following example shows a very simple scenario to understand the terminology better. It is based on the demo application (see Demo Application and Test Suite). Imagine a typical user visiting a blog. The user will possibly:

Test Case And Actions Test Case And Actions

In this example the test scenario is modeled as test case TBlogVisitor. A single execution of this test case is a single transaction. OpenHomePage, Paging, ViewArticleDetails, AddComment and ReturnToHomePage are the actions of this page flow to go from one page to the next. Validations after each page transition ensure that we arrived on the right page with the right content.

Available Programming Interfaces

Overview

XLT provides different approaches for writing test cases in Java. Several programming interfaces are available when using the XLT Framework. Built on each other, they represent different abstraction levels.

You can extend your test suite by using the XLT Scripting API. This allows you to start with Java code that was automatically generated from the recorded Script Developer test cases. This features a high-level command scripting API with a very intuitive syntax. XLT Scripting API is built on top of the WebDriver API, which can also be accessed directly to write more advanced tests.

XltDriver, as part of the XLT framework, is a WebDriver implementation extending the HtmlUnitDriver. Both in turn are built on the HtmlUnit API.. HtmlUnit is a headless browser that offers a low-level API that allows for full control when creating web tests.

To serve as the main framework when creating HtmlUnit based tests, XLT provides the XLT Action API which structures the code in action classes and test-case classes.

XLT Framework API XLT Framework API

XLT Test Cases are JUnit4 Tests

All XLT test cases use a Java test case class with one test() method, regardless of the chosen approach for writing tests, except for the pure Script Developer script test cases.

The test() method of each test case class has a @Test annotation (see TBlogVisitor code example line 12). XLT builds upon JUnit4 principles and its annotations to implement and tag test cases. This way each XLT test is in fact a JUnit test, so XLT tests can be executed just like any other unit test in the IDE or within an existing build process. The only difference between XLT and standard JUnit4 tests is that XLT tests can only have one active test method per test class. That means that although there can be an arbitrary number of methods within a class, only one method is permitted to be annotated with @Test. This limitation simplifies things and does not result in any actual limitations in real life.

Implementing the test case as a JUnit4 test also allows you to use standard JUnit assertion to validate the page, mainly when creating test cases by using the HtmlUnit API.

XLT Scripting API

When exporting a script developer test case to Java (see Export test case to Java) the resulting code can be based on the XLT scripting API (depending on the API selection during export). This API is an easy-to-use programming interface with a simple syntax based on the commands available in the XLT Script Developer.

Test cases can be written from scratch in Java using the XLT Scripting API. However, the most common way is to record a test case with Script Developer and extend the test case using the Scripting API after exporting it to Java.

The following screenshot shows what the test case TBlogVisitor introduced in the section above could look like if recorded with Script Developer. Note that this test case uses a very poor validation to keep the example short and simple. A real test should have more validations to be sure that the page is displayed correctly.

TBlogVisitor Demo Scipt Test Case TBlogVisitor Demo Scipt Test Case

Here is the automatically generated Java code of the TBlogVisitor test case after the export:

/**
 * Tests viewing articles and writing comments.
 */
public class TBlogVisitor extends AbstractWebDriverScriptTestCase
{

    /**
     * Constructor.
     */
    public TBlogVisitor()
    {
        super(new XltDriver(true), "http://localhost:8080");
    }


    /**
     * Executes the test.
     */
    @Test
    public void test() throws Throwable
    {

        //
        // ~~~ OpenHomePage ~~~
        //
        startAction("OpenHomePage");
        open("/pebble/");
        assertTitle("My blog");
        assertElementPresent("link=<< Previous");
        assertNotElementPresent("link=Next >>");

        //
        // ~~~ Paging ~~~
        //
        startAction("Paging");
        clickAndWait("link=<< Previous");
        assertElementPresent("link=<< Previous");
        assertElementPresent("link=Next >>");

        //
        // ~~~ Paging ~~~
        //
        startAction("Paging");
        clickAndWait("link=<< Previous");
        assertElementPresent("link=<< Previous");
        assertElementPresent("link=Next >>");

        //
        // ~~~ ViewArticleDetails ~~~
        //
        startAction("ViewArticleDetails");
        clickAndWait("//div[@id='content']/div[1]/h1/a");
        assertElementPresent("link=Send a TrackBack");
        assertElementPresent("//div[@id='content']/div/div[3]/a[1]");

        //
        // ~~~ AddComment ~~~
        //
        startAction("AddComment");
        click("//div[@id='content']/div/div[3]/a[1]");
        type("id=commentBody", "A comment");
        clickAndWait("//*[@id='commentForm']//input[@name='submit' and @type='submit' and @value='Add Comment']");
        assertElementPresent("//*[@id='content']//p");
        clickAndWait("name=submit value=Confirm");
        assertElementPresent("//*[@id='content']//div[contains(@class,'contentItemBody')]");

        //
        // ~~~ ReturnToHomePage ~~~
        //
        startAction("ReturnToHomePage");
        clickAndWait("link=Home");
        assertTitle("My blog");
        assertElementPresent("link=<< Previous");
        assertNotElementPresent("link=Next >>");

    }
}

You can see that the code is structured in blocks representing the actions. Each action starts with an startAction() command. The rest of the commands are very similar to the commands available in Script Developer. Each Script Developer command has a counterpart in the XLT scripting API. The generated test-case class extends the abstract class AbstractWebDriverScriptTestCase and inherits the methods that represent the scripting commands to interact with the page and perform validations.

See package com.xceptance.xlt.api.engine.scripting for more information about the available scripting commands of that API.

Script Modules

The Paging action in TBlogVisitor is implemented twice, which is not very efficient. Script Developer allows the user to extract parts of the script as a module. This would be a great idea for the Paging action.

When exporting the script to Java, XLT generates a class for the modules which extends AbstractWebDriverScriptModule and extracts the code to reuse it in the test case. Using the execute() method, the module can be called in the test case after creating an instance of the module class.

Modules can embed sub-modules. That means modules can be called by test cases or by other modules. This is why the module class needs two different constructors, one that takes a test case and one that takes a module.

This is the code for the extracted module class that can now easily be reused in any test case:


public class Paging extends AbstractWebDriverScriptModule
{
    //Constructor
    public Paging(final AbstractWebDriverScriptTestCase caller)
    {
        super(caller);
    }

    //Constructor
    public Paging(final AbstractWebDriverScriptModule caller)
    {
        super(caller);
    }

    /**
     * Executes this module.
     * 
     */
    public void execute()
    {
        //
        // ~~~ Paging ~~~
        //
        startAction("Paging");
        clickAndWait("link=<< Previous");
        assertElementPresent("link=<< Previous");
        assertElementPresent("link=Next >>");
    }
}

The resulting test case code instantiates the Paging module class and calls the execute() method twice to perform the paging actions:


		//
        // ~~~ Paging ~~~
        //
        final Paging _paging = new Paging(this);
        _paging.execute();
        _paging.execute();

WebDriver API

WebDriver is a tool for automated tests of web applications. It was originally introduced by Google. WebDriver features a simple and efficient API. The API permits control of real web browsers, such as Firefox, Internet Explorer, Safari, and the HtmlUnit headless browser.

The XLT framework integrates the WebDriver API, so external test cases using the WebDriver API can generally also run in the XLT framework. XLT-based WebDriver tests cannot be executed with a stand-alone WebDriver, because the XLT-WebDriver API integrates the concept of action names.

The Java code generated by XLT when exporting from the Script Developer is based on the XLT scripting API which is built on WebDriver. When you look at the TBlogVisitor example, you may notice that the strictly linear approach can show some limitations. Using the lower-level WebDriver API in combination with the XLT scripting API is a good way to overcome these limitations.

When running the TBlogVisitor example as functional test or load test the level of realism is low because the comments are always added to the same article. It would be nice to randomize the Paging action to browse to different pages and also to select the article randomly.

The exported code uses a XltDriver to simulate users. The WebDriver API also allows you to easily switch to other WebDrivers to simulate real-world browsers like Chrome, FirefoxDriver and Internet Explorer. The XltDriver is an extension of the HtmlUnitDriver implementation of the WebDriver API.

The following code shows an example using the WebDriver API to introduce random factors. To separate the different approaches in this example, the WebDriver functionality is re-factored to a method which also allows it to be reused for random paging and random article selection. It gives an example of a JUnit4 assertion and shows how you can use standard Java functionality like for loops to execute the paging several times.

Here is how the re-factored code described above could look:


/**
 * Tests viewing articles and adding a comment
 */
public class TBlogVisitor extends AbstractWebDriverScriptTestCase
{
    /**
     * Constructor.
     */
    public TBlogVisitor()
    {
        super(new XltDriver(true), "http://localhost:8080");
    }

    /**
    * This method randomly picks one of the web elements that match the given xpath
    * It uses the WebDriver API
    */ 
    private WebElement findWebElementsAndPickOne(String xpath)
    {
	// get all elements that match the given expression
	final List<WebElement> articleLinks = getWebDriver().findElements(By.xpath(xpath));
	
	//Make sure there is at least one element; This is a pure JUnit assertion
	Assert.assertFalse("No elements found", articleLinks.isEmpty());
	
	// grab one of the elements randomly
	WebElement element = articleLinks.get(XltRandom.nextInt(articleLinks.size()));
	
	//return the element
	return element;	
    }
    
    @Test
    public void test() throws Throwable
    {
        //
        // ~~~ OpenHomePage ~~~
        //
        startAction("OpenHomePage");
        open("/pebble/");
        assertTitle("My blog");
        assertElementPresent("link=<< Previous");
        assertNotElementPresent("link=Next >>");

        //
        // ~~~ Paging ~~~
        //
        //Loop the paging twice by using the extracted method implemented using the WebDriver API
        for (int i = 0; i < 2; i++)
		{
            startAction("Paging");
            clickAndWait("link=<< Previous");
            //Find the paging links by xpath, pick one randomly and click the link
            findWebElementsAndPickOne("id('linearNavigation')/a[. != 'Home']").click();
		}

        //
        // ~~~ ViewArticleDetails ~~~
        //
        startAction("ViewArticleDetails");
        clickAndWait("link=Another title text");
        //Find all article headlines by xpath, pick one of the articles randomly and click the link
        findWebElementsAndPickOne("id('content')/div[@class='contentItem published']/h1/a").click();
        assertElementPresent("link=Send a TrackBack");
        assertElementPresent("link=Add a comment");

        //
        // ~~~ AddComment ~~~
        //
        startAction("AddComment");
        //click("link=Add a comment");
        click("xpath=id('content')/x:div/x:div[2]/x:div/x:div/x:a[1]");
        type("id=commentBody", "A comment");
		clickAndWait("name=submit value=Add Comment");
        assertTextPresent("Please click the button to confirm.");
        clickAndWait("name=submit");

        //
        // ~~~ ReturnToHomePage ~~~
        //
        startAction("ReturnToHomePage");
        clickAndWait("link=Home");
        assertTitle("My blog");
        assertElementPresent("link=<< Previous");
        assertNotElementPresent("link=Next >>");
    }
}

The full documentation of the XLT framework with more information about the WebDriver support can be found in the file <XLT>/doc/apidoc.zip.

XLT Action API

In addition to the components already described, the XLT framework supports one more approach for modeling test scenarios in Java code. The test cases and action classes can be implemented in Java from scratch or they can be generated automatically during an export from XLT Script Developer. This approach is based on special action classes, so it is called the XLT Action API.

The foundation of XLT Action API for testing web-based applications is the HtmlUnit framework, which also includes the Mozilla Rhino engine for JavaScript support. Web tests are performed by a low-level web browser simulation. Basically this means that – as in real browsers – a web page is translated to a DOM (Document Object Model) tree. Analysis, validation, and any other access is performed afterward on the constructed DOM tree.

XLT Action API provides a programming paradigm to translate a test scenario into a unit test. The test scenario is implemented as a test-case class, which itself executes a sequence of one or more actions.

Test Case and Action Classes

All test-case classes should inherit the abstract class AbstractTestCase, which supplies some basic features like logging the test results to disk and easy access to properties.

Because a test case models a transaction and transactions rely on actions, defining the appropriate actions is the first step.

All actions must inherit the abstract class AbstractAction, which forces you to implement the three methods execute(), preValidate(), and postValidate(). As mentioned earlier, the preValidate() and postValidate() methods are used to perform validations before and after the execution of that action itself. Therefore the call sequence of an action generated by the XLT framework will always be:

This call sequence will be executed exactly once when the instance method run() of an action is called.

As you see, XLT Action API forces us to implement the validation methods and that is the whole purpose of testing: validating data. Therefore implementing the abstract validation methods in a non-trivial way (i.e. not leaving them empty) is strongly recommended. Otherwise you will sacrifice test quality.

Each of the three methods may throw an exception, which is always an indication of a problem. To check if an action can safely be executed, the abstract class AbstractAction provides a method called preValidateSafe(). This method internally calls preValidate() and catches any thrown exception. If no exception is thrown, preValidateSafe() returns true; otherwise it returns false. This helps you to determine if the prerequisites are fulfilled to continue the page flow in a certain direction. A simple example for that is the flow through a catalog with nested categories. Because you do not know the nesting level up-front, when you create a dynamic and random test execution, it might be necessary to call preValidateSafe() before trying to go to the next level of categories.

Note that AbstractAction does not offer any web support. Therefore, any web-based test should inherit the abstract class AbstractHtmlPageAction, which is a specialization of AbstractAction and offers support for web testing.

Validation

Assertion

JUnit provides the concept of assertions and we use this concept for all validations. Because XLT does not change JUnit in any way, you can use assertions just as you learned to do with JUnit.

Pre-validation

XLT offers two ways of using the preValidate() method. Any exception on the direct path will stop the test with an error message. In case we just want to check whether or not a requirement is fulfilled, we can call the preValidate in a safe way (by using preValidateSafe()), so that any exception will be caught and no error reported. Should we accidentally cause any Java exception different to AssertionException, such as NullPointerException or IndexOutOfBoundException, XLT will issue a warning, because the code might contain a problem from a programming point of view. Errors from the application being tested should always come up as assertion failures.

Post-validation

The postValidate() method works similarly to the preValidate() method. It is used to validate the page just loaded in execute() and ensures that the data matches the expectations. The full set of JUnit assertions is available.

The postValidate() method cannot be called explicitly, the framework does it. Additionally, error messages cannot be suppressed. If a page has different outcomes based on random data or states, you have to handle that explicitly in your validation code.

Validators

We strongly encourage you to write individual validation classes for easy reuse. If a certain check has to be done more than once, it becomes a candidate for a validator implementation. This simplifies the maintenance of the tests and makes them less error-prone, because copy-paste is one source of typical programming errors.

Some common validation routines are already covered by default validators, such as a HTTP response code, HTML end tag, and HTTP content length validation. See package com.xceptance.xlt.api.validators in the API documentation for more information about this topic.

Example

To explain the XLT Action API with an example, let’s imagine a web search engine. The most important action would be to “search”: to fill in the search phrase and click “Go”, “Search”, or something similar to load a list of results. The preconditions are the existence of the search input field and of an appropriate button labeled Search or Go. The execute() method should fill in the search phrase and click the button.

After the new page has been loaded, the result should be validated. This validation consists of general validation, performed by validators, and action-specific validation.

The resulting implementation of the search action would look like this:

public class Search extends AbstractHtmlPageAction
{
    // XPath to search input field
    private static final String searchInputFieldPath = "/id('search')/input";

    // XPath to search button
    private static final String searchButtonPath = "/input[@type='submit' and @name= 'Search']";

    // XPath to search result block
    private static final String searchResultPath = "/div[contains(text(), 'result')]";

    // Search input field HTML element
    private HtmlInput searchField;

    // Search button HTML element
    private HtmlInput searchButton;

    // search phrase
    private String searchPhrase;

    /**
     * Performs a new search.
     * @param prevAction
     *            Previous action.
     * @param searchPhrase
     *            Phrase to search for.
     */
    public Search(AbstractHtmlPageAction prevAction, String searchPhrase)
    {
        super(prevAction);
        this.searchPhrase = searchPhrase;
    }

    /**
     * Validation prior to execution.
     * @throws Exception
     *             if some of the required input elements couldn't be found.
     */
    public void preValidate() throws Exception
    {
        HtmlPage p = getPreviousAction().getHtmlPage();

        List result = p.getByXPath(searchInputFieldPath);
        Assert.assertEquals(1, result.size());

        searchField = (HtmlInput) result.get(0);
        Assert.assertNotNull(searchField);

        result = p.getByXPath(searchButtonPath);
        Assert.assertEquals(1, result.size());

        searchButton = (HtmlInput) result.get(0);
        Assert.assertNotNull(searchButton);
    }

    /**
     * Executes the search. Primarily this includes the input of the search
     * phrase and a click on the proper search button.
     * @throws Exception
     *             if some of the inputs have become invalid or setting the
     *             value attribute of the search input field has failed.
     */
    public void execute() throws Exception
    {
        // Fill in search phrase
        searchField.setAttribute("value", this.searchPhrase);
        // Click on 'Search'
        loadPageByClick(searchButton);
    }

    /**
     * Validation after search has become complete.
     * @throws Exception
     *             if no search result block element could be found or the
     *             search result block element doesn't contain the search
     *             phrase.
     */
    public void postValidate() throws Exception
    {
        HtmlPage p = getHtmlPage();

        // response code = 200?
        HttpResponseCodeValidator.getInstance().validate(p);
        // does the length match?
        ContentLengthValidator.getInstance().validate(p);
        // does the page end with </html>?
        HtmlEndTagValidator.getInstance().validate(p);

        List result = p.getByXPath(this.searchResultPath);
        Assert.assertEquals(1, result.size());

        HtmlElement el = (HtmlElement) result.get(0);
        Assert.assertNotNull(el);

        Assert.assertTrue(el.asText().contains(this.searchPhrase));
    }
}

Notice that the constructor of this class has two parameters. One of them is the search phrase that the action has to know about. The other parameter is the action performed previously. All actions that will be used in page flows have to provide a constructor with a parameter representing the previously performed action to enable a flow. Without passing the previous action, each action would be stand-alone and behave as if you had just opened a new web browser. Normally only the start action does that.

You’ll notice that the postValidate() method uses some of the predefined validators. XLT also offers a StandardValidator which performs the most common validations in one go. This includes:

Having the search action at hand, the implementation of a test case using this action is almost done. A very simple test case would be to repeatedly search for some phrases. These phrases can be stored in a data file and obtained using the XLT data provider mechanism.

public class TSearch extends AbstractTestCase
{
    // Container that holds all the search phrases
    private static CustomDataProvider phrases = null;

    @Before
    public void initialize() throws Exception
    {
        // Data container already initialized?
        if(phrases != null) return;
        // No. Go for it.
        phrases = new CustomDataProvider(
                        getProperty("searchphrases.filename", "phrases.txt"),
                        CustomDataProvider.DEFAULT);
    }

    @Test
    public void search() throws Throwable
    {
        // Start on Homepage.
        Startpage start = new Startpage();
        start.run();

        for (int i = 0; i < XltRandom.nextInt(10); i++)
        {
            // Take a random search phrase.
            String searchPhrase = phrases.getRandomRow(false);

            // Search.
            Search search = new Search(start,searchPhrase);
            search.run();
        }
    }
}

The above example also demonstrates the use of the XltRandom class, which offers some nice features regarding randomization. Note that the package com.xceptance.xlt.api.util provides more functionalities that can help implementing tests.

Each execution of the search action requires a proper search phrase, which is obtained from a CustomDataProvider object. This class provides a generic mechanism to handle and provide test data that is stored in a text file. The name of the text file, along with a Boolean value that tells the appropriate parser whether whitespace should be removed, are passed as parameters to the constructor. When the class is instantiated all data is kept in memory, allowing easy and fast access. XLT is shipped with a predefined set of data files which contain email addresses, first and last names, street and city names, and so on. This data can be obtained from the GeneralDataProvider class which uses the appropriate text files located in directory <testsuite>/config/data where <testsuite> refers to your test project directory.

The example also demonstrates the use of JUnit4 annotations, which can be used in the standard manner.

Data-driven Test

Sometimes a certain test case should be executed not just once, but multiple times, each time with a different set of test data. For example, to check not only the “happy path”, but also some border cases. You might even want to specify multiple test data sets, which are automatically recognized by the test framework and used to execute the test case once per specified data set. This concept is also known as data-driven test. Note that the test case executions are independent from each other and each produces a separate result in the test report.

XLT supports data-driven tests for any kind of test case (plain Java, Ruby, and scripts), but only when running them as a part of a functional test using a JUnit test runner. Data-driven test can not be used when replaying script test cases in the Script Developer. Also, for load tests, there are other parameters that define how long/often a test will be executed, such as measurement time and arrival rate.

Typically, test data can be classified as constant or variable test data. Constant test data is fixed for all runs of a test case in a data-driven test. Constant test data is either hard-coded into the test case or kept separate from the test code in a data file. Variable test data is different for each run of the test case, so variable test data is organized as a list of separate data sets. Each data set contains all variable test data needed for exactly one test run. The number of data sets determines the number of test runs.

But where do these data sets come from? For a data-driven test, XLT retrieves test data sets from an additional source. This could be, for example, another data file or a database. XLT accesses a data source via data set providers, which implement a uniform interface and return a data set as a simple key/value map.

During test execution, the framework reads the next data set from the configured data set provider and passes it to the test case instance and runs the test. The test instance is responsible to apply the test data appropriately. This process is repeated until all available variable data sets have been processed.

Data Set Providers

Built-In Providers

XLT supports three common sources for test data sets out of the box, which will be discussed in this section.

XML Files

Data sets can be stored in an XML data file with a three-level element structure. There is a single top-level element (as XML mandates it). The elements on 2nd level define the data sets, while the elements on 3rd level define the values. A data file with two sets of user data might look like this:

<?xml version="1.0" encoding="utf-8"?>
<data-sets>
	<data-set>
		<userName>fred</userName>
		<password>topsecret</password>
	</data-set>
	<data-set>
		<userName>wilma</userName>
		<password>cantremember</password>
	</data-set>
</data-sets>

Note that you can name the root element (here: data-sets) and the 2nd-level elements (here: data-set) as you like since the structure is important only. However, the tag names on the 3rd level always defines the parameter names, so these tag names must be used consistent across all data sets.

CSV Files

The data sets are stored in a data file, organized as lines of separated values. The values in the first line are defining the parameter names while the values of all following lines are defining the values of each data set. The previous example will now like this:

userName,password
fred,topsecret
wilma,cantremember

The separator used in the csv file (comma or semicolon are often used) can be configured using a XLT property. For example:

## Sets the field separator character for CSV files (defaults to ",").   
com.xceptance.xlt.data.dataSetProviders.csv.separator = ;

Note that there is no way to specify character encoding information in a CSV file. By default, XLT reads CSV files using UTF-8. To override this default use the following property:

com.xceptance.xlt.data.dataSetProviders.csv.encoding = ISO-8859-1
JDBC Data Sources

For JDBC data sources, there is also a “data file”, but this time the file does not specify the data sets directly, but contains an SQL query, which retrieves the data sets when executed. A query file for user data sets could look like this:

select login as "userName", password as "password" from users;

Each row returned from the database is converted to one data set. The alias names in the query define the resulting parameter names.

Note that the SQL data set provider needs some additional setup before it can be used. First, an appropriate JDBC driver needs to be present on the class path of your test suite. Simply copy the respective JAR file to <testsuite>/lib and you are done. Second, you need to configure the URL and credentials of your JDBC database connection that should be used when executing the query:

com.xceptance.xlt.data.dataSetProviders.jdbc.url = jdbc:h2:tcp://localhost/test
com.xceptance.xlt.data.dataSetProviders.jdbc.userName = sa
com.xceptance.xlt.data.dataSetProviders.jdbc.password = YourPassword

Custom Data Set Providers

If the built-in data set providers are not sufficient, you can write your own. Your custom data set provider must implement the general DataSetProvider interface: 

public interface DataSetProvider
{
    public List<Map<String, String>> getAllDataSets(File dataFile) throws DataSetProviderException;
}

To register your provider implementation with XLT, you just need to add a property to your configuration:

com.xceptance.xlt.data.dataSetProviders.foo = com.yourcompany.FooDataSetProvider

This will tell XLT to use the class com.yourcompany.FooDataSetProvider for data files with the extension .foo. Note that this way you can also override the built-in providers.

Test Data Set File Lookup

As you have learned in the previous section, there is always some kind of “data” file involved. Typically, these data files are named after the test case (the script name for script-based test cases, or the simple class name for Java-based test cases). For example, if a test script is named TAuthor, then the XLT framework will automatically look for files like TAuthor_datasets.<ext>, where <ext> is one of the file extensions, for which a data set provider has been registered. So the resulting list of file name candidates is: TAuthor_datasets.csv, TAuthor_datasets.xml, and TAuthor_datasets.sql (and TAuthor_datasets.foo, of course, if configured). The lookup order is:

  1. .csv
  2. .xml
  3. .sql
  4. (.foo)

However, test data files can also have arbitrary names or paths. In this case, you have to configure, which data file belongs to which test case. You do so by mapping the data set file name (or path) to the test case’s Java class using the following notation:

<class_name>.dataSetFile = <data_set_file_path>

For example:

com.mycompany.xlt.tests.MyTest1.dataSetsFile = Test1.xml
com.mycompany.xlt.tests.MyTest2.dataSetsFile = ./subdir/Test2.csv
com.mycompany.xlt.tests.MyTest3.dataSetsFile = c:/tmp/Test3.sql

Please keep in mind, that all property files are Java-style property files. So, when you use backslashes on Windows, you have to quote it with another backslash, e.g. c:\\tmp\\Test3.sql.

Whether or not the test data file name is given explicitly or has been derived from the test case name, the XLT framework looks for such a file in several locations:

  1. in the current directory (typically <testsuite>)
  2. in the directory specified by the property com.xceptance.xlt.data.dataSets.dir. This property is set to ./config/data) by default but is disabled as a comment. Enable this line by deleting the hash character when you want to use this location.
  3. in the default script test case directory <testsuite>/scripts (for script-based tests only)

As soon as a suitable file has been found, the lookup is stopped.

Note that if the test case has a qualified name, i.e. it has a package part, the data set file also needs to have that package to be found. So, if there is test case named your.package.TAuthor (be it a script or a Java test case), the framework will search the aforementioned directories for a corresponding data set file using the file path <dir>/your/package/TAuthor_datasets.<ext> or, alternatively, using the file name <dir>/your.package.TAuthor_datasets.<ext>.

Executing Data-Driven Tests via JUnit

Running data-driven tests via JUnit is not different in any way from running normal test cases. You simply add the test cases in question (directly or indirectly) to the list of classes to be run by JUnit. If XLT finds a data set file for a certain test case, it will be passed to the right data set provider, which returns all data sets. Then, the test case will be executed once for each data set. This works, because all test cases extend the class AbstractTestCase, and this class has all the magic built-in. The same is true for the generic script test case suite class ScriptTestCaseSuite.

See the demo test suite in directory <xlt>/samples/testsuite-pebble for an example how to configure Ant’s junit task to run XLT test cases.

If you want to temporarily disable a data-driven test and let the test cases run only once even though there are data set files, just configure the following setting in default.properties:

com.xceptance.xlt.data.dataDrivenTests.enabled = false

But note that your test case must provide some default test data in this case.

Demo Application and Test Suite

XLT ships with a real-world demo web application as the system under test (called “Pebble”) and a test suite to test this application. Both can be found in the directory <XLT>/samples.

Pebble is blog software written in Java and is under a BSD license. Because Pebble is small and easy to deploy, it is a perfect fit for our test application. Thanks to Simon Brown for this excellent piece of software and licensing it as Open Source.

Please do not use the provided installation for purposes other than testing, because it has been modified to run without problems from the sample directory. Feel free to download the latest version.

Running Pebble

To start the demo application, use the following command:

cd <XLT>/samples/app-server/bin
./start.sh

Windows users have to use the appropriate .cmd file which is located in the same directory, by entering start.cmd in your command prompt.

This will start an application server which contains the Pebble application. To access Pebble, simply open a browser with this URL: http://localhost:8080/pebble/. Please take some time to become familiar with Pebble.

Importing the Pebble Test Suite into Eclipse

After starting Eclipse and creating a workspace, if you have not already done so, you have to import the sample test suite that is part of XLT. Open the import dialog in Eclipse and point to the samples directory in your XLT installation directory (File > Import > General > Existing Project Into Workspace).

  1. Select the root directory to search in, and point to <XLT directory>/samples.
  2. Select the test suite project from the list.
  3. Click Finish and you are done.

Because the imported project has dependencies to the libraries of XLT, you have to adjust these dependencies.

  1. Simply right-click on your project and select Properties.
  2. Choose Java Build Path > Projects.
  3. Remove the required project xlt, if there is such a project configured.
  4. Select Libraries and click Add External JARs (if there are already other libraries besides the “JRE System Library”, remove them first).
  5. Go to <XLT>/lib and select all JARs. Then click Open.
  6. A list of all these JARs should be visible now. Close the dialog with OK.

Eclipse will rebuild the project now and will not report any build problems if configured properly.

Users of other IDEs have to carry out similar steps. Please be careful not to delete the JRE library as this may cause problems building the project.

The Directory Structure of the Test Suite

An XLT test project has a simple directory structure. The following directories have to exist to make everything run smoothly:

Understanding the Test Scenarios

Pebble is a blog software and so the test scenarios have to cover typical use cases of blogs:

Note that both scenarios share some common steps, allowing us to demonstrate the re-use of code across test cases.

Run the Tests

In order to run sample test cases of the demo application pebble, you can either use the Script Developer or import them into Eclipse and run them as JUnit tests. We will cover both of the approaches here.

Before running a sample test case make sure that Pebble is running.

Using Script Developer

Open the XLT Script Developer and open the configuration dialog by clicking XLT Script Developer|Settings and configure the settings appropriately:

All imported test cases and modules are visible in the script library. Double-click the TAuthor test case to see the list of commands and modules inside the work area.

Click the Play icon to start replaying your scripts inside Firefox.

Running Tests in Eclipse

Any test case can be run directly in Eclipse using the headless-browser mode. Go to package com.xceptance.xlt.samples.tests, select the test case class (e.g. TAuthor) and run it as JUnit test via the Eclipse class file context menu.

Writing Web Tests

The following sections will guide you through the essential steps when you want to write the first web tests on your own using XLT. The sections provide useful information that helps getting you started and helps to make essential the decisions regarding your test suite.

We encourage you to study the demo test suite carefully with respect to the approach as well as the code. Since all web tests are similar in structure, you will soon get a feeling for how to write your own tests. The sample test suite can serve as a template for your own test projects.

Creating a New Test Suite

In the XLT installation directory, you will find the demo test project “testsuite-pebble” which is mentioned above. To avoid creating the mandatory directory structure and configuration files from scratch, just copy this existing test project and strip it.

Copy the complete testsuite-pebble directory to a location of your choice and rename it, e.g. c:\test-suites\testsuite-MySite.

It is recommended to store your test suite in a directory that is not a sub-directory of your XLT installation. This makes updating to newer XLT versions much easier as it reduces the risk of overwriting your test cases during an update..

testsuite> will refer to this directory in the next sections.

This is the required directory structure of XLT test suites. It cannot be changed or altered.

Now it’s time to get rid of the leftovers from the testsuite-pebble project. You can delete the complete source code of the demo tests as well as the classes by emptying the following directories.

You should by no means delete the configuration files in <testsuite>/config as well as the files in the <testsuite> root. These files are mandatory to run the XLT test cases and must stay in these directories. Some configurations in these files must be changed to fit your project, especially when you plan to run load tests. You can delete the pebble specific properties here. You will find details for all of the properties and settings as comments in the files itself.

Before importing your project into Eclipse you have to edit the name element in the XML file <yourProject>/.project.

<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>testsuite-YourSite</name>
<>..<>
<>..<>
</projectDescription>

Choosing a Suitable Approach

As described in detail in the XLT Framework section of this document there is a number of different approaches available when you start writing your own test cases. The first step is to choose the suitable approach and API. To make the right decision you should answer yourself a couple of questions regarding your web test project:

Stand-Alone Script Developer Test Suite

Starting to record your tests using the XLT Script Developer and possibly continuing with just the Script developer and no Java code at all is a good option if...

The test cases can be recorded and manually replayed in the XLT Script Developer. There is no need to write any Java code and there is no need to use any tool or IDE different from the Script Developer.

Executing Plain Script Test Cases Outside Script Developer

When you need to run the tests outside Script Developer, e.g. in Eclipse or during an build process, then XLT Script Developer is the best starting point. You just have to enable generating Java wrapper classes for the script test cases. You can also run a load test with these automatically generated wrapper classes as long as you’re satisfied with the limitations listed above. Still you don’t need to write any Java code. Everything is based on the easy to use Script Developer commands and Firefox context menu validations.

As an additional benefit you can run data-driven tests and you can also simulate different browsers when running the script test cases outside the XLT Script Developer.

Script Developer Export, XLT Scripting API and WebDriver

If you need to overcome the limitations of the script test cases by switching to Java as the programming language then the Script Developer again is the best starting point because it’s fast and easy to create working prototypes for your test cases and then export these scripts to Java. The result is a test case class using the XLT Scripting API that can easily be expanded by using the WebDriver API. Another option is an export with resulting Java code based on XLT Action API. The TBlogVisitor test case introduced in the XLTFramework section is a good example for that approach with XLT Scripting API as export result. This approach is maybe the best choice if...

WebDriver from Scratch

To write your test cases you can also use the WebDriver API directly in the XLT context without using the XLT Scripting API abstraction level. This is a common approach for advanced users and this is a good choice if...

XLT Action API and HtmlUnit

Higher level APIs have a certain overhead that can influence the performance of the executed tests in comparison to a low level API. In some situations it makes sense to skip the higher level APIs listed above and program the test directly using HtmlUnit, for example when running load tests where heavy network traffic is required. For this case the XLT Action API serves as a framework to provide support for the action concept and validations and also helps to structure your code. You don’t even have to waive the Script Developer recording feature since exporting the script test cases to Java also allows to generate code based on the XLT Action API. The XLT Action API is the approach you should choose if...

If you decide to use the XLT Action API as the surrounding framework, then you will have to use the HtmlUnit API to code your tests. The other way round this is not true. You still would be able to go down to the HtmlUnit level for sections of your test cases, also if you decided to use one of the other approaches (e.g. WebDriver).

XLT Lightweight Mode

You should consider to choose the so called XLT Lightweight Mode to code your test cases if...

XLT provides the Lightweight Mode to code highest performance test cases. This maximum performance can be reached by omitting the creation of a DOM tree and JavaScript completely. Responses are available as html source and the test cases have to be coded on this level. One of the consequences is that some regular expression knowledge is required to identify elements and perform validations. Also the test cases become more complex and you will have to expect an increased programming effort.

Structuring Your Test Suite and Test Cases

Structuring in Script Developer

Naming and Tags

The easiest way to bring some structure into your test suite is to give your test cases a common naming convention. A common convention is to start the name with a capitalized T and then go on with the so called “camel case” with each element’s initial letter capitalized within the compound word. A best-practice example is to use the name’s elements to specify the test case’s purpose, e.g. TCartCheckoutCancel, TCartOrder and so on.

XLT Script Developer provides the possibility to tag test cases and modules. Each test case can have several tags. Tags help to group several test cases and make them easier to find. The list of available test cases can be filtered by tag and name.

Script Packages

Script Developer allows you to define packages to structure your test cases and modules in the script library. You can use this feature to bundle test cases for similar purposes. The packages can be created with a hierarchical structure but they are always displayed in a flat representation (e.g. testcases.cart.order).

Script Modules to Structure Test Cases

To structure a single script test case you should use modules. A module is a sequence of script commands and can be re-used in several test cases and can also call other modules, which means they can be nested. Using modules will prevent you from writing or recording the same sequence of commands again and again. But it also helps to keep the test case clear and understandable as a module call can be folded to hide the contained commands and show only the module’s name.

Action Commands

Script Developer inserts so-called Actions automatically while recording. You can also insert a new action manually at any position of your test case. Even though actions look similar to comments or general structuring elements they are not to be used to structure your scripts visually. Please read the action-related notes in the Script Developer section of this document.

Java Code Structuring

If you decide to choose an approach that requires to write your test cases in Java, you will have the same possibilities to structure your code as in any other Java program or software development project. Feel free to extract sequences of your test case code to methods, create Java classes and use packages to structure your test suite.

Package Suggestions

The <testsuite>/src directory contains subfolders with the structure of your Java packages as normal. Your source code should be organized in main packages. Typically you create one package for test cases, one for actions, one for flows, one for validators and one for utility classes.
A typical directory structure after you created your packages could be:

If you plan to use more than one of the approaches provided by XLT it is recommended to create packages for each of its test cases. This would result in up to three additional sub directories. You can use the following names as a suggestion for your own packages:

Beside these general possibilities, each of the approaches introduces XLT specific framework conditions that will give basic structure to your test suite and test cases. In particular, each test case is necessarily implemented as a Java class that extends a XLT test case class which is specific for the chosen approach and contains one method annotated with @Test. These specific framework conditions and further possibilities are explained in the following sections.

Structuring Scripting API and WebDriver Test Cases

When you export a script test case from Script Developer to Java using XLT Scripting API then it will be converted into a test case class which extends AbstractWebDriverScriptTescase. Modules are converted to classes extending AbstractWebDriverScriptModule. The test case class and all module classes are created automatically and you normally don’t have to deal with the creation of these classes.

If you decided to write the test cases from scratch using WebDriver API then your test case class should extend AbstractTestCase. Like all test case classes, this class can have several methods but exactly one method has to be annotated with @Test. This class normally contains the statements and lines of code that define the basic structure of test case, i.e. the page flow.

For both approaches, Scripting API and WebDriver API, it is recommended to structure the page flow by actions. This is important especially if you plan to run a load test because the load test reports are designed for analysis and evaluations based on XLT actions.

Scripting API offers a very simple command to start a new action: startAction("MyNewAction"). When using pure WebDriver API the following line of code can be used to start a new action Session.getCurrent().setWebDriverActionName("MyNewAction").

Again, actions should always be used with the basic concepts in mind. This means they should be used only to represent the page flow.

Structuring XLT Action Based Test Cases

As the name implies, test cases are closely related to actions when using the XLT Action API. Like all other approaches, an action interacts with the current page and as a result, loads the next page. That page is associated with this action and becomes the current page for the next action in the test case. But in contrast to previous approaches an action is implemented as a Java class extending AbstractHtmlPageAction. These XLT action classes can be seen as reusable building blocks to write your test case and define the page flow.

More information about how the API forces you to structure code and validations with methods you have to implement can be found in the XLT Action API section of this document.

Creating a Flow

When creating XLT test cases sometimes you may want to reuse blocks of code that contain more than a single action. As with modules, you can create an own class with one method that combines a sequence of several XLT actions as a flow. Different test cases can call this method now to reuse the flow. This is a concept for code structuring that can be implemented if needed, but there is no explicit support available or necessary in the XLT framework when creating a flow manually.

Flows will only be created automatically when exporting script modules to XLT Action API containing more than one action.

Test Suite and Framework Configuration

When using just the Script Developer for recording/writing and replaying script test cases, the following section does not apply. But configuring the test suite and framework by changing properties files might be necessary if you write or run test cases from inside Eclipse as JUnit tests or if you want to perform load tests.

To configure the test environment and test suite XLT uses Java properties files. Therefore the basic characteristics and syntax of that format apply also to the XLT properties files.

When reading the properties XLT distinguishes between the load test mode and development mode. As the name implies the load test mode is active when test cases are executed by the XLT master controller/agent controller as load tests. When test cases are executed as JUnit tests in Eclipse or any other JUnit test runner they run in development mode. Even though development mode is used mainly for developing the test cases it is also active if your test suite’s goal is an automated functional test that will be triggered manually from time to time or is integrated in a build process.

XLT uses a hierarchical file system so that properties can be distributed to several files with different priorities. Properties from different files complement each other. Furthermore, properties from a file with higher priority can overwrite identical properties from a file with a lower priority. With this mechanism general default values can be specialized for different test run scenarios or projects. Additionally it is possible to prepare several configurations in different files and activate one of these configurations by switching between the files.

All properties are read from the <testsuite>/config/ directory. The existing properties files are listed below sorted by priority from lowest to highest. See the following subsections for details.

Default Framework Configuration – default.properties

The properties in default.properties are general XLT framework settings for their respective default values. They are not specific for a single test project. If you need to change one of the properties listed in this file you should copy the considered property to the project.properties or test.properties and change the value there to overwrite the the value in default.properties. Overwriting the values is always possible because default.properties has the lowest priority. If the same property exists in one of the other files, the default value will be overwritten.

When updating XLT to a newer version the default.properties file should also be updated because newly available properties can be found here with their default value and description.
Even though it is not, this file should be treated as read-only. You can use this file as a documentation of available XLT framework properties which is also defining the default values for these properties.

The default.properties file contains properties in the following groups (see the file itself for details):

Test Project Configuration – project.properties

The file project.properties contains project specific settings. The first and most important property is the reference to the test.properties file to be applied (e.g. com.xceptance.xlt.testPropertiesFile = test-1.properties). By changing the value for this property you can easily switch between several different load test profiles configurations.

By default, it also contains the test case mapping that maps the test case class onto a load test name. The load test name will be referenced later in the load test configuration.

This file is also the best place for all your test case specific custom properties, e.g. urls, login information, search phrases or any other data you want to extract from your test cases as properties. The demo test suite pebble gives several examples how to use properties in the test cases.

Load Test Profile Configuration – test.properties

All settings to configure a specific load test profile are collected in a separate file. See load test chapter or the properties file itself for details of the available load test settings.

The default name for this file is test.properties. However, this name is variable and several files with different load test profile configurations can exist. The one file applied for a test run is referenced by a property in project.properties, which is mentioned in the section above.

Development Environment Configuration – dev.properties

The file dev.properties contains development mode settings. Use this file to modify the configuration such that it better suits your needs during development of test cases, i.e. when you create and debug the test cases from within your IDE.

This file is read in development mode only, but not during load testing. For development mode the values in this file have highest priority. Any setting defined here will overwrite the corresponding setting from the other properties files: “default.properties”, “project.properties”, and the test run specific properties file, e.g. “test.properties”.

A typical example for differing development settings is com.xceptance.xlt.loadStaticContent = true to enable loading images and other static content in development mode for debugging. Default value ( =false ) for this property is switching off loading static content to save resources during load test.

If the default values are sufficient as development settings for your test suite, the dev.properties file can also be empty.

Additional Configuration Files

In addition to the already described files there a three more files in <testsuite>/config/:

For more information about log4j settings also see Apache Log4j API Docs

Property Replacements

In all XLT properties files you can work with property replacements based on a ${} syntax. You can define a property and then assign a value to another property by referring to the first property.

This is a helpful feature especially for project.properties where properties are often defined for each test case to gain flexibility but e.g. the login data is identical for all test cases by default.

username = MyUsername
password = MySecretPassword
com.xceptance.xlt.samples.tests.TAuthor.username = ${username}
com.xceptance.xlt.samples.tests.TAuthor.password = ${password}
com.xceptance.xlt.samples.tests.webdriver.TAuthor.username = ${username}
com.xceptance.xlt.samples.tests.webdriver.TAuthor.password = ${password}

Load Testing

Typically, a distributed load generation environment is needed to generate enough load. For this we need a cluster of test machines. Install XLT on all these load machines.

Load generation environment Load generation environment

Load Test Environment Configuration

Before you can start the load test, some configuration work needs to be done. The configuration of XLT load generation environment itself is discussed in the next section, followed by an explanation of the configuration of your test suite.

The below mentioned property files are used to configure the main components of the XLT load generation runtime:

Agent Controller Configuration

Inside agent controller configuration file you can define the following properties.

Port Number

Port number, the agent controller is listening on. Default is 8500. You can pick any free port number, but make sure that the corresponding master controller entry matches that number. Also ensure, that the firewall rules in place allow unrestricted communication. The used protocol is https. If you would like to run more than one agent controller per machine, make sure that all controllers use different port numbers.

com.xceptance.xlt.agentcontroller.port = <portnumber>
Key Store Credentials

The credentials, your key store is encrypted with. You only have to change this, if your Java key store password has been changed from the default.

com.xceptance.xlt.agentcontroller.keystore.password = <password>
com.xceptance.xlt.agentcontroller.keystore.key.password = <password>
Agent Controller Logging

For the configuration of the agent controller logging facility, you can adjust these properties. These settings only affect the agent controller output and do not change the logging of your test code. Most of the time, a change here is not required.

log4j.rootLogger = info, console, file
log4j.appender.console = org.apache.log4j.ConsoleAppender
log4j.appender.console.layout = org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern = [%d{HH:mm:ss,SSS}] %-5p [%t] - %m%n ..............

For more information about log4j settings also see Apache Log4j API Docs

Master Controller Configuration

The second configuration file contains the properties of the master controller.

Test Suite Location

To determine which test suite should be used for the load test, you have to specify the location of the test suite relative to your XLT installation. It will be uploaded to the agent controllers from there.

com.xceptance.xlt.mastercontroller.testSuitePath = <location>
e.g. com.xceptance.xlt.mastercontroller.testSuitePath = samples/testsuite-pebble

When running on and from Windows, make sure that you use the correct encoding for backslashes, because property file format uses \ to quote other special characters, so you have to quote the \ with a \ to ensure its original meaning, e.g. c:\\test\\mysuite.

Depending on the operating system you are using, it can be necessary to specify an absolute path to the test suite instead of a relative one.

Update Interval

Defines how often the master controller updates the status of currently running load test.

com.xceptance.xlt.mastercontroller.ui.status.updateInterval = <time in seconds>
Status Display

Whether to display detailed status information for each simulated test user or not. If set to false, status information will be aggregated into one line per user type and vice versa. If you have a lot of test users running, it can be helpful to set this to false, otherwise you might become overwhelmed by the amount of information presented. This property will not change the data collection and final data presentation. This is a display property only.

com.xceptance.xlt.mastercontroller.ui.status.detailedList = <true/false>
Agent Controller Locations

This property lists the locations of the agent controllers, the master controller should use. 

com.xceptance.xlt.mastercontroller.agentcontrollers.<id>.url = <url> 
com.xceptance.xlt.mastercontroller.agentcontrollers.<id>.weight = <weight>

You can use any name for the <id> part of the property. Recommended are name and number combinations, such as ac1 for the first agent controller or blade01-02 for the second agent controller on the first blade. Make sure that the agent controller IDs differ from each other, otherwise a later entry in the file will overwrite the previous one.

In order to use load machines of different power together in a load cluster, you may specify a “weight” for each agent controller (defaults to 1 if not set). This value influences the automatic distribution of virtual users across the load machines. A machine with a weight of 3 will get 3 times the load of a machine with a weight of 1. 

com.xceptance.xlt.mastercontroller.agentcontrollers.ac1.url = https://localhost:8500
com.xceptance.xlt.mastercontroller.agentcontrollers.ac1.weight = 1
com.xceptance.xlt.mastercontroller.agentcontrollers.ac2.url = https://localhost:8501
com.xceptance.xlt.mastercontroller.agentcontrollers.ac2.weight = 3
Master Controller Logging

You can set a different logging behavior for the master controller. This can help to solve problems and also provides information in case of support inquiries.

log4j.rootLogger = debug, file
log4j.appender.console = org.apache.log4j.ConsoleAppender
log4j.appender.console.layout = org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern = [%d{HH:mm:ss,SSS}] %-5p [%t] - %m%n ......

Test Suite Configuration

The test suite itself is configured independent of the master controller. All properties are read from the <test-suite>/config directory. All files and most important properties are explained in detail in the Test Suite and Framework Configuration section of this document.

This section will discuss only the settings relevant to load testing.

Default Configuration – default.properties

Result directory location

Specifies the directory location where you want to store load test results. Normally it is not necessary to change this. 

com.xceptance.xlt.result-dir = <directory path>
Error Behavior

Specifies the framework behavior in case of an error – whether the framework should abort a transaction if any of the following error occurs:

com.xceptance.xlt.stopTestOnHttpErrors.page = <true/false>
com.xceptance.xlt.stopTestOnHttpErrors.embedded = <true/false>
com.xceptance.xlt.stopTestOnJavaScriptErrors = <true/false>
com.xceptance.xlt.maxErrors = <number of errors per agent controller>
Think Times

To specify the think time between two subsequent actions or transactions, use these properties. If a random think time is needed, set the deviation to a value greater than 0. It specifies the maximum deviation from think time in milliseconds. The respective value is added or subtracted from think time is generated with a pseudo-random, uniform distribution.

com.xceptance.xlt.thinktime.action = <time in [ms]>
com.xceptance.xlt.thinktime.action.deviation = <time in [ms]>
com.xceptance.xlt.thinktime.transaction = <time in [ms]>
com.xceptance.xlt.thinktime.transaction.deviation = <time in [ms]>

For example :

com.xceptance.xlt.thinktime.action = 100
com.xceptance.xlt.thinktime.action.deviation = 50
com.xceptance.xlt.thinktime.transaction = 0
com.xceptance.xlt.thinktime.transaction.deviation = 0

This will set the action think times between 50 and 150 ms and no transaction think time at all.

The deviation has to be smaller than the specified base think time.

Test Project Configuration – project.properties

To configure your test project you’ll have to edit the file named project.properties.

Test Properties File

XLT permits to prepare and use multiple test.properties files for easy maintenance of test setups. This makes switching between test setups easier and avoids configuration errors. This property does not allow the use of a path-specific file name. The test definition files reside in the same directory as the project.properties file. 

com.xceptance.xlt.testPropertiesFile = <filename>.properties
Test Class Mapping

test case mapping – specify Java classes mapped to the appropriate test IDs (fully qualified names). Note you can map the same class to multiple load test names if needed.

com.xceptance.xlt.loadtests.<name>.class = <fully qualified class name>

For example:

com.xceptance.xlt.loadtests.TVisitor.class = com.xceptance.xlt.samples.tests.TVisitor
com.xceptance.xlt.loadtests.TJSVisitor.class = com.xceptance.xlt.samples.tests.TJSVisitor
Test Class Specific Settings

Project specific settings – you can also define any test case specific settings as well, using the following syntax. 

<fully qualified name>.<property-name> = <value>

For example:

com.xceptance.xlt.samples.tests.blog-url = http://localhost:8080/pebble/
com.xceptance.xlt.samples.tests.TAuthor.username = username
com.xceptance.xlt.samples.tests.TAuthor.password = password
com.xceptance.xlt.samples.tests.webdriver.TAuthor.write-count = 2

Load Test Profile Configuration – test.properties

Test-run-specific settings – you can also configure an (optional) property file which contains the settings specific to a certain load test run. You can define more than one test property file, such as test-target-load.properties and test-2x-target-load. This way, many configurations can be defined and prepared in advance and used as needed. You switch between these files by changing com.xceptance.xlt.testPropertiesFile in the project.properties file.

Load test profile configurations are done inside test.properties file, where you define the test case name, number of virtual users and all other load test specific settings, which are to be run in parallel agents using the following syntax. 

com.xceptance.xlt.loadtests.<testID>.<setting> = <value>

For <testID> use any proper name. The supported values for <setting> are:

For example:

com.xceptance.xlt.loadtests = TAuthor
com.xceptance.xlt.loadtests.TAuthor.users = 5
com.xceptance.xlt.loadtests.TAuthor.iterations = 100
com.xceptance.xlt.loadtests.TAuthor.arrivalRate = 3600
com.xceptance.xlt.loadtests.TAuthor.initialDelay = 0
com.xceptance.xlt.loadtests.TAuthor.warmUpPeriod = 30s
com.xceptance.xlt.loadtests.TAuthor.measurementPeriod = 10m 0s

All time period values can be specified in one of the following formats (without the quotes):

If you want to run several test cases simultaneously, add the test case names to com.xceptance.xlt.loadtests seperated by space:

For example:

com.xceptance.xlt.loadtests = TAuthor TVisitor TCrawler
com.xceptance.xlt.loadtests.TAuthor.users = 5
com.xceptance.xlt.loadtests.TVisitor.users = 3
com.xceptance.xlt.loadtests.TCrawler.users = 4 
.........

The following figure shows the relationship between the various load test profile settings.

Load Test Profile Configuration Load Test Profile Configuration

A load test starts by generating a constant load based on the number of concurrent users where the maximum load is limited by the number of users. If an “arrivalRate” is given the system generates load with a cluster-wide constant arrival rate. Note that an arrival rate is only supported for non-iteration based load tests.

In order to minimize discrepancies that could be caused by applications and other systems starting up and not yet operating at an optimal level, an initialDelay and warmUpPeriod can be defined which is the time we will give the load to level out before taking measurements.

The “rampUpPeriod” setting dictates the time that it will take for the load to increase to 100% after the initial delay. If you run an arrival rate test, this will only lift the ceiling of the users permitted to run. Due to the nature of the arrival rate test, this might not necessarily mean that the users will run at this point in time.

The test will be measured during the “measurementPeriod”. To ensure a predictable load until the very end of the measurement period, a shutdownPeriod can be set in which the users will continue to run but the measurements have been stopped. This permits a fading out of the tests.

In case you want to modify the behavior of the logging facility of the load test agents, the test suite configuration directory contains a file named log4j.properties, which can be changed to meet your needs.

To launch the Java Virtual Machine that runs the agent with additional parameters, specify them in a file named jvmargs.cfg.

Run the Load Test

Running the load test consists of two steps:

  1. Running the agent controllers,
  2. Running the master controller.

Running the Agent Controllers

To start the agent controllers open a command line window/console and type the following sequence of commands:

cd <XLT>/bin
./agentcontroller.sh

Windows users have to use the appropriate .cmd file which is located in the same directory.

The agent controller will start running on the specified port. The output will look like that:

- Using "C:\Users\AppData\Local\Temp\vfs_cache" as temporary files store.
- Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
- jetty-6.1.19
- Started SslSocketConnector@0.0.0.0:8500

Running the Master Controller

Make sure all agent controllers are running on all respective load test machines before starting the master controller. The master controller cannot be started if the agent controllers are not running. Also check that the test suite has been compiled successfully to avoid errors when uploading the test suite.

Master controller can be started in either of the following modes

Interactive Mode

You can start the master controller in interactive mode with the following command line:

cd <XLT>/bin
./mastercontroller.sh

Windows users have to use the appropriate .cmd file which is located in the same directory.

A screen like this will appear and displays the command line menu:

Xceptance LoadTest 3.3.4 (Build 1999) - 1000 user license
Copyright (c) 2005-2009 Xceptance Software Technologies GmbH. All rights reserved.


(u) Upload test suite
(s) Start agents
(a) Abort agents
(r) Show agent status
(d) Download test results
(c) Create load test report
(q) Quit
=>

The following options are offered:

Once you have chosen an option (by pressing the associated key followed by ENTER), the appropriate action is executed. Afterward, you will return to the menu immediately (unless you have chosen to quit, of course).

A typical usage scenario for a load test is reflected by the order of the master controller menu items and might look like this:

  1. Upload the test suite (using (u) shortcut)
  2. Start the agents (using (s) shortcut)
  3. Check the agent status regularly (using (r) shortcut)
  4. Download the test results once the test has finished (using (d) shortcut)
  5. Create a report of the downloaded results (using (c) shortcut)
  6. Quit the master controller (using (q) shortcut)
Auto Mode

As you have seen in the previous sections, there is a typical sequence of steps to be executed when running a load test. It may quickly become tedious and error-prone to type the necessary keys over and over again. To avoid this repetition, XLT provides another operating mode: the auto mode. In this mode, all the steps mentioned above are executed automatically without any user interaction. To start XLT in this operating mode, use the following command line:

Unix based systems: 

cd <XLT>/bin
./mastercontroller.sh -auto

Windows: 

cd <XLT>\bin
mastercontroller.cmd -auto

If the test suite files were uploaded and the load agents started successfully, XLT automatically refreshes the agent status regularly. Once the test has finished, the test results are downloaded. Afterward, XLT quits.

If the command is followed by the option -report then additionally a load test and performance report is generated automatically after downloading all results is finished. 

cd <XLT>\bin
mastercontroller.cmd -auto -report

See below what the screen shows in -auto mode:

Xceptance LoadTest 3.2.0 (Build 1) - 1000 user license
Copyright (c) 2005-2009 Xceptance Software Technologies GmbH. All rights reserved.

Uploading agent files ...
 -> Agent Controller ac1 <https://localhost:8443> ... OK

Starting agents ...
 -> Agent Controller ac1 <https://localhost:8443> ... OK

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10            0      0,00 s      0,00 s      0:00:00           0        0         0%
TAddToCart      Running        10 of     10            0      0,00 s      0,00 s      0:00:00           0        0         0%
TCreateUser     Running        10 of     10            1      0,72 s      0,72 s      0:00:01           0        0         0%

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10           72      0,67 s      0,77 s      0:00:06           0        2         5%
TAddToCart      Running        10 of     10           55      0,70 s      1,03 s      0:00:06           0        0         5%
TCreateUser     Running        10 of     10           83      0,95 s      0,67 s      0:00:06           0       17         5%

.
.
.

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10        1.472      0,69 s      0,66 s      0:01:37          17       65        96%
TAddToCart      Running        10 of     10        1.412      0,66 s      0,68 s      0:01:37           0        0        96%
TCreateUser     Running        10 of     10        1.525      0,91 s      0,63 s      0:01:37           0      316        96%

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Finished        0 of     10        1.533      1,16 s      0,65 s      0:01:41          17       65       100%
TAddToCart      Finished        0 of     10        1.476      1,17 s      0,68 s      0:01:40           0        0       100%
TCreateUser     Finished        0 of     10        1.590      0,79 s      0,63 s      0:01:41           0      325       100%

Downloading test results ...
 -> Agent Controller ac1 <https://localhost:8443> ... OK

To abort the test prematurely, press CTRL-C to terminate the master controller. This terminates all running agents as well and triggers the download of all test results generated so far. This also means, that it is not possible to disconnect the master controller from the test cluster while keeping the load test running.

For long running load tests, it is recommended to run the test without the auto option, because this allows a disconnect from the test as well as inhibits accidental test termination.

Embedded Mode

Both interactive mode and auto mode, can be combined with the command line option -embedded. This option starts master controller together with an internal agent controller.

This is useful if you want to run load tests without a distributed load test environment but run just one agent controller together with the master controller on the same machine. There is no need to start an agent controller manually before you run the load test then. This helps to handle automated load tests started from within a build process. This option is also recommended when playing around with pebble demo for training purposes because it simplifies the process for running a load test.

When using -embedded mode the local agent controller settings will override the set of agent controllers configured in mastercontroller.properies.

Test Results and Reports

XLT Result Browser

When running test cases outside of Script Developer, either in Eclipse, as Load test, or an Ant build, you have the option to save the page output to disk. The relevant property is com.xceptance.xlt.output2disk. By default, this is set to never. If you want to enable page output to disk you should copy the following section to dev.properties or test.properties:

## Enables page output to disk. Possible values are:
## - never ... pages are never logged
## - onError ... pages are logged only if the transaction had errors
## - always ... pages are logged always 
com.xceptance.xlt.output2disk = always

All saved results can be found in the <testsuite>/results directory. See the following lines for details of the results sub-directory structure:

In the folders for each test run (results/[testcase]/[virtual-user]/output/[transaction-ID]) you will find an index.html. This opens the XLT Result Browser. The result browser offers an integrated navigation to browse the complete page output of the transaction, and allows viewing detailed information for every single request. The file last.html in the output folder results/[testcase]/[virtual-user]/output opens the result browser for the last executed transaction of this virtual user.

Result browser navigation only allows access to the single pages of a transaction if they are directly related to actions. Therefore defining actions properly is very important to make the most effective use of the result browser. For details of how to structure test cases and create acions, see also Basic Concepts and code structuring recommendations.

XLT Result Browser - Page Output XLT Result Browser – Page Output

Clicking on one of the action names in the navigation makes the result browser show the respective page. When double-clicking an action name, the navigation expands to list all related requests. The listed requests are color-coded with black for successful responses (HTTP status code 200), red for protocol errors (HTTP status code 404) and grey for redirects (HTTP status code of 301 or 302).

When selecting one of the requests from the navigation the page content is replaced by detailed information about the request and the related response accessible via four tabs on top of the page. The following information is available:

XLT Result Browser - Request Details XLT Result Browser – Request Details

Creating and Evaluating Load Test Reports

As the most important tool for analyzing the results of a load test run, XLT offers three different load test reports. Each of these reports is explained in detail in separate sub-sections.

To create the reports, you have to download all load test results from the agent controllers to the master controller. See the section Run The Load Test for details.

Once you have downloaded the load test results to your local disk, you can create the test reports with the XLT report generator. Enter a command in the console following the shown pattern:

cd <XLT>/bin
./<report-shortname>.(sh/cmd) ../results/<downloaded-results-dir> [options]

The <downloaded-results-dir> and <report-shortname> have to be replaced with the appropriate values. For example:

./create_report.sh ../results/20110503-152920

This tells the report generator to take the specified results directory as input for the report. By default, the generated report is saved to <XLT>/reports. The report sub directory is named after the respective results directory.

Options supported by report generator are:

-o <dir>: an alternative output directory (optional)
-from <time>: ignore results generated before the given time (optional)
-to <time>: ignore results generated after the given time (optional)

With the -o option, you can specify an alternative output directory. Please keep in mind, that you have to specify a target directory name including the final directory for your report. When using the -o option, the directory name is not set automatically but the specified directory will be created. For example: 

./create_report.sh ../results/20110503-152920 -o D:/Test_Reports/MyLatestReport

or if you are only interested in creating a report for a specific time range:

./create_report.sh ../results/20110503-152920 -from 20110503-152600 -to 20110503-152800

Note that <time> has to be specified in the format yyyyMMdd-HHmmss and it has to match the time zone of your local machine. The resulting report will be rendered using your machine’s time zone, too.

All this information is rendered to HTML pages that can be viewed using a standard web browser. Once the report is generated, which may take a while depending on the amount of data gathered during the load test, you will find the file index.html in the root of the appropriate test report directory. Open this file in a web browser to view the report.

Load and Performance Test Report

Load and performance test report is the main report that will give you the information needed for a detailed analysis after a load test run. It contains the several sections, each with a table and one or more charts showing graphic development of relevant measurements over time.

Load and Performance Test Report Load and Performance Test Report

Load and Performance Test Report - Charts Load and Performance Test Report – Charts

Create A Load And Performance Test Report

In order to generate a load and performance test report, use the following command:

create_report.(sh/cmd) ../results/<testDataDir> [options]

For example:

./create_report.sh ../results/20110503-160520

As an alternative to the command described above you can also create a load and performance test report with the (c) shortcut from the master controller’s command line menu. This shortcut creates a report of the last downloaded results after a load test.

Performance Comparison Report

Performance comparison report gives you a quick overview over performance improvements (green color tones) or performance decline (red color tones) between two test runs. The initial test run is labeled baseline. The test run that is compared to the baseline is labeled measurement run.

Every section of the comparison report displays a table with changes in performance and is divided into the following three parts:

When you hover the mouse over the columns of the report table, you will see the actual measurement results. This will give you a better impression whether or not the reported percentage change is significant or not.

Performance Comparison Report - Overview Performance Comparison Report – Overview

Performance Comparison Report Performance Comparison Report

Performance Comparison Report contains the following sections:

Create a Performance Comparison Report

You can generate a performance comparison report only between two existing load and performance test reports. This means you have to create both load and performance test reports first.

After that you can generate a performance comparison report using the following command:

create_diff_report.(sh/cmd) <testReportDir_1> <testReportDir_2> [options]

For example:

./create_diff_report.sh ../reports/20110503-152920 ../reports/20110503-160520

Performance Trend Report

A trend report shows the development of performance over time. Multiple measurements are taken into account and evaluated against each other. An XLT trend reports shows you how your system performs over time, how your tuning effort pays out, and how your live environment acts under changing load situation, if used as monitoring.

Two trend report types are available:

The Difference to the First Run reports the changes compared to your first test run, mostly referred to as baseline. Each table column will show you the difference between your baseline run and the run your are interested in. The quality of your baseline run defines how valuable this report may be. You can also see this as long-term performance trend report.

The Difference to Previous Run visualizes the improvements between two adjacent test runs. This shows you, how your last change or tuning effort payed out in comparison to the previous run. It helps you to see whether or not you are on the right track improving the performance of your application. The report also emphasizes sudden improvements or set-backs. This report can also be seen as a short-term performance trend report.

When you hover the mouse over the columns of the trend report table, you will see the actual measurement results. This will give you a better impression whether or not the reported percentage change is significant or not. Please keep in mind, that changes up to 10% are most of the time measurement fluctuation.

Performance Trend Report - Overview Performance Trend Report – Overview

Performance Trend Report Performance Trend Report

Similar to the other reports the trend report is divided into the following sections, each with the mentioned tables and charts:

Create a Performance Trend Report

You can generate a performance trend report over several test reports using the following command:

create_trend_report <testReportDir_1> ... <testReportDir_n> [options]

For example:

./create_trend_report.sh ../reports/20110503-152920 ../reports/20110503-160520 ../reports/20110503-161030

Custom Values

During a load test XLT is logging a large amount of data relevant to the test run. Nevertheless, sometimes it comes in handy to log some additional information about the system under test (SUT) directly during the load test run. For this purpose XLT provides custom values.

Example: An eCommerce application is typically connected to several third-party systems to use external services like creditworthiness check. The response time of these third-party systems can have a major impact on the SUT’s response to the client request. This application-internal information is not visible to XLT during a load test by default. A typical example for custom values in this context is logging the response time of requests to third party systems. To do so the user must write custom code to access the relevant sources, e.g. via remote connection to the application server. The additional data then can be logged by XLT while load test runtime and is automatically integrated into the load and performance test report.

Sampler

Custom Samplers enable the user to query custom sources and log data (samples) while load test runtime. For that purpose the user must provide a custom sampler class extending com.xceptance.xlt.api.engine.AbstractCustomSampler. The sampler gets configured in the test suite configuration files. Recommended location for the relevant configuration is project.properties.

The provided sampler must override the execute() method that is called after each interval time (see configuration). Furthermore the sampler might override the method initialize() or shutdown() that get called just once for the sampler. While initialize() is called before the first call of execute(), shutdown() is called on shutdown.

The logged custom value is the return value of the execute() method.

The AbstractCustomSampler can store any ‘double’ value. The stored value describes the absolute value at a certain point in time. The corresponding report chart is directly showing the logged value.

Configuration

Samplers get configured by providing the following properties:

com.xceptance.xlt.customSamplers.1.class=com.xceptance.xlt.samples.ValueSamplerDemo
com.xceptance.xlt.customSamplers.1.name=ValueSamplerDemo
com.xceptance.xlt.customSamplers.1.description=This sampler logs a custom value which is just a random number
com.xceptance.xlt.customSamplers.1.interval=1000
com.xceptance.xlt.customSamplers.1.chart.title=ValueSamplerDemo
com.xceptance.xlt.customSamplers.1.chart.yAxisTitle=Value
#com.xceptance.xlt.customSamplers.1.property.foo=123
#com.xceptance.xlt.customSamplers.1.property.bar=abc
...
...
com.xceptance.xlt.customSamplers.9.class=...
com.xceptance.xlt.customSamplers.9.name=...
...

Example

The following code shows an example of how to create two very simple custom sampler classes that are logging random values.

public class ValueSamplerDemo extends AbstractCustomSampler
{
    public ValueSamplerDemo()
    {
        super();
    }

    @Override
    public void initialize()
    {
        // initialize
    }

    @Override
    public double execute()
    {
        // generate random value based on the configured limits

        // get properties
        final String lowerLimitProp = getProperties().getProperty("generatedValueLowerLimit");
        final String upperLimitProp = getProperties().getProperty("generatedValueUpperLimit");

        // convert to integer
        try
        {
            final int lowerLimit = Integer.valueOf(lowerLimitProp);
            final int upperLimit = Integer.valueOf(upperLimitProp);
				
			// return the value to be logged
            return XltRandom.nextInt(lowerLimit, upperLimit) + XltRandom.nextDouble();

        }
        catch (final NumberFormatException e)
        {
            // log 0 in case of an exception
            return 0;
        }
    }

    @Override
    public void shutdown()
    {
        // clean up
    }
}

The resulting charts (one chart for each sampler) are automatically integrated into the XLT performance and load test report and can be accessed via the report navigation menu item Custom Values.

Custom sampler report Custom sampler report

External Data Report

As an alternative to custom values XLT report generator provides the ability to include external data available as files into the report. This can be used if it is not possible to access the external data source directly during load test runtime.

Parser

To read the external files and integrate them into the load test report a Parser class is needed for each type of file or format. XLT provides a set of predefined parsers for CSV files. If you plan to integrate some other file formats or reports you have to write your own parser class extending AbstractLineParser in com.xceptance.xlt.api.report.external.

XLT ships with a demo test suite demo-external-data to show the usage of external data in the load test report. As an Example of advanced parser classes two parsers for handling logs of the command line tool iostat can be found in the source directory of this test suite project <XLT>\samples\demo-external-data\src in package com.xceptance.xlt.report.external.

Configuration

Location

The configuration file externaldataconfig.xml is expected in the result directory of the respective load test run. If this file can’t get located there, it will get looked up in the master controller’s config directory. See the provided sample file <XLT\samples\demo-external-data\config\externaldataconfig.xml.sample and adapt to your needs after removing the .sample extension. At least adapt the source file and columns/indexes. The configuration file adapted for the demo-external-data project is located at <XLT>\samples\demo-external-data\results\20110621-101041.

Structure

The configuration defines files to parse by a specified parser, the column indexes to mark the relevant data and other settings like headlines, descriptions or colors to enrich the chart/table information for the report.

Currently there are two properties supported by the framework to convert a human readable date/time to a time stamp (parser.dateFormat.pattern and parser.dateFormat.timeZone). If the time stamp is already a UNIX time stamp, the pattern is not needed. It’s strictly recommended to adapt these properties to your needs. Please see java.text.SimpleDateFormat for more information on date/time patterns.
Another supported property is parser.csv.separator that describes the expected field separator char in CSV files. By default this is a comma (,).

Appendix

Script Developer Commands

CommandDescription
General Commands
openOpens a URL
closeClicks the close button in the titlebar of a popup window or tab
pauseWaits for the specified amount of time (in milliseconds)
typeTypes something to an input field
typeAndWaitTypes something to an input field and waits
clickClicks an element
clickAndWaitClicks an element and waits
doubleClickDouble-clicks an element
doubleClickAndWaitDouble-clicks an element and waits
submitSubmits a form
submitAndWaitSubmits a form and waits
setTimeoutSets the timeout for waiting commands
Assert Commands
assertTextPresentAsserts that the selected text is visible on the page (you have to select the respective text before)
assertElementPresentAsserts that an element is present on the page (you have to open the Firefox context menu directly on the element in question)
assertTitleAsserts that the page title matches
assertTextAsserts that an element is visible on the page and that the element has a certain text (you have to select the respective text before)
assertPageSizeAsserts that the size of the document does not exceed the given value
assertXpathCountAsserts that the number of matching elements are equal to the given value
assertLoadTimeAsserts that the time needed to load the page does not exceed the given value
assertNotElementPresentAsserts that an element is not on page
assertNotTextAsserts that text does not match
assertNotTextPresentAsserts that text is not on page
assertNotTitleAsserts that the page title does not match
assertNotXpathCountAsserts that the number of matching elements are unequal to the given value
WaitFor Commands
waitForElementPresentWaits for the specified element
waitForTextWaits for the specified element matching the specified text
waitForTextPresentWaits for the specified text
waitForTitleWaits for the specified page title
waitForPageToLoadWaits for page to load
waitForPopUpWaits for a popup window to appear and load up
waitForXpathCountWaits until the number of matching elements are equal to the given value
waitForNotElementPresentWaits for the specified element to disappear
waitForNotTextWaits for the specified element changing the specified text
waitForNotTextPresentWaits for the specified text to disappear
waitForNotTitleWaits for change the specified page title
waitForNotXpathCountWaits until the number of matching elements are unequal to the given value
Select Commands
selectSelects an option in single select
selectAndWaitSelects an option in single select and waits
selectFrameSelects a frame within the current window
selectWindowSelects a popup window using a window locator
addSelectionSelects an option in multiple select
removeSelectionRemoves an option in multiple select
CheckBox/Radio Button Commands
checkTicks checkbox or radio button
checkAndWaitTicks checkbox or radio button and wait
uncheckDeselects checkbox or radio button
uncheckAndWaitDeselects checkbox or radio button and waits
Cookies Handling Commands
createCookieCreates a new cookie
deleteCookieRemoves a cookie
deleteAllVisibleCookiesRemoves all cookies visible to the current page

Syntax and meaning of the commands is quite similar to Selenium. See the Selenium command reference for more information.

Script Developer Shortcuts

CommandShortcut
EditReturn
Edit DetailsAlt+Return
SaveCtrl+S
Save AllCtrl+Shift+S
CutCtrl+X
CopyCtrl+C
PasteCtrl+V
DeleteDelete
Manage Test DataAlt+D
Insert New Command (before currently selected command)Insert
Insert New Command (after currently selected command)Shift + Insert
En/Disable Module/CommandCtrl+Shift+C
Toggle StartPointS
Toggle BreakPointB
Move Command UpAlt+Up
Move Command DownAlt+Down
Select All CommandsCtrl+A
Multiple SelectHold Shift+Up/Down
Multiple Select SelectiveHold Ctrl+Up/Down+Spacebar
Reload Active ScriptF5
Close TabCtrl+W
Close All TabsCtrl+Shift+W

Terminology

Acknowledgments