Executing Tests

Previous Next Contents

5 Executing Tests

The JSONP TCK uses the JavaTest harness to execute the tests in the test suite. For detailed instructions that explain how to run and use JavaTest, see the JavaTest User’s Guide and Reference in the documentation bundle.

This chapter includes the following topics:

Note

The instructions in this chapter assume that you have installed and configured your test environment as described in Chapter 3, "Installation," and Chapter 4, "Setup and Configuration,", respectively.

5.1 Starting JavaTest

There are two general ways to run the JSONP TCK using the JavaTest harness software:

  • Through the JavaTest GUI

  • From the command line in your shell environment

Note

The <TS_HOME>/tools/ant/bin/ant command referenced in the following two procedures and elsewhere in this guide is the Ant build tool, which is included in the JSONP TCK bundle. The build.xml file in <TS_HOME>/bin contains the various Ant targets for the JSONP TCK test suite.

5.1.1 To Start JavaTest in GUI Mode

Execute the following commands:

cd <TS_HOME>/bin
ant gui

5.1.2 To Start JavaTest in Command-Line Mode

  1. Change to any subdirectory under <TS_HOME>/src/com/sun/ts/tests.

  2. Start JavaTest using the following command:

    ant runclient

Example 5-1 JSONP TCK Signature Tests

To run the JSONP TCK signature tests, enter the following commands:

cd <TS_HOME>/src/com/sun/ts/tests/signaturetest/jsonp
ant runclient

Example 5-2 Single Test Directory

To run a single test directory, enter the following commands:

cd <TS_HOME>/src/com/sun/ts/tests/jsonp
ant runclient

Example 5-3 Subset of Test Directories

To run a subset of test directories, enter the following commands:

cd <TS_HOME>/src/com/sun/ts/tests/jsonp
ant runclient

5.2 Running a Subset of the Tests

Use the following modes to run a subset of the tests:

5.2.1 To Run a Subset of Tests in GUI Mode

  1. From the JavaTest main menu, click Configure, then click Change Configuration, and then click Tests to Run.
    The tabbed Configuration Editor dialog box is displayed.

  2. Click Specify from the option list on the left.

  3. Select the tests you want to run from the displayed test tree, and then click Done.
    You can select entire branches of the test tree, or use Ctrl+Click or Shift+Click to select multiple tests or ranges of tests, respectively, or select just a single test.

  4. Click Save File.

  5. Click Run Tests, and then click Start to run the tests you selected.
    Alternatively, you can right-click the test you want from the test tree in the left section of the JavaTest main window, and choose Execute These Tests from the menu.

  6. Click Report, and then click Create Report.

  7. Specify the directory in which the JavaTest test harness will write the report, and then click OK.
    A report is created, and you are asked whether you want to view it.

  8. Click Yes to view the report.

5.2.2 To Run a Subset of Tests in Command-Line Mode

  1. Change to the directory containing the tests you want to run.

  2. Start the test run by executing the following command:

    ant runclient

    The tests in the directory and its subdirectories are run.

5.2.3 To Run a Subset of Tests in Batch Mode Based on Prior Result Status

You can run certain tests in batch mode based on the test’s prior run status by specifying the priorStatus system property when invoking <TS_HOME>/tools/ant/bin/ant.

Invoke ant with the priorStatus property.

The accepted values for the priorStatus property are any combination of the following:

  • fail

  • pass

  • error

  • notRun

For example, you could run all the JSONP tests with a status of failed and error by invoking the following commands:

ant -DpriorStatus="fail,error" runclient

Note that multiple priorStatus values must be separated by commas.

5.3 Running the TCK Against the RI

This test scenario is ensures that the configuration and deployment of all the prebuilt JSONP TCK tests against the Reference Implementation are successful, and that the TCK is ready for compatibility testing against the Vendor and Reference Implementations.

  1. Verify that you have followed the configuration instructions in Section 4.1, "Configuring Your Environment to Run the TCK Against the Reference Implementation."

  2. If required, verify that you have completed the steps in Section 4.3.2, "Deploying the Prebuilt Archives."

  3. Run the tests, as described in Section 5.1, "Starting JavaTest," and, if desired, Section 5.2, "Running a Subset of the Tests."

5.4 Running the TCK Against a Vendor’s Implementation

This test scenario is one of the compatibility test phases that all Vendors must pass.

5.5 Test Reports

A set of report files is created for every test run. These report files can be found in the report directory you specify. After a test run is completed, the JavaTest harness writes HTML reports for the test run. You can view these files in the JavaTest ReportBrowser when running in GUI mode, or in the web browser of your choice outside the JavaTest interface.

To see all of the HTML report files, enter the URL of the report.html file. This file is the root file that links to all of the other HTML reports.

The JavaTest harness also creates a summary.txt file in the report directory that you can open in any text editor. The summary.txt file contains a list of all tests that were run, their test results, and their status messages.

5.5.1 Creating Test Reports

Use the following modes to create test reports:

5.5.1.1 To Create a Test Report in GUI Mode
  1. From the JavaTest main menu, click Report, then click Create Report.
    You are prompted to specify a directory to use for your test reports.

  2. Specify the directory you want to use for your reports, and then click OK.
    Use the Filter list to specify whether you want to generate reports for the current configuration, all tests, or a custom set of tests.
    You are asked whether you want to view report now.

  3. Click Yes to display the new report in the JavaTest ReportBrowser.

5.5.1.2 To Create a Test Report in Command-Line Mode
  1. Specify where you want to create the test report.

    1. To specify the report directory from the command line at runtime, use:

      ant -Dreport.dir="report_dir"

      Reports are written for the last test run to the directory you specify.

  2. To specify the default report directory, set the report.dir property in <TS_HOME>/bin/ts.jte.
    For example:

    report.dir="/home/josephine/reports"
  3. To disable reporting, set the report.dir property to "none", either on the command line or in <TS_HOME>/bin/ts.jte.
    For example:

    ant -Dreport.dir="none"

5.5.2 Viewing an Existing Test Report

Use the following modes to view an existing test report:

5.5.2.1 To View an Existing Report in GUI Mode
  1. From the JavaTest main menu, click Report, then click Open Report.
    You are prompted to specify the directory containing the report you want to open.

  2. Select the report directory you want to open, and then click Open.
    The selected report set is opened in the JavaTest ReportBrowser.

5.5.2.2 To View an Existing Report in Command-Line Mode

Use the Web browser of your choice to view the report.html file in the report directory you specified from the command line or in <TS_HOME>/bin/ts.jte.

5.6 Running the Pluggability Tests

Use the following modes to run the pluggability tests:

5.6.1 To Run the Pluggability Tests in GUI Mode

When running JavaTest in GUI mode as described above, you are prompted to specify whether you want to run all or a subset of the test suite. Since the pluggability tests must be run separately from the other tests, select Yes. Select the tests you want to run from the displayed test tree, then select Next. You can select entire branches of the test tree, or use Ctrl+Click or Shift+Click to select multiple tests or ranges of tests, respectively. Select all tests except pluggability tests by unchecking the pluggability tests.

After running the other tests, run the pluggability tests, which require a separate test run:

  1. If the JavaTest GUI is running, exit the GUI.

  2. Change to the <TS_HOME>/bin directory and reconfigure environment to use the JsonProvider class that is supplied with the test suite and restart the JavaTest GUI:

cd <TS_HOME>/bin
ant enable.alternate.jsonp.provider
ant gui
  1. Select Configure, then select Edit Configuration.

  2. Select Tests to Run.

  3. Select the tests you want to run from the displayed test tree, then select Next.
    Select the pluggability tests only by unchecking all of the other tests.

  4. Select Done.

  5. Select Run Tests, then select Start.

  6. Exit the JavaTest GUI when finished.

  7. Disable the alternate JSON provider that you enabled in Step 2, before you initiated the test run for the pluggability tests:

cd <TS_HOME>/bin
ant disable.alternate.jsonp.provider

5.6.2 To Run the Pluggability Tests in Command-Line Mode

There are two ways to run the pluggability tests from the command line. It is highly recommended to use the first method, since it automatically calls the Ant targets to enable and disable the test provider. If you choose to use the second method, you will need to execute those Ant targets manually before and after you run the tests.

The first method is by executing the run.pluggability target in Ant:

  1. Execute the run.pluggability target using Ant:

cd <TS_HOME>/bin
ant run.pluggability

The run.pluggability target calls Ant targets that enable the test provider, run the pluggability tests, then disable the test provider.

The pluggability tests can also be run an alternate way, albeit one that requires more manual intervention:

  1. Execute the enable.alternate.jsonp.provider target using Ant:

cd <TS_HOME>/bin
ant enable.alternate.jsonp.provider

The enable.alternate.jsonp.provider target enables the alternate test provider. 2. Change to the <TS_HOME/src/com/sun/ts/tests/jsonp/pluggability> directory:

cd <TS_HOME/src/com/sun/ts/tests/jsonp/pluggability>
  1. Start the pluggability test run by executing the following command:

ant runclient

This will run just the pluggability tests under the <TS_HOME/src/com/sun/ts/tests/jsonp/pluggability directory, using a test-supplied JsonProvider Class for testing the SPI. All JsonProvider method calls will be invoked and verified by the tests to ensure SPI layer is working. 4. Execute the disable.alternate.jsonp.provider target using Ant:

cd <TS_HOME>/bin
ant disable.alternate.jsonp.provider

The disable.alternate.jsonp.provider target disables the alternate test provider.


Previous Next Contents
Eclipse Foundation Logo  Copyright © 2018, Oracle and/or its affiliates. All rights reserved.