App Test Framework Guide

Introduction

The App Test Framework provides a high productivity automated functional/integration testing framework for Transact Platform applications. This framework tests Transact Platform applications at the REST API layer by mocking the HTTP web requests to the Transact Application server, executing your applications code and then asserting system is in the correct state after the operation.

In terms of the pyramid or hierarchy of application testing, the App Test Framework sits between Unit Testing and UI Web Automation Testing.

It has the benefits of integration testing that UI Web Automation Testing provides in that it exercises the application very close to the HTTP protocol level, but also has the benefits of Unit Testing in terms of being very fast to run, not requiring extensive infrastructure, and can be created by application developers who don't have expertise in Web Automation testing frameworks such as Selenium.

The App Test Framework was inspired by property-based generative testing, but uses a functional specification approach to integration testing. Using the framework developers can write black-box functional test cases providing the inputs and specifying the expected results. This approach is much more productive than writing automated integration tests by hand in JUnit, greatly reducing the amount of test code you need to write and maintain.

Internally the Transact Platform uses a similar testing framework for QA of Transact Functions and the Open UX API.

Valid Test Scenarios

You should use the App Test Framework for testing the user journeys through your application, and ensuring the correct Transact Functions are called and correct state application transitions occur.

Here's what you should use the App Test Framework for.

• Developer-created application integration tests.
• Functional specification testing of user application flow.
• Fast CI-based integration testing (must run frequently).
• Testing the behavior of Transact Functions and their HTTP results.
• Testing application package configuration (prevent configuration drift)

Conversely, here are some test scenarios that you shouldn't use the App Test Framework for.

• Testing your application's HTML front end.
• Unit testing of lower level classes.
• Comprehensive 3rd party service integration testing.

Testing References

• Mike Cohn, The Forgotten Layer of the Test Automation Pyramid (Mountain Goat Software, 2009)
• Jonathan Rasmusson, The Testing Pyramid (Agile in a Nutshell, 2016)
• Ham Vocke, The Practical Test Pyramid (martinfowler.com, 2018)
• Mike Wacker, Just Say No to More End-to-End Tests (Google Testing Blog, 2015)

How it Works

The App Test Framework uses a data driven approach with JSON test suite definitions which provide a series of test cases to exercise application user flows. Each test case incorporates HTTP test inputs and test results which are evaluated once the test case is completed. The test results provide the assertions for each test case.

Below is a "Form Page 1" test case definition. Note the test input and result sections.

{
    "name": "Form Page 1",
    "input": {
        "command": {
            "type": "formUpdate"
        },
        "formXml": "test-data-1.xml"
    },
    "result": {
        "statusCode": 200,
        "response": {
            "formStatus": "Saved"
        },
        "txn": {
            "formStatus": "Saved",
            "deliveryStatus": "Not Ready",
            "emailAddress": "[email protected]",
            "formDataMap['First Name']": "John",
            "formDataMap['Last Name']": "Smith",
            "formDataMap['Email']": "[email protected]",
        },
        "appDoc": {
            "//Applicant/FirstName": "John",
            "//Applicant/LastName": "Smith",
            "//Applicant/Email": "[email protected]",
        }
    }
}

This test case tests the scenario where the user fills out the first page of a form and clicks on the next button. In an actual JavaScript client application, it would then make a HTTP call to the server performing an Open UX API formUpdate command including the XML form data. The Transact Platform Application server would process this HTTP command invoking any configured Transact Functions, perform data persistence operations and finally returning a HTTP response.

The App Test Framework does precisely this, it mocks up an actual HTTP request using the test inputs and invokes the Transact Platform Application server's FormCommandHandler. It then tests the HTTP response and persistent application state to ensure the test assertions are valid. The test result section includes:

• statusCode - the HTTP response status code
• response - the HTTP response JSON body
• txn - the Fluent API Txn transaction object graph which is full loaded to support deep graph test assertions
• appDoc - an array of XPath assertions against the persistent application XML document

Below is an example test case failure output where the wrong email address field was used.

 [app-test] -----------------
 [app-test]  HTTP Response
 [app-test] -----------------
 [app-test] statusCode: 200
 [app-test] message: 
 [app-test] {
 [app-test]   "formStatus": "Saved",
 [app-test]   "revisionNumber": 1
 [app-test] }
 [app-test] ------------------
 [app-test]  Assertion Errors
 [app-test] ------------------
 [app-test] txn.emailAddress:{ expected: [email protected], actual: [email protected] }
 [app-test] txn.formDataMap['Email']:{ expected: [email protected], actual: [email protected] }
 [app-test] appDoc[//Applicant/Email]:{ expected: [email protected], actual: [email protected] }

With a test suite you will typically define multiple test cases simulating a user journey through the the man pages of the application. In the brief example below, the final test case performs a userSubmit command to complete the application.

{
    "name": "Form Page 1",
    "input": {
        "command": {
            "type": "formUpdate"
        },
        "formXml": "test-data-1.xml"
    },
    "result": {
        "statusCode": 200,
        "response": {
            "formStatus": "Saved"
        },
        "txn": {
            "formStatus": "Saved",
            ...
        },
        "appDoc": {
            "//Applicant/FirstName": "John",
            ...
        }
    }
},
{
    "name": "Submit Form",
    "input": {
        "command": {
            "type": "userSubmit"
        },
        "formXml": "test-data-2.xml"
    },
    "result": {
        "statusCode": 200,
        "response": {
            "formStatus": "Completed"
        },
        "txn": {
            "formStatus": "Completed",
            ...
        },
        "formDoc": {
            "//Applicant/FirstName": "John",
            ...
        }
    }
}

As a developer or test automation engineer using the App Test Framework, you will also be learning the HTTP Open UX API wire protocol via the test inputs and results. This knowlege will help you when you build your applications and help you understand how the Transact Platform actually works. To learn more about this, see Open UX API.

It is important to note when you are using the App Test Framework that it is testing your application packages using the same Transact Platform Application server code that runs your production workloads. The principle differences between the App Test Framework and the Transact Platform Application server runtimes are:

• App Test Framework uses the H2 in-memory SQL database, while the Transact Platform Application server uses a big iron RDBMS.
• App Test Framework uses mock HTTP command inputs, while the Transact Platform Application server uses a full WildFly HTTP stack.

App Test Files

By convention app test suite JSON files are located in a tests directory under the application project root directory. A standard project layout is as follows.

project

• src
  • test
    • test-suite-1.json
    • test-data-1.xml
    • test-data-2.xml
  • app-package-def.json

In a Maven project, these files are located as follows.

project

• src
  • main
    • resources
      • app-package-def.json
  • test
    • resources
      • test-suite-1.json
      • test-data-1.xml
      • test-data-2.xml

These JSON test suites are then referenced by the app-test Ant Task using a fileset attribute. By customizing the fileset attribute you can tailor what app test suites are executed in your CI build process.

For more information, see the app-test Ant Task's configuration and test behaviour options.

Test Suite Definition

Example JSON

Below is a comprehensive Test Suite example to illustrate the available options.

{
    "name": "Test Suite 1",
    "description": "TODO...",
    "precondition": {
        "form": {
            "formCode": "credit-app",
            "formStatus": "Opened",
            "formXml": "test-data-1.xml"
       },
       "space": {
            "name": "Web Plug-in"
       },
       "user": {
            "loginName": "jsmith",
            "email": "[email protected]",
            "firstName": "John",
            "lastName": "Smith",
            "mobile": "0412 345 678"
       }
    },
    "testCases": [
        {
            "name": "Form Page 1",
            "input": {
                "command": {
                    "type": "formUpdate"
                },
                "formXml": "test-data-2.xml"
            },
            "result": {
                "statusCode": 200,
                "response": {
                    "formStatus": "Saved",
                    "revisionNumber": 1
                },
                "txn": {
                    "formStatus": "Saved",
                    "deliveryStatus": "Not Ready",
                    "emailAddress": "[email protected]",
                    "formDataMap['First Name']": "John",
                    "formDataMap['Last Name']": "Smith",
                    "formDataMap['Email']": "[email protected]",
                    "formDataMap['Mobile']": "0412 345 678"
                },
                "appDoc": {
                    "//Applicant/FirstName": "John",
                    "//Applicant/LastName": "Smith",
                    "//Applicant/Email": "[email protected]",
                    "//Applicant/Mobile": "0412 345 678"
                }
            }
        },
        {
            "name": "Submit Form",
            "input": {
                "command": {
                    "type": "userSubmit"
                },
                "formXml": "test-data-3.xml"
            },
            "result": {
                "statusCode": 200,
                "response": {
                    "formStatus": "Completed"
                },
                "txn": {
                    "formStatus": "Completed",
                    "receiptStatus": "Ready",
                    "deliveryStatus": "Not Ready",
                    "emailAddress": "[email protected]",
                    "formDataMap['First Name']": "John",
                    "formDataMap['Last Name']": "Smith",
                    "formDataMap['Email']": "[email protected]",
                    "formDataMap['Mobile']": "0412 345 678"
                },
                "appDoc": {
                    "//Applicant/FirstName": "John",
                    "//Applicant/LastName": "Smith",
                    "//Applicant/Email": "[email protected]",
                    "//Applicant/Mobile": "0412 345 678",
                    "//Applicant/Address/Line1": "404 Main Street",
                    "//Applicant/Address/Line2": "Middle Town",
                    "//Applicant/Address/State": "WA",
                    "//Applicant/Address/PostCode": "4312"
                }
            }
        }
    ]
}

Test Suite JSON Definition

Attribute	Description
name	String Required. The name of the test suite.
description	String This is useful for notes explaining what scenarios the test suite covers.
precondition	Object (TestPrecondition) Required. An object that provides the initial setup state for the test cases.
testCases[]	Object (TestCase) Required. The test cases to perform in series.

TestPrecondition

Attribute	Description
form	Object (TestForm) Required. The form object used for the test setup.
space	Object (FormSpace) The form space to associate the form transaction with.
user	Object (UserAccount) The user account to associate with the initial form transaction.

TestForm

Attribute	Description
formCode	String Required. An identifier that is used to create the setup transaction. The App Test Framework will resolve the form's current form version when creating the setup transaction. In future releases, we may add support for specifying the formVersion to test multiple form versions in the same application package.
formStatus	String The initial transaction form status. This must be one of the following. "Opened": for a new form transaction "Saved": for a previously saved form transaction
formXml	String The form XML document used to initialize the form transaction.

FormSpace

Attribute	Description
name	String Required. The name of the form space to associate with the initial form transaction.

UserAccount

Attribute	Description
loginName	String Required. The form transaction user's login name.
email	String Required. The form transaction user's email address.
firstName	String The form transaction user's first name.
lastName	String The form transaction user's last name.
mobile	String The form transaction user's mobile phone number.

TestCase

Attribute	Description
name	String Required. The name of the test case.
input	Object (TestCaseInput) Required. The test case input.
result	Object (TestCaseResult) Required. The test case result object.

TestCaseInput

Attribute	Description
command	Object (TestCaseCommand) Required. The test case input command.
formXml	String The filename of the XML to provide the HTTP command formXml parameter data. Do not use this parameter with the formFunction command.

TestCaseCommand

Attribute	Description
type	String Required. An Open UX API command type. This must be one of the following. "formFunction" "formIneligible" "formUpdate" "userSave" "userSubmit" "userCancel"
fileAttachment	String The filename of the file attachment to provide the HTTP command fileAttachment parameter data. The fileAttachment parameter is only valid for formUpdate commands.

TestCaseResult

Attribute	Description
statusCode	Number Required. The HTTP response status code.
response	Object (HttpResponse) The HTTP response JSON object.
txn	Object (Txn) The form transaction object to perform test assertions against. This object is a Fluent API Txn object with a fully populated object graph (except for PDF receipt data). Txn objects properties are evaluated using the MVEL expression language (MVEL Language Guide).
appDoc	Object (XmlDoc) The application XML document to perform XML XPath test assertions against.