Skip to main content

Nominal and Error Strategy

The objective of this strategy is to successfully test all the operations in the REST API that are to be investigated. To do this, an ordering based on the Operation Dependency Graph (ODG) is defined. For all other responsibilities (generation of input values, oracles and writers), this strategy is based on basic components available in the framework.

  • Operation Ordering: to generate the nominal test cases, the concrete implementation of the ordering of operations is a dynamic strategy based on the operation dependency graph. The next operation to be tested is taken by consulting the ODG, taking an operation that has yet to be tested and minimising the number of unsatisfied producer-consumer relations. In addition to this, the order of operations is arranged according to CRUD semantics (e.g. delete operations are the last to be tested).
  • Input Value Provider: the Enum and Example Priority Input Value Provider is used. In this way, an attempt is made to take advantage of all possible single strategies available.
  • Oracles: the results of the test cases are evaluated according to the two available oracles. Status codes are evaluated according to the Oracle Status Code, while the format of answers is evaluated according to the Oracle Validation Schema.
  • Writers: the test cases are written in RestAssured. A report in JSON containing the results of the various TestSequences is also produced. A reference to this component can be found here.

Once nominal testing is performed, error testing is also performed, which aims to test the API with scenarios that violate the specification to try to reveal faults in the handling of incorrect requests. This testing strategy starts with Nominal Tester requests that have been successfully executed (i.e 2XX status code). These are modified through customised mutation operations defined by this strategy. In addition, error testing provides a customised sorting and oracle. The other components are those available in the framework.

  • Operation Ordering: for each Test Sequence of nominal test length n, all sub-sequences are calculated from the first interaction of the original sequence, with incremental length from 1 to n. Mutation operators are applied to the last interaction of each of these sequences.
  • Mutation Operators: three concrete implementations of mutation operators are provided:
    • Missing Required: removes a mandatory input parameter from an operation to produce malformed input. It only applies to parameters marked as mandatory in the specification.
    • Wrong Input Type: changes the type of an input parameter, e.g. an integer parameter is assigned a string value. The random input value provider generates a random value of the new type.
    • Constraint Violation: modifies the value of a parameter to make it contrary to the constraints of the specification. For example, it may add more characters to a string to make it longer than its maximum length.
  • Oracles: the Oracle Validation Schema is integrated to check the validity of the response format. There is also a customised oracle called the Error Status Code. It is implemented to verify the outcome of error scenario testing, based on the assumption that an erroneous request should be rejected by a well implemented API. If a request is rejected (4XX status code), the test is classified as PASS. If a request is accepted (2XX status code), the test is classified as FAIL. If a server error is observed (5XX status code), the test is considered FAIL.
  • Writers: both the JSON report writerand the writer for writing test cases in RESTAssured are integrated.