Welcome to REST League 2027, the 2nd edition of the ultimate competition for automated black-box testing tools designed for RESTful APIs.
The goal of REST League is to evaluate the performance of open-source and proprietary tools in finding bugs, security vulnerabilities, and logic flaws in real-world REST APIs without access to their source code.
This year, the competition will be part of the 49th International Conference on Software Engineering (ICSE2027), contributing to the Competition Track. Details of previous editions can be found here.
Participants submit their automated black-box testing tools to compete against a benchmark set of REST APIs. These APIs contain a variety of intentionally injected issues, ranging from functional defects to common security flaws.
What is black-box testing?
In this context, black-box testing means the tools interact with an API solely via its public endpoints, documented by an OpenAPI specification. Tools generate test cases by leveraging the specification and the feedback from the API, without looking at the API’s internal implementation.
All tools in the competition will be provided with the OpenAPI specification (JSON/YAML) of the REST API to test, the API’s endpoint, and any required authentication material (e.g., API keys). Tools will not receive the API’s source code, database schema, or internal logs.
The output expected from a tool is a sequence of HTTP requests/responses (the test cases actually executed against the API) that are used to compute the evaluation metrics.
Tools must meet the following general requirements to participate
In particular, the tools must be compatible with RESTgym, a benchmarking infrastructure specifically designed for REST API testing tools. Check here the details of the competition experimental setting.
Tools will be scored based on a comprehensive set of criteria that focus on both quality of testing and resource efficiency.
5XX responses obtained by the tool2XX response)The final score will be a weighted aggregate of these metrics.
Uniqueness of 5XX is in terms of sufficiently distinct error messages, following the criterion used in literature1.
Testing metrics will be collected by using the Restats tool.
REST League features a badge-based challenge structure to highlight tool strengths along distinct dimensions.
The best-performing tool for each challenge will be awarded a badge.
🥇 Gold API Tester: Tool with the highest overall score, winner of the Effectiveness Challenge.
🥈 Silver API Tester: Tool with the second-highest overall score, runner-up of the Effectiveness Challenge.
🐞 Bug Hunter: Tool with the highest number of unique 5XX found, winner of the Fault detection Challenge.
⚡ Roadrunner: Tool that achieves the highest coverage with the lowest resource consumption footprint, winner of the Efficiency Challenge.
The tool earning the Gold API Tester badge is also the competition’s ultimate champion.
Winning tools and their authors will be prominently featured on this GitHub page, recognized at the Awards Ceremony, and invited to present their tool during ICSE2027 in Dublin.
Tools will be run by using RESTgym, with a time budget of one hour for each API. Testing sessions will be repeated five times, collecting average results.
All runs will execute in an organizer-controlled sandbox, deployed on proprietary hardware. Network access from within the sandbox is restricted to the target API’s endpoint only. Metrics will be collected by the independent, organizer-run Restats tool included in RESTgym.
Experiments will be executed on a 128-core machine with 386GB of RAM, enabling parallel testing sessions. To each tool, 8 cores and 16GB of RAM will be reserved. The machine used in the experiments is equipped with an Nvidia RTX 4090 video card, available for the tools.
LLM Adoption
To regulate language model usage, remotely accessed LLMs are forbidden. Competing tools can use small or large language models, given they can be locally run on the evaluation platform.
Tools will be evaluated on a benchmark consisting of ten real-world REST APIs. Half of them are APIs already used in testing literature (listed below), and half of them are fresh APIs never used in previous studies (not disclosed to participants).
The benchmark will consist of the following known REST APIs (all available in RESTgym):
Submit your tool via EasyChair by 9 October, 2026. For organizational purposes, we kindly ask interested participants to submit a partial submission (with at least the tool name and authors) even if the tool is not yet ready. Submissions can be freely updated until 9 October, 2026.
Upon submission, authors need to provide the tool source code and a Dockerfile enabling the building of a RESTgym-compliant container image.
Authors of non-open-source tools or tools under specific intellectual property constraints are asked to directly contact the organizers.
All authors of tools participating in the competition can submit a solution paper describing the tool’s approach used to tackle the competition problem. The paper should also discuss the evolution of the tool with respect to previous versions (if any).
Submissions will undergo a single-blind peer-review process to assess their technical correctness.
IEEE Publication
Accepted solution papers will be included in the ICSE2027 proceedings and published by IEEE.
Authors of accepted solution papers are required to present their tool during the competition conference session. The session will be part of the ICSE2027 program, scheduled from 25 April, 2027 to 1 May, 2027.
REST League 2027 will be hosted by ICSE2027 in Dublin, Ireland, with the following schedule.
| Date (AoE) | Event |
|---|---|
| 24 August, 2026 | Call for Participation Opens |
| 9 October, 2026 | Deadline for Tool Submission |
| 13 November, 2026 | Notification of Competition Results |
| 4 December, 2026 | Deadline for Solution Paper Submission |
| 16 December, 2026 | Solution Papers Response |
| 6 January, 2027 | Deadline for Solution Paper Revision |
| 13 January, 2027 | Solution Papers Final Notification |
| 20 January, 2027 | Deadline for Camera-ready Material |
| TBA | REST League Awards Ceremony |
TBA
To quickly develop your REST API testing tool, you can consider using the RestTestGen Framework. The framework provides many ready-to-use components to:
All components and the testing engine are easily customizable, to develop brand new testing tools in minutes. For more details, see the RestTestGen Framework Wiki.
Before submitting your tool to REST League, please check its compliance with RESTgym for easy deployment and running on our platform.
You can test your tool on the REST APIs provided in the apis directory of this repository (see the specific apis/README for details) and on the APIs available in RESTgym.
REST League is organized by
michele.pasqua@univr.itmariano.ceccato@univr.itsofia.mari@univr.itdavide.corradini@uni.luIf you have questions, feel free to reach out to us!
M. Kim, S. Sinha, and A. Orso. “Adaptive REST API Testing with Reinforcement Learning”. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering. ASE2023. IEEE Press, 2024, pp. 446-458. ↩