GSoC 2025: Specification Conformance Tests

Author: Anshu Kumar, a contributor to Google Summer of Code 2025
GSoC project link

1. About the Project
2. Problem statement
3. Solution Overview
   1. Description
   2. Key Features
   3. Core Working Example
   4. How it works
4. What lies ahead
5. Learnings and experience
6. PRs and Issues

This project aims to provide a concerto conformance test suite which will validate the semantic correctness of the core features of Concerto like the Modelmanager and Modelfile.

Concerto, the core modeling language of the ACCORD PROJECT, is critical for smart legal contracts and domain modeling. This project aims to migrate from the current testing framework mocha to vitest which will enhance the performance, execution time and debugging capabilities. This process can be automated by the combination of vitest-codemod, regex transformations, ai assistance and some manual scripts required for handling complex test cases which cannot be automated. Here are the deliverables: 1. Migration from Mocha, Chai, Sinon to Vitest for all Concerto implementation. 2. A new Concerto conformance package which will be a standalone validation test package for all Concerto implementations.

The Concerto Conformance Test Suite is a cross-language validation framework designed for the Accord Project’s Concerto modeling language.

It ensures that any modifications to core components like the ModelFile, or ModelManager continue to comply with a defined set of semantic validation rules.

Currently supporting JavaScript and Rust, the framework is extendable to other languages such as C#, and beyond. To test any Concerto implementation, only a corresponding step definition file is required.

1. Semantic Rule Validation
   The suite tests a wide range of semantic rules, including:

   1. Property and class naming conventions
   2. Namespace conflicts
   3. Invalid enum definitions
   4. Incorrect import structures

2. Model Fixtures (Valid & Invalid)

   1. Provides a curated set of valid and invalid model files (CTO) that serve as test fixtures.
   2. Valid models ensure that correctly defined Concerto structures pass validation.
   3. Invalid models are intentionally broken to verify that implementations catch and throw errors correctly.

3. Cucumber-Based Testing

   1. Uses Cucumber (.feature files + step definitions) for human-readable test scenarios.
   2. Enables structured validation with clear pass/fail outcomes.

4. Dynamic Source Loading

   1. Supports dynamic loading of Concerto implementations.
   2. Uses environment variables or package managers (e.g., npm for Node.js, cargo for Rust) to import ModelFile and ModelManager.
   3. Makes the suite flexible and language-agnostic.

5. CI/CD Integration

   1. Easily integrated into a project’s CI/CD pipeline.
   2. Ensures that semantic validation runs automatically on every commit or pull request.
   3. Helps maintain long-term consistency across multiple Concerto implementations.

Here is a simple example of how the conformance test suite works internally:

1. Valid CTO Model:

2. Invalid CTO Model (Supertype Missing):

3. Corresponding Cucumber Scenarios:

1. Model Parsing:
   CTO files are converted into Abstract Syntax Trees (ASTs).
2. Validation Execution:
   ModelFile and ModelManager implementations are used to validate ASTs.
3. Pass/Fail Checks:
   Valid models -> pass with no errors.
   Invalid models -> fail with clear error messages.

This ensures semantic correctness across all Concerto implementations.

1. More semantic rules:
   Currently the repository has tests on over 30+ semantic rules.
   Can have more rules if needed in the future to increase the robustness of the conformance test repository.

2. Support for more runtimes:
   The repository currently suppports Javascript and Rust implementation of Concerto but can support to more runtimes by adding only a step file corresponding to the runtime in the future.

This summer, as a participant in Google Summer of Code 2025, I had the opportunity to work with a very supportive organzation Accord Project under the guidance of my great mentors, Ertugrul Karademir and Dan Selman. During the journey of building a robust testing suite, I not only strengthened my technical skills but also learned the importance of collaboration and how a professional team functions. I gained experience in setting task priorities and working on creating something new, which has been an invaluable part of my learning journey.

While doing this project, I worked on and learnt about several things like Behaviour Driven Development and Test Driven Development. Along with this, I also got the opportunity to get to know about the insights about how Concerto works, what are it’s core components, what does they do , how does the components work together and more stuffs like this. While making the conformance tests, I had to ensure that it can run for different runtimes of Concerto like Javascript and Rust. So due to this I also got the opportunity to learn Rust and implement the requirements. This journey really helped me to increase my technical skills.

My mentors used to host weekly meetings to guide and support me throughout this journey.
Also apart from weekly meetings with mentors we used to have a weekly technology working group call where several members of the organisation along side with the GSoC participants discussed on some of the changes which have been made or which have to be made within the different repositories. I learnt a ton of things from these meetings.

I am really grateful to my mentors Ertugrul Karademir and Dan Selman as well as the whole Accord Project community for the constant guidance and support.

Overall GSoC 2025 was an amazing experience for me.
I’ll continue to contribute to Accord Project and remain as an active part of this amazing community.