# A Query Validator for the<br>Common Services of the<br>Once-Only Technical System

_The Once-Only Technical System (OOTS) enables Member States of the European
Union (EU) to exchange evidences such as diplomas, birth certificates, and
business permits. Its legal foundation is the Single Digital Gateway (SDG)
Regulation ([Regulation (EU) 2018/1724](eu2018/1724)). At the heart of the OOTS
are the Common Services (CS): three registries that store information about
where evidences can be found, when they are required, and how they relate to
each other._

_In this memo, I present a lightweight validator for query requests to the CS:
the OOTS CS Query Validator (OCQV). The development of the OCQV is intended to
deepen my understanding of the technical design of these registries and to
enable independent testing of query requests without relying on the
availability of the actual CS endpoint. It validates whether requests to the CS
conform to the documented interface and structure, and reports the results in a
structured XML format._

> **Acknowledgement:** At the time of writing this memo, I serve as the
> designated National Coordinator (NC) for the SDG on behalf of the
> Netherlands. However, the work that led to this memo and the associated
> software was carried out in my own time and is not supported or endorsed by
> the Netherlands or the European Commission (EC) in any way.
>
> While I do have access to information beyond what is publicly available due
> to my role as NC SDG, this memo and the software are based solely on public
> sources, which are cited accordingly. For the standing policy, please refer
> to my [vision on the SDG in the Netherlands][vision] (only available in
> Dutch).

## Specification

Before implementing the OCQV, I first need to understand how the CS of the OOTS
are intended to communicate the information they contain. The CS are described
at a technical level in documentation available on the [OOTS Hub][hub]. To make
my own life a bit easier, I've reproduced here the descriptions of the three CS
registries, as presented on the OOTS Hub:

- The **Data Service Directory** (DSD):
  > "The Data Service Directory maintains a catalogue of evidence providers
  > with the evidence types they are able to provide upon request using their
  > Data Services. It is used in the evidence exchange process by the evidence
  > requesters to discover the evidence providers that can provide the
  > evidences they require, and the required metadata."

- The **Evidence Broker** (EB):
  > "The Evidence Broker publishes which types of evidence Member States can
  > provide to prove a particular requirement of a procedure. Using the mapping
  > from criteria or information requirements to possible evidence types, it
  > can find the evidence types that can prove that the User fulfils the
  > requirements of the procedure."

- The **Semantic Repository** (SR):
  > "The Semantic Repository provides commonly agreed semantic specifications
  > for the exchange of evidences. The service provides the following
  > functionalities: ability to externally reference data models from other
  > components; ability to define and extract subsets of models; provision of
  > documentation."

### Queryable Registries

The DSD and EB expose a REST API based on the [OASIS ebXML RegRep V4][oasis]
standard. This standard defines a protocol for querying using HTTP `GET`
requests at each URL following a specific pattern:

```text
<server-base-url>/rest/search?queryId=<query-id>(&<param-name>=<param-value>)*
```

This means that each possible query has its own identifier (ID). Before listing
each available ID, it is useful to note that they follow a consistent structure
(where `<cs>` is either `dsd` or `eb`):

```text
urn:fdc:oots:<cs>:ebxml-regrep:queries:<query-specific-id>
```

Specifying the ID is mandatory, even though this diverges from the OASIS
specification, which states that the ID defaults to
`urn:oasis:names:tc:ebxml-regrep:query:FindObjectById` if omitted. All other
parameters are optional. The [OOTS API documentation][oots] confirms this: the
DSD and EB are only partial implementations of a RegRep server. As such,
several canonical queries and parameters defined by the RegRep standard are not
available and will result in an error. Therefore, I can limit my investigation
to the OOTS-specific documentation alone.

#### Data Service Directory

The DSD exposes a single endpoint for querying data services. Each query has
mandatory and optional parameters (with names that use a mixture of camelCase
and snake-case conventions). In the overview below, optional parameters are
displayed in _italics_.

| Query                                    | Parameter                      | Description                                                                                                                                       |
|:-----------------------------------------|:-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
| Find Data Services of Evidence Providers | `queryId`                      | `requirements-by-procedure-and-jurisdiction`                                                                                                      |
|                                          | `evidence-type-classification` | The classification code, encoded as a URL according to [RFC 3986][rfc3986]. Retrieved from the EB.                                                |
|                                          | `country-code`                 | The jurisdiction of the procedure, expressed using the [EEA Country subset of the ISO 3166-1 alpha-2][iso3166-1a2-eea] country code standard.     |
|                                          | _`jurisdiction-admin-l2`_      | Second-level administration code, expressed as a [NUTS code][nuts].                                                                               |
|                                          | _`jurisdiction-admin-l3`_      | Third-level administration code, expressed as a [LAU code][lau].                                                                                  |
|                                          | _`jurisdiction-context-id`_    | Only required when responding to a DSD exception, in which case its value is specified.                                                           |
| __                                       | _`specification`_              | The profile used to present the result of the query. Defaults to the OOTS EDM if not specified.                                                   |

#### Evidence Broker

The EB exposes two endpoints for queries. It provides the data that is used to
query both evidence types in the EB itself, and data services in the DSD.
The EB queries are listed below in the same manner as for the DSD.

| Query                    | Parameter                 | Description                                                                                                                                       |
|:-------------------------|:--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
| Get List of Requirements | `queryId`                 | `requirements-by-procedure-and-jurisdiction`                                                                                                      |
|                          | _`procedure-id`_          | A procedure code, as defined in the [procedures codelist][procedures].                                                                            |
|                          | _`country-code`_          | The jurisdiction of the procedure, expressed using the [EEA Country subset of the ISO 3166-1 alpha-2][iso3166-1a2-eea] country code standard.     |
|                          | _`jurisdiction-admin-l2`_ | Second-level administration code, expressed as a [NUTS code][nuts].                                                                               |
| __                       | _`jurisdiction-admin-l3`_ | Third-level administration code, expressed as a [LAU code][lau].                                                                                  |
| Get Evidence Types       | `queryId`                 | `evidence-types-by-requirement-and-jurisdiction`                                                                                                  |
|                          | `requirement-id`          | The requirement ID, encoded as a URL according to [RFC 3986][rfc3986]. Either known a priori or retrieved from the EB with a query.               |
| __                       | _`country-code`_          | The jurisdiction of the procedure, expressed using the [EEA Country subset of the ISO 3166-1 alpha-2][iso3166-1a2-eea] country code standard.     |

### Semantic Repository

The SR exposes its semantic assets and associated metadata via HTTP, where each
asset has its own URI, as defined in [RFC 3986][rfc3986]. In practice, this
means that every asset has a path associated with it that is appended to the
domain of the SR. The live version that is hosted by the EC is found at
`sr.oots.tech.ec.europa.eu`. In the following overview I've compiled all
available asset types and their associated routing (i.e. the path used to access
an instance of such an asset).

| Asset                       | Path                                    |
|-----------------------------|:----------------------------------------|
| Evidence                    | `/evidencetypeclassification/HR/<uuid>` |
| Procedure                   | `/codelist/procedures/<id>`             |
| Requirement                 | `/requirements/<uuid>`                  |
| Codelist                    | `/codelists/<label>`                    |
| Example                     | `/examples/<uuid>`                      |
| Schematron                  | `/schematrons/<label>`                  |
| Technical Specification     | `/specifications/<label>`               |
| Semantic Data Specification | `/datamodels/<label>`                   |
| Schema                      | `/schemas/<label>`                      |

## Implementing the Validator

The OCQV accepts the same queries as the DSD and EB and generates a report on
their completeness and validity. The SR, by contrast, does not accept queries
but only exposes resources. At this stage, I see little value in validating
those requests, so the SR is excluded from the OCQV.

### Returning a Response

The technical specification describes expected CS query responses in great
detail. For my purposes, however, these responses are not relevant: the OCQV
focuses solely on validating the requests made by a (prospective) evidence
requester.

The validator checks whether the parameters in a query are valid according to
the specification and produces a report in XML. An XML Schema definition for
these reports is included below.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
           targetNamespace="https://www.parcifal.dev/ns/oots-cs-query-validator/2025-08"
           xmlns:ocqv="https://www.parcifal.dev/ns/oots-cs-query-validator/2025-08"
           elementFormDefault="qualified"
           attributeFormDefault="unqualified">

    <xs:simpleType name="versionType">
        <xs:restriction base="xs:string">
            <xs:pattern value="\d+\.\d+\.\d+"/>
        </xs:restriction>
    </xs:simpleType>

    <xs:simpleType name="verdictType">
        <xs:restriction base="xs:string">
            <xs:enumeration value="valid"/>
            <xs:enumeration value="invalid"/>
            <xs:enumeration value="unknown"/>
            <xs:enumeration value="missing"/>
        </xs:restriction>
    </xs:simpleType>

    <xs:element name="ParameterEvaluation"
                type="ocqv:ParameterEvaluationType"/>

    <xs:complexType name="ParameterEvaluationType">
        <xs:complexContent>
            <xs:restriction base="xs:anyType">
                <xs:attribute name="name"
                              type="xs:string"
                              use="required"/>
                <xs:attribute name="value"
                              type="xs:string"/>
                <xs:attribute name="verdict"
                              type="ocqv:verdictType"
                              use="required"/>
            </xs:restriction>
        </xs:complexContent>
    </xs:complexType>

    <xs:element name="QueryReport">
        <xs:complexType>
            <xs:sequence>
                <xs:element ref="ocqv:ParameterEvaluation"
                            maxOccurs="unbounded"/>
            </xs:sequence>
            <xs:attribute name="version"
                          type="ocqv:versionType"
                          use="required"/>
            <xs:attribute name="apiVersion"
                          type="ocqv:versionType"
                          use="required"/>
            <xs:attribute name="createdAt"
                          type="xs:dateTime"
                          use="required"/>
        </xs:complexType>
    </xs:element>
</xs:schema>
```

The paths where queries can be submitted mirror those of the real CS. Below
is a description of the attributes of `ocqv:QueryReport` and
`ocqv:ParameterEvaluation`.

| Element                    | Parameter    | Description                                                                                                                 |
|:---------------------------|:-------------|:----------------------------------------------------------------------------------------------------------------------------|
| `ocqv:QueryReport`         | `xmlns:ocqv` | The namespace of the evaluation report, which allows for consecutive versions.                                              |
|                            | `version`    | The version of OCQV that generated the report, in the form `<major>.<minor>.<patch>`.                                       |
|                            | `apiVersion` | The version of the CS API against which the query was validated, in the form `<major>.<minor>.<patch>`.                     |
| __                         | `createdAt`  | The UTC timestamp of when the report was generated, in [ISO 8601][iso8601] format: `YYYY-mm-DDTHH:MM:SSZ`.                  |
| `ocqv:ParameterEvaluation` | `name`       | The name of the parameter, either as specified in the query or as defined in the OOTS documentation.                        |
|                            | `value`      | The value of the parameter, if provided in the query.                                                                       |
| __                         | `verdict`    | Indicates whether the query parameter is valid, invalid, unknown (i.e. not part of the official documentation), or missing. |

### Architecture

The validator is implemented in Python using the [Werkzeug framework
by Pallets][werkzeug]. I am well-versed in Python and familiar with Werkzeug,
mainly through [Flask][flask]. Since the evaluator does not require templating,
internationalization, or other Flask features, implementing it directly in
Werkzeug was the most straightforward approach.

The intention is to make a public instance of the validator available. New
versions are published through a GitLab CI pipeline. The code is open-source
and hosted on a [GitLab repository][gitlab]. In addition, a [Docker image][docker]
is available containing the validator as a WSGI application, and it is also
published on [PyPI][pypi], installable via `pip`.

#### Query Schemes

Internally, a `QueryScheme` is constructed for each version of each query.
For example, version $1.0.0$ of the evidence types query of the EB is defined
as:

```python
_QUERY_SCHEME_EB_EVIDENCE_TYPES_1_0_0 = QueryScheme({
    "queryId": defined,
    "requirement-id": url
}, {
    "country-code": COUNTRY_CODE,
    "jurisdiction-admin-l2": NUTS_CODE,
    "jurisdiction-admin-l3": LAU_CODE
})
```

Each query parameter is validated using a corresponding function, which
returns a verdict. For example, the `url` validator is implemented as:

```python
from urllib.parse import urlparse

def url(value: str) -> bool:
    try:
        result = urlparse(value)
        return result.scheme in ("http", "https") and bool(
            result.netloc.strip())
    except AttributeError:
        return False
```

#### Code List Validation

Some parameters, such as procedure ID, country code, NUTS code, or LAU
code, can be validated against pre-defined code lists. A generic code list
decorator checks whether a preloaded and cached list contains a provided
value. Caching is important because some lists, particularly LAU codes,
can be very long and slow to read. Optional regular expressions are used
to filter out obvious mismatches quickly.

The four code list-based evaluator types are defined as:

```python
PROCEDURE_ID = code_list( "Procedures-CodeList.gc", r"^[A-Z]+[1-9][0-9]*$")
COUNTRY_CODE = code_list("EEA_Country-CodeList.gc", r"^[A-Z]{2}$")
NUTS_CODE    = code_list(   "NUTS2024-CodeList.gc", r"^[A-Z][A-Z]+[0-9]+$")
LAU_CODE     = code_list(    "LAU2022-CodeList.gc", r"^[A-Z]*[0-9]+$")
```

#### Extending Query Schemes

Query schemes can be extended to define new versions with only minor
differences. For instance, version 1.2.0 of the evidence types query
made the requirement ID optional and removed jurisdiction levels 2 and 3.
Based on version 1.1.0 (which is identical to 1.0.0), the new scheme is:

```python
_QUERY_SCHEME_EB_EVIDENCE_TYPES_1_1_0 = _QUERY_SCHEME_EB_EVIDENCE_TYPES_1_0_0

_QUERY_SCHEME_EB_EVIDENCE_TYPES_1_2_0 = \
    _QUERY_SCHEME_EB_EVIDENCE_TYPES_1_1_0.extend({
        "requirement-id": None
    }, {
        "requirement-id": url,
        "jurisdiction-admin-l2": None,
        "jurisdiction-admin-l3": None
    })
```

Defining a validator as `None` will omit it from the extended scheme.

#### Query Validation Flow

Using these query schemes, an incoming query is validated in three steps:

 1. **Check for existence**<br>
    If a parameter key is missing, it is marked missing.
 2. **Validate value**<br>
    The appropriate validator is applied. True equals valid; False equals
    invalid.
 3. **Check mandatory parameters**<br>
    Any required parameters that were not evaluated are marked missing.

### Usage

Detailed installation and usage instructions are provided in the project
README on [GitLab][gitlab]. Here, the focus is on the preferred method for
debugging and development.

#### Installing from Source

Clone the repository and install the development dependencies:

```bash
git clone git@gitlab.com:parcifal/ocqv.git
cd ocqv
git submodule update --init --recursive
pip install .[dev]
```

This installs `ocqv` as a Python library and exposes the `ocqv` command,
which can run the validator as a simple HTTP server.

```bash
usage: ocqv [-h] [-H HOST] [-p PORT] [-r] [-d] [-t]

Run the OOTS Common Services Query Validator development server.

optional arguments:
  -h, --help            show this help message and exit
  -H HOST, --host HOST  hostname or IP address to bind to (default: localhost)
  -p PORT, --port PORT  port to listen on (default: 5000)
  -r, --no-reload       disable auto-reloader
  -d, --no-debug        disable interactive debugger
  -t, --threaded        enable multithreading
```

#### Running the Validator Directly

Since the source code is available locally, the script can also be run
directly:

```bash
python3 -m ocqv.cli
```

This loads all code lists into memory and starts an HTTP server on
`localhost`. You can then validate a query using a browser or `curl`:

```text
http://localhost:500/1.2.0/rest/search?queryId=urn:fdc:oots:dsd:ebxml-regrep:queries:dataservices-by-evidencetype-and-jurisdiction&evidence-type-classification=http://example.com&jurisdiction-admin-l2=spam&spam=bacon
```

#### Example Query Report

Accessing the above URL generates an XML report showing parameter
evaluations:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<ocqv:QueryReport xmlns:ocqv="https://www.parcifal.dev/ns/oots-cs-query-validator/2025-08"
                  version="0.1.0"
                  apiVersion="1.2.0"
                  createdAt="2025-08-12T20:47:00.581476+00:00">
    <ocqv:ParameterEvaluation name="queryId"
                              value="urn:fdc:oots:dsd:ebxml-regrep:queries:dataservices-by-evidencetype-and-jurisdiction"
                              verdict="valid"/>
    <ocqv:ParameterEvaluation name="evidence-type-classification"
                              value="http://example.com"
                              verdict="valid"/>
    <ocqv:ParameterEvaluation name="jurisdiction-admin-l2"
                              value="spam"
                              verdict="invalid"/>
    <ocqv:ParameterEvaluation name="spam"
                              value="bacon"
                              verdict="unknown"/>
    <ocqv:ParameterEvaluation name="country-code"
                              verdict="missing"/>
</ocqv:QueryReport>
```

This example demonstrates all possible verdicts. Queries for both the EB and
the DSD are submitted to the same endpoint, just like the real CS.
Currently, API versions $1.0.0$ through $1.2.0$ are supported.

#### Testing and Quality Checks

The project includes unit tests for comprehensive validation of OCQV
functionality:

```bash
python -m unittest discover -s test
```

Code quality can also be evaluated using `pylint`:
```bash
pylint ocqv test
```

## Conclusion

The goal of this project was to gain a deeper understanding of the architecture
and implementation of the CS of the OOTS. Implementing the OCQV required a close
study of the technical specification of queries for the DSD and the EB, which
provided a clear view of the information required to perform a request through
the OOTS.

As a follow-up, I am considering developing a JavaScript library capable of
interacting with both the OCQV and the CS. This could make these services more
accessible to a broader audience, e.g. serving as the backbone for a
user interface that allows CS queries to be constructed, validated, and submitted.

For feedback on this memo or the project, please reach out through any of the
platforms listed in the [colophon](/info.html) of this website. Issues with the
OCQV software can be reported via the [Gitlab issue tracker][issues], and
contributions to the codebase are welcome and encouraged.

*[OCQV]: OOTS CS Query Validator

*[OOTS]: Once-Only Technical System
*[SDG]:  Single Digital Gateway
*[SDGR]: SDG Regulation
*[CS]: Common Services
*[NC]: National Coordinator
*[EC]: European Commission
*[EU]: European Union
*[EEA]: European Economic Area

*[DSD]: Data Service Directory
*[EB]: Evidence Broker
*[SR]: Semantic Repository
*[EDM]: Evidence Exchange Model

*[API]: Application Programming Interface
*[REST]: Representational State Transfer
*[ID]: Identifier
*[URL]: Uniform Resource Locator
*[URI]: Uniform Resource Identifier
*[HTTP]: Hypertext Transfer Protocol
*[HTTPS]: Secure Hypertext Transfer Protocol
*[RFC]: Request for Change
*[DTD]: Document Type Definition
*[XML]: Extensible Markup Language
*[WSGI]: Web Server Gateway Interface
*[CI]: Continuous Integration
*[UTC]: Coordinated Universal Time

*[OASIS]: Organization for the Advancement of Structured Information Standards
*[ISO]: International Organization for Standardization
*[NUTS]: Nomenclature of Territorial Units for Statistics
*[LAU]: Local Administrative Units
*[EAA]: European Economic Area

[werkzeug]: https://werkzeug.palletsprojects.com/en/stable/flask
[flask]: https://flask.palletsprojects.com/en/stable/
[gitlab]: https://gitlab.com/parcifal/ocqv
[issues]: https://gitlab.com/parcifal/ocqv/-/issues
[pypi]: https://pypi.org/project/OOTS-CS-Query-Validator/
[docker]: https://hub.docker.com/r/pcfal/ocqv

[eu2018/1724]: https://eur-lex.europa.eu/eli/reg/2018/1724/oj/eng
[hub]: https://ec.europa.eu/digital-building-blocks/sites/display/OOTS/OOTSHUB+Home
[vision]: https://sdg.pleio.nl/page/view/2b1c9782-3342-4e79-9546-36d6855b19d3/meerjarenvisie-2025
[oasis]: https://docs.oasis-open.org/regrep/regrep-core/v4.0/os/regrep-core-rs-v4.0-os.html#__RefHeading__32747_422331532
[oots]: https://oots.pages.code.europa.eu/tdd/apidoc/
[iso3166-1a2-eea]: https://code.europa.eu/oots/tdd/tdd_chapters/-/blob/master/OOTS-EDM/codelists/OOTS/EEA_Country-CodeList.gc
[nuts]: https://code.europa.eu/oots/tdd/tdd_chapters/-/blob/master/OOTS-EDM/codelists/OOTS/NUTS2024-CodeList.gc?ref_type=heads
[lau]: https://code.europa.eu/oots/tdd/tdd_chapters/-/blob/master/OOTS-EDM/codelists/OOTS/LAU2022-CodeList.gc
[rfc3986]: https://www.rfc-editor.org/rfc/rfc3986
[iso8601]: https://www.iso.org/obp/ui/#iso:std:iso:8601:-1:ed-1:v1:en

[procedures]: https://code.europa.eu/oots/tdd/tdd_chapters/-/blob/master/OOTS-EDM/codelists/OOTS/Procedures-CodeList.gc