Stack
This chapter proposes engineering choices, naming conventions, and operational patterns for the NDE network which are not yet endorsed by NDE. Feedback welcome via the issue tracker.
Introduction
The NDE Stack is the working name given here to the ecosystem of components the Stack provides, the NDE-compatible products that fill its layers, and the operational patterns that compose them, grounded in the network’s shared standards: SCHEMA-AP-NDE, LDES, IIIF, DCAT-AP. The Stack’s goal is to help software developers in the NDE network solve shared functionality once rather than reinventing it in each project. It is what NDE offers builders to work with; the NDE-operated network services it works on – the Dataset Register, Network of Terms, and Dataset Knowledge Graph – are composed and consumed, not part of the Stack a builder deploys. Most of what appears here is new: proposed components and patterns yet to be built. The rest is existing software given a role in the Stack.
The Stack spans the whole path heritage data travels through the network, across all four layers of the NDE network architecture: from being managed and published in Data Platforms to being exposed and finally presented to end users in Service Platforms. Within that breadth, this documentation goes deepest on the Service Platforms, because that is where development work will focus in the near future. For Data Platforms, this documentation can rely more on existing solutions.
These chapters are also meant to create a shared language and understanding across the network: a common vocabulary for the components and patterns that builders, operators, and decision-makers can use when discussing what the Stack does and how it fits together. Naming the parts is the prerequisite for talking about them coherently across teams and organisations.
Reading guide
This is: the Stack described across the four layers above, at the engineering level: concrete components (existing and proposed), the default operational patterns that compose them, the standards they rest on, and the foundational technologies they depend on.
This isn’t: a description of any single product, nor a restatement of an existing document.
Relation to Van data naar dienst (NDE, November 2025): the report is a vision covering two of the four layers, discussing only what happens inside a Service Platform. For those two layers this guidance is a bridge from the report’s architectural vocabulary to running code: the report describes what should happen at each step, the Stack proposes how, with named patterns and packages. The Stack’s other layers have no report counterpart.
Goes beyond the report where useful: some practical engineering concerns are not in the report, such as semantic search, snapshot-CDC (change data capture) deletion handling, per-source outage resilience, declarative standards-backed pipeline configuration. The Stack picks these up as natural extensions of the report’s framework, flagged in context where they appear so a reader can tell report-grounded content from Stack-direction extensions.
Audience: primarily builders of Service Platforms and network services within the NDE network. Data Providers are a secondary audience: the Publication Layer is in scope, but what they publish is specified in Requirements rather than here.
Status of contents: many components are proposals (@lde/* or @ndes/* packages that do not exist yet). The function-mapping table marks them as such. Patterns labelled “Proposed” have been discussed but not endorsed.
Taxonomy
The Stack uses a small vocabulary consistently.
| Term | Definition | Examples |
|---|---|---|
| Component | Software the Stack provides | @lde/* packages, @ndes/* packages |
| Pattern | Operational mechanic | Blue/green Rebuild, SCHEMA-AP-NDE-first, Ports & Adapters |
| Service | A running instance of a Component, deployed with a specific configuration | A Service Platform running the search projection with its own SHACL and search configuration |
| Network service | An NDE-operated, network-wide endpoint the Stack builds on rather than provides: consumed via its canonical URL, not deployed by Stack users. DERA’s netwerkvoorziening | Dataset Register, Network of Terms, Dataset Knowledge Graph |
| NDE-compatible product | Third-party software that fills a layer by being NDE-compatible: part of the ecosystem, provided by its vendor rather than the Stack, its requirements specified in Requirements rather than here | Omeka-S, collection management systems, annotation stores |
| Standard | A network commitment the Stack adopts | SCHEMA-AP-NDE, LDES, IIIF, DCAT-AP 3.0 / Schema.org Dataset |
| Foundational technology | Upstream open-source dependency outside NDE governance | QLever, Typesense, nginx, Fastify, Mercurius |
Components
The Stack provides the components below – the software a builder deploys. These are distinct from the network services NDE operates and the Stack builds on: where a component below reaches a network service, the service is consumed, not part of the deployed Stack. The DKG, for instance, is a network service reached through the Knowledge Graph APIs, not a component a builder runs. They live in the Service Platform side chapter; this catalog is a quick index, and the Pipeline chapter shows how the pipeline components compose.
| Component | Layer | Brief |
|---|---|---|
| Search Pipeline (proposed) | Data | Builds a search index of records from selected datasets |
| Knowledge Graph Pipeline | Data | Builds a queryable knowledge graph from selected datasets |
| Search APIs (proposed) | Data | Search and filter API that Presentation Layers consume |
| Knowledge Graph APIs | Data | Query interfaces for the Stack’s knowledge graphs: DKG (operational), Term Backlink Graph (proposed), Knowledge Graph voor Termen (proposed), and any self-operated KG a Service Platform builds |
| Change Stream Producer (proposed) | Publication → Data | Detects changes in Data Layers that do not publish a change stream themselves, and republishes them as the LDES feed Service Platforms can subscribe to |
| Heritage UI Components (future) | Presentation | Reusable display components |
Foundational technologies
The Stack is built on top of mature open-source infrastructure that lives outside NDE/LDE governance. These are dependencies, not Stack components; release cycles, roadmaps, and breaking changes follow upstream projects. The Stack picks opinionated defaults and treats them as exchangeable for any conformant alternative; substitutes plug in behind ports defined by the Ports & adapters pattern, so a swap is a configuration concern rather than a code rewrite.
Defaults favour operational lightness: solutions that are resource-efficient and simple to run on NDE’s national infrastructure as well as by individual service providers on their own hardware, so the same Stack stays realistic to operate at every scale. That criterion is why the default search engine is Typesense (rather than the heavier Elasticsearch) and QLever the RDF store.
| Concern | Stack default | Realistic substitutes |
|---|---|---|
| RDF store / SPARQL engine | QLever – read-only-after-load, fast bulk-load, fits blue/green rebuild | Oxigraph, GraphDB, Jena Fuseki |
| Search engine | Typesense – used by the search pipeline | Elasticsearch, OpenSearch (each behind their own adapter at the search pipeline’s engine boundary) |
| Reverse proxy | nginx – default for proxy-level blue/green | HAProxy (runtime API for zero-reload switching), Caddy (admin-API hot reload), Envoy |
| Web / API runtime | Fastify – one runtime for both API styles: REST via @fastify/swagger for the OpenAPI surface (used by the Dataset Register API today), GraphQL via Mercurius (proposed @lde/graphql-server) | REST: any OpenAPI-capable framework. GraphQL: Apollo Server, Yoga, any GraphQL.js-based runtime |
Substrates
Three distinct bodies of source data (substrates) underlie the Stack. Each pipeline rides on exactly one. They differ in source, scale, cadence, and consumers; the layer pages refer back to them by letter.
| Substrate | What’s crawled | Source(s) | Scale / cadence | Feeds |
|---|---|---|---|---|
| A. Dataset descriptions | DCAT-AP metadata about datasets (titles, publishers, distributions, licenses, subjects) | Publishers’ DCAT-AP descriptions, harvested by the NDE Datasetregister | Small, metadata-only. Refresh frequently (daily) | Enumeration for B: which datasets exist and where their distributions live. Catalog Search Pipeline over the descriptions themselves (the Dataset Register browser), enriched with DKG facets |
| B. Metadata records | Metadata records inside each distribution (instances of CreativeWork, Person, Place, …) | Per-dataset SPARQL endpoint or RDF dump | Large, per-dataset. Refresh per source on last modified date | Object Search Pipeline (records inside distributions); Dataset Knowledge Graph; Term Backlink Graph (data-model-agnostic vocab walk) |
| C. Terms | Terms and the relations between them, across terminology sources | Terminology sources, aggregated by the NDE Network of Terms | Medium, vocabulary-scoped. The set of sources is stable, but terms change (GTAA tracks current events); refresh per source on update | The report’s “Knowledge Graph voor Termen” (function 5) |
In DERA terms. The substrates map onto DERA’s bedrijfsobjecten: C is Term (“aanduiding van een entiteit of onderwerp opgenomen in terminologiebron”) held in a Terminologiebron; A and B are both Metadata (“gegevens die context, inhoud, structuur en vorm van informatie en het beheer ervan beschrijven”), with B describing Informatieobjecten and Cultuurhistorische objecten.
One discrepancy worth signalling to the DERA-Architectuurraad: DERA does not distinguish A from B. Its Dataset is “verzameling van metadata, al dan niet aangevuld met informatieobjecten” – the collection itself – and it names no object for the DCAT-AP description record about a dataset, which is what the Datasetregister actually harvests. In DERA’s model both a dataset description and an object description are simply Metadata. The Stack splits them because they differ in source, scale, cadence, and consumers, as the table above shows.
Observations that fall out of this separation:
- A enumerates B, not C. B’s pipelines read the Register to learn which datasets exist and where their distributions live. C is enumerated separately, from the Network of Terms catalogue of terminology sources. Part of that catalogue may migrate into the Register over time, but external sources like GeoNames, AAT and Wikidata stay outside it, so C keeps its own enumeration.
- Object search over B is the norm; only the register indexes A. A Service Platform’s Search Pipeline reads the register (substrate A) only to enumerate which datasets to crawl; the records it ingests are the objects inside their distributions (substrate B) – each
CreativeWork,Person, orPlace. The dataset descriptions themselves never enter that index. The one exception is the Dataset Register’s own browser: being the catalog, it indexes substrate A directly – the dataset descriptions are its records – enriched with DKG facets. Same pipeline and stages either way; only the substrate and record grain differ. - B carries multiple projections. The same crawl feeds three structurally different sinks: an AP-aware search-document projection (Search Pipeline), a VoID (Vocabulary of Interlinked Datasets) statistical summary (Dataset Knowledge Graph), and a data-model-agnostic term-backlink projection (Term Backlink Graph). All three are enumerated from A but compute over B’s contents: what differs is the projection, not the substrate.
- Change cadences differ. B changes most often: records are added, updated, and removed at source as collections grow. A changes less often: dataset metadata is updated more rarely than the records it describes. C is most stable: vocabularies move slowly. Stack components inherit the change cadence of the substrate they ride on.