Skip to main content

We are the Hypermedia!

· 7 min read
Koen Van Daele
Onroerend Erfgoed

Within the Flemish Government, a central body coordinates policy on Information and ICT. Under its umbrella, several taskforces operate, including the OSLO Datastandards taskforce, which develops standards, semantic models and tooling for Flemish government agencies and partners. One of its 2018 projects was a Generic Hypermedia API standard — guidelines and building blocks to make REST services more hypermedia-driven and therefore more interoperable. Its first version, approved on November 8, 2018, covers three building blocks: language, CRUD and pagination.

Pagination. Web scale​

At Flanders Heritage we've started aligning our information systems with this standard, particularly the pagination building block. For now we're adopting the HTTP Link header method of pagination, as specified here and detailed in RFC 5988.

To see this in action, look at the Link HTTP header — via a command-line tool like cURL, a browser plugin, or a standalone REST client. We'll use cURL, since it's generally available or easy to install on Linux and Mac.

Say we're interested in books and documents about Knokke, a coastal town in Flanders. Looking at the REST API docs at https://bib.onroerenderfgoed.be/api_docs tells us we can call https://bib.onroerenderfgoed.be/werken with a titel parameter, so https://bib.onroerenderfgoed.be/werken?titel=knokke gives us books with Knokke in the title, or a chapter title:

$ curl -G -i -H Accept:application/json https://bib.onroerenderfgoed.be/werken?titel=knokke
HTTP/1.1 200 OK
Date: Mon, 16 Dec 2019 19:51:07 GMT
Server: Apache
Content-Type: application/json
Content-Length: 3933
Access-Control-Expose-Headers: Content-Range, X-Content-Range
Accept-Ranges: items
Content-Range: items 0-9/167
Link: <https://bib.onroerenderfgoed.be/werken?titel=knokke&per_pagina=10&pagina=2>; rel="next", <https://bib.onroerenderfgoed.be/werken?titel=knokke&per_pagina=10&pagina=17>; rel="last", <https://bib.onroerenderfgoed.be/werken?titel=knokke&per_pagina=10&pagina=1>; rel="first"
X-Content-Type-Options: nosniff
Access-Control-Allow-Origin: *
X-Permitted-Cross-Domain-Policies: none
X-XSS-Protection: 1; mode=block

[{"publicatietype": {"url": "http://purl.org/ontology/bibo/Document", "naam": "Document", "id": 1}, "titel": "Het landelijke Knokke", "self": "https://bib.onroerenderfgoed.be/werken/552291", "uri": "https://id.erfgoed.net/infocat/werken/552291", "auteur": "D'hont A.", "id": 552291}, "... 9 more results ..."]

This curl call fetches books and documents (werken) from our library catalog with knokke in the title, and asks it to print the headers. The one we care about here is Link:

Link: <https://bib.onroerenderfgoed.be/werken?titel=knokke&per_pagina=10&pagina=2>; rel="next", <https://bib.onroerenderfgoed.be/werken?titel=knokke&per_pagina=10&pagina=17>; rel="last", <https://bib.onroerenderfgoed.be/werken?titel=knokke&per_pagina=10&pagina=1>; rel="first"

We got results back, but only a partial set. The Link header tells us there's a next page, a last page and a first page — and since we're on page 1, no previous. We can follow these links to page through the results, or jump straight to the end. Importantly, a client shouldn't assume anything beyond what the links say: you might be tempted to infer, from 17 pages of at most 10 records each, that there are 161–170 results, but the server isn't obliged to make that arithmetic hold, or to put any other information in the link. This does expose a real limitation of Link-header pagination: there's no way to tell how many results there are in total, or how many pages remain. All the client knows is whether there's a next link; if there isn't, you're on the last page, and if next and last are identical, you're on the second-to-last. On the upside, the client needs zero knowledge of the URL's internal structure — no page vs. pagina, no limit/offset — which makes it easy to reuse the same client code across very different webservices. All it needs to know is: if there's a next link, there's more to fetch.

To get the second page:

$ curl -G -i -H Accept:application/json https://bib.onroerenderfgoed.be/werken?titel=knokke\&per_pagina=10\&pagina=2

All your data belongs to us!​

To show this off, we wrote a small Python script that queries several of our information systems and handles pagination via the Link header automatically. In Python, the requests library is the standard tool for REST calls, and conveniently it already has built-in support for the Link header, making it trivial to iterate over every result. For this example, we fetch all images (see Searching the image database), themes (see Themes, people and events), heritage and designation objects (see Heritage objects and designation objects) for the municipality of Knokke-Heist. It's a short script that drops a large pile of URIs at your feet, chaining calls to the various APIs and following each next link along the way.

Running it prints a long list of URIs — functional, but not exactly satisfying: a URI is great for a machine, but doesn't tell a human much on its own.

No context, no information​

Another interesting technique in the Linked Data world is JSON-LD (JSON for Linking Data), which turns plain JSON into linked data by mapping it to RDF through a context — defining which JSON keys correspond to which RDF properties. Here's a simple context you could use on our image database endpoint:

{
    "dct": "http://purl.org/dc/terms/",
    "title": "dct:title",
    "description": "dct:description",
    "uri": "@id"
}

This defines a namespace shorthand dct for the Dublin Core Terms namespace, so dct:title stands in for the full http://purl.org/dc/terms/title, and maps uri to @id, JSON-LD's way of marking a record's unique identifier. Applying this context to an image from our image database gives:

{
    "@id": "https://id.erfgoed.net/afbeeldingen/392715",
    "http://purl.org/dc/terms/title": "Villa Cools met kinesistenpraktijk",
    "http://purl.org/dc/terms/description": ""
}

This tells us there's a resource identified by https://id.erfgoed.net/afbeeldingen/392715 that can be described with a Dublin Core title. It's common in JSON-LD to also assign a @type, a URI pointing to a class in an RDF vocabulary, so records from different sources stay distinguishable once merged:

{
    "@id": "https://id.erfgoed.net/afbeeldingen/392715",
    "@type": "http://purl.org/dc/dcmitype/Image",
    "http://purl.org/dc/terms/title": "Villa Cools met kinesistenpraktijk",
    "http://purl.org/dc/terms/description": ""
}

Once more, with feeling​

Let's use JSON-LD to improve our script. We want a homogeneous dataset covering everything known about Knokke-Heist, with every record sharing the same structure — so we built two JSON-LD contexts, one for the image database and one shared across the three closely related inventory datasets. Since none of these datasets normally include their own resource type, we add that ourselves. Our inventory system also has a locatie_samenvatting field summarising an object's geographic location — the underlying summarisation logic can get fairly involved (some things sit in more than one street, municipality or even province), but it's very handy for a human glancing at results. The image database has no such field, so we compute a simple equivalent for it (image locations are much simpler than those of heritage objects).

The script fetches everything as before, adds the record type and a location summary for images, then expands it all into JSON-LD using the Python pyLD library. That leaves us with valid but unwieldy JSON-LD — every key is now a URI. So we compact it again, using a third context based on Dublin Core Terms, and print the result.

To run this script, install requests and pyLD first. It may take a while for larger municipalities like Gent or Antwerpen. As a bonus, we also wrote a variant that writes the resulting dataset to a CSV instead of printing it — feel free to adapt either script to build your own dataset for another municipality.

If Hypermedia APIs and Linked Data interest you, https://data.vlaanderen.be/ has a lot more background and points to the relevant standards and documents, and there's a whole community of Linked Data people happy to answer questions — you'll find a fair number of them on Twitter. As always, feel free to get in touch with further questions.