Check your Ingredients using Enduro - Artefactual at the iPres BakeOff

Have you ever cooked a meal only to realize at the end that one of your ingredients was off? Maybe your flour was a little past date, or maybe you used sugar instead of salt. This can ruin your whole dish!

At the iPres 2024 BakeOff, Managing Director Justin Simpson explained how Artefactual’s systems can help you feed a whole designated community with well baked AIPs.  Using the cooking analogy, an AIP is like a digital dumpling: a universal idea that is also specific to each culture and community in the form of gyoza, pierogi, wontons, manti, ravioli, samosas, etc. Your unique content serves as the filling, metadata is like spices and flavouring, and Archivematica cooks these into a dumpling by adding more metadata and packaging it all into a container.

At scale, the process of ingest - choosing and preparing the right content for your many, many dumplings - is critical.  Often you need to avoid bad ingredients that can spoil your AIPs by violating your organization’s “recipe” (business rules for digital preservation).  At Artefactual we have started to build a new set of tools we are using to make sure you are cooking with the right ingredients.  The main tool in this group is called Enduro; it is the core of a Secure Digital Preservation System, or SDPS, hence the name of the github organisation.

At the BakeOff, Justin focused on ingest, a key process included in the OAIS reference model, by demonstrating ingest with a SIP specification used by one of our clients.

Ingest may seem simple, but it is built from a couple key components. It requires organizations to have a recipe that includes organizational rules for required metadata, acceptable formats, and quality, that can then be “cooked” using technology that supports implementation of these rules – just the kind of technology Artefactual provides. And because it comes relatively early in the preservation process, good ingest can make or break your final dish.

Once Enduro has your SIP, it performs a few key steps:

• Checking the physical structure of the SIP

• Ensuring that all the files contained within the SIP are present, and verifying their checksums

• Ensuring that all of the files are in allowed file formats

• Ensuring that required metadata is present and uses the correct controlled vocabularies

• Ensuring that the SIP has been packaged properly by the Producer

Once these steps have been completed, Enduro passes the SIP on to Archivematica, to make a delicious AIP. 

At the BakeOff, Justin ingested content from the pantry (files offered as ingredients by the iPres organizers) that included a PowerPoint file. And the SIP was rejected because PowerPoint was a disallowed format under the business rules in this context. If this took place in a real organization, the Producer would have the opportunity to resubmit their content after removing the disallowed formats. Enduro and Archivematica are able to accept Powerpoint files, but the business rules that were defined in the workflow that was demonstrated did not permit them. 

Photo by iPRES 2024, CC BY-SA 4.0

We believe that supporting organizations at ingest in this way will help make sure you’re baking up high-quality AIPs for preservation, but we want to know what you think too! How does your organization manage ingest now? How might that change with a tool like Enduro to help?