Home

Introduction

ids.r2dfoo is a forked version of the main IcatProject IDS component (ids.server).

The name comes from the fact that it is designed to work with Read Only, 2 Level (archive/tape and main/disk storage), and Datafile oriented systems (rather than Dataset where all files in a dataset are stored together in a zip).

It was created in 2021 as part of the Diamond Datastore (DDS) project. Around 10 issues with using the IcatProject IDS had been identified over the years and had never been fully addressed. Initial work aimed to solve the problems with the IcatProject IDS, but after struggling to fix the first couple of issues, and with a better understanding of the codebase, it was clear that what was needed was a different design, better suited to the Diamond use case.

Improvements

The main improvements foreseen were:

• Not having to support two "StorageUnits" (Dataset and Datafile). The DLS IDS server uses Datafile.
• Not having to support writing of files via the IDS. The DLS data is ingested into StorageD directly and is not written via the IDS, mainly because this is faster. Making the IDS read-only means that the various types of locking used in the IcatProject IDS which introduce complexity and unnecessary delays in some cases are no longer a problem.
• Keeping each user's request for data together rather than splitting it up, adding it to a queue, and potentially mixing it in with other user requests, means that progress of the request can be more easily monitored and if necessary the whole request can be cancelled easily.

Approach

So the overall view was that the codebase could be significantly reduced and simplified, most of the identified issues resolved, and new functionality added to track the progress of requests and allow them to be cancelled.

Rather than re-write the entire component, it was decided that there was enough useful code that it was worth forking the IcatProject IDS. This provides the dual benefits of saving development effort by not having to re-write areas of code that were not causing problems, whilst also knowing that they had been well tested over many years of use.

Whilst the main aim was to create something to support the DLS use case, it was recognised that the new component may also need to be used by other RAL based Facilities, and that it may also become a more widely adopted IcatProject component in the future, although the primary goal of the initial development was to produce something that works for DLS.