Skip to content

validation.md

Latest commit

 

History

History
652 lines (604 loc) · 38.1 KB

validation.md

File metadata and controls

652 lines (604 loc) · 38.1 KB
Raw

Validation

Solid Memo checks that the data it reads from a Pod, and every change it writes, has the form the app expects. This page says what's checked and when, what happens to data that doesn't fit, and how it can be repaired.

Solid Memo checks the data it works on against its shapes and the DCAT-AP profile, in the browser:

  • Every instance is checked when it's opened. The user's invalid data policy says what happens to data that fails. What another app wrote only gets warnings (below).
  • Every write is checked before it's saved, so the app never adds invalid data to a pod (write check).
  • What can be repaired is repaired on request; the rest is shown, and can be removed (repair).
  • #/validate?instance=… shows the full report, linked as "Validate this instance" at the bottom of the workspace while developer mode (a per-instance preference) is on.
  • The Studio's health screen shows every user the findings and repairs for the instance or one deck (checkDeck, its entry and documents only), beside what else is wrong with a deck.

No shape constrains text in Markdown, and the app shows any text safely (markdown.md). A published library release is held to npm run library:check's Markdown rules on how its text shows.

Flow

flowchart LR
    accTitle: Validating an instance
    accDescr: The validation screen lists the instance's documents and checks each against its published shape with the SHACL engine.
    ui["ValidationContainer<br/>#/validate"] --> uc["validateInstance"]
    uc --> docs["instanceDocumentUrls<br/>meta, preferences, palettes, catalog,<br/>every deck's cards + reviews,<br/>every draft's documents"]
    docs --> port["ShapeValidator.validateDocument"]
    port --> pod["pod document<br/>(authenticated fetch)"]
    port --> shapes["ns/shapes/&lt;class&gt;/v&lt;N&gt;.ttl<br/>(published with the site)"]
    port --> engine["rdf-validate-shacl<br/>(lazy chunk)"]
Loading

  • validateInstance (application) lists the decks and names every document the instance may hold, including the answer log's month documents and the documents of the drafts its catalog links. It asks the ShapeValidator port about each, naming its context (a draft's is "draft", below). summarize (domain) counts the violations. Nothing is written.

  • Opening an instance runs checkInstance, the same check except:

    • a document still at a version that the instance's digest says conformed, by the same rules, is neither downloaded nor checked again (validateDocumentSince: a 304 is enough), and a conforming document gets a receipt;
    • the rules are named by a hash of every shape and vendored profile file, computed when the site is built (siteDefines in packages/site/src/siteBuild.ts, used by apps/web/vite.config.ts), so a site with other shapes checks everything again;
    • the answer log is left out: answers are checked as they're written, and in the full check, which the developer report always runs.

  • The format update and a library upgrade go through the write check: a document that would fail isn't written (dataNotConforming) and is named with why. The format update goes on with the other documents; a library upgrade stops, as at any failure. Another app's data never stops a write.

  • shaclShapeValidator.ts fetches the document and converts it to an RDF/JS dataset (toRdfJsDataset). For every subject, it picks the shapes by class and stored version as the mappers do (pickShapes), and by the values a shape fixes (an FSRS-6 review state's sm:scheduler, shapes.md). A subject of two classes is read as both, so it's held to the shape of each, a standard class's (foaf:Agent, dcat:DatasetSeries) as well as a Solid Memo one's. A deck group is the one exception: though a dcat:Catalog, it isn't checked as the catalog, which is read at its own address. In a release, a subject of a Solid Memo class only a pod holds (sm:Preferences, sm:Palette) is in violation (ReleaseClass). A subject is then:

    • checked, with its violations. A subject stamped with a format version below 1, which no shape has, is checked too: it's in violation (FormatVersionRange), as the app reads it as one that doesn't fit. So is a subject with a class that has no format of its version while another of its classes has one (FormatVersionClass);
    • newer, a format newer than this app knows, for every one of its classes: skipped, reported;
    • untyped, no Solid Memo class: listed, to show strays.

    A subject another app wrote is also marked foreign, and its results are warnings (below). A deck's document's own subject, when all it says is which deck the document is part of (isDeckBackLink in ownership.ts), is Solid Memo's and isn't listed. A document that doesn't exist is missing, which is normal.

  • Where a deck is is checked beside its shape, in an instance's catalog (DeckPlace, placeViolations in shaclShapeValidator.ts), for every subject that has sm:Deck among its classes. The entry is in violation when it's another document's subject, or puts a cards or reviews document at an address no deck's document may have (data-model.md). That's one a request or a server would take elsewhere, another resource's access control or metadata, or an instance document's. Folder and names are free. Such a deck isn't used at all, nor set aside: the notice names the problem, and removing the entry is all that's offered. A draft's deck isn't held to this.

  • A date the app can't read is checked beside its shape, wherever the subject is: an instance's, a release's or a draft's (DateTimeRange, dateViolations in shaclShapeValidator.ts). It's an xsd:dateTime in one of the shape's date fields that the app reads as no date (readDatetime in records.ts). One is past what a JavaScript date holds (after 13 September 275760, or before 20 April -271821), which SHACL finds valid. Another is longer than 64 characters, or isn't a date-time from its first character to its last: the app reads it as no date before the client's reader sees it, since that reader takes much longer on a long literal. It's a violation of the subject, with the date as its value. A value that's no date-time at all is the shape's to name, and named once. The app reads an optional date as absent, and a repair drops it. A subject that needs the date (an answer's time, a review state's reviews, the instance's creation) doesn't fit its shape, and is the user's to fix or remove. The release check names such a date as an error, and a release added from a link, or a copy's upgrade from one, that has one is refused. The write check doesn't look for it: the app writes no such date, and a subject that has one can still be written to.

  • A deck whose documents are unclaimed is named in the report's unclaimed (DeckRepository.unclaimedDocuments), with their addresses: they don't say they're the deck's, and are outside the instance's own folder (the same data-model.md section). It's no violation: nothing is blocked or set aside, and the deck is still read and studied from. Those documents aren't checked, under any spelling of their address, since a repair writes to the document. Each address is compared as a request sends it (canonicalUrlOf), the unclaimed ones brought to that form once per check, so a catalog of many decks costs one lookup per document, not a comparison of every pair. Solid Memo's data check and the Studio's Health screen show them (UnclaimedDecksNotice) with "These are this deck's documents", which claims them once the user confirms (claimDeckDocuments), for the next check. A document outside the instance's folder, and not in a storage of the user's found to hold the instance too (isInInstancePod), is named in outsidePod too, and its deck gets no such button. A check reads a deck's two documents at once. It reads those outside the instance's folder within one minute in all (OUTSIDE_READS_MS in instanceChecks.ts), whatever the read is for: to find which are unclaimed, to check them, or to read again one with something to repair. A catalog may name a thousand of them on servers that answer slowly, and every deck page waits on the check. The minute runs only while one of those reads is under way, so a slow read of the instance's own documents takes nothing from it. One not read by then counts as unclaimed when the check looks for those, and is unreadable when the check checks it. The documents in the instance's folder have no such budget.

  • A deck's document that isn't the deck's once read is left out of the report and isn't listed as unclaimed either. That's a release in the instance's folder named as a deck's document, or a document there that names only other decks. The check doesn't read a document in the instance's own folder to list it as unclaimed. It decides this (DeckRepository.isUnclaimedAsRead) only for a document with something to repair. A check of an unchanged instance still downloads nothing.

  • The shapes are fetched at their IRIs, under https://solid-memo.com/ns/shapes/ (SHAPES_BASE), not bundled: they're published with the site, so the browser checks against the document CI validated. siteFetch in appUseCases.ts reads them from wherever the site is served (for npm run dev and npm start, the repository's ns/), once per session.

  • The SHACL engine (engine.ts, the only module that imports rdf-validate-shacl) is a separate chunk, imported dynamically when a validation is asked for. It validates one focus node against one node shape (validateNode), so no sh:targetClass is needed.

  • The engine drops the data's rdfs:subClassOf statements before a check. The library follows them from a class to its subclasses for sh:class and sh:targetClass without bound, and the data is the document checked, which anyone may write. So a node is an instance of the classes its rdf:type names, and only those. No reference data states rdfs:subClassOf, and the CI cross-check refuses a document that does.

  • The engine reads a node with more than 32 rdf:type statements, far more than any Solid Memo or DCAT-AP document gives one, as having no types. The library goes through all of a node's types each time a subject linking to it is checked for its class (sh:class). So many subjects linking to a node with many types would take time growing with the square of the document. The node has one result more, a violation on rdf:type that says its types weren't read, and a subject that asks for its class finds it of none. In a pod, DCAT-AP's class result on the linking subject is a warning. The rest of the document is checked as usual, and a result about another app's node is a warning, as any other.

  • The engine checks xsd:decimal, xsd:float and xsd:double literals with patterns of its own, which it registers in the library's datatype checks (rdf-validate-datatype). The library's patterns can split a run of digits more than one way, so a literal that doesn't match takes time quadratic in its length, and the data checked may come from another app or a release's publisher. The engine's split a run one way only, so a check takes time linear in a literal's length. They take every value of the XSD lexical space (+INF too) and nothing else, so a value with two signs (+-1), which the library takes, is refused.

  • The engine counts how many times a check has gone over a node with one shape by the two terms, a key for each, and stops at 50, as the library does. The library counts by one string joined from the node and the shape. That string is longer than a term may be when the node is about that long (deck-library.md), and V8 hashes such a string by its length alone. So many such nodes of one length would take time growing with the square of their number. The library's own count is turned off.

  • A check keeps at most 1,000 results (MAX_RESULTS_PER_CHECK in validation.ts): of one subject against one shape, and of each node in a check of a whole graph against a profile. A constraint on each value, such as sh:datatype, gives a result for each, and a document may hold any number of values, and a constraint such as sh:closed gives all of a node's results at once. The engine makes no more than the limit of each severity of any one node, whatever the constraint, and leaves out the rest before they're made. The most severe are kept first, so a violation isn't left out for warnings found before it. It doesn't run a constraint again on a node whose left-out results are already as severe as any it could give. Past the limit, one more result (ResultLimit) says there are more. It's as severe as the most severe of those left out, so a subject with too many violations still doesn't conform, and one with only too many warnings still does. The library's own limit (maxErrors) isn't used: it counts only some constraints' results, and stops the whole check, every node's, not one node's. The places a deck entry names (DeckPlace, above) are held to the same limit. A subject's results, and a document's statements as the check reads them, are gathered one by one, never spread into a call, which a large list would overflow.

  • One engine per shape document is shared by every document checked against it. It runs one check at a time: the library keeps a single report per validator, so two documents checked together would mix up their results.

Profiles: DCAT-AP and SKOS

Data is held to two published profiles, vendored verbatim under packages/vocab/vendor/README.md, pinned by commit and sha256 in packages/vocab/vendor/manifest.json (which a test checks), and published with the site at /vendor/:

Profile Shapes Applies to
dcat-ap DCAT-AP 3.0.1 (SEMIC) catalogs, decks and deck releases, distributions, agents
skos SkoHub skos.shacl.ttl + skos.bestPractice.shacl.ttl, for SKOS the concept schemes in ns/vocab/
  • profiles.ts names each profile's files. Their shapes pick their own targets (sh:targetClass), so a profile checks a whole graph with the engine's validate, not one subject with validateNode.
  • The engine is SHACL Core only: coreOnly drops the SPARQL-based constraints (SkoHub has some) when the shapes are loaded. The CI cross-check runs them in full (below).
  • DCAT-AP's class checks (dcat:theme must be a skos:Concept, dcterms:language a dcterms:LinguisticSystem, …) look for the value's type in the data graph. The reference data in ns/vocab/external.ttl, the EU authority-table entries and media types Solid Memo uses, is loaded next to the data. A contributor adds a term there before data uses it.
  • In the browser, DCAT-AP's class check says nothing on a catalog's or deck group's link (dcat:dataset, dcat:catalog) to a member described in another document, in the data check and the write check alike (isMemberElsewhere in shaclShapeValidator.ts). A member of the same document still must be a dcat:Dataset or dcat:Catalog.
  • In a pod, DCAT-AP's result about a property Solid Memo doesn't write on one of its subjects is a warning, as another app's value (below). So is its class check on a value of a property Solid Memo writes, other than a catalog's member, unless a write adds that value and it's no blank node.
  • In node (npm run library, the tests), validateProfile (packages/shacl/node/shacl.ts) fails on any violation about a subject of the document, and, for Solid Memo's own concept schemes (held to SKOS best practice), on warnings (a profile's recommendations).
  • Fixtures under packages/vocab/fixtures/profile/<profile>/{valid,invalid}/ pin every property DCAT-AP makes mandatory on the classes Solid Memo writes. These are title and description on a catalog, dataset and dataset series, a publisher on a catalog, an access URL on a distribution, and a name on an agent. They pin the class of dcterms:source, dcat:theme and dcterms:language values, and of a catalog's or group's members: a deck (dcat:dataset) must be a dcat:Dataset and a group (dcat:catalog) a dcat:Catalog, so a member the document doesn't describe fails.

The report

Per document, the report gives its URL and status, and per subject one of these:

  • "conforms to deck format 2"
  • "deck format 3 is newer than this app knows; skipped"
  • "not a Solid Memo subject"
  • a subject only DCAT-AP has something to say about (a license, say)
  • "written by another app; Solid Memo only warns about it"
  • a table of severity, property, message and value (DCAT-AP's results marked as such)

The summary line says "All 9 documents conform" or "3 violations in 2 documents". A subject is checked in its stored format, so outdated but valid data passes. The format update is separate.

The invalid data policy

validateInstance runs when an instance is opened (Workspace, query ["validation", instance], shared with the developer view), and after a repair or a format update. What happens when the report doesn't conform is a preference, sm:invalidDataPolicy, a concept of sm:InvalidDataPolicies (vocab.md):

Policy What the app does
Block the instance (sm:blockInstance) The workspace waits for the check, then shows the Data check notice with the repair in place of every screen. Preferences and the validation view stay reachable, so the policy can be changed.
Set invalid data aside (sm:blockSubject, the default) A deck whose entry, cards or review states don't conform is set aside (deckSetAside): listed as "Set aside", not offered for study, its pages saying so. When the catalog or a deck group doesn't conform, the deck list's arrangement is set aside (arrangementSetAside): until it's repaired, the list shows every deck but can't be rearranged, nor its groups renamed or deleted, as with a list a newer version arranged. Any other subject that doesn't conform (a deck's agents or distribution, the catalog's publisher, the preferences, the instance record) sets nothing aside; the write check still refuses a write that leaves it not conforming. Everything else works. The notice names the decks set aside, says when the list can't be rearranged, and reports the rest.
Warn only (sm:warnOnly) The notice, and nothing else.
  • What fails is gathered once per report: the screens ask about each deck on every render, so the failing subjects and documents, and whether the arrangement fails, are read from a report once and kept with it. Each deck's answer is then a lookup, however large the report.
  • A deck entry or deck group counts only from the catalog: the deck list reads them from the catalog's document, so only a failing subject found there sets a deck's entry or the arrangement aside. A failing subject found in any other document, whatever its IRI, counts only through its own document: it sets aside the deck whose cards or reviews document it is in, if any. A deck's own check (checkDeck) and the Studio's health screen (deckOfSubjectsIn) count a subject the same way.
  • Preferences that don't fit their shape are kept. Preferences with a value their shape doesn't list, or without one it asks for, read as a misfit, not as none (getPreferences). So do ones whose format version isn't an integer, or that hold a value only a later format has: read as the older format, the migration would put the defaults in that value's place. The app studies by the defaults meanwhile, but no write puts the defaults over them. "Not now" to the invitation to FSRS-6 adds only its answer, as the Connected screen's choice does (onboarding.md), and the invitation stays hidden. The write check may refuse that write while the stored preferences still don't fit, and nothing is lost then. A choice of theme, palette or text size stays on the device. The format update leaves them as they are. Preferences says they don't fit, and a save or a switch of scheduler is refused (preferencesMisfit) unless the learner checks "Replace the saved preferences" there.
  • The default applies only where no policy is stated: an instance whose preferences were never saved, and preferences format 2, which the step to format 3 gives the default. Format 3 and later state a policy. Preferences saved with "Block the instance" keep it. In the vocabulary, the default concept's definition says "The default.", as sm:systemTheme's does.
  • Until the preferences are read, the workspace waits for the check as under "Block the instance", so a user who chose it never sees data before it's checked. Preferences that can't be read leave the default.
  • Under "Set invalid data aside", nothing is used before the check says what's set aside. Until then, a deck's pages (its browser, study, preferences and course) wait, and the deck list shows but can't be rearranged. So no write lands on data the check would set aside: a review state a reader drops would be studied as new, and the first answer would write over its history.
  • A document the check can't read (refused, say, or not Turtle, past the read's limits, or outside the instance's folder and not read within the check's minute) is in the report as unreadable, with why (unreadableDocument in validation.ts). What it holds isn't known, so it counts as one violation, and it holds back what it's part of, no more. Under "Set invalid data aside" the decks naming it are set aside (the arrangement too, when it's the catalog). Under "Block the instance" the instance is blocked. The validation view and the Studio's health name it, and why, with hidden characters shown as markers, as in a web address. No receipt is kept of it, and the app checks again when the user comes back to its tab.
  • A check that can't be made (the shapes unreachable, say) says so in a warning. Under "Block the instance" and "Set invalid data aside", the app then holds back what it would while the check is under way, until a later check is made. The app makes it again when the user comes back to its tab. What the check would have found isn't known, so nothing is taken as valid. The warning then says that choosing "Warn only" in Preferences, which the app always reaches, uses the data unchecked, and so does the Studio's line on a held screen. Under "Warn only" the warning is all.
  • The Studio holds to the same check and policy (useDataCheck in ui, which Solid Memo's workspace uses too), as studio.md says.

Data another app wrote

A subject is Solid Memo's when it has a Solid Memo class (sm:Deck, sm:Card, …), or carries sm:formatVersion, Solid Memo's stamp on every subject it writes. The catalog (catalog.ttl#catalog) is Solid Memo's, and so is a subject that one of Solid Memo's subjects names as its dcterms:creator, dcterms:publisher or dcat:distribution (ownership.ts), even when another app's rewrite drops its stamp. The catalog, a deck's agents and its distribution have no Solid Memo class. Such a link counts only where Solid Memo's shapes write it. In an instance's catalog.ttl, that's a creator or a distribution on a deck, and a publisher on the catalog or a deck group. In a release or a draft, a document that is an sm:Deck, it's a creator, a distribution or a publisher on the deck, and a publisher on its series. In a deck's cards or reviews document, or any other, no such link counts. A subject of a Solid Memo class counts only the links of that class, whatever other class it says it has. The subject it names must be of the class the link names, a foaf:Agent or a dcat:Distribution, and of no other class a shape describes. Only the #catalog of a catalog.ttl is the catalog without its class. Any other subject named that way stays another app's.

A review state of another scheduler is never Solid Memo's, whatever its stamp: any value of sm:scheduler but the sm:sm2 and sm:fsrs6 IRIs, a literal too. The rule is for review states only. A subject counts as one when it's typed sm:ReviewState, or has no Solid Memo class at all, and isn't the catalog. A subject with any other Solid Memo class (sm:Card, sm:Deck, …), and the catalog, is read as one whatever sm:scheduler it names, so it stays Solid Memo's and is held to its shape. Another scheduler's state only gets warnings (data-model.md). An FSRS-6 state, naming sm:fsrs6, is Solid Memo's own: it's checked against FSRS review state 1, its violations count as any other, and a half-written snapshot is repaired by dropping it, as an SM-2 state's is.

Any other subject is another app's, marked foreign. It's checked against the shape its class picks (a foaf:Agent against AgentV1) and DCAT-AP, but every result about it is a warning. Under every policy it sets no deck aside, leaves the arrangement as it is and blocks no instance. The document holding it still conforms, and gets its receipt. Only the developer report shows its results, and the repair offers nothing for it. A subject another app wrote with a Solid Memo class is held to that class's shape.

The Solid client keeps some IRIs for subjects it hasn't saved yet (LOCAL_NODE), so a document can't state one as data: its subject would have no URL to report. A read leaves out any statement that names such an IRI, so the check, the deck writes and every other reader never see it. The one exception is the library that makes a release public, which reads the text of the release's access control itself. Such a statement still counts toward a read's limits. A patch leaves it in the document; a write of the whole document drops it. A document that says nothing else reads as empty, so removing its deck deletes the document, whereas other foreign data keeps it.

A release in the library, or one added from a link, is wholly its publisher's: nothing in it is marked foreign, a link to a blank-node member counts as any other link, and every violation in it counts.

Deleting an instance, a deck or a draft removes from each document only Solid Memo's subjects of that document's kind, conforming or not, and deletes a document only when it's left empty (data-model.md). Another app's subject, and anything whose owner is unclear, is kept with its document, which the result names as kept.

A catalog's or deck group's link to another app's member is that app's data too. The member may be a subject of the same document that isn't Solid Memo's (a schema:Dataset, say, which fails DCAT-AP's class check), or a blank node (which fails the shape's sh:nodeKind sh:IRI). Either way, the result about the link is a warning, in the check and the write check alike: the catalog still conforms, the arrangement stays editable, and no repair unlinks the member. The drop-dangling-members repair, and writing the catalog whole, keep such a link. A link to a subject of the document that nothing describes is still the catalog's own problem.

A value another app added to one of Solid Memo's subjects is that app's data as well. The writer keeps every value of a property the subject's shape doesn't own, so another app's values survive a save. DCAT-AP constrains some of those properties, such as dcterms:accrualPeriodicity or a second dcterms:publisher on a deck's entry, or a second dcterms:type on an agent that several decks name. A DCAT-AP result about such a property is a warning, in the check and the write check alike (foreignValues in shaclShapeValidator.ts). The deck isn't set aside, and its entry, its agents and its distribution can still be saved and upgraded, with the value kept. The properties Solid Memo writes are the subject's classes, its format stamp, the fields of the shapes it's read by, and Solid Memo's own terms (sm:completedChapter, say). A result about one of those, or one with no property, still counts. A subject no shape is read by keeps every result.

Another app may also add a value to a property Solid Memo writes, such as a dcat:theme on a deck's entry that the deck's shape takes but that isn't a skos:Concept. The reader reads it back and the writer writes it again, so a result about it would refuse every write of the entry. The write check holds each value a write adds to its class, a theme copied from a release say, so a value in the document that fails the class check is another app's. So DCAT-AP's class check on a value of such a property is a warning in the check (isClassOfValue). It's a warning in the write check too, unless the write adds that value (addedValues, from the dataset's recorded changes). The recorded changes leave out every triple that names a blank node, so the class check on a blank-node value is a warning in the write check even when the write adds it. Solid Memo adds values by IRI or as literals, never as a blank node, so a blank-node value there is one it kept as it read it. Nothing may describe the value, the document may describe it as something else, or it may be one of Solid Memo's subjects in a role it never has. The class check on a catalog's link to a member of the document that nothing describes still counts, as above. So does every other DCAT-AP result about a property Solid Memo writes, such as a second value where DCAT-AP allows one.

Another app may give a node that a subject of Solid Memo's links to more rdf:type statements than the engine reads, so the engine checks it as having none. DCAT-AP's class check on a link to such a node, a catalog's link to a member included, is then a warning in both checks (isOverTypedValue), and the node's own result names the problem.

The write check

Every repository write is checked: a deck, its agents and distribution, the catalog, cards, review states, preferences, a palette and its color sets, the instance record. Only the subjects the write touches are checked, so an old problem elsewhere in the document doesn't block it. They're checked against their shapes, and in a document with DCAT subjects, against DCAT-AP, where a result about another app's value is a warning (checkSubjects in shaclShapeValidator.ts, wired into the repositories as checkWrite in appUseCases.ts). A write that wouldn't conform is refused with every problem named, and nothing is saved.

Every caller names where the write goes (WriteContext in writeCheck.ts), and that picks the shapes, as pickShapes picks them for a read:

  • "pod": an instance's documents, checked against the pod's shapes and, where they have DCAT subjects, DCAT-AP. Every write the app makes is one, except a draft's.

  • "draft": a release's draft. Its deck, chapters and steps are checked against the draft shapes (DraftDeckV1, DraftChapterV1, DraftStepV1, shapes.md), which don't yet ask what a release needs. Its cards, distractors and agents are checked against the shapes every context shares. DCAT-AP isn't run: a draft may still lack what a published dataset needs, such as a description. Every write of a draft is one (solidReleaseDraftRepository.ts), and so is the check of a draft's documents.

    The release check holds the draft to all of it before it's published: validateRelease checks the resulting release, moved to its address, against the library shapes ("library"), DCAT-AP and SKOS, with a library's index beside it for the library's policy. There, DCAT-AP holds the release's publisher and series to their class in the index. In a pod, nothing else is read (deck-library.md), so a publisher or series the release doesn't describe fails DCAT-AP's class check. The next version of a library release describes them, as the index does (nextVersionDraft). A subject of two Solid Memo classes is held to the shape of each; one of them with no shape at the subject's version is named as a subject no shape describes.

A published release is never written.

A release a learner adds from a link is checked in the "library" context (validateDataset(url, dataset, "library")): each subject against its library shape, no DCAT-AP. The check is of the read the library made (DeckLibrary.readRelease), which no login goes with, and the read that passes is the one imported. A violation refuses the release; a warning doesn't.

Repair

planRepair (domain/repair.ts) turns a report into repairs for the problems with one safe answer, written in the subject's own stored format:

Problem Repair
A deck without a description The default description, Flashcards: <title>.
A deck without a study direction, or an unknown one Front to back, as format 1 studied every deck
A review state with half an undo snapshot (an SM-2 or an FSRS-6 state's) The snapshot dropped (as the review-state 1 → 2 migration does), the four terms of an FSRS-6 state's or the five of an SM-2 state's
A malformed due day Recomputed from the last review and the bounded interval, or a long-past day, due at once, when those give no real day
An agent without a name Named after its IRI
A catalog or deck group listing a deck or group the document doesn't describe (DCAT-AP's class check on dcat:dataset or dcat:catalog) Those links dropped (a member typed sm:Deck or sm:DeckGroup alone is kept, and so is every link to another document)
A date the app can't read (DateTimeRange) of a field the subject's shape and format make optional Those dates dropped, every such date of the subject in one repair; a readable value of the same field, and every other field, are kept
A review state's undo snapshot with a date the app can't read The snapshot dropped, as for half a snapshot

Every other problem is listed with its document linked, to be fixed there or removed after a confirmation. Removing a subject drops only the catalog's and the deck groups' membership links to it (dcat:dataset, dcat:catalog), not a deck naming it as creator or the catalog as publisher. applyRepairs (solidRepairRepository.ts) reads each document once, applies its repairs in one edit, each to the subject as the repairs before it left it, and writes it once, then the instance is checked again. Many repairs take time that grows with them and the document, not their square. That read decides the write, as for any write to a deck's document: a release (<> a sm:Deck, other than a draft's release document) and a document unclaimed for every deck naming it are left as they are.

The CI cross-check

CI checks what Solid Memo publishes again with an independent SHACL engine, pySHACL (pinned in scripts/requirements-ci.txt), which runs SPARQL-based constraints too. scripts/shacl_crosscheck.py holds to DCAT-AP:

  • the deck library's index and every version;
  • the pod catalogs of the deck format 4, 5 (a title in any language) and 6 (tagged keywords) and deck group 1 (nested groups) fixtures;
  • a release published in a pod, with no index beside it: the library deck 6 fixtures, which describe their series and publisher themselves.

It holds that release, the draft deck, chapter and step 1 fixtures, and the review-state fixtures (review state 2 and FSRS review state 1 in one reviews document) to Solid Memo's own shapes. It holds the vocabulary's concept schemes to SkoHub's SKOS shapes, best practice included. The script picks the shape of each of a subject's classes by format version, context and the values a shape fixes, as pickShapes does, and targets it with sh:targetNode.

Each version is checked beside the index, only for the subjects it states something about, targeted the same way. DCAT-AP asks a node only about its own statements and their objects' class, so a version can't break an index node it says nothing about, and the index has its own check. The script stops if DCAT-AP starts asking more, or if any document states rdfs:subClassOf, since either would break that. The checks run in parallel, one process per CPU. A disagreement between the engines, or a constraint the browser's engine can't run, fails CI.

The script reads ns/ and decks/ from the repository, each file at the IRI the site publishes it under, without a build or the network, as its own job beside the build. Locally: pip install -r scripts/requirements-ci.txt, then npm run crosscheck. That runs python3 scripts/shacl_crosscheck.py, and says what to install if pySHACL is missing.