All notable changes to this project will be documented in this file. See also JsonSchemaChanges.md for a list of changes in the JSON Schema for API model files.
- Added HTTP services (defined in service-model.conf and OpenApi files) to the visualization graph
- Added HTTP services to the "Missing Items" reports
- Removed elements from "Missing Items" report and relationship graph that show published events with no subscribers, commands with no senders and sevices with no users
- Fixed versioning for OpenApi data stored in the icd database (previously the latest version was always used)
Note: run
icd-git --ingestto reload and fix an existing icd database. - Put context/interface diagram and missing item report in appendix of generated ICD SDBs (if the graph is not empty, if there are missing items)
- Updated dependencies, switched from akka to pekko.
- Replaced dependency on sjsonnet Java lib with a use of "jsonnet" command, which must be installed if the feature is used.
- UI Changes in icd web app: Moved buttons that depend on the selected subsystems or ICDs to the Select tab (Buttons are only enabled when applicable).
- Added Markdown/Paradox based documentation under ./docs directory (to replace Docushare PDF version).
- For ICDs, the summary table now displays the paths/routes that are used for HTTP services.
- For ICDs between components in the same subsystem, the version of the API is now displayed instead of "unpublished".
- Fixes for online documentation.
- Improved readability of service apis in SDB PDFs.
- Fixes for PDF display of HTML markup in OpenApi based service APIs.
- Fixed issue when, after renaming a component, the old name still showed up in the generated APIs.
- In icd web app: Removed display of components in the sidebar that are not in the ICD and added the subsystem names before the components.
- Changed ICD PDFs to include more client API information, such as a summary of subscribed events or sent commands.
- See the JIRA release page for details on the changes.
- Added "subsystem" constant in generated source files with subsystem name and version: For example: subsystem = "APS-1.4" (See Generate menu item in icd web app).
- Added file MacOs-Max-Files-Limit.md describing how to increase the max-files limit on recent MacOS versions (This is required in order to run MongoDB when running the ICD web server locally, since the default setting is too low).
- Include FITS keyword table in PDFs for APIs (was previously only displayed in web app)
- Added a new toolbar item: "Missing" that generates a PDF containing a list of subscribed events with no publisher, sent commands with no receiver, and referenced components that are not defined. The table refers to the selected subsystem(s) or all subsystems, if none were selected (in the Select tab).
- Added ability to specify FITS keywords in publish model event parameter descriptions. The keywords (and optional channels) must already be in the FITS dictionary.
- Added a command line option to icd-fits to generate a FITS dictionary JSON file by merging the one currently in the icd database with the FITS keywords defined for the event parameters in specified subsystems. This can be used to update the FITS dictionary to match the entries in the model files for a given subsystem (or all subsystems).
- Added "Alarms" toolbar item, to generate a PDF listing the alarms for the selected subsystem/component or all subsystems.
- Changed the web and PDF displays to include information about alarms in all ICDs with ESW.
- For graphs, added the option to display only subsystems (instead of the subsystem components). Added a new command line option for this to the icd-viz app and a new checkbox in the web app's graph options popup.
- For graphs: The graph generated for ICDs (two subsystems/components) now only includes connections between the two (previously connections to other subsystems were also included).
- Added new optional "category" field for events with possible values: [DEMAND, CONTROL, EVENT, STATUS] (See "Event Categories" at end of README.md)
- Replaced general usage of "TMT" with "TIO" (TMT International Observatory) in web app and docs.
- Added validation check for parameters that the "defaultValue" is valid for the declared type
- Set a fixed with for inline OpenApi display frames
- Added code to revert database if post-ingest validation errors are found.
- Changed the format of the FITS-Tags.conf file to allow one tag to inherit keywords from another. For example, the IRIS and MODHIS tags now "inherit" the keywords from the "DL" (diffraction-limited) tag, so these are automatically displayed when the IRIS or MODHIS tags are selected in the web app.
- Fixed scrollbar display in web app (was too thin or not visible)
- Fixed issue generating PDFs with the option selected: "Include only the details that are expanded in the HTML view"
- Display subscriber info in summary table if option "Include client API information" is selected, but not for ICDs
- Updated dependencies, code cleanup
- Added additional validation checks for duplicate or incorrect component, event or command names when manually ingesting subsystem model files
- Incremented major version to 3.0.0
- Fixed an issue where published images were not included in the generated documentation if no events were published.
- Added a validation check for conflicting component names when manually ingesting or uploading model files into the icd database. Also added a check for cases found in already published APIs.
- Minor performance improvements
- Added details table for published images to generated PDFs and changed the format used for the web version
- Added --documentNumber option to icd-db (and text field to web app's PDF options popup). If given, the text is displayed under the subtitle in the generated PDF
- Added options to icd-db command line app to generate code containing the event, command and parameter keys. (currently Scala, Java, Typescript and Python are supported)
- Added a new modelVersion "3.0", which defines the allowed parameter "units" matching the ones defined in the CSW framework and adds the parameter types "taiTime" and "utcTime" (same as taiDate and utcDate, which are still allowed: taiTime and utcTime should be preferred, matching the CSW names).
- Added jsonnet support. Model files with the '.jsonnet' suffix are processed with jsonnet before storing in the icd database. This can be used to avoid repetition in the model files. See this example.
- Changed the publish model to only allow predefined Observe Events
- Added support for documenting TMT CSW components that are HTTP web services using OpenAPI.
- Added a new, optional icd model file: service-model.conf, that contains information about the HTTP services that a component provides or requires. For each service provided, the model file also contains the name of the OpenAPI JSON file that describes the service.
- Updated dependencies
- Replaced deprecated ScalaJS Ajax usage with recommended dom fetch API
- Removed RaDec type from JSON schema for parameters in v2.0 and later model files
- Removed support for "struct" parameter type, since it was removed from CSW
- Fixed display of "default" parameter values
- Improved "busy" cursor display
- Removed support for uploading a zip file containing the icd model files, since modern browsers now all support uploading a directory
- Removed dependency on "less" and converted/renamed icd.less to icd.css
- Updated list of allowed CSW Units
- Updated all dependencies to the latest versions, including Bootstrap-5.2.0 and support for MongoDB-6.0
- Added support for documenting published and subscribed images (TMT VIZ)
- Added support for a FITS keyword dictionary, and the icd-fits command line app, FITS Dictionary tab (FITS Keywords
will be automatically imported once the DMS subsystem model files have been published. For now, you can
use
icd-fits -i examples/3.0/FITS-Dictionary.json --ingestTags examples/3.0/FITS-Tags.conf) or manually upload/ingest the DMS-Model-Files GitHub repo in the icd web app. - Updated to Play-Framework-2.9.0-M2, which means icd can now be compiled and run with Java-17 (as well as Java-11).
- Fixed issues in the handling of older subsystem versions when generating graphs
- Fixed issues dealing with "refs"
- Improved the title and description display for inner-subsystem ICDs (changed to avoid duplicating information, use component description where applicable).
- Added support for optional
${subsystem}-icd-model.conffiles (for example:IRIS-icd-model.conf,TCS-icd-model.conf) that add information about the ICD between${subsystem}and the subsystem being described. See TEST2-icd-model.conf or TEST-icd-model.conf for an example, or for more realistic examples: here and here
-
Fixed a minor issue that could occur in the web app when the uploaded model file versions in subsystem-model.conf and component-model.conf were different.
-
Changed size calculations for archived events: Use 12 hours/day (was 24), list hourly data sizes, changed event overheads to match CSW versions.
-
Added "Command: " to command description heading to match event headings.
-
Fixed an issue that could cause a newly published ICD version to not be automatically ingested into the local icd database in certain circumstances
-
Added a new component model file: alarm-model.conf, which replaces the "alarms" section in publish-model.conf (The old version is still supported for backward compatibility). See DEOPSICDDB-98 .
-
Added support for specifying an optional minLength and maxLength for string parameter types (to help in calculating the archived size for events)
- Changed the "attributes" keyword for events (in publish-model.conf) and the "args" keyword for commands (in command-model.conf) to "parameters", to match the CSW terms. The previous names are still accepted for backward compatibility. Also renamed the json-schema files used internally for validation. See DEOPSICDDB-113 . The generated APIs and ICDs now also display "Parameters" instead of "Attributes", etc.
-
Added "master" as an API version that is updated automatically from GitHub on startup or refresh of the web app.
-
Added support for LaTeX math formulas in description texts, delimited by
$`...`$ for inline or ```math...``` for a block. See examples/2.0/TEST/envCtrl/publish-model.conf and examples/2.0/TEST/envCtrl/command-model.conf for examples. The LaTeX string is converted to an image and then inserted in the HTML or PDF. -
Added support for UML (and Graphviz/Dot) markup in description texts, based on PlantUML, delimited by ```uml...```. See examples/2.0/TEST/envCtrl/component-model.conf. Note: The Graphviz apps needs to be installed on the server (or local machine, for the command line) in order to use this feature.
-
Added icd-viz command line app, based on Graphviz/Dot and Ed Chapin's
icdRelationships.py, that generates a graph of component relationships. -
Added a "Graph" toolbar item to the icd web app that displays a graph of relationships of the selected subsystems or components.
-
Added icd-db command line option "--clientApi" and a checkbox in the web app to include subscribed events and sent commands in an API document (the default was changed to only include published events or alarms and received commands) .
-
Added new option to icd-git command:
--ingestMissing: Ingests any APIs or ICDs that were published, but are not yet in the local database, plus any master branch versions. -
Added "ref" keyword for events, commands, attributes to enable reuse of all or part of another item in the model files. See "Reusing Event, Command and Attribute Definitions" at the end of README.md for more details.
-
Changed API layout to include only published events or alarms and received commands by default (with option to include subscribed events and sent commands as well, like before).
-
The checkboxes in the web app select dialog are only enabled when a single subsystem is selected (since they only apply to APIs).
-
Changed the JSON schema for attributes to allow "inf" as the value for "maximum" or "exclusiveMaximum" and "-inf" as the value for "minimum" or "exclusiveMinimum".
-
Changed the JSON schema for attributes to include the "allowNaN" keyword. Set to true if NaN values are allowed. Default value is false.
- Added icd-db command line option to increase font size for PDFs.
- Changed PDF and Archive buttons to display a popup with options for orientation and font size.
-
Fixed a bug that requires rebuilding the icd database. This is done automatically when you run icdwebserver, or you can run
icd-git --ingestto do it manually. -
Added the following subsystems back to the allowed list for the 2.0 model file schema: ENC, STR, M2S, M3S, TINS and created the related repos under https://github.com/tmt-icd/.
-
Added error message for duplicate component name in different directories
-
Added simple auth/login dialog for use on public servers (Use -Dicd.isPublicServer=true)
-
Added code to automatically tag API GitHub repos when an API is published (For example: "v1.2"), to make it easier to compare API versions
- Removed alarms from the "Missing Items" report generated by icd-db
- Alarm changes: Components can no longer subscribe to alarms
- Alarm property name changes (only for schema-2.0): "acknowledge" changed to "autoAck" (Opposite meaning)
Note: If you are using modelVersion 2.0, please update and run: icd-git --ingest to rebuild the local icd
database.
-
Fixed some issues with browser history after clicking on HTML links in the icd web app
-
Removed
-Dicd.allowUpload=falseoption, replaced with-Dicd.isPublicServer=true, which controls the features displayed when running the icd web server on a public server. -
Improved warning message displayed in HTML/PDF output for missing components
- Added an optional "role" field for received commands. The allowed values are "eng", "admin" or "user" (default: " user"). Sending a command to a component will require the correct user role for the subsystem in the CSW AAS auth service.
-
Minor CSS changes to support different screen sizes
-
Updated archive size for taiDate type to 12 bytes (was 16)
-
Updated icd-web-server.sh script (used to run on public server)
-
Removed unneeded subsystems
-
Added option to generate PDFs in portrait or landscape orientation (Landscape orientation is preferred, so that there is enough room in the many tables in the PDF output).
-
Updated itextpdf dependency version to itext-7
-
PDFs for published APIs and ICDs are now cached (Only for full, published APIs and ICDs without references to external subsystems - i.e.: The "Search all Subsystems" option is not checked).
-
For modelVersion = "2.0": Removed the "prefix" setting from component-model.conf, since after changes in CSW, prefix is always just $subsystem.$component (Component names can contain dots and spaces, except leading or trailing)
-
For modelVersion = "2.0": Component names can no longer contain a dash "-" (to match changes in CSW)
-
Fixed an issue where the version of the second subsystem in an ICD was ignored
-
Fixed an issue where after renaming a component, the old name could show up
- Fixed issue where subscribed telemetry and eventStreams were being ignored (Now they are automatically converted to events, since telemetry and eventStreams have been removed in the new JSON schema)
-
Made changes to allow inner document links in MarkDown descriptions (DEOPSICDDB-54)
-
Changed link targets to ensure unique names (in case two events in different components have the same name)
-
Fixed bug in web app dealing with "Search all Subsystems" option
-
Changes to allow embedded dots in component names
-
Updated dependencies to fix issue with PDF generation (DEOPSICDDB-93)
-
Added a new overview/status dialog that displays a table of the latest published APIs and ICDs for a selected subsystem.
-
Added Unpublish button to publish dialog (Use in case you published something by mistake).
-
Added confirmation popup for publishing or unpublishing an API or ICD.
-
Added an "Archive" item to the webapp that displays a PDF detailing the sizes of all archived items (events, etc.) in the selected subsystem and component (default: all subsystems).
-
Added the icd software version to the title in the Status page.
-
The Publish dialog access is restricted to those with write access to the ICD-Model-Files repository and is only enabled when starting icdwebserver with the
-Dicd.isPublicServer=trueoption. -
Fixed issue that could cause PDF generation to fail if embedded HTML in description text was not valid XHTML.
- Added total event size and yearly accumulation to the display for archived events (also added to the Archived
Items report produced by icd-db).
Note that for some types, such as strings, the sizes are only guesses, since the string length is not known ahead of
time.
If
maxRateis zero or not defined, 1 Hz is assumed. Note also that the actual space required to archive events may be much less, due to the storage format (CBOR), compression, etc.
- Removed
minRateandarchiveRatefrom the 2.0 JSON Schema for APIs. From now on, onlymaxRateshould be used, which is the maximum publish rate for the event in Hz. This is used to calculate the yearly accumulation or size of the data for archived events for a year.
-
Added a new Publish dialog to the icd web app
-
The
Uploadfeature in the icd web app. which allows you to ingest local model files into the ICD database, can now be disabled by changing theicd.isPublicServerconfiguration setting in icd-web-server/application.conf or by starting the web app like this:icdwebserver -Dicd.isPublicServer=true. The Upload feature should be disabled whenicdwebserveris running in a public network and should only display the actual API and ICD releases, which are stored on GitHub. -
Updated the list of allowed subsystem names for ICD model files (The old list is only allowed when modelVersion is set to "1.0". If it is set to "2.0", the new list is used).
- Bug fixes
- Added support for new types for attributes. The list of available types matches those implemented in CSW parameter sets: (array, struct, boolean, integer, number, string, byte, short, long, float, double, taiDate, utcDate, raDec, eqCoord, solarSystemCoord, minorPlanetCoord, cometCoord, altAzCoord, coord)
-
Changed the default JSON schema for model files. The old 1.0 schema is still supported. You can use the new model file formats by setting modelVersion="2.0" in component-model.conf and subsystem-model.conf.
-
The old and new JSON schema descriptions are now under icd-db/src/main/resources in the 1.0 and 2.0 directories. Examples of the old and new formats can be found in the examples directory.
-
With the new JSON schema for attributes, the values for exclusiveMinimum and exclusiveMaximum are the numerical values (previously it was a boolean value).
-
Changed the icd web app (icdwebserver) to support component to component ICDs as well as viewing selected components in an API.
-
Changed the icd web app to automatically ingest published APIs and subsystems from GitHub if missing on the first start (previously they were ingested only as needed, which caused delays and other issues)
-
Bug fixes
-
Removed telemetry events and eventStreams from the publish model (These are automatically converted to events when imported from modelVersion 1.0 files).
-
Removed archive field from alarm model.
-
Added observeEvents and currentStates as an event types for the publish model.
-
Added more required fields to the alarm model: location, alarmType, probableCause, operatorResponse, autoAck, latched.
-
Added support for struct types for attributes. See examples/2.0/TEST/envCtrl/publish-model.conf for some sample struct declarations.
-
New
icd-gitcommand line tool and features in the icd web app. Now published versions of subsystem APIs and ICDs are managed with GitHub repositories and JSON files ( See README.md). -
Now warnings are displayed for subscribed items where there is no publisher, or sent commands/configurations, where no receiving end was defined.
-
New
--target-componentoption to the icd-db command line app: Can be used together with the--componentoption to create a PDF for an ICD between two components in different subsystems, or just to restrict the document to items related to the target component. -
Added two new icd-db options: --archive (-a) to generate a report of all events that have "archive" set to true, and --missing (-m) to generate a report listing published events with no subscribers, subscribed events with no publishers, referenced components with no definition, etc.
-
Added a new primitive type "taiDate" that can be used in ICD model files to indicate a TAI date or time type.
-
Removed the Publish feature from the icd web app. Publishing is now done by a TMT admin using the
icd-gitcommand line tool. -
Changed the layout for APIs and ICDs to include a summary table at top with links to details below.
-
Changed the ICD PDF layout to display the published items from both subsystems, with links to the subscribers (instead of showing a list of the subscribed items)
-
Changed the way the PDF document titles are created, so that (for the command line at least) you can have component to component ICDs. For example, these two commands will generate an ICD PDF from a single component of IRIS to a single component of NFIRAOS and another one between two components in NFIRAOS (Note that if the versions are not specified, you get the latest unpublished version):
icd-db -s IRIS:1.5 -t NFIRAOS:1.3 --component csro-env-assembly --target-component encl -o csro-env-assembly-encl.pdf
icd-db -s NFIRAOS -t NFIRAOS --component dm --target-component rtc -o dm-rtc.pdf
-
Bug fixes and changes that were suggested in the last review.
-
Incompatible changes have been made, so existing Mongodb databases should be deleted (for example with the command: icd-db --drop db) and any existing ICD files should be validated against the new schema and reimported. You can do this using the web app. See the README.md files in the source code for more information.