Abstract template layer into its own dedicated module

[GuySartorelli]

https://github.com/silverstripeltd/product-issues/issues/875 (private repo) outlines a POC for abstracting the template layer into its own module. This issue is about taking that POC and implementing it properly for CMS 6.

This will provide the following benefits:

• improved separation between the view and model layers
• improved architecture for the view layer
• opens the door to further decouple the CMS from the template layer over time (e.g. could move all templates out into an optional 'admin-theme', and allow projects to introduce their own full 'admin-theme' similar to how the Drupal backend works)
• ability to swap out the rendering engine easily in the future, or per project, if that's desired

Related issues

Check if these can be sensibly resolved as part of this work

• Parsed template cache can't be configured silverstripe-template-engine#17
• Passing CustomFields as array to ViewableData::renderWith doesn't pass data in included templates #8580
• Data overrides in template hide chained object properties #9915
• Rendering lots of objects in nested templates can be really slow #10174 (see last comment - it's related to the implementation in $scope->locally())
• Better casting in ViewableData #11262
• Move ViewableData::Me() out of ViewableData and into the template layer #11261
• chooseTemplate sometimes do the work twice silverstripe-template-engine#18

Notes

• Check everywhere that uses ThemeResourceLoader::findTemplate() and SSViewer::get_templates_by_class() if those end up being moved or significantly refactored
• Don't forget to check what's going on with ViewableData_Customised and ViewableData_Debugger - at a minimum they need the same renaming treatment that ViewableData is getting.
• ContentNegotiator has some weird regex related to the output of <% base_tag %>

Acceptance criteria

• Code related to rendering .ss templates is moved into a new repository called silverstripe/silverstripe-template-renderer
  • To the extent that it's feasible, commit history for that code is copied into the new repository
  • The new module is a direct dependency of silverstripe/framework since that contains .ss templates.
• SSViewer remains in framework, and has the following responsibilities:
  • It delegates template rendering (i.e. you call process() on the SSViewer instance, which instantiates a template renderer and asks for the rendered HTML)
  • It keeps track of themes
  • It injects js/css from the Requirements API into the rendered HTML result
  • It wraps the rendered HTML (string) in DBHtmlText
• The content for the base tag and functionality for rewriting anchor links are either both retained in SSViewer or both moved to the new module
• There is an interface in framework that defines the minimum API required for SSViewer to abstractly interact with a template rendering engine
• If feasible with minimal upgrade pain, redefine how template candidates are structured (currently an esoteric associative array)
• In PHP for core and supported modules, templates are never referenced with the .ss extension
• A new ViewLayerData class is created.
  • All values must be wrapped in this class before being passed to the template renderer.
  • This class is responsible for the following:
    • Casting values to the appropriate DBField instance (plus arrays to either ArrayList or ArrayData)
    • Determining whether a value "exists" (most likely delegating to ModelData::exists() where that has been implemented)
• ViewableData is renamed to ModelData, since it really represents a "Model" in the MVC sense.
• Documentation is updated as appropriate
  • Any references to the API are updated
  • There's docs indicating how to replace the template renderer (and clearly highlighting the drawbacks of doing so)
  • As appropriate, distinguish between the model layer and the view layer in the docs
• Overall test coverage is not reduced
• There is a clear upgrade path

Explicitly excluded

• Support for multiple template rendering engines running concurrently (can be done afterward)
• Implementing alternative template rendering engine bridges (e.g. twig for silverstripe)
• Re-evaluating which classes are subclasses of ViewableData per https://github.com/silverstripeltd/product-issues/issues/875#issuecomment-2249358959 (can easily do as a follow-on card afterward)

Strong-typing PRs

Adding some strong typing to some existing API before I start the refactor so I have more confidence in the types of the values I'm dealing with and passing through to the template layer.

Kitchen sink CI run

• API Strong typing for the view layer #11351
• API Strong typing for the view layer silverstripe-cms#2994
• API Strong typing for the view layer silverstripe-reports#193
• API Strong typing for the view layer silverstripe-assets#634
• API Strong typing for the view layer silverstripe-elemental-bannerblock#160
• API Strong typing for the view layer silverstripe-elemental#1244
• API Strong typing for the view layer silverstripe-linkfield#322
• API Strong typing for the view layer silverstripe-registry#105
• MNT Fix unit test silverstripe-userforms#1321
• API Strong typing for the view layer symbiote/silverstripe-multivaluefield#118
• MNT Fix some unit tests silverstripe-gridfieldextensions#415
• API Strong typing for the view layer tractorcow-farm/silverstripe-fluent#881
• DOC Update typehints in docs developer-docs#569
• FIX Respect new typehints silverstripe-versioned#469
• FIX Respect new typehints silverstripe-restfulserver#133

Renaming PRs

CMS 5

• API Deprecate classes which will be renamed #11375
• DOC Document deprecating classes which will be renamed developer-docs#585

CMS 6

kitchen sink CI
The CI failure is unrelated to these PRs and was caused by a fluent PR being merged before the code it relies on.

• API Move various classes to more appropriate namespaces #11370
• API Use new names for renamed classes silverstripe-testsession#94
• API Use new names for renamed classes silverstripe-behat-extension#285
• API Use new names for renamed classes silverstripe-admin#1824
• API Use new names for renamed classes silverstripe-frameworktest#205
• API Use new names for renamed classes silverstripe-assets#640
• API Use new names for renamed classes silverstripe-asset-admin#1495
• API Use new names for renamed classes silverstripe-campaign-admin#330
• API Use new names for renamed classes silverstripe-reports#195
• API Use new names for renamed classes silverstripe-siteconfig#177
• API Use new names for renamed classes silverstripe-errorpage#122
• API Use new names for renamed classes silverstripe-session-manager#220
• API Use new names for renamed classes silverstripe-versioned#474
• API Use new names for renamed classes silverstripe-versioned-admin#363
• API Use new names for renamed classes silverstripe-cms#3003
• API Use new names for renamed classes silverstripe-externallinks#148
• API Use new names for renamed classes silverstripe-subsites#600
• API Use new names for renamed classes silverstripe-linkfield#336
• API Use new names for renamed classes silverstripe-tagfield#308
• API Use new names for renamed classes silverstripe-ldap#92
• API Use new names for renamed classes silverstripe-restfulserver#135
• API Use new names for renamed classes silverstripe-realme#163
• API Use new names for renamed classes silverstripe-staticpublishqueue#204
• API Use new names for renamed classes silverstripe-lumberjack#172
• API Use new names for renamed classes silverstripe-taxonomy#124
• API Use new names for renamed classes silverstripe-environmentcheck#113
• API Use new names for renamed classes silverstripe-blog#785
• API Use new names for renamed classes silverstripe-registry#108
• API Use new names for renamed classes silverstripe-graphql#606
• API Use new names for renamed classes silverstripe-mfa#566
• API Use new names for renamed classes silverstripe-iframe#98
• API Use new names for renamed classes silverstripe-gridfieldqueuedexport#125
• API Use new names for renamed classes silverstripe-userforms#1328
• API Use new names for renamed classes silverstripe-securityreport#89
• API Use new names for renamed classes silverstripe-contentreview#258
• API Use new names for renamed classes silverstripe-versionfeed#121
• API Use new names for renamed classes silverstripe-sharedraftcontent#261
• API Use new names for renamed classes bringyourownideas/silverstripe-maintenance#242
• API Use new names for renamed classes tractorcow-farm/silverstripe-fluent#889
• API Use new names for renamed classes silverstripe-elemental#1254
• API Use new names for renamed classes silverstripe-gridfield-bulk-editing-tools#307
• API Use new names for renamed classes silverstripe-gridfieldextensions#417
• API Use new names for renamed classes silverstripe-queuedjobs#448
• API Use new names for renamed classes symbiote/silverstripe-multivaluefield#120
• API Use new names for renamed classes silverstripe-advancedworkflow#554
• API Use new names for renamed classes silverstripe-elemental-bannerblock#165
• DOC Document renamed classes developer-docs#584

Refactor PRs

CMS 5

• DOC Document deprecations for template layer refactor developer-docs#594
• API Deprecations for template layer #11420
• API Deprecations for template layer silverstripe-cms#3012
• ENH Suppress deprecation notices for API we can't avoid silverstripe-admin#1835
• ENH Suppress deprecation notices for API we can't avoid silverstripe-assets#649
• ENH Suppress deprecation notices for API we can't avoid silverstripe-asset-admin#1501
• ENH Suppress deprecation notices for API we can't avoid silverstripe-contentreview#264
• ENH Avoid using deprecated API silverstripe-advancedworkflow#555
• ENH Avoid using deprecated API silverstripe-staticpublishqueue#207
• ENH Suppress deprecation notices for API we can't avoid silverstripe-userforms#1333

CMS 6

Kitchen sink CI run: https://github.com/creative-commoners/recipe-kitchen-sink/actions/runs/11584927001

• API Refactor template layer #11405
• ENH Update code to reflect changes in template layer silverstripe-admin#1833
• ENH Improve typing to support refactored template layer silverstripe-assets#647
• ENH Update code to reflect changes in template layer silverstripe-asset-admin#1499
• ENH Improve type safety to support refactored template layer silverstripe-cms#3010
• ENH Update code to reflect changes in template layer silverstripe-contentreview#265
• ENH Update code to reflect changes in template layer silverstripe-advancedworkflow#556
• MNT Update test to reflect changes to template/model layers silverstripe-elemental#1260
• ENH Update code to reflect changes in template layer silverstripe-userforms#1334
• ENH Update code to reflect changes in template layer silverstripe-staticpublishqueue#208
• ENH Update code to reflect changes in template layer silverstripe-mfa#570
• ENH Update code to reflect changes in template layer silverstripe-behat-extension#287
• DOC Document changes to template layer developer-docs#591

I've made an example of what a twig module may look like, including twig versions of the simple theme templates.
https://github.com/GuySartorelli/silverstripe-twig
The readme notes some ways template creators will need to workaround the way the abstraction works - notably having to use getRawDataValue() for conditional checks.
If you want to try it out to test the abstraction, follow the instructions in the readme to set it as the renderer for PageController

Migration PRs

CMS 5

• API Deprecate API which is moving into its own module #11454
• DOC Document deprecations for migrating the template engine developer-docs#623

CMS 6

Kitchen sink CI

• API Update API to suit its status as its own module silverstripe-template-engine#1
• ENH Add new module to CMS 6 supported list supported-modules#44

The above PRs should be merged before any others, since CI won't be happy on the others until the new repo is in packagist, and it can't be added to packagist until composer.json is added.
Supported modules PR is needed for GHA to generate a matrix so we can actually test the module itself.

• API Remove API which is now in silverstripe/silverstripe-template-engine #11451
• API Use class from new template engine module silverstripe-behat-extension#289
• API Use class from new template engine module silverstripe-assets#654
• DEP Explicitly require and test silverstripe/template-engine recipe-core#103
• MNT Add silverstripe/silverstripe-template-engine tests to test suite recipe-kitchen-sink#75
• DOC Document moving template engine into its own module developer-docs#622
• MNT Run module-standardiser silverstripe-template-engine#2
• MNT Run module-standardiser silverstripe-template-engine#3

[emteknetnz]

Main question for me is just how limited this is, particularly given that there's a bunch of pjax requests made in the cms that use .ss templates as part of the server rendered HTML, which presumably aren't getting changed any time soon.

Using twig as an example and let's pretend there's a working twig renderer for Silverstripe, let say that a 3rd party module uses twig templates, or a website project uses twig templates. Will the CMS still be functional, or is there now suddenly a need for the module / project to override ALL core .ss templates with equivalent .twig templates?

Also the basically the same question, this time with one third-party module using .ss templates and another third-party module using twig templates in the same project?

If all .ss templates need to be replaced, then while this does clean up the code, it seems like it has nearly zero practical real-world usage

[GuySartorelli]

With this approach, ALL templates for a project must be in the same template syntax.
So if you're using the ss template renderer for your project, ALL templates must be in ss syntax.
If you've implemented a twig template renderer that follows the abstract API, ALL templates in your project (including ones usually provided by framework, admin, cms, etc) must be in twig syntax.

This is as per the description in https://github.com/silverstripeltd/product-issues/issues/875:

This would mean people would be free to replace the rendering engine with one of their choice if they really want to - with the caveat that they then need to convert and maintain maintain all templates from supported and third-party modules in their template syntax of choice. It gives people freedom, but doesn't give the CMS Squad responsibility for those choices.
...
after all of the above, we can completely replace the template engine in a project (with the caveat that all ss templates would also need to be replaced in that project)

If we wanted to, we could do some follow-on work to allow defining different rendering engines for different controllers, or something like that, so LeftAndMain could use .ss templates and everything else (i.e. the frontend) could use .whatever templates. But the abstraction itself has to happen first.

[emteknetnz]

Yeah isolating the CMS templates away from everything else seems like it should be done as part of this card, it's absolutely crucial for this to have any real world usage

So just to be clear ... something like this - https://github.com/silverstripe/silverstripe-elemental-bannerblock/blob/3/templates/SilverStripe/ElementalBannerBlock/Block/BannerBlock.ss - if you have website project running twig, you now CANNOT use this module because the template would be nested on a PageController which uses twig rather than .ss? ... unless you re-implement BannerBlock.twig in your website codebase?

[GuySartorelli]

Yeah isolating the CMS templates away from everything else seems like it should be done as part of this card

I would prefer it be a separate card. This is going to be a big change as it is.

it's absolutely crucial for this to have any real world usage

If by real world usage you mean someone implementing a different template engine with the abstracted API, I agree. I'm not sure that's a primary driver for this work, though Jackson can confirm either way.

if you have website project running twig, you now CANNOT use this module

No, you can use the module. You just need to supply a template in the syntax of your choice. Even changing template engine per controller, or however else we want to make that distinction, there will be cases like this where you still have to reimplement some templates.

In this particular case you would have to implement a controller specifically for that elemental block and declare that it uses ss templates, assuming you declared that the main elemental controller uses twig.

But imagine this block uses an include that is shared across some other things, and you've replaced those other things (and therefore also the include) with twig templates. Now you have a discrepancy where there's two versions of that include. You have to replace both if you make changes and want them in sync.

[emteknetnz]

In this particular case you would have to implement a controller specifically for that elemental block and declare that it uses ss templates, assuming you declared that the main elemental controller uses twig.

I'm thinking we'd probably want some sort of backwards compatible logic where if there's controllers (or just namespaces) that don't have any "config" defined that it would just fallback to using .ss templates

With this particular banner block example, it's "nested" in PageController:

templates/Layout/Page.ss (PageController)
  ...
  $ElementalArea 
  templates/DNADesign/Elemental/Models/ElementalArea.ss <<< third-party module (elemental)
    ...
    <% loop $ElementControllers %>
      $Me
      templates/DNADesign/Elemental/Layout/ElementHolder.ss
      ...
	$Element
        templates/SilverStripe/ElementalBannerBlock/Block/BannerBlock.ss <<< third-party module (banner-block)
           ...
           $File, $Title, etc

If you were running Twig for your site, presumably in this case you'd need to
a) Replace all the elemental templates with twig templates
b) Replace the banner-block template with a twig template?

Seems like for a "regular" site, where frontend content is basically just pages (SiteTree) + elemental blocks, that you need to be running purely .ss or twig (or something else), though not a combination? Basically it's breaks a lot of the "plug and play" of installing silverstripe modules?

This wouldn't be an issue for an XHR style site (e.g. statically generated site) where little bits of content comes in from individual controller endpoints

Do I have this correct?

[GuySartorelli]

I'm thinking we'd probably want some sort of backwards compatible logic where if there's controllers (or just namespaces) that don't have any "config" defined that it would just fallback to using .ss templates

Regardless of how we do that, that's a separate card that would follow this one. There's a lot to decide about how we'd do that, including deciding if it even would be per controller or a simple CMS vs front-end separation, or some other way to separate it. Or frankly if we want to do it at all.

This card is just about the initial abstraction. We can worry about follow-on work in follow-on cards.

[michalkleiner]

I understand the benefits of doing the work, but those are, obviously, benefits of doing the work. Are there real world problems that we need to or must solve here? Could the linked issues this would supposedly solve be addressed differently, without this work? I'm definitely not against progressing the CMS in these areas, however I still somehow feel like we are pulling random big impact changes out of a hat without a longterm strategy. Maybe there is one, maybe it just wasn't shared, not sure. Do we have a vision or a roadmap where we want the CMS to be in 5 years? E.g. full React, no graphql, modularised, fully decoupled, replaceable components using Symfony components etc.. Some goal, target, which we can relate all the work to, with some purpose and meaning. Maybe this is not the best issue to be raising this here (not an RFC), but it felt related.

[jakxnz]

👍🏻 this scope sounds great to me

[GuySartorelli]

@michalkleiner

This is part of a wider overall but very vague direction to make Silverstripe CMS architecture cleaner, less coupled, and easier to pull apart without the whole tower collapsing.

I don't think anyone currently has a specific, well articulated long-term vision or strategy for Silverstripe CMS such as could populate a clear roadmap where we can forecast exactly when various features will drop.

At the same time, we have scoped some items like this that help set us (Silverstripe CMS maintainers) and the community up for success. Another example of work like this that has been scoped is various issues related to refactoring LeftAndMain/CMSMain (see silverstripe/silverstripe-cms#2933 and the various issues that have linked to it).

Yes to some degree these are "random big impact changes" - though we are keenly aware that all API breaking changes have the potential to cause upgrade pain, and are intentionally implementing the changes in such a way to minimise that pain.