Programmer's guide

Overview

The Excess compiler follows the following principles:

• declarative: extensions writers will inform the compiler of the changes to be made, they will not perform any parsing, semantic binding or code generation.
• extension agnostic: There is no knowledge of extensions in the compiler. All extensions will be injected into the compiler at run time. As such, a compiler without extensions is little more than a host language compiler.
• pipelined: All phases of the compiling process will be available to developers: lexical, syntactical and semantical passes will be performed linearly. Developers will be able to intervene in all such passes.
• match and transform: In general, developers will be asked to describe a pattern for the engine to match in any pass. Conversely, developers will instruct the compiler how to transform said patterns when matched. These transformation will mainly be specified as free functions.
• document based: Source code will be treated as an evolving document unto which changes are made. Such changes can be scheduled at any point in the future and passes shall not yield until all its changes have been applied. Changes can be either global (applied to the whole document) or local (applied to single nodes)
• scoped: A hierarchical state will be kept throughout the compilation process and will be available to transform functions. This scope will provide access to compilation items (such as the current document) as well as custom data stored by extensions.

Pipeline Overview

The pipeline in Excess follows the traditional path of compilers, the source document is parsed into tokens, then a syntax tree is built and then semantic meaning is assigned to the nodes in the tree. In addition, the compiler offers access to an environment interface where the user can configure compilation level properties, such as dependencies.

The pipeline sections are detailed in the following pages:

• Lexical Reference
• Syntactical Reference
• Semantical Reference

Standard Extensions

Although the platform is general enough to handle micro cases (see the xs language), we acknowledge the need for certain standardization in extension writing. This would benefit not only users of said extensions, but also writers, who shouldn't be needlessly writing the same matching code over and over.

As such, we propose a standard way to write most extensions and provide the API for it:

<extension name> <extension id> ( <parameters> )
{
    <extension body>
}  

This is a fairly standard way of doing things in the industry (see C#'s using, lock, etc). It is easy to recognize and implement. Additionally, we make the extension id to be optional and in the future the parameters will be optional as well.

There is also the issue of where in the source code such extensions are allowed. For many cases, the extension makes only sense in certain context (you wouldn't want to see a match extension used as a method) and only in rare cases it makes sense to have it in multiple contexts. As such, all API dealing with extensions receives a parameter kind which can take one of the following values:

• Code: The extension will appear inside user's code (as opposed to declarations)
• Expression: The extension will appear inside user's code, but representing an assignment. You do not have to be specific in your declaration. An extension kind of value Code will be recognized as expression but your transform functions must be ready to make the distinction.
• Member: The extension will look (syntactically speaking) as a method.
• Type: The extension will look (syntactically speaking) as a type.

Scheduling changes

As previously mentioned, Excess treats source code as an evolving document to which changes are applied. The primordial source of those changes is extension definition, injecting these global changes (match and transform) into the documents. However, this is not the only source. Every transform function is free to schedule changes to the current document to occur either in the current pass or in future passes.

All such functions receive a parameter named "scope", which represents the global state of the compilation process and provides access to, among other things, the document being currently processed. Once obtained, you can invoke the change method to schedule changes:

var document = scope.GetDocument<Token, Node, Model>();
return document.change(node, transform);

The change method has a few variations and depending on the type of the transformation function it will schedule the change in the appropriate pass. Additionally, the change method supports named changes. These serve the purpose of scheduling changes into specific pipeline points inside passes. You should consider this internal at the moment.

Compilation Environment

Eventually, Excess documents must be bundled and compiled to produce executable code. This setup is usually called a compilation. At the moment we do not offer engine level interfaces to deal with compilations. We, however, provide a Roslyn-specific compilation object and a general mechanism to address whatever layer is on top of the document. This mechanism is called the environment.

Currently we have two uses for the environment:

• Dependencies: One clear use of Excess is to provide syntactical alternatives to software libraries. As such, it is quite likely a final compilation will need to include foreign dependencies. Additionally, successful compilation needs the proper modules (usings in c#).

• Display: Extensions might want to be displayed in a particular way, the clearest example being which keywords should the text editor highlight.

The functionality of the environment will certainly change, but we offer some examples of the current functionality for reference:

environment
    //type dependency plus usings
    .dependency<Compiler>(new [] {
        "Excess.Compiler",
        "Excess.Compiler.Core",
        "Excess.Compiler.Roslyn"})
    //hardcoded dependency, no usings
    .dependency(string.Empty, path: Path.Combine(assemblyPath, "System.Runtime.dll"))
    //keywords
    .keyword("contract")