Skip to content

Release 0.1.25 - Major changes to internal representation and some new examples

Latest
Latest
Compare
Choose a tag to compare
@sparkslabs sparkslabs released this 03 Feb 19:06
· 5 commits to master since this release
v0.1.25
9ab8298

Overview

Summary

The focus of this release is the result of the independent intermediate
node refactoring project I was undertaking. This was to transform the list
oriented data structure for representing a "pure" intermediate form of
code into a more stuctured/generalisable format.

The motivation behind this change was the realisation that implementation
of end user functions (ie def) would be very difficult with this more
structured approach.

Rationale behind changes

Before this change, pyxie used the following data structures, and code
phases when parsing, analysing, transforming and performing code
generation.

These were not ideal:

  • Code phase: Parser - read in the source, parsed it, and created a
    data structure
    • Data structure: pynodes - these represent the concrete python
      syntax, and are generated during the code generation phase. These
      are python objects in a class hierarchy. Once created they are
      analysed and placed into a context to analyse datatypes.
  • Code phase: Analysis. - Analyses the code, and decorates the existing
    pynodes with type information to understand the programs data and data
    types. (does not create a new data structure).
  • Transform phase: walks the pynode CST, and generates an intermediate
    data structure intended to represent the program in the abstract
    independent of the language used. An independent intermediate form
    if you like.
    • Data structure: independent intermediate form - This is used
      to model the program in a "pure form" - which isn't constrained
      by the source language, and contains enough information for the
      target language output to be generated. That was the theory. In
      practice it was a nested list of list of lists ... Not ideal.
      More on this below.
  • Code generation phase: This walked the independent intermediate form
    (the lists of lists), and created an output data structure representing
    the concrete final program.
    • The third/final data structure is intended to represent the final
      output language. ie it is intended to be a concrete representation of
      the output language. This final data structure is then capable of
      creating the output. Again, forms a hierarchy and could be called
      "CppNodes" (Though they weren't before this change)

The problem here is that while the pynodes are currently well defined (to
a large extent) and that the CppNodes are pretty well defined (even if
they could be better), the independent intermediate form sucked because
it was just nested lists. This meant in practice the code was fragile,
difficult to change, and increasingly difficult to work with. In the early
days of Pyxie this simplistic structure made sense. However as data
analysis becomes more complex and tricky. This mirrors the fact that
the same was true in the early days of the parsing side.

So the focus of this sub-project was to replace the intermediate form,
and tighten up the differences and make clearer in the code base that
we have PyNodes, iiNodes, and CppNodes, where:

  • Pynodes - are the current structures, unchanged - note these are
    currently prefixed Py - PyDefStatement for example.
  • CppNodes - are map to objects/hierarchy/etc that represents C++ programs,
    but made clearer (slightly). These are now prefixed Cpp rather than the
    previous C_ which some of the objects use.
  • iiNodes - represent the independent, intermediate nodes. Since iiNodes
    need to be used in source code where there are either PyNodes + iiNodes
    or CppNodes + iiNodes, they have the prefix "ii" enable differentiation.

Since all 3 of these are actually models, the code for these has all
moved to sit below pyxie.models.

At some point there will be a need to restructure the code more generally.
Parsing, transforming to IINodes and code generation are actually all
transforms, and should sit together. That was out of scope for this
already relatively major internal change.

Changelog

New

  • doc/Versioning.md - semantic versioning as it applies to pyxie
  • doc/WIPNOTES/6.Models.md - start of some docs around the models in use
  • Added explicit notes on licensing of pyxie's output.
    (Short version: I view the output as being derived from your code by you)
  • Language focussed examples / acceptance tests added:
    if if-else pass print while-break while-continue
  • Change arduino profile to support the Adafruit_NeoPixel library
  • neopixel example added

What's been fixed? / Improved

  • Handling if if/elif/else improved/fixed
  • added clean.sh to arduino example
  • added clean.sh to servo example
  • added README.md to simplest_function example (won't work yet)

Internal Changes

  • bin/pyxie now pulls in functionality from pyxie.api to be clearer
    about what the API is
  • added pyxie/api.py - public API for embedding pyxie. (Assumes
    ubuntu host) Core contents:
    • initialise_API(profile="default") - Call first
    • set_profile(profile_name) - Call later
    • PyxieAPI.parse(filename) - parse file, output goes to console
    • PyxieAPI.analyse(filename) - parse & analyse flle, output to console
    • PyxieAPI.codegen(filename, result_filename=None) - parse through
      code generation. Output to console
    • PyxieAPI.compile(filename, result_filename=None) - Compile file,
      result ends up as "result_filename"
  • pyxie/core - changed over to drive the conversion of pynodes to
    cppnodes via iinodes. (aim is to simplify pynodes down the line and
    make iinodes our main location for analysis/etc)
  • Minor changes to .gitignore
  • Minor change to the Makefile to make editting simpler for me...
  • Update clib.py based on changes to clib
  • pyxie/codegen/simple_cpp.py - functionality shifted out to
    pyxie/models/cppnodes.py to allow a model driven approach
  • pyxie/model/cppnodes.py Created. A better code representation
    model for C++. Code generation therefore means transforming from the
    iiNodes to CppNodes
  • pyxie/model/iinodes.py - Introduces iiNodes - which are used to
    represent an intermediate version of a program such that it can bridge
    the conversion between languages
  • pyxie/model/pynodes/operators.py - added args() method to a number
    of classes to unify behaviours
  • pyxie/model/transform.py - Major change - converts PyNodes to
    iiNode representation rather than json objects.

Other

  • doc/newsletter/07-20161110.Pyxie-Release-0.1.24.md corrected release date
  • doc/newsletter/XX-TBD.Pyxie-2017-Goals.md - Unreleased newsletter.
    Interesting notes, will get reused if appropriate later. Rather moot
    contents now though.
  • doc/newsletter/08-TBD.Focus-Or-Pyxie-Release-X.X.XX.md Template for
    next newsletter
Assets 2
Loading