rewrite the whole of nav

[plastikfan]

extendio was the first largish project I wrote in go, so I made many mistakes; although in its current state it does what it was designed to do, but the implementation needs a re-write incorpating the lessons learnt from experience. One of the biggest problems in that nav is too big. There are many features all mixed together. We should have more sub packages which are more focused/cohesive.

Actually, I think there is a good case to make a new repository walker (hike, stride, march, ramble, tread) that only does navigation, and leave extendio to be a platform package.

Things to improve

• have a new "trav" top level package (to replace nav) which has many sub packages, eg filter, resume, depth, notify, hooks, async, persist(marshalling), measure (metrics), sample, options, listen, enum (actually enum should stay in extendio)
• make sure that log and config are created as soon as possible within the bootstrap phase
• no inheritance (the agent idea was a poor choice and just reflected my OO background, not making the full switch away from inheritance). We can still have an agent (but should be renamed to be a verb rather than a noun) but its just a function to which you pass a navigator, no inheritence is required
• change the virtual file system to be compatble with os.fs
• make sure all file system operations go via the virtual file system rather than using native directly
• improve the bootstrap process (maybe dependency injection); define a set of clear phases and stick to them so it clears what happens and when; to get rid of the confusion about what type of operation can happen when
• improve the filters with respect to sampling. The sampling feature was much more complicated than it needed to be because of filtering; see issue: implement directory sampling with filtering #373 (FILTERING MUST HAPPEN BEFORE SAMPLING, this will eliminate a lot of the complexity, which manifested itself in the lookahead/caching mechanism that was desgned in as a work-around)
• use code genertors for i18n errors and compose them in such a way so they can be identified by errors.Is
• the LISTEN decorator must precede the FILTER decorator, that way, we can listen for an event that can't be filtered out, by accident, or perhaps make this explicit. It may be that LISTEN should only be dependent of events that are filtered in - this could be an option; actually impelement this in such a way that it is clear to see how this is compsed, ie in a functional/fluent way. Also consider this in the content of resume, which uses the listen feature.
• improve the decoration process especially with respect to filtering, perhaps this could be improved via the observer
• there is currently a silly bug with respect to filtering when a worker-pool is being used and the metrics. The file/folder counts can mis calculated, purely because of the way the decorators are composed. This should be an an easy bug to fix, but it isnt because of the current structure
• ensure the context can be handled better without having to store it; perhaps define a WithContext operator/function, similar to the way its done in charm.log
• navigation frame has some 'model' like properties, so introduce a model and clarify the purpose of the frame (actually, I have changed my mind about a model in the navigator, this is a client side concern only, but we should clarify the purpose of the frame as it is currently a bit muddy)
• be VERY strict about errors, make consistent. If there is any chance an method/function can result in an error, perfer to return the error if it has external significance. I did not recognise the importance of this until fairly recently, and it's time to do something about it, in version 2
• each filter should be able to have multiple patterns, combined logically with AND op
• make use of internal packages more
• get rid of the callback model for options, we only want 1 way of setting options and that is the provided options model
• get rid of the refernce pointers, this was experiemental and of little value
• get rid of the navigator/traverse confusion, use one form of the terminology and make it consistent, or make it clear the difference between navigator or traverser
• build in cancellation feature from the start that includes for sequential processing
• build in the observer concept to make filtering easier
• come up with a better name for the listen feature (waypoint/hibernation) as that was mis-named; The key concept behind Listen is that client callback is invoked on a conditional basis, ie we don't start invoking until a particular condition is met; ie we find a specific fie or directory and we can also stop invoking when another condition is met.
• make sure that listen feature is implemented to make resume easier; part of the rationale of the listen feature was for resume, but it I am not happy with the way these 2 feature interacted; should be improved
• get rid of Enum/En from variable names (this is an anti-pattern)
• get rid of item extension, there's no need for it
• get rid of the nil concept, perhaps we borrow the option concept from rust, but rename it to optional
• change options to work more like they are in charm glamour - TermRendererOption; NB: also consider reactive options style, perhaps we have both style mixed together; ultimately there will be a master options object that can be marshalled into json, for the purposes of resume. Filtering would be a reactive style options as it affects observable events, where as behaviours would be the functional style.

Things that are good and should be retianed:

• hooks
• extended glob filter
• poly filter
• marshalling
• periscope for depth management

Also see 🚀 awesome-go