`AnchorStart` and `AnchorEnd` do not match start and end of **chunks** as expected.

[hippietrail]

These are needed to work on chunks because chunks are the subdivisions of a document over which PatternLinters/ExprLinters run. I don't think chunk is a properly defined concept though. Instead, the AnchorStart and AnchorEnd refer to the concept of a token stream, which I think is also not a defined concept:

/// A [`Step`] which will match only if the cursor is over the first word-like of a token stream.
/// It will return that token.

The importance of the chunk concept is seen in lint_group.rs:

impl Linter for LintGroup {
    fn lint(&mut self, document: &Document) -> Vec<Lint> {
        let mut results = Vec::new();

        // Normal linters
        for (key, linter) in &mut self.linters {
            if self.config.is_rule_enabled(key) {
                results.extend(linter.lint(document));
            }
        }

        // Pattern linters
        for chunk in document.iter_chunks() {
            let Some(chunk_span) = chunk.span() else {
                continue;
            };

            let chunk_chars = document.get_span_content(&chunk_span);
            let config_hash = self.hasher_builder.hash_one(&self.config);
            let key = (chunk_chars.into(), config_hash);

            let mut chunk_results = if let Some(hit) = self.chunk_expr_cache.get(&key) {
                hit.clone()
            } else {
                let mut pattern_lints = Vec::new();

                for (key, linter) in &mut self.expr_linters {
                    if self.config.is_rule_enabled(key) {
                        pattern_lints.extend(run_on_chunk(linter, chunk, document.get_source()));
                    }
                }

                // Make the spans relative to the chunk start
                for lint in &mut pattern_lints {
                    lint.span.pull_by(chunk_span.start);
                }

                self.chunk_expr_cache.put(key, pattern_lints.clone());
                pattern_lints
            };

            // Bring the spans back into document-space
            for lint in &mut chunk_results {
                lint.span.push_by(chunk_span.start);
            }

            results.append(&mut chunk_results);
        }

        results
    }

A lint group is an arbitrary bunch of lints, and crucially is used for the set of all lints enabled by the user when linting a document through the LSP or any of the plugins.

You can see that the way a document is linted is in two steps:

1. First run all of the "normal" linters over the document as a whole.
2. Iterate over each chunk and run the pattern linters* (including Exprs) over each chunk.

Why do we need anchors at the start and end of a chunk?
Because Patterns and Exprs run over chunks and (just as in regexes) sometimes a pattern needs to know if it's at the start or end (just as ^ and $ in regexes.)

For instance, many patterns are only applicable if not preceded by or not followed by certain words.

To check if there is not a specific prev/next word, there are two possibilities:

1. There is a prev/next word, but it's not one of the words we need. (And there will be whitespace too.)
2. There is no prev/next word at all (and no whitespace). Because we're at the end. But the end of a chunk, not the entire document (or "token stream") since chunks are what Patterns and Exprs operate on.

The only way to be able to check for condition 2. is to be able to check if we're at the beginning or end of the chunk.

---

This is why my linter to change "reach out" to "contact" is incomplete.

Please let me know if any of this is still not clear.

[hippietrail]

In fact this behaviour is quite hidden from tests too unfortunately, since the testing assert macros such as assert_suggestion_result run a single lint over the whole document, regardless of whether it's a "normal" linter or a pattern/expr linter. The tests are not run chunk-by-chunk.

Then again, I seem to be misunderstanding something since AnchorStart seems to work in the linters it's used in. AnchorEnd is another story. No linters currently use it. You can see my attempts to use it in my "reach out" PR branch: #1620

I also find I keep getting confused by how these anchors are different to regex ones. In regex ^ and $ match 0-width (no characters) anywhere, but AnchorStart and AnchorEnd match 1-width (one token) but not anywhere - only when the token is "wordlike". This also makes .then() in SequenceExpr a bit counterintuitive since anchorstart-then-word will actually match the "word" twice - once for the anchor and once for the word. So instead of "a then b" it's "a and b simultaneously".

[elijah-potter]

The "token stream" language is somewhat abstract on purpose. In reality, I probably should have used the word "slice" instead. I wanted (and failed) to make it clear that the size and context of the sequence of tokens was not defined or restricted by the Anchor start/end.

For example, if you pass an AnchorStart an entire document, it would match the first word in the document. On the other hand, if you passed it a chunk (as is the case in an ExprLinter), it would only match the first word of the chunk.

The tests are not run chunk-by-chunk.

I want to make it clear that the tests replicate the behavior of the whole document and thus run chunk-by-chunk. While the tests do not use the lint_group code you mentioned, the underlying ExprLinter blanket implementation runs on a chunk-by-chunk basis as well. We try to use the LintGroup implementation whenever possible because it includes caching, but the fundamental behavior is the same regardless.

impl<L> Linter for L
where
    L: ExprLinter,
{
    fn lint(&mut self, document: &Document) -> Vec<Lint> {
        let mut lints = Vec::new();
        let source = document.get_source();

        for chunk in document.iter_chunks() {
            lints.extend(run_on_chunk(self, chunk, source));
        }

        lints
    }

    fn description(&self) -> &str {
        self.description()
    }
}

I also find I keep getting confused by how these anchors are different to regex ones. In regex ^ and $ match 0-width (no characters) anywhere, but AnchorStart and AnchorEnd match 1-width (one token) but not anywhere - only when the token is "wordlike". This also makes .then() in SequenceExpr a bit counterintuitive since anchorstart-then-word will actually match the "word" twice - once for the anchor and once for the word. So instead of "a then b" it's "a and b simultaneously".

You've nailed it on the head. I would like to make something clear, if it isn't already. When you use anchorstart-then-word, only one copy of the token will end up in the resulting match.

[hippietrail]

I've added a draft PR with a test for each showing the kind of matching I expect to be able to do.

AnchorEnd doesn't seem to work or is very counterintuitive. I can't decide whether AnchorStart is partially or fully working.