gh-151303 : Improve SyntaxError suggestions for common operator typos and cross-language mistakes

[Aniketsy]

fixes #151303

• Issue: Improve SyntaxError suggestions for common operator typos and cross-language mistakes #151303

[StanFromIreland]

Won't this break when __future__.CO_FUTURE_BARRY_AS_BDFL is true?

[Aniketsy]

hmm , yes

aniket@DESKTOP-074O80J:/mnt/d/cpython/cpython$ ./python.exe -c "
__futu> from __future__ import barry_as_FLUFL
> 1 < > 2
> "
  File "<string>", line 3
    1 < > 2
      ^^^
SyntaxError: invalid syntax.  Maybe you meant '!=' instead of '<>'?

please let me know your thoughts on this ...

aniket@DESKTOP-074O80J:/mnt/d/cpython/cpython$ git diff Grammar/python.gram
diff --git a/Grammar/python.gram b/Grammar/python.gram
index 61a6d5ea90..19ef406437 100644
--- a/Grammar/python.gram
+++ b/Grammar/python.gram
@@ -805,7 +805,11 @@ compare_op_bitwise_or_pair[CmpopExprPair*]:
     | invalid_eqeqeq

 invalid_diamond_op:
-    | a='<' b='>' { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax.  Maybe you meant '!=' instead of '<>'?") }
+    | a='<' b='>' {
+        (p->flags & PyPARSE_BARRY_AS_BDFL)
+        ? RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax.  Maybe you meant '<>' instead of '< >'?")
+        : RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax.  Maybe you meant '!=' instead of '<>'?")
+    }
 eq_bitwise_or[CmpopExprPair*]: '==' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Eq, a) }
 invalid_eqeqeq:
     | a='==' b='=' { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax.  Maybe you meant '==' instead of '==='?") }

output:

aniket@DESKTOP-074O80J:/mnt/d/cpython/cpython$ ./python.exe -c "1 < > 2"
  File "<string>", line 1
    1 < > 2
      ^^^
SyntaxError: invalid syntax.  Maybe you meant '!=' instead of '<>'?

aniket@DESKTOP-074O80J:/mnt/d/cpython/cpython$ ./python.exe -c "
__futu> from __future__ import barry_as_FLUFL
> 1 < > 2
> "
  File "<string>", line 3
    1 < > 2
      ^^^
SyntaxError: invalid syntax.  Maybe you meant '<>' instead of '< >'?

[skirpichev]

hmm, yes

No, this example doesn't break __future__ import. But I suspect, that __future__.CO_FUTURE_BARRY_AS_BDFL is the reason that your code doesn't work without spaces. This can be "fixed" by this patch:

diff --git a/Tools/build/generate_token.py b/Tools/build/generate_token.py
index 9ee5ec86e75..28a0ce1ff78 100755
--- a/Tools/build/generate_token.py
+++ b/Tools/build/generate_token.py
@@ -178,7 +178,7 @@ def generate_chars_to_token(mapping, n=1):
 
 def make_c(infile, outfile='Parser/token.c'):
     tok_names, ERRORTOKEN, string_to_tok = load_tokens(infile)
-    string_to_tok['<>'] = string_to_tok['!=']
+#   string_to_tok['<>'] = string_to_tok['!=']
     chars_to_token = {}
     for string, value in string_to_tok.items():
         assert 1 <= len(string) <= 3

Look at the difference:

$ cat a.py
1 =! 2
$ cat a.py | ./python -m tokenize
1,0-1,1:            NUMBER         '1'            
1,2-1,3:            OP             '='            
1,3-1,4:            OP             '!'            
1,5-1,6:            NUMBER         '2'            
1,6-1,7:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''             

vs

$ cat a.py
1 <> 2
$ cat a.py | ./python -m tokenize
1,0-1,1:            NUMBER         '1'            
1,2-1,4:            OP             '<>'           
1,5-1,6:            NUMBER         '2'            
1,6-1,7:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''             

Regardless on a joke, it looks as a tokenizer bug. I'll open a separate issue.

[Aniketsy]

thanks for looking into this, i'll revert the changes

[StanFromIreland]

No, this example doesn't break future import.

It still does break it, we're suggesting != for < > which is invalid syntax when __future__.CO_FUTURE_BARRY_AS_BDFL is true.

[skirpichev]

Yes, this is wrong, see my comment below. But nothing to do with __future__.barry_as_FLUFL.

Currently, we have a hidden token (<> is alias for !=) in the grammar. So, this patch will not produce an error without a space between. See #151464.

[skirpichev]

Sorry, this doesn't look right for me, just as in another your PR: #150906 (comment)

[serhiy-storchaka]

Like in the previous PR, check that there is no gap between tokens.

[serhiy-storchaka]

This is more likely "is" than "==".

[serhiy-storchaka]

There should not be space in the middle.