AttributeError: 'NoneType' object has no attribute 'char_pos'

[fmigneault]

Description

The following line seems to raise an error on None line:

lark/lark/lexer.py

Lines 615 to 616 in f79772c

	line_ctr = lex_state.line_ctr
	while line_ctr.char_pos < lex_state.text.end:

Running lark=1.3.1 produces the error, while it does not occur with lark==1.2.2.

Example

https://github.com/crim-ca/weaver/actions/runs/18957464025/job/54137521089

   File "/home/runner/work/weaver/weaver/tests/wps_restapi/test_swagger_definitions.py", line 425, in test_collection_input_filter_parsing_error
    sd.ExecuteCollectionInput().deserialize(input_data)
  File "/home/runner/work/weaver/weaver/weaver/wps_restapi/swagger_definitions.py", line 1761, in deserialize
    self.convert(filter_expr, filter_lang)
  File "/home/runner/work/weaver/weaver/weaver/wps_restapi/swagger_definitions.py", line 1724, in convert
    parsed_expr = self.validate(filter_expr, filter_lang)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/weaver/weaver/weaver/wps_restapi/swagger_definitions.py", line 1713, in validate
    return self.parse(filter_expr, filter_lang)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/weaver/weaver/weaver/utils.py", line 401, in wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/weaver/weaver/weaver/wps_restapi/swagger_definitions.py", line 1697, in parse
    parsed_expr = ecql.parse(filter_expr)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/pygeofilter/parsers/ecql/parser.py", line 211, in parse
    return parser.parse(cql_text)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/lark.py", line 677, in parse
    return self.parser.parse(text, start=start, on_error=on_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/parser_frontends.py", line 131, in parse
    return self.parser.parse(stream, chosen_start, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
    return self.parser.parse(lexer, start)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
    return self.parse_from_state(parser_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 100, in parse_from_state
    for token in state.lexer.lex(state):
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/lexer.py", line 689, in lex
    yield lexer.next_token(lexer_state, parser_state)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/lexer.py", line 616, in next_token
    while line_ctr.char_pos < lex_state.text.end:
          ^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'char_pos'

[MegaIng]

Could you please create a small isolated example to reproduce this?

[erezsh]

@fmigneault Thanks for reporting. Are you using a custom lexer, by any chance?

[fmigneault]

I am relying on the lexer definitions found in https://github.com/geopython/pygeofilter/tree/main/pygeofilter/parsers

The test is defined here: https://github.com/crim-ca/weaver/blob/cba028a11d1bee42d5b7c81211ad6ec8b1a717bf/tests/wps_restapi/test_swagger_definitions.py#L403-L426

The error log displays pygeofilter/parsers/ecql/parser.py, so the code should be equivalent to:

from pygeofilter.parsers import ecql

filter_expr = {"test": "bad"}
parsed_expr = ecql.parse(filter_expr)

The operation is expected to fail due to invalid format (purpose of the test), but it was somewhat handled before with (TypeError, ValueError) being raised (https://github.com/crim-ca/weaver/blob/cba028a11d1bee42d5b7c81211ad6ec8b1a717bf/weaver/wps_restapi/swagger_definitions.py#L1710-L1719) rather than the unexpected AttributeError: 'NoneType' access.

[erezsh]

@fmigneault Can you please try running it with PR #1567 and see if that restores the expected behavior?

[fmigneault]

@erezsh
It did not fix it, but the error is different.

====================================================================================================================================== short test summary info ======================================================================================================================================
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data3-cql-text] - AttributeError: 'HashDict' object has no attribute 'end'
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data4-cql2-text] - AttributeError: 'HashDict' object has no attribute 'end'
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data5-ecql] - AttributeError: 'HashDict' object has no attribute 'end'
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data6-cql] - AttributeError: 'HashDict' object has no attribute 'end'
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data7-simple-cql] - AttributeError: 'HashDict' object has no attribute 'end'
============================================================================================= 5 failed, 2923 passed, 3 skipped, 516 deselected, 3 xfailed, 1 xpassed, 342 warnings in 235.52s (0:03:55) =============================================================================================
make: *** [Makefile:420: test-unit-only] Error 1

❯ pip show lark
Name: lark
Version: 1.3.1
Summary: a modern parsing library
Home-page: https://github.com/lark-parser/lark
Author: 
Author-email: Erez Shinan <erezshin@gmail.com>
License: MIT
Location: /home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages
Requires: 
Required-by: pygeofilter

❯ cat /home/francis/dev/miniconda/envs/weaver-py312/lib/python3.12/site-packages/lark-1.3.1.dist-info/direct_url.json
{"dir_info": {}, "url": "file:///home/francis/dev/lark"}%                                                                                                                                                                                                                                            

❯ cd /home/francis/dev/lark && git status
On branch issue1565
Your branch is up to date with 'origin/issue1565'.

Error for one of the failed test:

___________________________________________________________________________________________________________________ test_collection_input_filter_parsing_error[filter_data5-ecql] ___________________________________________________________________________________________________________________
Traceback (most recent call last):
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/runner.py", line 344, in from_call
    result: TResult | None = func()
                             ^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/runner.py", line 246, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
    yield
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/skipping.py", line 263, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/runner.py", line 178, in pytest_runtest_call
    item.runtest()
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/python.py", line 1671, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/python.py", line 157, in pytest_pyfunc_call
    result = testfunction(**testargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/weaver/tests/wps_restapi/test_swagger_definitions.py", line 425, in test_collection_input_filter_parsing_error
    sd.ExecuteCollectionInput().deserialize(input_data)
  File "/home/francis/dev/weaver/weaver/wps_restapi/swagger_definitions.py", line 1761, in deserialize
    self.convert(filter_expr, filter_lang)
  File "/home/francis/dev/weaver/weaver/wps_restapi/swagger_definitions.py", line 1724, in convert
    parsed_expr = self.validate(filter_expr, filter_lang)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/weaver/weaver/wps_restapi/swagger_definitions.py", line 1713, in validate
    return self.parse(filter_expr, filter_lang)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/weaver/weaver/utils.py", line 401, in wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/weaver/weaver/wps_restapi/swagger_definitions.py", line 1697, in parse
    parsed_expr = ecql.parse(filter_expr)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pygeofilter/parsers/ecql/parser.py", line 211, in parse
    return parser.parse(cql_text)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/lark.py", line 677, in parse
    return self.parser.parse(text, start=start, on_error=on_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/parser_frontends.py", line 131, in parse
    return self.parser.parse(stream, chosen_start, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
    return self.parser.parse(lexer, start)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
    return self.parse_from_state(parser_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 100, in parse_from_state
    for token in state.lexer.lex(state):
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/lexer.py", line 691, in lex
    yield lexer.next_token(lexer_state, parser_state)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/lexer.py", line 618, in next_token
    while line_ctr.char_pos < lex_state.text.end:
                              ^^^^^^^^^^^^^^^^^^
AttributeError: 'HashDict' object has no attribute 'end'
--------------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------------------------------------------------

STATE STACK DUMP
----------------
0) {<_binary_spatial_predicate_func :  * INTERSECTS>, <wkt__geometry :  * wkt__multipoint>, <predicate :  * attribute __ANON_3>, <wkt__multipolygon :  * MULTIPOLYGON LPAR LPAR wkt__coordinate_lists RPAR RPAR>, <wkt__geometry :  * wkt__point>, <_binary_spatial_predicate_func :  * CONTAINS>, <_binary_spatial_predicate_func :  * TOUCHES>, <temporal_predicate :  * expression DURING period>, <literal :  * ewkt_geometry>, <_binary_spatial_predicate_func :  * OVERLAPS>, <temporal_predicate :  * expression AFTER DATETIME>, <wkt__multilinestring :  * MULTILINESTRING LPAR wkt__coordinate_lists RPAR>, <predicate :  * expression LESSTHAN expression>, <predicate :  * expression ILIKE SINGLE_QUOTED>, <number :  * FLOAT>, <sum :  * sum PLUS product>, <attribute :  * NAME>, <condition_1 :  * LPAR condition RPAR>, <ewkt_geometry :  * wkt__geometry>, <sum :  * sum MINUS product>, <predicate :  * expression IN LPAR expression __predicate_star_4 RPAR>, <atom :  * MINUS atom>, <wkt__linestring :  * LINESTRING LPAR wkt__coordinate_list RPAR>, <predicate :  * spatial_predicate>, <predicate :  * expression NOT IN LPAR expression RPAR>, <condition :  * condition AND condition_1>, <wkt__geometrycollection :  * GEOMETRYCOLLECTION LPAR wkt__geometry __wkt__geometrycollection_star_0 RPAR>, <predicate :  * expression LIKE SINGLE_QUOTED>, <literal :  * SINGLE_QUOTED>, <predicate :  * expression IS NOT NULL>, <wkt__geometry :  * wkt__geometrycollection>, <literal :  * BOOLEAN>, <product :  * product STAR atom>, <predicate :  * attribute EXISTS>, <wkt__geometry :  * wkt__linestring>, <temporal_predicate :  * expression DURING OR AFTER period>, <spatial_predicate :  * _binary_spatial_predicate_func LPAR expression COMMA expression RPAR>, <start :  * condition>, <predicate :  * EXCLUDE>, <atom :  * literal>, <spatial_predicate :  * BBOX LPAR expression COMMA full_number COMMA full_number COMMA full_number COMMA full_number RPAR>, <sum :  * product>, <predicate :  * expression IN LPAR expression RPAR>, <atom :  * attribute>, <predicate :  * expression EQUAL expression>, <condition_1 :  * predicate>, <spatial_predicate :  * RELATE LPAR expression COMMA expression COMMA SINGLE_QUOTED RPAR>, <_distance_spatial_predicate_func :  * DWITHIN>, <predicate :  * expression NOT BETWEEN expression AND expression>, <predicate :  * expression IS NULL>, <product :  * atom>, <condition :  * condition_1>, <_binary_spatial_predicate_func :  * WITHIN>, <wkt__point :  * POINT LPAR wkt__coordinate RPAR>, <_binary_spatial_predicate_func :  * DISJOINT>, <wkt__multipolygon :  * MULTIPOLYGON LPAR LPAR wkt__coordinate_lists RPAR __wkt__multipolygon_star_2 RPAR>, <predicate :  * expression NOT ILIKE SINGLE_QUOTED>, <_binary_spatial_predicate_func :  * CROSSES>, <predicate :  * expression __ANON_0 expression>, <$root_start :  * start>, <atom :  * NAME LPAR expression RPAR>, <predicate :  * expression __ANON_2 expression>, <attribute :  * DOUBLE_QUOTED>, <wkt__multipoint :  * MULTIPOINT LPAR LPAR wkt__coordinate RPAR RPAR>, <spatial_predicate :  * BBOX LPAR expression COMMA full_number COMMA full_number COMMA full_number COMMA full_number COMMA SINGLE_QUOTED RPAR>, <temporal_predicate :  * expression BEFORE OR DURING period>, <wkt__geometry :  * wkt__multilinestring>, <product :  * product SLASH atom>, <condition_1 :  * NOT predicate>, <atom :  * NAME LPAR RPAR>, <wkt__geometry :  * wkt__polygon>, <number :  * INT>, <predicate :  * expression __ANON_1 expression>, <atom :  * NAME LPAR expression __predicate_star_4 RPAR>, <wkt__geometry :  * wkt__multipolygon>, <literal :  * envelope>, <predicate :  * INCLUDE>, <predicate :  * expression NOT LIKE SINGLE_QUOTED>, <wkt__polygon :  * POLYGON LPAR wkt__coordinate_lists RPAR>, <temporal_predicate :  * expression BEFORE DATETIME>, <condition :  * condition OR condition_1>, <wkt__multipoint :  * MULTIPOINT LPAR wkt__coordinate_list RPAR>, <wkt__geometrycollection :  * GEOMETRYCOLLECTION LPAR wkt__geometry RPAR>, <predicate :  * temporal_predicate>, <literal :  * number>, <spatial_predicate :  * _distance_spatial_predicate_func LPAR expression COMMA expression COMMA number COMMA distance_units RPAR>, <predicate :  * expression BETWEEN expression AND expression>, <wkt__multipoint :  * MULTIPOINT LPAR LPAR wkt__coordinate RPAR __wkt__multipoint_star_1 RPAR>, <attribute :  * QUALIFIED_NAME>, <ewkt_geometry :  * SRID EQUAL wkt__INT SEMICOLON wkt__geometry>, <_binary_spatial_predicate_func :  * EQUALS>, <_distance_spatial_predicate_func :  * BEYOND>, <predicate :  * expression NOT IN LPAR expression __predicate_star_4 RPAR>, <expression :  * sum>, <atom :  * LPAR expression RPAR>, <predicate :  * expression MORETHAN expression>, <envelope :  * ENVELOPE LPAR number number number number RPAR>}