Skip to content

Fix crash on \c/\M- escapes with an invalid byte on a UTF-8 source #1092

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nabsei wants to merge 1 commit into
base: master
Choose a base branch
from
+62 −3
Open

Fix crash on \c/\M- escapes with an invalid byte on a UTF-8 source #1092

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions lib/parser/lexer-strings.rl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,7 +445,11 @@ class Parser::LexerStrings
end

def read_post_meta_or_ctrl_char(p)
@escape = source_buffer.slice(p - 1, 1).chr
# \c and \M- escapes operate on the raw byte value of their target, not on
# a Unicode codepoint, so read it as a raw byte here: the source buffer's
# own encoding (eg. UTF-8) would otherwise make `.ord` below raise on a
# byte that isn't valid on its own in that encoding (eg. \c\xFF).
@escape = source_buffer.slice(p - 1, 1).chr.b

if @version >= 27 && ((0..8).include?(@escape.ord) || (14..31).include?(@escape.ord))
diagnostic :fatal, :invalid_escape
Expand Down Expand Up @@ -477,11 +481,16 @@ class Parser::LexerStrings
end

def slash_c_char
@escape = encode_escape(@escape[0].ord & 0x9f)
# @escape may already carry the source encoding (eg. via encode_escape,
# which force_encodes a raw byte without validating it), so read its
# value as a raw byte here rather than as a character in that encoding:
# a \c/\M control/meta escape operates on bytes, and .ord on a string
# that isn't valid in its tagged encoding raises ArgumentError.
@escape = encode_escape(@escape.b[0].ord & 0x9f)
end

def slash_m_char
@escape = encode_escape(@escape[0].ord | 0x80)
@escape = encode_escape(@escape.b[0].ord | 0x80)
end

def emit_character_constant
Expand Down
50 changes: 50 additions & 0 deletions test/test_parser.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11070,6 +11070,56 @@ def test_control_meta_escape_chars_in_regexp__since_31
SINCE_3_1)
end

# Same escapes as test_control_meta_escape_chars_in_regexp__since_31, but
# from a UTF-8 source (the common case, and what these literals actually
# have by default unless the source is forced to ascii-8bit as above).
# \x9F on its own isn't valid UTF-8, so this can't successfully parse to a
# :str node the way the ascii-8bit case does; it must instead raise a
# graceful diagnostic rather than crash with an unhandled ArgumentError.
def test_control_meta_escape_chars_in_regexp_from_utf8_source
assert_diagnoses(
[:error, :invalid_encoding],
%q{/\c\xFF/},
%q{},
SINCE_3_1)

assert_diagnoses(
[:error, :invalid_encoding],
%q{/\c\M-\xFF/},
%q{},
SINCE_3_1)

assert_diagnoses(
[:error, :invalid_encoding],
%q{/\C-\xFF/},
%q{},
SINCE_3_1)

assert_diagnoses(
[:error, :invalid_encoding],
%q{/\C-\M-\xFF/},
%q{},
SINCE_3_1)

assert_diagnoses(
[:error, :invalid_encoding],
%q{/\M-\xFF/},
%q{},
SINCE_3_1)

assert_diagnoses(
[:error, :invalid_encoding],
%q{/\M-\C-\xFF/},
%q{},
SINCE_3_1)

assert_diagnoses(
[:error, :invalid_encoding],
%q{/\M-\c\xFF/},
%q{},
SINCE_3_1)
end

def test_forward_arg_with_open_args
assert_diagnoses_many(
[
Expand Down
Loading