Libhtp rs v20 #12240
Draft
Libhtp rs v20 #12240
+27,228
−1,950
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ticket: OISF#2696 There are a lot of changes here, which are described below. In general these changes are renaming constants to conform to the libhtp-rs versions (which are generated by cbindgen); making all htp types opaque and changing struct->member references to htp_struct_member() function calls; and a handful of changes to offload functionality onto libhtp-rs from suricata, such as URI normalization and transaction cleanup. Constants renamed to correspond to cbindgen generated names: HTP_OK => HTP_STATUS_OK HTP_ERROR => HTP_STATUS_ERROR HTP_SERVER_* => HTP_SERVER_PERSONALITY_* HTP_REQUEST_INVALID_T_E => HTP_FLAGS_REQUEST_INVALID_T_E HTP_REQUEST_INVALID_C_L => HTP_FLAGS_REQUEST_INVALID_C_L HTP_HOST_MISSING => HTP_FLAGS_HOST_MISSING HTP_HOST_AMBIGUOUS => HTP_FLAGS_HOST_AMBIGUOUS HTP_HOSTU_INVALID => HTP_FLAGS_HOSTU_INVALID HTP_HOSTH_INVALID => HTP_FLAGS_HOSTH_INVALID HTP_AUTH_UNRECOGNIZED => HTP_AUTH_TYPE_UNRECOGNIZED HTP_M_UNKNOWN => HTP_METHOD_UNKNOWN HTP_M_GET => HTP_METHOD_GET HTP_M_POST => HTP_METHOD_POST HTP_M_PUT => HTP_METHOD_PUT HTP_M_CONNECT => HTP_METHOD_CONNECT HTP_STREAM_ERROR => HTP_STREAM_STATE_ERROR HTP_STREAM_TUNNEL => HTP_STREAM_STATE_TUNNEL HTP_REQUEST_LINE => HTP_REQUEST_PROGRESS_LINE HTP_REQUEST_HEADERS => HTP_REQUEST_PROGRESS_HEADERS HTP_REQUEST_BODY => HTP_REQUEST_PROGRESS_BODY HTP_REQUEST_TRAILER => HTP_REQUEST_PROGRESS_TRAILER HTP_REQUEST_COMPLETE => HTP_REQUEST_PROGRESS_COMPLETE HTP_RESPONSE_LINE => HTP_RESPONSE_PROGRESS_LINE HTP_RESPONSE_HEADERS => HTP_RESPONSE_PROGRESS_HEADERS HTP_RESPONSE_BODY => HTP_RESPONSE_PROGRESS_BODY HTP_RESPONSE_TRAILER => HTP_RESPONSE_PROGRESS_TRAILER HTP_RESPONSE_COMPLETE => HTP_RESPONSE_PROGRESS_COMPLETE HTP_PROTOCOL_1_1 => HTP_PROTOCOL_V1_1 HTP_PROTOCOL_1_0 => HTP_PROTOCOL_V1_0 HTP_PROTOCOL_0_9 => HTP_PROTOCOL_V0_9 Functions introduced to handle opaque htp_tx_t: - tx->cfg => htp_tx_cfg(tx) - tx->flags => htp_tx_flags(tx) - tx->is_protocol_0_9 => htp_tx_is_protocol_0_9(tx) - tx->parsed_uri => htp_tx_parsed_uri(tx) - tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx) - tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx)) - tx->request_auth_type => htp_tx_request_auth_type(tx) - tx->request_headers => htp_tx_request_headers(tx) - tx->request_hostname => htp_tx_request_hostname(tx) - tx->request_line => htp_tx_request_line(tx) - tx->request_message_len => htp_tx_request_message_len(tx) - tx->request_method => htp_tx_request_method(tx) - tx->request_method_number => htp_tx_request_method_number(tx) - tx->request_port_number => htp_tx_request_port_number(tx) - tx->request_progress => htp_tx_request_progress(tx) - tx->request_protocol => htp_tx_request_protocol(tx) - tx->request_protocol_number => htp_tx_request_protocol_number(tx) - tx->request_uri => htp_tx_request_uri(tx) - tx->response_headers => htp_tx_response_headers(tx) - tx->response_line => htp_tx_response_line(tx) - tx->response_message => htp_tx_response_message(tx) - tx->response_message_len => htp_tx_response_message_len(tx) - tx->response_status => htp_tx_response_status(tx) - tx->response_status_number => htp_tx_response_status_number(tx) - tx->response_progress => htp_tx_response_progress(tx) - tx->response_protocol_number => htp_tx_response_protocol_number(tx) - htp_tx_get_user_data() => htp_tx_user_data(tx) - htp_table_get_c(tx->request_headers, header) => htp_tx_request_header(tx, header) - htp_table_get_c(tx->response_headers, header) => htp_tx_response_header(tx, header) - htp_table_get_index(tx->request_headers, index) => htp_tx_request_header_index(tx, index) - htp_table_size(tx->request_headers) => htp_tx_request_headers_size(tx) - htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101 and “Upgrade: h2c" header. Functions introduced to handle opaque htp_header_t: - header->name => htp_header_name(header) - bstr_ptr(header->name) => htp_header_name_ptr(header) - bstr_len(header->name) => htp_header_name_len(header) - header->value => htp_header_value(header) - bstr_len(header->value) => htp_header_value_len(header) - bstr_ptr(header->value) => htp_header_value_ptr(header) Functions introduced to handle opaque htp_headers_t: - htp_table_size(headers) => htp_headers_size(headers) - htp_table_get_index(headers, index) => htp_headers_get_index(headers, index) Functions introduced to handle opaque htp_tx_data_t: - d->len => htp_tx_data_len() - d->data => htp_tx_data_data() - htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t - htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty. Functions introduced to handle opaque htp_connp_t: - htp_list_get(connp->transactions, txid) => htp_connp_tx(connp, txid) - htp_list_size(http_state->conn->transactions) => htp_connp_tx_size(connp) - htp_connp_get_connection(connp) => htp_connp_connection(connp) - htp_connp_req_data(connp) => htp_connp_request_data(connp) - htp_connp_req_close(connp) => htp_connp_request_close(connp) - htp_connp_res_data(connp) => htp_connp_response_data(connp) - htp_connp_get_jn_tx(connp) => htp_connp_get_request_tx(connp) - htp_connp_get_out_tx(connp) => htp_connp_get_response_tx(connp) - htp_connp_req_data_consumed(connp) => htp_connp_request_data_consumed(connp) - htp_connp_res_data_consumed(connp) => htp_connp_response_data_consumed(connp) - htp_connp_get_user_data(connp) => htp_connp_user_data(connp) Functions introduced to handle opaque htp_conn_t: - conn->in_data_counter => htp_conn_request_data_counter(conn) - conn->out_data_counter => htp_conn_response_data_counter(conn) Other changes: Build libhtp-rs as a crate inside rust. Update autoconf to no longer use libhtp as an external dependency. Remove HAVE_HTP feature defines since they are no longer needed. Make function arguments and return values const where possible htp_tx_destroy(tx) will now free an incomplete transaction htp_time_t replaced with standard struct timeval Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t as separate arguments. This means the connection parser is no longer fetched from the transaction inside callbacks. SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which now provides normalized URI values. The normalized URI is available with accessor function: htp_tx_normalized_uri() Configuration settings added to control the behaviour of the URI normalization: - htp_config_set_normalized_uri_include_all() - htp_config_set_plusspace_decode() - htp_config_set_convert_lowercase() - htp_config_set_double_decode_normalized_query() - htp_config_set_double_decode_normalized_path() - htp_config_set_backslash_convert_slashes() - htp_config_set_bestfit_replacement_byte() - htp_config_set_convert_lowercase() - htp_config_set_nul_encoded_terminates() - htp_config_set_nul_raw_terminates() - htp_config_set_path_separators_compress() - htp_config_set_path_separators_decode() - htp_config_set_u_encoding_decode() - htp_config_set_url_encoding_invalid_handling() - htp_config_set_utf8_convert_bestfit() - htp_config_set_normalized_uri_include_all() - htp_config_set_plusspace_decode() Constants related to configuring uri normalization: - HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT - HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT - HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID htp_config_set_field_limits(soft_limit, hard_limit) changed to htp_config_set_field_limit(limit) because libhtp didn't implement soft limits. libhtp logging API updated to provide HTP_LOG_CODE constants along with the message. This eliminates the need to perform string matching on message text to map log messages to HTTP_DECODER_EVENT values, and the HTP_LOG_CODE values can be used directly. In support of this, HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE values: HTTP_DECODER_EVENT_UNKNOWN_ERROR => HTP_LOG_CODE_UNKNOWN HTTP_DECODER_EVENT_GZIP_DECOMPRESSION_FAILED => HTP_LOG_CODE_GZIP_DECOMPRESSION_FAILED HTTP_DECODER_EVENT_REQUEST_FIELD_MISSING_COLON => HTP_LOG_CODE_REQUEST_FIELD_MISSING_COLON HTTP_DECODER_EVENT_RESPONSE_FIELD_MISSING_COLON => HTP_LOG_CODE_RESPONSE_FIELD_MISSING_COLON HTTP_DECODER_EVENT_INVALID_REQUEST_CHUNK_LEN => HTP_LOG_CODE_INVALID_REQUEST_CHUNK_LEN HTTP_DECODER_EVENT_INVALID_RESPONSE_CHUNK_LEN => HTP_LOG_CODE_INVALID_RESPONSE_CHUNK_LEN HTTP_DECODER_EVENT_INVALID_TRANSFER_ENCODING_VALUE_IN_REQUEST => HTP_LOG_CODE_INVALID_TRANSFER_ENCODING_VALUE_IN_REQUEST HTTP_DECODER_EVENT_INVALID_TRANSFER_ENCODING_VALUE_IN_RESPONSE => HTP_LOG_CODE_INVALID_TRANSFER_ENCODING_VALUE_IN_RESPONSE HTTP_DECODER_EVENT_INVALID_CONTENT_LENGTH_FIELD_IN_REQUEST => HTP_LOG_CODE_INVALID_CONTENT_LENGTH_FIELD_IN_REQUEST HTTP_DECODER_EVENT_INVALID_CONTENT_LENGTH_FIELD_IN_RESPONSE => HTP_LOG_CODE_INVALID_CONTENT_LENGTH_FIELD_IN_RESPONSE HTTP_DECODER_EVENT_DUPLICATE_CONTENT_LENGTH_FIELD_IN_REQUEST => HTP_LOG_CODE_DUPLICATE_CONTENT_LENGTH_FIELD_IN_REQUEST HTTP_DECODER_EVENT_DUPLICATE_CONTENT_LENGTH_FIELD_IN_RESPONSE => HTP_LOG_CODE_DUPLICATE_CONTENT_LENGTH_FIELD_IN_RESPONSE HTTP_DECODER_EVENT_100_CONTINUE_ALREADY_SEEN => HTP_LOG_CODE_CONTINUE_ALREADY_SEEN HTTP_DECODER_EVENT_UNABLE_TO_MATCH_RESPONSE_TO_REQUEST => HTP_LOG_CODE_UNABLE_TO_MATCH_RESPONSE_TO_REQUEST HTTP_DECODER_EVENT_INVALID_SERVER_PORT_IN_REQUEST => HTP_LOG_CODE_INVALID_SERVER_PORT_IN_REQUEST HTTP_DECODER_EVENT_INVALID_AUTHORITY_PORT => HTP_LOG_CODE_INVALID_AUTHORITY_PORT HTTP_DECODER_EVENT_REQUEST_HEADER_INVALID => HTP_LOG_CODE_REQUEST_HEADER_INVALID HTTP_DECODER_EVENT_RESPONSE_HEADER_INVALID => HTP_LOG_CODE_RESPONSE_HEADER_INVALID HTTP_DECODER_EVENT_MISSING_HOST_HEADER => HTP_LOG_CODE_MISSING_HOST_HEADER HTTP_DECODER_EVENT_HOST_HEADER_AMBIGUOUS => HTP_LOG_CODE_HOST_HEADER_AMBIGUOUS HTTP_DECODER_EVENT_INVALID_REQUEST_FIELD_FOLDING => HTP_LOG_CODE_INVALID_REQUEST_FIELD_FOLDING HTTP_DECODER_EVENT_INVALID_RESPONSE_FIELD_FOLDING => HTP_LOG_CODE_INVALID_RESPONSE_FIELD_FOLDING HTTP_DECODER_EVENT_REQUEST_FIELD_TOO_LONG => HTP_LOG_CODE_REQUEST_FIELD_TOO_LONG HTTP_DECODER_EVENT_RESPONSE_FIELD_TOO_LONG => HTP_LOG_CODE_RESPONSE_FIELD_TOO_LONG HTTP_DECODER_EVENT_FILE_NAME_TOO_LONG => HTP_LOG_CODE_REQUEST_LINE_INVALID HTTP_DECODER_EVENT_REQUEST_LINE_INVALID => HTP_LOG_CODE_REQUEST_BODY_UNEXPECTED HTTP_DECODER_EVENT_REQUEST_BODY_UNEXPECTED => HTP_LOG_CODE_RESPONSE_BODY_UNEXPECTED HTTP_DECODER_EVENT_REQUEST_SERVER_PORT_TCP_PORT_MISMATCH => HTP_LOG_CODE_REQUEST_SERVER_PORT_TCP_PORT_MISMATCH HTTP_DECODER_EVENT_URI_HOST_INVALID => HTP_LOG_CODE_URI_HOST_INVALID HTTP_DECODER_EVENT_HEADER_HOST_INVALID => HTP_LOG_CODE_HEADER_HOST_INVALID HTTP_DECODER_EVENT_AUTH_UNRECOGNIZED => HTP_LOG_CODE_AUTH_UNRECOGNIZED HTTP_DECODER_EVENT_REQUEST_HEADER_REPETITION => HTP_LOG_CODE_REQUEST_HEADER_REPETITION HTTP_DECODER_EVENT_RESPONSE_HEADER_REPETITION => HTP_LOG_CODE_RESPONSE_HEADER_REPETITION HTTP_DECODER_EVENT_DOUBLE_ENCODED_URI => HTP_LOG_CODE_DOUBLE_ENCODED_URI HTTP_DECODER_EVENT_URI_DELIM_NON_COMPLIANT => HTP_LOG_CODE_URI_DELIM_NON_COMPLIANT HTTP_DECODER_EVENT_METHOD_DELIM_NON_COMPLIANT => HTP_LOG_CODE_METHOD_DELIM_NON_COMPLIANT HTTP_DECODER_EVENT_REQUEST_LINE_LEADING_WHITESPACE => HTP_LOG_CODE_REQUEST_LINE_LEADING_WHITESPACE HTTP_DECODER_EVENT_TOO_MANY_ENCODING_LAYERS => HTP_LOG_CODE_TOO_MANY_ENCODING_LAYERS HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER => HTP_LOG_CODE_ABNORMAL_CE_HEADER HTTP_DECODER_EVENT_RESPONSE_MULTIPART_BYTERANGES => HTP_LOG_CODE_RESPONSE_MULTIPART_BYTERANGES HTTP_DECODER_EVENT_RESPONSE_ABNORMAL_TRANSFER_ENCODING => HTP_LOG_CODE_RESPONSE_ABNORMAL_TRANSFER_ENCODING HTTP_DECODER_EVENT_RESPONSE_CHUNKED_OLD_PROTO => HTP_LOG_CODE_RESPONSE_CHUNKED_OLD_PROTO HTTP_DECODER_EVENT_RESPONSE_INVALID_PROTOCOL => HTP_LOG_CODE_RESPONSE_INVALID_PROTOCOL HTTP_DECODER_EVENT_RESPONSE_INVALID_STATUS => HTP_LOG_CODE_RESPONSE_INVALID_STATUS HTTP_DECODER_EVENT_REQUEST_LINE_INCOMPLETE => HTP_LOG_CODE_REQUEST_LINE_INCOMPLETE HTTP_DECODER_EVENT_LZMA_MEMLIMIT_REACHED => HTP_LOG_CODE_LZMA_MEMLIMIT_REACHED HTTP_DECODER_EVENT_COMPRESSION_BOMB => HTP_LOG_CODE_COMPRESSION_BOMB New log events to describe additional anomalies: HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH HTP_LOG_CODE_DEFORMED_EOL HTP_LOG_CODE_PARSER_STATE_ERROR HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED HTP_LOG_CODE_CONNECTION_ALREADY_OPEN HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA HTP_LOG_CODE_INVALID_CONTENT_ENCODING HTP_LOG_CODE_INVALID_GAP HTP_LOG_CODE_ERROR The new htp_log API supports consuming log messages more easily than walking a list and tracking the current offset. Internally, libhtp-rs now provides log messages as a queue of htp_log_t, which means the application can simply call htp_conn_next_log() to fetch the next log message until the queue is empty. Once the application is done with a log message, they can call htp_log_free() to dispose of it. Functions supporting htp_log_t: htp_conn_next_log(conn) - Get the next log message htp_log_message(log) - To get the text of the message htp_log_code(log) - To get the HTP_LOG_CODE value htp_log_free(log) - To free the htp_log_t
catenacyber
requested review from
jasonish,
jufajardini,
victorjulien and
a team
as code owners
Closed
|
WARNING: ERROR: QA failed on SURI_TLPW1_files_sha256. ERROR: QA failed on SURI_TLPR1_alerts_cmp.
Pipeline 23701 |
|
@ct0br0 could I get the pcaps for what is left to do for SURI_TLPW1_files_sha256 ? |
|
Wonder if the issues in #12186 (comment) may be relevant here. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Link to redmine ticket:
https://redmine.openinfosecfoundation.org/issues/2696
Describe changes:
Rebase from #12214 + with
Draft to see CI/QA feedback with latest libhtp-rs changes, and not have too many conflicts