Missing MQTT v5 property protocol validation and decoding buffer safety checks

[kaabia]

This issue addresses critical omissions in MQTT v5 protocol compliance and buffer safety within the property encoding and decoding logic in wolfMQTT:

1. Missing Protocol Validation: The implementation lacked the mandatory check to ensure that a given property is only used within the specific MQTT v5 packet types allowed by its definition (as outlined in the gPropMatrix).

• Impact: This allowed protocol violations where properties could be sent or received in unauthorized packet types, leading to non-compliant MQTT v5 communication. This resolves an explicit TODO in the code.

2. Buffer Safety Hazard: The property decoding logic in MqttDecode_Props lacked explicit boundary checks before decoding the property identifier (VBI) and before decoding property string data.

• Impact: If a received packet's length field was malformed or maliciously manipulated, the lack of checks could lead to buffer overruns or reading beyond allocated memory during the decoding process.

Area: Protocol, MQTT v5, Property Decoding, Safety

The bugs are protocol compliance and safety logic errors verifiable via static code analysis.

Write:

1. Protocol Validation: Examine MqttEncode_Props and MqttDecode_Props and confirm the original logic did not validate whether a property's packet_type_mask allows its use in the current packet type (e.g., trying to send a Session Expiry Interval in a PUBLISH packet). The fix introduces this check and uses the new MQTT_CODE_ERROR_PROPERTY_MISMATCH error code.

2. Buffer Safety: Analyze MqttDecode_Props to confirm the lack of boundary checks against the total packet length before attempting to read variable-byte integers (VBIs) and property string data. This vulnerability requires a malformed packet to exploit.

Additional Context

Fix Proposal is presented in the following PR: #440