Section 3.3.3 Payload of the MQTT 3.1.1 Specification states that, "The Payload contains the Application Message that is being published. The content and format of the data is application specific."
In order to reliably dissect the payload then, Wireshark must be told how to do that. In other words, it must be told what protocol dissector to hand off the payload to, when applicable. The MQTT Message Decoding preferences serve that purpose, and it's based on the MQTT Topic. You basically tell Wireshark for a given topic pattern and match criteria, which dissector Wireshark should call in order to properly interpret/dissect the payload.
The "Match criteria" of the topic pattern can be one of, "Equal to", "Contains", "Starts with", "Ends with", or "Regular Expression".
The "Topic pattern" is the topic of interest, or some part of that topic that will be used by the match criteria.
"Decoding" is either "none" or "compressed", and this tells Wireshark whether the payload needs to be decompressed first or not before handing off the payload to the appropriate payload protocol dissector.
Finally, the "Payload protocol" is simply the name of the protocol dissector that will get handed the payload once a match of the topic pattern is made.
Again, the payload is application-specific, so you must know how to match the topic to the payload in order for Wireshark to be able to dissect it.
See the Protobuf section of the Wireshark Wiki and 11.21. Protobuf Search Paths in the WSUG - Wireshark User's Guide.