I assume you mean that you've decided that the frame in question isn't really your protocol?
In that case, yes, you should return 0. An example from README.dissector:
static int
dissect_dnp3_tcp(tvbuff_t *tvb, packet_info *pinfo, proto_tree *tree, void *data)
{
if (!check_dnp3_header(tvb, FALSE)) {
return 0;
Note that you should return 0 before adding anything to the protocol tree or modifying any columns.
If, on the other hand, you're in the middle of dissecting a frame when you realize something's wrong then generally you'd want to add an expert info indicating that/why you're giving up dissection.