What should be done when detecting faulty frames?

edit

I assume you mean that you've decided that the frame in question isn't really your protocol?

In that case, yes, you should return 0. An example from README.dissector:

static int
dissect_dnp3_tcp(tvbuff_t *tvb, packet_info *pinfo, proto_tree *tree, void *data)
{
    if (!check_dnp3_header(tvb, FALSE)) {
        return 0;

Note that you should return 0 before adding anything to the protocol tree or modifying any columns.

If, on the other hand, you're in the middle of dissecting a frame when you realize something's wrong then generally you'd want to add an expert info indicating that/why you're giving up dissection.