tcp[12] yields a single byte (8 bits); by dividing the value by 16, you move the higher four bits to the position of the lower four, losing the lower four completely due to integer division. So when you then multiply by 4, you move the original upper four bits to bit positions 5 through 2, while bit positions 1 and 0 are set to 0. This is what you want because the upper four bits express header size in 32-bit units and you need it in 8-bit units.
Unlike that, by using >> 2, you move the upmost 6 bits (bit positions 7 through to 2) to bit positions 5 through to 0, so you do not mask out the value of orginal bits 3 and 2, so if the are not 0, your calculation of the beginning of TCP payload gets broken - unless you apply something like & 0x3f on the result before using it.
So tcp[tcp[12]/16*4] means "the first byte of the TCP payload", tcp[tcp[12]/16*4+5]means "the sixth byte of the TCP payload" (or 0th and 5th if counting from 0 as programmers do).
However, the reason why the capture filter above does not always work is that TCP is not a packet-oriented transport but a stream-oriented one. So the TLS record carrying the certificate does not necessarily start at the first byte of TCP payload, and can span across several packets.
So in a common case, the Server Hello record starts at the first byte of the TCP payload, and the Certificate record starts right after the Server Hello still in the same TCP packet, while the biggest part of it comes in the next TCP packet (whose payload doesn't even begin with a TLS record header).
In another words, even if you would cook a capture filter allowing you to identify the beginning of the Certificate record (there is a length value in TLS records so you could skip the Server Hello), there is no way to capture the second (and further) packets carrying the rest of the certificate as there is nothing like "capture also N packets following the one matching the condition" available in capture filters. Nor is in display filters, except that display filters work with dissection results and dissectors do PDU reassembly, so the complete PDU is available as part of dissection tree of the last packet carrying that PDU.
