How do I know what the upper layer protocol is above TCP or UDP?

edit

Most dissectors that run on tcp or udp explicitly register their association on specific ports, usually those registered with IANA or commonly used in practice and some dissectors have preference settings to allow ports (or a range) to be specified.

The association can be overridden by a user with the "Decode As ..." option.

Other dissectors are heuristic, in that they will inspect a number of octets in the data and determine if this is "their" protocol. Options exist to allow heuristic dissectors first run at data, rather than registered ones (tcp\udp; "Try heuristic dissectors first"). Heuristic dissectors can be fooled and wrongly dissect traffic that isn't "theirs", that's why the option to try them first is off by default.

The default port range for the HTTP dissector is 80,3128,3132,5985,8080,8088,11371,1900,2869,2710

What is the algorithm wireshark uses to know what the next protocol is to decode?

It's a combination of testing the port numbers of the endpoints and looking at the packet data to see what it looks like.

I assume WireShark will decode it properly

There's no guarantee of that. If, for the protocol in question, there's a "heuristic dissector", which looks at the packet data, then, if the data matches what the heuristic dissector is expecting, it'll be dissected properly if neither a by-port-number match succeeds first or another heuristic dissector match succeeds first.

It depends on your configuration.

Edit => Preferences => Protocols. Then pick your protocol and see on which port(s) it will normally detect.

If you hit traffic on another port then it will not be decoded b default. However you can just right click on a relevant packet (SYN packets are not the best for this) and tell Wireshark to decode traffic on this port as the protocol you want.

If this hapens a lot I suggest you add the port to protocol settings. If it only happens once I wouldn't bother with that.

Regards, Hugo.