How does Wireshark distinguish the client vs server packets?

edit

Usually the client is the endpoint that initiates the connection. Easy for TCP, difficult for UDP.

(No luck finding the previous discussion.)
Sometimes it is assumed client port (ephemeral) number is larger than server port number.

And there ya go - hit send and suddenly Google foo works.
Why are some TCP conversations shown backwards/reversed?

I have captured pcap file from OT environment where many devices are talking at once. I need a solid base on how wireshark distinguishes the thing. Is it just port number or something else as well.

See also TCP: Conversations Statistics loyalty to connections initiators

When I do follow tcp stream, I see some packets being recognized falsely as client packet. For example, x11 event messages are generated by the server but it was put in the client side. The comparison of port seems good but does it always work like client ports are always larger. All I want is to distinguish incoming vs outgoing message belonging to the tcp stream. The first packet's incoming vs outgoing is an important part as the later packets can be labeled based on the first packet's information. what would be the most logical way to do so? @Chuckc and @grahamb

epan/follow.c:
The follow code chooses client by who send the first packet. So Wireshark is not consistent in choosing client and server.
Would your captures always have the initial connection setup or would there be mid-stream captures?
Does the protocol include any information that indicates who is client and who is server?

    if (follow_info->client_port == 0) {
        follow_info->client_port = pinfo->srcport;
        copy_address(&follow_info->client_ip, &pinfo->src);
        follow_info->server_port = pinfo->destport;
        copy_address(&follow_info->server_ip, &pinfo->dst);
    }

Thank you very much @Chuckc. The packet captures are from OT environment and lots of devices are talking. Different application protocols are also used and of course some can be clearly identified as server and client. The packet captures when I see it are missing the complete handshake as well. I need a consistent and clear approach. What would you suggest ? My thinking is for the handshakes available, do things best on handshake and for others do best on port information.