Well, for proprietary protocols that you don't have a specification document for there are two options:
- reverse engineer the protocol by playing with the client to see what kind of data is sent as an reaction to whatever you do, e.g. "if I move in this direction, data looks like this, if I move in another direction, those bytes are different". It's basically a differential analysis
- reverse engineer the game binaries to find the part that generates the network data. This is extremely hard to do unless you are very skilled in reverse engineering binaries.
