Ask Your Question
0

Use wireshark to scrape data stream used by ToS

asked 2021-12-05 17:03:47 +0000

Y6v7dKdxYt gravatar image

https://www.reddit.com/r/algotrading/...

I see some discussion above to use wireshark to scrape data stream used by ToS. It is still an unsolved problem. It seems to be a difficult problem to solve.

Let me know when you get to the ssl encryption part in Wireshark. That’s where I am stuck right now.

The encryption seems to be hard to solve. Let alone the binary data form that needs to deciphered, too.

Bear in mind, this is a binary format they use too, not some god awful JSON/XML format that ToS probably uses which consumes 100x more bandwidth.

Does anybody have any idea on whether it is feasible for a person experienced in reverse engineering to decipher the underlying communication data used by ToS? If so, how to tackle this problem effectively?

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted
0

answered 2021-12-05 18:30:37 +0000

grahamb gravatar image

Seems like there are two technical issues here let alone the legality of reverse engineering the data which is presumably protected by some form of contract.

The technical issues are; decrypting the traffic and dissecting the data protocol.

Assuming the connection uses TLS, which is easily determined using Wireshark, the first might be done using some form of MITM proxy but that does depend on the client co-operating with a certificate that isn't issued by the data vendor. If proxying doesn't work then it's much tougher as you need to extract the pre-master secret from the client and this depends on many things including the TLS library used by the client. If TLS isn't used, then you have a tough task working out what encryption is used and how to decrypt the traffic.

If the decryption issue can be overcome and the keying material provided to Wireshark, then the raw traffic data will be visible. You might be lucky and it actually uses a protocol already dissected by Wireshark, if not then it will be plain data as a series of hex bytes. A custom Wireshark dissector could then be used as the protocol is reverse engineered, but again the reverse engineering may be a tough task depending on the complexity.

edit flag offensive delete link more

Comments

Thanks. I will start with the process one step by one step.

First, I listen to my interface en0. (I am not sure there is a way to just listen to ToS app traffic. Is there a way to do so?) But I do something in ToS to make sure it generates some traffic. Then I see the following two IP addresses (with their domain name shown below) are accessed.

https://whatismyipaddress.com/ip/198.... -> analytics-tx-tmp.tdameritrade.comhttps://whatismyipaddress.com/ip/198.... -> tda-tx.thinkorswim.com

http://tda-tx.thinkorswim.com responds with {"error":"symbol is not futures"}. It sounds like an HTTPS server so the communication between ToS desktop and this server is HTTPS.

http://analytics-tx-tmp.tdameritrade.com is not responsive. Now, I try to use host thinkorswim.comto filter the traffic on en0. But I don't see anything. I guess there is something wrong when ...(more)

Y6v7dKdxYt gravatar imageY6v7dKdxYt ( 2021-12-05 19:38:50 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2021-12-05 17:03:47 +0000

Seen: 574 times

Last updated: Dec 05 '21