There is no generic solution.
I assume that your goal is service detection or possibly banner gabbing. Both of these are things that nmap does (among others), so the best way to understand how it works is to see how established tools do it.
How to do service detection
nmap can detect which service is behind a given port by using the -sV flag. The procedure is simple: nmap sends a payload, then checks the response. Sometimes that response tells nmap what service is running.
For example, some HTTP servers will respond to invalid requests like this:
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=UTF-8
[...]
From this response, nmap can tell it's an HTTP server. But what if the server is less talkative and just closes the connection after an improper request? In this case, nmap will try out every single kind of payload that is known to nmap. So if a server only understands SMTP, nmap will try all kinds of payloads until it finally replies to a SMTP request.
Doing this by hand is possible, although very very tedious. You can attempt some common services by hand (HTTP, FTP, SMTP, etc.), but trying out every somewhat popular proprietary protocol will take time.
How to do banner grabbing
Banner Grabbing works in similar ways. Once you figured out what kind of service is on the other side, you will have to try to figure out the version of that software. Sometimes this is baked into the protocol in some way, and it will be easy for you to figure out that the other side is a server running nginx in version 1.10.0 or Minecraft in version 1.7.10. Other times, the server will not be very talkative, and perhaps you need to do some educated guessing.
For example, you could check which features are enabled. This could give you an indication as to how old the version is (e.g. feature X was introduced in version Y), and provide a lower bound. You could also check how the server responds to certain malformed requests.
Again, tools like nmap have big databases that can do all of this for you, so in practice, there is no reason to do this "by hand", at least not until you know established tools didn't find anything. And then again, it may just be some proprietary in-house software.