howto://macgyver netstat into a sniffer-part one

by Ed Fisher on 2010-03-31

in Infrastructure

cli

 

We’ve got a couple of good posts here at RetroHack about networking tools like portquery (here, and here,) and of course WireShark is my go to app every time the network is involved. But sometimes adding tools is just not an option. Either you don’t have/don’t want to take time, or you don’t have permission to install anything on a client’s server, or nasty little things like change control pop up, and you’re stuck with a network problem and no tools at hand. As much as I think WireShark should be a part of every default server build, it’s at times like these that I’m grateful for cmd-line gems like netstat. In this two-part post, we’re going to dive head first into the netstat goodness, see what it can do, and then how you can use it as a pseudo-sniffer when nothing else is available.

The netstat command has been around since the early days of NT, and is almost an exact match function and switch to the same command in Linux and Unix operating systems. It is short for network statistics, and is very useful in base lining, troubleshooting, and general information gathering about how a system is performing on a network. It can tell you what ports are open on a host, what active connections you have(in and out,) routing table information, and even layer two stats. Let’s start with a brief review of the basics first, then we’ll get into how to MacGyver netstat into a basic protocol analyser.

 

The Windows Netstat command

Displays protocol statistics and current TCP/IP network connections, ports on which the computer is listening, Ethernet statistics, the IP routing table, IPv4 statistics (for the IP, ICMP, TCP, and UDP protocols), and IPv6 statistics (for the IPv6, ICMPv6, TCP over IPv6, and UDP over IPv6 protocols). Used without parameters, netstat displays active TCP connections.

netstat [-a] [-b] [-e] [-f] [-n] [-o] [-p proto] [-r] [-s] [-t] [interval]

  -a           Displays all connections and listening ports.
  -b           Displays the executable involved in creating each connection or
                listening port. In some cases well-known executables host
                multiple independent components, and in these cases the
                sequence of components involved in creating the connection
                or listening port is displayed. In this case the executable
                name is in [] at the bottom, on top is the component it called,
                and so forth until TCP/IP was reached. Note that this option
                can be time-consuming and will fail unless you have sufficient
                permissions.
  -e           Displays Ethernet statistics. This may be combined with the -s
                option.
  -f            Displays Fully Qualified Domain Names (FQDN) for foreign
                addresses.
  -n           Displays addresses and port numbers in numerical form.
  -o           Displays the owning process ID associated with each connection.
  -p proto   Shows connections for the protocol specified by proto; proto
                may be any of: TCP, UDP, TCPv6, or UDPv6.  If used with the -s
                option to display per-protocol statistics, proto may be any of:
                IP, IPv6, ICMP, ICMPv6, TCP, TCPv6, UDP, or UDPv6.
  -r            Displays the routing table.
  -s           Displays per-protocol statistics.  By default, statistics are
                shown for IP, IPv6, ICMP, ICMPv6, TCP, TCPv6, UDP, and UDPv6;
                the -p option may be used to specify a subset of the default.
  -t            Displays the current connection offload state.
  interval    Redisplays selected statistics, pausing interval seconds
                between each display.  Press CTRL+C to stop redisplaying
                statistics.  If omitted, netstat will print the current
                configuration information once.
  /?           Displays help at the command prompt.

Most of the above works the same in Linux or Unix, except that –p displays the process…much like the -b does in Windows. The switches can be combined or listed separately, ex. "–an: is equivalent to "–a –n".

Here are some examples. Work through these with me. If you are on Windows 7, use an administrative cmd prompt or some of these will fail.

Say you want to see all the open ports that are connected on your system.
netstat [enter]
image

From this simple command, we can tell that my computer’s name is kaos, the ip.addr assigned to my system is 192.168.198.89, that I have several established connections, as well as some that were recently established but have since closed. Let’s look at these a bit closer before we move on.

The column headers define the protocol, the local ip.addr and port, the remote host and port, and the connection state.

Look at the first few lines, starting with this one.
TCP     127.0.0.1:1025     kaos:5354     ESTABLISHED
Basically, this says that I have a service listening on TCP port 1025, and that I am talking to myself from TCP port 5354. Yes, it’s true, I talk to myself. Sometimes when writing this blog, there is no other choice. But what is this? Windows opens dynamic ports starting at 1025, and makes any number of Local Procedure Calls (LPC) to itself for various services. This particular one is for the AirPort Base agent, but we’ll have to go a little further into this before you’ll see how I figured that out. The next several which reference a listening ip.addr of 127.0.0.1 and a source of kaos are all like that…just more conversations with myself.

The first line that shows a conversations with someone other than myself is this one.
TCP     192.168.198.89:1164     www:https     CLOSE_WAIT
This means that I had a conversation where the source ip.addr was mine (192.168.198.89) and the TCP source port was 1164, the destination hostname started with www, and the destination port was TCP 443 (https) but the conversation is over, and was closed down gracefully (CLOSE_WAIT.)

You see, netstat is more interested in connections than full details. If I wanted to know the destination ip.addr, or the FQDN of whoever I was talking to, I would have had to use some switches to tell netstat I wanted to know more. We also see in this line another state that a TCP connection can be in, CLOSE_WAIT.

There are twelve for TCP connections, but most of then are transient in nature, so you will seldom see them. Since there are no states in raw mode and usually no states used in UDP, this column may be left blank. Here is a list of them all:

    ESTABLISHED
           The socket has an established connection.

    SYN_SENT
           The socket is actively attempting to establish a
           connection.

    SYN_RECV
           A connection request has been received from the
           network.

    FIN_WAIT1
           The socket is closed, and the connection is shutting
           down.

    FIN_WAIT2
           Connection is closed, and the socket is waiting for
           a shutdown from the remote end.

    TIME_WAIT
           The socket is waiting after close to handle packets
           still in the network.

    CLOSE 
           The socket is not being used.

    CLOSE_WAIT
           The remote end has shut down, waiting for the socket
           to close.

    LAST_ACK
           The remote end has shut down, and the socket is closed.
           Waiting for acknowledgement.

    LISTEN
       The socket is listening for incoming connections. 

    CLOSING
           Both sockets are shut down but we still don’t have
           all our data sent.

    UNKNOWN
       Dude, wait, whut?

 

Suppose you want to see not just the active and recent TCP connections, but all the TCP ports that are listening.

netstat –ap tcp [enter]

image

We use the –a to show all connections, and the –p to specify a protocol. The switches –a and –p are combined as –ap, and then I specify TCP. Since I am using Windows 7, I see that my computer is listening on RPC, CIFS, MSTSC, NetBIOS, and a bunch of others yet to be identified, and has an established connection to itself on a few ports (Windows talks to itself more than even I do) and also has an established connection to Zeus, which is my domain controller. You could omit the –p to see all the UDP ports you’re listening on, but that would make for a really long screenshot, and not add much to our example. Want to see only the UDP, change your switch to –p udp to get that.

Same thing, but this time, but you just want ip.addrs, not names. This will be very useful later…

netstat –anp tcp [enter]

image

The –n switch means to show only numeric addresses, not names.

Wondering what the heck process has some of those ports open, and to where they are connecting?

netstat –afbop tcp [enter]

image

Here, we add the –f switch to show FQDNs, the –b switch to show the binary, and –o switch to show the PID of the owning process. You could have chosen to leave out either the b or o switches, but I like to see the binary so I get an idea of what the process is, and the PID in case I need to use Task Manager to kill it.

Want to see some error stats? Try these…

netstat –s –p tcp | more [enter]

image

621 fails and 5799 resets aren’t great stats. Since I know I’ve been running a port scanner from this machine, I’m not worried, but if you don’t have a reasonable explanation for a high count, you need to look deeper at your network. The " | more " is just to keep the output from scrolling off the screen, since after the statistics, it will list all of the TCP connections. You can do the same thing with UDP.

netstat –s –p udp [enter]

image

And finally, for ICMP.

image

 

Switching gears, here is how to check your routing table with a little more flair than route print.

netstat -r [enter]

image

Notice that we have a routing table for both IPv4 and IPv6. And do you see that line about persistent connections? That means someone added an IPv4 route that will survive a reboot, probably by using the route add command.

How’s your error count and congestion looking, and do you have anything like IPX still on your LAN?

netstat -e [enter]

image

This shows us bytes and packets in and out, and if there were any discards, errors or unknown protocols.

  • Unicast packets are exactly what they sound like…packets sent directly to your machine, or sent by your machine to a specific host.
  • Non-unicast packets are broadcasts and multicasts.
  • Discards should never show any…if they do, it means the network is so busy your computer cannot keep up with the incoming traffic and is dumping frames, or the network is so busy that it cannot transmit frames on the wire and is timing them out.
  • An occasional error is nothing for concern, especially if you are on a wireless network, but keep an eye on it. A double-digit number in under a day, or a rapidly climbing number, is a sure sign that there are problems at the physical layer. That could be interference, a bad Ethernet cable, a faulty switch port, or something along those lines. The key here is that they are physical layer….don’t go looking at routing, or other VLANs…this problem is local.
  • Unknown protocols indicates that there is a protocol suite on the wire that you computer doesn’t understand. This is common if you still have some HP JetDirect cards SAP’ing IPX on the wire, but is otherwise harmless.

Come back for part two of this post to put it all together. Link will be live on 20100402. While you are waiting, check out this Cyanide & Happiness short, "Waiting for the Bus." language mildly NSFW

You’re welcome to post any questions on what we’ve covered so far by leaving a comment.

You might also enjoy:

  1. howto://macgyver netstat into a sniffer-part two
  2. howto://connect clients to exchange-part one
  3. howto://publish a web farm using TMG 2010-part one
  4. howto://connect clients to exchange-part two

Leave a Comment

Previous post:

Next post: