So I’d like to share a word or two about firewalls. More specifically, their configuration and the responses they provide (or don’t) when they encounter traffic that is, shall we say, less than desirable. My goal here is to bring some rationality to the way our maligned little friends act on the wire, simplify our network troubleshooting, and maybe, just maybe, make it easier for you to convince management that the screws are a little too tight.
Let’s make sure we frame this post properly. I am talking about, and a big fan of, packet filtering hardware firewalls. We’re talking about the layer four devices that sit on our network between the Internet and our internal network, between the DMZ and everything else, and sometimes even between sections of our WAN. They are both our first and our last line of defence, make sure that administrative misconfigurations don’t lead to incidents, and can provide us with great information about what’s happening on the wire. While my personal favourite will always be Cisco, I’ve used Juniper, Checkpoint, and others, and the concepts I am about to discuss apply equally to all, even if my vocabulary might seem Cisco-centric. While software-based firewalls (like Windows built-in firewall, ZoneAlarm, et al.) have their place and purpose, they are not a part of this conversation.
Do we even want them anymore? YES!
See the above paragraph. Firewalls are a key part of a defence in depth strategy, and act as a filter to block traffic coming into our network, or leaving our network, that we don’t want on the wire. They are also great at alerting us to the existence of that traffic, so that if we have an infected host on the internal network, we will know immediately if the box tries to connect to an IRC channel. As choke points, firewalls also offer us an excellent way to shun attacking hosts or compartmentalise different areas of our network, increasing our security posture. Just remember (and remind management) that they are not magical. Firewalls filter based on protocol and port. An attack against a vulnerable web server over HTTP will fly right through a firewall that is configured to allow HTTP traffic to pass. Just ask all those Nimda infected machines out there how much good the firewall did them. A firewall should block protocols we don’t want on our wire, should prevent an admin from accidentally firing up an anonymous write FTP service on a DMZ host, and give us the juice to shun an attacker or infected host. It should not be looked upon as our only line of defence.
Who should manage them? NOT IT!
(Well, actually, usually I am it, that’s why I can write this article.) This is a sticky question in a lot of environments. Is the firewall a security device, or a network device, or something in between? The short answer is that ‘it depends.’ The longer answer is that there is no concise and simple response, but these points should help guide you to an answer that works for your environment.
- More than one person-the worst situation you can find yourself in is the one that starts with a single person being able to do X, and ends with "they’re not answering the phone." Only slightly less a bad thing than this is when you do have multiple people, but they work for different teams. Conflicts of interest, playing "who did it?" and inconsistencies in how things are done are just some of the fun you can look forward to in that situation. If you have a dedicated security team with the expertise to manage the firewall, have them do it. If not, the network team is the next best choice.
- Only the most experienced-considering how much trouble a misconfigured firewall can cause, and how often the firewall is implicated in problems simply because it is there and convenient to blame, you want your firewall admins to be on their A game at all times. They need to understand networking, client-server application behaviours, to be able to take and read a trace, to white-board explanations to folks who don’t understand any of this, and sometimes even to take those 2 AM phone calls from the CIO asking "what the hell is going on?" You don’t want a FNG in that role if you can avoid it. If you do have to go with raw talent, get them some training so that they can succeed. OJT may be more valuable, but issues involving the firewall are always time-sensitive, and usually impact everything. They need a good starting point.
- Separation of duties-server admins should NOT be firewall admins. It’s a direct conflict of interest. The firewall is designed to restrict traffic to only that which is explicitly permitted. Server admins should be able to define those requirements. If they cannot, sometimes the time crunch of deploying a new system can lead to the ip any approach, with the intended but never realised "we’ll fix it later" promise.
But no matter who manages it…for Pete’s* sake, make sure they all log in with unique credentials. No one should be logging in as admin or root. *Pete is my imaginary friend, who gets really upset when he sees audit logs with admin, root, administrator, etc.
Deny all has never cost someone their job.
But it has saved their arse. It’s a fact. Firewall’s default posture is to implicitly deny everything that is not explicitly permitted. Use that. Embrace that. Love that. Never, EVER, try to get around that. Sure, an ip any might seem like a good idea at the time…it might even facilitate quickly ruling the firewall out as the cause of some problem, but it is not the right way to go. Look at it another way…there are 65,536 TCP ports, 65,536 UDP ports, maybe a dozen or so ICMP types actually in use, and about 8 or 9 IP types commonly encountered. Do the math…it is far easier to block all and permit only what you really want, then to permit all and try to block the bad packets. Evil bit aside, you just can’t win if that is your approach.
Bypassing the firewall? How ’bout "NO."
Watch your design, especially with DMZ servers, VMs, and so-called backup networks. The idea of having a firewall is to provide a separation between the zones you don’t fully trust (the DMZ, the Internet) and the zone you do (the internal network.) Multihomed hosts, VM hosts with NICs on both the DMZ and internal networks, and backup networks in particular, can all provide a path into your internal network, bypassing the controls of the firewall completely. If you are going to have a backup network, and OOB management network, or the like, make sure that network is as ‘untrusted’ as the DMZ, and filter it from the internal network.
ICMP Echo is not the enemy.
So tell me, why is it that you block ICMP? No, I understand that we don’t want redirects, or subnet requests, or timestamp requests, or other bad things like that. That’s not what I asked. I asked….why is it that you block ICMP? Maybe I should make myself more clear…why is it that you block ICMP ECHO? Ping, while rudimentary at best, still proves very useful in troubleshooting basic network connectivity. It confirms name resolution and routing in one command….heck, how many of you ping a hostname to get the ip.addr instead of doing a host or an nslookup? Come on, show of hands…how many?
See, here’s the thing. The regular users are getting wise to the ways of the net. Ping may not make much sense to them, but they know enough to open a cmd prompt and ping www.yourwebsite.com. I’m not saying to open the floodgates of ICMP, but allowing ECHO to your website is not the end of the world. You can rate limit it if you fear a ping flood, size limit it if you fear a DoS, and your IPS can shun a source that is doing a ping sweep. You will find utility greatly outweighs any reduction in security by doing the following
- allowing ICMP ECHO requests to your web server,
- allowing ICMP ECHO requests to your VPN concentrator, and
- allowing ICMP ECHO requests throughout your internal network.
- Of course, allow the corresponding ICMP ECHO REPLIES to all of the above.
The Ping of Death is so old that if you are still running susceptible systems you deserve to have them bluescreen. Heck, you probably want them to so folks will finally upgrade! And while your security guys (the audit ones, not the ones who are technical) may fear folks using traceroute to map your network, remind them that Linux boxes use UDP, not ICMP, to trace, and the awesome Windows tool TCPING uses TCP and incrementing TTLs to do the same thing as an ICMP based trace does. So while you are defending the realm from n00bs and wannabes, you’re doing it wrong.
NAT is EVIL!
So much so that I used to have a blog tag dedicated just to that point. They’re all under older rants now if you’re curious. Hey, NAT’ing traffic going from the internal network out to the Internet is just fine. You should be doing that. NAT’ing traffic from the Internet into the DMZ is okay too. But never, ever, EVER NAT traffic from the internal network to the DMZ, or the DMZ to the internal network. Trust me on this one folks. Between the fun you will have with DNS entries, the protocols you will have to fix up because they include network addresses in the application layer, pinging an internal name to see a response from an external address (harmless itself, but you will be questioned about it ever week from now until the end of time,) and the protocols you will just plain damn break because they don’t NAT (IPSec, RPC DCOM, SIP, RTP, TLS, et al.) it is just not something you want to do. If you don’t want your internal users hitting your website over the Internet, just split your DNS! Personally, I would rather they do…you’ll find out about any issues a lot quicker if they are experienced internally, and you might even get the chance to fix them before an external customer notices.
When to drop? On the Internet.
Dropping packets into the bit bucket means that the firewall does not respond to them. To the sender, there is no perceptible difference between packets that are dropped by the firewall, and packets that are sent to a down, or non-existent host. On the Internet access-list, we want to drop silently all unsolicited packets that are not explicitly permitted. That way, we’re not giving away any more information about our network, our hosts, or our configuration than is absolutely necessary. We’re making the bad guys work for it. Of course, legitimate clients will not know that their packets are being dropped, and may continue to send packets until the application time-out is reached, but that will be the exception and not the norm.
When to block? Everything internal.
Blocking packets means that the firewall responds to undesired packets with some sort of message back to the sender. Usually this is an ICMP UNREACHABLE (3,9 or 3,10) message, that includes the first several bytes of the offending packet. A client will react immediately to such a message by stopping, and sending no further packets, as opposed to dumbly retransmitting as if that will help. If you are using a protocol analyser, you can see not only traffic is being blocked by the firewall, but what traffic is being blocked. With that you can go to the firewall admins to open required ports instead of continuing to wonder where the hell your traffic is going. Or, to put it another way…hey firewall admin! You know how every single time something doesn’t work on the LAN, some n00b blames it on the firewall? And how you get stuck with trying to figure out why it’s not working, since, you know, it might be the firewall? Dropping shit on the inside network is why that happens to you. Start blocking instead, and you can throw it back at the apps guys with a simple question…‘did you see an UNREACHABLE? No? Then it’s not the firewall!’
I hope you have found this article to be informative, useful, and perhaps even of help in convincing management of some things you’ve been saying all along. I learned all of the above the hard way. I’d like to hear from you if this saved you any pain or suffering, or even if you just finally found a kindred spirit. In parting, I’d like to send a shout out to all those security folks who, after reading this post, think I’m a complete nutter. This one’s for you.
direct link for rss and email readers http://www.youtube.com/watch?v=rpRiSb_Ir-s
What do you think? Am I a complete nutter (about firewalls; nothing else is relevant THIS time.) Sound off in the comments and let me know how you feel.
No related posts.
{ 2 comments… read them below or add one }
I don’t think you’re a complete nutter, I think you hit the Nine Inch Nail on the head. I think you just said something many in your field have been afraid to say. I do agree that ‘Paranoia’ needs to be put in check. Much has been said about the effectiveness of ‘Fire Walls’ but not much has been said about how easily big bad ‘Fire Walls’ can be ‘subverted’. When will more books on the subject be published? What is your definition for “Maligned Little Creatures” ? Everybody seems to be getting paid Big Bucks to help Information Security become more effective, but can’t the term ‘Iron Curtain’ be used to describe the term ‘Virtual Fence’ ? In other words, when does Information Security stop serving it’s basic function? I agree Security Clearances are ‘Paramount’. However what good are they if they can’t be ‘secured for the long term’ or put in ‘suspended animation’ without accidently trampling on the ability of the every day average ‘Curious American Joe’ to surf the net freely? Shouldn’t the Internet be FREE of ‘Leviathian-like trip wires’ ? It’s very apparent to me that’s how it was originally designed. Are we developing ‘Fire walls’ or ‘Mine Fields’ ? Shouldn’t we be able to tell the difference?
Thanks for that Adrian, nice to know there’s a kindred spirit or two out there!