43 things IT administrators should never do

Administrators have a tough job on their hands to manage, maintain and protect the network they are responsible for. Armed with the latest tools, they do an excellent job; however, at times, often due to pressure, they make mistakes – sometimes easily avoidable. In this post I am going to present the top 43 common mistakes administrators have made, as a reminder of what they shouldn’t do. All are based on firsthand experiences.

1. Leave the trailing dot off a zone file in DNS

The first and most significant mistake a new BIND administrator can do is forget to end each zone with the trailing dot, leaving their zone dangling in the breeze as clients try to append their own domain name, and fail.

2. Implement HOSTS files instead of fixing DNS

HOSTS files may be necessary for troubleshooting, but should never be used in production to get around a DNS issue. Six months from now, no one will remember that server with the HOSTS file, until they spend a few hours troubleshooting why it keeps trying to connect to an old ip.addr.

3. Implement recursive forwarding in DNS

Forwarding is for when a DNS server doesn’t have the answer to a client query, so it can ask another server who might. Set two servers to forward to one another, and you will quickly take down your network with the resulting UDP traffic that the looping queries generate.

4. Allow unrestricted zone transfers

No sense making a potential attacker’s job any easier. Only permit zone transfers to your DNS servers.

5. Leave out WINS

Eleven years after Windows 2000 came out, Microsoft networks still rely on NetBIOS for several functions. A well designed WINS solution will greatly improve performance, while the lack of one can cause all kinds of client issues.

6. Implement LMHOSTS files instead of fixing WINS

Much like using HOSTS files for DNS, a LMHOSTS file should be used for troubleshooting a specific client, not because your WINS infrastructure doesn’t work.

7. Implement a disjoint namespace

There are many things you can do, but should not. This is one of them. The inconsistencies that can occur when you use a disjoint namespace outweigh any political or legacy reason to do so.

8. Bypass the firewall

Firewalls are there for a reason – to prevent bad things from happening and to separate security zones. Bypassing a firewall makes a bad guy’s job that much easier, and can provide them an express lane straight into your network.

9. Bridge networks

Whether it bypasses the firewall like in point eight, or just starts spewing internal wire traffic over your wireless network, bridging is a good idea that always turns bad.

10. NAT internal traffic

If you think NATing internal traffic to an external address is easier than using a split DNS, you need to reconsider. Between the protocols that can break when NATed and the user issues that can arise when trying to troubleshoot, it is far better in the long run to implement a split DNS. It also makes your firewall configure that much easier to manage.

11. Apply a patch without testing

No vendor can fully test a patch in your environment. That’s your job. Applying an untested patch is marginally safer than not patching, but eventually it will break a critical application. Bite the bullet and build a test environment.

12. Make a change without testing and having a backout plan

Here’s a similar concept. Untested changes will eventually break something, and not having a backout plan in place means downtime.

13. Make several changes concurrently

The first thing you ask when troubleshooting is “what changed?”, because often the easiest fix is to change it back. When the answer is ten items long, it’s much harder to do this.

14. Bounce a box figuring no one will notice

Trust me, they will, and they will scream to high heaven that they were right in the middle of something when you rebooted the server. If you cannot wait for a maintenance window, you need to at least send out an email giving them a couple of minutes notice.

15. Use unsupported characters in any name

Here’s another case of “just because you can do something, doesn’t mean you want to”. Whether it is “$”, underscores, “”, or spaces, including anything other than letters and numbers, it will eventually break something – be it a script, or a new application.

16. Run services using their own user account

I once saw a case where an administrator installed a cluster service to run using his own account because it was the easiest way to get a new service running. 45 days later, when he went to change his password, the service died. He set his account to never expire, and six months later when he quit and his account was disabled, and the service died again. Give each service its own service account, and never use your own account for anything but your own login.

17. Enable anonymous FTP uploads

Unless you really do want to host illegal warez that will burn your bandwidth and use up all your disk space, never allow anonymous uploads on FTP servers.

18. Configure an open relay

Configuring an open relay is the easiest way to stop your users from sending email to anyone; which is also the fast path to having your mail servers put on every block list on the planet.

19. Leave default credentials intact

Default credentials are published, well known, and scanned for by free tools. One of the fastest ways to get hacked is to leave default credentials alone.

20. Use dictionary passwords

Here’s the second fastest way to get hacked; using dictionary words for passwords. It only takes the simplest of tools a few minutes to run through every word in the dictionary, making password cracking a trivial exercise.

21. Use non-expiring passwords

The main reason we expire passwords is so that, if they have been compromised, eventually that door is closed. Trust me, no matter how good you think your password is, it’s not that good. Change your password regularly the same way as you make your users do it.

22. Use shared/common credentials

Check the log to see who made that change. Who did? The administrator? Great, who knows the administrator password? Every administrative user should have their own set of credentials, and the administrator/root/supervisor account passwords should be locked up and only pulled out in case of emergency. Anything less breaks accountability and any hope for auditing who did what.

23. Run unverified downloads

Checksums are there so you can be sure what you think you downloaded is what you have. Always verify things you downloaded from the Internet before running them, especially when you are going to be running them using privileged accounts.

24. Use outbound permit ACLs instead of a proxy

Do you like herding cats? Me neither, and I also don’t like trying to tighten a screw with a hammer, or driving a nail with a wrench. Outbound ACLs should block things you don’t want all systems to do, like send SMTP or NetBIOS traffic to the Internet. If you want to control web access, use a proxy, which is purpose built for the task and can deal with FQDNs and URLs instead of ip.addrs.

25. Block PINGs

The PING of Death is over a decade old. Blocking ICMP echo and response does almost nothing for security, but breaks a tried and true method for testing connectivity and troubleshooting network issues. It also violates an RFC.

26. Deploy open Wi-Fi networks

Hiding your SSID does nothing to secure your network. Deploying an open Wi-Fi network is as dangerous as running an Ethernet drop into the alley out back of your shop, and is an open invitation for attackers to run amok on your network. At a minimum, implement WPA, and segment your wireless network from your wired.

27. Surf the Internet while logged on as an administrator

When you are logged on as an administrator, every program you run is a risk. Unless you have a sandboxed browser, a compromised website could lead to a compromised client, or worse, network. Surf the web using your regular account to reduce your risk from zero-day attacks.

28. Read email while logged on as an administrator

Much like surfing the web, running your mail client with your privileged account runs the risk of compromise from malware attachments, embedded scripts in email, etc. The best antivirus and antispam products in the world still rely on signatures, which can only be developed after a zero-day attack becomes a known attack.

29. Skip documentation

Show me an administrator who likes to document, and I will… well, I won’t have to do anything because no administrator on the planet likes to document, but it is a necessary part of the job. Even you won’t remember everything you did six months later, having documentation to refer to can make the difference between a simple task, and weeks of reverse engineering or reinventing the wheel.

30. Skip change logs

Much like documentation, change logs make it easy to answer that troubleshooting question “what changed?”. This is especially beneficial when that question is being asked by your assistant while you are on vacation. Unless you want to answer the phone while you’re on the beach, document changes.

31. Implement a new system without a scheduled maintenance window

Any new system you deploy, whether a simple file server or a complex application farm, needs to have a maintenance window established so you can do upgrades, patching, etc. Unless you like staying up until 02:00 on a Sunday morning, try to get that window approved for daylight hours.

32. Implement a new system without including redundancy

Having redundancy means never having to get that 02:00 call because a service went down. You may not be able to add redundancy to legacy systems, but anything new you deploy should include redundancy.

33. Run backups without verifying restores

“I don’t care what the backup logs say…” – until you take that tape, restore the data from it, and verify you can access the restored data, you don’t have a backup you can count on. Do you want to tell the CEO that you cannot restore his mailbox because of a bad tape?

34. Skip a patch

I have worked over one hundred security incidents; more than 90 of those have been hacks against known vulnerabilities for which a patch existed, but wasn’t applied. Patch regularly, patch often, and never skip a security patch.

35. Monitor too little

If you rely on users complaining about outages to let you know when a system has failed, you won’t last for long in this career field. Monitoring your critical systems is a vital part of administering a network.

36. Monitor too much

But monitoring too much leads to information overload, and pretty soon you are ignoring all the monitoring emails, which means you miss the important ones that warned you of an imminent failure. It’s going to take a lot of effort to get the right balance, and no two companies will be quite the same, but a good starting point is to get an email alert immediately only for those things that show an actual failure, or a condition that indicates an imminent failure. Anything else should be a daily summary.

37. Email when angry

Whether you are sending out an email bcc all, or replying to an upset user or clueless PM who has riled your feathers, emailing angry does no one any good and can damage your reputation. Take a deep breath, go grab a cup of coffee, or even put it off until the next day, but if you find yourself pounding on the keyboard while you are composing an email, don’t dare hit send.

38. Keep information a secret

If you are the only one who knows how something works, you are not creating your own job security; you are guaranteeing you will get called on your day off, while you are on vacation, and that you will never be able to pass it on to someone else. The best administrators are the ones who share information with others, and cross train them to reduce any human as a single point of failure.

39. Update information inconsistently

Any update is better than no update, but inconsistent information can be confusing, lead to mistakes, and generate even more questions that you will have to answer. Establish a format or template for any information, whether it is for your change log or for user accounts in Active Directory, and make sure all administrators follow it consistently.

40. Violate licensing agreements

Some risks are just too great to take, and knowingly violating licensing agreements not only exposes the company to legal action and financial penalties, it can quickly end your career.

41. Practice other than they preach

Users, junior administrators, and bosses alike, are not nearly as stupid as you may think. Telling them to do one thing, while you do something else, is a very easy way to lose their respect, as well as their trust. Follow the rules and lead by example.

42. Abuse their privileges

It doesn’t matter that you can access that file folder, should you? Administrators are in a position of very high trust, and violating that trust can quickly end your career.

43. Test in production

Even if the only testing you can do is in a VM running on your workstation, you need to test any changes before deploying them to production. Failing to do so is just asking for trouble, will kill your SLAs, and tarnish your reputation – it’s not worth it.

If you can avoid doing the 43 things in this list, you will save yourself, and others, time and money, avoid headaches, and minimize downtime. In other words, you will be doing a great job and taking your game to the next level. Almost anyone can be an administrator; but very few can be great administrators.

Learned any lessons the hard way yourself? Share them with us!

SHARETweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Pin on PinterestDigg thisShare on RedditShare on TumblrShare on StumbleUponEmail this to someone

Emmanuel Carabott CISSP heads security research at GFI Software. He has over 12 years’ experience in the security field and is a regular contributor to several websites and blogs. For more information about the benefits of using email usage reporting.