Michael Johnson

Note: I’ve been asked to post this from a reader that asked to remain anonymous but would like to inject some thoughts into our dialog here. This is the second in this series (the first was here). If you have thoughts you would like to insert into the discussion feel free to contact us. -mj

When our network was evaluated by an external expert, the results were both encouraging and humbling.  The encouraging part was that at 45% of the Critical Security Controls implemented, the consultants told us we were actually in better shape than most organizations.  The humbling part was just how many very basic things we had missed.  Without going into specifics,

• We needed a LOT more logging.  A SEIM, DHCP logging, DNS logging, change logging.  You name it. If a very secure organization logs it, we probably didn’t.
• There was a decided lack of formal policies and procedures.  While there was lots that we were doing right, our processes typically grew out of round table discussions, and were never documented.  This lead to a lot of mistakes and shortcuts.
• We weren’t doing any vulnerability scanning, which meant that we didn’t know how much risk we had from poorly patched machines.  And patches were being applied ad-hoc, so some servers were way behind
• We had no security training program for staff
• 15 years of work with no defined policy and procedure had left us with a substantial mess to clean up.

With the report in front of me, and needing to show my manager that I was making progress in my new role, I immediately went to my techie roots and started cleaning up the mess in our Active Directory system.  I wrote reports to show us accounts that didn’t have expiry dates, that hadn’t been logged in for a very long time, or that had non-expiring passwords.  I disabled thousands of un-used accounts, added the expiry dates to the remaining accounts, wrote a report to show us which accounts were about to expire so they could be reviewed.  I connected employee accounts in Active Directory to the employee’s record in our HR system, and wrote scripts to automatically disable an employee’s account when they were terminated.  I did a lot of very good work cleaning up a long time of neglect.  It was, of course, entirely the wrong thing to do.

The work was good and necessary, but out of sequence.  Since we still had no formal policy and procedure, as I was cleaning up the mistakes of the past, the staff who were responsible for creating and maintaining our Active Directory were happily going about their work as they always had.  Creating fresh messes behind me as I cleaned.  Embarrassingly, I can’t really blame this on my status as a security rookie, but it’s probably obvious that I’ve never held an Administration role either right about now.

That’s the first lesson I learned in a security role, and one that hasn’t been taught in any of the courses I’ve been taking.  As a technical guy, I’m drawn to the technical solution.  And security tools are really cool and exciting to implement.  But the first tool you need is well defined procedures, approved from high enough up the corporate chain to make them stick.  Document how things should, and will be done.  Put the processes in place to monitor that they are being done properly.  Ensure that your staff aren’t still making the mistakes that you need to clean up.  Then start your cleanup.  Or as a colleague of mine wisely put it, turn off the taps before you mop up the water.