This interview is one of several that I’m working on that deal with the broad area of security operations. Jeff Lahann has extensive experience in building and leading high-performance security operations teams. We’re in an era in information security where the battlefield doesn’t favor the defender so it’s up to innovative leaders such as Jeff to create and lead information security teams that are up to the challenge. Jeff is one of the sharpest and most even tempered people that I’ve run across in security operations leadership and I’m grateful that he took the time to participate in this interview.
Professional Biography of Jeff Lahann
Jeff Lahann currently works at a multinational company building out the company’s first Security Operations department which has responsibilities for 24x7 security threat intelligence, threat monitoring & response, as well as, vulnerability scanning, vulnerability management, and data loss prevention missions. Before coming to his current company, he evolved and ran a SOC at a large multinational manufacturing company where he worked with several government agencies to combat threats targeting government contractors. He has been doing information security operations work for over 10 years starting out as a Security Operations Center analyst at IBM watching out over 220 fortune 500 companies. Jeff has been a digital investigator and incident responder for several Fortune 100 companies, an IT threat intelligence analyst for IBM’s Managed Security Services Delivery team, and a Senior Security Analyst in a large multinational’s SOC. Jeff was also an adjunct instructor at Colorado University - Boulder teaching IT Security classes to masters candidates. He was also a corporate instructor training investigators/incident responders and has developed several beginning and advanced level courses. Jeff has earned an MBA and a Masters of Science in Information Technology & Security. Prior to getting into securing bits & bytes, Jeff served in the United States Air Force as an Explosive Ordnance Disposal Journeyman.
AFoD: So how did you go from being an Air Force EOD person to a career in information security? That's not exactly a natural career progression.
JL: What do you mean? I hear that going from "Bomb Squad Tech" to "Computer Security Tech" is quite common, no? All kidding aside, you are correct, not the most natural of career progressions but one that my wife is far happier with than I. Don't get me wrong, I have thoroughly enjoyed working in the IT Security field doing Security Operations work, incident response, digital investigations, security analysis, and I have even enjoyed my time on the dark side in security management. However, not much else matches several thousand pounds of explosives going off or hovering over an interesting device wondering what you are dealing with.
AFoD: What was your first job in information security and how did it influence you to keep going down that career path?
JL: My first job in information security was riding a console as a Security Operations Center Level 1 Analyst in a SOC that was servicing a couple hundred Fortune 500 companies. Those were some long 12 hour shifts trying to stay vigilant watching hundreds of log events roll by the screen for each customer. Back in those days SIEM's were just coming on to the market so we had all sorts of paper based systems to keep track of suspicious events and correlations. Back then knowing how to make military style coffee, the kind that eats away the cup if you don't drink it fast enough, was just how you survived the shift. There were several things that really influenced me to keep going down this career path. First was that it lined up with my general life calling of doing security and protection work. Prior to IT Security, I was doing all types of security work in and around private sector, law enforcement, and military arenas. Second, one of the best things about IT Security is the requirement of continual learning you have to engage in just to keep up with the bad guys and the field. The final influential aspect was the people I worked with back in my starting days. We had a very good team and it is always good to work with and for great people. Even as we all spread out over the years, those folks are still a foundational part of my network.
AFoD: You eventually made the decision to move from being a skilled individual contributor to a leader. Why did you decide to go down the professional security leadership path?
JL: Well I was wondering when the question of why I would sell my soul and join the "dark side" was coming. It was actually a pivotal point that I still remember well, but for the record, there are days when I'm slogging through a couple hundred emails and working on a power point that I miss the days of spending my hours on the command line doing something a bit more "real". I had worked my way up the various skill paths and levels into a senior analyst and team lead role at a company when I had figured it was time for a change of scenery. So I up and moved companies, states even, to land in another senior analyst role. It was only about 4-5 month into that gig that I had hit the top of my game again and being the type of person that never likes his feet falling asleep, I was already looking at the proverbial, "what's next for me" question. Then one night it just kind of hit me; I figured it was game changer time when I could move around and hit my stride so early on. It wasn't anything as dramatic as the devil showing up and offering me to sign in blood, it was more timing than anything. The guy that had started the department I worked for was looking to move along and I had decided to try to look at positions up the ladder. Things feel into place and the next thing I knew I had been running the shop for a year taking myself and the department to the next level.
AFoD: As part of your professional development, you obtained an MBA from Arizona State University. How has that benefited your ability to build and lead your teams?
JL: First, here is the back-story on getting an MBA. I was a couple years into the management side of things and was still having issues with what I still see as a primary function of a security manager, no matter where you are on the ladder of security management, and that is the function of translator. What I mean by translator is the job of taking highly geeky security specialist jargon and problem sets and turning it into more general terminology that others outside of security can understand. I typically find that real world analogies work the best to describe to non-security types, complex attacks and impacts. I was tasked with protecting the business but was struggling to get my urgent message into terms that business folks understood and wanted to take action on. I figured that to go to battle over priorities and budgets with business types to get my projects funded, I needed to understand their world and speak their language. Thus came the idea to get what I termed, a "weaponized education" (hat tip to RM). As far as the other benefits of my MBA, it has helped not only with what I mentioned previously so that my teams see progress in the fight to protect the company, but also in helping translate backwards from “business priorities” and “bottom lines” to what that means to security projects and security specialist geek types. Overall, it has been a very eye opening experience, especially after getting through the universal term, "MBA Hell".
AFoD: Your career path has led you to a place where you are highly skilled in building and leading security operation centers. Can you explain what a security operations center actually does?
JL: Security Operations Centers or more commonly referred to as "SOCs" can be associated with several functions but the most common one tend to be some sort of security event monitoring. Much like Network Operation Centers (NOCs) do for the entire IT environment/infrastructure by way of looking for problems in network flow, system outages, and the like that potentially cause business impact, SOCs do this from a security perspective. Security monitoring can be done in many different ways but the best way is via pulling all relevant logs into a centralized logging solution and then feeding this aggregate information into a SEIM (Security Event and Information Management) system. These systems allow you to correlate and make sense of all the various forms and types of event logs of interest to bring about alerts of suspicious or malicious activities. Many SOCs will have expanded missions and taskings ranging from operational support of security type devices to incident response to threat intelligence and research. But to the core, SOCs, and those that work in them, are performing some level of analysis of data to bring about action to protect the enterprise.
AFoD: Successful teams are made up of important elements such as people, processes, and tools. I'd like to start by asking about the people aspect of how you go about building your teams. What sort of people do you like to hire and how do you organize them into an effective team?
JL: People are a good part of the triad to start with as they usually make or break the team; you can have bad people with good processes and tools and you'll still fail. Now if you have good people and bad processes and tools, you still have a chance at succeeding because of the ability for good people to adapt and work with what they have. I've always been lucky to surround myself with good people and we always manage to pull off some pretty amazing things. What types of people I like to hire depends largely on the environment the SOC is residing in and/or the charted missions. But in a general sense I am always doing the obvious thing of trying to find the most talent I can afford for any position I am hiring. In more specific terms, if I'm looking for entry-level analysts for the SOC (a great way to break into the security field) then I don't expect them to have any security experience, just good technical expertise (if I find someone that has been doing security learning above and beyond the day job it is just bonus). In this case I am looking for someone that has aptitude to learn quickly and has a passion for the security field. I've found that these two attributes for a level 1 analyst are what usually prove out to be the senior level analyst years later. You have to have a passion and desire to continually learn in the security field because the learning curve is so steep all the time its more like scaling a vertical wall of ice that has had oil poured over it. For level 2 analysts I am looking for someone that has been on the job doing security work for a company for at least 2-3 years, preferably in/around operations work; again I want to see the passion for the work and aptitude to learn. Senior level analysts and higher is just more time in service and experience in several domains. A final trait I look for across all the levels of analyst is that desire to keep pulling at loose threads, picking at the loose corner of a wrapped gift , the trait to keeping digging on something until an answer is found that makes sense. In relation to shift organization, I pair two level 1 analysts and a level 2 analyst per shift in a 24x7 SOC and then spread the Senior level talent across multiple shifts as best I can. From a manager perspective I try to keep the ideals of work hard/play hard, everyone has each others back, don't be afraid to say I don't know (something I picked up on the Bomb Squad for obvious reasons), and finally I try to enable the team to do the work and get out of their way. By the way, did I mention I'm currently hiring several analyst and management positions for day and night shifts?
AFoD: Gosh, Jeff, you hadn't mentioned that! What a great opportunity! How would someone go about finding more information about your option positions?
[AFoD: Okay, dear reader, you caught us. I had the idea for this interview when I saw that Jeff was looking to bring some new people onto his team. I was in the mood to interview an experienced security operations leader and offered up the interview to Jeff as a way to get the word out for his new positions. Jeff is a very sharp fellow and if you have an interest in getting involved in this particular portion of the incident response and digital forensics world, he's someone you should talk to about your interests.]
JL: Well since you asked (*wink*) people interested in joining a newly built and deployed security operations center can hit the job site: http://jobs-apollo.com/arizona-jobs.
AFoD: The second leg of the triad is processes. Can you talk about what role processes serve in your teams? For example, what tasks do you create processes around and what tasks are treated with more of an unscripted case-by-case approach?
JL: Ah yes, processes. These are very important but not so much the sexy side of things that people like to work with. Processes and procedures are important to a SOC for repeatability and standardization of those things that need repeatability and standardization. I make this circular distinction because I am a big proponent of standard operating procedures (SOPs) where they make sense. However, it is important to note that you don't want to become imprisoned by processes or worse yet, take away an analysts ability to apply common sense to the situation. There are many things in a SOC that you want described in a step-by-step procedure or overarching process. For example if the SOC is responsible for updating IDS signatures on the sensors, then you want a clear process stating when and why you do this and a step-by-step procedure as to how this is done so you end up with the same result no matter which analyst completes the task. SOC controlled or initiated change management, inbound security report handling, shift turn over, and system rebuilds are more examples of tasks that can be controlled by process and procedure. On the contrary are those situations where process and/or procedures can hinder the analyst or strip away the analysts' intellectual contribution. Incident response, compromise response, threat intelligence research, and malware analysis are all examples of situations or tasks that don't follow a script and trying to overlay anything more than loose guidelines can hamper the right efforts.
AFoD: I agree completely, of course. There's a time and a place for scripting repeatable processes in the name of standardization. However, analytical work can be just as much of an art as a science and it doesn't lend itself well to scripting. That's one of the fundamentals that I learned as a police officer. Every department has a SOP, but there is only so much standardization you can get away with when you are doing police work. Good luck creating a repeatable process for handling a shots fired call.
What use do you make of concepts such as Six Sigma and IT Infrastructure Library (ITIL)?
JL: Six Sigma is great for manufacturing processes to which it was designed but I find it hard to completely extrapolate it into the IT world. Though I have found use for the various before-ToBe process flows, TMAPs, and some of the statistical analysis where it fits. As far as ITIL, this is a bit more geared to IT and my current organization is big on its structures. It helps to get things into standardized methodologies and terminology but you have to be careful not to be too rigid or be a slave to the framework or you end up with too much overhead.
AFoD: How do you structure your security operations centers? We've talked about disciplines such as incident response, malware analysis, and threat intelligence. How are you putting these pieces together to create a high-performance team?
JL: I typically approach building out a full SOC on what I call the "virtuous circle"; depicted as a self-feeding circle type diagram. The three main ideas or missions I build a SOC on are Threat Identification, Threat Detection, and Threat Response. You have to have threat intelligence research and malware analysis functions being done to provide data on the current threat landscape and how it applies to your current company setting. This data should then feed into the detection and monitor mission by way of custom IDS rules, firewall rule sets, web filtering blacklists, etc.; this is commonly seen as the foundational SOC mission. But once you have detected the threats you have to respond to them in a standardized method with specific tool sets. I tend to favor the SANS incident handling methodology and tailor it to my current company setting as needed. At the end of your response tasks, you typically end up with additional intel in which to feed back into your Threat Intelligence mission and thus we end up with the self feeding circle. Of course you have to have reporting with all of these missions, something that all my former students at CU Boulder used to groan about as well as my current staff. I still preach that if you have all kinds of good data points about the strengths and weaknesses of your security, it is worthless if you don't leverage it by reporting it to various stakeholders to get action. The foremost function a SOC should be aspiring to is to use the front line trenches information it gathers in the execution of the virtuous circle to continuously improve the security posture of the enterprise it serves.
AFoD: That brings us into the last part of the people, process, and tool triad. What tools are your teams using to accomplish their goals?
JL: Tools and technologies tend to evolve with the general IT landscape and business needs as do security tools and technologies but they also have to try and keep up with that ever evolving threat landscape. I look to base my tool purchases on a solutions approach, not a product approach. Too many times over the years vendors have come at my teams trying to sell product versus trying to help us solve a problem and sharing honest and open dialogue with what they can do to help. You help me solve a problem and support my team well and you start to earn some loyalty. Philosophies aside, some of my favorite tool sets for a SOC executing the missions we've discussed previously are Sourcefire IDS, ArcSight SEIM, Encase, Mandiant Intelligent Response (MIR), Splunk, and good old fashion Linux. Having worked in more budget downturns than "golden years" I've had to get creative trying to solve problems and I am always impressed with what you can do with a skilled security analyst who is passionate about protecting the company and a simple Linux system.
No comments:
Post a Comment