One of the things that I have enjoyed immensely about my information security career is that I have had the opportunity to meet and work with some amazing people. Rob Lee is one of those people. I have learned a tremendous amount from him over the relatively short time that I have known him. He’s someone that I have come to trust as a friend and as a professional peer. Anyone who has the good fortune to know Rob knows why I frequently use the hash tag #giantpersistentfriend when referring to him. I count getting to know Rob as one of the highlights of my personal and professional life so far.
Professional Biography of Rob Lee
Rob is an entrepreneur in the Washington D.C. area specializing in Information Security, Incident Response, and Digital Forensics. Rob is also the curriculum lead for digital forensic training at the SANS Institute. Rob has more than 15 years' experience in computer forensics, vulnerability and exploit discovery, intrusion detection/prevention, and incident response.
Rob graduated from the U.S. Air Force Academy and served in the U.S. Air Force as a founding member of the 609th Information Warfare Squadron, the first U.S. military operational unit focused on information operations. Later, he was a member of the Air Force Office of Special Investigations where he conducted computer crime investigations, incident response, and computer forensics. Prior to starting his own firm, he directly worked with a variety of government agencies in the law enforcement, U.S. Department of Defense, and intelligence communities as the technical lead for a vulnerability discovery and exploit development team, lead for a cyber forensics branch, and lead for a computer forensic and security software development team. Rob Lee also was a Director for MANDIANT for four years prior to starting his own business.
Rob co-authored the bestselling book Know Your Enemy, 2nd Edition. Rob earned his MBA from Georgetown University in Washington DC. He was awarded the Digital Forensic Examiner of the Year from the Forensic 4Cast Awards. Rob is also an ardent blogger about computer forensics and incident response topics at the SANS Computer Forensic Blog. Rob is also co-author of the MANDIANT threat intelligence report M-Trends: The Advanced Persistent Threat.
AFoD Blog: Why did you choose to attend the United States Air Force Academy?
Rob Lee: I grew up in an U.S. Air Force family. My father, Col (Ret) Robert E. Lee USAF, and my grandfather, BGen Travis M. Hetherington USAF, both served full careers in the Air Force. My grandfather was a West Point grad ('33) both are originally from Texas as well. In fact my Grandfather was deputy director of the NSA when it was formed. I was really inspired by both of them. The U.S. Air Force is a part of my blood and it has never left it. I attended USAFA as I felt it would give me the best chance to work in the space operations career field. I loved anything to do with rockets, space shuttles, and the possibilities in space. I was one of those "Space Camp" nerds. Unfortunately, my personality was not geared for that entire "listening to orders" thing. I liked to do things "my own way" and USAFA was the antithesis to that mentality. I had former teachers and even my own parents sit me down and ask me if I truly understood what I was getting myself into. In the end, I went because I felt that if I didn't try I would always end up regretting it. I wanted the challenge.
AFoD: You earned your degree in space operations while you were at the academy. Can you tell us what sort of courses you took as part of that program?
LEE: Every USAFA grad graduates with a Bachelor of Science degree regardless of major, so we have a very intensive core curriculum. Most cadets end up taking between 18 and 21 credit hours per semester with most in engineering degrees have to take 21 credit hours. Those in the Astronautical Engineering majors usually had to take at least 2-3 semesters taking 24 credit hours back in the mid-90s. For the degree program, there was a mix of computer programing with astronautical engineering courses. I also took the first information operations (IO) course offered at USAFA which really had me redirect my thinking for career options when I graduated. My senior year, I took an extra course, for "fun", from the USAFA Computer Science dept. in Computer Security. While I had a large affinity toward computer science and I was good at the programming, most of the career advisers told cadets to choose a major with an operations focus and many felt those with computer science degrees wouldn't probably make rank as easily as your peers. I don't regret my choice as I really was able to get the best of both degrees in my studies.
AFoD: One of the advantages of attending a United States military service academy is that in addition to getting a great education, cadets have a rich academy experience outside of the classroom. These service academies are some of the finest leadership training institutions in the world. Can you tell us what sort of extracurricular activities you participated in when you weren't in class and how your overall academy experience prepared you for your future career?
LEE: My 1st year, I was on the USAFA Ski Team. I was on racing ski teams in high school on Team Breckenridge and Copper Mountain teams in Colorado. However, trying to manage racing with school ended up being too much and I had to pull back. Every cadet had to go through Survival Training call (SERE) training as well. I also participated in a program my 3rd summer where I led newly enlisted airmen through basic training at Lackland AFB. Leadership and management are very different things and the academy provided much of the environment to practice that. One thing that clearly helped me out in my career is my participation in the cadet acting group called Bluebards. Being able to stand up in front of your peers and communicate effectively is a skill I feel everyone should master if they plan to lead and not just manage.
AFoD: So how did you go from a snow skiing Air Force space engineer to a career in information warfare and digital forensics?
LEE: Hahaha... So... as I was entering the first semester of my senior year we had to select our chosen Air Force careers where you chose your warrior class. I had chosen space operations. However, as I found out MUCH too late, Space Operations had a color vision requirement. ACK! Ironically, it was the same reason that eliminated me from becoming a pilot too. I was torn apart internally as I had studied so hard to be in Space Command. I wanted to be a Star Trek nerd. It was around this time that I interested in Information Warfare and was taking a course in it. I decided in that course, that I was going to make a career of this.
During an Information Warfare (IW) conference held at the academy in Nov 1995, I ended up meeting two members from the 9th Air Force 609th Information Warfare Squadron. Little did I know that, at that point, there were the only two members that existed. Lt. Col. "Dusty" Rhoads and Maj. Andrew Weaver. I expressed my interest in being assigned to their unit and wrote a letter to them expressing my desire. They told me to apply to be a Communication Officer (33C) career field. I did.
You have to understand though, back in 1995, not many had even heard of IW or IO at this point. What I didn't know was that Lt. Col. Rhoads had "by-name selection authority" given to him by the Chief of Staff of the U.S. Air Force. Early in the spring I was called into the assignments office at USAFA. Apparently cadets are not supposed to go find their jobs and I was given a "talking to." Apparently, orders came down from the Pentagon to assign myself and another cadet to the 609th IWS. I didn't realize how irregular this was until much later, but I found out that it was one of the first assignments generated for my graduating class, but I couldn't tell anyone about it till assignments day. They had called me to the office to find out how I "gamed" the system.
I knew that being selected to go to the 609th would be a career changer for me and spent the rest of my senior year studying every programming book, every security book, and reading online as much as I could. I also gave up part of my summer after graduation to intern at DISA in Washington D.C. to merely absorb as much as I could. I came to understand that no one really knew that much about internet warfare or defense and that we ended up taking a really good stab at figuring it out. The 609th was a great experiment that ended up getting tabled due to politics. But I do consider the two years I spent at the 609th IWS as my graduate degree of sorts. I read more books, learned more on the job, and the unit engineered more solutions without manuals. I’m not sure I could have received a better education in information security through any university. We just didn't receive a paper to hang on the wall.
AFoD: I had a similar experience. I wanted to be a United States Navy Surface Warfare Officer when I was growing up and it turns out the Navy frowns on officers who were legally blind without their glasses commanding warships. Thus, they invited me not to join them and that was how I ended up in law enforcement. What caused you to leave the Air Force and what did you do after you left?
LEE: Leaving the Air Force was not an easy decision. Both my father and grandfather were career officers. However, the services simply did not have a career path for officers that both wanted to stay technical and lead troops although they had many examples of that in pilots, space operators, doctors, etc. In a nutshell, the personnel center told me that if I intended to be promoted I would have to expand my horizons out of the Information Operations (IO) side of things. I wrote to many Generals and Colonels that I knew and each told me that it was too soon for a specific IO track in the AF. However, they also told me that there are other ways to serve my country. I separated with the intent of going to work for one of the intelligence agencies and ended up working the next 7 years between the CIA and the NSA.
This is also around the time I started teaching at the SANS Institute. I attended my first SANS event in Orlando in March of 2000. It was at this time that they were introducing the intrusion analysis (GCIA) exam. I was decent at examining packets by hand as a result of my time at the 609th IWS so I wanted to simply challenge the exam without taking the course. I wrote to Stephen Northcutt and asked him for permission. He hesitated initially, but approved. When he approved the waiver via email he mentioned that it was a really difficult test and not many who took the class passed it. He kind of implied that I would probably fail since I didn’t take the training first. Apparently, almost everyone was failing their initial exam because of the difficulty by the people that took the courses first. I took the test and scored just somewhere 96%. I think Stephen must have been receiving scores in his email as I received a call no more than 10 minutes after I finished the test. He said, "Who are you?" And I explained my background working at the 609th IWS and AFOSI. He wanted to meet at the SANS conference in DC that summer and asked me to give a 2 hour presentation on IR and Forensics at "Capitol SANS 2000". I have been teaching ever since.
In my other position, after I separated from the Air Force, I worked in a very specialized group in a group at a government contractor ManTech called Computer Forensics Intrusion Analysis (CFIA) Division. I was convinced by Travis Reese that joining a government contractor allowed me to work with more projects than becoming a .gov civilian. I asked to be assigned to the IOD (Intrusion Operations Division). I was a researcher where I accomplished vulnerability enumeration and discovery. In a nutshell, we tried to break things. Although rare and time consuming, we were usually successful at it. I also led a team of developers working for a variety of projects in the IO world of the intelligence community. The most wonderful thing about CFIA was the people and my co-workers. CFIA was a pool that ended up having under its roof some of the most talented individuals you have probably never heard of. All I can say was that we did more during my years at CFIA to help this country than I did in the 5 years of service while in uniform. Many from CFIA eventually left to join other companies such as Kyrus-Tech. I ended up leaving in 2007 to go to business school at Georgetown University.
AFoD: So what would a highly accomplished information security leader such as you need with an MBA?
LEE: I really wanted a master’s degree and I was torn between technical and getting a business degree. Why a business degree? 2 reasons really:
1. Every organization that is compromised is run by business leaders to an extent. Being able to understand their concerns from the business angle has made it incredibly easier to sit in board rooms and not only tell them I’m a geek, but a geek that understands them. They have tended to respect that I have gone out of my way to speak their language and understand the business impacts to a compromise.
2. I wanted to do something that I wasn't good at. Heading the business direction allowed me to explore areas that I wasn't familiar with, particularly business finance. It was important for me to grasp these concepts as I am purely fascinated by how organizations truly operate. I was growing tired of not being able to truly look at 10K and not know what is good or bad in a business. To that end, I hope to start my own organization/business soon and I simply wanted to learn more in an area I never really focused on before.
AFoD Blog: I couldn't agree more. One of the key deficiencies that I see in the larger information security community is a failure to understand how the business world works. Some of the best training I've obtained in my own career has been through an executive education program that I went through at Dartmouth's Tuck School of Business. There has been a considerable amount of effort put forth by business schools to develop future generations of business leaders that the information security community should be taking part of and embracing. There is much more to creating and leading effective security organizations than just technical knowledge.
So let's shift gears a bit here and talk about one of the hot topics in the information security world which is advanced persistent threat (APT). What is your definition of APT?
LEE: The APT is a cyber-adversary displaying advanced logistical and operational capability for long-term intrusion campaigns. Its goal is to maintain access to victim networks and exfiltrate intellectual property data as well as information that is economically and politically advantageous.
The APT is not a bot-net. It is not a car. It is the DNA of an adversarial group.
AFoD: Do you limit your definition of APT to nation-states acting as the cyber adversary or do you allow, for example, organized crime groups acting independently of a nation-state to also be defined as APT?
LEE: Any group able to display the logistical and operational control for a long term intrusion would fit. Scale wise, nation states tend to have enough of the people employed to pull this off. Organized crime -- more people needed to pull of long term attacks = more mouths to feed. It also doesn't align to their goals. It is much easier to offline a CC database than to remove IP from a network. APT to an extent has not been as interested in financial and card data theft operations. Organized crime tends to not be interested in information that the APT tends to focus on. Therefore, it could be anyone, but it tends to be nation state actors over organized crime as a result.
AFoD: So what is your advice to an organization that finds itself targeted by an APT adversary?
LEE: Don't over react. Organizations should gather as much information about the situation they are in before they take any action. Early execution could simply make the problem worse. This basically means that you might need to "let the adversary" have their way with your network while you assess exactly where they are prior to strategize a plan for their removal. Long gone are the days of "Pull the plug and reinstall from backup." Most organizations start blind, but progress to having very good intelligence on exactly where in their enterprise they will find the APT as a result.
Once an organization has decent intelligence on how to identify compromised systems by the APT, they should begin planning the remediation actions. The remediation actions should not be gradual, but a sudden and deliberate plan of action designed to cut off communication between the APT malware and their operators. In addition, remediation should focus on the removal of the malware and the addition of new security measures designed to prevent additional beach head systems and degrade the ease lateral movement by the APT. This is sometimes difficult if remediation occurs too early. If a single piece of malware survives, that footprint could be used to gain control back over the network and malware that is more difficult to find and identify is usually deploy by the adversary.
Too often, an organization responds blindly and ends up making the problem much worse than it needed to be. To the adversary, they cannot tell if you have your act together or if you are fire into the darkness. They respond as if they are about to be removed and as a result dig deeper into your infrastructure.
AFoD Blog: People like you and I are spending a considerable amount of time getting the message out to senior executives and others that controls aren't enough and that they need to get much better at detecting, responding, and remediating. All that said, controls are still very important. What do you tell people who ask what sort of controls are the most effective when it comes to combating advanced actors like nation-states and organized crime groups?
LEE: That is such a great question. I get asked this all the time in the form of "What is the simplest and easiest thing I can do to improve my security to defend against the APT?"
The SIMPLEST answer? I usually answer, plan migrate to Windows 7 and Server 2008 as quickly as possible. Most enterprise organizations are still running a Win2K or a WinXP workstation with a Sever 2000/2003 base. These technologies are over 10 years old now and were created prior to Microsoft's secure computing initiative. While not full proof, the ability to freely move in a Win7/Server 2008 vs. a WinXP/Server2K environment is night and day. The amount of capability and additional protection pre-baked into the latest releases of those operating environments will slow the advance of adversaries. I won't get into each specific as there are many, but the simplest (not the easiest) answer is to "upgrade your enterprise."
As a part of this, once you upgrade to a somewhat homogenous IT environment, it is easier to establish decent application and system controls since there is a reduced common baseline. Instead of managing upwards of 30-40 different system configurations, an organization could reduce their common desktop environment to less than 10 variations. With the reduction, application and host based controls can easier track and monitor application white and black lists easier. In addition to this, the host/server auditing found in Win7/Server 2008 can be configured easier and at a depth that is effective. WinXP didn't allow for the specific tuning that a Win7/Server2008 environment might hold.
So the benefits to upgrading are not only a better security baseline, but trickle down security benefits from the upgrade as well. There is no silver bullet in APT defense. But you can make it more difficult for them to create a beach-head and laterally move.
The EASIEST answer? Hire the smartest and most experienced people you can get your hands on. Recruit and court a good leader and give that leader the authority and funds to build the best team possible. Give them a blank check to do so. True APT defense is not a technological answer, it is a technology and a people one. Typically organizations that have the best people will end up creating their own solutions on the fly to combat the APT. Give them leverage, delegate responsibility, and remove the internal roadblocks that will impede their success. The key is the team lead. Take your time and consider executive compensation for this individual as they are extremely difficult to find if they are good. I could get into a series on how to create a structured team that promotes creativity, champions hard work, and empowers the IT security operators to get the job done. The biggest gripe from most IT security professionals? Very few listen to them. Block some time in your schedule. Sit down. Listen.
AFoD Blog: So what can we expect to see from Rob Lee during the next year or so?
LEE: Over the past 3 years, we have been updating the forensic and incident response courses at SANS to include the latest tactics at finding and defeating the APT. The course where I have focused the majority of my efforts to train forensicators to deal with the threat has been FOR508: Advanced Computer Forensic Analysis and Incident Response. One of the biggest things I am working on currently is an update to FOR508 - Advanced Forensics and Incident Response. I cannot give away too much about what is coming, but it will be the go-to course for investigating APT and advanced incidents. This update is big and has been introduced in stages starting last year. For example we added a section on enterprise and remote forensics already. In fact, we hand out F-Response to each student who attends class.
On a professional front, I’m starting my own company early next year. I kind of have a unique idea that I’m working off of and am excited to see it come to light.
Next summer we will be doing the 2012 SANS DFIR Summit in Austin TX again. I am very excited about it. Leading up to it, we are accomplishing the first SANS DFIR lighting talks on Dec 13th 2011 in DC. The talks are completely open to the public but you must register. The Washington D.C. area is known for its density of talented professionals in the field of Digital Forensics and Incident Response, and SANS is bringing 10 top industry experts to you for the first SANS360: DFIR Lightning Talk. In one hour these Digital Forensics and Incident Response experts will discuss the coolest techniques and solutions they have discovered in 2011.If you have never been to a lightning talk it is an eye opening experience. Each speaker has 360 seconds (6 minutes) to deliver their message. This format allows SANS to present 10 experts within one hour, instead of the standard one presenter per hour. The compressed format gives you a clear and condensed message eliminating the fluff. If the topic isn't engaging, a new topic is just 6 minutes away.