An Interview With Hal Pomeranz

Welcome to the first AFoD blog post for 2011. I’ve decided to start the new year out with an interview with Hal Pomeranz. Hal is very well known in the information security community and is probably best known for this work with the SANS Institute as the driving force behind the Linux/Unix security content for SANS. Hal is also part of the SANS digital forensics team and has recently been spending quite a bit of time writing, researching, and teaching on various digital forensic topics.

Professional Biography of Hal Pomeranz

A dynamic and experienced technology authority, Hal Pomeranz is the Founder and Technical Lead of Deer Run Associates, a consulting company focusing on Computer Forensic Investigations and Information Security.  He has spent more than twenty years providing pragmatic Information Technology and Security solutions for some of the world's largest commercial, government, and academic institutions.

Hal is a Faculty Fellow of the SANS Institute, and it's longest-tenured instructor.  He is the track author and primary instructor for Sec506: the Linux/Unix Security certification track (GCUX).  He is also a GIAC Certified Forensic Analyst (GCFA) and an instructor in the SANS Computer Forensics curriculum.  Hal frequently contributes to the SANS Computer Forensics blog, and is a co-author with fellow SANS instructor Ed Skoudis and Tim Medin  of the weekly on-line Command Line Kung Fu column.

A leader in the community, Hal has served on the Board of Directors for USENIX, BayLISA, and BackBayLISA.  He is a co-founder of the IT Professionals Forum.  He is a frequent presenter at national and local technical gatherings, and the author of numerous books and articles on subjects ranging from Computer Forensics to Information Security to System and Network Management to Perl Programming.  Hal also served as the Technical Editor for Sys Admin Magazine during its last four years of publication.  He is a recipient of the SAGE Outstanding Achievement award for his teaching and leadership in the field of System Administration.

Prior to founding his consulting company, Hal's career included  a variety of roles from System and Network Operations, to Network and Security Architecture, and even Software Development.  He has worked for an equally diverse set of employers including AT&T Bell Laboratories, NASA, the University of Pennsylvania, TRW Financial Systems, and even an Internet start-up, NetMarket, which was the first company to conduct a secure financial transaction on the World-Wide Web.  As a consultant for Deer Run Associates, Hal's client list includes Cisco, Microsoft, eBay, the FBI and several other government agencies.

AFoD Interview with Hal Pomeranz

AFoD: Many of the top level people in our community come from a law enforcement or military background.  The paths that these people take into digital forensics are generally defined by traditional law enforcement and military career development processes.  You are part of the top level of people who have entered digital forensics from the information technology world.  Can you describe your journey into the digital forensics community?

HP: I started my career as a Unix Sys Admin.  My first boss, an early mentor of mine, saw my interest in Computer Security and nurtured that.  So I ended up as a Unix Admin who was also an InfoSec person.

As an InfoSec professional, you're bound to have at least some
interaction with computer crime events, even if it's just in the course of defending your own networks.  As it turns out, my second job was at the University of Pennsylvania, whose dial-up services were at the time being used by non-University folks for a lot of nefarious activity. This led to some interaction with law enforcement, including the local FBI office.

There were a couple of other watershed moments in my career that started me down the digital forensics path.  The first was incorporating forensic material into my SANS Linux/Unix security track when it was conceived in the early 2000's.  John Green-- who did incident response at NSWC Dahlgren and is now the CISO for the state of Virginia-- wrote and taught the original material in the track, and patiently answered lots of newbie questions from me.  When John went off to pursue other opportunities, I took over teaching and updating the material.  This forced me to learn a lot.  I'm also now one of the instructors in the SANS Computer Forensics curriculum, which has also been a great opportunity to help others, but also expand my own
knowledge.

But I still wasn't doing digital forensics as part of my professional
consulting practice, though I did occasionally help out friends who were in difficulty and I certainly worked on engagements where I was part of the IT/InfoSec cleanup crew in the wake of an incident.  A couple of years ago, however, Rob Lee approached me about an opportunity with Mandiant.  Essentially they were looking to develop relationships with other skilled consultants who they could reach out to on an as-needed basis for short-term help during unexpectedly busy periods.  "Surge Staff" is the name of the program.

As it turns out though, "unexpectedly busy" seems to be more the rule than the exception.  I've spent a lot of time in the last year and a half working on cases through Mandiant.  It's been a very positive relationship for both of us, and I've certainly seen my skills expand enormously with the benfit of this experience.  Long live the Surge!

AFoD: Becoming part of the SANS digital forensic team is a relatively recent development for you.  Doesn't your involvement with SANS predate your involvement with the digital forensics community?  I think I remember you saying that you were one of the original SANS Fellows.  Can you tell us about your SANS career and how that has impacted your professional development?

HP: Of currently active SANS faculty, I think I can lay claim to being the longest-tenured.  Randy Marchany, Gene Schultz, and I have a good-natured argument going about who's the "oldest" SANS faculty member, but we've all been presenting at SANS conferences since the beginning of the 1990's (before the organization was called SANS, by the way).  I taught my first paid tutorial for SANS in 1994-- a class called "Securing Solaris: Step-by-Step". Over time, that course gradually morphed in what is now SANS' "Sec506: Linux/Unix Security" track, which is the basis for the GIAC GCUX cert. We rolled the first version of the track out in 2001 I believe.  John Green and Lee Brotzman originally contributed material to the track, and much of John's forensics material is still visible.  But I've been pretty much the only instructor and author for several years now.

As far as professional development goes, mastering material well enough to teach it to others seems to me to be the highest order of technical expertise you can obtain.  I learn a huge amount updating and teaching my courses.  And not just from my own research: SANS attracts some incredibly bright experts as students, and I learn something from the folks I teach every time I do a class. Teaching a class in front of a high level technical audience and being open to admitting that there's something you don't know may be one of the most difficult tightrope acts I can imagine.  "How are you going to learning anything if you know everything already?", is a motto that appears in my SANS "author's statement" and I think it's true.  We are all experts and yet also students at different times and in different subject matter areas.  There is so much we can learn from each other!

In the IT community that I grew up in, there was a lot of knowledge transfer going on, both formally and informally.  I was lucky enough to be mentored by some truly famous people.  Their expectation was that when I achieved a decent level of proficiency, I would educate the next generation-- "pay it forward".  I've tried to live up to that. Aside from stimulating my own learning, the other obvious benefit to being out there in the community and teaching/writing, is the personal "networking" and reputation enhancement aspects.  For example, my current business association with Mandiant wouldn't have happened if I didn't know Rob Lee because of our mutual involvement in SANS. And certainly I've had former students who have become clients.

AFoD: Ovie Carroll is good at articulating the point about learning from others also. I agree with him when he says he can sit down with anyone learn a couple things from them.  There is just so much to know in our field that it's impossible to really master all of it.  That's why it's important for people to share even if they don't think they are at the level of someone like you or Rob. Now the fact of the matter is that you are in the top tier of digital forensics people.  What sort of skills did you develop over the years that helped you reach this level in the field?

HP: On the technical side of things, there's a whole body of low-level knowledge about file systems, networking, system devices, etc that System Admins in my generation were expected to master.  For example, one of the first posts I wrote for the SANS Forensics Blog was a trick that uses alternate superblocks to allow you to mount "dirty" EXT3 file systems.  How did I know about that trick?  Because twenty years ago recovering broken file systems from alternate superblocks wasn't just a cool trick-- it was a necessary survival strategy!  In any event, it turns out that a lot of this technical knowledge is directly applicable to understanding forensic artifacts.

The other technical item that Sys Admins were expected to master was the ability to write code to automate repetitive tasks.  I'm dealing with a case right now where I need to process three or four dozen system images.  If I tried to do this serially using some of the standard commercial tools, I'd never get it done.  But because I can quickly write small shells scripts, Perl programs, etc I can be doing lots of automatic processing in the background while I'm working on specific artifacts that pop out as "interesting".  Also, I can read other people's code, find bugs and vulnerabilities, understand what malware is doing, etc.  The ability to program provides so much leverage for forensic analysts!  It's a shame that more time isn't devoted to teaching these skills in typical forensic curricula.

Looking at non-technical skills, there's a problem-solving strategy that successful IT people tend to develop that's very similar to the process you go through when responding to an incident or working a case.  You end up asking the same sorts of questions: "What was the tip-off that something has gone wrong?", "What were you expecting to see instead?", "When was the last time the correct behavior occurred?", and so on.  Good IT shops spend a lot of time doing "root cause analysis"-- figuring out exactly what went wrong so that it never happens again.  But as forensic analysts we also spend a lot of time figuring out "what went wrong", and the skills sets overlap.

As an IT/InfoSec consultant, I'm also used to coming into lots of
different environments and quickly assessing system and network
architecture, getting an handle on the site's policies and procedures, and so on.  Sometimes I even have to try and deduce how things were supposed to be configured, and then try and figure out where bit rot and entropy have set in-- I call this "IT archeology".  You do the same sorts of things when you're pulling apart a forensic image and trying to figure out what the machine was supposed to be doing and what's been happening on the machine.

From a psychological perspective, succeeding in IT over the long term generally means you've developed a certain level of confidence in working with strange problems without much external support or documentation.  Computer systems are deterministic-- if you fully understand the technical underpinnings, you should be able to understand past failures and/or predict future behavior.  Certainly you develop strong research skills and good testing methodologies, but confidence is key.  In the current forensics space, where so much is undocumented and you're often left to your own devices to figure things out, this confidence that you will be able to find a solution is supremely important.

AFoD: There are plenty of people who are in information technology who are good problem solvers and can do some scripting of repetitive tasks, but we both know that doesn't automatically make them good candidates for a role in digital forensics.  What do you think are the fundamental building blocks that someone needs to have to be turned into a good digital forensic examiner?

HP: In a nutshell, you want to look for the ones who do the root cause analysis and don't just reboot the machine when something goes wrong. They're the ones who want to figure out *why* a problem happened, not just make it go away.  They're going to learn more and learn faster than their peers, and they've got the kind of staying power that comes in handy during investigations.

It also helps if they can write.  Do they produce coherent documentation?  Are they writing articles or putting their ideas out in the community via other means?  Can they convey information to non-technical people?  Communication is key, because what we do is a "team sport" that involves lots of people, both technical and non-technical.

And to echo a theme from earlier interviews, I think you're looking for "passion".  Is working with computers a job or a calling for your candidate?  Do they spend significant amounts of time (especially their own time) on continuing education?  I've often said that if I won the lottery tomorrow and didn't have to work for a living, I'd still keep on doing what I do just because I have so much fun with it. There are plenty of frustrations around what we do, so I think you've got to love it in order to get over the rough spots.

AFoD: So lets say that we have someone who has all of the necessary fundamental skills and passion to get into digital forensics work. Let's talk about how you think they should go about it.  What would you tell the high school student who wants a career in digital forensics? 

What would you tell someone who already has an established career in information technology, but who wants to break into the field of digital forensics?

HP: One piece of the puzzle is having the necessary skills.  The question pre-supposes that our candidate has the necessary technical chops to handle the position.  But there's a difference between having a large body of technical knowledge under your belt and being able to apply it to an investigation.  Some sort of specific training in computer forensic tools and techniques plus some legal background to understand the laws as they apply to forensic investigations is clearly warranted. Obviously, I think SANS training is pretty darn good, but there are lots of other programs out there as well.  Caveat emptor.

Another important piece is getting some actual case experience.  This can be the hardest part for somebody trying to break into the field. If you're just starting out, then you'll probably have to pay some dues.  You might look into large e-discovery firms and contracting shops that do a lot of forensic work.  They're more likely to hire junior analysts.  Another avenue is law enforcement.  It seems like the FBI is interested in bringing on more qualified civilian employees in the computer forensics realm, and state and local law enforcement would probably like a few as well.

If you're already working in a company as an IT professional and would like to make the move into forensics, then you should start cultivating relationships with the people who do incident response for your company. In some cases, there might not be anybody currently in your firm who has that job.  In which case, you might talk to your own management about sending you to training so that you can become that person. There's lots of scare stories in the news these days that can bolster your case for creating an internal resource for forensics and incident response.

In the meantime, there are lots of forensic challenges out there that you can use to practice your skills.  If you do well in those challenges, it will be good resume material and will likely bring you to the attention of people who are needing to hire forensic experts.

Which brings us to career development item #3.  You need to start linking up with the community and getting your name out there. Networking is always the best way to find a job.  There's a really interesting forensics community developing on Twitter-- check out Joe Garcia's "Follow Friday" list for a good starting point to find people in the community.  Read what these people are saying and the links that they're posting (to their own research and others) and I guarantee you'll learn a lot.

But networking has to be more about giving than taking.  So start
doing your own research and writing your own blog posts, white papers, etc.  There are a lot of great Open Source forensics projects that you can contribute to.  Log2timeline, regripper, and volatility are all examples of projects where you can easily contribute small modules to expand the power of these tools.  Or take on a bigger project-- such as adding support for a new file system type in the Sleuthkit, like EXT4 or XFS (both of which are desperately needed, IMHO).

Try and find local groups in your area-- whether tech groups, ISSA, InfraGard, HTCIA, or what have you-- and find other people with common interests.  Get some practice speaking in front of these groups. Submit talks to security conferences, BSides, etc.  If you're doing interesting work and can be articulate about it, you'll get a job.

And finally, never ever stop learning.  The computer industry in general is all about constant retraining because technology changes so rapidly.  The computer forensics field is so new that there's an enormous amount that we don't know and research to be done.  That means it's even more important to stay up on the latest knowledge. It's like the sharks that die when they stop swimming-- if you stop learning and updating your skills then your career is going to die. Celeste Stokely taught me, "Learn one big new thing every year." It's good advice and I've tried to stick with it.

AFoD: One of the things we discussed recently on Forensic 4cast was whether someone who is interested in going into digital forensics should pursue an actual degree in digital forensics or something broader like computer science.  As I think more about it, it seems if you have a burning desire to get into digital forensics and you want to gain some academic training, it's reasonable to get a degree from one of the handful of good programs that are available.  However, this isn't going to be an option for many people.

For example, take a high school student here in the United States who doesn't have a lot of money to spend on college or who wisely don't want to start their professional career with a six figure debt burden.  Those students might be limited to situations where they can get in-state tuition at a school in their state of residence and those schools will likely not have a digital forensics program.  However, there are many state universities that have excellent programs in areas such computer science and engineering.

What would you recommend for those students? Regardless of their degree option, what topics would you encourage them to study to become better digital forensic examiners?

HP: When I think of specific topics from my educational background that help me in my daily practice it would be things like programming, operating systems, compiler design, algorithms, and so on.  I went to a pretty "Ivory Tower" Liberal Arts school, so some of the really practical stuff I had to get on my own by hacking around in the school computer labs (much to the detriment of my GPA).  In fact, my major was actually math because my school only offered a minor in computer science and not a full degree program.  But I learned a lot from the math curriculum too-- like how to model problems and estimate the computational complexity/processing time requirements, and how to construct proofs and logical arguments.

And I think this latter part-- learning how to think about problems-- is one of the most important things you can get out of college. The specific technologies I studied in school (Pascal programming anybody?) are no longer in vogue.  But I "learned how to learn", I learned how to write, and I was exposed to a broad range of topics that end up helping at the oddest times.  Heck, the Shakespeare I studied in English Lit once helped me defuse a raging flame-war on an intra-company mailing list!

Undergraduate programs are your chance to get a broad background and explore lots of different subjects.  Use this time wisely. If you're able, I would recommend doing your undergrad at a smaller, teaching-focused school where you can get more one-on-one time with the faculty, as opposed to a big research university where you get lost in the crowd.  If you get really interested in a field and want to study it in depth, then get into a graduate program-- probably at a well-funded research university, if possible-- and really dive in to your research.

AFoD: Let's talk a bit about the current state of digital forensics.  What is your impression of where we are as a community? What do we do well and what do we need to get better at as a community?

HP: We are so young as a discipline!  It would be easy to focus on what we don't know, the tools that we wish we had, short-comings in different curricula, and so on.  But being a "glass is half-full" kind of person, I have to say that it's pretty amazing what we can
do today as compared to a decade ago-- just take memory analysis as one example of many.  And we're getting better every day as the result of sterling work by the community.

What we're trying to do now is engineer solutions to problems on platforms where the vendors give us little or no direct support or documentation.  In the late 70's and early 80's, Unix people who were faced with a similar situation vis a vis a lack of Unix support from AT&T.  My people dealt with the problem by creating a community to share information that they had gleaned through their own research.

We're starting to see some of that in the forensics community, but we need to do more.  I'd like to see an ecology of forensics gatherings at least as rich in number and variety as the conferences for people who are interested in breaking systems.  And I'd like to see more forensics gatherings that are not sponsored by a single vendor of forensic tools, and which avoid government entanglements that inhibit the free flow of information.  Let me be clear that I'm not looking for people to share IoC's or technical details from actual cases.  I'd just like to see us sharing basic "block and tackle" type technology notes that will make everybody's lives easier.

We need better cross-platform tools.  The mature tools in the field
run largely on Windows and do a good job analyzing Windows-- Mac and Linux, not so much.  The Mac folks have a bunch of tools for analyzing Macs (and other Apple devices) that largely only run on the Mac platform.  Linux wizards mutter arcane command-line incantations and make data appear.  I shouldn't need three different computers to do my job! And that's not even counting the the mobile device insanity that's already threatening to overwhelm us.  I have a feeling that things are going to get worse before they get better here, because everybody's so focused on simply understanding all the details of their chosen platforms, but it really does make life hard for analysts who have to deal with cases involving multiple platforms.

And I think we have some more "bridge building" to do.  Within our own community we have people with a background in law enforcement or from a military service academy.  And then there are folks like me coming at this out of the "computer geek" community.  Both sides are still trying to understand where the other is coming from and what valuable stuff each side is bringing to the party.  But then there are also all of the external communities we touch: law enforcement, the Bar, Human Resources and other corporate gatekeepers, and even just normal citizens who have need of our services.  There's a lot of education and outreach that needs to be done so that these folks understand what we do and what help we can provide, and increase our understanding of what they need from us.

AFoD: You've been involved in the information security and digital forensics communities for a long time.  This has given you a unique position where you have had considerable exposure to people who have come into digital forensics from different paths whether from law enforcement or a more traditional information security background. What have learned about the differences between a law enforcement perspective on digital forensics compared to an information technology perspective? Are there any differences that you can see?

HP: From my perspective at the far end of the "nerd" part of the spectrum, I see a lot of differences.  One basic thing I really respect about my law enforcement colleagues is their ability to manage their case loads. As an IT person, I was always proud of my ability to "multi-task" and juggle multiple projects simultaneously.  But I realize now that I had nothing on an attorney who is simultaneously managing a couple of dozen cases all headed to trial or plea agreements on different schedules, or my LE friends who are having to work so many different kinds of crimes in parallel.

I've also gotten a terrific education from the LE and legal side of the house on what evidence is useful and relevant, as opposed to stuff that I think may be cool from a technical perspective, but which doesn't help them make the case.  This has helped me streamline my investigations, as well as my report writing.

Now the plus side of my being a computer person, and the reason I get called in to help on cases, is that I can often extract evidence that the LE folks lack the skills or processes to get at.  Sometimes I have to create new tools to do this, which is something I'm comfortable with because I'm confident around information technology.  The dark side of this ability, though, is that sometimes we computer folks get sucked down technological rat-holes, where we spend a huge amount of time producing evidence that doesn't really end up being that useful.  Somebody with less computer skills, but with more case experience might just move on to some other, more fruitful portion of the analysis.  Letting go of an interesting technical challenge and moving on to another piece of the investigation is a skill I'm only slowly getting hold of.

On the other side of the coin, I see my LE friends spending so much time on manual tasks that could be solved by a little scripting.  Like many people who aren't primarily technologists, they don't know what things should be easy to do with technology and what's hard-- or they just don't have the skills to implement solutions.  I once helped an agent who was manually cutting and pasting EXIF data from an image viewer into a spreadsheet-- about 30 minutes of my tinkering with awk and shell and the output of exiftool saved countless hours of investigator time.  It's a huge compliment to me when the LE guys say, "We like having Hal around, because he gets things done faster."

AFoD: You're bringing up a point that I'm seeing emphasized more and more by people in the top tiers of the profession.  Conducting an efficient investigation that answers specific questions that our customers want answered is the core of what we should be doing in digital forensics, isn't it? How do you go about scoping out and planning your investigations so that you get your customers what they want?

HP: I think the type of engagement has some bearing on whether you're targeting specific questions or going for a more wholistic approach. Lately I've been doing a lot of "lead generation" work for law enforcement, and that's definitely about honing in on very specific kinds of information.  E-discovery is much the same-- winnowing out particular types of information your client is interested in from huge volumes of data.  On the other hand, if I were working up a report on a system that was going to be crucial in a court case, I would want to do a more thorough job just to make sure I got at all the evidence: both inculpatory and exculpatory.

As far as the scoping question goes, I think the key is high levels of communication and short turn times.  With my law enforcement clients we'll start with a briefing for me on the particulars of the case, and I'll try and understand from them what sorts of information they're interested in.  For example, if they're looking for a pattern of on-line behavior, then I'll probably focus my initial efforts on browser artifacts, on-line chat, email, and so on.  If they need attribution data then maybe I'll look more at network data timelines, file meta-data, contact lists, the social networks, and so on.

This is not to say that I won't work up other data from the system, but I want to get into the case as quickly as possible and start extracting data.  I then take the data that I've found in my first pass which I think is "interesting" and show it to the client.  Maybe they say, "Great! We want more of that", or maybe it's uninteresting. Either answer helps guide my investigation.  Also, of course, evidence that I turn up sparks a whole new line of thought which may lead to them asking me for other kinds of information.

At this point I return to the images I'm analyzing for another pass.  That will turn up more information that I'll take back to the client, and so on.  Early on in the investigation I may talk with the client multiple times per day if they can spare the time. As we go through more cycles, we figure out more clearly the kinds of data we're interested in, and I need to interact with the client less often because I know what they're looking for.

Even if I don't need guidance on what to look for, however, I still
like to give the client quick updates on what kinds of information I'm finding, just to keep them up-to-speed on how the investigation is proceeding.  This is just quick status updates, mind you—possibly via short emails rather than live conversations-- I'll typically save full detail for the report.

The other advantage to these status updates is that it makes it pretty clear to everybody when we've reached the "point of diminishing returns" in the investigation.  Everybody's cost-conscious, and clients really appreciate it when they feel like you're looking out for their bottom-line.  They tend to remember things like that when they're needing to hire somebody to help with the next investigation.

AFoD: I'd like to discuss the topic of criminal defense work in the
digital forensics community. I have observed an increasing resistance to the "pariahization" of digital forensics people who do criminal defense work.  When I first entered the field, there wasn't much pushback on the idea that examiners who engaged in criminal defense work had gone over to "the dark side" and that it was essentially acceptable to push these examiners to the margins of our community. Now I'm seeing that attitude becoming much less prevalent. I suspect this is because we are seeing more people in the law enforcement community leave to start second careers in the private sector.  I also think one of the things that is causing a change in attitude is that we have more people from the traditional information security world entering the digital forensics field who don't view forensics through the lens of the justice system. This gets back into our previous discussion about the differences of people who enter the digital forensics community from an information technology background as opposed to a law enforcement one. What are your thoughts on digital forensic examiners who engage in criminal defense work?

HP: Maybe it's because I'm coming at this from a non law enforcement background, or maybe I'm just hopelessly naive, but it seems to me that forensic analysis should be first and foremost about science, and not about choosing sides.  Prosecution or defense, we as analysts should report on observed facts.  Where we're required to draw conclusions based on our observations, we need to be aware of our biases and avoid letting them distort our findings.  And because we sometimes have to make judgments based on incomplete evidence, it's incumbent upon us to acknowledge the gaps in our understanding and be scrupulous in our research and experimentation.  As scientists we should welcome peer review and encourage transparency.

Also, I was raised to believe in "innocent until proven guilty", and that everybody is entitled to a "robust defense" regardless of their ability to pay.  To me that means that the defendant should also have access to the same level of forensic experts that the prosecution enjoys and not just the people who have "lowered themselves" in some way to work defense cases.

OK, that's the "perfect world" view anyway.  The history of jurisprudence is littered with examples of witnesses, forensic analysts (both digital and traditional), attorneys, and law enforcement personnel who have-- intentionally or otherwise-- manipulated a case and achieved an outcome that was directly contradicted by the available evidence.  And rising legal costs mean that justice can be trumped by deep pockets.

I don't know how to solve these problems in our current justice system. But I do think that reducing the adversarial nature of the current system would be a good first step.  Shouldn't a trial be a collective search for the truth rather than a slap fight?  Wouldn't cooperation reduce costs for both sides?

As forensic analysts, we can't control how attorneys, judges, and other members of the justice system behave, but we should at least begin by unwinding artificial "us vs. them" divisions in our own field. Let's not ostracize fellow practitioners based on the types of cases they work on.  Instead, let's grow the size of our community so that we can expose-- through peer review and science-- those members who act unprofessionally or unethically regardless of which side of the aisle they're currently sitting on.

AFoD: What can we expect to see from you in 2011?

HP:  The SANS Forensics curriculum is just exploding in popularity and all of the qualified faculty are scrambling to meet the demand.  As an independent, I've got a bit more "flex" in my schedule than some of the other faculty who've got full-time employers, so I expect to be teaching more for SANS in 2011.  In particular, I seem to be picking up some of the dates for Lenny Zeltser's excellent Reverse Engineering Malware class--particularly the international training.  Also, Chad Tilbury and I are co-teaching For508 via SANS' vLive! distance-learning technology later this year, and I'm really looking forward to that.

I'm also getting out to some non-SANS forensic events in 2011.
For example, I'm giving a couple of talks at the DoD Cyber Crime Conference in January, and a couple more at CEIC in May(apparently I need to find a non-SANS venue for the Fall as well).  And while I'm on the road so much, I've been trying to make appearances at various local user groups-- for 2011 I've got confirmed dates to speak at the ISSA meeting in Tacoma, WA, NECERT's Cyber Security Forum in Omaha, and the Linux User Group in Boulder.  (you can follow my travels on the Deer Run Associates home page).  There are so many folks that I've been corresponding with via email and Twitter that I'm hoping to meet in person in 2011!

I've also got a plan to continue writing articles for the SANS Forensics Blog every 4-6 weeks.  I've got a backlog of topic ideas-- it's the research and the writing that can be difficult to find time for!  For example, there's at least two more articles planned in the series I started recently on EXT4.  And of course there's the weekly Command Line Kung Fu blog that I'm co-writing with Tim Medin.

The big one, however, is that I promised Rob Lee I'd develop a Linux Forensics class for SANS.  This is a huge undertaking-- hundreds of hours of work-- that I need to find time for in 2011.  I'm cheating a bit in that the research, blog posts, and presentations I'm doing lately are all development work that's going to end up feeding into the final version of the Linux Forensics class.  But it's still going to be a massive effort to give birth to this baby.

Thank You

I want to thank everyone from the bottom of my heart for their generous support of the blog this past year. From about the time I first started in digital forensics, I've been active in communicating with my peers through forums and all of the great email lists that we have available to us. I decided to start this blog so that I could have a more permanent place to store things that I  wish to share with the community. When I started it, I didn't know how well it would be received given the number of excellent blogs in the information security and digital forensics space. I've been amazed and humbled to see the blog’s metrics in areas like monthly page views increase from being counted in the dozens to the thousands.

My hope is that I will continue to meet and exceed your expectations in 2011. To that end, I have some excellent interviews that I’m hoping to accomplish including ones that are almost complete with people such as Hal Pomeranz and Ryan Pittman. I will to use next year’s interviews to do things like introduce to you people in the community who you might not know and to promote the excellent work of others. I will continue to blog about the issues of the day that impact our community such as regulatory issues. I will also use the blog to share my research efforts. For example,  I am hoping to start on a memory forensics project next year.

I will also start using the blog to periodically post book reviews that I will be placing on the Amazon website. For example, I expect to get a proper review completed for Hacking Exposed Wireless, Second Edition early next year. I’m finding writing book reviews to be a new challenge that I’m happy to pursue. A good book review needs to be pithy, but also provide the reader with more than simply saying, “This is a great book. You should buy it.” 

As always, I’m particularly grateful to those who take the time to leave comments on the blog or to contact me privately. It means a great deal to those of us who blog to get feedback and suggestions from our readers. Thank you very much for your support this year. Merry Christmas and Happy New Year to you all!

Standing Athwart Information Technology

I read a discussion recently where a group of very sharp information security professionals were discussing the topic of deploying mobile devices in an enterprise environment. The discussion quickly turned to a variety of “what if” scenarios that we love do to in information security. During this discussion someone made the excellent point that we could “what if” almost any bit of technology to death and come up with reasons why adopting that technology is a bad idea.

One of the classic faults of information security people is to automatically look for reasons to tell our customers not to deploy new technologies or to greatly limited their usefulness if deployed. Security people are fantastic for coming up for reasons not to do something and creating sometimes elaborate doomsday scenarios that could come to pass if our advice is not taken. While it is understandable that a community of people who spend their careers thinking about and responding to serious security incidents would think like this, it is not an attitude that is in the best interest of our customers.

Our job as trusted advisors is to facilitate the secure use of technology. As information security professionals, we should not to be standing athwart information technology yelling stop. It is not good for our customers and it is not good for our careers. We are in a time of rapid and exciting technological advances whether it is something such as “Cloud Computing”, social networking, or mobile device technology. We should be technology enablers rather than preventers.

The invaluable Mike Cloppert wrote a fantastic piece recently where he argued that we should be working to enable “Cloud Computing” for our customers rather than working against it in the name of fear of the unknown. We should take this same attitude with mobile device technology. It’s here now and it is a very powerful tool for our customers to utilize in advancing their objectives. As digital forensics and information security professionals, we should be continuously looking over the horizon to discover and understand technological advances early so that we can work with our customers to adopt, secure, and maximize their potential.

In the digital forensics community, we have been paying a lot of attention to mobile devices because they are playing an increasingly important role in our investigations. Because we’ve spent so much time studying this technology, we are in an excellent position to not only work with our customers to secure it, but to encourage them to adopt it.

We live in an era where powerful mobile devices are cheap and accessible to large numbers of people. We’re also entering an era of widely available high speed data connections for these devices. For example, Sprint has had their high speed mobile network up for some time now and Verizon’s LTE network just came online. This means there are going to be millions upon millions of people around the world with inexpensive, portable, and powerful devices that will be connected to increasingly fast and affordable data networks. We should be encouraging our customers to quickly embrace this technology so as to obtain an advantage over their competitors. As Margaret Thatcher might advise us, this is no time to go wobbly.

Did We Make a Mistake?

The comments from my last blog post were excellent and you can read them here.  Troy and Neil are quite correct. There is another accreditation issue looming over the digital forensics community other than digital forensic certification.  The accreditation of digital forensics labs is something that we need to start talking about more as a community.  As it stands right now, accreditation of digital forensic labs is voluntary and relatively rare. There are a small percentage of labs that have become accredited through organizations like ASCLD/LAB.  I’m curious about what others think about this issue.  Neil makes a very articulate argument, but I find myself sympathetic to Troy’s position.

My initial thought is that voluntary accreditation against a standard that is specifically tailored to digital forensics labs sounds reasonable enough.  However, I have concerns about the concept of mandatory accreditation. For example, it could easily be used to establish a guild system similar to what we see with some state licensing standards.  I am also concerned that mandatory lab accreditation standards could stifle innovation.  The way we do things in digital forensics changes so quickly that standards would almost certainly not keep up. Remember it wasn’t all that long ago where we were automatically pulling the plug from the back of Windows machines as a best practice. Now we’re in the age of live response and the tools and methods available have changed rapidly.

I wonder if we have made a mistake in the digital forensics community by calling our work areas “labs”. I started in traditional law enforcement where crime labs were places that forensic scientists tested all sorts of very perishable evidence that could easily be destroyed or contaminated if great caution wasn’t taken.  For example, it makes a great deal of sense to have strict controls in place when you are working with blood samples.  Improper storage and handling are likely to result in destroyed or tainted evidence.  

While there are very valid concerns relative to tainting digital forensics evidence that need to be continuously addressed, we’ve got it a lot easier than our colleagues in traditional crime labs.  We can easily create digital storage containers like forensic images with free and widely accessible tools that can be safely used outside of a controlled environment such as traditional crime lab.  One of the greatest gifts to digital forensics examiners is the simple hash value.  You can’t hash a blood sample, but you certainly can hash an image of a hard drive.  I can make an unlimited amount of identical copies of my digital evidence.  You can’t do that with blood. 

You can also put a forensic image of a hard drive on your laptop, bring the laptop down to Starbucks, and do a proper and defensible digital forensics exam while sipping your Gingerbread Latte.  Do that with a blood sample and you’re going to have a very uncomfortable court experience in your future. With digital evidence, I can take my evidence, put it on a external hard drive, leave it unsupervised on the floor of a busy shopping mall for days on end, and I can still show that nothing was altered by using hash values.  Blood? Not so much.

Consider an independent digital forensics consultant who works out of his house while traveling most of the time doing incident response work. Does he need to have his “lab” accredited?  Does that make any sense? What exactly constitutes his “lab”? His laptop where he does most of his forensics work in some hotel room? His home office where he spends less time than on the road?

How does this sort of thing scale into the future? What if a digital forensics lab uses some sort of Software-as-a-Service type provider for some of its examination work? Does that outside provider also need to be an accredited digital forensics lab?

I understand why traditional crime labs need to have very strict standards and why ASCLD/LAB accreditation style standards are embraced.  What I’m having a problem with is equating what we do with digital evidence to what these traditional forensic science labs do with their evidence.  If we adopt artificially stringent standards that weren’t originally intended for digital forensics, we could put a lot of private entities and smaller law enforcement organizations out of business at a time when we need more capacity to keep up with the increasing demand for digital forensics.

Certification, Licensing, and Accreditation in Digital Forensics

Considering the subject matter that I’m going to be wading into with this blog post, I want to start off by doing some full disclosure.  I’m a member of the Board of Directors for the Consortium of Digital Forensics Specialists (CDFS) and I’m also in the orbit of the SANS Institute. I’ve done both volunteer and paid work for SANS and the Global Information Assurance Certification (GIAC). I’m hoping to teach my first Community SANS class for them sometime in 2012 which would be a paid engagement.  As always, I speak only for myself on this blog and what I write does not necessarily reflect the views of any organizations that I’m associated with such as CDFS or SANS.

Some of the hottest topics of discussion in the digital forensics community are the issues of certification, accreditation, and  licensing.  In fact, one of the most common errors that I see in these discussions is confusing the terms and their goals.  In the digital forensics community, these terms have specific meanings that I would like to try and define up front.

Certification takes the form of an outside entity who certifies that an individual has met some sort of minimum standard of competency in an area of digital forensics.  The entities that do this inside of the digital forensics community are legion and include organizations such as the International Society of Forensic Computer Examiners (ISFCE), the International Association of Computer Investigative Specialists (IACIS) and GIAC.

Accreditation, for the purposes of this discussion, is an outside entity such as the Forensic Science Accreditation Board (FSAB) or American National Standards Institute (ANSI) who through an accreditation process validates that  a digital forensics certification or organization meets its minimum standards.  For example, GIAC has several of it’s certifications accredited by ANSI including the GIAC Certified Forensic Analyst (GCFA) certification.  There are several entities such as the Digital Forensics Certification Board (DFCB) and IACIS who are interested in pursing FSAB accreditation.

Licensing is a government entity regulating a particular profession in such manner where it becomes unlawful to engage in certain professional activities without a license. There are a whole host of professions that are regulated in this manner to the extent that a person needs government permission to engage in activities such as private investigation, practicing medicine, cutting hair, giving therapeutic massages, and a long list of other activities.

Two out of the three of these things are good ideas for the digital forensics community.  Certification of practitioners and the accreditation of the bodies that certify them are vital to professionalizing the industry and helping us progress as a community.  The licensing of digital forensics practitioners is a bad idea regardless if digital forensics practitioners are required to be licensed as private investigators or specifically as digital forensics examiners.

I’m not an absolutist when it comes to licensing.  I understand that in certain limited cases pertaining to critical issues such as public health and safety, there is an important role for government to play in regulating certain activities.  However, it’s important that we as community understand that the history of professional regulation has not been a rosy one.  Much of what we see here in the United States relative to professional licensing is just a modern day version of the guild system where professions use licensing  to keep out competition and control the market.

The common case that is made by those who support the licensing of digital forensics is that it will somehow increase professionalization by weeding out those who are unethical or incompetent.  This gets into a common mistake that is made by supporters of licensing which is to assume that licensing is a measure of competency.   While it’s true that, licensing arrangements frequently mandate some sort of training in the professional area, this is not necessarily a measure of professional competence. In the cases when testing is performed as part of the process, it is generally used to validate regulatory knowledge rather than professional competency.  It’s that mandatory training requirement (if one exists) that allegedly insures professional competency. Not coincidentally, it’s also what is used to establish modern day guilds that we see in professions like law, medicine, and even massage therapists. 

Because digital forensics is a convergence of technology and law, we already have measures in place that protect the public from unethical and incompetent examiners and methods.  We have standards like Daubert and an adversarial legal process that has well established methods of vetting those who would act as expert witnesses during legal proceedings.  Licensing of digital forensics people is unnecessary in the face of well known and accepted gatekeeping processes for legal proceedings. 

Not only is it unnecessary, but it’s harmful for both the profession and the public.  This is because licensing will likely result in a digital forensics guild system where the government will decide who can practice digital forensics and who can’t.  It will do this without much serious thought to the issue of professional competency which is the banner in which proponents of digital forensics licensing frequently rally under.

One argument is that a digital forensics licensing system can be established that would provide for competency assurance by requiring that licensees have a certification in digital forensics from an approved entity.  This is unhealthy for the community because it could very well result in the various certification organizations having to put a lot of time and money into lobbying the various government entities to allow their certification to be one of the approved certifications.  It gets worse if a government regulatory body were to decide that they were only going to accept one digital forensics certification as the standard for licensing.  That will put the certification bodies in direct adversarial competition with each other to make themselves the standard for that regulatory body.

There also is the issue of law not keeping up with technology which is a frequent occurrence in the digital age.  Even if I were to allow myself to be swayed by some siren song of licensing, how does state specific licensing work here in the United States?  Licensing systems are generally done at the state level.  Digital forensics is very much an interstate and international issue.   What if you have a case that requires you to engage in regulated activities in many states where a license is required for each one?  What if each of those states not only requires a license, but they also require different digital forensics certifications as part of that licensing process?

We don’t need a modern day digital forensics guild system.  We are capable as a community to regulate ourselves through collaborative efforts like the CDFS, the various well established and respected organizations like ISFCE and IACIS, and through the legal system’s standards in vetting people who provide testimony in legal proceedings.

Just say no to digital forensics licensing.

Certification and accreditation are something that we should embrace as a community in part to help ward off any licensing efforts by the government.  This should be an area of common ground between those who support licensing and those who support industry self-regulation.  For example, if one supports licensing of digital forensics professionals as a way to ensure basic competency, there has to be some sort of competency testing component to that process. That component can be achieved by professional certification through the various digital forensics certification bodies.

If we are going to be taken seriously as a profession, we ourselves have to take our profession seriously.  That means coming together as a community to establish minimum standards of competency for digital forensics examiners and providing methods in which examiners can show that they have met these standards.  We have many respected organizations who have spent a lot of time and effort doing that very thing and judging by the amount of people I see who hold digital forensics certifications, we have embraced those efforts as a community.

It’s important to understand that certification does not mean mastery.  It just means that an outside organization has validated that an individual has met the minimum standards as defined by the organization.  In fact, certification doesn’t necessarily even mean professional competency.  Ask any digital forensics hiring manager and they will be able to provide you with stories of certified applicants who failed their hiring process because of a lack of technical competency.  Doing a week of digital forensics training and then obtaining a certification doesn’t mean that someone is necessarily a competent digital forensics examiner, but it’s a start especially someone who is interested in getting into the field.

Accreditation is a key component of certification.  It’s essentially the certification bodies being certified themselves by a trusted outside entity such as the FSAB or ANSI. As a community, we should be pushing the various certification organizations to advance the cause of digital forensics professionalism by pursing accreditation.   We should do this because our professional organizations and their associated certifications will be taken more seriously if these organizations can show that they are following industry standard practices when it comes to the credentialing of digital forensics practitioners.

GIAC went the ANSI route and I think that means that the GCFA certification might be the first digital forensics certification that has achieved accreditation from a well recognized standards organization.

I know IACIS (I’m an associate member) is interested pursing FSAB accreditation.  That’s great to see because IACIS has spent a lot of time and effort into making their CFCE certification into a well respected certification in the digital forensics community.  They recently made the decision to open up that certification process to those who aren’t members of IACIS which is part of what needs to happen for FSAB accreditation.  The FSAB prohibits membership in an organization as a requirement for certification.  I’m not sure when the certification will be available to the public, but IACIS is working on getting that done.

One of the primary premises behind the DFCB is to establish an industry standard digital forensics certification that would achieve FSAB accreditation.  This effort hasn’t gone all that smoothly, unfortunately.  The “Founders” Digital Forensic Certified Practitioners (DFCP)  process that I went through to achieve my DFCP certification was disorganized and understaffed.  Since that time, I haven’t seen much in the way of improvement when it comes to communication and organization on the part of the DFCB.   They haven’t been very good when it comes to communication of what is going on with the organization and what progress is behind made towards their ultimate goals. Transparency hasn’t been a hallmark of the organization.   For example, I would like to know who makes up the various committees.  The website lists who leads their committees, but not who are members, what the committee goals are, and what progress has been made towards those goals.  Early in their history they posted some documents of this nature pertaining to early organizational meetings, but that has not occurred in some time. I’ve yet to find a DFCP certified person who is happy with the organization. They mean very well, but they’ve clearly had some trouble when it comes to communication and execution. I’m hoping things will get better for them as they pick up some momentum because their stated goals are laudable. I would also like to see at least one digital forensics organization achieve FSAB accreditation.

Interview with Dr. Gary Kessler

Future of Digital Forensic Tools Follow Up

Thanks for all of the comments both in public and private relative to my last post about the future of digital forensic tools. In a nutshell, we’re going to be approaching the point where digital forensic leaders like myself are going to have to make hard choices about where we spend our limited resources.  If I have five head count available to me, do I really want to devote the equivalent of one FTE to the care and feeding of increasingly sophisticated and complex enterprise sized digital forensic tools? That is going to cost me twenty percent of my analytical productivity.  Outsourcing the administration of my enterprise level tools through a Software-as-a-Service (SaaS) model in a cost effective manner will be a compelling option and I think it’s one that will be coming relatively soon.

CEIC 2011

‘Tis the season to start thinking about 2011 digital forensics training conferences and Guidance Software has worked very hard to make CEIC a very compelling choice.  I attended my first CEIC last year and enjoyed it immensely. The CFP period is open with a November 15th deadline.   I will be putting into present on a couple of different topics.  Hopefully, one will get accepted and I’ll see you there in Orlando.

SANS Forensics and Incident Response Summit 2011

Rob Lee has done an absolutely fantastic job turning this event into an amazing offering.  Because he is so well known in the community and has so many relationships with the A list digital forensics and incident response people, he has the ability to put together the best lineup of presenters that you’ll find at any digital forensics conference.  I’m hoping that I’ll be able to attend this event which will be held in Austin, Texas.

Dr. Gary Kessler Interview

I decided one of the best ways to follow up the “Take Vienna” blog post was to interview someone who has a background as both an academic and a practitioner in the field. I’m a big Gary Kessler fan and since he is both a skilled academic and sharp digital forensics examiner he was the clear choice.  It’s hard not to like him and he’s done a considerable amount of work over the years advancing the cause of digital forensics as a science, including being heavily involved with the creation of the digital forensic program at Champlain.   While Gary is no longer with Champlain, he continues to contribute to the digital forensics community through through a variety of ways which you can read about at his website.  Gary is a very active in several efforts to organize and professionalize the practice of digital forensics.

AFoD: What attracted you to the field of digital forensics?

GK: I have been involved with information security, in general, since the late-1970s. Computer forensics, as a form of infosec incident response, seemed to come into vogue in the late-1990s.

Meanwhile, the Internet Crimes Against Children (ICAC) Task Force was being formed in VT and the leadership all knew me and thought that having a computer techie (my M.S. is in Computer Science) helping out might be useful. So, in 1999 or so, I was asked to join the VT ICAC as a pro bono consultant.

As I got more involved with the DF community -- in 2002, it was mostly law enforcement -- I found myself meeting some of the finest folks I have ever worked with professionally. And I like investigative work, problem solving, working puzzles, and helping others understand what the computer has to tell you...

AFoD: What lead to you to getting involved in digital forensics in the academic world?

GK: I joined the faculty at Champlain College as an adjunct in early 2000 and full-time in the summer of 2000. I was already involved with the ICAC and participated in training activities.

In late 2001, the Task Force commander and I thought that it would be interesting to teach a course at the college in CF. The course was offered in the fall 2002 semester and filled during preregistration. During that semester, we became aware of a variety of NIJ studies that, among other things, suggested a gap between what LEOs actually knew about CF and what
they needed to know. At the same time, we were getting questions about our CF "program" -- yet we only had one course!

That lead to the development of an undergraduate CF program that started in 2003 and the online undergrad program in 2004. CC started a graduate program in CF management in 2009.

This all said, there is work afoot to come up with curriculum guidelines for DF. The project started about five years ago, sponsored by NIJ. For some reason, the output from that group never got published. After the AAFS adopted DF as a forensic science, the work started again and should be adopted/published, I would guess, within the next six months.

AFoD: What was your role at Champlain college and what makes that program unique from other digital forensics programs?

GK: I was the program director of the undergrad CF programs at their inception. Eventually, the online division took over the online CF program (in about 2007) and then I moved into managing the graduate program (2009). I was the
program director of the M.S. in Digital Investigation Management when I left the college in the summer of 2010.

I think the thing that made our undergrad program unique in 2003 was a) I don't know of another undergrad program that existed at the time and b) it combined computer courses, criminal justice courses, and CF courses.

AFoD: What makes up an ideal digital forensics academic program?

GK: This is hard to answer because it depends so much on the goals of the program. At the undergraduate level, I think that academia needs to prepare students for life-long learning. The undergrad of today might well have three or more *careers* -- not merely *jobs* -- in their lifetime so higher ed.'s first responsibility, IMO, has to be to make sure that students know how to learn.

Second, the curriculum should prepare the student both to enter the workplace or graduate school. So this is a bit long but I think that a CF/DF program needs to teach some general education to round out a student, and a broad spectrum of computer science (including fundamentals of operating systems), law, networking, and, of course, CF (processes, file systems, mobile devices, cyberlaw, cybercrime, e-discovery, testimony, etc.).

Graduate programs are a bit harder to nail down. At the graduate level, a technical program, IMO, is advanced, specialized computer science. This is a program for individuals who will be next generation tool creators, process
developers, tool testers, etc., etc. A management program, such as the one at CC, is designed for those aspiring to manage CF labs and people, and understand the business aspects of such activities.

In either case, DF students need to know how to write well, speak well, and *read*!

AFoD: What should the end goals be for an academic digital forensics program?

GK: Pretty much as stated above. Produce generalist learners, specialists in DF as a multidisciplinary science, and prepared for life, the universe, and everything!

Since I got this far, *I* have never taken the posture that CF graduates could be able to immediately walk into a CF shop and be able to work on exams unsupervised. I have always felt that the programs should concentrate on the process and introduce a plethora of tools rather than produce a student who is expert in one tool. The latter is the purpose of training. I observe that students getting a CJ degree still go to a police academy and
then get additional on-the-job training prior to pushing a cruiser on their own. A CF graduate should be able to quickly get up to speed but will still need some training.

AFoD: Other than teaching, what role should academic digital forensics program play in advancing digital forensics?

GK: I think that academicians can play a critical role in advancing the science. They should also be practitioners so that they are aware of the real problems faced by people in the field. They can then be in a good position to help work with the practitioner community to advance standards, tools, research, legislation, local training efforts, and more.

AFoD: Is digital forensics a science? Is it an art? Both?

GK: DF had better be a science now that the AAFS has adopted Multimedia and Digital Forensics as a new branch! :-) Sure, there is some art to the practice but we *MUST* define and adopt processes for DF that are, in fact, based upon science. For this, it's worth reading Fred Cohen's books and learning about information physics!

AFoD: You mentioned the American Academy of Forensic Science (AAFS) has added a Digital and Multimedia Sciences section.  Why is that significant for the digital forensics community?

GK: If the DF community wants to be taken seriously as a forensic science, then this nod from the AAFS is incredibly important. DF is the only forensic science that has been largely driven by the practitioner community rather than the computer science community. But the examination of computers is,
fundamentally, computer science.

That is *not* to say that one needs to be a formally trained computer scientist in order to practice computer forensics. Not only do I not believe that but it would fly in the face of the reality of the profession today. But DF needs to become more of a science and less of an art!

AF0D: What digital forensic programs other than Champlain could you recommend to students who are interested in studying digital forensics?

GK: There are now a bunch of program depending upon where you want to study and what approach you'd like to take to your studies. Certainly the undergrad programs at Daytona State, Defiance College, Bloomsburg University, University of Rhode Island, Utica College, Univ. of Alabama Birmingham, Univ. of Mississippi, Johns Hopkins, Fountainhead College of Technology, and Univ. of Advancing Technology are well-known and worth
investigating. There are others, too: see
http://www.e-evidence.info/education.html

There are also grad program worth looking into... programs at
Daytona/UCF, Purdue, John Jay, Univ, of Maryland University College, and California Sciences Institute leap immediately to mind.

And these are just the programs in the U.S.!

AFoD: Is there a career path for people interested in digital forensics,  but who want to practice it as a full time academic discipline?

GK: Yes, I believe so... but accreditation requirements of colleges and universities will demand that anyone with a full-time job in academia hold at least a masters degree and, preferably, a doctorate.

AFoD: What should be the role of the scientific method in digital forensics?

GK: Well, that couples with the question above. The Daubert and Kumho Tire rulings guide the introduction of scientific and technical evidence in federal courts and about half of the state courts. We need to have a science that answers the tests. One Daubert requirement is that the procedures have a known, or knowable, error rate. It is unclear that we even know how to
calculate the error rates in DF practices.

Again, I am *not* saying that DF work is sloppy or error-prone or anything like that. I am suggesting that we know that we're not seeing 100% of everything and we have no way to prove that what we're misisng doesn't change the bottom line.

We need more science and more research.

AFoD: What is your view on the role of digital forensics certifications?

GK: I think that certifications are ONE part of professional credentialing but, in the end, speak to one's training. I also think that academic credentialing is important, as well. Unfortunately, an academic degree may not demonstrate one's practical knowledge/skills and certifications don't demonstrate a person's fundamental and theoretical knowledge -- things that
I believe are essential for life-long learning and professionalism.

GK: I think that professionals need to demonstrate a combination of appropriate training and education. Certification is a part of that.

AFoD: Should the digital forensic community standardize on just one digital forensics certification or continue to have multiple
certifications from different organizations?

GK: Even if I felt that one standardized certificate was the right thing to do, I don't see how we could choose which one, given that the barn door is already open! (If I can mix the metaphors.)

I would like to see some standardization is what the generic industry certs actually show. In response to the NAS report from 2009 about forensics, I think it imperative that any DF certification include a practical portion. I think that being able to communicate one's findings in a report need to be a part of the certification. I think that for our own credibility, the certs that are respected demonstrate experience and practical competence and NOT be ones that you could read a book for and pass. Vendor neutrality, IMO, is key as well as being available industry-wide.

I also see different levels of cert coming. A general DF cert is
great. I see specialty certs also coming, such as mobile forensics and e-discovery.

AFoD: What advice would you give to those who want to break into the digital forensics field?

GK: Well, it would depend up the age of the person and the area where they live. DF is no easier a profession to break into than information security; you can't just get some training, hang up a shingle, and start working. If I were 40 years younger, I would say go to school. If making a career change, I would survey the local practitioner landscape and try to find a mentor. So many people say, "I want to learn CF and volunteer with local police and
catch child perpetrators." Well, that may be noble but it is very hard to find in practice! It's easier to find a private firm. Look for local DF organizations, such as a local HTCIA chapter; it's a great way to learn and to network. In some cases, it means thinking about moving; there are a lot of CF jobs but they are not equally distributed geographically.

The Future of Digital Forensics Tools?

Access Data released the newest version of it’s popular FTK Imager tool this week which incorporates a variety of new features including the ability to mount images as a drive or physical device.  A key feature of FTK Imager is that it can be used as a very basic file system analysis program.  By adding the mounting feature, Access Data has taken another step in moving this tool beyond being just a nice acquisition tool towards something that will commonly be used in examination work.

I think this small event could signal the beginning of the end of forensic software manufacturers charging high prices for comprehensive digital forensics suites such as EnCase and FTK.   This doesn’t mean that digital forensics tools are going to be cheap in the future, but I think the future is starting to become clearer.

The way I see the evolution of digital forensics tools goes something like this:

The Zero Generation: The Mesozoic era

In the beginning, there was nothing.  Seriously, nothing. This was before I entered the field, but I know enough people who started in this era to have a good feel for it.  Examiners during this time had to use tools like hex editors and system administration type tools because of the lack of tools specifically designed for digital forensic purposes.  As the market expanded for digital forensics tool, we entered…

The First Generation: The Enhanced Hex Editor Era

We had tools like Expert Witness (which later became known as EnCase) created in this era that were designed to be digital forensics tools.  The dominant tool of this era was EnCase.  The core of EnCase was the ability acquire forensic images in a court defensible manner and to examine the resulting images. When being used for analysis, EnCase was essentially an very specialized read-only hex editor that could parse file systems.  Guidance Software’s innovation path was to increasingly add useful features that parsed different types of artifacts.  Users had the ability to create their own features through the EnScripting language. 

Access Data’s FTK became a very popular tool to use alongside of EnCase because it handled email very well and also incorporated the DtSearch indexing engine.  However, FTK was generally not considered to be as good as EnCase when it came to disk level examination functions so it tended not to be used as replacement to EnCase.   This was fine for tactical level digital forensics work, but for eDiscovery and for larger data set digital forensics cases, the hex editor model didn’t scale well which brought us to…

(Okay, I have to stop here because I know I’m going to have people screaming at their monitors shortly if they haven’t already started.  I know I’m grossly oversimplifying this, but I don’t intend for this post to be a comprehensive history of digital forensic and eDiscovery tools.  Sleuth Kit rocks and the price is right, you also have great tools from this era like ProDiscover, X-Ways, and SMART. However, at high level, they all are essentially the same type of forensic software. I’m also assuming that the people reading this blog post have a working knowledge of how all of these tools work.)

The Second Generation: The Database Era

The eDiscovery people really pushed this and were the first people to develop tools that used databases to manage data and allow for scalability. On the digital forensics side, Access Data was the first traditional digital forensic company to really embrace this by releasing Oracle based FTK 2.  As we know, FTK 2 was an abomination (it didn’t actually work), but FTK 3 followed shortly and has become a dominant second generation digital forensics tool.  There are plenty of eDisco tools that aggressively use database technology as well as other unique technologies such as concept analysis, but most digital forensics companies are still largely in the first generation era.

Access Data and Guidance Software have been aggressively involved in the enterprise level eDiscovery and digital forensics market for quite some time.  Guidance still appears to approach things from a first generation view which I think is one the reasons why Access Data has gained so much traction recently.  Access Data has embraced the explosion of innovation in the eDiscovery market up to and including merging with CT Summation.  They understand that scalability is going to be a key issue that digital forensics companies will have to face and they clearly understand that first generation digital forensics tools are not the future.

This is why I think the release of FTK Imager 3 is a small, but key event.  If a company like Access Data can be profitable with second generation tools and enterprise focused strategies, they may decide to put downward pressure on their first generation-centric competitors by offering up their own first generation technology tools for free or very low cost.  We may very well be seeing the beginning of the end of paying thousands of dollars for first generation style hex editor tools because…

The Third Generation: Digital Forensics Software as a Service

The eDisco people have already been here for awhile so it’s logical that the digital forensics world will follow.  I bet you see Access Data start moving to this model at some point in the near future.  They’re already pushing the limits of what a database layman can do and one of the consistent complaints I hear about FTK 3 is that it’s very resource intensive.  Access Data already sells expanded versions of it’s FTK suite to customers who need more horsepower and capabilities, but this requires additional hardware resources and personnel to administer it.

The next logical step will be for a company like Access Data to embrace the cloud based SaaS model for digital forensics tools.  In this model, Access Data would manage all of the hardware and software and also act as the custodian of the data for a case.  The customer’s analysts would work with the data remotely without having to manage forensic hardware or software.

I’m not saying third generation digital forensics tools will replace first and second generation tools.  For example, I think we will have the enhanced hex editor type tools with us for a very long time because they work well for cases with small data sets.  However, the increasing size of data sets coupled with the need for advanced features like data analytics and more powerful forensics software will usher in this generation of digital forensics tools.

Access Data gained a competitive advantage by beating Guidance to the second generation. If were Guidance Software, I’d be working on third generation of digital forensic tools so that I could return the favor.

Work For Lenny

I had the good fortune to attend Lenny Zeltzer’s introductory malware analysis presentation at the HTCIA Northeast chapter meeting today.  I have been looking forward to attending this presentation ever since I learned about it.  Lenny is an accomplished instructor and did a remarkable job explaining a complex topic like malware analysis in terms that made it very approachable for the layperson.

Lenny breaks down malware analysis into two main parts. The first part is behavioral analysis.  This is where the examiner works with the malware in a safe environment to learn about it through interaction and observation.  The second part is code analysis which involves using tools like debuggers to examine malware at an assembly language level.  It’s important to note that knowing assembly language is not a prerequisite  to becoming a malware analyst or attending Lenny’s training.   That said, if you want to be excellent at it, you’ll need to add knowledge of assembly language to your skill set. 

Lenny is going to be teaching his SANS malware analysis course in New York this month and there are seats still available. COINS-LZ is a discount code that will reduce the cost of the class by ten percent. 

Lenny is also in the market for a security architect to come to work for him. If you are interested in a great job in the NYC metro area, this is a fantastic opportunity.

SANS Network Security 2010

I had the pleasure of being Rob Lee’s Teaching Assistant for Computer Forensics Essentials (AKA FOR408) last week at SANS Network Security 2010 in Las Vegas.  It was more than a surreal experience for me.   About six years ago I took my first SANS class taught by Ed Skoudis. Ed showed how great information security training could be when you match a fantastic instructor with great material. I’ve been a SANS fan ever since.   Part of what made it a surrealistic week was that the event was held at Caesars Palace in Las Vegas.  Factor in a high level of excitement on my part, minimal sleep, the almost cartoon like Vegas surroundings and interacting with all of the great SANS instructors, students, and staff and it’s quite a difference from my normal work week.

One of the nice things about being on the instruction staff, even just as a TA, is access to the speaker’s ready room where the instructors eat breakfast and lunch together.  I essentially got to act as a fly on the way watching how the instructors abused interacted with each other.  They clearly spend a lot of time together during the year at these conferences and outside of them and have a high sense of camaraderie.   Being able to hear what people like Stephen Northcutt, Lenny Zeltzer, Ed Skoudis, Mike Poor, Kevin Johnson and the rest talk about when they’re together was worth the trip alone. I was finally able to meet Lenny Zeltzer, Hal Pomeranz, and Kevin Johnson in person for the first time and enjoyed their company immensely.

If you’re at a SANS conference or any other venue where Kevin Johnson is speaking, I highly encourage you to attend one of his “Social Zombie” presentations.  Kevin is not only a very sharp fellow, but it’s more than a little bit of a showman.  His material is excellent and he’s an innovator when it comes to the convergence of social networking and penetration testing.   I enjoyed not only the content of the presentation, but watching how Kevin works a room. He provides a very high energy presentation and is almost constantly in motion when he’s talking.  The audience was very engaged with his presentation because of how well he can connect with a large group of people.

I also finally got to meet Scott Moulton in person.  His hard drive repair class was in the room next to where Rob was teaching FOR408.  Scott’s class is nothing short of awesome.  He brings an amazing amount of equipment with him and it looks like an outrageous amount of fun.  If you were were the kid who liked to take things apart just to see how they worked, this would be the class for you.  I saw legions of hard drives in various states of assembly and watched at least one student trying to solder his way into bringing a drive back to life.  The class room looked like a hard drive civil war had occurred there.  It was hard drive Gettysburg.

Being a Teacher’s Assistant  for Rob was a great experience.  We had around 50 students in the class and I enjoyed helping Rob introduce them to the world of digital forensics.  I was surprised by how many of the students were new to digital forensics.  One of the things I found most fulfilling  was being able to share my own experiences learning digital forensics with the students.  It was a long time ago that I started on this path myself and it quite a bit of fun watching people start on the same path with so much enthusiasm.  We even had a student destroy their first hard drive (complete with “magic smoke”) while trying to image it. I felt like a proud father watching his son score his first touchdown. If you do digital forensics long enough, you’re going to kill your fair share of hard drives.  Imaging can be really rough on a drive and if you have one that is already on death’s door knocking loudly, the imaging process is more than capable of opening the door.  Now that I think about it, we should have rushed it next door to Scott’s class…

Coming Soon

The interview with Richard Bejtlich was very well received and I’m grateful for all of the positive comments that were sent in response.  One of the positive things that came out of the interview is that I have been approached by several really high caliber people who liked the interview and who I will be using as future interview subjects.

Great Digital Forensics Job Opportunity

I don’t normally do this, but I want to bring to the reader’s attention a great job opportunity in digital forensics.  The United States Department of Agriculture’s Office of the Inspector General (USDA OIG) has a GS-13 position open in their digital forensics lab in Kansas City, Missouri.

This is a great opportunity because you’d get to work on a team lead by Mike “Jake” Jacobson.  Jake is a former local law enforcement officer who has had a distinguished career in digital forensics including being assigned to Heart of America RCFL.  He’s a very sharp fellow who is passionate about digital forensics and he’s finalizing his team over at the OIG.  This is a high paying job in a low cost area with a great team.

I have nothing to do with this process and any questions should be directed to Jake at mike.jacobson a/t/ oig.usda d0t gov.  You’ll want to act fast if you are interested because the application period is only open for two weeks.

Interview with Richard Bejtlich

I had been working on a “guru post” for the longest time where I examined backgrounds of top tier digital forensics people in an attempt to find common trends on how they got to where they did in the field.   I found common paths like obtaining technical degrees from good universities and getting direct job experience in the military or law enforcement. However, no matter how I wrote the post up, it just didn’t work well.  Telling someone to get an electrical engineering degree from the US Air Force Academy and then trying to get assigned to the Air Force Office of Special Operations is an interesting bit of advice, but it’s only going to work for a limited amount of people and it doesn’t provide any broadly applicable lessons for the rest of us.

What I decided to do instead is to just interview some selected gurus and focus on how they decided to go into digital forensics and what paths they took to get there.  I took a page from the British Special Air Service and went with the “Who Dares Wins” approach by asking Richard Bejtlich to be my first interview subject.  Richard was kind enough to agree and what follows is the result. 

Richard went well above and beyond the call of duty with this interview and I’d like to thank him publically for putting up with what essentially turned out to be a beta test of the concept.   He salvaged more than one bad question for me (the Hoover institute one was a dog. I knew what I wanted to do with it, but setting up a question with content unfamiliar to the interview subject is a bad idea) and was very patient with a process that I’m going to make much shorter in the future.

Richard’s bio is available here and it briefly documents his career progression during his Air Force service and into the private sector.  You should also follow his Taosecurity Blog which is a must read for anyone involved in digital forensics.  Lastly, Richard was recently interviewed by Gary McGraw of The Silver Bullet Security Podcast.

The Interview

AFoD: Like many leading digital forensic and information security experts, you chose the United States Air Force as your starting point. Can you describe what motivated you to become an Air Force officer?

RB: After seeing Star Wars in the theater in 1977 I decided I wanted to be an astronaut.  Once my eyesight failed I realized I couldn't be a pilot, so I decided to be a Mission Specialist who designed spacecraft.  I looked for programs in astronautical engineering.  I told my parents I would put myself through college.  An Air Force ROTC program appeared to be my best option.  I wanted to attend MIT and have the Air Force help pay for the program.

AFoD: What was your path to the Air Force Academy?  Did you participate in any programs like Air Force JROTC or anything similar?

RB: I am an Eagle Scout, but I did not participate in JROTC.  My family had very little prior military experience and no awareness of the service academies.  I learned about USAFA while attending an Air Force ROTC event at Hanscom Air Force Base.  Some USAFA recruiter earned his or her pay that night!  The USAFA video they showed hooked my attention, and I applied to USAFA.  I also participated in the "Summer Scientific Seminar," a pre-Academy summer event to recruit cadets. Although MIT accepted me and the Air Force provided a ROTC scholarship, my USAFA acceptance arrived first.  I accepted the appointment and sealed my fate!

AFoD: What was your Eagle Scout project?

RB: A high school friend succumbed to childhood leukemia while we were freshmen in high school.  To honor her memory and to raise awareness and funds for childhood leukemia I organized a road race in 1989 as a high school senior.  I believe 4 to 6 more happened during the 1990s; I helped with a few but was away in Colorado for most.

AFoD: What was it about that USAFA video that so attracted you to the institution?

RB: The tennis courts.  I saw something like 30 of them and thought, "Wow."  On a serious note, USAFA seemed like THE place to go if you wanted to be an officer, and especially if you wanted to be an astronaut.  I didn't apply to any other military academy.  People asked "what if you don't make it?  Shouldn't you apply to West Point and Annapolis too?"  I replied "I don't want to be in the Army or Navy."

AFoD: You're one of the leading digital forensic and information security thought leaders in our community.  Many of your peers who became similarly prominent obtained degrees in disciplines like electrical engineering and computer science from top quality schools like the Air Force Academy, VMI and MIT.  Why did you decide to study history rather than a technical discipline?

RB: I was ready to study astronautical engineering at USAFA.  My placement tests landed me in Calculus 243 with juniors and seniors.  However, my freshman history teacher, Captain Ruffley, made a big impression on me.  He was an intelligence officer who focused on the Soviet Union. His work sounded a lot more interesting.  I also met professors who were officers and who hoped to be astronauts, but they seemed so *old*.  I could do military intelligence right out of the Academy. When we started bombing Iraq during the first Gulf War in early 1991 I knew intelligence was the right role for me.  I selected history as my
major, and later added political science as a second major and French and German as minors.  I was a little too ambitious back then.

AF0D: Can you describe how your studies in history and political science at the USAFA prepared you for your future roles in the Air Force and  the private sector?

RB: These are three of my favorites: 1) People now are NOT smarter than anyone who live before.  People who think they are smarter will likely assume they can overcome history's lessons.  Their hubris enables failure. 2) Writing is very important.  Solid writers often prevail. 3) Nation-states are not monoliths.  Read Essence of Decision: Explaining the Cuban Missile Crisis by Graham Allison.

AFoD: Hoover Institute Fellow Peter Robinson recently conducted an Uncommon Knowledge interview with Ambassador Charles Hill.  The interview was an exploration of Hill's idea that academic institutions are failing to teach "grand strategy" to our future leaders.  He states that students are disappointed when they undertake studying a discipline like political science expecting to be taught how to tackle big problems, but wind up being presented with small problems such as voting trends for a particular congressional district.  Hill also thinks that one can not learn "Grand Strategy" without an appreciation of literature. You are a proven leader who clearly understands how to tackle "grand strategy" type problems.What taught you how to think about how to attack a large problem such as information warfare in a corporate environment? Did learning history at the Air Force Academy and your graduate work at Harvard lay the foundations of where you are today or was is something after your formal education?  Would you recommend the Air Force Academy to a high school student who wants to become a future leader
in private industry?

RB: As a history and political science double-major I confronted lots of "big problems" in school.  After graduation in 1994 I was thankful to be selected to attend the Harvard Kennedy School (as it's called now) to work on a Master's degree in public policy.  As a lieutenant I shared the class with colonels and enjoyed instructors who were former National Security Council advisors, generals, and so on.  My USAFA and Harvard experiences contributed to my development, but everything I needed to know about leadership I learned as a Patrol Leader.

AFoD: Your experiences in the Boy Scouts mirror my own a bit in that one of my formative experiences was as a Police Explorer (which is a  program that is part of the Boy Scouts). I learned a lot about leadership early by being exposed to a program like the Explorers. What would you recommend to someone who is reading this interview while they are in college and doesn't have the opportunity to join an organization like the Scouts or the Explorers, but wants to learn  about leadership first hand?

RB: Lead something, anything -- say, organize an event.  If you're a  security person, organize a group or a con.  There is no substitute  for being on point!

AFoD: The Kennedy School is one of the nation's most prestigious schools of government and public policy. Graduate school tends to come much later in the career process of the average US military officers.  How did a junior officer such as yourself get selected to attend that program?

RB: Since the 1970s USAFA and Harvard have shared an arrangement whereby they accept 4 or 5 graduates each year.  I applied and won a slot.

AFoD: You have a passion for reading, writing and reviewing information security books with Amazon being your chosen platform for your book reviews. What constitutes a five star book?

RB: Five star books 1) change the way I look at a problem, or properly introduce me to thinking about a problem for which I have little or no frame of reference; 2) have few or no technical errors; 3) make the material actionable; 4) include current research and reference outside sources; and 5) are enjoyable reads.

AFoD: What causes you to remove stars from a review?

RB: Failure to meet the previous. I also subtract for plagiarism, poor production quality, and repetition of previously published material.

AFoD: Anyone who follows your blog or your Twitter feed knows that you are less than enthusiastic about Power Point based presentations.  What sort of presentations do you advocate as a replacement?

RB: Focus on the message not the medium.  Don't think "I need to create slides on topic X."  Think "how best can I communicate topic X to the audience?"

AFoD: What advice would you give to someone who is going to give a presentation before a large audience on a technical subject like information security?

RB: Consider using handouts instead of slides.  Attend a class by Edward Tufte.

What I Learned From The Interview

Richard is clearly a very smart and driven person.  I knew that going into the interview, but I really wanted to learn how that manifested itself in his formative academic years.  If you are smart enough and driven enough where you end up with the US Air Force Academy and MIT offering you an opportunity to study with them, you’re clearly someone who will be successful in your chosen field.  That’s an obvious lesson.  Apply yourself and utilize the gifts and opportunities that are available to you and you maximize your chance for success in any field. 

I was also taken by how flexibility was a theme with his professional development.  He didn’t set out to be an information warfare leader, but when presented with setbacks and new opportunities, he readjusted and continued on his path. This is something you’d expect from our war fighters, but it’s a lesson that all of us can learn from and apply to ourselves.

The leadership aspect of the interview was something that resonated with me because of my own experience in law enforcement and police exploring. One of Richard’s most early formative experiences with leadership was his experience in the Boy Scouts.  There are fundamental qualities of leadership that can be learned early in life and do not necessarily require formal training in a service academy to obtain.  While these are qualities that are drilled into those of us who served in the military or law enforcement, they are also attainable by learning from proven leaders like Richard.

Another thing that stands out from both this interview and his professional life is passion.  If you aren’t passionate about something, you are unlikely to reach the top of that profession. Richard shares a quality that appears to be universal with the top tier people in digital forensics and that’s passion for the field.  The top players in our field aren’t people who just punch a time clock and then forget about digital forensics when they go home. Richard is as an excellent example of this.  Not only does he direct the incident response function for one of the biggest corporations in the world, but he reads, writes, and reviews information security books.  He conducts research and teaches.  He also finds time to frequently update his blog and indulge people like me when we ask him to do an interview.

Lastly, his perspective on book reviews is a natural progression of what he spoke about early in the interviews and what I learned from him.  He approaches his book reviews as a learning experience where you can see his intellectual flexibility on display when he speaks about how a good book can change the way he looks at a problem.  This ties into his comments about the study of history earlier in the interview where he stated that “People now are NOT smarter than anyone who live before.  People who think they are smarter will likely assume they can overcome history's lessons. Their hubris enables failure.” This is an important lesson in digital forensics because our field has so much technological complexity.  An open mind and a healthy degree of intellectual honesty will go a long way in allowing one to remain open new ideas and methods in this ever changing field.