Saturday, December 4, 2010

Did We Make a Mistake?

The comments from my last blog post were excellent and you can read them hereTroy and Neil are quite correct. There is another accreditation issue looming over the digital forensics community other than digital forensic certification.  The accreditation of digital forensics labs is something that we need to start talking about more as a community.  As it stands right now, accreditation of digital forensic labs is voluntary and relatively rare. There are a small percentage of labs that have become accredited through organizations like ASCLD/LAB.  I’m curious about what others think about this issue.  Neil makes a very articulate argument, but I find myself sympathetic to Troy’s position.

My initial thought is that voluntary accreditation against a standard that is specifically tailored to digital forensics labs sounds reasonable enough.  However, I have concerns about the concept of mandatory accreditation. For example, it could easily be used to establish a guild system similar to what we see with some state licensing standards.  I am also concerned that mandatory lab accreditation standards could stifle innovation.  The way we do things in digital forensics changes so quickly that standards would almost certainly not keep up. Remember it wasn’t all that long ago where we were automatically pulling the plug from the back of Windows machines as a best practice. Now we’re in the age of live response and the tools and methods available have changed rapidly.

I wonder if we have made a mistake in the digital forensics community by calling our work areas “labs”. I started in traditional law enforcement where crime labs were places that forensic scientists tested all sorts of very perishable evidence that could easily be destroyed or contaminated if great caution wasn’t taken.  For example, it makes a great deal of sense to have strict controls in place when you are working with blood samples.  Improper storage and handling are likely to result in destroyed or tainted evidence.  

While there are very valid concerns relative to tainting digital forensics evidence that need to be continuously addressed, we’ve got it a lot easier than our colleagues in traditional crime labs.  We can easily create digital storage containers like forensic images with free and widely accessible tools that can be safely used outside of a controlled environment such as traditional crime lab.  One of the greatest gifts to digital forensics examiners is the simple hash value.  You can’t hash a blood sample, but you certainly can hash an image of a hard drive.  I can make an unlimited amount of identical copies of my digital evidence.  You can’t do that with blood. 

You can also put a forensic image of a hard drive on your laptop, bring the laptop down to Starbucks, and do a proper and defensible digital forensics exam while sipping your Gingerbread Latte.  Do that with a blood sample and you’re going to have a very uncomfortable court experience in your future. With digital evidence, I can take my evidence, put it on a external hard drive, leave it unsupervised on the floor of a busy shopping mall for days on end, and I can still show that nothing was altered by using hash values.  Blood? Not so much.

Consider an independent digital forensics consultant who works out of his house while traveling most of the time doing incident response work. Does he need to have his “lab” accredited?  Does that make any sense? What exactly constitutes his “lab”? His laptop where he does most of his forensics work in some hotel room? His home office where he spends less time than on the road?

How does this sort of thing scale into the future? What if a digital forensics lab uses some sort of Software-as-a-Service type provider for some of its examination work? Does that outside provider also need to be an accredited digital forensics lab?

I understand why traditional crime labs need to have very strict standards and why ASCLD/LAB accreditation style standards are embraced.  What I’m having a problem with is equating what we do with digital evidence to what these traditional forensic science labs do with their evidence.  If we adopt artificially stringent standards that weren’t originally intended for digital forensics, we could put a lot of private entities and smaller law enforcement organizations out of business at a time when we need more capacity to keep up with the increasing demand for digital forensics.

13 comments:

  1. Eric,

    I'm not seeing where the "Did we make a mistake?" post title fits in with the contents of the post...can you clarify?

    Thanks.

    ReplyDelete
  2. Thanks for dropping by, Harlan. It's funny (okay, a lot less funny now...) that I had that same thought before I published the post. I wondered if I made it clear enough in the content itself. I decided that the first sentence in the third paragraph took care of it and stuck with the title.

    I'm wondering if we made a community-wide mistake by using the word "lab" to describe our work areas. I'm concerned that using this term sets ourselves up for standards and restrictions that are more appropriate for traditional forensic analysis than for digital forensics.

    ReplyDelete
  3. I guess it depends on who you're talking to, or who's reading the post.

    When I started at ISS, our "lab" was in our home office. We were expected to follow certain procedures...admittedly, some clearly didn't.

    However, you could say "lab" to another person and they'd think "ASCLD certification". It's just like when you talk to some folks about "digital forensics" and their first thoughts go to expert witness testimony, Daubert, and defense attorneys.

    It's funny...there are "labs" funded, maintained and inspected by government entities that have massive backlogs and it takes a considerable amount of time to get reports on analysis back. However, there are others (such as Chris Pogue, myself, etc.) who have taken data in and turned around comprehensive reports in a matter of days, many of which have resulted in plea agreements.

    I don't think that it's mistake to call something a "lab"...IMHO, the mistake is in what a "lab" has come to mean.

    ReplyDelete
  4. Here are a few thoughts on accreditation. Starting with disclosure, I am a technical assessor with Australia’s accrediting body, NATA and I have spent the past several years working in a lab built from the ground up to meet the ISO17025 standards. So my opinions may be biased.
    First some background, the standard Forensic Labs are accredited to is ISO17025, the title of this standard is “General requirements for the competence of testing and calibration laboratories”. As the title implies this standard does not just apply to forensic labs, it can and is used by a range of labs performing tests from soil samples to DNA testing. As such it is a fairly general standard, with two main components. The first is a focus on quality management (the management requirements are aligned with ISO9001) and the other is on the technical soundness of the lab processes. When seeking accreditation the lab must also comply with a Field Application Document (FAD), this is developed by the accrediting body addressing the specific issues of the field the lab is to be accredited under. In Australia there is a FAD for all forensic sciences which includes two categories for digital forensics. The FAD can be downloaded for free from here: http://www.nata.com.au/phocadownload/publications/FADS_Amendments/ForensicScienceFAD.pdf if you are interested, there was also an update published in 2008 to include electronic evidence that can be found here: http://www.nata.com.au/phocadownload/publications/Technical_publications/Policy_Tech_circulars/technical_circular_9_nov08.pdf . ASCLD was originally accrediting under their own standard. They are now accrediting using ISO17025 (called International Laboratory Accreditation). They do not make their FAD equivalent available for free download.
    The most important thing to understand about all this is that when seeking accreditation you design your management practices to meet the standards requirements. The standard does not specify the processes you must use, nor does it dictate where you must operate. The main focus is on having documented procedures in place to ensure that you have addressed the requirements of your field, and that having documented those procedures you then make sure that they are followed. Of course it is the role of a technical assessor to examine your procedures and make sure that they are acceptable.
    So if you want to perform your analysis down at the local coffee shop and write that into your procedures then go for it. Of course that process would have to be approved by the assessors, which might prove a little challenging. Also in the case of your independent examiner it would be possible for him to achieve accreditation by designing processes to meet his needs. Provided he can demonstrate that he is using sound practices.

    ReplyDelete
  5. The rest of the post due to size limitations....

    The big concern with labs considering accreditation is the cost. I am not sure of costs in the US but in Australia you are mainly paying for travel and accommodation expenses for the assessors to attend for the inspection, we work on a volunteer basis (or more to the point our employers volunteer our time). The other big cost is the time it takes to put everything together, in this the smaller organization may be at an advantage as they have less people to organize, although a larger organization is likely to have the benefit of corporate policies already in place. In either case in terms of time to develop policies you are looking at a considerable investment. I was looking at setting up a private lab in Australia last year, and having roughed out what needed to be done estimated that it would take about 3 months of part time work to get everything together. This was setting everything up from scratch, so a bit easier than trying to document existing procedures or having to change them.
    The ongoing costs may result in a slight increase, but this should simply be due to the fact that you have to be more rigorous in your work. If it is anything else then you need to review how you have set things up.
    Personally I think the benefits of accreditation outweigh the negatives. The process of preparing for accreditation forces you to examine your procedures and ensure that they achieve their expected outcomes. Having said that it should be up to the industry and market, to decide if accreditation is appropriate or not. If labs start getting accredited and the courts, or customers start demanding it then other labs will start looking into it. I do not believe there is great value in having government mandated accreditation or certification for that matter.
    With regards to certification under ISO17025 accreditation you are required to have education and training requirements for the staff, along with regular proficiency tests. This really addresses the whole certification issue. You could accredit the organization, not the individual. Of course that is a whole other argument.

    ReplyDelete
  6. I should also point out that ASCLD are not the only body in the US conducting ISO17025 Forensic accreditation.

    ReplyDelete
  7. Eric,

    Yes, I do believe we’ve made a mistake referring to digital forensic facilities as “labs”.
    I think most normal people equate a lab with a group of white-coat scientists conducting experiments on rats, or in the case of forensics, a wet lab, i.e. DNA, blood, etc. It may not seem terribly important what normal people think but sometimes society trumps Webster’s Dictionary. And they sit on the jury.

    How we define ourselves matters. A little research indicates Webster’s Dictionary and others define a “laboratory” and “scientist” in sufficiently broad strokes to cover just about anyone who uses scientific principles in their job and any place with a roof and electricity. That doesn’t help at all. We must decide whether we’re going to call ourselves analysts, investigators, or scientists. We aren’t all scientists, we’re not all investigators in the classical sense (society’s definition again), but we all analyze data, therefore I choose “digital forensic analyst.” Furthermore, you really don’t have a lab if you can’t stock it full of “scientists” so I choose “facility”.

    That brings me back to society. Remember the debacle a decade or two ago where we decided we’d append “engineer” to everyone’s job? That misguided project in self-esteem was an epic fail that brought derision upon the domestic-engineers of the world, among others. Similarly, we seem focused on legitimizing our field of digital forensic analysis by referring to ourselves as scientists and our facilities as labs. Just because you can do something, does that mean you should?

    Most digital forensic facilities, for the purposes of this discussion, are typically for-profit or a government entities charged with supporting law enforcement. The reality both types of facilities face is the same; they’re an assembly line. These facilities must work as efficiently as possible to provide the best possible customer service in the shortest amount of time. They are definitely NOT a playground for geeks. They have business objectives and must be accountable for their performance and the quality of their product. What free-time exists is spent increasing efficiency, training and testing tools developed by engineers and scientists. Yes, some research and experimentation occurs as a natural byproduct of the job, and the techniques used should follow scientific principles, but in my opinion that doesn’t make us scientists or most of our facilities a laboratory.

    For those of you who are formally trained scientists and engineers, all my love to you guys and gals. It’s a privilege to have one or more of you on a team for sure. However, most of us are not, and don’t need to be, scientists. Contrary to the NAS report, a scientist on every street corner, lab and crime scene is unrealistic and just a little amusing. Okay, that’s my interpretation of their report but I don’t think I’m that far off base. We don’t have enough scientists and engineers to go around as it is, let alone the money to pay for them.

    In conclusion, most of our facilities exist to make money or put bad guys in jail. They’re not designed for research, testing and development of major forensic tools. This job is not a daily science project, it’s a grind. Let’s not call ourselves scientists unless the label truly fits. Also, let’s not call our facilities “labs” unless they’re heavily staffed with formally educated scientists conducting traditional scientific research or providing work product based on the natural sciences. The public may not appreciate our unnecessary attempt to elevate our self-esteem and dismiss us accordingly.

    Jake

    ReplyDelete
  8. I think lab is fair. A place where a scientific methodology is employed. If you look at the history of discovery and science, labs have certainly evolved over time. The main focus on proving an area where test can be performed over and over again to achieve measurable and repeatable results. There is certainly a want and need to have sterility and security in labs and I think that is where we do have an advantage as we can control the digital workplace much better than organic material investigators. We still have the need for policy, procedures and protocols to maintain a level of accountability and control. But, in all honesty I have a personal touch to my environment that meets forensic "soundness" but might not feel comfortable at first glance to another investigator. So does the personality of the user affect the neutrality of what can be considered a "lab"?

    Ovie Carol at a recent NYC4SEC meet-up (shameless plug) even said we have to stop referring to ourselves as scientists as it denotes a perception of white lab coats and overly sterile thought processes on investigations as digital forensics is an art as much as a science.

    So, maybe we do have labs - digital forensic labs - and they are different than traditional forensic labs but still meet the criteria to carry the moniker.

    ReplyDelete
  9. Eric,
    Great article! At a recent forensics meeting I attended, this very issue was brought up about ASCLD/LAB standards amongst forensic laboratories. Digital Forensics is dynamic and evolving. Just as you mentioned, we can validate our forensic evidence (e.g.MD5/SHA-1 hashing).
    At this point, I think any form of "governing body" enforcing accreditation, would negatively impact the community and diminish creativity in numerous ways. This is another reason why it is so important for forensic practitioners to get involved, take a stand, and be an advocate for their community.

    ReplyDelete
  10. I've worked in an ASCLD/LAB digital evidence lab operated by a federal law enforcement agency. I can't begin to estimate how many hours are devoted to the bureaucracy of the concept, quality control and quality assurance, mandated training, policy development and review. It is all so permeated in the science of digital examinations that its difficult to tell where one ends and the other begins.
    Since then, I worked in the private sector, not in a certified forensic lab. I've done work on-site, in hotel rooms, offices and even in the rent car. Lab isn't about the structure or walls or desks or where you store the evidence. Its about a state of mind, a way of doing things, an adherence to a set of rules and policies and a set of ethics and doing things in a way that others would agree was good and a best practice.
    I guess you could say that you are the lab, you take it with you. When you stop doing the best job you can... you aren't a lab anymore.

    Always do the best you can and don't give anyone an inch to test your ethical conduct. Do a good job as if your Mom is watching you.

    ReplyDelete
  11. I think that Law Enforcement labs should be accredited as their findings could send someone to jail. So their processes and procedures should be strictly accountable.

    On the other hand, for consultants that work in civil cases, I don't see where accreditation is necessary. Like Eric said, the consultant may work out of his home or take the drive to Startbucks and do his examination there. If the consultant did not follow procedures, the other side will definitely bring this out and the court will have to decide as to what evidence is admissible.

    The difference in criminal cases is that the findings may be used to file charges, that could lead to someone's arrest. That is not the case in civil cases. Even if the examination findings are used to file the original complaint, the other side will have ample time to examine and contest the findings if necessary and the Court would have to decide on the admissibility of the digital evidence.

    My .02 cents...

    ReplyDelete
  12. I don't think we make a mistake by calling the area we work "labs". That being said, your point about accreditation slowing innovation is on the mark. I have talked with an examiner who works in an ASCLD forensic lab and that is a big issue that they have - lack of innovation because of the restrictions of accreditation.

    And yes, while criminal labs can put people in jail, civil labs have the ability to change lives because the work they do can call for sanctions and other monitary awards, some quite drastic.

    ReplyDelete
  13. Accreditation is a humorus topic. Where were the "forensics guys" when computer data cases started accumulating for LE. They weren't there. I was there from the start in my department, and the "forensics guys" would not help, could not help, and ridiculed our help request. The rest is history. Now they want a piece of the action.....hmmmmmmmm. This field is not a science. It's a computer technician skill. Labs are not necessary, or even applicable. Licensing and regulation are necessary to protect the public from the abuses in computer forensics going on now by private sector experts and LE examiners. It's becoming a mess.

    ReplyDelete