Sunday, November 25, 2012

AFoD Interview With Carlos Cajigas

I have spent the last several months working on relocating from the New York metropolitan area to Tampa, Florida. Now that I’m starting to get settled into Florida, I will be blogging on a more consistent schedule. Carlos Cajigas is one of the many sharp Florida-based digital forensics people that I have had the privilege to meet in my travels around the state. Carlos is an accomplished digital forensics examiner as well as a bit of an entrepreneur. He’s very passionate about the use of Linux in digital forensics and it didn’t take talking to him for very long to realize that he would be a great interview subject for the blog.

Carlos Cajigas Professional Biography

clip_image002Carlos, a native of San Juan, Puerto Rico, is the Training Director and Senior Forensic Analyst for EPYX Forensics. Additionally, he is employed by the West Palm Beach Police Department (FL) as a Detective/Examiner assigned to the Digital Forensics Unit with over 9 years law enforcement experience. He has conducted examinations on hundreds of digital devices to include computers, cell phones, and GPS devices to go along with hundreds of hours of digital forensics training. His training includes courses offered by Guidance Software (EnCase), National White Collar Crime Center (NW3C), and the International Association of Computer Investigative Specialists (IACIS).

Carlos holds B.S. and M.S. degrees from Palm Beach Atlantic University (FL). In addition, he holds various certifications in the digital forensics field to include EnCase Certified Examiner (EnCE), Certified Forensic Computer Examiner (CFCE) from IACIS, and Certified Digital Forensic Examiner (CDFE) from Mile2. Carlos is a Florida Department of Law Enforcement (FDLE) certified instructor with experience teaching digital forensic classes. He is an active member of both the International Association of Computer Investigative Specialists (IACIS) and Miami Electronic Crimes Task Force (MECTF).

Most recently, Carlos has endeavored in writing a blog for EPYX Forensics that would assist other digital forensic examiners in using free open source Linux-based tools to do their jobs. He hopes to develop and implement course training in this area in the belief that there are alternatives to expensive commercial software and training.

A Fistful of Dongles Blog: What led you into becoming a law enforcement officer?

Carlos Cajigas: Although police work has always appealed to me, the decision to join law enforcement didn’t enter my mind until late in my 20’s.  At the age of 17, I moved from Puerto Rico to Palm Beach County, Florida to pursue a career in baseball. At that time my priorities were anything baseball and my responsibilities were simple: keep making good grades and go to practice. Although I was a fairly talented baseball player, I also knew that I wasn’t the most gifted. From an early age, I learned that hard work could make up for the areas where one lacks talent. That is a lesson that still holds value even to this day. So the answer was simple, I played and practiced as much as I could while working hard every day. Subsequently, I received a baseball scholarship to Palm Beach Atlantic University. 

I continued to work hard and had some success on the field. I broke a few records and became an MVP. It truly was a great experience that I will always remember. Unfortunately, my collegiate baseball career ended after 4 years of eligibility. So there I was - 22 years of age with a decision to make unsure of what I really wanted to do.

On September 11th 2001, halfway through grad school, the events of that day changed many lives forever. The impact that day had on me, led me to join law enforcement. Although law enforcement work always appealed to me, that day I decided that I wanted to make a career out of giving back to the community. I wanted to be part of another team with similar interests and values of mine. Baseball in many ways prepared me for that jump. I finished grad school and applied to the West Palm Beach Police Department. I have been a police officer now for nine years. Throughout my career, I have been part of multiple units and many teams. I have been given opportunities to do some good and have taken advantage of them. Our city is a great city and our Department is top notch. Joining law enforcement is another decision that I am glad I made. 

AFoD: What happened once you joined the West Palm Beach police? What was your initial training like and how did you end up doing digital forensics work?

Cajigas: After completing the Academy and once at the Department, I went through an initial series of stages that I had to pass before being allowed inside of a patrol car. They included training in defensive tactics, driving, and firearms, among others. These were the core primary skills that were taught to trainees. The department was very strict about their minimum qualifications. 

I then progressed into a multi-month field training program that required me to go on patrol while a qualified senior officer sat next to me and evaluated me. My trainers were very good, no-nonsense, seasoned officers.  Learning at this stage was done at a very fast pace. I was taught radio procedures, report writing and everything else.  Every new call brought about a new challenge. On-the-fly problem solving skills were a must have and strict emphasis was given to safety. 

Upon graduation from the field training program, I began responding to calls by myself. I completed a one year probationary period and remained as a road patrol officer for about four years. I then joined a specialized unit created to reduce crime in a specific area of our city. Our team was made up of 6 officers and we were responsible for just about any crime or issue in this area. This unit provided me with an opportunity to conduct investigations from beginning to end. Some investigations required travel to neighboring cities and others undercover work. 

When a position opened up for the Digital Forensics Unit, I interviewed.  After a few months, I was notified that I had been awarded the position.  That was a very exciting day. I have always had a passion for computers and now I was being given the opportunity to combine police work with that passion. A few years later with a few hundred hours of digital forensic training under my belt, I enjoy computers even more.   

AFoD: What is life like working on the Digital Forensics Unit?

Cajigas: Life in the Digital Forensics Unit is full of activity. Our unit is part of the Palm Beach County Internet Crimes against Children (ICAC) Task Force and the Palm Beach County Regional Forensics Task Force. The task forces are made up of investigators and examiners from different participating agencies in the county. Our ICAC task force has a very proactive approach towards pursuing individuals who hurt children. As a result, we conduct an average of one search warrant every other week involving child exploitation. 

The operations that we conduct can take us from one side of the county to the other with very short notice. We travel in our mobile forensics van, and we triage and preview devices on location. In most cases, we can retrieve the necessary evidence that the investigator needs to make an arrest on scene.

As part of our duties to the Regional Forensics Task Force, we provide assistance to neighboring agencies without forensic units. The cases that we assist with can range from simple thefts to homicides and everything in between. On any day, it is quite common that we might get a request to process ten phones and five computers. The amount of devices that we see at the lab daily poses challenges and opportunities to learn something new every day. This is the part of the job that I enjoy the most - the process involving identifying a problem, looking for a solution and then implementing that solution.

Recently during a case, we came across a 3TB hard drive with a corrupt GPT. Once the drive was imaged, our tool of choice was unable to see the directory structure of the volume. We needed specific files, and we wanted to be able to access the volume inside of the drive without restoring the image or editing the image. After a little bit of research, we ended up accessing the volume by mounting the E01 image in Linux and using the program ‘Testdisk’ to point us to the starting sector of the volume.  The mount command then mounted the volume, and the volume became accessible.    

I have learned that Linux can be a little complicated; however, it is powerful and free. In this instance, once we got past the complicated part, we were left with powerful and free. I see the usefulness and versatility that Linux has when used in forensics. On the days that we have time to catch up, I dedicate a few hours to learning Linux in the hopes that it can become another tool in our tool belt in the battle against online criminals.  

AFoD: So how did you discover that Linux could be a powerful tool for digital forensics examinations?

Cajigas: I first got introduced to Linux back in 2007, before learning forensics. I stumbled across Ubuntu version 7.04 out of curiosity and necessity. During those years, I used to spend a lot of time building and fixing computers. I decided to try Ubuntu when a friend requested my help with recovering family photos from his BSOD’d Windows PC.  I burned the ISO to a CD and booted his “dead” computer from the CD drive. Ubuntu loaded and his computer “magically” came back to life.  His drive was still healthy and the directory structure was intact.  All of his family photos were there waiting to be copied. My friend was happy and I was hooked. To this day it still amazes me how the entire OS can run from a CD, just for the cost of the CD.

I began testing and installing as many variants of Linux on as many computers as I could just to see what I could learn. For instance, it took me about a week of testing different distributions before finally getting the right version of xubuntu working on a PS3. As a result, I installed Ubuntu on a flash drive and carried it with me, just in case of a “dead PC” emergency. 

Fast forward a few years and after some forensics training, I decided to try Linux for forensics. I did it out of curiosity and necessity. I saw that great tools like the SIFT workstation were already out there, so I was curious as to how to use them. I needed a second method of doing forensics, so that at the very least, I could use them to validate procedures. I downloaded as many of the forensic distributions as I could and began testing the programs. Just like with any tool, there is always a learning curve.  Unfortunately when it comes to Linux, that learning curve can sometimes be curvier. Just learning the commands for a specific program could take hours of research and trial and error. But once you learn the program, the results can be very rewarding.

There are programs in Linux for just about every action needed in forensics. Some of the smarter minds in forensics build these programs and release them for free for the benefit of the community. Unfortunately, the documentation on how to use programs in Linux can sometimes be difficult to find. I have found myself reading blog after blog gathering bits from one site and pieces from another, while teaching myself how to use these tools.  As a result, I have decided to document procedures on how to use forensic tools on your own standalone Ubuntu 12.04 machine. My intent is to help other digital examiners in using open source tools during the course of their investigations for free.

I started documenting these procedures at the beginning of the year, and I plan on adding many more. So far, I have documented procedures on how to recover Win7 passwords, using Testdisk, acquiring and mounting E01’s, recovering IE history, file carving, extracting files by record number, registry analysis, and parsing the MFT with analyzeMFT. My attempt is to outline the procedures from beginning to the end for users new to Linux, while explaining every step with screenshots. You can find them at

AFoD: What can Linux-based tools do for a digital forensics examiner that the increasingly wide range of Windows-based tools can't do?

Cajigas: They might just save you some money! Windows based tools like EnCase, FTK and X-Ways are simply excellent tools.They combine the  processes of acquiring, indexing, parsing, searching, recovering, and reporting all into one suite. In my opinion, there is no equivalent single program available in Linux that can compare to these great suites. Every lab should have at least one of these tools. What is available in Linux is an accumulation of different tools that when put together can accomplish almost all of the same things that these suites do. Some of the tools can do some tasks better, others, not so well. 

But the tools will always accomplish their tasks for free, and they might help you when you don’t have the commercial tool needed for the job. 

Let’s say for example that you and your case could benefit from an analysis of a timeline and you do not have the tool of choice to build that timeline.  Log2timeline is an excellent open-source framework for automatic creation of super timelines. Log2timeline received the 2012 Forensic 4cast, computer forensic software tool of the year. 

Every examiner that I have talked to can remember that time when their Windows forensic tool of choice crashed and failed to accomplish some sort of task. I can personally recall instances in the lab when Windows based tools failed to image devices, and I reverted to using Guymager in Linux with absolute success. And best of all, these tools can all be run from a ten cent DVD. I recently participated in a Rob Lee webcast were he so accurately described the SIFT workstation as your own mobile computer forensics lab.

Linux-based open-source tools alone can be used to complete forensic examinations. Many of these tools have helped me during my investigations.  They were released free to the community, and I believe that we can all benefit from them.  

AFoD: Let's pick out a couple tools to use as examples. I've sung the praises of log2timeline here on the blog and will continue to do so in the future. Let's focus on a tool that might not be as well known. What is Guymager and why should someone consider using it over a Windows-based tool?

Cajigas: Guymager is an open source forensic imager with an easy to use graphical user interface (GUI). The tool, created by Guy Voncken, was designed to be fast especially on multiple processor machines. It produces DD, E01, or AFF images and conducts verifications upon completion. In addition, it creates an .info file that stores acquisition details to include hashes, bad sectors, and SMART data. Because it runs on Linux, it often succeeds at acquiring those pesky drives that make Windows freeze and/or have trouble showing up in Disk Management.

Since acquisition is the one process that has to be done in 99% of examinations, Guymager is a tool I find myself using a lot of the time. Guymager can be downloaded from the Ubuntu Software Center.

Another open source Windows analysis tool with an easy to use GUI is the Forensic Registry EDitor (FRED). FRED is a registry hive editor created by Daniel Gillen. It can navigate the directory structure of a hive and has a built in hex viewer and data interpreter. Another cool feature built into the tool is that it has automated reporting functions that can give you the “RecentDocs” and the “TypedUrls” out of an NTUser.dat registry hive. FRED can be downloaded from

For their ease of use, these two tools are a good start and a must try for those interested in using Linux based tools.  

AFoD: What do you recommend for someone who wants to learn Linux and get to a point where they can comfortable leverage it for digital forensics examinations?

Cajigas: There will be a lot of reading involved, but these three steps will get you going in the right direction. The first step towards becoming familiar with Linux is to begin using it. As simple as it sounds, installing and using your favorite distribution will teach you a lot about how the OS works and how the directory structure is laid out. Once you know the layout, you are now able to spot the things that look right and the ones that don’t.

The next step is to become familiar with manipulating the shell commands. This is the stage where you learn commands like “cd, cp, rm, mv” and terms like input/output redirection. To redirect the results of one command into a second command and get only one set of results is the linux equivalent of killing two birds with one stone. Redirection is one of the most useful features of the shell. A well written website on learning the linux shell can be found at linuxcommand dot org.

The last step in the familiarization process is to start playing with the forensic tools. This is the fun part. I have written some articles to get you started with the basic forensic tools, and many more can be found on the web. Just like with forensics, in Linux there is something new to be learned every day with tools that are available for just about any task.

The more you play with the tools, the more comfortable you get. Figure out what your need is, and learn how to accomplish that task with Linux. Chances are the next time that you need the same task completed, you will revert back to accomplishing it in Linux.

AFoD: Are their any Linux distros that you recommend over others for the beginning Linux user?

Cajigas: In my opinion, before using any of the forensic live distributions, anyone starting on Linux should start with Ubuntu. Ubuntu was first released in 2004 by a UK based company called Canonical. Canonical provides support, patches, bug and security fixes for a period of eighteen months on each of their new releases, keeping Ubuntu up to date. 

Ubuntu was designed with ease of use in mind and comes with GUI based tools for installation, updating, personalization of the OS, and many more. It has built in support for a lot of different hardware, which translates to a good chance that it will boot and recognize the hardware in your computer. Due to its popularity, a web search will often point you in the right direction towards solving most of the problems you may encounter.

During basic Ubuntu use, mandatory interaction with the terminal is minimal. This gives the user time to get to know the OS before being forced to use the terminal for non-GUI tools. Ubuntu has been selected as the platform for popular live DVD distributions like the SIFT and DEFT. After you become comfortable using your installed version of Ubuntu, graduating to using live distributions no longer feels like unfamiliar territory.  

AFoD: Is there anything else that you'd like people to know?

Cajigas: Open source (Linux) forensic utilities are very useful as a supplement to commercial tools. However, there is the good and bad when it comes to open source. The good - tools are free and they are just as, if not more powerful than commercial tools. The bad – there is a learning curve, and they are harder to use. With that said, as part of EPYX Forensics, my colleagues and I want to bridge that learning curve gap so that forensic examiners are able to take full advantage of open source forensic tools. As I spoke about earlier, I have begun doing this by posting tutorials through the EPYX Blog. We are also currently putting together training courses for law enforcement, government and private sector personnel that we are planning to launch in early 2013.

The world of digital forensics is constantly evolving, and I believe there is a shift towards increased usage of open source forensic utilities, especially with the expenses that come with commercial tools. My hope is for all forensic practitioners to sit down and at least try the open source passage – you just might like what you find.

Saturday, August 25, 2012

Cyber Pearl Harbor

I’m an amateur student of military history who has written several blog posts in the past discussing physical warfare concepts and how they can be applied to the cyber world. For example, I’ve written about how the Battle of the Atlantic and the Battle of Britain provide lessons that can be applied to cyber warfare. Thus, I’m the last person who will say that it’s inappropriate to apply lessons from the physical world to the information technology world. So while it can be an appropriate thing to do, it can also be done in a haphazard manner that doesn’t correctly respect the historical record. The term “cyber Pearl Harbor” is one that is increasingly being used in a manner that just doesn’t make sense from a military history perspective.

120702-N-VD564-016 The Pearl Harbor attack involved the nation of Japan engaging its military to strike a substantial blow against Pacific Fleet of the United States Navy. The damage that was inflicted on the US Navy was such that it provided the Japanese military with a decisive military imbalance in the Pacific that it exploited until the American military was able to rebuild and regroup. In my mind, for an attack to be accurately labeled “cyber Pearl Harbor” it would need to involve a cyber attack that accomplished a similar objective. A reasonable example of “cyber Pearl Harbor” could be a nation-state using a cyber attack to substantially degrade another nation’s ability to respond to future attacks in either the physical or cyber world. An incident where an attacker unexpectedly brings down a power plant without any further attacks isn’t “cyber Pearl Harbor”. That’s certainly a serious incident, but it’s not equivalent to the Pearl Harbor attack.

Another term that I’m having even more trouble with is “cyber terrorism”. The American English version of the Oxford dictionary defines terrorism as “the use of violence and intimidation in the pursuit of political aims.” The Oxford British & World English dictionary defines it as:

the unofficial or unauthorized use of violence and intimidation in the pursuit of political aims:
the fight against terrorism
international terrorism

For a cyber attack to be accurately defined as cyber terrorism, the attack would have to have a violent result or at least some sort of intimidating effect. I just don’t see how a DDoS attack or even something destructive like Stuxnet clears this bar. In my mind, for something in the information technology world to be considered “cyber terrorism”, you’d need a result where you had the loss of life or a substantial and intimidating impact such as taking down a power grid of a major city. An action like a major urban power outage could very well result in indirect loss of life (heat related deaths during summer months) and violence (riots). It’s not that this couldn’t happen, but we just haven’t seen it yet.

Google Takeout

Tom Thomas is the marketing director over at IACIS. He posted to the IACIS email list recently about a Christopher Null authored article he discovered in PC World that explained Google Takeout. Tom was nice enough to give me permission to pass along what he posted to the rest of the team through the blog.

The Google Takeout webpage explains that “Google Takeout allows you to download a copy of your data stored within Google products.” Null explained in his article that:

Google wants you to keep using Search, Docs, and Google+, so it’s trying to play nice, and last June Google introduced a service designed to let you see, in part at least, what Google knows about you with a single click.

Eric Zimmerman’s Tools

Unsurprisingly, the Eric Zimmerman interview generated a tremendous amount of interest in Eric and his tools. One of the popular questions has been how to obtain the various tools that Eric has created and made available to the law enforcement community. You can contact Eric through the FBI’s Salt Lake City Division or you can just send me an email and I’ll pass it along to Eric.

Digital Forensics Email Lists

Another one of the questions that came out of that interview was a request to know which email lists that Eric was participating in since that was mentioned in the interview. I won’t disclose what lists that Eric participates in, but what I will do is contact some of the people who run the various lists that I am on and get permission to post their membership requirements and how to join. Some of these lists have few requirements for membership, but others are more restrictive such excluding people who do criminal defense work.

About The Photo

Photo credit information from the United States Navy:

120702-N-VD564-016 PEARL HARBOR (July 2, 2012) Sailors man the rails aboard the aircraft carrier USS Nimitz (CVN 68) as it passes the USS Arizona Memorial in Pearl Harbor. Nimitz is participating in the biennial Rim of the Pacific (RIMPAC) exercise 2012, the world's largest international maritime exercise. Twenty-two nations, more than 40 ships and submarines, more than 200 aircraft and 25,000 personnel are participating in RIMPAC exercise from June 29 to Aug. 3, in and around the Hawaiian Islands. (U.S. Navy photo by Chief Mass Communication Specialist Keith W. DeVinney/Released)

Thanks to the United States Navy and all of the other services who make these photos available for people like me to use. Thanks to Chief Mass Communication Specialist DeVinney for his service and his excellent photographic work.

Friday, August 17, 2012

AFoD Interview with Eric Zimmerman

Eric Zimmerman is one of the most amazing digital forensics people that I have run across in the last few years. He’s a combination of passion, digital forensics skill, and sharp programming abilities. He has created a whole host of digital forensics tools to help the good guys catch the bad guys. He’s a credit to the digital forensics community as well as his employer who happens to be the Federal Bureau of Investigation (FBI). As Eric points out at the end of this interview, his answers reflect his views and not the views of the FBI unless clearly stated otherwise.

Professional Biography of Eric Zimmerman

Zimmerman Eric 18 Eric Zimmerman is a Special Agent assigned to the Cyber Squad of the Salt Lake City FBI field office where he has been investigating child pornography and computer intrusions since early 2008. He is a member of the Utah ICAC and has provided training and assistance to dozens of local, state, federal and international law enforcement agencies. Eric has a degree in Computer Science and has developed several computer programs to aid in the investigation and prosecution of child exploitation matters.

Eric is an EnCase certified examiner and has several other certifications from CompTIA and SANS.

A Fistful of Dongles Blog: What was your path to the FBI? What made you decide to join?

Eric Zimmerman: My path into the FBI was somewhat out of left field. I moved to Chicago in late 1998 to work at a 3rd party logistics start up company called Con-way Integrated Services. I was the 7th employee and over the years we grew the company to about $70 million/year in revenue. Around 2005, we started the process of merging with a sister company of ours, Menlo Logistics. They were a billion dollar company and as such, their culture won. After this happened, what used to take me two days would take two weeks. Things slowed down dramatically and I was no longer able to move at a pace I was comfortable with. As the years went by, I started asking myself "Who am I benefitting being here?"  I applied to a few other places and did some interviews, but nothing panned out.

In late 2005, my brother, who has been in the Army since he graduated from high school in 1994, recommended I look into such places as the CIA and Secret Service. Neither of those agencies appealed to me for various reasons. I then took a look at the FBI in January 2006 and felt it was a much better fit. I sent in an application that same month and soon after the pieces started falling into place. In September, 2007, I was given a slot in New Agents Class 08-01. I graduated on March 5, 2008 and reported to Salt Lake City soon after that.

The reason I wanted to join the FBI was to be a part of something bigger than myself and to have the opportunity to serve our country. I had lost my passion for what I was doing in the private sector and if you do not love what you are doing, it becomes very hard to get up every morning to go to work. Now I look forward to going to work every day (well, almost every day!).

In short, I left the red tape of corporate America behind to dedicate my career to serving the public, and now, as a FBI special agent, I am able to help the most innocent among us -- our children.

I know there are a lot of other people who go to work day in and day out in the law enforcement field who have enormous responsibilities and more work than they can address effectively due to budget and time restraints. I try to think of ways to make their jobs easier and provide tools and techniques to make their work more efficient at very little to no cost.

I have been given amazing opportunities at the FBI to solve problems and design things to help a lot of people both in the United States and across the world. It hasn't been without its challenges along the way, but it has definitely been worth it and I think the results speak for themselves.

AFoD: Can you tell us what you are currently doing for the FBI and how you came into that position?

Zimmerman: I am currently assigned to the Cyber squad in Salt Lake. We are responsible for both criminal and national security investigations. I have been assigned to this squad since arriving in Salt Lake City as I am a cyber agent. By this I mean my career path is designated ‘cyber’ as opposed to counter intelligence, counter terrorism, and so on. This isn't to say we do not regularly help other squads with things as there is a computer involved at some level in just about any crime these days. My day to day work involves general case work (both criminal and national security as we do not have dedicated squads for each), some programming and the occasional bout of reverse engineering various things.

I have also been actively involved with the Utah Internet Crimes against Children (ICAC) task force since arriving in the division. A lot of the forensic programs I have written were due to my involvement with the ICAC and the FBI Innocent Images program.

Due to the success of these programs, I also spend time supporting these tools to include basic tech support, training, and so on. I provide my cell phone number and email address to everyone who uses my programs so they can contact me any time should they have an issue. I recall sitting in a presentation at the last ICAC conference in Atlanta where a vendor, in all seriousness, essentially said “If you aren't paying for software, its junk.” My guess is he was insinuating free software isn’t supported or kept up to date.

Regardless of what he meant, I feel confident the vast majority of the more than 2360 users (in more than 40 countries) of my software would disagree with that statement. What commercial vendor can you call and speak directly to the developer about a problem? I feel a great sense of responsibility to the users of my software and feel I would be doing a disservice to them and the community if I didn’t make myself available when questions or issues arise, especially when the users are in the field conducting a search and whatnot.

I also teach at several national conferences in the United States (specifically, the national ICAC conference and the Dallas Crimes against Children conference) and at a few international ones as well. I will be teaching at Europol's conference in October of this year for the first time. I am also a member of the Interpol peer to peer technical working group which is comprised of people from various law enforcement agencies all over the world who meet regularly to discuss the most efficient way to combat the exploitation of children on the Internet.

I initially started writing the various software programs out of necessity. Other Special Agents and law enforcement personnel were finding themselves in situations where there were no tools or techniques to aid in their investigations or the tools that were available did not work very well. A lot of the tools we currently have started from a simple phone call and a few hours of work. Over time, more features were added. This process has culminated in a whole suite of cutting edge tools that provide features no one else has at any price. Some of the tools are simple one offs and others have been in active development for several years. I maintain all the tools (currently I maintain 13 programs) in addition to my case load, so it can get quite hectic!

AFoD: So this gets to the heart of why I wanted to do this interview. You're very active in the digital forensics community and anyone who is on the same digital forensics email lists as you can see how knowledgeable and helpful you are. I also want to talk about your development work and the award you were recently given for this work. However, before we dig into that, we have many people who follow the blog who are interested in breaking into digital forensics and cyber investigations. We know your path into the FBI now, but can you explain how you ended up getting assigned to the Cyber squad? Is that something that you can choose to do as a condition of joining the FBI or is it up to fate and the Bureau once you have completed the academy?

Zimmerman: There is currently a list of "critical skills" that the FBI is looking for, at least in regard to the Special Agent position. Some of these include accounting, engineering, and computer science/information technology. If you have a strong background in one of these fields it tends to help move you through the process a bit quicker. My education is in mathematics and computer science, so I came in under the computer science critical skill.

When I attended the FBI Academy, I was assigned to the cyber career path. Other possibilities included counter intelligence, counter terrorism, and so on. While I had some say in my career path in that I ranked my preferences, at the end of the day someone else made the decision for me based on my background and education.

Part of that decision is also based on how many agents the FBI requires in the various career paths as well. While I was sure I would be tracked cyber due to my background and the fact that I had put cyber as my number one choice, there were some people who were tracked cyber who had little knowledge of computers and whatnot. All hope is not lost though. There is opportunity to change career paths once you graduate.

Once you are assigned a career path, FBI Headquarters will assign you to a field office. As with career paths, there is an opportunity to rank the field offices you want to go to, but at the end of the day it is the needs of the Bureau that make the final determination. In my particular case, Salt Lake City was 17th on my list (out of over 50 field offices).

After getting my orders at the academy (about a third of the way through the 21 week course), I received an email message from my soon to be Supervisory Special Agent in Salt Lake welcoming me to the cyber squad. I arrived in the division in March, 2007.

There really isn't a lot that is set in stone before you go to the academy and, without a compelling reason, your orders are your orders. Worst case you could always quit but with how hard it is to get into the FBI academy (last I heard 1 in 5000 Special Agent applicants are given a slot at the FBI academy), very few people choose to do so (at least based on the people in my class). There are always opportunities to go to a different field office as well should people want or need to do so.

For people looking to get into the FBI and work cyber matters, majoring in computer science or some other information technology discipline would certainly be a good start. Being able to program is also a huge help for solving problems we run into every day but I do not know how much that goes into the selection process.

Beyond one’s choice in a particular major, in my mind it is much more important for a person to have a passion for computers. This, more often than not, leads to quite a bit of personal time being spent improving one's skills beyond what is taught in college. Most college classes are not at the cutting edge of what is out there and so, in my experience at least, personal experience and learning often trump what is taught in school.

In short, the FBI decides your career path based upon your background and college major (you must have a bachelor’s degree or better) once you are at the FBI academy.

AFoD: So what does a FBI cyber special agent in the Salt Lake City Field office do on a day-to-day basis?

Zimmerman: Hmm, that’s a difficult one to provide a succinct answer to.

A typical day will almost always involve paperwork of some kind, from
reading email to documenting investigative activities to requesting
authority for something. Other common activities include various types of training, meetings, and operational stuff like search warrants and arrests.

The squad you are assigned to will determine how many arrests and search warrants you are involved with. I have been lucky to have been involved with the Utah ICAC so we do search warrants and/or arrests just about every week. I have been on hundreds of arrests and searches in the time I have been in Salt Lake City. Depending on the nature of the case, you may have to travel for operational needs as well. I recently executed search warrants and an arrest on an Anonymous related case which required traveling to Ohio.

As for day to day activities, I usually have a mountain of email to answer both from internal and external sources. A portion of those emails are related to support questions for the programs I have written, others are asking for direction or how to best handle a particular computer related
matter, and others are for direct support of some initiative in the
office. I also get quite a few phone calls which come at all hours of the day.

Of course sometimes there are fires to put out so those have to be dealt
with as they come up. For example, a few weeks ago, several of us were called out to a search scene to conduct interviews and that ended up consuming the entire day.

Things tend to move in cycles a lot of the time, so there are times when
I will be buried in case work for a few weeks, followed by a period
where you are basically waiting for things to come back (a subpoena
request, search warrant return, etc.) When I am in the slow cycle of
case work, I typically focus on getting some extra programming done or
start looking into a new problem area we are seeing. I like the research
and development (R&D) side of things as it constantly presents new challenges. I tend to get bored doing the same thing over and over, so the time I get to spend doing R&D counteracts the often mundane nature of paperwork.

There are a lot of opportunities to use cutting edge software and
techniques as well. Some of these were developed by the FBI and some are commercial off the shelf software packages. For example, in a recent case I had a need to examine dozens of computers on a network but had to do so in a way that limited our exposure, both physically and on the network. Because of this, walking around to various computers and hooking up hardware to it was out of the question. I ended up using F-Response to facilitate access to all the target machines. This allowed me to view any of the hard drives on the network that were of interest. This access was completely transparent to the users of the workstations. Once the drives were exposed locally, I had a huge amount of flexibility in analyzing and reviewing computers, from taking a forensic image to pulling a few files for review.

Some of the other common activity would be reviewing evidence, imaging computers, reviewing intelligence products, going to firearms to qualify every quarter, taking various training such as legal training, defensive tactics, and specific computer related training (general classes or more specialized classes which result in industry certifications, etc.)

Cyber agents have quite a bit of mandatory training which range from A+
certification to a wide range of SANS courses. The training is divided up into four stages that roughly correspond to the first five years of a cyber agent's career. Outside of the mandatory training, there is opportunity to take various elective courses such as malware analysis and so on. The malware analysis classes are very challenging courses and, like a lot of the other classes, are taught by world class instructors and very accomplished professionals. There are also ways to pursue training which isn’t a part of the official curriculum if it can be articulated that said training is directly applicable to one’s job. For example, I was able to get my EnCase Certification since I do a lot of forensic work.

With all that said, and as I mentioned at the opening of this question, there really isn’t a typical kind of day (and that’s a good thing)!

AFoD: You are making quite a mark in the digital forensics community through your research and development efforts. What can you say in public about the work that you are doing and the investigative benefits that have come from it?

Zimmerman: The response to the tools and techniques has been overwhelming. As mentioned above, several thousand users in over 40 countries have downloaded one or more of my tools. I regularly get email from law enforcement officers and examiners with success stories as a result of using my software. These stories range from successful prosecution stories to the "I never knew that was there" kind of thing. I also regularly hear from people that they couldn't do their job without some of the software I have written.

To date I have released 13 programs ranging from file parsers and hashing tools to network monitoring tools. All of the software I write is provided 100% free of charge and, in most cases, comes with extensive documentation (some of the simpler programs do not require documentation).

Most of these programs sprung from either necessity or the fact that I was not happy with the tools currently out there. A good example of this is my hashing program. In one of the training classes I was in, we were provided a Java based hashing program which handled one kind of hash algorithm and was very unintuitive to use. Over the course of a few evenings in my hotel room, I wrote a replacement for it which included many more algorithms as well as many usability improvements. Over time I kept adding features and polishing the interface. As it stands now, Hasher is the fastest and easiest to use hashing program I am aware of.

I strive to provide intuitive interfaces in my programs which are, as we say in the FBI, "Agent proof." By this I mean the software programs are easy to use and hard to break. This isn't to say people haven't found creative ways to use the programs which I never thought of! Because of this, I have invested a significant amount of resources in providing a way to automatically report errors to me so I can fix issues as quickly as possible. This process includes sending an error report with a single mouse click which includes the complete stack trace, value of all variables, and even the line numbers where the error occurred. With this kind of information, correcting issues is much easier.

Related to this is the automatic updating feature of almost all my software. Gone are the days of having to manually check for updates on a website every few months. Now, assuming you are connected to the Internet, the software will tell you if any updates are available, download the update for you, and, when you tell it to, will apply the update and restart the program for you. This has saved thousands of hours of productivity as people do not need to be trained on the particulars of how to look for and apply updates.

Hundreds of people have downloaded my parsing and hashing utilities, but the most popular piece of software I have released is osTriage. To date, thousands have downloaded osTriage and used it to greatly extend their capabilities in the field. I am not aware of any other software at any price that provides as much information as osTriage does in such an easy to understand and consistently presented format. Most of the newer features are the result of user feedback. In most cases, new features can be added in a few hours. I am currently doing about four releases of osTriage a year, but the amount of new stuff depends on the rest of my case load.

Another major benefit is the improvement from the time a search warrant is executed to the time charges are brought against someone. Previous to osTriage, investigators may have to wait months to get back results of a forensic review. With osTriage, you have most of that data available to you before you leave the scene which can be used to pursue charges. In some states, a full forensic review is not even required anymore due to how well osTriage works.

A good portion of the forensic research I have done to date is directly available to law enforcement and forensic examiners for no cost. In any of my programs which deal with various kinds of forensic artifacts, detailed manuals are included which not only explain how to use the program, but also explain the exact layout of the files the parsers are dealing with. This lets examiners validate the tool using whatever means they choose to.

Most of this material is law enforcement sensitive and as such, I cannot get into details about the specifics of my work in this kind of open interview format. I fear we may lose some readers if we went too far down in the weeds anyways!

What I can say is that most of my work involved reverse engineering proprietary, closed source binary files and network protocols. The tools I used to reverse the files and protocols include packet capture tools, a hex editor, and custom testing programs I wrote to aid in decoding various chunks of data from the raw network captures. Some of the files were more trivial than others to reverse depending on what the purpose of the file was. Reversing the wire protocol was a significant investment of time over several weeks, but the result of the work has significantly improved our investigative efforts on certain networks.

I have also taken open source data (from Twitter for example) and written code to parse that information into a usable format. While Twitter provides a vast amount of data (as the result of a search warrant for example), it is essentially unusable in the format they provide it in. Rather than spend hours trying to correlate userIDs to usernames by hand, I spent a few hours and wrote a program to do it for me. Once that was done all it took was a little time working on a nice front end for it and now any investigator can immediately start using the return data for case work instead of busy work.

This last example is a good illustration of another major area of improvement my software has provided: the ability to automate tasks that used to be very time consuming. This includes such things as deconfliction systems which are used by law enforcement around the world to coordinate efforts related to investigating the exploitation of children online. Previous to my software, deconfliction was handled manually at just about every agency and no one had a means to share this data with each other. This resulted in a lot of duplicated effort and wasted time.

Even with all the gains in efficiency and the reduction of the more tedious aspects of various tasks, by far the best thing to come from my work is the rescue of at least 200 children who were being abused in 2011. We also saw over 300 arrests in 2011 in countries across the world.

AFoD: Can you provide a list of your tools and what they do? 

Zimmerman: Sure, here is a generalized list and a brief description where applicable (in no particular order).

osTriage: Live response tool that, among other things, finds image, video, encryption, virtual machine, archive, and P2P files fast. Live response data is pulled from the registry, via WMI, and various other files. This is by far the most capable live response tool available in my opinion. While it was primarily built for investigations related to child exploitation, it can be used for any investigation involving a computer as it provides such details as every USB device ever plugged into the computer (including make, model and serial #), full browser history for all major browsers, browser search history, dozens of registry keys such as MRU, PIDL, FirstFolder, TypedPaths, and many more, extracts passwords from p2p, email, chat, and other sources, full network details, ARP cache, DNS cache, open ports, running processes, installed software, and on and on. osTriage is very customizable and allows for adding additional items of interest beyond what is included. Anything 'of interest' is highlighted in red and moved to the top for easy visibility. Supports SHA1 base32 or MD5 hashes for image and video matching. There is MUCH more osTriage is capable of, but a full description would be a dozen paragraphs alone.

Hasher: Calculate file hashes using a multithreaded, easy to use interface. I have not found a faster or easier to use tool to date. Supports SHA1 base16, SHA1 base32, MD4, MD5, eMule/eDonkey, TIGER, WHIRLPOOL, SHA-256, SHA-512, RIPEMD-256, and CRC32 hashes and can recursively process files/folders. Also allows exporting of hash results directly to osTriage supported formats, Excel, etc.

Pingaling: Pings one or more IP addresses and/or host names and plays an audible alert when an IP address/host is available. I wrote this tool for use during surveillance of a suspect using a particular IP address at a hotel. Rather than watch command windows and the output of ping, this tool lets you put in the IP, start monitoring, and do whatever else you like. When the IP comes up, you will be alerted and can respond accordingly.

WMISpy: Monitor processes via WMI locally or across the network. Allows for setting up a list of executables to watch. Watched executables have their start and stop times recorded.

Web log parser: Parses Apache access logs, Apache error logs and IIS web server log files and allows for easy analysis. Also supports SSH logs and can perform DNS lookups. Has the ability to set up keywords to search for and will then give you totals of how many times those keywords were seen. I wrote this tool when I was working an Anonymous related case which involved a lot of SQL injection attacks, so I added keywords like sqlmap and Havij and then processed the web logs. I then knew exactly where the attack took place without having to do anything but open the log files. This saved me hours of time.

Twitter Parser: Takes a Twitter search warrant return and cross references Twitter IDs to usernames, allows searching and sorting of messages, and exporting into a variety of formats. Without this tool, Twitter search warrant returns are very hard to analyze.

I have several forensic programs which parse artifacts from various programs. I cannot provide details on those as they are law enforcement sensitive.

I have also written programs to generate certain types of files, monitor networks, and provide for global deconfliction of investigative efforts.

AFoD: Who is eligible to get these tools and how would they go about getting them?

Zimmerman: Anyone working for a law enforcement agency (or a company that directly supports law enforcement) as well as the military is eligible to get the tools. With very few exceptions, my software is available to every country in the world. All of the software is available free of charge.

We also provide hands on training at various conferences such as the national ICAC conference and Dallas Crimes against Children conference as well as more specialized training such as the FBI's Innocent Images basic classes.

There are also many subject matter experts around the country for the software which can provide regional training to agencies who cannot afford to send their personnel to conferences or other remote training venues. We are also looking to do some webinars for some of my programs which will provide another low cost method to provide training to users.

While live training is often ideal, great effort has gone into the manuals in order to allow for self-paced training. In the case of osTriage, reading the manual is the minimum requirement for FBI agents to use the software.

I understand everyone is at different levels of ability and knowledge and as such it was important to me to be able to provide a wide range of training options to people so as not to prohibit the use of a tool until some type of classroom training occurred. I also realize the things learned at training are not used all day, every day after the training is over. This is another reason why detailed, precise manuals are critical as they allow users to refresh their understanding of a program without having to attend some kind of refresher course or follow up training.

As to how to get the software, the URL to access the site will be provided in training or I can provide it directly to people who are interested. Since the site is not a public site, I cannot divulge the URL here, but anyone interested can email or call me and, after vetting the requester, I will provide direction for how to access the installer which allows for downloading all the software. I am easy enough to find on the mailing lists but calling the Sale Lake City field office is another way to contact me if needed.

AFoD: Your work has been so beneficial to the law enforcement community that you were recently recognized with a major award. Can you tell us about that award?

Zimmerman: I don't think I can explain the award any better than the first
paragraph from the press release:

"Every year the National Center for Missing & Exploited Children honors
law-enforcement officers who have demonstrated exceptional efforts in
the recovery of missing children and combat of child sexual exploitation."

My particular award was under the "Law Enforcement Excellence Award
Recipients" category. I was nominated for this award by a peer who
filled out an application which was sent to NCMEC and then reviewed by
NCMEC personnel.

My wife and I were flown to Washington DC where the award ceremony took place. After the ceremony, we were given a tour of NCMEC headquarters in Alexandria, VA. It was a wonderful experience and it was an honor to even be nominated let alone selected as a recipient of the award.

The full press release from NCMEC can be found at:

AFoD: What advice would to give someone who is in high school or college and wants to break into the digital forensics field?

Zimmerman: The best advice I can think of is to have a passion for the subject matter. Without a passion for the work, it will quickly become stale and tedious. Passion is what will take you through the mundane and grinding side of forensics. You can only look at hex for so long without passion before you go crazy.

My particular background education wise is mathematics and computer
science and I (perhaps biased I realize) think those two disciplines
serve as a strong base for a career in forensics. Both require
significant analytical skills which directly translate to working in the
field of forensics.

In my experience I have found much more satisfaction in pursuing things
beyond what is found on a curriculum. Planning and getting a curriculum
approved takes time and by the time those two things happen, the
information is rarely at the forefront of the discipline. With that
said, formal classes certainly serve to provide foundational knowledge
and provide the framework for more original work.

I also see huge value in computers being a hobby outside of formal
studies. Are you interested in protecting a network? Then setting up and
maintaining a mail server, web server, DNS server, etc. on a domain you
control is a great way to get your feet wet and learn the basics.
Protecting systems and data becomes a lot more meaningful when they
belong to you! Similarly, I find it helps immensely in terms of
motivation to have some kind of personal stake in your areas of
research. For me, part of my job is combating child exploitation. It
is very hard to be motivated about something you do not care about.

Another piece of advice I would recommend is this: Do not get hung up in
the ivory tower of pure academia. What works on paper is rarely 100% possible to accomplish in the real world. There are just too many unknowns (especially in the field; on a search warrant, etc.) to not be somewhat flexible in one's approach to forensics. You will quickly find yourself alone on an island (perhaps with a few others who cannot adapt) if you do not accept the fact that it is perfectly fine to deviate from standard operating procedure if the situation calls for it AND you can justify your decisions. As long as you can articulate your position and back it up, I do not see a reason to be fearful of going outside the box.

In other words, be a forward thinker and do not be afraid to question what is accepted as a standard just because it has been done that way forever. I think huge opportunities are missed more often than we realize because of a perceived need to "stay in the lines at all costs."

A good example of this is the movement toward live response from the
"pull the plug and take it to the lab" technique which most of us have
been trained to do for so long. The argument for the latter technique is
of course to avoid making changes to a computer and so on.

In reality, things are changing on a computer regardless if the plug is
pulled or not, so why lose data that can be of value? To me, forensics
should be based on being "minimally intrusive" vs. "do not change
anything." Of course minimally intrusive means being able to show what
changes you made to a system and the easiest/safest way to be consistent is via automation, i.e. live response programs.

Finally, I would like to comment on certifications. When it comes to
certifications I have mixed feelings. Before I joined the FBI, I was burned by people who looked good on paper but in reality could do nowhere near
what their certification made it look like they could do. Competency in
any field is much less a function of being able to pass a test than it
is to have a practical working knowledge of a subject AND the ability to
apply that knowledge to a problem.

I think there is value in certifications as it can be used to demonstrate knowledge and of course some employers like to see candidates with certain letters by their name, but do not get so hung up on getting a certification that you short change yourself in the long run by simply cramming information in your head to pass a test. For those that do, it is readily apparent to others when it is time to deliver.

AFoD: Let's talk a bit about tool development. Let's also go with the same scenario again. You have someone in high school or in college who not only wants to make digital forensics their career, but they also want to follow in your footsteps and develop great tools. How would they prepare themselves to create useful digital forensics tools? Are their certain classes they should take? Are their certain programming languages they should learn?

Zimmerman: As to how to best prepare themselves to create useful tools, the first and most important thing is this: have a problem to solve. I cannot overemphasize that point. It can be anything really, but without a problem to solve, teaching oneself to code is next to impossible as the motivation will just not be there as there will always be other things vying for your time. If spending a few dozen hours learning to program can save you and others around you hundreds of hours, it becomes easy to justify learning to program. Your goal may be as grand as finding the end of the Interwebs to making a simple GUI for your kids that has a few buttons on it that tells jokes when clicked. Either way, having that goal is critical.

Some examples of typical problems in law enforcement include something like taking a flat file and converting it to XML, parsing an Apache log file looking for signs of compromise, or writing a class to parse a binary file to be displayed to an end user.

Knowing your audience is also an important aspect to being a good programmer. Anyone can learn to throw some code together, but creating powerful programs that are easy to use is a skill that takes a lot of trial and error to get right. The only way to achieve this is code, code, code! After a few years, you will look back at your first programs and laugh at what a hack you were!

In my experience, one of the biggest weaknesses in programmers in general is the belief that their users are always nice people. One need look no further than the rampant issues with SQL Injection attacks these days to see how devastating blindly trusting end users can be. What this problem boils down to is programmers not doing adequate input validation (a note to any aspiring programmers: Never, EVER concatenate strings to build SQL statements. ALWAYS use parameterized queries!) of user supplied data.

Closely related to the issue explained above is the belief that end users will use a program exactly how you, the programmer, intended it to be used. It is almost universally true that the programmer will rarely if ever make a mistake when using their own program. By this I mean the programmer knows exactly what they were thinking when writing it and therefore know the exact steps to take when using it.

However, once you give a program to someone else, all bets are off as people are very good about using a program in ways the programmer never thought of. This ranges from everything to not checking the types of data being passed into the program (string vs. integer for example) to ensuring all necessary data is available before continuing.

Finally, finding a mentor or someone to bounce ideas off of is recommended. Anyone who has built anything of significance has failed many times along the way and being able to leverage such experience can do much to improve one's skills than the cruel lessons of trial and error.

While experience is often the best teacher, finding the balance between asking questions and being pointed in the right direction by a mentor can do wonders to keep motivation high and prevent burnout.

When it comes to classes to take, almost every computer science curriculum will include several programming languages as a requirement, so people will more than likely get exposed to the basics in a few different languages. C and/or C++ will almost always be included in courses as well, especially if the class involves Unix or some BSD derivative. While I took a few C classes here and there, I never really developed anything of any significance with it because it takes a lot of work with little to show as far as GUI development goes (at least for the kind of problems I was trying to solve).

A good database class is always helpful too as you will almost always have to persist data in some way in every application. Learning how to properly create a database schema and then manage that schema is a critical skill which will save a lot of pain down the road. When it comes to databases, it is not normal to not normalize.

Taking a class on file systems is highly recommended as you will almost always be dealing with a file system when it comes to forensics. File System Forensic Analysis by Brian Carrier is a fantastic resource for jumping into file systems outside of any class that is offered in a formal curriculum.

There are a lot of excellent resources out there for things other than file systems too. Harlan Carvey's Windows Forensic Analysis is one such example that touches on a lot of key areas not only in regard to Windows, but the incident response process in general.

Personally, I bet I came close to reading one or more books like the ones mentioned above for every college class I took. More often than not, I got more out of the extracurricular reading than what was presented in class.

Finally, which programming language someone chooses to focus on really depends to some degree as to what you want to accomplish. If you plan on writing Windows programs with full blown user interfaces, either C# or would be great choices. I am a developer myself. I started using Microsoft Access and VBScript about 15 years ago and once I outgrew that, moved to VB6. Once .net came out I migrated to that platform.

Similar to the PC vs. Apple debate (the PC is clearly better of course!), there is quite the battle between C# and in some circles. C# has traditionally received new features before VB but once Visual Studio 2012 comes out, the gap between VB and C# will be nearly closed. At the end of the day, any .net language would work as it all ends up as the same intermediate language (MSIL) at some point.

If I was going to recommend a language to someone wanting to learn to program it would almost certainly be Python. Not only is it incredibly powerful, but it can be run just about everywhere and that flexibility will come in handy. I would not want to develop any kind of GUI in Python (though I am sure it’s possible, it just doesn't look polished nor does it have the powerful 3rd party components that are available on the .net platform), but for general scripting and parsing programs, Python is hard to beat. Once you learn the basic constructs of programming (loops, if then statements, functions, etc.) you can then simply learn a slightly new syntax in other languages. By choosing something as commonplace as Python you are ensuring you will be able to execute your code on a wide variety of platforms.

At the end of the day the programming language one decides to use doesn't matter as much as what the program can do for people. You can be an amazing programmer and solve difficult problems, but if your program is hard to use or doesn't provide meaningful information to end users you will find people just won’t use it.

For example, if your program pulls a 64-bit timestamp out of the registry or some binary file and reports it as such to an end user, you just alienated the vast majority of computer users out there as most won’t have a clue what to do with such a number.

Now if you are writing some kind of low level API that carves files into objects or something that is one thing (and it can be argued that you should convert it to a meaningful datetime regardless), but if you write programs with the intention of "normal" users consuming such data you have missed the boat.

As I mentioned earlier, the moral of the story is to know your audience and tailor the output to that audience. Just as with teaching a class or lecturing, if you aren’t bringing things to a level where most can understand it, you aren't being as effective as you should be.

Finally, I am a believer in agile development vs. the classic waterfall approach. Simply put, agile development means letting the actual end users of the product use the tool as it evolves vs. waiting for a program to be 'finished' and then having end users request a myriad of changes as the program doesn’t do what they envisioned it to do.

Remember, the end users are not programmers (if they were they wouldn’t need you), so they will, more often than not, lack the ability to describe exactly what they want in a way that translates to crafting a program. By putting incrementally changing versions of your program in the hands of the people who will ultimately be using the product you will, in my experience, find it much easier to come up with a product people want to use at the end of the development cycle.

My recommendation after identifying a problem is to find a group of people who understand what they want to get from a program in order to accomplish their job. For example, if your problem is developing software to track pedophiles online, find a group of law enforcement officers who are experienced enough to know what they need through all stages of the investigative process AND be able to identify the gaps they currently have to deal with. It is this group of people who should be the ones using your programs as incremental changes are made so they can steer the development along to best solve the problem that caused you to undertake writing a program in the first place.

AFoD: Any final thoughts? 

I would like to close by saying that it has been a privilege to be a part of the forensics community for the past few years. I have met many dedicated professionals who deeply care about their craft and are passionate about computers as well as highly motivated law enforcement officers who are underpaid and underappreciated. Your work impacts people far beyond what get to see every day.

Be safe out there everyone!

P.S. The opinions stated, unless clearly indicated otherwise, are my own and not that of my employer.

Thursday, August 2, 2012

AFoD Interview with Jimmy Weg

Jimmy Weg is one of the most talented people that I know in the digital forensics world. One of the things that I recommend to digital forensics examiners is that they work hard to establish a network of sharp people who they can call when they get stuck on something. There is just too much for any one person to know and it is important to have a group of people who are smarter and more talented you who you can reach out to for help. This interview with Jimmy is a fine illustration of why he is one of the first people I reach out to when I’m really in a bind and can’t figure something out.

Professional Biography of Jimmy Weg

I’m a graduate of Fairleigh Dickinson University, Rutherford, NJ.  I was a senior examiner for the National Association of Securities Dealers in New York City between 1973 and 1977.  From 1977 to 1996, I was Chief of Enforcement for the Montana Securities Department, where I headed the agency’s law enforcement efforts against securities fraud.  In 1996, I became the first Supervising Agent of the Medicaid Fraud Control Unit, Division of Criminal Investigation (DCI), Montana Department of Justice, and supervised investigations of Medicaid fraud and patient abuse.  Since November 2000, I’ve been Agent in Charge of the Computer Crime Unit of DCI.  I conduct and supervise computer forensics for state, local, and federal agencies, and I am an IACIS Certified Forensic Computer Examiner (CFCE). 

Jimmy Pic I’ve received hundreds of hours of training in computer forensics and computer crime, beginning in 1994.  My training includes courses offered by SANS, the Federal Law Enforcement Training Center, National White Collar Crime Center, International Association of Computer Investigative Specialists (IACIS), AccessData Corp., Guidance Software, Inc., New Technologies, Inc., the National Center for Missing and Exploited Children, X-Ways Software Technology.  I‘ve been an instructor on computer forensics for the Montana Law Enforcement Coordinating Committee, Montana County Attorneys Association, Northwest Association of Forensic Scientists, Project Safe Child, Carroll College, University of Montana, University of Nevada-Las Vegas, Flathead Valley Community College, and the Montana Law Enforcement Academy.   I’ve testified as an expert in computer forensics in state and federal courts throughout Montana. 

AFoD Blog: How did you get involved in digital forensics?

Jimmy Weg: I always was short on patience.  So, back in the days of the Commodore, the prospect getting information quickly, or "right now," fascinated me.  I started my career as a white collar crime investigator and headed the securities fraud law enforcement program in Montana, after working on Wall Street in regulation (I’m a native New Yorker, raised in Jersey).  I can remember tracing money and documenting financial transactions on columnar pads with a pencil.  I’ll bet that there are people on the lists who don’t what a columnar pad is, and they should be grateful for that fact.  There was no “Delete” key, and an eraser was your friend, as long as you didn’t wear out the paper.

Anyway, a company named Wang showed up with a program named 20-20 Spreadsheet.  It was amazing.  (Remember Lotus 1-2-3?)  It added, subtracted, and I could “erase” my errors and start anew in a matter of seconds.  The next thing we found, was that crooks also could use electronic paraphernalia, and what were we going to do when we got hold of a box of floppy disks?  I heard a tale that deleted files on a floppy weren’t really deleted.  That concept was the one that really piqued by curiosity.  So, in 1995, I was off to the Federal Law Enforcement Training Center for a two-week course of electronic investigations.  We learned the basics of FAT and DOS and all about the most powerful forensic tool of the day: Norton Utilities.  I still have a copy of Norton DiskEdit around somewhere.  Following the FAT and cluster chains was thrilling, I thought.  I came back to work and preformed magic tricks by recovering delete files for my coworkers, just for fun.  They were shocked, to say the least.  It’s funny, but even today, many folks don’t realize what lurks on their computers. 

AFoD: I'm not worthy. You started back in the hex editor days of digital forensics. I never cease to be impressed with the folks who were doing digital forensics in the era where there were few specialized digital forensics tools. What was the first actual specialized digital forensics program that you remember using?

WEG: My first “all-in-one” was iLook, which was law-enforcement-only back then.  Elliott Spencer was the developer and was aided by some remarkably talented people, like JB (Jim Bob Baker) and others whose names escape me.  It also was free, as major funding came from the IRS, if I recall correctly.  So, not only did I have a tool that, IMHO, rivaled the top commercial tools, but also was supported through a great online forum.  I should add that iLook still is around today in a commercial form and, from the previews that I’ve seen, it’s still quite a tool.  Many law enforcement agencies would not have been able to get into forensics, but for iLook.  Of course, it’s easy to fire up a debate among our colleagues when it comes to tools. 

I also should mention the pioneer of forensic imaging: SafeBack.  It ran from a floppy and we added drivers and the like for SCSI support and often imaged to tape.  Otherwise, imaging usually meant cloning and booting with a floppy that locked the drives.  That reminds me of Digital Intelligence, which offered a suite of very handy DOS tools, like PDBlock, which was my forensic boot disk. 

Way back then, I also recognized that I needed variety in my toolbox.  I started early with DataSniffer, as it was called at the time.  Steve Payne and Randy Becker developed it, and it was a great file carver.  It actually was quite advanced for its time, and is around today as DataLifter.  I also was one of the early users of NetAnalysis, and swear by it to this day.  Craig Wilson, the developer, is obsessed with accuracy, and his passion for his tools always has given me a great deal of comfort.

It’s fun to think back to the old days, but scary to think where we would be today without the advances that we’ve made.  Aside from the folks whom I’ve mentioned, there are a number of forensic pioneers. There are lots of great tools out there.

AFoD: I'm getting close to being in the digital forensics field for a decade now. We've seen quite a bit of development when it comes to tools and more than a little bit of competition recently between the various vendors. What are the primary tools that you find yourself using today? I'm curious, for example, what your favorite tools are for file system digital forensics work and well as more specialized work like Windows registry examination. 

WEG: Anyone who has seen my posts on this topic knows that I’m a huge fan of X-Ways Forensics (XWF).  I can’t say enough good things about it, but I will make only a few comments so that I don’t make this answer as lengthy as War and Peace.  XWF does everything that each (FTK and Encase) of the major tools does, but I liken it to FTK 4.x on steroids.  Imagine running four instances of EnCase or FTK on one machine, let alone from a thumb drive.  Imagine asking a tech support question and receiving an answer on Sunday afternoon from the guy who wrote the program.  That “guy” is Stefan Fleischmann, who is on my list of “all-time forensic gods.”  I submit that he one of very few expert authorities on file system forensics.   I use XWF on every case. 

Nevertheless, an examiner needs a toolbox that contains a variety of applications.  Since you mentioned the registry, I believe that this is one area that requires special attention.  We should be grateful to Harlan Carvey for making examiners more aware of the wealth of information that lies within the registry.  I use RegRipper on many cases, and I also use AccessData’s registry Viewer.   I think that we need to remember that case work among practitioners varies.  As a law enforcement examiner, my assignments differ from yours.  We have different needs, insofar as evidence is concerned.  In my average image/video cases, XWF’s registry viewer works superbly.  However, when I really need to explore the bowels of the registry and focus on a variety of artifacts, I call upon RegRipper.  For reporting results to my clients, I like Registry Viewer.  I also use Mark Woan’s RegExtract frequently.  

I should add that most of my testing scenarios involve the registry.  I want to see how real life actions affect the registry.  So, I also rely upon the Sysinternals (MS) suite and tools such as RegShot and RegDatXP.  I’ve found that it’s easier to test than to research an issue; it’s more reliable, too. 

I mentioned NetAnalysis before, and I use it on almost every case.  I’m becoming more accustomed to using Jad Software’s Internet Evidence Finder as well.  I have a ton of specialty programs at my reach, and I have a Start menu folder named Forensics, which contains them.  They may not be “forensic” applications, but they’re essential.  Like SamInside. 
I also want to give a plug to Mark Woan, of Woanware.  His  RegExtract and JumpLister are superb tools and are free!  I think that Windows 7 jump lists are the most valuable forensic artifacts to arise in recent years.  The accuracy of Mark’s tools is spot on, and he has implemented “wish list” requests overnight.  He is a great resource. 

I use Paul Sanderson’s tools as well.  I really like SkypeAlyzer for Skype cases.  Paul’s research, alone, is worth the price of all of his tools. 
That also brings to mind that we shouldn’t consider tools, alone, as resources.  We should be grateful to those folks who publish papers based upon many hours of research.  Aside from validation, I never use a tool unless I understand what it does and what the artifact means.  So, I do my homework, determine why I should look for a given artifact, and then look for a tool that finds it reliably. 

AFoD: You are bringing up a great point. As much as we rely on our tools in this field, it's ultimately the person behind the tools that makes the difference.  What do you think makes a good digital forensic examiner? What qualities will someone who wants to be successful in this filed possess?

WEG: A competent examiner possesses many qualities, so I’ll try to limit my list to some that I believe are essential.  The questions go hand in hand.  It’s understood that a sense of fairness and adherence to ethical standards go with the job.  I won’t recite them, but the IACIS standards are a good model for cops and private practitioners alike.  This business is not a contest to see who wins, but is a quest for facts, regardless of whom they favor.

Most everything else relates to skills.  A good foundation is an understanding of file and operating systems.  After all the years that NTFS has been around, we’re just learning now how different artifacts can be valuable in certain cases.  Just think about the different time stamps and system files that, a few years ago, were never considered.  Back then, none of the major tools presented these artifacts, at least not in a friendly fashion.  However, if you know the file/operating system, you’ll find a way explore beyond what’s presented. 

A successful examiner understands that no field changes more rapidly than computer forensics.  We must set aside time every day to learn and explore.  “Remember what the dormouse said: feed your head” (Grace Slick, loosely paraphrasing Lewis Carroll’s Alice in Wonderland).  What I mean is that a successful examiner will have a hungry mind.  What’s great about our profession is that there are so many sources to satisfy our minds’ appetites.  Just look at the forums and lists that you and I have joined!

Speaking of forums and lists, never be hesitant to ask questions.  I’ve heard anecdotes about the opposition using our questions as a sign of ignorance, yet I haven’t seen anyone cite an example where the opposition scored points with such an approach.  I suggest that, if an opponent had, the examiner or the prosecutor/counsel did not respond as well as he or she could have. 

Try to answer questions as well.  If a question pops up and you don’t know the answer offhand, do some research and offer your findings, at least if you find the question intriguing.  You’d be surprised at how much you can learn and retain by doing just that.  In other words, you don’t need to know the answer at the moment to answer the question!  A successful examiner is willing to help others because doing so pays dividends to all concerned. 
Next, never be satisfied with a tool out-of-the-box.  A successful examiner takes nothing for granted.  Validate and re-validate.  Be thorough.  Double check everything before it leaves your shop.  Don’t be satisfied with mediocrity, but be persistent so you know that you met the goals of the assignment. 

In my view, a successful examiner writes well.  I think that our clients and opponents size us up by what we produce and how we present our product.  Good reports are critical.  I’ve found that my audience enjoys reports that are concise and cater to non-techs.  I encourage everyone who puts pen to paper to get a copy of Patricia T. O’Conner’s Woe Is I: The Grammarphobe's Guide to Better English in Plain English.  I don’t like 50-page reports that include 40 pages of screen shots, most of which are “bulk.”  I rarely put them in my written reports, though I won’t fault those who use them wisely.  Unless you’re a one-person shop, let a teammate review and critique your reports.  A touch of peer review is a great asset. 
A good examiner thinks on his or her feet, yet doesn’t rush to conclusions.  Learn to speak well, too.  Most folks who enter this field will end up in court one day.  A lesson on courtroom testimony is beyond the scope of your questions, but anyone who is comfortable in addressing an audience and is a good listener will have an advantage. 

A successful and examiner is fair minded and is not afraid to change his or her opinion.  Today’s research may be proven wrong tomorrow.  Count on the unexpected.  It’s how you deal with it that can make you rise above the average. 

Lastly, before I go too far in terms of length, a successful examiner is a planner.  He or she approaches an assignment by studying the case background and setting out a plan to cover everything that may be relevant.  The examiner also will cover the basics in every case.  The job’s not over when your report leaves the shop.  Follow up and be sure that your client understands what you submit. 

AFoD: It's timely that you mention learning and writing because you are combining both of these in your new JustAskWeg blog. What are your plans for the blog and why did you decide to create it?

WEG: To be honest, it just seemed like a neat thing to do.  I've received many questions requests for advice (which I welcome) on creating virtual machines.  I already had sent out quite a few copies of some of the information on my first blog post.  I thought that videos would provide a great way to explain the process, so a blog seemed like the ideal way to reach the largest audience.  I’ve also seen folks struggle with creating VMs when they really don’t need to face any hurdles. 

I had started a long time ago with VMs, and did some testing for a fellow named Michael Penhallurick, the developer of Virtual Forensic Computing (VFC), which, IMHO, is the leading tool for automating the process of creating VMs from images.  I learned much from Michael, and some of his research may still be out there.  Nevertheless, many of my colleagues, particularly in law enforcement, can’t afford a variety of commercial tools, but also had problems with the major free, open source, VM-creating tool.  So, why not lend a hand? 

I suggest that creating VMs is one of the greatest advancements in forensics in recent years.  I use them almost on every case.  In many instances, it’s the quickest, fail-safe way to check configuration settings of applications.  You can run programs in their native environment and even hook up a network adapter and do some testing.  In that respect, readers need not go beyond my first blog post. 

As I go forward, I’ll use a generic, Windows 7 VM to examine shadow volumes.  It’ll be a VM that includes X-Ways Forensics and a few other tools, and the reader can add anything he or she desires.  I call it the “SEAT” workstation: Shadow Examination and Analysis Toolkit .  It’s nothing technically special, but affords an easy way to examine shadow volumes, which should be around in Windows 8, too. 
My blog plans are short term.  Getting through the shadow volumes is going to take a little time, which is a scarce commodity for me.  Looking ahead further, I hope to do some tutorials on forensic approaches to issues, principally with X-Ways Forensics.  However, I’m no competition for Ted Smith, who’s published some remarkable tutorials on X-Ways Forensics.

At the moment, I have no plans to write a regular forensics journal of a nature like yours and Harlan Carvey’s.  You guys (and several others) do an incredible job of putting out a wealth of information.  I simply don’t have the time, or at least I don’t manage my time as well as I could.  Then again, if I think of useful topics about which I can write a few paragraphs, I may post more often. 

AFoD: For the benefit of people reading this who might not be up to speed on the topic, can you explain what shadow volumes are and why they are important?

WEG: I’ll try to answer with a concise, not-too-technical explanation, as there is a lot of stuff out there on shadow volumes and I don’t want to suggest technical completeness and be technically imprecise.  So, let’s start with the statement that a shadow volumes (SV) is a creature of Windows Vista that has carried over into Windows 7 and, from what I understand the moment, will appear in Windows 8.  Simply put, SVs are time machines that the system creates periodically based on elapsed time or events.

The Volume Shadow Copy Service (VSS) is a process that makes backup copies (snapshots) of files and folders on a volume at specific points in time.  It’s used by Windows System Restore, which allows a user to undo changes made to the operating system and recover from system failures.  The System Restore feature automatically creates “restore points,” which users can employ to revert to a previous time.  Restore points are created at the time of significant system events (certain program installations) as well as periodically or they may be created manually.  VSS also supports a feature known as Previous Versions.  Certain editions of Vista and Windows 7 allow a user to restore previous versions of specific files with that feature.

VSS allows an individualized restoration by creating block level, differential backups of files.  In simple terms, changes to a file are recorded, and the file can be restored to how it existed at a chosen point in time by assembling the original file plus subsequent changes.  Using System Restore will recover the objects that System Restore includes.  System Restore does not necessarily include every file that was backed up by VSS, so it may not recover a previous version of a certain file.   

Although System Restore and Previous Versions derive their contents from the same snapshot data, each feature provides only the recovery of data specific to the feature, and the data differ from one feature to the other.  As restore points are created, files are backed up to those points in time.  A file that has been deleted irrecoverably today may be available within VSS.  VSS is akin to a “time machine,” which allows a user to travel back in time to visit a file, as it existed previously. 

A case in point: Joe routinely downloads videos through peer-to-peer (P2P) file sharing programs and deletes them periodically.  Moreover, Joe deletes each file with a robust file wiper.  So, Joe download files on Friday, and a SV is created on Saturday.  Joe wipes the files on Sunday.  On the following Thursday, the cops arrive and seize Joe’s machine.  Although Joe’s day is ruined, he feels secure in the thought that he’s wiped all of his videos. 

On Friday, Mary Examiner observes no videos in the current P2P downloads folder.  She tries to carve videos, without success.  Mary knows that videos can be difficult to carve, as they’re large files, possibly fragmented, and more likely to be overwritten, at least in part.  However, Mary’s aware of the power of SVs.  Using her method of choice, Mary accesses the SV that the system created last Saturday.  There, she finds all of the videos that Joe wiped on Sunday.  The files are intact, with all of their metadata.  Of course, Mary also can study link files, the registry, and other artifacts, within or without the SV, to determine whether the videos were viewed. 

One more case: After he’s released from prison, Joe buys another system and returns to his former ways.  This time, he keeps his laptop in a box buried in the back yard.  However, after tripping over the dog one chilly evening while retrieving his stash, Joe decides that there must be a better way.  His former roommate at Butner had mentioned a program named TrueCrypt.  So, Joe gets the app and creates an encrypted container.  This is pretty slick, Joe thinks.  “Just let ‘em try to find my videos now,” he exclaims.  However, the cops come back the following week.  I’ll cut to the chase.  Many of the videos that existed on the system before TrueCrypt was used are in the SVs. 

The possibilities are endless. But, so is the amount of data now presented to an examiner.  A SV is just that; it’s a volume.  The number of SVs can be quite large, depending on a few factors, and I’ve seen more that 30 on one system.  Do you want to image 30 volumes?  One case turns into 30.  Fear not.  There are a few approaches to manage the load, ad my blog will review some. 

AFoD: So you mentioned that Mary should use her method of choice for accessing the system volume. What are the various options available to her? Has anyone created any tools that make life easier on examiners doing system volume work?

WEG: I have my current method posted on my blog.  I simply have a base VM and add the target system (virtual disk) to my VM.  Then, I’ll use X-Ways Forensics and Dan Mares’ VSS to examine the SVs.  You also can boot the target system VM and run tools like X-Ways and VSS from a thumb drive directly in the target VM. 

If you don’t have VMware, I understand that there are similar virtualization tools that may work.  I haven’t used them, so I can’t comment on their effectiveness.  To be honest, I think that every examiner needs a virtualization tool.  One always can clone the image to a disk and attach the disk to a Win 7 box.  To me, that takes too much time.  I also believe that EnCase has as native virtualization add-on or the like that works, but I’m not an EnCase user. 

ProDiscover from Technology Pathways does a nice job of mounting SVs from an image added to the case.  Chris Brown kindly gave me a temporary license to work with the tool and propose suggestions, and I think it’s coming along nicely.  A user can choose to mount any or all SVs on a system and then examine them within ProDiscover. 

There are some Linux mounters out there, which I haven’t tried.  Of course, the point is to examine SVs after mounting them, and you could do so in the SANS SIFT Workstation.  I’m a huge fan of SANS and Rob Lee. 
There’s also a tool named Shadow Scanner, which is produced by EKLsoftware.   I did a little testing of it, and it works quite well.  It’s designed to do an object comparison among SVs, and I think that’s the primary focus of a SV exam.  I should reiterate that “focus” is critical in SV exams.  We’re (or I’m) not going to image 30 volumes that exist on one system.  I also hope to do a blog post using Shadow Scanner, as my approach is a little different from the method that the authors describe on their site.  The site has some great videos, and it’s worth a look. 

There probably are other tools out there, but the above tools or methods are what come to mind.  I also should add that I don’t examine shadow volumes in every case.  It’s a matter of considering the nature of your case and what’s important to all concerned.  Then you can judge the likelihood of evidence existing in the SVs.  Moreover, you should review the timing of SVs.  You may find several that were created within a short period.  Consider whether you should study all of them. 

AFoD: Thank you for being so generous with your time and knowledge. Is there anything else that you'd like to share with the readers as we conclude this interview?

WEG: Actually, I don’t have much to add, which speaks well for the thoroughness of your interview.  I’ll re-emphasize that learning and sharing should be part of our everyday routines.  There is way too much information out there for anyone to master completely.  Fortunately, many practitioners out there know almost everything about a few subjects and share their knowledge.  Note the word “almost.”  I point out that word because even super-forensicators ask questions now and then.  Perhaps that’s part of the reason why they’ve achieved such success. 

Nevertheless, I also think that we should recognize that many of us have more to do than ever before, and resources, at least in government, aren’t expanding as quickly as our caseloads.  I understand why some examiners simply have to be “consumers” of list and forum posts, and that’s okay.  Read what you can, and save what’s important.  I use a little app that pops up with every (100+ per day) email/post that I receive and displays the subject and first few lines.  If it’s something that interests me, I go to the message right away.  I have an extensive library of list posts (in a PST) that go back as far as 2000.  I search it often. 

I think the future holds great change in the way we approach our jobs.  I see a need for more specialization in the systems and devices that come through our doors.  Today, a significant number of list posts concern smartphones.  What a difference from a couple of years ago!  A fellow from the RIAA mentioned that more children own cell phones than books, and that includes the entire world.  Didn’t the Encyclopedia Britannica go out of print recently? 

When all is said and done, after a long day at work, I look forward to heading home and having a pint of a local microbrew and dinner with my wife, Kelly, who’s quite tolerant of my schedule and work habits.  I also treasure my nightly conversation with my daughter, Kristen, who lives in Las Vegas, and checking on my one-year-old granddaughter, Zoey, who is the newest apple of my eye.  In closing, I’ll share what I told Kristen years ago.  Kindness is one of the greatest human attributes.  Use it generously.  Thanks for taking the time to consider my thoughts.