Saturday, August 3, 2019

The Application Era of Digital Forensics

I started in digital forensics in 2002 which seems like a lifetime ago when I look at the digital forensics world today compared to back then.  One of the primary changes is that the center of gravity for host-based digital forensic examination has moved from file systems to applications.  There has been quite a bit of discussion about this trend over the years so it’s not a unique observation on my part, but I’ve certainly watched in wonder as digital forensics methods and tools have changed to adapt to this reality.

Around the turn of the century, the primary focuses of host-based digital forensic examinations tended to be web browser artifacts, document and photo metadata, file system artifacts, operating system log files, and email parsing.  Forensic examinations weren’t exclusively focused on these areas, but most digital forensics exams would involve at least some of them as their core components.  If you had proficiency in these areas, there was an excellent chance that you were a competent digital forensics examiner.   In my mind, the key differentiator at that time was whether an examiner understood file systems well enough where they could understand and articulate concepts such as how date and file data manifested itself inside of a master file table.  If I interviewed someone for a job and they told me that there were only four date stamps inside of NTFS master file table record, I knew they were likely just someone who drove a digital forensics tool and didn’t really understand much more beyond that.

As I look back on it now, the relatively lack of diversity in digital forensic tools at that time makes more sense to me. It was very common for a digital forensics shop to use EnCase as their primary digital forensic tools along with FTK as their indexing/email forensics tool.  FTK was also nice to have around for a second opinion on what EnCase was telling you in regards to the file system. Both tools did web parsing reasonably well, but many of us used the very fine NetAnalysis tool for web browser history forensics.  We didn’t have a vast diversity of tools because there just wasn’t the business case for them. Our existing tools generally did what we needed them to do.

If someone were writing the history of digital forensics, the advent of Magnet Forensics Internet Evidence Finder (IEF) would likely show up at the start of a chapter talking about the switch of focus from file systems to applications.  IEF became very popular, very quickly because it was designed with a focus on parsing application information whether that application was part of the operating system such as a native web browser or whether it was a third-party application such as a third-party web browser, chat program, email client, P2P client, and the like.  Of course, it also did a great job parsing a whole host of operating system artifacts just like the other tools did, but the long-term secret sauce of IEF was that it was focused on application artifacts in an era when apps were becoming the primary focus of consumer technology use and spending.

IEF was a great product and, as it turns out, the sign of a great business strategy in the making.  The computing world was moving from one where user activity was focused on things such as web browsers and office applications to one where users were using an amazing diversity of applications primarily on their mobile devices.  It’s not that we gave up using web browsers and office applications, but they were just part of the greater mix of applications being used.  Just take the Apple store as an example.  It opened up in 2008 with about 500 applications available on it and by 2017 it had over two million applications.  

This rise in applications drove changes in the development of digital forensic tools. The companies that have focused their development work on applications have generally done very well for themselves.  Magnet is an obvious example since it went from a one-person shop to a global digital forensics company by riding this wave.  Other examples are MSAB, Cellebrite, and Oxygen Forensics who have done very well for themselves by also capitalizing on this trend and creating products to address it. 

It’s not that the core digital forensics skills such as file system forensic analysis are obsolete. Far from it.  You still have to load yourself up on materials such as the magnificent Harlan Carvey books especially if you are investigating network intrusion cases in an enterprise computing environment, but now you also have to understand how operate as a digital forensics examiner in this application-centric mobile device era.

What do I mean by this?  We’re in this new era of digital forensics where examiners are going to have to get comfortable with being even less able to rely on their forensic tools for support than before.  You can’t rely on your commercial tools any more than you did in the past file system focused era.  In that era, the name of the game was going beyond your tools and understanding the underlying technology well enough so that you could validate the output from your tools.  That’s still the case now.  In this new era, the commercial forensic tool developers will use their finite development capacity to support only the applications that are broadly used and whose support will drive people to purchase their tools. Because of this, the digital forensics community will need to provide their own support for applications that aren’t covered by commercial digital forensic tools. 

What will this support look like? It’s going to come in the form of enterprising digital forensics people using their knowledge of parsing technologies such as JSON and SQLlite to create their own tools and scripts to investigate these artifacts.  In some cases, this will take the form of creating scripts that leverage existing digital forensic tools or creating tools/scripts that work as standalone solutions.  As always, we’ll still need to have the ability to double check what these tools are telling us so that we can validate the results. I use the term “have the ability” because I understand that an individual examiner can’t be expected to know everything such as having the ability to comfortably parse a particular Python script or JSON artifact.  The ability to do these things might take the form of being able to leverage someone you know to do this work for you.  Digital forensics is a team sport and one of the most important tools in your inventory is a list of people who are willing to help you out when you are in a bind.

Cryptocurrency wallets are a good example of all of this.  Cryptocurrencies are the primary payment system of the online underground economy and wallets are applications that allow people to interact with cryptocurrency blockchains so they can send and receive transactions. There are just too many wallets for the digital forensics companies to provide support for all of them and it’s going to take the community creating tools to parse them to provide the necessary support for cryptocurrency related examinations.  This is not an original point on my part since Jad Siliba made this observation about cryptocurrency forensics at the 2019 Magnet User Conference (MUS2019) this year and those comments have stuck with me ever since. 

Various and Sundry

I’m still trying to get back into a monthly blogging tempo and I have another AFoD interview in the works as I write this.  I’m coming towards the end of a very heavy conference presentation schedule this year that started about the time of MUS2019 and will end for me at the end of the upcoming Dallas Crimes Against Children Conference.  Thanks to everyone who came out so see me speak on Business Email Compromise and Virtual Currency Investigations over the past four months or so.  I’ve enjoyed getting to meet so many new people as well as to finally meet some people who I had only known electronically through the years.   I’ll still be doing some presentations in 2019, but I think the heavy digital forensics conference season is pretty much done for me and most of the people I know who do speaking on this circuit.  I’m going to be working on developing some new talks for the 2020 season.

I basically wrote this blog post in my head while at the MUS2019 conference after listening to people like Jad and attending some great talks.  For example, Alex Brigoni did a magnificent talk called “Unsupported Apps. What Can Be Done? A Methodological Approach to Mobile App Forensics” that covered must how digital forensics people should be approaching this new application-centric era.  He’s a razor sharp fellow and you can find him over at his blog and on the Twitters.  You should also read an interview he did on the Magnet blog regarding application forensics.

Thursday, March 14, 2019

The End of the Golden Age of Incident Response Billing

If you squint, you can see the beginning of the end of the golden age of incident response billing. I’ve seen this movie before and I know how it ends because I lived through the golden age of eDiscovery billing.  Incident response will no more go away than litigation requiring the production and review of electronic documents, but the current billing gold rush won’t continue indefinitely.

I left law enforcement and entered the private sector around the time electronic discovery was really gaining steam and interest in the legal world.  This resulted in legions of eDiscovery consulting outfits of various sizes and abilities getting into the game and charging confiscatory prices for their work.  The billing was such during this period where it took nothing for litigation to result in some eDiscovery consulting outfit making six or seven figure sums for their work.  Law firms and their clients eventually rebelled against being ridden like ponies off into the sunset by the eDiscovery industry and started to bring as much of the work in-house as they could get away with to avoid expensive outsourcing. Electronic discovery cost containment became a very important buzzword in the legal world. 

The gold rush also brought in more competition and interest from giant consulting firms who could offer competitive pricing and performance because of their economies of scale and ability to invest in technology and utilize their existing infrastructure. This resulted in quite a few small to medium sized eDiscovery firms being bought up, merging with other firms, or just going out of business entirely.  It wasn’t that eDiscovery went away or that it suddenly became inexpensive, but the market eventually worked things out where the larger and more efficient firms could offer better speed, cost, and quality to the legal world and their customers.

We’re going to see something very similar in the incident response world. We’re still very much in the information security version of WWII’s Happy Time where the field of battle still greatly benefits the attacker.  That isn’t changing anytime soon and maybe it never will change.  I wrote about this information security happy time in 2011 and very little has changed since then.  We just have to look at the headlines to see the near constant reports of major breaches in all sectors of business and government. These successes are going to continue to result in high demand for incident response services and these services are not cheap.  Many a fortune has been made in recent years by sharp people who set up incident response consulting practices and billed themselves into a king’s ransom. The costs associated with a breach can be immense due to the costs of the technical response itself, resulting litigation, paying for identity theft protection if personally identifying data was involved, and everything else associated with recovering from a breach including potentially rebuilding all or some of the impacted organizations information technology infrastructure.

These costs have created a growing cyber insurance market where organizations are making cyber insurance part of their risk management process and basically paying the insurance companies to help shoulder the risk for them.  The key rule to understand in an arrangement like this is the age old one that says that “He who pays the piper calls the tune.”  When a breach happens, the insurance companies will be the ones dictating the response since they are the ones shouldering the cost. These firms will have already entered into agreements with trusted incident response providers to provide their services at pre-determined billing rates.  The insurance companies will be driving cost containment in this area because their financial health will depend on it.  This will put an end to the current golden age of incident response billing which will put downward pressure on the profits of organization providing incident response capabilities and the salaries of those who work in those organizations. I expect that we’ll see similar consolidation on the industry where it will be hard for smaller incident response firms to survive unless they develop practices based on providing affordable response services to smaller entitles that might not have insurance and the resources to pay expensive incident response fees. That said, there will still be plenty of money to be made in this area and it’s still going to be a great industry to be in if you are interested in developing the incident response skills that will be in demand for a very long time to come. 

In the short term, the gold rush is going to continue because the insurance market is still developing in this area.  The sun will start to set in the medium term as the insurance industry becomes more mature in this area and an increasing amount of breach victims are covered under some form of cyber insurance.  I think we’ll also see legislation helping drive some of the cost containment where organizations that take certain proactive steps such as being compliant with some information security standard or another will have their liability capped and that will also help drive costs down.  In the long term, stick a fork in the golden rush that is the current incident response market. It will be done.

Sunday, January 20, 2019


Image by geralt via Pixabay
I think the last AFoD blog post in the Life After Law Enforcement series will end up being one talking about the interview process. It’s the logical next step after the resume post and it’s also the one that I’ve been dreading and dragging my feet in doing.  The resume post was enlightened self-interest in that I can help others out by teaching them what a proper resume should look like, but it also helps my teams and me getting better written and more informative resumes for our future open job postings. Win-win.  The interview blog post will go against my self-interest because I’ll go into some of the things that can torpedo a candidate and ways to deal with at least some of the more common interview questions that can trip people up. In other words, it’s me giving out the answers to some of my test questions, but it’s necessary stuff to know if you are plotting that move into the private sector so I’ll have it out at some point soon.

The good news is that Richard Bejtlich’s recent and excellent burnout blog post gives me an excuse to write about something else and procrastinate on the interview blog post just a bit longer.  The first thing you should do is go and read his blog post.  I’ll wait. 

So, now that you are back, I can tell you that burnout is a very real thing in the incident response world.  My first case of career burnout actually happened very early in my career when I was in law enforcement.  It’s a good laugh line that I use early in my conference presentations when I tell people that I used to be a police officer until I got tired of living the Jerry Springer show. However, it’s also an illustration of an early personal and career failure on my part. I had all of the training in the world on how to deal with law enforcement stress and burnout, but I didn’t use those tools.  Throw in some early medical problems dealing with chronic pain issues and I ended up making a decision that I ultimately regretted which was to leave law enforcement and enter the private sector way too early in my career. It’s not that I haven’t enjoyed many aspects of in the private sector and I’ve certainly made more money than I ever would have had in the law enforcement world, but chasing money isn’t worth it and there would have been plenty of time to do that after a longer law enforcement career. 

I’ve experienced a couple different versions of burnout in my private sector career from different sources.  The second case of burnout was one that many people in the digital forensics world experience which is simply too many hours and too much travel.  I started my post-law enforcement career with a small digital forensics and electronic discovery startup.  It was a fantastic experience and I’m glad I did it. I still have an immense amount of respect for the people who started and ran that business. They put their capital (and sanity) on the line to build a business and made a great run out of it. Since it was a new startup business with a lean staff, it required an immense number of hours on the part of everyone. Once it got to the point where I was traveling 100 percent of the time, I had reached the burnout point and was ready for a change particularly since I was newly engaged to be married.

The third case of burnout was just doing one too many major incident responses. I left the consulting world and entered the world of enterprise high-technology investigations. I don’t regret that aspect of my career path and I leaned an immense amount from my time in massive corporate environments.  I built and led some teams made up of some of the finest people I’ve ever met, but the tempo and politics of giant corporations eventually wore me down to the point where I was ready for a change.  In my case, I was presented with the opportunity to leave the digital forensics and incident response (DFIR) world and enter the world of fraud investigation.  It was an easy call to make at the time because of being burned out on DFIR and the opportunity to learn about how banking, payment systems, and fraud actually worked.  I’ll admit to missing DFIR work after I spent a couple years away and wondering whether I had made a mistake or not more than a few times along the way, but it ultimately all worked out well. What I ended up with was this weird unicorn skill set where I can have the ability to build and lead teams and projects that involve both the finance, cybercrime, and information security investigation worlds.  It’s not that I recommend becoming burned out as a tool for career diversification, but it can be the inspiration to change your path and end up with a better result than if you had stayed on your current path.

So after about a half century being alive, I’ve learned that burnout is a thing and it’s important to be able to manage the stress to avoid burnout.  If you do find yourself burned out, there isn’t anything wrong with making a change and finding different pastures, but it’s best to make those changes when it’s not a response to getting the point of being burned out in the first place.  One of the best hedges against this is having activities outside of professional life that help manage stress and give you an opportunity to do something meaningful outside of career life. 

Richard recommends doing something physical outside of your career and I think that’s a fine idea.  In my case, I picked up practical shooting several years ago and it’s a nice way to get some physical activity when I shoot matches periodically throughout the month or when I’m doing dry fire practice at home.  There is very little digital involved with practical shooting beyond the scoring technology.  Pistols are basically springs, levers, and chemistry coming together in a small package that is deceptively hard to excel at when the buzzer goes off. I shoot better and understand firearms much better now than I ever did when I was a police officer. 

I’ve also picked up amateur ornithology now that I’ve found myself living in Florida.  Birds are a testament to the wonder of God’s creation and we’ve got a bumper crop of different types of bird species here in Florida alone. I enjoying going to our local beaches and see what our various birds and sea life are up to and I’ll post pictures from time to time on my Twitter feed if I get a good picture of video of some seabird doing something interesting.

The biggest thing that made a difference my life is my relationship with Christ.  The best stress relief I’ll ever get is attending various services and events at my church.*  I grew up and attended various churches throughout the years, but it was only in recent years when I really started to take my faith seriously and that’s been the most lifechanging hedge against stress and burnout that I’ve ever experienced. It’s allowed me to see my career as something I do rather than something than I am.  One of my favorite Bible verses is Philippians 1:21 (I even have it on my Twitter profile) which says that “For to me, to live is Christ and to die is gain”.  I even have that verse listed on both sleeves of the shooting shirt that I wear to major shooting matches to remind me what the purpose of my life is sanctification (an ongoing struggle since I’m really good at sinning early and often just like everyone else), following His example, and sharing the transforming gospel of Jesus Christ through my words and deeds. 

(*Unfortunately, even though I’m a member of a church centered on Calvinist doctrine, I don’t have a proper Calvinist beard. My facial hair just won’t cooperate to get a proper beard, sadly.)