Sunday, April 25, 2010

The Ballad of Grayson Lenik

Grayson Lenik is a relatively new member of our community who has made the decision to move from a systems administration focus to a digital forensics focus. You can follow his journey at his blog "An Eye on Forensics".

Grayson is clearly a sharp fellow. From what I can tell, he passed the SANS GCFA exam via the challenge process rather than taking any of the SANS course content. That's impressive considering the scope and difficultly of that exam. Grayson has been encouraged to contribute to the recently started Into The Boxes digital forensic online magazine and, to his credit, he's accepting the challenge and looking for a topic to research.

His comments on the research issue made me think of my decision making process regarding engaging digital forensics research. I've been doing digital forensics for a relatively long time now, but it was only last year when I decided that I'd start to contribute to the community in a meaningful manner in this area.

The reasons why I hadn't done so were largely due to intimidation. I look up to people such as Harlan, Rob, Jesse and Eoghan (otherwise known as the people who don't need last names to know who they are) and the work that they have done advancing the field with their research, training and tool development efforts. Who was I to even think that I could play on the same field as them? I also fell into a common trap that I see with IT security people which is that because I didn't know everything, I thought that I didn't know anything.

Last year I stumbled across Adobe Flash Cookies while doing an examination and started to dig into them. I began to learn that some of these cookies can provide a treasure trove of information for a digital forensic examination and started to parse them out as well as I could. I made a couple phone calls to some very experienced examiners and asked them if they had heard of them before and was told that they had not. One of those examiners was actually able to take what I told them over the phone and put it to use in a criminal investigation they were using so I knew I had something that would be beneficial to the community.

So I decided to just plow ahead and start writing something up with the goal having something to present at a conference like CEIC. I started to create an early overview paper.
I was lucky enough to have people like Cindy Murphy, Gary Kessler, Jimmy Weg and Mark Johnson review that paper and make suggestions on how to improve it. Cindy even managed to carve out some time from her busy schedule to do some additional research in regards to a particular kind of cookies that really helped fill out my knowledge. I briefly distributed the paper through some of the email lists like IACIS and HTCC hoping that it might get the word out and generate some additional research leads.

I sent it out to the community and heard....nothing much. I later learned this is a pretty common occurrence in our community even for Those-Who-Only-Need-A-First-Name. A digital forensics researcher will put a lot of work and effort into a project, release it out for free and ask for feedback...and will rarely get any back. I would get people thanking me for providing them the paper after I sent it to them, but then no response back to my requests for feedback on whether it was useful, whether they found any errors, how I could improve the final product, etc.

One of the notable exceptions to this which was Jesse Kornblum. Some time after I had released the paper, I checked my email to see a request from Jesse for the paper. It was a classic good news\bad news situation. The good news was that Jesse Kornblum wanted to see the paper. The bad news was that Jesse Kornblum wanted to see the paper. I'll admit a certain amount of dread when I hit the send button. The short version of the story is that Jesse liked what I had done. He offered encouragement and suggestions on how to proceed. Very cool!!!

So bolstered with my new found confidence, I pressed forwards with the research project and hit a major sticking point when I encountered some very odd metadata behavior that I absolutely could not figure out. I was saved by Eoghan Casey who helped me determine that the odd behavior I was seeing was due to File System Tunneling (which I will explain at my CEIC presentation next month). Yet another of my forensic idols riding to the rescue!

Around January or so, however, I was starting to realize that I was over my head. I able to parse out the header information for these artifacts, but I didn't have the knowledge to completely parse everything out. My hex-fu was okay, but it wasn't good enough to completely finish the project the way I wanted to complete it. The way I saw it was that I could either crawl back into my hole and admit defeat or just publish what I learned so far and hope that someone else could run with the research at a later date. I decided to do the second option with an eye on getting what I had completed published in some form.

Then on Feb 17th, 2010, I got lucky. Kristinn Gudjonsson posted some of his Adobe Flash Cookie research on the SANS Forensic blog. My initial reaction was that I had been too slow, too unknowledgeable and had been just wasted months of my research life because what he had done was so fantastic that it was better than I could have ever done. I even found that I had made at least one major error in my original header research. Woe is me, right? However, when I started to look closer, I realized that we had approached the research from different standpoints. Kristinn is an amazingly sharp incident responder and forensic examiner with an engineering background. That means he spent a lot of time looking at the hex level view of these cookies and did an exceptional job parsing them out. I approached the research from a more traditional investigative digital forensics perspective which means I concentrated on the metadata (which is why I discovered and overcame the file tunneling issue) and a lot of the higher level aspects of the research such as how and when Flash cookies tended to appear on a machine. I became excited about the prospect of merging the research, but would someone like Kristinn be willing to talk to little old me? (There's that self doubt again...)

As you know from my previous blog entries, yes, he was more than willing to talk and after a flurry of emails comparing our various notes on the project, we decided it made good sense to team up and create a final research project.

The moral of the story?

1. Be like Grayson Lenik, not Eric Huber. Grayson has been a member of our community literally only for a matter of months and he's already sharing what he's learning through his educational process and he's going to do a research project for ITB. It took me years before I decided to do what Grayson is doing now.

2. Research what you know and if you get stuck, get help and continue on. There is a vast amount of research opportunities in digital forensics for all skill levels. Harlan wrote a particularly pithy bit of advice for Grayson when he said "...start writing about what you know...we'll work with you." That's essentially what I have been doing. I plow through the best I can within the range of my abilities and if I get stuck, I go ask for help. Grayson will do great because he's a sharp fellow who has the desire to do the work and he'll have people like Harlan and Don Weber to help him when he needs it. What I've found is that the gurus like Harlan and Don are very helpful if you approach them in the right way.

3. If you don't have time to complete a project, even partial research is helpful and someone else might take what you have done and run with it. I did that with my Kindle forensic research. I knew I wasn't going to have the time and probably the knowledge to completely parse every aspect of what one can find on a Kindle so I posted what I learned on this blog.

4. Provide feedback. If you don't have the time or desire to do digital forensic research, no worries. However, one thing that you can do to help those who are doing it is to provide feedback when you have found something useful that helped you in your job. Did you like a particular digital forensics book? A nice thing to do would be to post review at a site like Amazon. Even negative feedback is welcome as long as it's constructive. If I made a mistake, I want to know about it. If what I wrote didn't make any sense, it doesn't help me develop as a writer or a researcher if I don't know what I'm doing wrong.


  1. Eric,

    Thanks for writing this and for your research on Flash cookies, two great contributions to the community.

    I've found even the bad research I've done has been beneficial for both myself and others I've shared it with. We learn from success and from failure.

    The community is like stone soup, I suppose. If everyone brings something to the table, we'll all benefit. The field is huge and there are many unknowns just waiting for someone with the time and inclination to tackle them.

  2. Thanks for the kind words, Dave. I agree. There's something for everyone out there who is interested in doing digital forensics research. Even if it turns out you can only partially complete a project, it's still something that you can put out someplace like a blog for someone else to use and expand on.

    For those of you who don't know Dave, he's part of the SANS Digital Forensic family ( and has contributed quite a bit himself ( to the community. Thanks, Dave!

  3. Eric, Thanks for sharing you experience. I know there are many of us out there that have been here a while but not contributed back as much as we could have. Great advice and keep up the great work.


  4. Great article Eric!


  5. Eric,
    Thanks for some great material. Like you, I discovered Grayson early and have been following his work. He definitely is an interesting person, and I too expect great things from him in the near future.

    Thanks for your reminder regarding thinking you don't know anything cuz you don't everything. Great encouragement.

    There was so much to appreciate about your post. Aside from the technical aspects, I like writing style, humility, and humor. Many can be techies, but few have the skill and guts to admit feelings of doubt and fear. Even fewer admit their mistakes. You're obviously a whole person.

    And especially for technical people, that's refreshing.

    I want to be like Grayson AND Eric Huber!

  6. Thanks for the kind words. I consider myself a graduate student of digital forensics. I know I'm not at the guru level, but I have come to realize that I have some kung fu that I can share with a community that has been very good to me over the course of my digital forensic career.

    What I have discovered is that the more that I share, the more that I'm getting back in return by making new relationships and learning new things that I don't think I would have discovered if I had just decided to be a consumer of other people's work rather than trying to produce some of my own.