DHS, the government, and "cyber" security

The White House issued their "CyberDream" last month titled the "International Strategy for CyberSpace." The strategy is chock full of wonderful thoughts and ideas about what cyberspace should look like and even has a few really excellent threats leveled at people who may want to attack the US. The majority of the document seems to be a plea for other nations to share "our" vision. On one hand it would be amazing to have multiple nations sharing information and standing against those who would seeks to do harm or gain information illegally via the digital realm. On the other hand, why don't we just defend our own assets better. If I had an extra hand I might suggest a blend of the two.

Really, I doubt it will matter since the government will want to be in charge of the whole program including some of the highly technical aspects. This is evidenced by DHS being funded with over 40 million dollars and their attempts to continue to hire infosec personnel. What's wrong with that? I am so glad you asked, nothing would be wrong with that IF DHS is hiring new top-level decision makers along with the technical folks. If the same policy makers are in charge, nothing will change. They will hire bright and talented young technical folks then ignore their recommendations and lose them to the private sector. Also known as "status quo." Reading the DHS, and other government, initiatives on hiring they appear to believe that the only problem is the pay. That may be part of the problem, but the real issue I have seen is the promotion of the wrong personnel over time. The last government place I worked had dug a hole so deep and wide they didn't even know they were in a hole. Several manager level personnel possessed zero technical skill or knowledge yet were making decisions regarding technologies they didn't understand. They were also responsible for hiring but didn't know how to hire since  this field has been overrun with certifications, "policy analysts", and general confusion.

How bad is it? That used to be difficult to quantify but take a look at the major breaches that have happened in 2011. HBGary, RSA, and Lockheed are security-centric shops with deep technical talent that were pwned with techniques taught in any pen testing class you might want to take. The Oak Ridge National Lab was taken hostage and had to be disconnected from the Internet for weeks. Many of these intrusions were blamed on "APT" but they were not advanced techniques and they certainly didn't persist for very long. What really happened is the age-old theory of "acceptable risk" is being put to the test. If your entire enterprise can be jacked when one person clicks a link or opens a file something is not quite right. I have performed enough penetration tests to know that this is common and the excuses for not implementing deep, separate layers of security are many both valid and otherwise.

What's the fix? Get red teamed or pen tested using real techniques. Make sure the people performing the tests offer mitigations and remediations for the vulnerabilities they discover and exploit. HIRE THE RIGHT PEOPLE and GET RID OF THE WRONG PEOPLE. I cannot stress that enough and I know hiring is difficult. If you don't know how to hire the right people, then you shouldn't be doing the hiring. I don't mean that at all in a rude or arrogant manner, simply ask for help and the infosec community will help you. I would be glad to assist your hiring, resume review, and interview process. I watched a government site two years a go replace a talented mid-level technical cyber ninja with a junior in college who knew nothing about security. Now I am all for bringing up new folks and teaching them but you must keep good technical people and hearing the phrase "we just need to hire people who get along, not anymore smarty pants" was discouraging. I believe the government CAN do it right if they want to. Challenge the status quo, admit when mistakes are made, and persevere. Hire the right contractors to assist you. For some of us it isn't about money, it's about doing security right to protect assets as a higher calling. You do it right, because it's right.

APT = inAPTitude please pass the FUD

So, the Oak Ridge National Laboratory was absolutely pwned. How do we know it's "that bad" you ask? If they had to unplug from the Internet (read Facebook) for any amount of time, you know it's not good. I feel like we should address the immediate assigning of the breach to a determined nation state. Here's why I don't think you can call this APT now. For the record APT is defined well here: http://taosecurity.blogspot.com/2010/01/what-is-apt-and-what-does-it-want.html

1. The incident analysis is ongoing. There's no way they have had time to give full attribution already. If they could, you wouldn't need to completely disengage from the entire Internet.

2. A phishing attack with and alleged IE 0-day is not that advanced. Anyone with basic skill in Metasploit or the Social Engineering Toolkit could have done this with a 10 year old payload and completely bypassed whatever A/V they think protects them.

3. That place is wide open. For the most part it's an academic research facility full of foreign nationals several of whom are from "sensitive countries", whatever that means. One walk across the lawn and you will see enough to know what I am talking about. This is not to say that anyone form a foreign country means the US harm.

4. That place gets owned all the time. I mean just use Google and you can read about it happening every few years. Despite this the same personnel and outdated protection schemes remain instantiated, welcome to the government.

5. The lab director states there was no large-scale exfiltration detected. If you had an advanced determined attacker, they could exfiltrate without you knowing. Seriously, what ports are you allowing outbound from your users. When they SSH tunnel out to avoid your proxy do you know what data is in said tunnel?

6. Even if you get some IP address in a nation we aren't cozy with, that isn't immediate attribution to said government. That could be someone in this country using one of their IP addresses as a pass through. Jumping to conclusions during an investigation is unhealthy and may give a false sense of closure.

I was saddened to learn this happened at ORNL. It was worse to have the director of the lab jump on the bandwagon of APT before the post-mortem analysis is complete.

Smart Grid Standards Groups

One of my co-workers estimated there are approximately 46 groups working on standards for the smart grid. Above is a partial list of the folks trying to work this out. I, and others from EnerNex, regularly contribute to these groups within our own areas of expertise. It will be interesting to see how things boil down once you have to migrate from a standard to an actionable solution. So far the cooperative effort is excellent. For all those involved I offer a free Internet high five.

Mid Atlantic Collegiate Cyber Defense Competition (CCDC)

Wow! Let me say that once more WOW! Thanks to all the companies and other entities that sponsor this event as the next generation of "cyber warriors" is being educated. Boeing contacted my company, EnerNex, to see if we would be interested in assisting with certain aspects of this year's competition. This was shipped over to me since I am a penetration tester/security analyst. I was unable to help with the smart-grid scenario that had been planned for the competition and felt really bad about that. I flew up there on my birthday feeling just terrible that I had not been able to assist in any way. When I landed I sent a message to Casey O'Brien and Tim Rosenberg offering to help however they needed. Big note to self, do that more often. I was moved in the White Cell for the competition, specifically I played federal law enforcement for incident response in an effort to teach the blue cell how to submit accurate actionable information to law enforcement. Now on to how the event played.

The Teams:

Red Cell: Attackers, crackers, hackers. Their goal is to penetrate your systems, gain and keep access and wreak havoc.
Blue Cell: Defenders, their goal is to respond to current attacks and prevent future attacks. There were blue cells from different colleges and universities.
White Cell: These were the folks judging the business injects and observing the team. Additionally, certain members were designated as federal law enforcement. LE members were allowed to give limited guidance if a team was really struggling.
Gold Cell: Operations. These members were responsible for making the equipment work, scoring the game and keeping everything running. 

So what did they have to do? I am so glad you asked. Blue cells were given several nodes to defend based on a viable business scenario:

Business Scenario: Haven Electric CoOp (HEC)
Each year, the CyberWatch Mid-Atlantic CCDC presents a new exercise scenario and cutting-edge technologies that mimic those in the real world. This year’s scenario involves student teams working for the Haven Electric CoOp (HEC), a national electricity provider. With operations spread throughout the United States, HEC is a leading electric grid manager and reseller of Power Management Units (PMUs).
Because of risky investments, HEC has fallen on hard economic times and has been acquired by the U.S. government. Given the unstable future of the company, most of the IT staff has left for other jobs, while those remaining are less than effective. As a result, the government has brought in contractors to replace all the IT staff. The student teams are these contractors.
The student teams will be charged with maintaining and securing the network, while providing critical services and responding to the demands of clients, end users, upper management, and others. As employees of HEC, the students will also have access to the HEC Credit Union, where they can conduct their day-to-day banking.

Now what do they defend:

inside the firewall: 

Ms-Exchange 2003

MS Win 2008 AD server - 10,000 user accounts

Open PDC manager

Hadoop

Splunk - Ubuntu 10.x

MyBanco - Ubuntu 10.x

OpenPDC DB - Ubuntu 10.x

LibkiWikiID - Fedora 14 

outside the firewall:

Nagios

Kiosks

Splunk

Red team had a 30 minute head start so, if you haven't attempted to defend a network before, everything was already compromised by the time blue even "got to work." In addition to the aforementioned devices, each contestant wore a badge with an 802.15.4 ZigBee radio which beaconed every ten minutes with a predefined integer. The integer was power usage data so that, in effect, all players were wearing a smart meter that updated itself regularly. The meters were also in play and at the end of the first day, one blue cell member had somehow managed to use over 1 billion kilowatt hours. Larry Pesce built the badges and wrote the software that was used on them at a final price of $32 per badge, most excellent job by Larry. Some other nodes that weren't readily noticeable were two Cisco 7960 IP phones and a web-enabled surge protector which some red cell members took great delight in attacking.

At the end of day 1, there was a "corporate meeting" business inject requiring all blue cell to immediately leave the competition floor. For 10 minutes the red cell was allowed physical access to the blue pit where they wreaked havoc by taping Ethernet cables, swapping cables around, and running custom tools to add users and acquire password hashes. In ten minutes, the red cell successfully touched every blue cell node.

The days were long yet quite rewarding. I enjoyed helping the blue cells learn how to submit incident reports. Though I frustrated many of them by continually rejecting them for lack of evidence they began to learn that I needed who, what, when, how, and maybe why to give attribution to an actor(threat). They also had to learn that it isn't what they think they know, it's what they prove by providing corroborating evidence such as logs, files, and screenshots. Additionally, if a team was really struggling, I could provide hints and suggestions or in dire cases I could take the blue cell member aside and have some teaching moments as they struggled with the complexities of being assaulted not only in the cyber world, but in the business world as well.

Did you say the business world? Yes, I did. The "CEO" flew in to interview the team captains after he learned that some of his assets had been compromised. Each captain was given the "opportunity" to sit and tell the CEO the state of affairs of his network and data. Some young folks responded with poise, others literally shook in their chairs, and still others refused to have their follow-up meeting. This was also excellent training that should help close the gap I have seen where non-technical people are not getting promoted. This also gave me an opportunity to speak with some of the captains outside of "the pit" (competition floor) to explain some of the terms used by their CEO and help prep responses in his language.

This was the best training a future IT security professional can receive and I truly appreciate that EnerNex was kind enough to send me. Being able to assist in events like this gives me hope that things can get better in InfoSec. It's always a pleasure to share experience and knowledge with those who are seeking a career in this field. Many dedicated educators are attempting to do this but they need practitioners from the real world to assist and fill in the gaps. I look forward to assisting more in this area both at our local schools here and with some of the colleges and universities I interacted with at this competition. I met some great students, faculty, parents, and sponsors. I also had the wonderful privilege of working with Casey O'Brien, Tim Rosenberg, Matt McFadden, Gary Stoneburner, and many others. Please keep in touch everybody.

For the curious:

Blue Cells: http://www.midatlanticccdc.org/CCDC/students/

Sponsors: http://www.midatlanticccdc.org/CCDC/sponsors/ - we can't thank you enough

Pictures I took - https://picasaweb.google.com/griffse/MidAtlanticCCDC#

Smart Grid Security East

I had a great time at this conference and got to meet some great folks. I also had the opportunity to be the first speaker on the first day which really helps the other talks as I set the bar pretty low. My two favorite talks were Travis Goodspeed leaving the crowd in stunned silence with some of his hardware ninjary, and Ido Dubrawsky from Itron during the AMI security workshop. Ido gave a great talk that was grounded in facts which is sadly lacking sometimes. Stephen Chasko and Ed Beroset also gave good talks from a vendor perspective. For the panel-style talks, I naturally enjoyed the penetration testers over most of the policy and strategy panels. I will have to say my favorite moment was a vendor offering perhaps a tiny bit of marketing hype being asked from an audience member "Are you saying you are guaranteeing absolute security from that point forward?" Of course, the vendor was not offering that and the talk proceeded smoothly. I enjoyed that because it represented the spirit of the conference. People spoke openly and disagreed with each other with facts and perspectives without anything devolving into chaos. There was even a meter vendor panel where they seemed to be working towards common goals regarding smart-meter security. A goal going forward is to get SCADA vendors involved and provide utilities with a way to share security-related information if they are experiencing an incident.

I was also privileged to connect with several gentlemen and ladies considered by many to be the leading experts in the efforts to make the smart grid secure. I cannot list them all but literally everyone I met was making a contribution to this effort. I can't wait to see the next event, it should be even better.

The Song Remains the Same

So Stuxnet was a "game changer" because we saw a private separated network get JACKED! Let me share some of the responses I have heard:

"They shouldn't have been using Windows"
"Stuxnet was no big deal if you weren't the target"
"There are enough other people that are vulnerable, they probably won't come after us"
"We have firewalls, IDS, and AV."

These comments come from vendors, CISOs, and security architects. Hi, you are missing the point. If you focus on the specifics of the attack these are somewhat accurate statements. If you look at the framework of the attack it should make you aware that you are at risk. Some components of Stuxnet were very generic and can provide a framework for future attacks. Check out this page by Ralph Langner: http://www.controlglobal.com/articles/2011/IndustrialControllers1101.html

 Here's a question to ask your CISO or security team lead or whoever you have entrusted your security to.:

"How can our firewall (also include AV, IDS, etc) be defeated?"
"How can an attacker exfiltrate data once they are inside?"
"Can you (security d00d) exfiltrate data without anyone knowing?"

If you saw the report on Night Dragon, you saw another example of energy being targeted. The target was compromised via SQLi and the attack progressed using fairly standard simplistic techniques. No ofeense to the target is meant here, I am targeting the mentality mentioned above. These folks had firewalls, AV, proxies, and policies. Their controls were overcome at every step with what the incident responders called "simple" techniques. Simple is a relative term and the timeframe of the attack is not discussed. If this attack took place over a span of weeks it is relatively easy to recreate. If this attack was done in a matter of days or less, it was well-planned and executed. 

Critical Infrastructure Hacking FUD

Let's take a minute and talk about some of the FUD being slammed all around regarding critical systems hacking.  We are talking about the electric power system, water, and other utilities or critical infrastructure. This article came out last week: http://www.wired.com/threatlevel/2011/02/hoover/ Stating that hackers can't do weird stuff to Hoover Dam. That article is accurate. Twitter exploded the same day with infosec and pen testers screaming "yes we can!" This is also accurate. We have to temper some of the almost outlandish claims we attackers make with the "you can't touch us" claims of infrastructure. 

Why is the wired article true:

1. Separated networks - The Hoover Dam (critical infrastructure sites) are not web apps that you can just stick in a web browser.

2. Infrastructure stuff breaks all the time - These people are trained to respond to outages a lot better than the IT in some organizations.

3. Hackers aren't breaking news - Infosec incidents get published all the time and, sometimes, utilities take notice and plan for these things.

Why what the hackers are saying is true:

1. Remember Stuxnet? - Those targets were air-gapped and didn't touch the Internet.

2. Resiliency != Security - Infrastructure people will say "when was the last time your lights went out?" when the question really is "When was the last time someone wanted to make your lights go out?"

3. Hackers evolve - When people start figuring out patching, web apps and client side attacks shift to the front. When people get leery of those techniques bring on insider threats and social engineering.

You have to get both sides of the story to understand the problem. If you are using computers, networks, and software you have risks. Reducing your attack surface by using air-gapped and private networks is an effective layer of defense. That said, security is never "done." It is an ongoing issue and it must be tested continuously. Insiders cannot be trusted, sometimes this is because of bad intentions, and sometimes it is because people make mistakes. We also have instances where you have say a SCADA operator granting remote sessions and connections for service or maintenance on the system, or they figure out some way to surf the web from their console. In case you have the world's best workers who never look for a way to goof off, we have the removable media attack vectors. I will leave a nasty USB drive in your parking lot or Starbucks and watch who picks it up etc.

Does your blue team tell you they can't be breached? If so, go find a red team and let a real-world scenario play out with them, you might learn that your team is as great as they say they are. You might find that they are unaware of certain vectors into your systems. For example, let's pretend you are performing a test of a "closed system" and everything initially seems to indicate that this is true. Then you notice you can resolve DNS names like Google, but you cannot not get to the Internet via a web browser, the system isn't touching the Internet right? WRONG! Your assigned DNS servers, initially RFC 1918 addresses,  become public IP addresses when you reboot while connected to the "private" network. Out of curiosity, you try to touch those servers from your home ISP and you can. This is news to your client since they had been assured otherwise by the provider. Maybe it even said that in their SLA.

If you read the link regarding the Hoover Dam, someone who appears to be from the public affairs office is posting comments about how that cannot happen. You will see other folks asking how employees communicate and are part of the electric smart grid if they are so isolated. You cannot have it both ways. There's an example of someone touring a power generation facility and asking about security and the operator saying "We aren't connected to the Internet." The person touring asks how they receive communications and directives from their main facility which is several miles away. The operator points out that they receive e-mail on the control system machine. Now this is where perspectives will really diverge. For me, it's not the same to say you don't touch or use the Internet when you are, hopefully, using some sort of VPN tunnel. I view separate as not touching, tunneling, sharing a switch/router, or even the same network rack. SEPARATE. Don't get me wrong, I understand how extremely cost prohibitive it would be to build out your own personal WAN but it can be done. For the govie "cyber" security architects, there are a lot of good models to look at. Companies who have customers and dollars to lose take security pretty seriously.  

So can hackers open the gates of the Hoover Dam? No one has let me test it so all I can say is "maybe." The attack probably won't be attempted from some kid's basement but that doesn't mean it cannot be done. A lot of people say they aren't connected to Internet when they really are. All systems have vulnerabilities but not all vulnerabilities can be exploited with the same level of ease. Be a critical thinker and get both sides of every story.