One of my co-workers estimated there are approximately 46 groups working on standards for the smart grid. Above is a partial list of the folks trying to work this out. I, and others from EnerNex, regularly contribute to these groups within our own areas of expertise. It will be interesting to see how things boil down once you have to migrate from a standard to an actionable solution. So far the cooperative effort is excellent. For all those involved I offer a free Internet high five.
Wow! Let me say that once more WOW! Thanks to all the companies and other entities that sponsor this event as the next generation of "cyber warriors" is being educated. Boeing contacted my company, EnerNex, to see if we would be interested in assisting with certain aspects of this year's competition. This was shipped over to me since I am a penetration tester/security analyst. I was unable to help with the smart-grid scenario that had been planned for the competition and felt really bad about that. I flew up there on my birthday feeling just terrible that I had not been able to assist in any way. When I landed I sent a message to Casey O'Brien and Tim Rosenberg offering to help however they needed. Big note to self, do that more often. I was moved in the White Cell for the competition, specifically I played federal law enforcement for incident response in an effort to teach the blue cell how to submit accurate actionable information to law enforcement. Now on to how the event played.
Red Cell: Attackers, crackers, hackers. Their goal is to penetrate your systems, gain and keep access and wreak havoc.
Blue Cell: Defenders, their goal is to respond to current attacks and prevent future attacks. There were blue cells from different colleges and universities.
White Cell: These were the folks judging the business injects and observing the team. Additionally, certain members were designated as federal law enforcement. LE members were allowed to give limited guidance if a team was really struggling.
Gold Cell: Operations. These members were responsible for making the equipment work, scoring the game and keeping everything running.
So what did they have to do? I am so glad you asked. Blue cells were given several nodes to defend based on a viable business scenario:
Business Scenario: Haven Electric CoOp (HEC)
Each year, the CyberWatch Mid-Atlantic CCDC presents a new exercise scenario and cutting-edge technologies that mimic those in the real world. This year’s scenario involves student teams working for the Haven Electric CoOp (HEC), a national electricity provider. With operations spread throughout the United States, HEC is a leading electric grid manager and reseller of Power Management Units (PMUs).
Because of risky investments, HEC has fallen on hard economic times and has been acquired by the U.S. government. Given the unstable future of the company, most of the IT staff has left for other jobs, while those remaining are less than effective. As a result, the government has brought in contractors to replace all the IT staff. The student teams are these contractors.
The student teams will be charged with maintaining and securing the network, while providing critical services and responding to the demands of clients, end users, upper management, and others. As employees of HEC, the students will also have access to the HEC Credit Union, where they can conduct their day-to-day banking.
Now what do they defend:
inside the firewall:
MS Win 2008 AD server - 10,000 user accounts
Open PDC manager
Splunk - Ubuntu 10.x
MyBanco - Ubuntu 10.x
OpenPDC DB - Ubuntu 10.x
LibkiWikiID - Fedora 14
outside the firewall:
Red team had a 30 minute head start so, if you haven't attempted to defend a network before, everything was already compromised by the time blue even "got to work." In addition to the aforementioned devices, each contestant wore a badge with an 802.15.4 ZigBee radio which beaconed every ten minutes with a predefined integer. The integer was power usage data so that, in effect, all players were wearing a smart meter that updated itself regularly. The meters were also in play and at the end of the first day, one blue cell member had somehow managed to use over 1 billion kilowatt hours. Larry Pesce built the badges and wrote the software that was used on them at a final price of $32 per badge, most excellent job by Larry. Some other nodes that weren't readily noticeable were two Cisco 7960 IP phones and a web-enabled surge protector which some red cell members took great delight in attacking.
At the end of day 1, there was a "corporate meeting" business inject requiring all blue cell to immediately leave the competition floor. For 10 minutes the red cell was allowed physical access to the blue pit where they wreaked havoc by taping Ethernet cables, swapping cables around, and running custom tools to add users and acquire password hashes. In ten minutes, the red cell successfully touched every blue cell node.
The days were long yet quite rewarding. I enjoyed helping the blue cells learn how to submit incident reports. Though I frustrated many of them by continually rejecting them for lack of evidence they began to learn that I needed who, what, when, how, and maybe why to give attribution to an actor(threat). They also had to learn that it isn't what they think they know, it's what they prove by providing corroborating evidence such as logs, files, and screenshots. Additionally, if a team was really struggling, I could provide hints and suggestions or in dire cases I could take the blue cell member aside and have some teaching moments as they struggled with the complexities of being assaulted not only in the cyber world, but in the business world as well.
Did you say the business world? Yes, I did. The "CEO" flew in to interview the team captains after he learned that some of his assets had been compromised. Each captain was given the "opportunity" to sit and tell the CEO the state of affairs of his network and data. Some young folks responded with poise, others literally shook in their chairs, and still others refused to have their follow-up meeting. This was also excellent training that should help close the gap I have seen where non-technical people are not getting promoted. This also gave me an opportunity to speak with some of the captains outside of "the pit" (competition floor) to explain some of the terms used by their CEO and help prep responses in his language.
This was the best training a future IT security professional can receive and I truly appreciate that EnerNex was kind enough to send me. Being able to assist in events like this gives me hope that things can get better in InfoSec. It's always a pleasure to share experience and knowledge with those who are seeking a career in this field. Many dedicated educators are attempting to do this but they need practitioners from the real world to assist and fill in the gaps. I look forward to assisting more in this area both at our local schools here and with some of the colleges and universities I interacted with at this competition. I met some great students, faculty, parents, and sponsors. I also had the wonderful privilege of working with Casey O'Brien, Tim Rosenberg, Matt McFadden, Gary Stoneburner, and many others. Please keep in touch everybody.
For the curious:
Blue Cells: http://www.midatlanticccdc.org/CCDC/students/
Sponsors: http://www.midatlanticccdc.org/CCDC/sponsors/ - we can't thank you enough
Pictures I took - https://picasaweb.google.com/griffse/MidAtlanticCCDC#
I had a great time at this conference and got to meet some great folks. I also had the opportunity to be the first speaker on the first day which really helps the other talks as I set the bar pretty low. My two favorite talks were Travis Goodspeed leaving the crowd in stunned silence with some of his hardware ninjary, and Ido Dubrawsky from Itron during the AMI security workshop. Ido gave a great talk that was grounded in facts which is sadly lacking sometimes. Stephen Chasko and Ed Beroset also gave good talks from a vendor perspective. For the panel-style talks, I naturally enjoyed the penetration testers over most of the policy and strategy panels. I will have to say my favorite moment was a vendor offering perhaps a tiny bit of marketing hype being asked from an audience member "Are you saying you are guaranteeing absolute security from that point forward?" Of course, the vendor was not offering that and the talk proceeded smoothly. I enjoyed that because it represented the spirit of the conference. People spoke openly and disagreed with each other with facts and perspectives without anything devolving into chaos. There was even a meter vendor panel where they seemed to be working towards common goals regarding smart-meter security. A goal going forward is to get SCADA vendors involved and provide utilities with a way to share security-related information if they are experiencing an incident.
I was also privileged to connect with several gentlemen and ladies considered by many to be the leading experts in the efforts to make the smart grid secure. I cannot list them all but literally everyone I met was making a contribution to this effort. I can't wait to see the next event, it should be even better.