risk management – Kevin L. McLaughlin, PhD

Build It, Buy It, or Somewhere in Between: Honest Questions Before You Stand Up a SOC

Thinking about building a security operations center or hiring someone to run one for you? Before you look at a single price quote, here are the plain questions worth sitting down with first.

There’s a question that comes up a lot in our field, and it sounds a lot simpler than it really is. Should we build our own security operations center, or should we pay someone else to run one for us? The honest answer is that it depends, and I know that’s not a very satisfying thing to hear. So instead of leaving it there, I want to talk through what it actually depends on, because those are the conversations that save organizations a lot of money and a lot of heartache down the road.

The first thing I’d ask anyone is what they actually need the SOC to do. A SOC is not a room full of big screens. It’s people, process, and tools working together so that when something goes wrong, somebody notices and does something about it quickly enough to matter. Before you look at a single price quote, write down in plain words what good looks like for you. What hours do you need covered? Which systems matter most? Who gets the call at two in the morning, and what are they allowed to do once they answer it? If you can’t answer those questions yet, you’re not ready to build or buy. You’re ready to have a meeting with the business, and that meeting is worth having first.

The second thing is to be honest about people, because people are the hard part. You can buy tools. You can sign a contract for a platform in an afternoon. What you can’t do in an afternoon is hire, train, and keep a team that knows your environment well enough to tell the difference between normal and trouble. Building in house means recruiting, scheduling around the clock, covering vacations and sick days, and giving folks a reason to stay once they’ve learned your business. Overnight shifts wear people down over time, and good analysts get offers. If you can’t staff it and keep it staffed, the most beautiful SOC design on paper turns into empty chairs. I’d rather see a leader admit that up front than find it out the hard way a year later.

Buying has real advantages. A good managed provider can give you coverage much faster than you could build it, and the right partner brings people who’ve seen a lot of different situations. But there’s one thing you can never hand off, and that’s accountability. The provider can watch your environment, but your organization still owns the risk and still owns the decisions. So when you talk to a potential partner, ask the plain questions. What will you actually do for us, and what will you only tell us about? How do you escalate, and to whom? Who do I call when something goes sideways on a holiday weekend? How will you learn how our business works? And if we part ways someday, what do we get to keep, such as our data, our detection rules, and our documentation? A partner who answers those questions clearly is worth a lot. A partner who gets vague is telling you something too.

A lot of organizations end up somewhere in between, and there’s nothing wrong with that. You might keep the knowledge of the business and the big decisions inside your own walls, and let a partner carry the overnight watch or some of the specialized work. That can be a very sensible arrangement. The thing to watch in a blended model is the seams. Things fall through the cracks at the handoffs, between your team and theirs, between one shift and the next. Talk through those handoffs before you need them, and then practice them. A tabletop exercise on a quiet afternoon is a lot cheaper than learning where the gaps are during a real incident.

Whichever way you go, think carefully about how you’ll know it’s working. It’s easy to count alerts, tickets, and hours. Those numbers have their place, but they don’t tell you much on their own. The better questions are whether the right people found out about the right things quickly, whether the team acted, and whether leadership got reporting that helped them make a decision. Good reporting should help the organization get a little safer every month, and own it when it didn’t.

The last piece of advice is to start smaller than your ambition. It’s far better to watch a handful of truly important things well than to watch everything poorly. Get the basics working, build trust with the business, and grow from there. A SOC that earns its keep on a few critical systems will have a much easier time making the case for the next step.

There’s no single right answer to build or buy. The right answer is the one that fits your risk, your budget, and your people, and that you can live with at two in the morning when the phone rings. I spent a good deal of time on these same questions when I wrote Cybersecurity Operations and Fusion Centers: A Comprehensive Guide to SOC and TIC Strategy. It walks through choosing between building and buying, picking a partner, building the team, and running the day to day operations and reporting. If you’re wrestling with this decision right now, it may save you a few headaches. You can find it on Amazon.

Small and Medium Size Organizations at Risk

The bad guys are now starting to target mid-tier companies with their hacking activities. In many cases these companies have not yet come to realize that having a Certified Chief Information Security Officer (C|CISO) or a Virtual Cerified|CISO (vC|CISO) is something that needs to be accepted as a part of doing business. The recent Presidential report on how Cyber Attacks impact the economy of the U.S.A. makes it clear that the cost of having a security expert on staff or on retainer is well worth it if the smaller company wants to remain in business. Cyber crime is putting many of these smaller companies out of business simply because they cannot recover from the post breach losses. At the same time these companies may not have the dollars to pay for a full time C|CISO, so their alternative is to explore the vC|CISO concept. The vC|CISO model can be leveraged to effectively manage security risk for small to mid-tier companies.

If a company is going to explore the hiring of a vC|CISO they need to make sure to do their due diligence and ask about the certifications (such as the one in the graphic below) and experience of the vC|CISO they are going to have giving them advice. While there are a lot of good vC|CISO options available there are just as many vendors and security professionals touting themselves as vCISO’s who lack the background or experience to be of much help. I lack quantifiable data on this point but I would say you should expect to pay between 3 and 400.00 per hour for a vC|CISO, the good news is you can choose what % of their time you want them to work for you (10%, 20%, on retainer, etc.) and then only pay for that portion of time.

Certified CISO

Security Steering Committee – Members Needed

Steering Committee Story

The Office of No, Really?

Successful ransomware attacks are at an all time high, we are losing the cyberwar, cyber criminals are making more money than ever before and it is only going to get worse, a cyber attack could be as damaging as a nuclear war – the headlines abound with comments such as this. Yet, there are still a lot of postings about how information security should not be the office of No. Really? Information security is a security function, as an information security professional are you really going to say – Yes, sure, go ahead and port that un-scanned software code into our production environment? I would hope you are going to say no you cannot port that un-scanned software code into our production environment. No is not a bad word, it is one that the job requires us to say. If you are offended when someone says you are the office of No- don’t be. To be a security professional no is a good response. I train my team members to say “yes, but” in order to soften the perceived impression of the word no ….. (here’s a clue – “yes, but” is still a no – no matter how you spin it).

However, I have yet to sit in a business meeting and see any of my peers simply say no without being willing to engage in dialogue that would lead to a good and secure solution. So for those who don’t like to say no then do it this way – yes you can put that server with 2 catastrophic exploitable vulnerabilities into the environment after we help you remediate the vulnerabilities or work on additional adequate controls that will allow us to use the server in a safe and secure manner. For me, information security professionals can and should say no, but after doing so we need to be helpful and smart enough to engage in conversations that help the business figure out how to do what they want to do in a way that is cost effective, safe and secure.

In many ways with the types and amount of successful attacks we are experiencing across the U.S. infrastructure being the office of Yes is a far more scary response than saying No to items that put your business at risk.

Why Protect End User Workstations Instead of Servers First?

I have been lucky enough to spend most of my Cyber Security career doing startup operations for large companies. I thrive on the energy and passion that teams get when they are given the opportunity and support to design and implement security protections for their company.

One of the things that I am often asked is why do I focus some of my first efforts on locking down the end user system before locking down the servers and databases. This is a great question and one that can spark many hours of debate. Please don’t send me a lot of comments telling me that Databases and servers are where the information is – of course I know that.

For me, and remember my work has been mostly in very large global enterprises with a mix of blue collar and professional staff, it is a matter of evaluating risk. In large companies it is often hard to know where all the servers are and who owns them but it is a pretty safe bet that the person running the server is a techie and has been running and protecting these types of systems for a number of years ergo they have much more knowledge of how to protect a system and how not to fall for a bad actors attack than most end users. I need to consider whether 50,000 or 100,000 attack vectors (i.e. end user systems) , all with email accounts, usually with admin access to their systems and wanting to open attachments and PDFs pose more of a risk than 4-5,000 attack vectors (i.e. servers) that typically don’t surf the web or get email. My choice in the first 12-18 months is to say that the end user systems pose more overall risk than the servers. This leads to aggressively putting in controls and protections for those systems first. Please don’t take this out of context – of course in parallel I drive initiatives to patch server vulnerabilities, get servers logging into a Security Alerting system, setting up a SOC, etc. but given choices with limited resources and time I choose to deploy end point encryption, good AV and HIPs, taking away admin rights, software such as Tanium, 2 factor authentication for email, etc. on the end user system first.

Critical Manufacturing Legacy Systems

One common item that information Security Professionals working in Critical Manufacturing environments have to deal with is that of legacy systems. You see, in Critical Manufacturing environments it is very common for the systems that run and control factory lines to remain in place for a very long time. Some of these systems can be running Operating Systems(OS) that are 10 to 15 years out of date. In many cases these OS are no longer vendor supported and cannot be patched to remediate known exploitable vulnerabilities. These older systems are often used to run production lines and they still do a great job at doing what they were purchased to do. It is difficult, rightly so, to convince the leadership team at a factory to spend money to replace something that is old but that is still doing the job it was purchased to do. It cost money and reduces potential profits to replace these old and outdated systems with new systems. Because they continue to do the job they were purchased to do justifying the new spend can be a difficult thing to do.

From an Information Security point of view these systems pose a large risk to the overall manufacturing environment and if hacked could cause a large scale production outage. In smaller companies this type of major Cyber attack can result in no longer being able to conduct business and permanently closing the doors. Legacy systems that are commonly found on the shop floor are often 3, 4 or even 10 years out-of-date when it comes to standard Information Technology patching. Information Security Professionals look at these systems as attack vectors while the people working in the Critical Manufacturing environment view them as cost effective work horses that are getting the job done. While cyber attacks on networks at Sony, Target, Home Depot and the US Government are getting all the press, the greatest cyber vulnerability is in manufacturing. “By raw numbers, and by the numerous manners of attacks, manufacturing is the most targeted area now, even compared to financial services,” Chet Namboodri, senior director of Global Private Sector Industries at Cisco, told Design News. “Financial services gets more press, but industrial networks get more attacks.” Attacks and warnings such as Stuxnet, Armaco, SolarWorld and U.S., Steel to U.S. regulators and security experts sending out an official warning that hackers could now access critical medical equipment including pacemakers and insulin pumps with potentially deadly results make the threat to Critical Manufacturing a real one. Determining what to do in order to lock down and protect the Legacy systems while at the same time allowing them to continue doing the work they have been doing is a major part of an Information Security professionals job.

“You can’t scan that system (or you can’t put AV on that system) because it is old and fragile and if you bring it down we will not be able to produce our product” Is way too common of a phrase in the Critical Manufacturing environment. In many cases Information Security Professionals are asked/told to please just leave the systems alone, do not run vulnerability scans, do not put antivirus on them, do not put a light firewall on them, do not patch them, do not put updates on them, etc. This type of thinking by Information Technology and Factory leadership teams is shortsighted and is putting their entire production capability at huge risk of catastrophic failure. As these legacy systems are outdated and no longer being supported by the vendor they are hugely exploitable to any blackhat or hacker that wants to take advantage of their exploitability. The reality is that the risk is real, the risk is great and from past events we know that these systems pose easy to use attack vectors for blackhats, fraudsters and competitors seeking to cause negative business impact to the company.

Information Security professionals working in Critical Manufacturing should take the approach shown in Table 1 for dealing with the Computer Systems residing in the factory environment and that are on the plant floor. By following this methodology the legacy systems will be protected while at the same time be able to continue doing the job they are good at and that they were purchased to do. In most cases this approach will also reduce the overall risk that these systems pose to an acceptable level.

Legacy Table Table 1

This approach when combined with the network segmentation and smart firewall approach discussed in my previous blog on Critical Manufacturing is the start of a successful recipe in securing a Critical Manufacturing environment.

Critical Manufacturing Cyber Security Defense Program – A Starting Look

Cyber security professionals are often tasked with trying to defend more than they have the resources to defend. This is especially true in the area of Critical Manufacturing as there tends to be a strong desire to protect everything. In many cases when we try to protect everything we end up protecting nothing. A Cyber Security Defense Program (CSDP) needs to be very focused and strategically deployed to protect the systems and areas which need protected.

Critical Manufacturing as defined by the Department of Homeland Security(DHS) is primary metal, machinery, medical, electrical and transportation. Primary metal is Iron, Steel, Aluminum non-Feros metals; medical includes facilities and devices, machinery is engines, turbines and power transmission, within electrical is electrical equipment manufacturing, transportation is : vehicle, aviation and aerospace parts and vehicle manufacturing and railroad rolling stock. Products made by these are essential to critical infrastructure sectors. The Critical Manufacturing sector focuses identification, assessment, prioritization and protection of nationally significant manufacturing industries within the given sector that may be susceptible to manmade or natural disasters. This area is one that is critically in need of a strong CSDP. The discipline for CSDP must not be twisted to taint what should be a strategic approach to the industrial base (“Critical Capabilities At Risk,” 2009).

One critical component to the protection of a Critical Manufacturing environment is that of using next generation, aka smart, firewalls with intrusion prevention and with some form of malware prevention and detection, updated via global feeds, enabled. This firewall environment should be set up in such a way as to segment the Critical Manufacturing infrastructure. Each individual plant or factory within the Critical Manufacturing environment should be on their own local area network (LAN) segment. A very basic way of thinking about this is captured in Figure 1 below.

Manufacturing Network Segmentation

Figure 1

Because of the method in which business decisions are made for factory systems it is critical that organizations involved in Critical Manufacturing use strongly firewalled network segmentation methods to ensure that each of their factory sites sit behind their own firewalls and on their own network segment (DHS, 2009). This allows better control of the environment and enables the isolation of one factory without impacting the rest of the factories owned by the organization. In this way if one factory is compromised by a hacker or group of hackers it can be removed and isolated from the organization’s global IT infrastructure. Network segmentation, also called Network Control (NC) is a primary bastion for a defense in depth strategy and it can also be used to help prevent a blackhat from pivoting from a relatively unsecure factory system to a core business system. Combining an Intrusion Prevention Appliance (IPS) or functionality with the firewall inside of a network segment provides a strong front line of perimeter defense for factories involved in Critical Manufacturing. These IPS’ do not have to be overly tweaked and analyzed to be effective, running them in default signature mode is an adequate layer of defense for the network segment protecting and controlling a factory environment (DHS, 2009).

Cyber Security – A basic look at Defense in Depth

A basic somewhat quirky and fun look at Defense in Depth.

The Highest Risk Ever….. Not!

BlogWordle I just received one of my daily news reports about Cyber Security and it said something to the effect of such and such a Research group has found out that Application Security holes are at highest numbers ever and this number is expected to increase even further in 2016! Companies really need to focus on fixing the risk incurred by these applications. I thought, “Wow, Really”? I mean yet another highest number, highest risk, most vulnerable, more danger than ever before message. What a surprise it was to get yet another news feed about a Cyber Security vulnerability being the worst ever. (that last sentence is dripping with sarcasm – just in case you couldn’t tell). I really just want to shout and scream out that No It Isn’t the highest total ever! None of these items being touted as worst ever are really the worst ever. The simple reality is that we are still in our relative infancy when it comes to Cyber Security as a profession and as something to write about. Being in our infancy means that we are still finding lots of this things that we couldn’t find before.

None of these vulnerabilities appeared overnight. The vulnerabilities that we are now finding and that are generating these, at times horrific, Blog- Overnight reports about numbers of vulnerabilities have always been there. Not only have these vulnerabilities been there but here is a secret I will let you in on – there are many more that we still haven’t discovered! Companies, researchers, etc. now have the proper tools to discover concerning Cyber Security vulnerabilities in large numbers. These are items that in the past no one knew about even though they were there. As more and better tools come out we will find even more of the worst vulnerabilities ever.

Vulnerable OS, vulnerable Webpages, vulnerable coding, vulnerable hardware or firmware, these items have been around for decades and will remain around. So while it is great that we can now find these and do something about fixing and remediation of these vulnerabilities they aren’t new. Maybe newly discovered or newly recognized but they have always been there.

Why am I going on about something that is basically hype, and in some ways good in that it publicizes exposure to these items so that Cyber Security professionals can use them to raise awareness, funding and support for remediation? Because of the way the reports and news articles are written. In too many cases these articles raise awareness and hype of issues that are much lower risk than the ones the organization’s Information Security team are already working on. I have sat in many meetings where executive management has looked at the Chief Information Security Officer (CISO) and said “I know your team is really busy remediating core vulnerabilities across our Financial landscape but I read something yesterday about mobile devices being very vulnerable and I think we need to re-focus our resources…. the article really made this mobile risk sound like the most dangerous thing ever.”

Don’t get me wrong, I appreciate the awareness being generated but …. the WORST EVER …. no in most cases it really isn’t. There are lots of Cyber Security items just as bad or worse that most organizations need to fix and the decision on where the highest Information Security risk sits inside of the company should not be decided by reporters and vendors but should rest on the shoulders of the CISO and the experts they have hired to analyze and determine what risk items need immediate attention.

Home Depot Breach 43M loss posting on LinkedIn

I just read an article on LinkedIn about the negative impact caused by the recent Home Depot breach. Here are my thoughts about how all the 2014 breaches should change some C level and Information Security paradigms. These could cause a major C suite thought shift and maybe even a large paradigm thought shift in how our profession thinks about the skills and talents really needed by Information Security professionals.

Note – I know this opinion will not resonate well with some CISO’s – Major IT departments continue to hire CISO’s that have little to no security background. Hackers and blackhats are associated with organized crime and IMO until we put more of an emphasis on the word SECURITY in the CISO title and until CISOs stop walking a tightrope by always trying to compromise and keep the business happy by letting the business need trump the security need (“ok, don’t patch that extremely vulnerable system I understand your business need not to” ) the crooks will continue to win.

Also, it appears that too many companies are basing their CISO hiring decisions on the candidate’s business acumen. Wouldn’t it be better to focus on how well the candidate knows how to fight and deter criminals?
I’m not saying CISO’s don’t need to understand the business I am saying that a stronger focus should be on how much the CISO knows about criminals and crime fighting. I also realize that thought would take a major paradigm shift in the thinking of the C suite.