Speaker 1: 0:04

All right, you are listening to the Audit presented by IT Audit Labs. I'm Joshua Schmidt, your co-host and producer Today. We're joined by Nick Mellom and Eric Brown of IT Audit Labs, and our guest today is Dino Busolacchi. He is currently in St Louis, missouri, and he's coming to us today to talk about OT-IT convergence. And, dino, thanks for joining us.

Speaker 2: 0:25

Thanks for inviting me.

Speaker 1: 0:28

Absolutely. I know this isn't your first podcast, but we're excited to talk about a topic that you're familiar with today that OT meets IT and a lot of other things that go hand in glove with that, so thanks for joining us. Can you give us a little background? You have so many titles here on LinkedIn I didn't know where to start, so I'll hand that over to you.

Speaker 2: 0:49

Yeah, that comes with the gray hair right. So I'm currently a director here at the Barry Wimmler Design Group. The Design Group is a manufacturing engineering organization been around for 100 years and they acquired my firm last year. So I've been a design group now since July of 2024.

Speaker 2: 1:12

Prior to that, I owned and operated a OT cybersecurity systems integration company. I was focused just with these OT cyber securities. Prior to that, I spent several years working for Rockwell automation and their network security services group. Prior to that, I was in the Rockwell channel working with their distributors, a lot of system integrators and OEMs, and I'll touch a little bit on that then.

Speaker 2: 1:36

Prior to that, I spent 20 years working for one of the world's largest adult beverage manufacturers based out of st Louis you can probably guess who that is working building out breweries of the future and so I spent a lot of time around control systems, implementing what I want to call IT technologies on the plant floor, which is why, when I was at Rockwell when some of these OT cybersecurity technologies started coming to the market, I knew there was a niche in driving cybersecurity down on that plant floor the myth of those machines being air-gapped it just doesn't exist anymore. Prior to that. I worked for General Motors under Ross Perot back in the 80s. I worked for Monsanto and various others. I actually started in the banking industry in the early 80s, working in data centers and got involved with building out networks to banks and putting in ATM machines back in the day when the ATM machine was actually at the bank. So that's a condensed version of my background.

Speaker 1: 2:32

That's awesome. Do you get a lifetime supply of free beer then? Does that come along with the territory?

Speaker 2: 2:38

If you know the right people. The answer is yes.

Speaker 1: 2:42

One of the things that stuck with me is you shared that manufacturing is one of the largest attack vectors and maybe you could explain why that is and how that has met with your experience working in OT.

Speaker 2: 2:54

Attacking a manufacturing facility and getting to the point where you have a material breach and you're able to shut down that plant is very costly to shut down that plant is very costly, right. So when I worked for that large adult beverage manufacturer, if we just had one packaging line not putting beer in a can or a bottle that was $80,000 an hour of loss, right. And so when you think about a manufacturing environment, you know their plants aren't operating because it's very expensive, very quickly, right. And cybersecurity insurance doesn't cover business disruption? Right, it'll cover the aspects of doing the remediation, mitigation work that you got to do, but they're not going to pay for the loss that you incurred from being down.

Speaker 3: 3:35

So then, dino, probably the most famous example of that air gap breach, so to speak, is probably Stuxnet right, right With the sandworm. So, in your experience, have you seen other attacks like that when you were working at those previous organizations, and how did you combat it?

Speaker 2: 4:00

Not that targeted. I mean, it was a highly specialized, developed malware, you know, attacking a specific Siemens set of control systems that, whatever nation states were involved with it, had an inherent knowledge of what they were using within that nuclear facility. But what I will say is that a lot of the manufacturing environments are impacted by breaches that come through the IT environment. Right, because we use a lot of the same technologies on the plant floor that you have in the enterprise or in the office space. Right, windows, for example, that are running HMIs. You know your human machine interfaces, your historians, your application servers, your engineering programming terminals those are all Windows machines. And you also have a lot of networking equipment out there too. Right, you've got all stripes and types out there inside that manufacturing facility and they all have vulnerabilities. So if you're accustomed and used to what I still call Patch Tuesday for Microsoft, right, we don't patch very often on the plant floor because it's very disruptive.

Speaker 2: 5:04

The life cycle of control systems is measured in decades. Right, it's not measured in a three to five or seven year window. So when control systems, you know you're putting in a $20 million packaging line, the intent is that packaging line is going to be out there for 20 years doing what it does, right, and so, and even then, you know, the frames of that equipment may still remain the same. But you might do a recontrol right and recontrol efforts. That means that you're replacing the PLCs, you're replacing the SCADA. Software might be doing some upgrade on the applications, maybe the network. It's very, very costly, right, and those are usually capital projects. They're not OPEX projects, they're capital projects. And so the dynamics are different in that particular space. And if you're buying from an OEM, right, they may not stay as current on patching for their control systems, right, they just again, because it's disruptive and there's costs involved with it, you know, and and so we don't see a lot of that. And so then you gotta, you know we subscribe to, you guys are familiar with, you know, the sans institute, right, so think of um the the sans.

Speaker 2: 6:14

There's five critical control points for ot cyber security right in the it world there's like a dozen or more, but on the ot side we can kind of focus on five with incident response, defensible architecture, vulnerability management, remote access, which is really big, and we talk about that in a second because COVID kicked the door open on that and then continuous monitoring right, which is not something that groups are used to doing. Because most engineering teams, when you start talking to them about putting technology inside their control system network to monitor it, the first thing they think of is that you're scanning and dumping a lot of data on that control system network and it can be very disruptive. And so they've lived through that because IT even though sometimes they're really trying to do their best on due diligence you know putting in IT cybersecurity tools and they start scanning that network or implementing EDR on those HMIs you know, like a CrowdStrike or Sentinel-1, and all of a sudden you're inducing delay around those machine centers that require very specific you know time sinks and communications for a couple of reasons, safety being one right and depending on the nature of whatever you're doing, you know if you're not watching that stuff it can be very impactful and sometimes IT doesn't even realize they're doing it right and the plant floor folks don't necessarily have the technology as a place to see this stuff. We have that technology now, right when you think of the Armises and the Clarities, and then the Zomis and Cisco Cyber Vision and Dragos, if you're familiar with those types of intrusion detection systems that were specifically built and geared for the OT space, right, and they're passive, right. They sit there and listen to the network. They do do packet inspection to build out that baseline of those assets, to build a profile on that PLC, to tell you the make and the model and the firmware on it, and based on the firmware version, now I know the vulnerabilities that are associated with it, just like you know the operating system level on a Windows machine I can start building that CVE list, right, and then from there I can start building a strategy on how I can improve my cybersecurity posture around those control systems.

Speaker 2: 8:23

Because, back to my earlier point, the stuff is connected, right, it's just, whether it's for support purposes or feeding MES systems or manufacturing execution systems or your ERPs or other enterprise applications, they're pulling data up and off that plant floor and then you have people showing up to the plant floor. So, and then you have people showing up to the plant. You know, if I go and if I show up to a plant, eight out of 10 of them will let us go in there with our own laptops or computers and plug into their environment, right, and they have no idea what's on these machines, right, and then a lot of OEMs. When you buy a machine from them, so think about buying a new vehicle, right? And it's got, you know, onstar on, it's got satellite, it's got cellular, it's got Bluetooth, it's got Wi-Fi.

Speaker 2: 9:09

Well, these machines aren't any different on the plant floor, and we'll find cellular cradle point modems in there. We'll find VPN concentrators from SACOMI or EWON or Toshiboxes, right, because somebody is connecting in to provide support of that equipment down on the plant floor when it's not running correctly. Or if they want to do some type of enhancement, maybe they're moving to a different product, so they're doing some programming, and so there's always connectivity in those environments. They're very dynamic, they are not static at all.

Speaker 1: 9:39

That was going to be one of my questions and you've pretty much illustrated it. But for the people that aren't, you know, familiar with manufacturing environments or or maybe just you know are tuning in for the tech aspects of this show, can you give us kind of this like a general sense of what these manufacturing floors look like?

Speaker 2: 9:55

yeah, and it's significant right, when you think about the number of you know you've probably heard the term iot or iiot for the industrial Internet of Things. We obviously have the term OT. But yeah, when you think about motors and drives and robots and pumps and sensors and fillers and conveyors and just the presses and all you know dryers and fans, that can go on and on and on and all of the different types of connected physical systems, that's what we kind of call it, you know, I think of connected physical systems and in the IT world you have data driving data outcomes. Right. In the OT world you've got data driving physical outcomes. And so you know the two priorities.

Speaker 2: 10:42

In any manufacturing facility, safety is always job one. Right, there's no doubt about that. You got to operate safe and it's not negotiable, right. So you have to be a safe environment. You have to demonstrate that you're doing it.

Speaker 2: 10:52

Number two is what I spoke about earlier. It's unplanned, unscheduled downtime, because it's expensive and it's costly to demonstrate what the target is for this Right, because it's usually not in their purview, meaning that IT doesn't own those assets. You know. Nowhere else in the business, especially on the enterprise side, it has a responsibility of the technology that the company is using to run their business right. But on the OT side it's not so much. You know the decision to put in a new packaging line and what's going to consist on the control systems around that packaging line, and what computers are we going to choose and what networking equipment are we going to choose, and operating systems and firmware and applications. It's an entirely different group of people, right? So I worked at Rockwell. Even when I worked for Anheuser-Busch, it wasn't sitting in the room when we were getting ready to determine, you know, to build a new brew house for $500 million, should they have been sitting in the room.

Speaker 3: 11:51

Ideally, Eric.

Speaker 2: 11:52

yes, we want that. Yes, here's the way I kind of describe it, Eric. I want them on the field, I want them in the huddle. They just can't be the quarterback, if that makes sense right, but yeah, they play a tremendous role right Governance policy, technology resources, standards.

Speaker 2: 12:10

They bring a lot to the table, but they're not always invited in through that CapEx project, on whatever it is that company's doing an expansion, brownfield, greenfield, lifecycle replacement, whatever the re-control, whatever tech project they're taking on we don't typically see it sitting in the room participating, you know, in something that might start two years prior to that machine being built and running. And I'll give you another good example erica, we were just in atlanta a couple weeks ago and the client, the it, the networking team, came on site and met us there. They were 20 minutes from this plant and out of the three it professionals out of this company, only one of them had been at the plant in the last five years. The other two had never been there.

Speaker 2: 12:58

So you know it's very difficult if, from an it perspective, if you're not engaged in what's going on in that plant operationally, from a process perspective, what technologies are in there, it's very difficult for you to start putting a cybersecurity strategy around that. You've got to think globally but you've got to act locally. And if you've got 50 plants in your fleet, no two are the same fleet, no two are the same, and so you have to have some inherent knowledge of what's going on inside that facility and what technologies do they have in there and how that stuff is put together.

Speaker 4: 13:31

You know we're seeing this too that IT is not invited to the party. I guess in your opinion, you know, especially in your industry. Why do you think that is? Why are they being left out of these meetings that are happening, even if they're happening two years beforehand? Why are they not being invited? In your opinion?

Speaker 2: 13:49

Yeah, good question. You know, one of the things that we look at when we start these engagements is we look at the organization. One of the things that we look at when we start these engagements is we look at the organization. And sometimes the first time that IT and OT have the same manager leader is usually at the president or the CEO of the company. Think about that for a second right and that's not necessarily right either right. And then we have to think about the size and the scope of how many people you need. So if you're a CIO and let's say you got, you know, 200 people working for you, right, and you've got 100 plants, well, how big is that that OT practice need to be from a technology standpoint? And who's going to own that Right? Is it going to be the CIO? Is the CIO going to become the new plant manager or general manager of the plant?

Speaker 2: 14:35

Because they operate as fiefdoms, these plants do their own little kingdom. You know meaning they decide who they're going to work with from an integrator perspective or whose machines they're going to buy. You know what their partner community is. On the OT ecosystem as I call it right, and the OT side is not any better right, as I call it right, and the OT side is not any better right. The OT ecosystem, that supply chain on that side of the house, needs to get better at this right. Because if you go in there and start talking to them about cybersecurity, who do you think they're going to point their fingers at and say is responsible for cybersecurity? That's IT. They're going to come. That's IT's problem, right. And then we show these machine centers and IT sitting in the room. They're going yeah, that's not us, I don't even know what that stuff is Right. And so there's a gap.

Speaker 3: 15:21

We see it a lot where we'll go in to help an organization maybe restructure how they're doing information security because they're at an inflection point, Right, Whatever that is. And a lot of these organizations have multiple disciplines of OT, whether it's around transportation or refuse or control systems that are doing something related to critical infrastructure. And I don't think I've gone into an organization yet that has had a what I would call even a standard plan for how they're going to operate and secure the technology that the plant is responsible for. Like, at the end of the day, don't really care what's in the plant, what the plant does, but it's about how are those systems that are operating that equipment being secured. And, like you said, there's communication happening outside of that. Quote unquote, closed network. I don't think I've ever seen truly closed network yet either, and I've been doing this a while. But yet there's, you know you, some of them are museum pieces. You go in there and there's windows XP, there's windows seven, there's, there's all of these um, none their their desktop applications that, for whatever reason, these OT organizations have decided to run their infrastructure on, See very little Unix, Linux flavors.

Speaker 3: 16:59

It's a lot of Windows flavors and a lot of legacy Windows flavors and you mentioned tools like Dragos or specific to OT that can help look passively for things going on on that network, which you know in some cases when the horsepower around the technology was smaller or there was less horsepower available to do non-ILC processing. Then nowadays, though there's technology that could sit quietly on the network, could look, decrypt, whatever, for things that might be calling out or attempting to call out or attempting to move laterally, and unless that, that, that OT part of the organization is bringing apart the discipline, which it's really hard to have, the discipline to know how to run a tier one cybersecurity shop and run a plant that is responsible for operating that line of business. It's really hard to do both. So you've got to kind of outsource the IT side to either internal IT or a third party to bring about that best practice and facilitate it. And I don't think it's acceptable to say, yeah, it's okay to run on 20-year-old technology just because the plant itself is running on old technology, because it can last 20, 30 years.

Speaker 3: 18:37

Those manufacturers need to be held accountable to bringing the technology along. It's not a secret. We all know that Windows expires over time. We've got to stay on top of it. So bringing governance in early into that two-year process at the contract level to hold the vendor accountable for making sure that those updates there is at least a vehicle to do the updates. That's kind of where I am now in my headspace, versus saying, oh, it's an OT environment, let's just leave it, let it run and hope nothing happens.

Speaker 2: 19:13

Yeah Well, and the way what you're describing is very expensive, right, sure To replace, to move from Windows 7 or XP and do a recontrol or maybe move a piece of equipment. You might be talking about hundreds of thousands or millions of dollars in order to do that right. And so there has to be the client has to recognize that.

Speaker 1: 19:34

Where are you right?

Speaker 2: 19:35

Because right now they don't even have a really good idea where they are. They don't have a good baseline. So you got to get that figured out first and there are. There are technologies that help through that. While they're building their capital plan eric to your point to be able to go after that stuff um, like virtual patching right or defensible architectures around that machine center so that you can at least improve your cybersecurity posture, harden it while you're putting together that multi-million dollar CapEx plan to go after and replace that.

Speaker 2: 20:07

The other thing that I'll tell you is that I call it the OEM blockade. They won't let you touch their stuff right. If you want their support and you want the warranty and maintenance that comes with that machine center, they don't want you touching it right, and that'll because otherwise you void the warranty. So think about if you bought a car and you know you're getting the warranty and support from GM and then you go in there and you modify the exhaust system and you decide to swap out the transmission and you try to you know, maybe do something with fuel injection and you modify it and then you take it back to GM to say fix it, they're going to go no right. And these OEMs have that mindset, because they don't even understand that even if we just want to put something in there as simple as a sensor right to collect the metadata within that control system, they would say, if you touch that, you know. If you add anything to that switch to do that, or change the configuration of that switch to create a span port to pull that stuff out of there, you void the warranty and a lot of the clients are like they don't want to buck the system with that OEM. And so now you've got to get ahead of the game, right. Now you've got to be proactive and so if you're going to buy a new piece of equipment, then you need this and you're let's say you're a drago shop or an armist shop or clarity shop, you should be planning ahead to put in the right technology in that machine center, right.

Speaker 2: 21:26

I always like using the automobiles. You don't put the safety, the seat belts, on after the fact, or the anti-lock it breaks on after the fact or the sensors of the backup, backup cameras, the airbags. You do it when you're building the car, right. And it's the same concept which is one of the other big disconnects that we see is those OEMs and a lot of these system integrators, which the firm I work for is both of those trying to change that narrative in the machines and the integrations that you're doing and implementing that up front, right. It's much more cost effective to do it then as it is to do it on the retro, on the backside. So two things it can be expensive to try to do a recontrol, replace equipment, and then you've got to deal with the OEM. That can be a little pig-headed sometimes, right, and you've got to try to break through that.

Speaker 4: 22:08

There's a lot of moving parts here. I'm thinking back to one of my first IT jobs, relatively not that long ago 10, 12 years ago and it was at a large manufacturing company in the Twin Cities and, being a junior IT, I was on a help desk there and I'd run out and help all these different lines get back up that were going down or do any sort of tasks. But there we had so much legacy software and I'll never forget one of the more senior guys. I was out running tasks with him and we had one machine over in this corner. I don't remember what it was doing, I think it was printing labels or something and he said we don't touch that thing because if it turns off it might not ever come back on. It had been there for maybe 30 years out of support, obviously, legacy software. The vendor doesn't even exist anymore.

Speaker 4: 22:57

But it was always so confusing to me why don't we just change it out? And I'm hearing from you now. You know just the expense to it, but you know, is it a lack of auditing? Are we not asking these questions? Is that? Is that the root problem? Are we not, as IT professionals, are they? Are we not digging into the sand, letting them know these issues. I guess my question is are we going wrong somewhere or is it just because we're not invited to the table is my earlier question. But it's happening so often that we're inviting the wolf into the hen house, not being compliant or having these discussions that we're having now, which it seems like we're all on the same page, but it seems from a simple discussion earlier on. Like you said, putting the seatbelts in early, we could solve a lot of these problems down the road.

Speaker 2: 23:45

It's the timing of it, in my view, right. How do you get that IT group engaged and involved in that OT environment right and become part of that ecosystem?

Speaker 3: 23:57

What makes the most sense.

Speaker 2: 23:59

So think of shadow IT right.

Speaker 2: 24:01

Eric, I know, you know what that is, you've been probably we go back a number of years. Right, and that was one of the things we were trying to chase out was shadow IT. But when it comes to that plant floor, it's the one area that they've just never been able to take on, and a lot of it is because they don't own the asset, right. Who owns that asset? Like I said earlier, all of the enterprise stuff, there's no question, it's it. They own that right. They make those decisions, whether using third parties or it's an internal team or a mixture of both, but at the end of the day, they make those decisions. They make that, those buying decisions. They decide the resources and the responsibility to take care of it, just like that OEM does on the plant floor. But when it comes to that PLC, it's a programmable logic controller, right, and so that's the instrument.

Speaker 3: 24:48

And it's usually running on a.

Speaker 2: 24:50

Linux-based operating system of some sort. Right, it's not running on a Windows OS, it's usually a Linux type of OS but it's the one that's receiving signals and making calls to make that motor go faster or to turn the flames up on that burner or to speed up that filler or whatever that might be. That's what that controller is doing, right, and IT just never put their hands on it, even to the extent that engineering teams would design their control system architecture to put multiple network cards into that PLC to make it a network segmentation device. Because one side if you're familiar with the Purdue model I don't know if you guys are familiar with the Purdue model, but think of a PLC with an Ethernet card on it, because we want information sent to our historian, we need remote access to it for programming and I want to send data up to my MES, and so there's a connection to what I want to call the enterprise side of the network, because even though it's within the plant, it's the side of the network that IT usually manages. But then there's other cards in there for control that are on that PLC and that's the space that you never ever find IT in, because that's actual control.

Speaker 2: 26:00

That's the one that's sending the signals, you know, because we're back to process integrity and we're back to safety, and if you don't understand what that thing is doing, it's very hard for you to take on a role of trying to be I call it, the round peg square hole and applying a security measures in there that could be disruptive to that environment. How are you going to get a group with the boots on the ground knowledge of what's going in there to implement? All right, I got Siemens over here, I got Rockwell over there, I got Honeywell, I got GE, I got Emerson, I got Yasago, I got Mitsubishi. They're not very standardized. Right, it's done a really good job of standardizing everything right Compute platform, network platform, databases.

Speaker 2: 26:41

you know ERP systems. They standardize you get into that plant and it's seven ways to Sunday in there. You know I'll find eight different access methods into the plant for remote access and I'll find several different automation technology vendors in that one facility.

Speaker 1: 26:59

One thing you said that stood out to me, dino, is that even just within a company, the plants are vastly different from location to location as well. Not just the equipment that's running in them and how they get things done and where they're positioned, and all that.

Speaker 2: 27:13

but yeah, the whole layout, the design, and culturally and behaviorally I mean depending on what part of the world they're in or what part of even in this country they're in you have different mindsets on how they operate that are tough to get through. And then you have a lot of mergers and acquisitions, a lot of these plants you know that have been bought and sold multiple times over the last 30 or 40 years and you just don't have a lot of good continuity and the way we pick up on that really quickly is lack of documentation.

Speaker 2: 27:43

They don't know what they got. There's no network drawings, there's no good network drawings. Whatever documentation they have, you know 20%, 30%, 40% of it's wrong, it hasn't been marked up, can't find it, don't have it. Right, it hasn't been marked up, can't find it, don't have it. So we do a lot of investigating and hunting. When we're in there trying to determine, we go in there and drop one of these tools in to start collecting a baseline, and then we walk the plant and we open up those panel doors to see what's in there and we document that.

Speaker 2: 28:16

So I want to be able to reconcile what did I visually see on the plant floor and it's not in my baseline on my tool. So I can figure out how do I get that system over here on this box Right, and make those determinations of whether I got to put a sensor out there or I got to change the network to some extent. Do they even have cabling to get me out there? Right, so, and so it's the dynamic of it. Aspect of it is what can be challenging Right, so, and so it's the dynamic of it. Aspect of it is what can be challenging Right, and and not everybody's prepared. You know they're behind. They're 20 years behind IT on cybersecurity 20 years.

Speaker 4: 28:46

Dean, are you familiar with CMMC, the cybersecurity maturity model from Siemens, remodel from Siemens? This is from the Department of Defense that you would have to follow these guidelines to be able to be a contractor to work, build a piece of a bomb, build a piece of an airplane or something like that, and I think we're all talking about the same thing here, but with a set of standards for many different organizations. Now this would be very hard to implement and govern. But I think CMMC was one of the first entries into the space to try to standardize this space. But we're not seeing it for non-federal contractors, somebody like Anheuser-Busch. We're not seeing it for that space at all. Would implementing something like that solve some of these?

Speaker 2: 29:34

problems. But that's one of the things we look for is are they following a security framework, right? You've got NIST. It depends on the vertical one, right? Some are highly, more highly regulated than others, whether using the IEC 62443 or MITRE or the ISO 27002 or one. And then you got NIST, which for the BOK for you know, the Boko we do is food and chemical, life sciences, heavy industries, semiconductor.

Speaker 2: 30:04

We do a lot of data center work around power systems, and so from a regulatory standpoint, not everybody has to even follow regulation, right. I mean, you think about if you're a publicly traded company today and you get a material breach. You've got like 48 hours, 72 hours. You got to tell your investors that you got pop, right, and even then the government may give you some leeway depending on who you are like, if you're Boeing, right, compared to you know, some soda manufacturer, right, and the government may even follow up on that. But my point is is that we use the security frameworks you know to go after the identification, detection, you know, response, recover, you know even an incident response plan.

Speaker 1: 30:49

What is your?

Speaker 2: 30:50

IR plan at this plant, right, and you'd be surprised how many companies don't have that. They don't have an IR plan. So then you get back to the IT conversations. Well, how do you get IT to drive that? Because IT's probably got an IR plan, you know, and they probably practice it, you know. I mean, I can remember the, you know the IR plans.

Speaker 2: 31:07

You know, back in the day where you'd run off to some third-party site to spin up your business somewhere else right, but in a plant. Spin up your business somewhere else, right, but in a plant, their only option is to either move their volumes to other facilities to make up for the loss of this one if it's not up and running right, but they don't have one. And I will tell you that if I go into 10 plants, two of them are going to have malware in them. Every one of them is going to have a rogue asset or a series of rogue assets in them. You know even the guy that's been there for 20 years and we're telling him hey, there's a WAP sitting over here on your packaging line.

Speaker 2: 31:42

He goes no, there's not, it's like yes, there is we can see it. It's on the network. Let's go hunt it down and find it. And they have no idea how it got there. We found stuff in ceiling tiles and plants that somebody put in there.

Speaker 4: 31:58

There's all kinds of goofy things and they don't even know that they're here.

Speaker 2: 31:59

It was probably the server guy wearing umbros, just to give you an idea between Gartner and Cisco and Rockwell and Siemens and these OT ideas. Guys, 60% of manufacturers out there are just. They're either unaware or just beginning to understand that OT is a thing. That's 60%. You got 30% that have actually started doing something. Maybe they did a POC, Maybe they've had some demos. Somebody came in and started talking about it. Maybe they've actually got a couple of people who are in the organization, have been tasked with cybersecurity for the plant floors. And then you have 10% that actually have a strategy and a plan or starting to implement it and going down the line.

Speaker 1: 32:39

That's where we are today. I wanted to circle back really quick and kind of get Eric's take on just integrating the IT in any kind of organization but more specifically bigger manufacturing or larger entities like Dino's talking about mostly. What's your take on that? Is it mostly just kind of just a knowledge base or does a cultural thing of why there's a disconnect there? And I know you spend a lot of time on boards and things like that. You know showing people what the risk actually is, because no one really sees it until it happens. So I was kind of curious to see what you thought about that.

Speaker 3: 33:15

I've seen a couple of cases where more than a couple and Dino alluded to this where the IT organization or the leadership of the IT organization has never actually been into the plant. Maybe they've toured the plant, been on the grounds but never really looked at the technology in the plant. And, as as Dino was telling the story about um rogue devices or potentially malware in the systems, I've walked these, um, the, these plants or these operations and the. We ought to do a picture slideshow on this because there's just some crazy pictures where it's in a hospital and the amount of cabling that was just across the face of the rack probably weighed more than me. It was just this huge rat's nest and there was dead switches that had just been cabled around that were in that environment. There was dead rodents that were completely squished underneath a piece of technology that you know hadn't been moved in 10 years.

Speaker 3: 34:29

It was way up in the rafters in this production facility that is responsible for making gift cards and we were doing this network assessment and we were kind of tracing back where we were seeing devices on the network but we couldn't find them.

Speaker 3: 34:50

And way up in the rafters on top of an I-beam was like some form of a Linksys device that was cabled into the network. There's no documentation, of course, on it. Nobody knew how it got there or what it was for, or if they were trying to add a new machine that they had long since taken out and forgot about. But the amount of undocumented technology that you know sometimes people that have been around the plant for years, they may remember, or might be some tribal knowledge, but I think part of it, josh, is if you're responsible for an environment or you're providing some form of responsibility on the technology side. You got to know what's in your environment, right Back to that CIS one and two. Know what the hardware is in the environment, know what the software is in the environment, because if you don't know what's out there, you really can't even begin to try to protect it.

Speaker 2: 35:51

That's the first candidate in any cybersecurity framework. Is that right there? Your asset inventory? Yeah, the question is whether you want to get it in a continuous fashion. Like IT generally does is collecting information consistently, constantly, versus a snapshot in time. But that's the number one tenet in any security framework is asset inventory. Even to the point where critical infrastructure organizations, where the TSA and Department of Homeland Security are coming to them, and even now insurance Now the insurance companies are coming in because they're getting smarter going you need an accurate asset inventory and I want to know the vulnerabilities and risk associated with those things. That's the first thing, right Before you get into anything else. Let's just do that Right.

Speaker 3: 36:32

Do you know? The first year that I had to fill out a cyber insurance form it must have been like 2013 or 2014 working with a customer, we sat down over a lunch with the insurance form and they're like yep, you know, we have this form that we need you to fill out. It was half a page. I've done a few this year and none were less than like 20 pages of details around multiple tabs. It's all online now. Right Of all of these things that the insurance company wants to know in order to rate you, and it seems every year there's more and more questions going into it.

Speaker 4: 37:14

Understand what's your network, but I think after we do that, I think organizations fall short of documenting everything they don't document. They rely so heavily on tribal knowledge. It's the operator that's been in there for 20 years and then they get a replacement, you get somebody junior and there's that gap of 10, 20 years. But if you spend a little time, you know, bring a third party in, go through these processes, build your policies and procedures, document everything from all these controls. I think the maturity of an organization would go through the roof instantly if you started to document these processes.

Speaker 3: 37:49

Dino. So how do you get them to maintain their environment? So you come in, you document it, you come back with a great study. How do you keep it so that you don't show up five years later and there hasn't been any updates to all that work that you've done?

Speaker 2: 38:06

So what we do is we don't sell technology and just leave. So I back up the truck, push it off the dock and give it to them, or just stand it up and get it running and leaving. We pursue managed services because, you know, managed service is big in the IT world, right, and most 80 percent or so of the manufacturers today don't have a practice. They don't have an OT practice and they have an IT team, but the IT team is is limited in the number of resources that they have. They've already got full-time jobs taking care of a whole bunch of other stuff, much less dumping this net new data stream that looks like a fire hose pointed at their head. Right, it's almost like they get alert fatigue with the stuff that comes flying out of there. So our goal is to get a managed services piece in place for them, at least for the first year, eric, until they decide that they want to build their own practice.

Speaker 2: 39:00

Um or uh, some clients, just because they're in that managed service mindset, um will flat out say, hey, I just want to hire you guys for three years, I don't even care what tool you use. You know here's, here's my requirements, and if you can, and then we just come in and run it and manage it for three years for them, yeah, and we catch new assets coming, because these tools will catch new assets. They'll catch new applications, they're going to catch new protocols, they're going to catch changes in the control systems, new networking devices, it. So we're constantly watching and building to your point of five years. You, if I see a new asset come out, well then that's going to be a trouble ticket. That's going to be created and somebody's going to have to go resolve. It's going to get assigned and somebody's got to go resolve that.

Speaker 1: 39:42

Bringing it back to the first question I asked at the beginning of the podcast about attack vectors, I think we've thoroughly mapped the landscape of how things are a bit disjointed. Maybe, Dino, you could speak to why that is on just a practical level, maybe coming from the hacker or malicious actor side of things. Why is this such a prime target in manufacturing?

Speaker 2: 40:07

Because they'll pay. That manufacturer will write a check and get it paid to get back up and running. Those that don't will spend months and months and lose hundreds of millions of dollars trying to unwind whatever got in there. Those that don't have the weather, all skill, resources to deal with it will just write a check.

Speaker 1: 40:25

And that goes back to your earlier point about the amount of money being lost per hour on the manufacturing floor. And are you seeing just a lot? Is that just a lot of malware? Is there the ransomware and things of that nature, or what's the most common?

Speaker 2: 40:40

Yeah, malware that comes in, ransomware that makes its way in through some email system that gets its hands on those HMIs out there on the plant floor. Because the HMI, that is a human machine interface and if you can't gain access to that then you're blind to whatever the machine's doing. You can't control it, you have to shut it down.

Speaker 1: 41:00

And who's typically catching those things.

Speaker 2: 41:03

Well, if you've got a clarity in the environment or an armistice in the environment, it's going to catch it on that side of the fence, or IT may have caught it on their side. So think of Colonial Pipeline, right, caught it on their side. So think of Colonial Pipeline, right? You guys are probably familiar with Colonial Pipeline. You know, in their particular case, you know when it came into their environment, the reason why they shut down on their side, on the moving of the fuel side, on the plant side, because they had no visibility, they didn't know whether they were safe or not, and because they're moving, you know petrol, gasoline and diesel and all kinds of stuff, over 5,500 miles of pipe. They shut down on precaution because they didn't know the extent of the breach. They had no idea. If you look at Clorox, they decided to fight it themselves and they lost several hundreds of millions of dollars trying to eradicate it themselves. I've got clients that have been fighting stuff on their plant floor for almost coming up on two years, right, because they're trying to get it themselves and try to eradicate it. And then you have those that will write the check.

Speaker 2: 42:04

But getting back to what Eric said at the very beginning, you know in regards to something like Stuxnet. It was a very targeted control system. There's been a couple others. There's one at Saudi Aramco had one. It was Red Typhoon or something like that. It was targeting control systems. We don't see a lot of that as much as we see the malware that's hitting the Windows system. It bleeds over into the plant floor where you have Windows tools, right Windows systems. But it doesn't mean that you mean that they're not out there. They are, it's just a lot of them is not even advertised. You've got clients out there that just keep it quiet and find it themselves.

Speaker 1: 42:44

So we've touched on it briefly and, as we're looking to wrap up this awesome conversation, I'm just wondering, as we look to the future, what are some quick ideas or bullet point thoughts that how we can improve upon this in the future, going forward both from the OT and the IT side?

Speaker 2: 43:01

Well, obviously you want it to be collaborative, right? You want IT and OT to be aligned and collaborative and recognize each other's weaknesses in regards to dealing with this problem. You're probably going to need a third party to come in and help you with some of that, and you've got to develop. You've got to bring in a security framework. You need to pick one, whatever it is, if it's specific to your vertical, if you're in critical infrastructure and IEC 62443 and NERC separate, then adhere to that. A lot of regulatory stuff is making you do that anyway, but you need to get a framework in place and you need to get the right tools and you got to bring the right people to the table, and you need to have people that have experience doing this.

Speaker 2: 43:43

It's surprising to me how many people just try to fight through it themselves, and I saw Gartner came out with a report at the end of this year that said by 2027, was it? A third or 25% of the cybersecurity professionals out there are just going to quit and go do something else. My playbook would be my choice. How come Did it say why? Because of the stress, you know, and it was just too difficult. You know, if you're a CISO. The average tenure of a CISO is like less than two years, something like that.

Speaker 4: 44:15

You know that we better get our rakes. We've got a lot of work to do when that comes up.

Speaker 1: 44:19

I thought you were going to say they were going to work on their golf game.

Speaker 3: 44:25

Eric, how about you? Yeah, yeah, sorry. I think it's the organization's understanding that they can't do it themselves. Dino, you alluded to a couple of customers that you'd worked with trying to do the cleanup on their own. A lot of the customers that we've gone into recently kind of had this. I don't want to call it an anti-contractor mindset, but it's almost like the us and them of like the internal full-time employee staff versus the contractors, who the contractors are? Essentially, they're just paid out of a different bucket. So why do you care how they're paid if they're there to help you?

Speaker 3: 45:17

And I apply it to a hospital model You're sick, you go to the hospital. The ER doctor is not an employee of that hospital. The ambulance that brought you there is not an employee of that hospital. The anesthesiologist that is making sure you're comfortable while you're under surgery, if you have to go through that, it's not an employee of that hospital. That cardiologist that's taking care of you is not an employee of that hospital.

Speaker 3: 45:51

So I look at organizations that have been running. You know, fat, dumb and happy. They have an inflection point. They need to bring in some outside assistance to help them get better, to help them operationalize and continue to run their business. But really looking at those third-party contractors as people that are really specialized in an area, that can come in and are honestly going to be able to help you faster than you could go out and try to staff it on your own, you're going to have to rely on that external assistance and you're going to have to pay for it. So that's where I think there is no free lunch in this whole cybersecurity and even IT game that we're in. Right, you just understand what you need to run your business and if you don't have it internally, look externally.

Speaker 2: 46:44

Yeah, and the plant floor is not very much different than what you just described, eric, because a lot of them don't have their own engineering teams. They don't build those machines, right? If you want a packaging line, they don't engineer and build that packaging line. They go out there and get the three or four or five different groups together to build that packaging line and the engineering teams to do that, right. It's the same concept, you know, just different.

Speaker 2: 47:03

But yeah, that's why you end up with a lot of different components on the plant floor of Siemens and Rockwell and Honeywell, and you know, hirschman switches versus Ciscoisco switches versus, you know, phoenix contacts or red lions or whatever you know, because of the different groups that are building that equipment and just bringing it into your plant. So the very similar analogy that he gave, that's the plant floor, um, and to your and it's very standardized, right. They think that they can standardize everything and they're just accustomed to working from that premise and you're just not going to get that on the plant floor. Sorry, it's just, it's not that, not that way. Especially when you got 10,000 assets out there right on and just one plant, you got 10,000 assets managed by gosh, who knows who, right?

Speaker 1: 47:49

it's really cool talking with you, dino. Uh, it feels like you have a really good beat on what's going on out there in the manufacturing world and and it's really interesting for me, even just being in a completely different industry in the entertainment and audio video it makes a lot of sense and it just impresses me on how these things even get accomplished. There's so many moving parts and there's so many intricacies to all of it. So hats off to you for being a leader in that space, and I'd like to pass it around for any final thoughts today before we wrap up.

Speaker 1: 48:23

I think we could stay on for the rest of the afternoon, but I was going to say, like we didn't even get into like incident response and things like.

Speaker 4: 48:30

Yeah, like emerging threats yeah.

Speaker 1: 48:38

Maybe we'll have to have you back down the road, dino, to talk about, uh, what could drill down on something like incident response, particularly how how you handle that. But, um, at the very least, eric your question that you always ask at the end of the podcast. Do you want to? You want to do it?

Speaker 3: 48:47

what's my question?

Speaker 1: 48:48

okay. Are you planning on going to any security conferences? Do you know? Do you go to? Going to?

Speaker 3: 48:53

any security conferences? Dino, do you go to any conferences? Security conferences I do.

Speaker 2: 48:59

I'm actually sitting on the steering committee for Manusec that's coming up here in May in Vegas. I also go to Rockwell Automation Fair. It's a big one that we attend S4. I've been going to S4 in the last few years down there. I don't know if you're familiar with Dale Peterson that runs S4X out of Florida once a year. It's a good one. So from a cyber and it's specific on OT cybersecurity, it's called S4. He's been running it for over 20 years. Nice, that's a good one to look up. I haven't really hit RSA or Black Hat. To be honest with you, I tend to lean towards the ones where I'm focused on the manufacturing organizations.

Speaker 1: 49:45

Do you do a booth at your conferences? You go to Dino or do you just go and do the cocktail hour? I speak a lot of them.

Speaker 2: 49:51

Obviously, we have clients, the cocktail hour, I speak a lot of them. So we take obviously we have clients that are there partners We've got a lot of partners, so yeah, Because I like your musical. I use the instrument a lot. It's like I can give you a guitar. The question is whether you know how to play it or not.

Speaker 3: 50:08

It comes with these tools Like you.

Speaker 2: 50:10

Take a clarity, for example. Right, there's those that know how to play that instrument and then those uh, don't you know, and we've got. People have been working with this stuff for almost a decade, so it's going to be hard to beat them. If you're just starting today with the tool you might as well leverage, somebody's been working with it for for several years you know that's a great analogy, yeah well.

Speaker 1: 50:32

Thanks so much for your time today, Dino. It's been a great conversation. It's been very informative. Yeah, Thanks for being our guest. I'm going to wrap it up here, guys, and then we can debrief and, if you'd like, you've been listening to the Audit presented by IT Audit Labs. My name is Joshua Schmidt, your co-host and producer. Today, our guest has been Dino Busolacchi and he was speaking about OT and IT and the convergence with Nick Mellum and Eric Brown from IT Audit Labs. Please like, share and subscribe wherever you source your podcasts. We have video every two weeks on YouTube and Spotify and we'll see you in a couple days. Thanks so much for tuning in. Don't forget to like, share and subscribe. If you have a moment, leave us a comment on a YouTube channel or give us a review on an Apple podcast. It really helps others find the show. Thanks so much for joining us and we'll see you in the next one.