A (2:47)

Absolutely. I went through the challenges of how do you get it off site, how do you get it backed up, how can you actually recover it? And then it was, oh no, you know, somebody deleted it and, but they're supposed to be able to delete it, but that wasn't me. So yeah, I've gone through so many times, like say I'm the guy that was in the room when it went bad and learned from things we just didn't know about at the time.

B (3:10)

So you're, you're a previous diamond in the rough now you're a pure diamond.

A (3:14)

Right?

B (3:14)

Because you've, you've got all the, all the hard spots knocked off. You know, you learn through, through going through it.

A (3:22)

Oh, absolutely, yeah. Absolute hive knowledge here. Learned and benefited from other people's mistakes.

B (3:28)

That's the best way to learn instead of making your own. So hey, but let's look at this from the perspective of an executive, right? Let's say I got a mid sized company because those are probably a mid size and small companies are probably the ones that are at highest risk of this sort of thing. Because big corporations, they've got, they've got big doctor and VC business continuity programs, they've got the money that they can do it. Right, but what about the mid sized guy? The small guy as an executive, what do I plan for, why do I plan and you know, how do I get started? What do I do? Tom?

A (4:08)

Well, honestly, most organizations think they're protected or they know they're not protected and both are dangerous. So if you're not protected, obviously you've got to do something. And ultimately I don't think you are going to have the specialty on your staff, your main IT person or people. They may be really great at things. Like I was a generalist, you know, they're knocking eight out of tens and everywhere. If it's a decathlon, man, they're winning because they're hitting the scores across the board. Board. But with Dr. It has to be more targeted. You need a 10 out of 10 in it all day long. So if, if you don't have anything, you've got to find someone to help you. This is not something you learn on, on your own. If you have something and you think you're okay, the honesty is you're not. This is an ever evolving need. It is organic because everything in your environment changes. Think about how many times someone gets a new phone. Well, how many times is there a new security Patch. How many times does someone save a document? It's forever changing. So you have to have the mindset as an executive that unfortunately we don't like that line item cost that says the R&IT services, it's forever in your budget. It is ever growing, ever evolving. And so that was my advice to any executive is it's not a one and done. You didn't put in a new operating system and you walk away for three years. It's this is every day being vigilant about it.

B (5:38)

So is there some point in that there's a return on investment you have to be worried about? I mean, there's a certain amount of risk that maybe you're willing to take for disaster because it could get very expensive, right. To do it, right?

A (5:55)

Oh, absolutely. We're great at spending people's money in it. Give me an open checkbook. I had a client that was like, well know, can you give us a price with this? And you know, I joked, I was like, no, I'm going to spend all your money. Really what you have to look at is business drives this, okay? And we forget that. We let the IT people drive us into fear. Oh, everything you know is going to destroy. They're not wrong. But a business impact analysis is really, really important. And I'll give you a quick, high level example of something. Let's say you work in law, I worked at a big law firm. And people would say, what's the most important thing? Billing. So all these accountants would get together and talk about how quickly invoicing had to come back up. The reality is from a business impact analysis, okay, accounts receivable goes out a couple weeks. Big deal. What do we need to do? Accrued billing, who's talking, who's writing, who's producing revenue? So now you look at those systems and that's true of Dr. What we think is I need it on now. Do you really? So it starts with the business. The business drives this, right?

B (7:00)

I mean, you've got to prioritize then and then put a time. It sounds to me like put a time frame on it too. Like you said, accounts receivable, two weeks is okay, but you know, accounting, you know, accrual, I want that in six minute increments.

A (7:18)

Right, exactly.

B (7:20)

All right, so, all right, that makes sense. So that's a good place to start is looking at your business and saying time sensitive. Like, I'll give you a great example. When I was cio, I said, hey, we've got an email. We were running our own exchange Server. And we were. It needed an upgrade. It absolutely needed an upgrade. It was cobbled together. And I said, it's going to cost us a quarter of a million dollars to upgrade this thing. Three weeks later, it goes down. There wasn't enough money in the company's coffers that they were willing to offer me to get that up and running right away. Right. So time sensitive. Yeah. If your company runs on email, then having email up is, Is critical. Right.

A (8:15)

But I think it comes down to two concepts when we talk about it from an executive and it loves to use these acronyms. And so I'll say them and then I'm going to tell you what they mean. They'll make sense. All right, so you have RTO return to operation. How long can this system be shut off? I need it in five minutes or 10 minutes or two hours or three days. Right. So your email was a very low RTO, 15 minutes. Right. That accounting system. Nah, two weeks. I mean, we're not shooting for that goal, but we're prioritizing. If everything's on fire, nothing's on fire. Then we have recovery point, objective rpo. Basically, what point in time does the data come back? So I turned it on within 15 minutes. Yay. How much data did I lose?

B (9:02)

Oh, interesting.

A (9:03)

Those are the two factors. And so when you talk about cost and analysis and we can run up the gamut on this, think about this. You could say, no, I need that email up with zero data loss and I need it to turn on in two minutes. There are systems that will do that. What if it was a million dollars? Suddenly you'll get sticker shock and go, you know what? Maybe 15 minutes isn't so bad if that's only $15,000. So again, your business is going to drive it. And what's the takeaway, executives, is this. You need to work with someone who helps you make a bad informed decision. In other words, you are aware of all options. You weigh your risk analysis. You know what you're gambling when you go into it. And when. And if it happens, you're prepared because you chose that. You may not like it, but you actively chose a situation. It's a better place to be than a, you know, poorly informed. I didn't know exactly. I just didn't know. For $30 more. You mean I could have.

B (10:03)

Yes.

A (10:03)

I should have told you.

B (10:05)

Well, this, this sounds like this is going to take some time to put together a disaster recovery program. It takes time. It's not like something I could do in an afternoon.

A (10:16)

No, and when you work with somebody, they'll really help you slipstream it. When I get involved with clients, one of the things that I like to think about is we have to protect the now, even if it's not the best solution. We need a solution. Something is better than nothing.

B (10:29)

Something right away, whether it's exactly. Got it.

A (10:32)

Minimum viable product, as we like to say in those product roadmaps, right? Get something into production. At least you have something. You know, it's not ideal. Now you can roadmap it. Now you can ratchet it up. I have no backups. Put a backup. Oh, but somebody could get in and delete them. You're right. But at least you have layer one covered. You have something. So some things can go in relatively quick for people, and that's what somebody should really plan at is it's a phased approach. It's a roadmap. You won't have every bell and whistle if you watch cnbc, you won't have every tick box marked. But you know what? When people don't take action, they blink away three years. When they take small steps, in a year, it's completed. It's kind of like, how do you eat an elephant? One bite at a time, at a time.

B (11:17)

So it's. So that's your approach. Your approach is to say, let's. Let's do it piece by piece, come up with our plan. We walk through the plan, adjust the plan as. As priorities might change or as new threats, you know, come up, right? That. Because there's new threats to disaster. Disaster recovery is not just a tornado hit my building. There's lots of different, you know, disasters, right, that I can talk about.

A (11:45)

In fact, that's kind of the rarity nowadays. I lost power. I. I had a tornado. Most people have some servers somewhere. Is it in the cloud? Is it in the Colo. They're providing you power and uptime and. And generators. You're not buying those things. You're not building it yourself. And if you are, stop. Unless you're, you know, a Fortune 100. But go to somebody that does that, there's reasonable costs to that, and it's on. It's the human error. It's the malware. It's the security. Like you're saying, those are the ones that we're mitigating. So when you think of Dr. It really goes hand in hand with security. They interlock. You cannot have security without doctor you cannot have doctor without security. They are just like an infinity loop together.

B (12:29)

All right, so that's really interesting. Because you talked a little bit about cloud or colo. They handle some of your disaster. Well, they handle more business continuity than disaster recovery.

A (12:39)

Right.

B (12:39)

They have backup generators, they have backup Internet connection. They've got those things pretty much covered. But that's not the same thing as having your own disaster recovery business continuity plan.

A (12:53)

Right.

B (12:53)

Because cloud service providers go down. People don't believe that, but they do. And they actually, their, their percentages are not great right now.

A (13:06)

No talking.

B (13:06)

Maybe only 99, 98% uptime. What?

A (13:12)

Well, you're, you're talking denial, right? You look at having to architect your own system and go, I don't know all these things and I'd have to spend all this money if I just write a check. It's absolutely taken care of.

B (13:23)

Taking care of.

A (13:24)

Rule number one. Exactly. Rule number one. Zero trust. I don't trust them. They're my friend, they're my vendor. I'm not mean to them, but I'm a paid pessimist. I have to have another route even so far as. Well, I'm in Amazon and I have multiple zones in Amazon. What if some random thing goes wrong in an Azure, an Amazon, a whoever, and I'm not picking on them, but what if that one thing goes and then you're down? What priority do you have? You have to have a backup outside of that main production space. Backing up to yourself is no different than when I started in this business is you'd put your little backup tape in your server, it back up all the data and the admin would stick it on top of the server and go home. Well, where's your backup when it burns down? What happens in that cloud? If the backup is in the same cloud, you're not really winning. You can have one and should have one there, but it's the tertiary that makes the difference outside of the environment, something else. And you have to plan on it.

B (14:35)

You know, I'm glad you brought that up because there was a, there was a shipbuilder, I won't name who they were, but they're in Pascaloosa, Mississippi and their backup, they had a business continuity disaster recovery plan. They had a backup data center five miles from their main one. So when the hurricane came, it wiped it. It wiped it all out. Yeah, well, luckily they had tape backups further on, but you know, it ended up being somewhat.

A (15:06)

Okay.

B (15:06)

It took them like four days to get back online, but I mean, their shipyard was completely destroyed by the hurricane, so I mean, some of the stuff they didn't have to worry about for some time. But it goes to show you've got to be smart about, about that. You know, backups in the same cloud service provider, probably not a great idea.

A (15:29)

Think about your SaaS as well. Well yeah, PeopleSoft, you know, PeopleSoft, everybody has them on prem then everybody's like they do it in the cloud and they just do it for us. And they say they do whatever. Where's your audit? Where's your test? How do you really know what they're doing? And you may think, oh, it's no big deal. You have to understand that right now in every business your data is everything. If tomorrow you had access to absolutely no data, no matter what system you're using, would your business survive? You could survive downtime. But what if it never came back? That's the big challenge with these SaaS vendors. These are your Atlassians making Jira and whatnot. Love those products. Major software developers, if they lose that, they're done. There are products out there that will back that up. Outside of it, even if it's kludgy to come back again, remember it's minimum viable product. You have something that shipbuilder would be happy if the data was two weeks old, a month old, a month old, zero. How do they even call their customers to go hey, we're back online or hey can we get this shipment? Imagine you can call no one. That's the scariness. It's complete obliteration.

B (16:40)

So all right, so let's talk about our reliance on these SaaS providers a little bit because yeah, there is this illusion of protection and maybe sometimes we relegate, we relegate that to them thinking, you know, we don't have to worry about it anymore. What you're saying is we do, it's still our, we have to remember it's still our stuff and it's still important to us. So we've got to, we've got to do a backup. So is it easy for most of these SaaS providers? Do they all provide a way of backing up outside of their system? And if so, how do I go about learning about how to do that?

A (17:26)

They actually don't. Most of them don't. Some will. I'll use like Salesforce. There is a way in which you can extract some files and somebody's going to have to re architect the environment to come up. Then there's third party tools. Veeam has them I think SaaS is sure has some, there's some other providers out there that you would subscribe to and hopefully Use someone like me to architect it. Where? Listen, it's not that much. Let's go in, let's back it up and have it recoverable because we have to back up and we have to recover it. It's a two stage double headed sword here, if you will. So even if you have the backup, how do you get it back? Third party is really the way to go right now. I'm not really aware of many that make it very clean to be able to do that. So it is third party data, right? Like Salesforce's spreadsheets upon spreadsheets upon spreadsheets. The last time I looked and I'm dating myself at about three years at that mark because now there's a third party tool we use and we're just like hooking it, just so.

B (18:27)

Got it.

A (18:28)

It's, it's hard to architect your solution with one tool. What I'm going to say is one tool will do most of it. Most of the pain points are those SaaS providers. It's finding which backup company software do I use. Do they have this product in the roadmap? Is it live? Oh wait, somebody else has it. So, okay, I've got vendor number one. I love them, but now I need vendor number two. And one day maybe vendor one releases it and I roll it back into single pane of glass. One vendor architecture. But you're going to have to be a little bit robust going back to the business systems. You have to know the data that you touch and where it lives to be able to protect it. So executives, all your reports, head chiefs and directors, they need to know what systems do we use online or not and how are we backing up. It comes up with every system we design, not just the newest flavor. Dr. Is part of that conversation.

B (19:22)

Okay, so that's why, that's why it's a continuous thing. You need someone that's sitting there saying, all right, we added this new tool, new data, who uses it. All that needs to go into a disaster recovery plan, right? It can't just be in someone's head, Right?

A (19:41)

It's the Superman theory. So how many times in our environments do we have that one person that looks like Clark Kent every day of their life, right? And the emergency happens and boom, they tear open their shirt and they're like, I'm Superman. Well, think about employee turnover lately. It's absolutely skyrocketed by choice or not. All of a sudden you're just used to a Superman being there. But remember, you don't know who Superman is. You've trusted. He's in your trenches. What if Superman went to work somewhere else? Now you're in the middle of the emergency, Superman's not there somewhere, like you're in trouble. So you're right. It's documented. So now it's the people on the process side. It's not just I bought software. It's written down, it's executed, it's tested and it's ever organic. Growing as we do new things. Every new system, every upgrade is what do I do for Dr. Is it backed up, is it protected? The end.

B (20:42)

You know, that reminds me of a story. I was working at Lucent Technologies and we, we had a guy on our team that was a cyclist and he got hit and by a truck and killed. Very, very sad. He was a critical person on our team and he had a lot of the process on building our software in his head.

A (21:06)

Oh yeah.

B (21:08)

And so we, we coined a term called the trucking factor. I don't know if you've ever heard of it.

A (21:14)

Yes. Yeah.

B (21:15)

Trucking factor of one is dangerous, right? One person has the knowledge or one. And, and that's very true in disaster recovery. If you got one person that knows how to do the disaster recovery, that's bad. You need another person or you need at least documented. So we always said one and a half minimum half is documentation, but two and a half is whole lot better than, you know, one and a half having, having that there. It's a sad situation. But these things do happen in, in life, right? Especially if there's a true physical disaster. You want, you want someone, anyone to be able to help, help you get back your business back to where it needs to be.

A (22:00)

Well and I think that's where kind of I found my niche in the world was looking at fractional backup and disaster recovery. So rather than hiring the full time person, you're going fractional, you're paying a fraction of the cost. Somebody's spending some time on your site, if you will, virtually, and this person's working and their specialized target and you may not need them for 40 hours, you might need them 10 hours a month. However, now multiply that into say, well, when you hire a firm, maybe you're getting 3, 4, 5 fractional people. So you don't have that trucking factor. So not only did you fill the seat for less, but now you have the elimination of trucking. As long as they're transparent and documenting. Because don't even trust your doctor vendor, if you hired me tomorrow, I'd be like, worst thing you could do is trust me. You should respect me, but zero trust. I'm not insulted.

B (22:51)

You're right. Yeah. Zero trust. But. What was the word you. Zero trust. But.

A (22:57)

But respect. Respect. Right.

B (23:00)

There's a new shirt. I'm going to do a new T shirt. You can get. You can get it on my. On my T shirt shop. Right. Zero trust. Zero trust. But respect. I kind of like that now. And. And I see where you're coming. Ultimately, it's my business. I'm responsible. Right. So if I don't want to spend the money, then I shouldn't be mad that I lost everything because I decided not to spend money to do it.

A (23:27)

Right. Well, and look at your contracts, too. From an executive position, when legal is going through and redlining, I guarantee they're looking at a lot of SAS vendors with boilerplate contracts. First of all. First of all, my corporate attorney would tell you you can negotiate several of those things people don't understand, but come back to the value of loss. If you're paying a vendor a couple thousand dollars a month, and let's say you're a pharmaceutical company, and I had one in this situation, you know, $6 billion worth of intellectual property, and they're spending $1,000 a month, and right away they're like, well, what if we lose our data? You're our last great hope. Well, for $1,000 a month, nobody's going to insure your data for $6 billion. But even if they wrote you a $6 billion check, you're basically just cashing out at this point. You're paying everybody else, paying everybody, walking away and trying to recreate. So you have to look at your limit of liability. And you mentioned about your incident. I can tell a different Persona from a person who has never been in an incident and how much they're willing to spend and what they're willing to go through versus the person who's gone through it, which, like you said, unlimited budget. I don't care how long it takes. Get us back alive.

B (24:38)

Yeah.

A (24:39)

The incident is worse than the preparation.

B (24:41)

Yeah, absolutely. So, I mean, so how do you. How do you determine how much to spend? Because, I mean, like you said, I could spend an unlimited amount. As a. As an executive, you just weigh that risk, I think. But where do I start? That's the hardest part. I mean,

A (25:00)

it's your revenue. You have to figure out how much you're losing per minute, hour, whatever. Okay. The very large manufacturer, they're all over the world. And one of the things that I've Noticed about them is they have surprisingly low return to operation because their factories are segmented off. If a factory is down, it's not really in my area if a robotic arm goes down or something, but they have their own backups but for those systems. But let's just say that data isn't flowing into the factory. Well, they're preloading seven days worth of manufacturing. So they essentially could have seven days worth of time. So do they need those cutting edge systems? No. Do they need the secondary redundant crazy levels? No, they just need to say, listen, if this goes down, can somebody get me this piece of equipment within 24, 48 hours? Can somebody rack it up? Can somebody recover back two days, three days, we're okay. If I'm running a dot com processing transactions per hour, you know, if I was like PayPal or Stripe, think of how much they do in a second. They need to have it. And so it comes down to that. Business impact analysis. What does it cost us when we're down? What now when we build Utopia, how much does that cost? Can we roadmap to Utopia and do it in stair stepping over a couple years and realizing that, listen, these are key critical systems. Like you said, email was everything. Well, that's user shock, right? It really wasn't generating revenue per se, but it's inconvenient, of course. But on the flip side, you know, what was the revenue driver for that business? So it's coming down and understanding your business. Then it's coming down to what systems generate those revenues and designing the architecture of first up, second up, third up, full roll, if you will. And that's where it's then going to come into your planning of. I mean, you have to think about where are people going to work. We all assume at home, but do they have access to those? That's, that's the, the planning side of it is what will we do sort of thing.

B (27:14)

Well, I think it's interesting because I, I'm, I'm wondering if Covid really helped people understand business continuity better because holy cow, all of a sudden, Worldwide, everyone had to start working from home like overnight.

A (27:31)

Yeah, they sure did. And our phones ran off the hook overnight. Customers that we couldn't beg them to test anything. Like, you've been a customer for five years, can we please spin up a sandbox to test these things so you can see wouldn't return our calls. We're suddenly beating down our doors. We need it now, we need it now. And it was like, yeah, you do. I wish you would have Planned, so it's not an emergency on my part. But you're right, that scared a lot of people. However, they're starting to get lax.

B (27:57)

They're getting lax again.

A (27:59)

Yes. Yep.

B (28:01)

Yeah, I'm seeing that. I. Yeah, I think you're right. We've seen, we've seen some of that happen. So this constant continuous doctor investment is something that you should write into your business plan. It sounds like. And saying, hey, this is a. It's almost like buying insurance. Almost.

A (28:23)

Correct.

B (28:24)

Got it.

A (28:24)

It's buying insurance, but you're the one arc or executing the insurance policy. When you have an insurance and you have to beg your vendor, please pay me. At least here you've kind of. It's like self insuring is what I would say. You've got this chunk of money sitting aside and you need to go to the doctor. You just draw off of it, and every month you're putting it back. You know it's there. That's what the R is. It's self insurance insurance. Because if you run into a ransomware situation, this is very important. You know, first of all, it's very randomized. Nobody is targeting you. If they are, your goose is cooked. They've been doing it every which way.

B (28:56)

Right.

A (28:56)

But when we come into the randomize of a ransom and they want to be paid, the fact is we're finding out several things. Number one, even when people pay now, they're not giving out the codes. They're like, hey, oh, you gave me a million dollars. That's awesome. Now I want to. So you're lucky if you even get your data back. Second, you're now targeted because you're a known payer. So everybody's like, hey, go after Tom. He just paid. Awesomeness. So you have to be able to build your insurance with this. And one way or the other, you're going to pay. And this is a risk minimization. It's not a fun and sexy topic. When you put out a new web interface, everybody loves it. When I do my job, nobody knows I even work.

B (29:34)

Well, if you do it right, yeah, you're invisible.

A (29:37)

Exactly.

B (29:38)

Right. Interesting. So how do I prevent. Ransomware is a scary thing, right? Because they've, they've infected my data. They've quarantined it, they've encrypted it and all that. How do I know how far I need to go back to bring back my data so they're not just right back in again?

A (30:00)

Well, that's the challenge. So modern systems, and I'm a big veeam proponent. Just to let everybody know there's some key players, but they've really done a solid job for the longest time. But what I would say is a lot of these technologies, including Veeam, they have the ability that the moment you're taking a backup, it's mounting it and scanning it and it's looking for malware.

B (30:21)

So they're checking. So that's a check for malware as you're storing your backups.

A (30:27)

Correct. That's a big factor to it. A lot of times what I've seen is more of you've been surveilled for a long period of time and somebody now decides to enter in through a backdoor and now they begin the encryption. So sometimes it's time to time delayed fuse where it's automatic. A lot of times it's, oh, I've gathered enough information, let me enter. Once I enter, let me impersonate. Oh, now I'm Darren. I have Darren's God rights on the network. Now let's. What's the first thing I'm going to hit those backups because they know if you have insurance, they need to deplete your account.

B (31:02)

Right.

A (31:03)

So that's where we get into the specialty of backups. And saying how do we protect our backups even from ourselves? That's kind of that Tier two.

B (31:11)

Got it. All right. This has been, this has been. It's bringing back memories. A little bit of P. PTSD from being a cio. Yeah, right. Because everything, the whole weight. And people don't understand this about CIOs. The whole weight of the organization is on your shoulders when it comes to information management. And it, it can be difficult, especially nowadays. So thank you, Tom for coming on the show, going through this. If people want to learn more about what you do and how, how that all happens, where do they go to find out more?

A (31:49)

So we have a website, differentdev.com so it's like different development. Com. You can find me on LinkedIn. Thomasjmay. I'm out there under different dev. Clearly we'll have some backlinks between us. Easy. Email Tom may like the monthifferentdev.com so I'd love to speak to you. I don't charge for every minute that we're out there. It's prospecting and learning what you need. I've learned sometimes in business, Darren, you know, you make a connection, you give someone a penny's worth of advice and three years later they've earned your trust and they come back. So if you're worried about it. Let's just have a conversation and see it doesn't cost you anything other than time.

B (32:33)

Well, hey Tom, thanks for coming on the show. This has been wonderful.

A (32:37)

Hey, I appreciate being here today. Thank you. It was a great time.

B (32:44)

Thanks for listening to Embracing Digital Transformation. If you enjoyed today's conversation, give us five stars on your favorite podcasting app or on YouTube. It really helps others discover the show. If you want to go deeper, join our exclusive community@patreon.com embracingdigital where we share bonus content and you can always connect with other change makers like yourself. You can always find more resources at Embracing Digital.

A (33:09)

Org.

B (33:10)

Until next time, keep embracing the Digital Transformation.