Transforming Your Priorities

00:02 — Jamie Parker
We all make choices every day, some more consequential than others. When it comes to digital transformation, few choices stand alone. Imagine you make a choice thinking you know what the outcome will be, but find out later—too late—that it had consequences you didn't anticipate but could have. Now imagine you're working within a government agency or financial institution, and that those consequences have a profound impact on the lives and livelihoods of countless people. These organizations are understandably slow to change because the stakes are higher. Being cautious and methodical isn't actually a failure, even when working with an industry whose recent motto has often been to move fast and break things. But on the other hand, there are avoidable setbacks that make it seemingly impossible to move it all.

00:53 — Jamie Parker
In this episode of Code Comments, account executive Adam Timm of Crunchy Data shares his experience working with government agencies to overcome hangups with modernizing their database operations and what you can do to better anticipate the potential consequences of your development choices.

01:15 — Jamie Parker
Adam Timm works with the Department of Defense to modernize their database systems.

01:20 — Adam Timm
The government doesn't have necessarily the best track record with delivering capabilities on time. And so, quite often, when a capability is slated to be delivered is not when it actually gets delivered. Oftentimes it's months, sometimes years late, depending on the scoping and complexity of the application that they're trying to get delivered. And so the fundamental value proposition of digital transformation was to turn that on its head and be able to start delivering capabilities sooner.

01:51 — Jamie Parker
Do for the Department of Defense what digital transformation has done for countless other organizations—reduce the time to develop and deploy, allowing for more iteration and incorporation of user feedback. It's been done before. Why not for the government? Well, there's an expected standard for reliability that might be quite a bit higher than for other environments.

02:14 — Adam Timm
And that outcome is delivering mission capability to the warfighter in as short amount of time as possible, but with all the enterprise features that the mission has come to expect from your agency or service. So if you think about it, getting software from development into production, it can be a pretty straightforward task. But once it's in production, especially in a mission environment, the end user needs to have assurance that the application's going to be running, that the data's going to be available, that if something happens, there's going to be minimal or no disruption to the end user.

02:49 — Jamie Parker
The situations in which some of these systems are used have much higher stakes than at the average companies, as are the security requirements. That can put a damper on the speed at which working systems can be upgraded to something new. On top of that, you can layer the reputation that databases themselves have for complexity and there comes a certain reluctance to touch them.

03:13 — Adam Timm
Databases were always viewed as these big, clunky, monolithic type things that required mainframes and massive servers to run, and they were complicated to install, and once they were installed, they were very fragile to maintain, and upgrades were a pain. Upgrades were things that had to be planned out in a methodical manner.

03:40 — Jamie Parker
To run properly, databases needed a lot of uptime. They needed stability. With a monolithic server, that wasn't an issue. Then, IT modernization by way of containerization came along. Since containers come and go by design and distributed systems are governed on the premise of replaceability, it left a big question mark on the function of the database.

04:04 — Adam Timm
If an application dies or if your container dies, well, we're just going to tear it down. We're going to spin it back up from a known good state. And your application is running, the platform's going to manage all that for you, and the developer just worries about building and deploying their application. But if a container gets destroyed and rebuilt, well, any data that resides within that container also gets destroyed and you start from a clean slate.

04:27 — Jamie Parker
That's a problem. You can't run a reliable application if data loss is a routine outcome of your system working as intended. That would lead to errors, missing information, and worse. That's unacceptable for most organizations. It's definitely a non-starter for the Department of Defense. Luckily, that problem is overstated. It turns out there are ways to run a database with a cloud-native system.

04:54 — Adam Timm
The legacy perception of databases combined with the initial philosophy of running applications and platforms, I think leads to that perception and that myth of Kubernetes is just not a good environment for running data. And in reality, if you are using database technology, i.e., Postgres, that's been architected to take advantage of all the capabilities of Kubernetes and handles data in an appropriate way to make sure that the data persists, even if your container disappears on you, then Kubernetes is absolutely the right environment to handle data-intensive workloads.

05:35 — Jamie Parker
So there's a technological solution to the alleged database roadblock. Databases can be integrated with Kubernetes, but the solution being available isn't the end of the road. You still need to make sure you get it right. Crossing your T's and dotting your I's is a big deal. And for a government organization, you're likely going to have someone checking your work. So you've done the job, you built the app, connected all the systems, and ensured it'll survive all but the absolute worst case scenarios. All you have left to do is to go through an authority-to-operate checklist process. The ATO checks for clean CVE scans, making sure the system is architected according to the security technical implementation guides, which are often called STIGs.

06:22 — Adam Timm
How do you configure a database according to proper security guidelines? And then it also looks at your overall business and mission process that goes above the code level of what's your continuity of operations plan. So if something happens to the datacenter or to a particular location, what is your plan for failing over or what is the amount of time it will take for you to recover so you're mission-ready?

06:51 — Jamie Parker
That's a lot of complex questions to answer. Any system that satisfies those requirements and more is going to need careful planning and execution.

07:04 — Jamie Parker
You need to make a lot of choices to build a modern IT system. As we've previously covered, data isn't always the top consideration when making decisions.

07:14 — Adam Timm
It didn't really fully dawn on me until I had left my agency and joined Crunchy Data. Oftentimes, the data is the overlooked aspect of digital transformation. There's a strong focus on containerization, a strong focus on being able to deliver new features, but the data tends to be a bit of an afterthought in a lot of cases from a perspective of developers choosing the database that they prefer, not necessarily the database that is best from an organizational perspective.

07:51 — Jamie Parker
Adam is going to make this point several times. When making choices, it's vital to consider which option is the best for the organization, not simply for your own convenience. If you choose a database because it's what you're familiar with but don't have any idea about what the operations team would prefer, or if it's the most efficient tool available, you might be setting up your organization for some pain later on, and that could come to a head during that last step.

08:21 — Adam Timm
When it's time to go through the final ATO process, they start reviewing that, and oftentimes, they look at that and they go, well, wait a minute. You built it—let's just say—a developer may have built something using Azure services, but those aren't available on the military domains. And so a lot of times oversights like that can slow things down, or they may have built it using one particular database on the low side, and then when it gets delivered to operations, the operations team doesn't have a ton of experience operating that database. So a lot of times, systems aren't optimized on the production side for those things. So organizational misalignment can happen through just fundamental oversight on asking some of those lower-level questions.

09:11 — Jamie Parker
So keep in mind what's best for the organization. Also keep in mind what kind of security and reliability requirements you're going to need to address, ideally early on in the development process so you don't get stuck having to make huge changes when it comes time to get that ATO certification.

09:30 — Jamie Parker
A big part of those considerations is finding whether the off-the-shelf components meet all of those needs. Adam has seen agencies give developers the green light to develop on certain platforms that have tools available to work with, but not all of those tools were actually cleared for use.

09:46 — Adam Timm
Developers would come in and there'd be a handful of services available in the development environment, but only a few of them actually had the enterprise capabilities built into them. But not every developer was choosing to use that. They'd use another single-node instance because the developers didn't understand that at some point they were going to have to move it into operations. They thought just because it was built in the platform, everything in the platform was built to the same level. And so when it came time to actually move their application into production, they started getting asked questions like, well, what's your data assuredness? Is your database configured according to the STIG?

10:24 — Jamie Parker
This level of mismatched expectations is apparently kind of common. And when faced with those questions, the developers don't always have good answers.

10:34 — Adam Timm
Oftentimes, that answer was either they didn't know because they didn't have control over that service, so they didn't know how it was being built or configured, or two, it was no. And so quite often, they would have to go back and re-architect their application to use the Crunchy service or another one that met the needs.

10:52 — Jamie Parker
But you can't just blame one side of the organization. Expectations need to be clear early on, especially if the tools made available don't meet all the criteria to pass the final checks.

11:05 — Jamie Parker
This all makes it sound like choosing off-the-shelf systems are more trouble than it's worth, but that's not the case. They just need an upfront level of scrutiny according to the standards of your organization, knowing those at the outset can help you make an informed decision, or you could choose the other option on the table.

11:23 — Adam Timm
A lot of times, they'll default to the, well, if we build it ourselves, we know what's going into it, and therefore we have complete control over it. Which on the surface sounds right, but once you start digging into it—especially from the open source side of things—an organization like Crunchy, we put a lot of time and effort into making sure that we've got a consistent build process that everything is done according to a defined standard.

11:48 — Jamie Parker
Adam is arguing against building things in-house. There's some value to choosing the vetted, certified, supported system that has a whole organization and community behind it to find and address problems. After the break, we're going to evaluate the option of building a system yourself.

12:19 — Jamie Parker
So why not build the system yourself? A lot of developers and in-house developer teams have the capability to build systems, even enterprise-ready production-ready systems. They can even build it to the organization's standards to pass the ATO requirements with flying colors. But there's a cost to consider too.

12:39 — Adam Timm
There's open source utilities out there that they can bring in. They can architect it out themselves and they can run it in the platform. But I go back to focusing on what are the outcomes that the developers are supposed to be focused on on behalf of their end customer and their mission user? And is it in the best interest of the mission—not the developer, not necessarily the end user—but is it in the best interest of the mission for the developer to spend development cycles building all that stuff out themselves? Or is it more in the interest of the mission to take something that's off the shelf, ready to go, and supported by an organization that is going to ensure that all the vulnerabilities are patched from a security perspective, that there's new features being constantly introduced to benefit the mission?

13:30 — Jamie Parker
Spending development cycles means spending time. For this project, likely a lot of time. And timeliness is something that Adam is helping the Department of Defense with. But timetables aren't the only considerations when building a system in-house.

13:45 — Adam Timm
If a developer is going to go out and they're going to build a highly available Postgres architecture in Kubernetes, can go out and use all the same components, open source components that we would put into our distribution, but they might architect it in a slightly different way. So you've got some variability there that only that developer knows how it was architected. So the supportability of that becomes suspect in the future.

14:09 — Jamie Parker
Those developers are likely not going to be around for the whole lifespan of the system they've built. If any issues come up with their unique approach to the system, hopefully they've extensively documented their work for the next team to be able to understand the system's inner workings. To some, documentation isn't an issue, and the time spent is worth it for the upsides of building something to the desired specifications. There's also the argument that it could save the organization money by avoiding licensing support or subscription fees. Adam argues that in the long run, it's likely going to end up costing more to build something yourself.

14:48 — Adam Timm
Is it in the best interest of the mission for the developer to spend the time and effort to build all that stuff out themselves with the goal of once they build that process, then the government "owns" it and it'll save the government money in the future, but at what cost? Yeah, it might potentially save the government money in the future, but in reality, I can go down the whole rabbit hole of that actually isn't the case because if it's saving you money, then odds are you're paying for it with time.

15:18 — Jamie Parker
What you're not paying in cost to vendors, you're paying in development time, which is also a scarce resource. While your developers are working on this system, they're not working on the rest of the application.

15:31 — Adam Timm
I'm a skilled developer, Postgres is open source. There's all this stuff out there. I can put all this stuff together myself. Sure, they can, but is that what's really in the best interest of the mission?

15:45 — Jamie Parker
So the questions we end with are the ones we've been considering the whole episode, and hopefully those you keep in mind throughout your own digital transformation. How do you best serve the organization and the end user? For organizations with a reputation for inertia, how do you overcome that reputation for blowing past deadlines, especially when working with notoriously difficult components? If you choose an off-the-shelf solution, make sure it complies with the organization's security standards. If you choose to build your own system, make sure you do it in a way that you can support it, and don't take too much time away from the rest of the work you have to accomplish. There are a lot of weighty decisions to make. Finding the right path forward is key to making sure your organization moves at all. But with the right mindset and clear communication between all teams throughout their journey, progress is achievable for any organization, no matter the weight of their processes or the height of their stakes.

16:47 — Jamie Parker
You can go to redhat.com/codecommentspodcast to learn more or visit redhat.com to find our guides to digital transformation. Many thanks to Adam Timm for being our guest. Thank you for joining us.

17:00 — Jamie Parker
This episode was produced by Johan Philippine, Kim Huang, Caroline Creaghead, and Brent Simoneaux. Our audio engineer is Christian Prohom. The audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Carrie da Silva, Mira Cyril, Ocean Matthews, Paige Stroud, Alex Traboulsi, Boo Boo Howse, and Victoria Lawton. I'm Jamie Parker, and this has been Code Comments, an original podcast from Red Hat.