DevOps Decrypted: Ep.25 - Automating Observability, and Safety in Open Source

It's episode 25 of DevOps Decrypted – and we're talking about some big news in DevOps. It was recently revealed that a backdoor had been intentionally planted in XZ Utils, an open-source data compression utility available on almost all installations of Linux and other Unix-like operating systems.

We talk about the ramifications and ask, surely things would be better if they weren't open source? We'll let Rasmus stew on that one for a bit… The conversation leads us to the importance of observability and alerts, and we circle back to a past episode about SBOMs. We wrap up with some events on our calendar, including Team 24 – and as always, we're eager to hear your feedback.

Tell us what you think of the show at devopsdecrypted@adaptavist.com!

Hey, everybody! Welcome to DevOps Decrypted, where we talk about all things DevOps!

I'm your host, Laura Larramore. I'm here with our Adaptavist panel; we have Rasmus, Jobin, Matt and Jon here today – quickly of note, we'd like to ask you to leave us some feedback at devopsdecrypted@adaptavist.com – that's where you can email your feedback to us.

We're going to kick off today with some DevOps in the news. And the first thing we're going to talk about is the backdoor that's been planted in the XZ Utils open-source project. So, Rasmus, do you want to kick us off with that a little bit? Give us a little overview of what's going on there.

Sure – as the resident open-source nerd and enthusiast that I am, you know, well into these kinds of things and have been reading about it a lot. It's a deep thing I encourage anybody with a curiosity to look into it more because - we don't really need to talk about what XZ does – because this is more about the social engineering, about, you know, open source products.

And there's this famous xkcd comic about how the internet is this beautiful superstructure built on top of a bunch of things. But there's this one little teeny, tiny thing in the lower right corner that's just thanklessly maintained by some guy in Kansas or whatnot, for who knows how many years.

And in this case, XZ was kind of like that. And there was just this one stressed maintainer – I can again relate there – and over time, somebody wormed their way in that looked like a person, but they were really probably a state actor of some sort.

And they managed to obfuscate and disable some checks and put something in that you couldn't tell was actually a sneaky backdoor about SSH, which could give attackers access to a wider range of servers because it got into, you know, Linux distros and so on.

Is it… XZee, X Zed? Should I be saying "X Zee" for those of you on the other side of the pond?

I have no idea. Zeb-ra, zee-bra!? It's a stifle, you know.

But what is it? So it's a compression utility, right?

Yeah, yeah, yeah. And again.

Like, zip up files and squidges things really, really small. What I don't quite get is how it's so core to so many things and so many people's infrastructures. Are there libraries that are like deeply embedded into – like Linux? Is that how it works, Rasmus?

That's really it – it's like, unlike something like Windows where everything is written by Microsoft and is just in this big giant code base somewhere, with Linux in general, it's been put together by hundreds and thousands of people just over the years, and they're all these little itty, bitty pieces – like file system approaches you've never heard of like… well, that's in there.

So there's all this kind of stale or semi-stale and not necessarily well-maintained stuff in there. And attackers are beginning to realise, you know, if we could sneak in there and plant an unrelated backdoor – even though it's a compression utility, how do you attack something through a compression utility?

Well, when it has a sneaky sneaker that goes in and interacts with SSH through some other issue. Well, then, you start being able to work out that attack vector there.

So it's crazy.

And it is definitely an easy way to get into the system, right? This actually reminds me of one of those episodes where we discussed in detail about SBOMs – the software bill of materials.

The main reason SBOMs become so popular these days is, you know, you need to know exactly what is embedded in your systems. And you know what kind of utilities we're using, what libraries were use. And you know who wrote it, and obviously do the scanning and figure out all the details about those libraries.

I think, you know, if there is one takeaway again from this whole situation it's, you know, make use of SBOMs. Know what you're embedding in your application, I mean. Obviously, nobody suspects Linux. So this is the next level. What if they get into the operating system? But I think there's also a lesson for us to take away from this as well.

We need to be careful with all the applications we are developing and the products that we are developing as well.

Yeah. And this does get into licensing, which is a good next topic. Because I, the social engineering angle of this is just spot on because I have been that stressed open source maintainer.

I have had weirdos show up for my projects and contribute code like, what is this?! "Well, I thought this would be useful". I mean, okay, maybe.

But there have been times when I have been desperate for more contributors and would even let in kind of poor-quality code just to activate more contributors, just to get help, just to get people to work with me.

But what's a better way? Because that's what we keep hitting on these other topics about HashiCorp and now Redis and all that, like… What is a sustainable, open-source mode that doesn't rely on a bunch of stressed-out maintainers like me when I'm wearing my open-source nerd hat?

This is like the tragedy of successful projects, I guess, isn't it? Because…. to my discredit, I have not really been very much involved in doing open source stuff since about, well, since literally the 1990s. Where I had all sorts of fun, it was very kind of a fundamental part of building my career when I was messing around with Unix utilities like, back in, yeah, in the literal late 1990s.

And you always found some sage-like, guru-type person at the top of the tree, evaluating pull requests, or it wasn't using pull requests, "back in the day", it was like patch files, etcetera, and anything grows. And then you say, well, you take this thing on as well, you know, you seem to know a bit more about testing than I do. I'm going to focus on the core functionality of this thing.

So you take care of that. And then you get this Balkanized responsibility for checking things that are submitted and merging them. Is it even a tragedy of the Commons? I'm not sure if that's the right analogy. Something like that.

Surely this would be much better if things weren't open source, right, Rasmus?!

Heheheheheh… Yeah, okay!? And here's the fun part, because plenty of people say the solution is just to pour money at open source maintainers. And I have a weird response that, no, that's not at all it. My open-source organisation actually had money, and we couldn't use it – because you still have to review code, and you still have to have people motivated by money.

But then they might not be motivated to actually work on the project for the sake of the project, which is really kind of like the spirit of open source.

So if you can't fix it by pouring money at open source contributors or necessarily just like handout developers like, here's the developer, you go work on this thing over here on your own time for a little bit that doesn't necessarily fix the issue. It helps it, you know, adds more activity.

But it doesn't fix it.

And I have, like a sneaky side tangent, hope for the Venue project that I'm working on these days – because some of the concepts have grown out of my struggles over the years as an open source maintainer in that I wish this was easier.

I wish it were easier to keep people organised. I wish it were easier to make things visible so that you could see these things.

The things that happened with XZ were super sneaky, like an automated test that used a binary file like it was obfuscated off to the side; that's why it nearly made it.

But what if there was just like a best practice for unit testing, like everything was just beautifully laid out for you? Just just plug these things in. You're good. And then you hooked up to CVEs and all these things that, you know, scan things for you, and you can trust in this kind of like foundation system that makes things really easy for you, as opposed to, it's a guy in his garden shed trying to make this thing work with, you know, old calculator computers, it's just like… eurgh!

I mean, I like your idea about unit testing everything but who takes ownership of testing the underlying operating system? I mean, we don't even have time to test the products that we develop or the applications that we work on. So, how much visibility are we getting into this underlying infrastructure or the operating system itself? Right?

So, who takes ownership of these libraries that they're using in Linux?

I think that's kind of a really interesting part of this, because this is a small little library. The things there. And there may be the other two things that might be in the news at the moment is kind of like HashiCorp and Valkey and Redis.

And those two are completely different in terms of the scale of things. But you both paint them with you, paint them both with the open source brush. Right? And one is about business models. You know, so with HashiCorp and Redis, and being able to be able to make money from open source.

And these small libraries that just underpin how everything works. I think that's kind of a completely different thing. I'm just reminded of this – Max Howell from Homebrew has put together the Tea protocol, so tea.xyz, which is about trying to reward project contribution. So about rewarding, community and things as a project gets popular. So I find that that's a fascinating initiative to drive that a little bit. I hope it's successful. But at the moment, a lot of certain small-scale open source is based on hope and goodwill.

And he said the right thing to do. I mean, I'm probably going to say a controversial opinion, and Rasmus probably wouldn't talk to me after this?!

But – HashiCorp – can you actually blame them for trying to make money off it because they actually own the code, and they're maintaining it? They're making sure that there are no CVEs coming or no backdoor entries happening there.

We, as a company, do we really rely on you know, individual contributors who can likely plant these backdoor entries? Or do we rely on companies like HashiCorp and maybe pay them some bucks, so we are sure that you know they're giving secure software?

Yep, I think a big thing here is whether you get into bait-and-switch situations or not. That's what upsets me. Like proprietary source code doesn't upset me – like Windows is proprietary, and it's a relatively good product.

But you know, if HashiCorp had started out the way they did and said, we are a proprietary commercial product, I have no problem with them. It was that whole like, building their product and their community on open source and volunteerism, and all this good, happy time stuff… And then, like, oh, by the way, now it costs money. That's my big problem with these things.

I suspect, and I'll bring up an example from my open source project a long, long time ago, where somebody tried to contribute something which I found interesting.

We were trying to adopt semantic versioning, which is one of these things that can help you make it more visible whether a version of the thing is going to have a bigger impact or smaller impact.

And we got into thinking about it. Well, if we make individual developers try to determine, just like on the fly, whether a pull request is a major, minor or patch release, it will become a complete and utter mess.

So what can we do that kind of like automates this process? And we got into thinking about… Well, you would kind of deduce this by just looking at a method signature like, is this actually a major or a minor? And so on. At which point, you take that away, the responsibility of figuring out whether something is really the thing, and you just like, make it, make a computer do it, make a robot do it.

And at the chagrin of Matt being here and loving AI, I suspect AI will end up playing a big role in this. Take away some of this monotony. And you just leave the humans there to figure out, okay, what is it we want to do? And then, just make me the base foundation, and then we will add that in pieces on top of it.

So the TL;DR on the XZ, I think, is, yeah, it's just shining a light on things that have probably been bubbling under for a while, right and uh…

That's not the only one…

Just see some, some, well, less of these scary things. We haven't had a like really scary exploit for a while. So I guess we were due one, perhaps.

And other projects are not now suddenly realising, you know, wait a minute… This person over here is contributing weird things. It's like, Hmm.

I think that's a good thing, even if it will keep me awake tonight.

Yep, So I wonder if this kind of foundation relates to anything with infra from code? Does anybody want to highlight what that is, what that means, and how it's different, as infrastructure as code?

I'm probably the big kind of infra from code nut, so, my software engineering background goes like this: I don't really want to deal with infrastructure; I just want enough infrastructure provisioned in order to be able to run my application.

And there's that’s what we want. So if there's something that can deduce what's necessary to run my code and provision that, then I'm all for that.

So that's the kind of basis of infra from code. But I have yet to see a really good implementation of it that doesn't introduce a brand new programming language. That isn't general purpose, and it doesn't, like, you know, require you to do a whole rewrite of all your stuff. So yeah. So that's the background of those things.

Rasmus – what's in the news about it?

Well, it's news to me, just because I don't know all the things. Surprisingly, there's this not new, but a new thing called Winglang that's trying to do infra from code. And just, you just write your code, and this magical infrastructure will appear – which I would love if that were, you know, the reality of things, although for me, it's kind of like security.

I don't want to think about security. Just do it for me, justmagically make security appear, which is kind of what I was talking about earlier.

I wish all these things would just happen in the background as I pick a framework and start.

Whether this Winglang thing would do that? I don't know. I posted it on our Slack, and it got some mixed responses.

It is an interesting prospect, for sure, because of all these different cloud providers coming up, you know, AWS, Google Cloud, and even on-prem, right? I mean, I don't want to be sitting there writing different code for different cloud providers and stuff like that. I mean, definitely not!

It's difficult because everybody uses different services, and you know, if there is a way to abstract it all, it looks like that's what infra from code is doing right? You know you write one piece of code, and you know you just hook it into different cloud providers, and it will magically work.

Am I getting the wrong impression here, or is that how it is supposed to work?

Yeah, I mean, that's how it's supposed to work. You declare your app, and you run something over it, and boom, it arrives. So Ampt – getampt.com – is like, that's the best implementation I've seen of this from things out there, but it's very specific to AWS.

It supports a small number of libraries and things. So you have to build in a way that's sympathetic. But it is just Javascript, and you can run Node. So this, for me, is like the best, the best, so far, where we're at in terms of just being a general programming language, and it working.

Let me clarify for the listeners – so when you say infra from code, we are writing code for our application, not for the infrastructure itself, and it is actually going through a code detecting what kind of infrastructure our application would need, and then bring up that infrastructure wherever we need, right?

Yeah, yeah. Exactly. So if you write a so like you've got writing Node code, you want an express backend, it knows as you can go through, and it will pull out the endpoints, create a gateway for all of the various – whatever target you have, it goes, and then it goes and creates those things. You’re not have to write anything which explicitly says, I need an API gateway, I need this, that and the other.

So, no more writing in Terraform, no more writing CloudFormation – none of that?

That's the dream. Yeah.

That's the dream. I like it!

So is it like an evolution from infra as code? So again, from infra as code, where the cloud engineers would write Terraform, write cloud formation, or Pulumi, or whatever, and putting this entirely in the hands of the developers to bring up all their infrastructure, have I got that right?

Yeah, that's the dream – and it sounds scary, doesn't it?!

It's kind of funny because I came onto this recording, not thinking I'd mention Venue once, letalone twice. But it feels like this is kind of what we're trying to do with Venue.

But it's not infra from code.

It's kind of just like old-fashioned automated infra as code, in that, you know you log into a thing, you say I want to work on a Node app – click button. Okay, fill in some parameters – and then magic happens.

And then, because it runs through Jenkins and SonarQube and Nexus and gets deployed to Kubernetes, and all these magic things in the background as part of the platform, it's kinda the same thing just like Ampt is from code to fully managed workloads in seconds.

Like, yeah, I got that button.

It's the same thing happening.

I mean, I would say the user experience is the same. Although you have behind-the-scenes, you have your Terraform code and cloud formation doing this stuff for you in this particular case, that's not the case anymore. There's no Terraform code. There's nothing sitting underneath.

But yes, for the user. Absolutely. Yeah. You have a platform. You're clicking the button in both cases.

That's why. So I think you need to think of a level of abstraction prior, right? So with most of the solutions at the moment, you're like, I have, I have a node app, and I have a MySQL database, and I need a queue.

So, from something the infra from code just goes, I have storage. So I need to read. I need to read and write SQL stuff.

I have a queue I want to read and write to. And I don't want to specify what they are or how to interact with them, except through my API calls in my code.

I don't even want to know what the implementation is under the, like, which products I'm using, or whatever; it just sort of does it. So that is that level above…

But it is somewhere like I – my understanding from Winglang is that while you do write this plain generic code, and it's agnostic when you run some step, it actually does generate some Terraform code on the fly that goes and deploys to AWS and whatever it is.

And likewise with Venue like, yep, here's a template. It just contains the code you wanna write and do things. But then, when you let it run somewhere behind the scenes, there's a bunch of nerds like me and like wizards that do magic things to your cloud, you know, infrastructure, and just magics it up.

As a developer here, I appreciate the magic people who are back there, like Rasmus! So, yeah, yeah, that sounds really cool. I think that kind of goes into… …o – if you're looking at… Oh, okay, I'm writing this code, and all these things are happening back there. But you do need to see into that sometimes. So, one of our main topics today was observability.

So, who wants to lead us off on that?

Matt, do you have any thoughts on observability?

Yeah, so I think this is very relevant – make quite a good segue here to what we were just talking about, which is that looking at the way that observability is evolving into 2024, kind of goes hand in hand with a lot of these closed-down abstractions, and I say that in a good way, that we're offering to developers these days.

So yeah, I mean, just throwing back to what we were saying about the infra from code stuff, it's yet another thing where we're moving towards greater abstraction. We're having alleged "magic." I know this is a podcast, so you can't see my air quotes in the background.

So, from alleged magic actually happening so that the likes of you, Laura, can just get on with stuff.

And I think there's quite an interesting emergence of techniques and also a maturity of them where those sorts of things are okay. Because you've got the developer's dream, which is, well, I just want to write my code.

Which is I think, honestly, any platform team should be trying to provide that for developers. And you want developers to be able to write their code and get on with it. Have a paved road, whether through something like Ampt or whatever platform thing you've got, Venue, Backstage, etc, providing all that for you.

And then, well, actually, if we're providing all of this stuff for you as a platform; cloud infrastructure, developer experience, etc. – why the hell aren't we doing the observability as well?

Observability, as a thing, kind of seems to be evolving into what, three or four different areas now?

You have logging and monitoring of system resources. You've got monitoring your application and then probably some APM, you know, some application performance monitoring stuff going on there.

We could be doing all this as code. I think.

I would love that. We were just like saying infra from code. We are saying security from code. I would love that one and dashboards from code. I would love to just magically have this beautiful Grafana dashboard that just shows me everything without ever having done it myself!

So yeah…

You're scaring people…

No, but we need to check this.

Laura, we can't possibly have you develop your code and then just deploy it on your own.

No. What if it goes wrong?

Folks, these are solved problems, aren't they right? Even the security stuff. It's like we talk about DevSecOps and getting people to shift left and codify all your things, including involving developers in conversations earlier. Sorry, involve security people in conversations earlier. Maybe that's why it didn't happen… I get that wrong all the time.

Yes, let's codify that we can put in the right firewall rules, the right security groups, etc. Apologies for the AWS-leaning terminology.

Compliance. Maybe we're doing something that actually involves the transfer of money, and we need to log various things. That's all codifiable. If codifiable is a word. So yeah, should we get on with it?

I think platforms are already getting there. Right? I mean platforms, right, you know, really.. I have seen blogs about observability as code, and you know, anything as code is a good thing. Because it comes with all the advantages that you know in something as code provides – putting them in Git repositories, you know, as the scalability of these things, you know, how we can have consistency across platforms, versioncontrol, all of that.

So, I don't think there is a concern about “is observability as code a bad thing?”.

And again, observability – working with different customers, we have seen over the last couple of years that every big customer wants to have more monitoring, not less monitoring.

And it's been a challenge as well. Because then, you see this, you know, “alert fatigue”. There were so many statistics that we have come across, such as businesses with more than 10,000 employees. 56% that said, you know they chase more than 1,000 alerts a day. Can you imagine, you know?! How many alerts are we getting? What exactly, how exactly should we be dealing with this, what we call the alert fatigue?

And so, should we talk about that as well? I mean, observability is a good thing, but how much of it is a good thing?

Yeah, I mean, you've reminded me there, Jobin, of a situation that I was involved in. But just for once, not the ultimate mistake maker, where we – I was part of building a system which is fully automated using Puppet, and we put in some automation using a tool called Sensu, which probably everyone has forgotten about…

Yes, some blank looks on the call – and I think we managed to send just over 1,300 text messages to the CTO’s mobile phone in one evening.

That was 2016, though. And you know, we get this same sort of responses to like, oh, we can automate this for monitoring things as we did then. And because of that, you know, the thinking has moved on.

We can be using… We can get two steps ahead before we start making mistakes like that and avoid that alert fatigue. It's like, you know, we can figure out what we need to monitor in advance. We can also figure out, in a sort of postmodern retro style, what an alert storm would look like, what alert fatigue would look like.

You can use software these days – I won't mention any names – that will do things like dampen alerts at a certain time and damping groups of alerts so that you don't get this absolute torrent of stuff.

I think it also combines with some of the stuff you need to do to get good developer experience, anyway, which is to give people good guidelines to give developers good guidelines of what good alerts look like and also to keep you on the straight and narrow with things like logging standards.

So if you're going to log stuff off to a system that alerts based on various patterns, then don't just let every bit of debug logging enter into that stream. From there, you can work on getting some, yeah, some good postmodern observability going on.

How does that sound?

Yeah, well, I've got a riff on that a little bit more, Matt, because one of the things that we see a lot like you, you know like post-incident reviews and retrospectives and things is like, Oh, we need. We need to be monitoring this, or we need an alarm on that metric, or whatever it is.

We never take them away.

We're really scared of removing them and actually having it an acceptable thing to go like, actually, that alarm is not useful anymore, or is, are all those things like – it's almost like you should have a one-in-one-out policy for like alarms on things, so you don't get this like crazy kind of storms and things.

Yeah, it's like tests.

It's like, Oh, let's add more tests. And let's have hundreds of tests. And oh, by the way, Ops folks – why does my build take like 3 hours?

Yeah, yeah. Maybe we should.

I mean, I don't take it kindly that you're pointing fingers at developers here!

I mean, obviously, you know, there are so many debug logs that we put into the code, I agree. But at the same time, you know, there are these complex infrastructure systems. You know the complexity of cloud-native systems; so many different services on AWS, GCP, or Azure are used, and every single thing sends alerts. It's not just the code; the alerts are coming from the code. Every single service you're using has a modern thing involved. They are sending alerts as well.

And how many of these are we handling? And a lot of them are false alerts. Yeah, you're right. I mean, developers send a few false alerts as well, but infrastructure does as well, right?

I think the observability thing is only as useful as it is. You can observe everything, or you can observe what's really useful.

Like, we're working on a plugin that we maintain that we're trying to create like an audit log for at the moment. And so we're trying to decide what is exactly useful for this because we don't wanna slow down everything. We don't wanna overload everybody. We want to be able to use the information as it's useful, and I think that the evaluation, when you add something new, to say, is there anything I could take away?

You know like that's a very good kind of – in my own life, I'm a little bit of a minimalist. That's a kind of minimalist approach to coding and data instead of being like, give me all the data.

It's like, well, let me think about what exactly I need here. You know?

What you said is actually a data-driven approach that most companies are now doing. A lot of these platforms, you know, they actually take the data, see which ones actually make sense, like Jon was earlier saying about, Hey, does this metric even make sense anymore?

Should we take this a lot out because it's not applicable anymore? And I think artificial intelligence can play a big part in this. And I'm sure Matt will love this. But because, you know, humans cannot in any way go through 1,000 or 10,000 or 100,000 alerts and figure out which one is more important and which one we should keep.

So I think that's one area where AI could substantially help us, you know, and figure out which alerts should be important – ultimately, what message, what text message I should be sending to the CTO's mobile phone? Right? That has to be decided by probably computers, not humans.

And it's nuance, isn't it, Jobin? Because, so Laura mentioned audit logs a few minutes ago. And so you've got this dichotomy of things because there are some things that you want to be minimalist about, because a lot of it is just noise.

And you kind of know what they need to look like and what you're going to be looking for.

But then there are other things on the other side of the coin which are, well, actually, we might need to come back to this data in like four months' time for some audit reason or compliance reason or something – and keeping those two things very, very separate, I think, is important. I think there's also a load more nuance and alerts on those lines.

Yeah.

Absolutely.

I've seen some initial promising results out of Google Cloud, because in there sometimes, I need to look for logs. And it seems like there are some smarts going on, like trying to surface the stuff that's actually meaningful to me. And I can only imagine that gets better. It's like we also need your intelligence or meaning from code. Ultimately, we need a platform of everything that does all the things, yet it still is a balance between, like handing it over to an AI and doing the thing as a human, and just that constant balance of hide the abstraction, or rather abstracting the complexity or moving the complexity.

I guess that's just our career.

Yeah. And the AI thing, orr I am actually gonna say something positive about AI. We're seeing that in the observability platforms coming in. Where, so I like to think of like doing postmodern alerts things – so things like if the system goes down, and then it always comes back like eight minutes later, then maybe some piece of software could be intelligent enough to draw the dots there, and that like work out what's actually going on there, or at least mitigate the alert. Put a note on it.

That sort of stuff, I think, is starting to emerge into some of these observability platforms, and it's all a whole load of data crunching going on in the background on our behalf.

So, we start to see these patterns and trends. And yeah, if they're not using AI in the background, then I'll be very surprised.

And that's a good thing.

It is.

So what else is up?

A bunch of us will be in Vegas at Team 24, Atlassian's big conference. And we've got a big booth there; we're doing loads of lightning talks. We'll be doing one on DevOps and another on AI, and plenty of other people are talking about some really interesting things.

And this could be a really good, really good conference. So if you're there, come and say hi!

I'd love to meet you.

And I guess I can say that here on 23 April, I will be at the Shift Conference in Miami with a few other people.

I think we both have some Adaptavist people. We're doing some Venue stuff there. It should be fun!

We wanted to mention that on April 25th, Rasmus, Matt and I will host the expert hour live on Adaptavist's Linkedin, and we will also have a special guest from Gitlab, Bart Zhang. We'll discuss transformation for the curious – so join us and hang out there live.

For Jobin, Matt, Rasmus, Jon and myself – this has been DevOps Decrypted – thanks guys.

Why not leave us a review on your podcast platform of choice? Let us know how we're doing or highlight topics you would like us to discuss in our upcoming episodes.

We truly love to hear your feedback, and as a thank you, we will be giving out some free Adaptavist swag bags to say thank you for your ongoing support!