Transcript: DevOps Decrypted Ep. 11 - Flaming hot source

In this month's episode, we are joined by Adaptavist CTO Jon Mort as we discuss Google's latest announcement around Open Source maintenance and our thoughts on Open Source security, maintenance, and culture and what this means for your business.

Hello everybody and welcome to DevOps Decrypted. This is episode 11: ‘Flaming hot source’. I'm your host, Romy Greenfield and joining me today i've got Rasmus, Jobin and a new guest member, Jon Mort, say hi everyone.

Hello. Hello.

Hello.

Hello.

Cool. So, today we're going to get sourcey. We are talking about open source. We know that Google has announced a large chunk of money into maintaining open source software, a maintenance team. So, guys, what do you think about that?

Well, I mean, this is something that had to be expected, right? We have all heard about the Log4j vulnerability. We actually talked about it in our podcast, I think a few weeks, months back.

Yeah, we definitely did.

Yes and I think this got attention in the tech community, but also even for governments, right? And, there was this big summit that happened at White House. They held a meeting with the big companies like Google, Apple, Amazon, IBM, Apache, all the big companies came together. White House actually started talking about how these vulnerabilities could be a national security issue, right. And, they started talking about how we can prevent such vulnerabilities and any cyber threats coming out of those, exploit those vulnerabilities. And, I think, that's where it all started. It was a good move and major companies like Google started exploring that further. Obviously, these companies were looking at open source even before, much before this, but the focus is now shifted and they started looking at vulnerability specifically, and how it could affect the products that they are developing, right? So, I guess that's where it all started. Jon, Rasmus, what do you think?

I mean, I think it might be worth just stepping back a little bit and looking at the impact of open source in general, right? So, they open so... Almost everything you, every device you touch pretty much is built using open source software and relying on the things. So, this isn't the small thing, open source is huge and it's transformed the industry. I was thinking the other day, I can't really imagine what developing or using a computer would look like if you couldn't use open source software to do that or building things, it'd be pretty much impossible. So, it's fundamental to everything we do and the security of that obviously, is massively important. And, it feels like the world is only just waking up to something that lots of people have been shouting about for quite a long time.

There seems to be a pattern with human behaviour about this. A few select people notice that there's a massive issue. You have to wait till it snowballs till enough people higher up go, "Oh, actually this could be a problem for us." And, it takes a company like Google, Amazon, a big giant to actually go, "Do you know what? Maybe we should throw some money at this problem. It's affecting us just as much as it's affecting the little people." I mean, I know I haven't even been in software engineering where I haven't had access to open source. It's always been open source for me. It's what I started using almost from day one of learning how to code. So, yeah, it's massively important.

There's so much good and bad involved here. I could talk about open source for hours and I'll talk about regular in the view, but it's both incredibly good. Like Jon points out what open source has allowed us to do at this point in time, it's a wonderful, wonderful feeling of all these enthusiast projects that have been put together by people just because they wanted to. It's not driven by profited breed and all those kind of things, which is good, but it's also bad in that this level of volunteering means that a lot of the times you will end up with corners and spots that get less attention when they really warrant it, is especially when half the world starts using it. So, it's good when companies like Google and others put money into fixing actual security issues to help put band aids on the wounds, but it's also bad because that doesn't fix open source.

We talk about DevOps a lot on this thing, of course and, I think, we emphasise in DevOps over, and over, and over again, that it's actually about culture. It's very, very much about culture, not so much about tools and things, and the same goes to open source, where it's really the culture of how you get people on board, how you run the projects, no amount of money is going to be able to fix that. So, this is going to be a helpful act at one point in time, but there's a bigger topic of how open source can remain healthy.

Yeah. I mean, I know when we've done hack days before we've got projects where people have literally just contributed their time to working on open source project maintenance. And, I almost feel like every company that uses open source, which is probably every company, should make a commitment of giving their developers X amount of time, X amount of days a year, where they should solely be working on an open source project that is used in their products.

And, that might actually be one of the almost better solutions in just that, sure, the companies can donate money and get tax benefits. That's the cold cynical view on it, but encouraging a culture within their organisation of contributing to open source. Open source that their employees are excited about, not necessarily actually using it, but just that they're excited about improving, that can help grow the field of people involved in open source. And, it follows the whole thing about all bugs are shallow if you have enough eyes, so you just, you need more people and more time in open source to fix all the things.

And, I think that is actually what Google is doing here, right? I mean, Google had been contributing to open source from a long, long time ago, but now what they're doing is they're coming up with something called the open source maintenance crew, which is a dedicated set of resources, specifically looking at opensource scores and trying to fix vulnerabilities in it. It's not like Google hadn't been doing that earlier, but now they have a dedicated team doing it. Are we starting to see more and more companies doing that? It seems to be going that way.

Yeah. So, because I wonder whether this is a societal issue, this is big enough that this is something that should be dealt with at government level. And, it's almost an ethical must for companies to be involved and to be contributing to the things that they rely on. Yeah. Always reminded of the, well, xkcd, has got a cartoon for everything, right? But, there's... So, there's the famous one of that, of the layers of architecture with that, the little tiny piece that there, is just maintained by so run, a project some random person in Nebraska has been thanklessly maintaining since 2003, and an hour things pile on top of it. So, that's xkcd 2347. And so, I think that if you're a large organisation, or medium size, relying on that, you should be contributing to it, and you should be encouraging staff to be doing that, and backing that financially, as well.

So yeah, Jo, I think it's you spot on with that. There's a duty there almost to be contributing, and it is a societal issue.

Yeah. And I mean, that's reflected in the fact that what happened with Log4Shell, I can remember what it's called now.

Log4j.

Log4j. It was reflected in that because one vulnerability in that, I think it was from two, 2003 or something really early that vulnerability had been in there. And obviously, no one had been concerned with the security of that. There hadn't been a full security test to know whether that had a vulnerability in it. And, if we had people investing, if you use it, you invested the time to check that it was secure, or that it was signed off by a different company. You'd check that it was secure then maybe, could have saved ourselves a lot of hassle.

Yeah, absolutely and Google actually is doing a lot more than just promising money against it. As I was saying earlier, I think, they first promised $100,000,000 towards the cost of supporting open source in the first place, but then they created a framework around it. I don't know if you know about the ‘Know, Prevent, Fix framework’ that Google introduced. Basically, the thing is, first, you know about all the vulnerable distances of software. I mean, that's not easy because the vulnerabilities keep popping up, but somebody knows how to actually create a shared database of all the non vulnerabilities like that. But, the first thing is to know about the vulnerabilities in your software. Again, you have so many open source libraries that you're using in the software, some may now be as easy as you think, but that's the first step, right?

And then, prevent the addition of new vulnerabilities, so that's the ‘Know, Prevent, and then Fix’, right? Fix or remove the vulnerabilities. But, in order for you to do that, as I was saying earlier, you have to have a centralised place where all the vulnerabilities are documented. Then, you have dedicated resources who are identifying those vulnerabilities and fixing those. I think that's where this new maintenance crew will affect you, right? We cannot wait for individual maintenance to come and fix the vulnerability. It's going to take a lot of time. Whereas, if you have a dedicated team from companies like Google, who can constantly look for vulnerabilities and fix them, that will help the open source community as a whole, and definitely improve security.

Yeah. Again, just like DevOps it is a combination of people and tools, and culture, and so on. So, as much as I like talking about open source culture and so on, it certainly would be nice if there was a supporting underlying framework where you had all these nice automated methods too, did you know your open source product so and so has a vulnerability identified over here. Here is a pull request that actually increases the dependency on the vulnerable component you're relying on, just click the button, and you're done. That can help take out some of the toil cost of open source products, especially the ones that get less attention because people don't want to deal with them. But, if you can just go hit a button, because somebody else was clever enough to make a tool that could find the thing, and propose a fix, that helps. And, this kind of investment from Google and others certainly will be a step on the way there.

Yeah. I think, it's healthy to put a team together whose sole purpose is to look at this because I know from previous experience, when you find a bit of open source code that you really want to use, but it just has one little feature that would be really helpful for you, that you know that you could go in there and add yourself, but you just don't have the time. You don't have the resources from your workplace in order to do that. So, the delay in whatever you were working on to do that, you end up just moving on and not doing it. So, having that dedicated team, allowing that team to see actually this would be really great improvement for this library that multiple users could get benefits from and actually having the time to dedicate to do that, which is awesome.

Which makes me think about what you said earlier about whether companies themselves encourage their employees to take some time, 5% time, 10% time, or whatnot, to go do good things for open source. If they happen to find something that's, "Hey, this one feature for this one open source project would really help in this spot over here and I can use work time on." That would be awesome.

And, it's interesting that it's not just Google doing this, right? I mean, there are a lot of other projects contributing to it. There is the open source Alpha-Omega Project, I'm reading from the website. Its purpose is to improve global open source software supply chain security by working with project maintenance to systematically look for new as-yet-undiscovered vulnerabilities in open source code. So, other companies are doing it. And, there's another one, interestingly for me, the Linux Foundation secure open source, or the SOS project, the interesting one is, there's a $1,000,000 funding from Google for it. SOS is actually offering $10,000 in rewards to developers for hardening software. So, if you're a developer and if you can harden the software, you can probably claim for, I know, up to $10,000 from Google. So, there is monetary benefit in creating secure software, that's interesting, right? So, I think, there's a lot of good initiatives happening in the tech community for making sure that we are developing secure software.

And, this is where I have another topic that's not a pet peeve, but a pet concern just because I have seen over the years working in open source how bounty programs can work, and how bounty programs can be abused, because it's one thing, I worked with Bountysource that come at one point, which was really just, "Hey, you see an issue on some open source product somewhere and you want to encourage people to fix it, put a bounty on it." And, in concept, that sounds like a good idea, but then you can get to a point where you have, you get bombarded by these supposed fixes to a thing you didn't know about, because some outside party just put a bounty on it and you're, "Wait a minute. No, no, no, this is not how I want to fix this at all."

Or even, if the fixes are good, it's, "Here's a pile of pool written code that will fix your problem. Can you please review it and merge it so I can get paid, please. Thank you." And then, you get into as a maintainer, well, do you have time for that? Do you want to educate this user and how to fix and link their code, and make it acceptable? It gets so complicated. Money in open source is such a double-edged sword.

Fascinating point because I've never thought of it like that.

Yeah and, I'm sure, probably Jon, you have some stories to talk about the Atlassian bounty program because we also do some bounty programs for our own internal products, right?

Yeah. So, when we... There's bug bounties, which I think are, I mean, they're a really powerful tool, and they're really great for that, but in order to have one of those running, you need to make sure that you have a team that's able to respond quickly, and things that, you are inviting attention and you, I got the engagement there. And so, you need to have that. So, I mean, as Rasmus said, if you've got a bounty put out and your project and you are not expecting it, and you're not prepared for it, you can get thoroughly overwhelmed and it's totally, it's not a helpful thing to do. So, those things you need to do super carefully. And, it's that people aspect of the things, it has to be a conversation that has to be willingness to participate and to contribute.

And, that be, and to look at that. It's interesting looking at the 10 points and open SSFs. So, plan is there, the plan for, whatever they call it, there's streams of investment. So, one of them is security education, which is, it's one of the things that can so easily get missed. And, it's really difficult to keep on top of all the various different, new potential vulnerabilities in classes of risk. And, this plan goes over these 10 different areas. And, roughly looking at about half of them are about culture and team support. And, about another half are technical solutions to things, like digital signatures and better scanning, the tools for those things.

So, I quite like their approach on how they're going to do that in a way that doesn't put pressure onto those maintainers, who are like that poor maintainer in Nebraska, who has been doing it. And, the other thing I think, that's worth calling out on this is that the majority of people who are contributing to open source, well, maybe that's not true, but a decent number of people contributing to open source, they're not doing it for the money. So, money's not potentially, not really the motivator that's going to work well here. So, I think that looking at motivation, and looking at why people contribute, and why people run a project, I think that's really important in looking at how we raise the bar as an industry for security in open source.

That is just so extremely important. That level of motivation makes it so easy to mess it up, and money does not necessarily make it better. I had one example that I thought was spot on in this, one of the projects I work on in open source man is a video game that has in game user interfaces that can be rendered with HTML. At some point in the past, somebody who did that just took an existing library and just dumped in the source code as actual source code, not an external dependency, and included all kinds of things that weren't used, like JavaScript rendering, and whatnot, but because it was in our repo on GitHub, years later down the road, when nobody was maintaining that code anymore, somebody came out in the woodworks saying, "Hey, our scanner detected a vulnerability in the JavaScript pausing in your project."

And, if you approve this thing, we will pay a researcher for having found the thing. And then, we will pay you for fixing it, which is, that's novel and interesting, but you realise this is unused code in embedded source that is woefully out of date in a video game that could be exploited, if it was a web server. So, I completely miss it, and you can tell that the person who shows up, "I want money and you can have money, too." But, there was nothing about open source in that whole interchange. There was no understanding of the project and it was a completely pointless fix. So, we turned it down and said, "This is a cool initiative, but you really need to sharpen your focus and understand what you're trying to do in the motivation board."

It is an interesting story though, right? So, people are actually exploiting, not just the vulnerabilities, but also vulnerabilities, the opportunity to fix it and may probably make money out of it. That's an interesting perspective. But again, the crew that Google started, maybe more focus on certain teams to maintain these softwares will eventually eradicate the need for individuals trying to fix vulnerabilities for the sake of money, or trying to make money out of it.

It seems more like a one time thing than that. It really feels like over time open source just acquired all these wounds and bruises, and cuts, and things. And, this is Google, and other companies just flying in with a ship in band aids, and they slap them on there, and it helps. But, it doesn't fundamentally change the journey open source was on in the first place, which is what put it into all the brambles and things that cause those problems in the first place. So, I am thankful to Google and others for jumping in. And even, sometimes it's weird to be told, "Hey, you can be paid to fix bugs in your own code", which, as an open source maintainer, that's, "Wow, that's really cool." But, it's also not helpful at times. It's weird.

So, it's great that we are coming up with all these different ways to fix the vulnerabilities in the open source world, but we know that it's not going to be the story, right? There will still be new vulnerabilities. So, as people who are practising DevOps, as companies who are developing products used in these open source libraries, there's always this question about, okay, how do we keep our products secure, safer, right? So, that's where the DevOps principles will come in and say that, okay, you have to start doing shift left. You have to start catching these vulnerabilities earlier. You have to incorporate security scanning and vulnerability tests into your pipelines, those kinds of things, right? So, what do you think about it? I mean, is that one of the ways to do things?

I mean, I think it's always a good start to have vulnerability scanning in your pipelines, even if you're not contributing to the open source itself, at least you'll know where there's vulnerabilities, and protect yourself that way.

Yeah. There's a lot of the logistics and meta work that really could use some of this help, because that is a one time investment that pays off over the long term. So, lots of open source projects are just a random report story on GitHub, that somebody just threw some code at one point. GitHub itself has helped in the way that they automatically enable the GitHub advanced security on report stories, which helps find some of the common vulnerabilities.

They have the dependabot thing, which is useful. We need more of that because that is a forced amplifier. Even if there's only one guy in Nebraska working on this particular library over here, if he's inside a tool ecosystem that just makes it super easy to point out, here's some quality standards and things. Here's some recommendations, here's some linting tools. Here's some simple warnings about what's wrong. That not only makes it easier for you as a person to go in there and click a button to apply a fix, but it also helps you encourage others to come in by saying, "Welcome to my project. Here's a simple to-do list of nice things, it would be cool to have people's help with. Come on in and then help out, feel good, and help make the world a better place."

Yeah. And, in the story that you were saying earlier Rasmus, so another person comes in starts hardening stuff into your open source project, and then boom, suddenly the bills fail and you see that, okay, there are some vulnerabilities identified in the new code that you pulled in, or the new library that you pulled in. And, that actually notifies not just the internal, but also the person who committed the code, saying that, okay, there is a problem with the code that I checked in, so I need to fix that before I can merge it or move it upstream, right?

Yep. And, that's also where there is a cultural ill that affects some projects. And, I will be completely volunteering of having done this myself, but sometimes if you are maintaining open source project and you see somebody new come in, and you are desperate for more help and more contributors, and things like that, you might be tempted to let in some tech debt just to encourage these new people to come on board as ongoing maintainers. And so, that is a very, very tricky balancing act.

All right. I can't forget the statistics that I read about open source saying it will take them at least $147.9 million to fix all the non vulnerabilities that we have identified in the next two years. That's a lot of money and that's a lot of vulnerabilities that we have in the pipeline to fix.

Yeah. Scary numbers.

Scary numbers, indeed.

Yep. And, as usual, it's just that tech debt building over with time. So, anything that helps you prevent that tech debt from accumulating in the first place, and anything that makes it faster and easier to fix it. I would love to see more support for improving, but sometimes this has to just come from within the projects relevant from outside, because changes from outside, support from outside with literal code or money can very much be short term, it can be false salvation. I mean, if somebody comes in and brings this awesome piece of everything, they'll just fix it. But then, it turns out that nobody knows the language that is written in and it can't be maintained over time. Now, you suddenly doomed the project you were trying to help.

So, White House, or any government can come up with an executive order, but it still has to be, there is still a job to be done by technologies and companies around the world, to keep our software safer.

Yep. So, if you're not maintaining any open source and you are using it, get onto that now. Give your staff members time in their day to contribute, do it. Be the initial, I can't remember what the name is for the curve, the initial adopters, and get out there.

That's right. And, I think, that's a question that we always ask at the end of the podcast, right? What's the one thing that you would want to do? So, Romy, that's your one thing?

My one thing is we've just been adding automations into our pipeline for licence checking, as well. So, I would, we did it manually first to check that it works, but I'm always pro automate that, if you can get that into a pipeline and you don't have to do it every single time, just do it, automate those tasks that are easy to automate, get it in there, save yourself a lot of hassle.

Sounds great. I mean, I would go with vulnerability testing. I mean, we had been talking about vulnerabilities, right. So, if you have a pipeline already, obviously if you don't have a pipeline, that's the first thing that you should be doing, build a pipeline around your code, but if you already have a pipeline, yeah, make sure you add vulnerability testing at some point before obviously, you merge into your mainstream. Yeah. That's the one thing I would say, because it's a scary world out there. I mean, open source is great. Everybody's using it. But, as you saw from the numbers, I mean, there's hell a lot of vulnerabilities out there, and it's going to take a lot of money and time, and probably commitment from different companies to fix it. So, what we can do is, as Google said, right? Know your vulnerabilities, and then fix them. But, the first step is to know them. So, add vulnerability testing to your pipelines.

And, I'll chip in actually by stealing Romy's item from earlier, I really want to highlight that. Make companies invest in this by promising their employees they can spend X% of their time contributing to open source products of their choice, that does not have to be relevant to their work. It has to be something they are enthusiastic about, that they want to do, because open source, and on the cultural side of things, it's like a garden. If you do what you can to just make the soil fertile and make it really good to grow things, it doesn't matter what grows. That's part of the beauty of it, because people will just do things they think are beautiful, and that's what will grow in the garden. Whereas, if you try these heavy-handed approaches and just send people in with giant weed rackers and pesticide, and so on, because we are trying to de-weed this thing, that doesn't help the garden. It just makes it look prettier for a little bit.

Oh, and I guess, my one thing, maybe go tip your favourite, or the open source project, which you rely on... Buy them a coffee or show them that you appreciate the work that they're doing. Because, I think, yeah, it can be a pretty thankless task. So, handing around a bit of appreciation, I think would be a great thing to do.

Awesome. Thanks for joining us today to discuss everything open source, and all of the exciting money that's being thrown at it by big companies. We hope you're enjoying this show, let us know what you think on social media @Adaptavist. We look forward to keeping you interested and having a conversation with you there. We really want some feedback. So, if anyone has a big, good, bad, evil to say, whack it on there. So, from me, Romy, your host, and Jobin, Jon, and Rasmus, thanks for listening today. This has been another episode of DevOps decrypted, which is part of the Adaptivist live podcast network.