Episode 23: Kubernetes and Cloud Security with Andrew Martin

Committing to Cloud Native

Sponsored by Reblaze, creators of Curiefense

Panelists

Justin Dorfman | Richard Littauer

Guest

Andrew Martin

Show Notes

Hello and welcome to Committing to Cloud Native Podcast! It’s the podcast sponsored by Reblaze where we talk about the confluence of Cloud Native technology and Open Source. We have a great guest today, Andrew Martin, joining us from London. He is the CEO of Control Plane, a Cloud Native security consultancy training and pen test firm. We learn more about Andrew’s background, how he got involved in Kubernetes and Cloud security, and more about Cloud Plane. In 2019, Andrew made some Kubernetes predictions, and we find out today if any of them came true. We also find out how he keeps updated on what’s going on with open source in Cloud Native and other things. Since he has such a wealth of knowledge, Andrew fills us in on his book coming out soon called Hacking Kubernetes: Threat-Driven Analysis and Defense, and what chapter he’s most looking forward to people reading and why. We couldn’t let Andrew go without asking him for his “Predictions for 2023!” Go ahead and download this episode now to learn so much more from Andrew!

[00:01:34] Andrew tells us what Control Plane is, what does it does, and how many people they have working there.

[00:02:13] What is the average size of company in this space and why would someone need extra security on top of Cloud Native?

[00:06:58] Andrew tells us how he got involved with Kubernetes, Cloud security, and more about his background.

[00:10:22] We find out why Andrew thinks Kubernetes succeeded and Docker Swarm didn’t.

[00:11:57] In 2019, Andrew made some predictions and Justin wants to see if any of them came true. First prediction, did hosted services catch up with GKE?

[00:12:59] Second prediction, did non-container VM-based isolation improvement happen?

[00:16:39] With Andrew’s vast knowledge Richard wonders what he uses to keep updated on how open source works in Cloud Native and if there’s a Medium Blog that he’s subscribes to. Also, he shares which conference he will be attending this year and others he recommends. Justin gives a shout-out to TAG Security and their meetups.

[00:20:05] Andrew’s book he co-wrote with Michael Hausenblas, Hacking Kubernetes, is discussed and he tells us the chapter he’s most looking forward to having people read.

[00:23:49] Justin wonders if any of Andrew’s colleagues reviewed the book or if it’s all done with O’Reilly.

[00:25:26] Andrew explains what he does to make sure that people at Control Plane are actually getting the best of the open source world without which it wouldn’t exist.

[00:29:03] Richard is curious to know what method Andrew uses to find an interesting problem and how does he do security research in a way that makes him feel really excited about doing that sort of work.

[00:32:22] We hear one last 2019 Kubernetes prediction and that is, if the tangle of YAML was going to unravel by 2019? He also talks about image and build metadata security matures which was another prediction.

[00:35:53] Richard asks Andrew if he’s worked with Dan Lorenc in the Sigstore Project and Justin gives a shout-out to Dan and Episode 20 on this podcast to check out.

[00:36:14] Andrew shares his predictions for 2023.

[00:39:27] Find out where you can follow Andrew and the work he does.

Quotes

[00:03:21] “The shared responsibility model gives us a different level of interaction with our cloud provider based upon what is ultimately platform as a service or infrastructure as a service or software as a service as well.”

[00:04:03] “But when it comes to how we behave operationally the cloud provider can make no guarantees that we’re not shipping bad code to production.”

[00:10:51] “And service meshes were being shipped by Docker Swarm before they were cool.”

[00:11:29] “So, from a networking perspective, Docker Swarm was much better out of the box because it was batteries included, but changeable, and came with its own networking paradigm.”

[00:11:40] “However, the inability to run multiple containers in a pod meant that there was no flexibility of application to Pology, and that’s really where Kubernetes stole the show.”

[00:13:14] “Google had this huge infrastructure, all these Borg cells, Flexible Compute to host Gmail and Google Search and calendar, and maps.”

[00:14:10] “And so still definitely harboring loads of zero days that are probably being exploited somewhere by somebody.”

[00:25:32] “Democratization and open sourcing of security, tooling, and information has been a constant source of utter amazement to me.”

[00:26:13] “At some point, I have wondered, and I noticed the case for other security researchers in the Cloud Native space as well, if actually we’re a bit further ahead with the art of the possible than the current state of the art.”

[00:26:36] “So we try very hard to open source everything.”

[00:29:23] “I think the most important think speaking personally, but also infusing teams and displaying leadership, is to infuse or inculcate a shared sense of passion for thing.”

[00:37:11] “So actually, one of the things that I really love is the function as a service approach, on top of STO, on top of Knative, on top of Kubernetes, because you then get the full observability of the whole platform.”

[00:37:23] “You can apply this intrusion detection, you can use your namespace, aware tools in order to introspect more deeply, and you can also satisfy what ultimately the kingmakers, the developers require because there’s no point building a secure system if the developers can’t ship business functionality through it.”

Links

Curiefense

Curiefense Twitter

Curiefense Blog

Cloud Native Community Groups-Curifense

[e-mail]

Reblaze

Justin Dorfman Twitter

[e-mail]

[e-mail]

Richard Littauer Twitter

Tzury Bar Yochay Twitter

Andrew Martin Twitter

Andrew Martin Linkedin

Control Plane

Hacking Kubernetes: Threat-Driven Analysis and Defense by Andrew Martin and Michael Hausenblas_

Hacking Kubernetes: Threat-Driven Analysis and Defense by Andrew Martin and Michael Hausenblas (Amazon UK)

Committing to Cloud Native Podcast-Episode 20: Taking Open Source Supply Chain Security Seriously with Dan Lorenc

SANS SEC584: Cloud Native Security: Defending Containers and Kubernetes-Course with Andrew Martin

O’Reilly Kubernetes Threat Modeling Course with Andrew Martin

Credits

• Executive Produced by Tzury Bar Yochay
• Produced by Justin Dorfman
• Edited by Paul M. Bahr at Peachtree Sound
• Show notes by DeAnn Bahr at Peachtree Sound
• Transcript by Layten Pryce

---

Transcript

[00:02] Andrew Martin: One of the things that I really love is the function as a service approach on top of STO, on top of K native on top of Kubernetes. Because you then get full observability of the whole platform, you can apply this intrusion detection, you can use your namespace, aware tools in order to introspect more deeply, and should also satisfy what ultimately the kingmakers, the developers require. Because there's no point building a secure system if the developers can't ship business functionality through it. Platform for functions with Kubernetes is underneath.

[00:33] Richard: Hello, and welcome to committing to cloud native, the podcast where we talk about the confluence of cloud native technology and open source. Super excited to talk to you today, been a while since we've had a podcast, very happy to be back at the track or whatever. I don't know something. Anyway, our other panelists we had today besides the illustrious Richard [inaudible 00:55]. That's this guy, is Justin Dorfman. Justin, how you doing?

[01:00] Justin: Doing good. How are you Richard? It's good to see you again. It's been a while.

[01:04] Richard: Always good to see you, too. Oh, the hat still has something on it. It's a smudge, kind of looks like LA, should say New York City. I don't know what's going on there. Anyway, we have a guest on this podcast, which is good for all of us listeners. Because it'd be the worst if Justin and I just talked. I'm talking about Andrew Martin. Andrewio is, Andrewio?

[01:23] Andrew Martin: I'll be whoever you need me to be.

[01:27] Richard: We have Andrew Martin on today who has just said he'll be whatever he needs me to be. Andrew, I need you to be the CEO of control plane, can you tell me what control plane is and does.

[01:38] Andrew Martin: Control Plane is a cloud native security consultancy, training and pentest firm. If it looks like cloud native, and it smells like security, we are interested.

[01:49] Richard: Awesome. I understand what that is pretty intuitively. How many people do you have there?

[01:53] Andrew Martin: We have about 30 people and still expanding, taking on security architects and threat modelers and all that good stuff.

[01:59] Richard: I'm familiar with the security field. I kind of know what pentesting means. I know that there are people out there who are literally hired to like, take axes and wire clippers and go through fences and figure out how to get to servers, which is the coolest job ever. For my benefit, because I don't know, what's the average size of company in this space? Is it like five people? Is it like a solo Polish dude and working out of his apartment? Does that have like 1000 people? What's going on?

[02:26] Andrew Martin: There's a huge variance. Actually, we should be just as concerned about the solo dude in his basement, who may have an incisive way into our systems as some of the bigger organizations, of course, in the UK for penetration testing, you have organizations like NCC who do it, a huge proportion of the market. But then you also have the smaller boutique firms that focus very specifically on the cloud native aspects and the interactions between the cluster and the clouds. The IM integrations, how things configured in CICD, getting into the supply chain and all that good stuff. Yeah, we we like to think we operate on more of a intuitive than check mark based basis.

[03:06] Richard: Cool. that makes a bit more sense. Now, dumb question. Security for cloud native, I thought cloud native depends upon actually using other people's code. When I'm using AWS, I'm dependent on AWS security, why do I need extra security on top of that?

[03:22] Andrew Martin: The shared responsibility model gives us a different level of interaction with a cloud provider, based upon what is ultimately platform as a service, or infrastructure as a service, or software as a service as well, where Gmail, you don't see the servers, but for the infrastructure from for example, GCP while you're actually interacting with the servers, and you have a responsibility to secure part of that stack. We trust the cloud provider to encrypt our data in transit, to make sure that there's suitably advanced encryption and folds envelope encryption, so the keys can't be retrieved and so that our data at rest is safe. But when it comes to how we behave operationally, the cloud provider can make no guarantees that we're not shipping bad code to production.

[04:09] First line of defense for any cloud-based application is the socket that faces the internet that's behind the load balancer, or whatever that would be. As we go to just serve our application code, well, that's developed by us, not developers. If we have a struts like remote code execution, in that web facing tier, well, then all of a sudden, our adversary can get in, to our infrastructure. As I say, that's not something that the cloud provider is able to give us any guarantees over because they have no insight into what we're deploying. That's very much the delineation of the model, how it's supposed to work.

[04:45] What do we do? Well, we can use the raft of modern cloud native, specifically kind of containers, namespace aware tooling, to run lots of processes on one host, let's say like a Kubernetes style. Lots of containers. For each of those containers, we can attach a granular security profile. If we think back to the LAMP stack, we would have a VM with, obviously, it's running Linux, we have an Apache web server, which needs capabilities and permissions to bind to a port below 1024. That's a privileged operation, then we've also got my sequel, which wants to bind to a port above 1024. It's a 3306 if memory serves.

[05:25] Applying same policy to the whole VM, isn't practical, you have to do that on a per process basis. That just gets laborious. Traditionally, we've seen, we've got stuff like Dan Walsh, turn off SC Linux, it's a kind of lifetime crusade, SE Linux has this level of granularity, but it's difficult to figure. Cloud Native gives us containers, containers gives us a single process per container model. That container is an easy, and relatively well hardened security boundary. We can configure that per process. We have very high fidelity of policy, very granular application or per process level. When we get that remote code execution into our container, because we've got that vulnerable application library, we can go in and save the baseline behavior for this process has deviated, we've spawned a shell, we're making other strange network requests we wouldn't do normally.

That's the delineation between the cloud security that we get from the vendor, which those underlay encryptions, and data at rest. The lifeblood, the data when it's actually passing through the system, and narrowing down the attack surface, and then running cover with tight policy for each process that run.

[06:43] Richard: Oh, sure enough CEO slash Chief Marketing Officer, cause it's just very clear. Thank you so much. You mentioned Kubernetes at one point, and I do want to get to your book that you wrote Hacking Kubernetes, which is going to be very exciting. Before I get there now that we have enough background to know that what you're talking about. How did you get involved with Kubernetes? How did you get involved with cloud security? What's your background?

[07:05] Andrew Martin: I was very privileged to come straight out of university and into a startup where I did the traditional hat juggling that start-ups are well known for. That gave me the opportunity to migrate bare metal servers on to Amazon, AWS at the end of 2008 to deal with PCI audits to do database query tuning, to expand into multiple different types of programming language. That gave me the kind of generalizing specialist angle on things. That really placed my inquisitiveness, which is probably a superpower and a huge distraction, and meant that I had the ability to kind of learn about various different domains. I moved to London about 10 years ago.

[07:42] We're just kind of looking for difficult problems, you can build high traffic websites, and fine tune them and deal with your requests per second that all the different layers of the stack. But actually, when things get really difficult, that's when regulation turns up that specifically for example, it puts an extra hop in between things, and you're looking at trying to do a low latency service. But actually, you have to deal with some sort of reverse proxy, some sort of step up gateway kind of thing. The nature of those problems really appeal to me as an engineer, because of course, all engineering is a compromise. It's about finding the right thing to fit into the hole. When that set of compromises is regulators, then it becomes a slightly more difficult and interesting set of problems.

[07:38] Subsequently ended up moving through various financial institutions ended up at the UK home office, and they were ahead of their time. Thanks to what was called the GDS, government digital services, oh and incidentally, the home office in the UK is a governmental department. It's a ministry of the interior, rather than sort of the ubiquitous back room, which we all share these days. The home office were advanced by virtue of these previous projects, and they were running Kubernetes 1.2. I turned up, dealing with critical national infrastructure deployments onto Kubernetes. At that point, it was clear there was obviously 1.2, kind of pre our back, pre network policy, pre dynamic admission control. There's a huge number of things that were widely known security foot guns in Kubernetes.

[09:15] But because I've been deploying containerized systems for a couple of years preceding that, it was clear to me that this was the direction of travel. The pod concept enabled any application deployment topology. That really differentiated from Docker Swarm, which looked like it had a horse in the race for a while, and so I double down individually and sort of commercially. Seeing that we had this fully declarative interface of Kubernetes where YAML as far as the eye can see, I built a static analysis tool called cube sec, at cubesec.io, which uses a beyond call risk models style for Kubernetes resources. This pattern of static analysis on resources on plans kind of terraform style infrastructures, code, as well as the deployment code, sort of brings us to a place where we have stuff like opa today, that generalized open policy agent.

That sort of talk on that about the same time, it was very clear the direction of travel of cloud native security. It takes the journey to almost four years next week. For those of us listening in the past or the future, so well, yeah, perseverance is key.

[10:23] Justin: You brought up Docker Swarm? Why do you think Kubernetes succeeded and Swarm didn't?

[10:32] Andrew Martin: That's a great question. Potentially, the answer is to do with the lack of pod support. With a single process per container, Swarm was only designed to run single containers, which meant that we had to run more processes in one container, so we don't have to use full in it. Side cars, we don't have side cars for logging or security or service mesh envoy style, and service meshes were being shipped by Docker Swarm, before they were cool. They created a mutual TLS overlay network across the base network between the Docker Swarm nodes over which the micro segmented application networks were able to communicate. That's a pretty advanced networking model, compared to the more flexible, but less well defined CNI model in Kubernetes. Of course, we only got network policy in Kubernetes, because Calico shipped it and as with other things, the interface was extracted from the Calico implementation and baked into Kubernetes core itself.

From a networking perspective, Docker Swarm was much better out of the box because it was batteries included, but changeable, and came with its own networking paradigm. However, the inability to run multiple containers in a pod meant that there was no flexibility of application topology. That's really where Kubernetes stole the show.

[11:54] Justin: Thank you. That's a really great answer. You're an expert with Kubernetes. In 2019, you made some predictions, and I want to see what came true and what didn't. Hosted services, catch up with GKE. Did that happen or not?

[12:12] Andrew Martin: To some extent, I think GKE, and full disclosure, we wrote the CIS benchmarks for GKE, in this year, and contributed back some sort of thoughts and ideas around the supporting services. GKE still continues to push the envelope in terms of how Kubernetes is run, and how it integrates with the cloud. A good example would be, we have the kind of [inaudible 12:30] hybrids in Amazon AWS, Google's kind of GKE response to that is GKE autopilot, which removes the node abstraction. It's advanced thinking, and it's definitely pushing the paradigm. I think GKE continues to sort of lead slightly at the things like binary authorization, supply chain security before it was cool again. I have a soft spot in my heart for GKE of course. I still think it edges ahead somewhat.

[12:59] Justin: Shout out to Dan Lawrence there, the second one, non container VM base isolation improvement. Did it happen?

[13:07] Andrew Martin: Oh, yes, this blew up. We've now got things like gVisor, which is ultimately for App Engine to run. Google had this huge infrastructure, all these Borg cells, flexible compute to host, Gmail and Google search and calendar and maps. That was how Google deployed itself globally, and the self-contained cells. Amazon turns up and starts renting compute in the form of AWS and Google, say, we can do this, why aren't we doing this, and the first thing that they build, the app engine is running untrusted third party code. That is my code and your code on Google's Borg infrastructure on the backplane. In order to isolate those workloads from the underlying Linux kernels, Google built first version of gVisor, and gVisor is a user space kernel reimplementation, which means it catches system calls before they hit the kernel, and handled as much as that is possible, and a golang based kernel emulation.

[14:04] This moves the untrusted code further away from the Linux kernel base, which of course is written in C, and still definitely harboring loads of zero days that are probably being exploited somewhere by somebody. It gave the Google security team, that warm, fuzzy feeling of security. They do a couple of other things with gVisor including dedicated IO dedicated TCIP stack. You can run it in various configurations, but they realized they were onto something here and gVisor is now also runnable as a container runtime for Docker. It will run in GKE. If it's installed on a host node, you can start containers that are wrapped with gVisor. If an untrusted process is running, and by that I mean stuff perhaps or video transcoding because they're very complex is often somewhere that you can see bugs.

[14:56] Image tragic was a good example of how difficult it is to do that safely all time. Maybe it's also particularly risky workload, as in, perhaps we allow untrusted code to be executed within. If somebody gets remote code execution, or we gift it to them, or whatever it is, they can't run exploits against the kernel, because they've got this Shim, this gVisor layer in between. It's a kind of VM based isolation. We've also got things like firecracker, which is the AWS runtime for lambda, which is the same kind of idea, it's a very stripped down Virtual Machine Manager inside the virtual machine. It also creates container namespaces. It hangs set comp off not only the VMM process, but also the process inside the VM. These are really crazy hybrid approaches that take the best of both, they stripped down the QEMU, and KVM, virtual machine drivers, and give us this kind of hybrid existence where we can still take advantage of some of the container guarantees of horizontal scalability and the resistance to churn and add these virtual machine guarantees, which include a more difficult runtime to escape. Yes, it developed and

improves, I think, is suitably in specific to claim victory.

[16:15] Richard: Andrew, you have a lot of knowledge, right? I feel like a lot of this podcast of ours listening to you, well, you're very articulate and may just be the English accent, I don't know. What's like interesting to me is that you're basically saying all this extra stuff. You're giving all this context for where this team moved here. This team moved there. I think about the average listener, and wondering what there may be wondering, and I think the main thing is, where do you find this stuff out? Like, what do you use to keep updated on how open source works in cloud native? How do that say, Google realized that we're on to a good thing with gVisor. How do you know? Is there like a medium blog I should subscribe to what's going on?

[16:56] Andrew Martin: It is a great question. There are multiple layers. One is that I've been very fortunate to attend many conferences and meet some of my heroes. I mean, I met Eric Brewer at one of the cube cons, obviously, the guy invented CAP theorem. But even so many really remarkable minds have contributed into open source and code bases. Just going to conferences, and having enough of grounding or an understanding of people's work to ask them incisive questions, is a great way to just kind of dig in a bit deeper and get some context. We're very fortunate in that we've been able to provide services, some of the cloud providers and some of the big industry entities that also sort of provides them access to this, but really good and assiduous use of Twitter lists is really key.

[17:43] Instead of just following my timeline, I have a tech list. I have a sec list. Between those, I also trigger some RSS notifications. Sifting through those things, top Hacker News comments, not all the Hacker News comments, just helps. There's definitely some diamonds in the rough.

[18:02] Richard: So staying keyed into the ecosystem is a good way. Twitter really does continue to be in fantastic way if you can avoid timeline fatigue. Using list is a good idea using conferences is a good idea. Which conferences are you say the best ones to go to? Or do you have any that you're looking forward to this year?

[18:19] Andrew Martin: Well, if all the stars align, I'll speak at cube con in person. Just prior to that is the cloud native security day, the final masterstroke, the community understanding is of course, getting involved and places like tag security as it is now and the CNCF six security, the open SSF. These are all open-source organizations that helped to push the envelope by virtue of people just contributing their time and kind of collaboration. Those are the most fascinating and interesting things. Even just combing through meeting notes to catch up on a month being away is a useful way to maybe there's a GitHub issue. Maybe you can chase a pull request down and kind of dig through some of those things. In terms of conferences though, yeah, absolutely keep on the tag security day. Prior to that, we'll run the CTF there as well. Sign up for tag security. You also have the opportunity to run some nefarious, I don't know how much I can reveal but red pill blue pill, style CTF.

[19:21] Justin: Do you ever attend the tag security weekly meetups?

[19:23] Andrew Martin: Yes, I do.

[19:26] Justin: I learned a lot. My boss was like, check this out. I'm like rolling my eyes like oh God. But then I went to him. I'm like, wow, it's a lot of great information. I haven't run into you yet or maybe we did. I just didn't see you because there's so many people on it. Yeah, shout out to tag security for sure.

[19:44] Andrew Martin: Yes, I managed to be a member from early doors. Then my attendance is not what it could be.

[19:50] Justin: I wasn't trying to throw you under the bus or anything like that. I think Richard brings up a great point like you're expert in Kubernetes and I think that's why O'Reilly probably went to you and said, Hey, bro, we need to write a book. Let's talk about this Hacking Kubernetes book. It's not out yet. You wrote it with a fellow named Michael. When does it come out? Let's talk about this book.

[20:15] Andrew Martin: I discovered by virtue of searching for it on Amazon the other day, that is due out on the 30th of October, November. Yes, November, I think those briefly concerning and yeah, I wrote it with my excellent and studious co author, Mr. Michael Hudson [inaudible 20:31], who is a prolific writer already, and took me under his wing to guide me through the process. It was tremendously enjoyable to write. The opportunity for consistent research and making sure that I understood a concept sufficiently to write a few sentences on it is really positive reinforcing learning cycle for me. Really as I said, this acquisition of knowledge and interesting things, very much powers my kind of intellectual engine. I really enjoyed that part of the process.

[21:05] On the flip side, it required excessive dedication to the cause. Some sacrifices of various things, such as enjoying a drink from time to time, I was just able to hand in the final draft a few weeks ago. The cognitive unbundling that occurs when that happens, because everything that turns up is potentially something for the book and how do I contextualize this? Cross reference. Did I make that CV in there? It was a tremendously enjoyable exercise to undertake. What we've attempted to do is analyze the threat landscape against Kubernetes bathed in the light of historical vulnerabilities, and then try and paint some kind of future ideal hardened security state based on the impact of remediating risks, rather than kind of Iron Maiden lock everything down and cut move. Hopefully, people find that in some way useful.

[21:59] Richard: What do you think the most accessible chapter is that you look forward to having people read?

[22:03] Andrew Martin: I really enjoy the first chapter. Because it starts off with the pod and it decomposes the pod looks at a container. The great thing about Kubernetes container and Linux security is they are microcosms and microcosms of each other. The more that you zoom in on Kubernetes security, but it's actually container security for the process, of course, rather than the orchestrator. Then you drill down there and container controls are just Linux kernel controls, and then we go all the way down, and then we're back into some archaeological spelunking through why specific design decisions have persisted for however long. The book features my archetypal eight bit nemesis, the nefarious Captain hash jack. His prime directive is to plunder the data, that courses through the veins of the example organization of which the reader is a fictional seizer.

[23:00] We contextualize a number of attacks with the intent of a threat actor of Captain hashtag description. He's state sponsored, he's organized crime level, he's where a lot of organizations today will find they potentially have problems. Hopefully, this despite being slightly fantastical, anchors it somewhere close to reality and makes it more of a useful set of recommendations, rather than something a bit more abstract. Starting with the pods, is my favorite piece. We actually had to move Captain hash jack attack on a container from the first chapter into an appendix, because it's a content of 12 pages to truly explore the periphery. Combination of chapter one, and the appendix helps to set the scene for the rest of the book and ecosystem.

[23:48] Justin: Do you have any colleagues of yours, like review the book? Or is it all done with O'Reilly?

[23:55] Andrew Martin: I was so grateful for some incredibly detailed reviews. Editor, O'Reilly, [inaudible 24:00] was awesome. She connected us with some really established and advanced authors who were very competent with their reviews, and obviously preeminent in the space. I then reached out personally to a couple people in the scene who really, again, that they will get called out beginning but provided some invaluable feedback and also some philosophical views on some points that were made with a kind of higher level generalist perspective, which was, it was challenging to read. It meant that I had to sort of exercise my own thought patterns on the thing. That was one of my favorite parts of the whole process. Then finally, I obviously disseminated the "work" to the control plane team internally, and a number of people came back, what I would describe as critical bugs in parts of the book. We ran an internal bug bounty, and numerous people scored.

[24:57] Richard: This reminds me of one of the questions I was asking myself earlier while you were talking, which is control plane has 30 people in it. Security firm, obviously, you can't have all your tools be open source, because that doesn't work for security at all. One of the questions I have is how do you disseminate open source knowledge about cloud native infrastructure, cloud native programs through control plane, you just mentioned internal bounties, which is a really great idea for things like a book. But then a book is also a closed product that doesn't have to be re updated every time. What do you do to make sure that people that people at control plane are actually getting the best of the open source world without which it wouldn't exist?

[25:33] Andrew Martin: Democratization and open sourcing of security, tooling and information has been a constant source of utter amazement to me, ever since sort of starry eyed, I found, OWASP, when I was about 20, or so. As I said, I had this very privileged position of being in this company and having responsibility. I would drive to London, Bristol, which is 100 and something miles to attend OWASP events. At one point, I received the big thick book, and the last sort of 100 pages for all XXS attacks. People printed out all the Unicode strings that would cause problems. It just blew my mind. At some point in control planes existence, I have wondered, and I noticed the case for other security researchers in the cloud native space as well. Actually, we're a bit further ahead with the art of the possible than the current state of the art. There is potentially potentially exposing things that are not yet target for attackers.

[26:33] But clearly they will be in some period of time. We try very hard to open source, everything. We open sourced the threat models that we built for the financial services use group for Kubernetes. Of course, CIS benchmarks, open source, we've just built a new training course for O'Reilly called threat modeling Kubernetes, which has all of our ways and means to do threat modeling. Including, kind of default list of threats. Obviously, we have absorbed work that community's done, I folded that in with our own particular slant. I mean, we claim no credit for it, we're just happy to sort of push back. Where the difficulty comes, I think is the open core versus hosted security tool balance.

[27:13] We see this with a number of tools, whereby the core enforcer or scanner, or security tooling is open source. You can take that and you can deploy it in your infrastructure, and you can have a good result. But if you want to do that at scale, and have an awareness of what these things are doing over multiple, perhaps clusters, or nodes, or availability zones, that's where the secret sauce kind of SAAS hosted piece comes in. The only thing we don't open source is the infrastructure behind the accumulated simulator project, which is what we use to run the Capture the Flag events at cube con. We've been running that with tag security for the last two or three events. We have a wonderful time that the community contributes and totally good fun.

[28:03] Justin: Yeah, no, I hear that's like one of the biggest side events there. Now it's so cool talking to the creator.

[28:12] Richard: You don't mind, I want to see if we can get behind the creators brain a bit more to help out future participants of capture the flag. Earlier on in this podcast, you were talking about how you enter the back control plane, and you said I was looking for a difficult problem. You've also mentioned several times that you're incredibly privileged, which this is to me, those go together, once you've solved the bread and butter, once you have rents covered. A lot of people, in our circumstances, I include myself in this, are looking for difficult problems. I think for you is a particularly interesting question because you're a security researcher, and security researchers, by default are looking to find ways around things. Are looking to find problems, questions, answers, which are not public, which are hidden, which are not going to be written somewhere in an easy one to three step guide. I'm curious, I'm not sure how to best phrase this question and thinking about it, whether to ask what's interesting to you? Or what method do you use to find an interesting problem? How do you do security research in a way that like, makes you feel really excited about doing that sort of work? What does the difficult problem mean to you? How do you approach that question?

[29:20] Andrew Martin: That certainly triggers some self reflective thoughts. I think the most important thing, speaking personally, but also in fusing teams and displaying leadership, is to infuse or inculcate a shared sense of passion for a thing. I'm very fortunate again, that everything in cloud native is shiny, and so it's easy to find things that are, and we have this concept of flow, from a psychological perspective, something is a little bit more difficult, and we're a little bit less comfortable with it than the average. We can get into the state of psychological flow where we're kind of chasing the dragon of intellectual stimulation, and being in that quadrant of the graph is where I try and organize my time as an individual, where I try and infuse the organization that I'm very lucky to be a part of.

[30:08] Finding those things that inspire passion in people, and then performing research on them, it's kind of a happily, conducive feedback loop into itself. Finding that level of challenge, rears its head up with the interest levels of technology. Right now, there's a lot of interest in next generation, process isolation, and those kind of gVisor and capture containers and all that firecracker, good stuff. There's also a lot of interest in EBPF base kernel instrumentation and process analysis. There's been so much momentum building up for this, and the tooling is all available. We've just been waiting for the kernel versions in our deployed servers to catch up with the kind of five mainline, which brings in all this new functionality. Additionally, this gives us a whole new way to rewrite a number of tools that have had durational problems.

[31:05] I mean, of course, when Kubernetes started, IP tabled rule sets that were kind of in the hundreds or 1000s, when they reloaded, that was a two to three second downtime on all netfilter traffic, and you just have an outage. They had to rewrite IP tables with EBPF, which is the NF tables kind of shim. We can still interact by IP tables. But the netfilter code in the kernel that deals with that is now this hyperfast performance EBPF. These kind of things where we know the problem space, but there's something revolutionary, can really lead to lots of new code, we want new code because it brings us new features. But of course, new code is less tested or more potentially subject to bugs. Then we're into the constant software is never finished. It's part of an ecosystem that's always moving Moxie marlinspike can be credited greatly for that. These are the things that really piqued my interest and kind of keep me entertained from a professional perspective.

[32:00] Richard: Love that answer also [inaudible 32:01] is the coolest just straight out saying that. Excellent. You've personally answered one of the questions I was going to ask next, which is what are you most excited about in the cloud native security space coming up. You've already covered a few things, this may be a good point to go back to your 2019 Kubernetes predictions. The last one of the ones that we're going to get to is that the tangle of yellow was going to unravel by 2019. Did that happen?

[32:28] Andrew Martin: To some extent, there were, I don't know, leaps, bounds and steps that these taken in the right direction. We see things like customize, that make things at least easier to template across different environments. But I think the underlying issue of kind of meta templating or like higher order templating over YAML is still almost where we are.

[32:52] Richard: Okay, so it has nothing yet but maybe in the future, we hope. What about image and build metadata security matures? Has it matured?

[32:57] Andrew Martin: Yeah, that's a great one. Well, looking back on it, I appreciate the fact that I wrote it, let's say. What we seen is tools like cosine move into the space. Control plane have been hugely supportive of work out of the NYU cybersecurity labs, like tough and in toto over the past few years. But they were always lacking adoption. Which was able to bring content trust, as they called it into Docker, that required a secondary server to be deployed alongside an OCI registry. That overhead it was too much for most organizations. It's not a not a great deal of difficulty. But there was no stimulus. Suddenly, we see the solar winds loss of attacks. We see the kind of rebill pipeline attacks as well this year. Broadly, supply chain just enters the public consciousness at 1000 miles an hour, and starbursts when it's there.

[33:54] Suddenly, everyone's talking about supply chain. What we've now seen is this use of image and build metadata in all sorts of different ways. There's this amazing project called recall, that will take the hashes of things that we've built, and put them in a append only transparency look. We can now do this for binary artifacts. Then you've got tools like cosine, which can integrate the update framework with container signing, and give us this brand new signing style that actually encompasses all of the existing state of the art with one slight innovation, which is to put the signed metadata and signature into an OCI image and push it to the same registry as the image that it's signing.

[34:41] Because we're using that unique identifier for the image means that we have an immutable or a hash based link between the two. We can then deploy to production, validate that some evidence lake or recall or some external system has proof that thing was signed revalidate the signature to ensure that it's signed by somebody that we trust. Then finally, that's our admission control into Kubernetes. In production, the state of the art is now integrating tecton chains, whereby in toto is baked into every build step, you get the end of the build, build a container, you sign it with six store that implicitly trusts all of the cumulative in toto stages before that. We're into an amazing place where we can add software bills of materials into here and sign those two. While we may not have fixed the ultimate problem, which is third party code risk is very difficult to hedge against.

[35:36] GitHub is full of potentially untrusted code. We're now in a position where we are much stronger against insider threats. in the event of a discovery of potentially malicious code, we know which containers or which systems have that code deployed, and we can operate and affect patching strategy.

[35:52] Richard: Are you working with Dan Lorenzo on that? The six door?

[35:55] Andrew Martin: Yes, we have worked with Dan and Luke Hinds in the six door project.

[36:03] Justin: Shout out to Dan and Episode 20. You can hear more about that.

[36:07] Richard: Cool. You had one more prediction in 2019. But I think in the interest of time, I want to ask a different question. Which is predictions for 2023?

[36:18] Andrew Martin: Okay, that's a great question. Kelsey Hightower has been saying that Kubernetes is for building platforms. It's not the platform itself, building platforms, I think we're moving closer and closer towards that. What's happened over the past five years is that big companies who've built platforms on top of Kubernetes have been subsumed into larger organizations. We still haven't quite seen the thing. Rancher is doing an amazing job, but again, sort of [inaudible 36:40] last year. Still some sort of abstraction over Kubernetes, I think kind of makes that a little bit easier. I really love the GCP cloud run functionality, restates container and just runs it by itself. We're back into the same Swarm question there because of course, it's a single container, you don't have the lambda, secondary security container thing. But GCP cloud run runs on App Engine, so through gVisor onto App Engine, but it's k native compatible.

[37:11] Actually, one of the things that I really love is the function as a service approach on top of SEO, on top of K native on top of Kubernetes, because you then get full observability of the whole platform, you can apply this intrusion detection, you can use your namespace, aware tools in order to introspect more deeply. You can also satisfy what ultimately the kingmakers the developers require, because there's no point building a secure system if the developers can't ship business functionality through it. Platform for functions with Kubernetes is underneath.

[37:43] Justin: When we had Kelsey on a couple months ago, that's what he was saying. He says lambda and in open FAAS Functions as a Service, that's the future of like software development. I don't know if it'd be 2023. But who knows, I mean, maybe it will be. It could totally shift.

[38:01] Andrew Martin: We certainly see that with some of our customers now, there is a hybrid deployment approach when they're trying to build global platforms in multinational organizations, where they're interested in targeting everything from a virtual machine, through a Kubernetes into a server less platform. Of course, developers will go for the least friction, operations teams probably want to have some balance of uptime and availability and observability for things, which traditionally have been slightly more difficult with Functions as a Service. We've had to do things like keep functions warm by pinging them every so often, just because the underlying runtime garbage collect them. There are still operational and developer frictions when it comes to running these things in production. Certainly, it's the direction of travel.

[38:47] Richard: Awesome, great answer. Thank you so much for coming up with that on the fly. In fact, thank you for talking just in general during this entire podcast. It's been great to have you and to learn about your background, learn about control plane does learn about your new book coming out October, November 30 to you who are listening, this is going to be called Hacking Kubernetes. You have to say it with a British accent from now on. You can't say Kubernetes that's over. That'll be very fun. Please check it out. That's coming out of O'Reilly it will be available on Amazon and we hope that your predictions 2023 come true along with Kelsey Hightower. Now, Andrew Martin before you leave, we know that we can access you on Twitter and we can put the Twitter handle sublimino on our favorite Twitter list for tech security and probably other things if we wanted. We know we can see control plane @control-plane.io. Is there anywhere else on the internet where we can follow along with your thinking your words and the work that you do.

[39:52] Andrew Martin: There are a couple of interesting escapades you may wish to follow along with. One is sans sec. 584. Which I was very lucky to have the opportunity to write with some excellent co authors and co trainers. That is attacking cloud native and Kubernetes. It's a three day deep dive into red teaming, and then blue team responds. It ends up being a purple mix of the two. There is also a course for a rally called threat modeling Kubernetes, which is the paper-based view of the hands on sans lab. It teaches you things like connectivity matrices, how to model systems, and that default set of attacks and threat objects that we look to remediate against. It should be a jumpstart into Cloud Native defense.

[40:43] Justin: Is that based off their [inaudible 40:44]

[40:47] Andrew Martin: It's not actually in tune with [inaudible 40:46] yet. We do have one exercise in there. It will all be embedded in the future, though. At the end, we set up a lab, shout out to Ben Hall, the [inaudible 41:00]

[41:03] Richard: Well, follow along with those audience. Thank you so much for listening. Andrew, thank you so much for talking. It was great to have you on.

[41:11] Justin: I've learned so much like I don't think I can hold it all in my brain. But that's the beauty of this podcast is I can listen over and over. T hanks so much for coming on. I really appreciate it.

[41:23] Andrew Martin: Absolutely. It's been my pleasure. Thank you very much.

Special Guest: Andrew Martin.

Sponsored By:

• Curiefense: Protect Your Cloud Native Applications 👉 Try Curiefense today!