Episode 6: transcript

EPISODE 6: How to secure AI systems

“BD: Now we’re in this stage of, ‘Oh my, it works.’ Defending AI was moot 20 years ago. It didn’t do anything that was worth attacking. Now that we have AI systems that really are remarkably powerful, and that are jumping from domain to domain with remarkable success. Now, we need to worry about policy. Now, we need to worry about defending them. Now, we need to worry about all the problems that success brings.”

[INTRODUCTION]

[00:00:36] SM: Welcome to Deep Dive AI, a podcast from the Open Source Initiative. We’ll be exploring how artificial intelligence impacts free and open-source software, from developers to businesses, to the rest of us.

[SPONSOR MESSAGE]

[00:00:50] SM: Deep Dive AI supported by our sponsor, GitHub. Open-source AI frameworks and models will drive transformational impact into the next era of software, evolving every industry, democratizing knowledge and lowering barriers to becoming a developer. As this revolution continues, GitHub is excited to engage in support toy size, deep dive into AI and open source and welcomes everyone to contribute to the conversation.

[00:01:18] ANNOUNCER: No sponsor had any right or opportunity to approve or disapprove the content of this podcast.

[EPISODE]

[00:01:22] SM: Welcome to this special episode of Deep Dive AI. Today, I’m joined by co-host, Director of Policy at the Open Source Initiative, Deb Bryant. Welcome.

[00:01:31] DB: Thanks.

[00:01:32] SM: This is a new experience for me having a co-host. In this episode, we’re talking to Dr. Bruce Draper. He’s a Program Manager of the Information Innovation Office at the Defense Advanced Research Projects Agency, which is hard to pronounce, but it’s also known as DARPA. Much easier to pronounce for me. He’s a professor at Colorado State University in the Department of Computer Science. He has an impressive curriculum in areas that range from knowledge-based vision, to reinforcement learning, evaluation of face recognition, unmanned ground vehicles and more. Especially at DARPA, he is responsible for the GARD Project. This is an acronym that stands for guaranteeing AI robustness against deception.

Dr. Bruce, if I understand correctly, GARD’s objective is to develop tools to understand if a machine learning system has been tampered with, how wrong am I?

[00:02:29] BD: Well, it’s designed to develop tools that will defend an AI system against opponents who are trying to defeat it. So developing tools to make AI systems more robust, particularly against malintention adversaries, and to try to make those tools available to the larger community, so that all the AI systems that are out there will hopefully be secure.

[00:02:52] SM: Wonderful, so it sits perfectly into DARPA, the defense mission. Speaking of missions and stuff of DARPA. What is the mission of DARPA? People are familiar with it. I’m sure that our listeners remember ARPANET and all the research that came out of it. But what has DARPA done for us recently?

[00:03:11] BD: Well, of course, so DARPA was of course founded many years ago when Sputnik was launched. It terrified the American government and the American government decided did not want to be surprised again by technological change. The role of DARPA, our official mission is to anticipate and prepare for technological change and disruption. Along the way, we’ve done a lot of things. You mentioned the old ARPANET, things like global positioning satellites, was another hit of ours. If you ask the question, what have we done for you lately? I think mRNA vaccines are recent and very powerful example of work done at DARPA.

[00:03:46] SM: I didn’t know of the mRNA tied to DARPA, which kind of leads into the open-source aspect of this. So DARPA developed this technology that went into the vaccines, but then how these technologies being monetized and shared with the rest of the world?

[00:04:05] BD: Well, in the case of the mRNA vaccine, Moderna started as a spinoff from DARPA. DARPA got that technology out into the community through the pharmaceutical industry, with the hope of being able to vaccinate large numbers of people, which turned out to be really important with the onset of the COVID crisis. Similarly with the what we’re trying to do here with the GARD program, is sort of get technology out there that will be robust and safe against an adversary, before adversaries attack. So that as AI becomes more and more part of our everyday processes, and we’re all sitting in self-driving cars and all the rest, we can be reassured that the AI systems will work and work correctly.

[00:04:44] DB: Is open-source license a technology then a strategy to get your research or the product research out into the environment more quickly?

[00:04:55] BD: Absolutely. What we’re trying to do is get defensive AI, AI that will be robust against attack out into the community as quickly as possible before large scale attacks start happening. We’ve already seen small scale attacks on commercial entities and another entities. We would like to be able to sort of get out and you can’t really – inoculate is a strong word. But we’d like to go out and get defensive tools in people’s hands as quickly as possible before large scale attacks start happening.

[00:05:22] SM: How important is the open-source aspect of this? In other words, how important it is to the mission of DARPA and the mission of GARD project, to have technology that is available with licenses that basically provide no friction to use or no limitations.

[00:05:41] BD: So the mission of DARPA is, of course, to protect the United States and more generally, the free world. If you think about a scenario in the relatively near future, you can imagine a city where almost all or significant percent of the cars on the roads are self-driving cars. Now, those are not military systems as a privately owned, that’s your car, and my car, and all that kind of stuff. But if an adversary can tie that up, an adversary can defeat that and start causing crashes all over city, they could tie up any major city they wanted to. Not by attacking the military structure, but by attacking the civilian infrastructure. It’s not important to us that we defend not just the military systems, but that we defend all systems that are out there.

[00:06:25] SM: This is a very fascinating scenario, because a lot of the technology that we have right now without being a machine learning system is extremely vulnerable anyway. So what’s the difference between a machine learning system and a general IT computer science system?

[00:06:42] BD: Well, as you know, cybercrime has become a major problem and all kinds of systems out there are vulnerable. I think part of what happened is, we went out many years ago and started networking all the computers together and creating this great interconnected digital world that we now live in, before we really thought about the problem of what was going to happen with malicious people with malicious intent. The result was, the people doing cyber defense are constantly playing catch up.

People are constantly attacking old systems that don’t have all the latest defenses and that sort of stuff. What we’re hoping to do in the case of AI systems, is to get out in front, right? AI systems are there in the media a lot, you’re seeing them more and more often, but they’re not yet as widely spread as the sort of more conventional systems. But they’re going to become much more widely spread. They’re going to become something that everybody uses. So we’d like to go and get some defenses out into the world before the adversaries can get in front of it.

[00:07:43] SM: So you’re basically seeing a future that is already painted, right? In your mind, there is absolutely no question that the world has changed with the introduction of these new, more modern AI machine learning tools, reinforcement learning and all the pieces of it. They’re absolutely going to take over. There’s not going to be another sunset or upset like in previous generations of AI tools. Do you see it different this time?

[00:08:08] BD: I think what we’ve seen before, people talked a lot about AI summers and AI winters, right? Growth periods for AI and then periods where AI dropped back. But what’s interesting is it, with every summer, it sort of got a little more prevalent and that sort of stopped going for a while. Now, what I think we’ve finally done is we’ve hit a tipping point. I mean, just think about all the publicity we’ve had in the last years, whether it’s ChatGPT, or DALL-E, or any of these other programs that have gotten so much publicity. These systems are becoming very, very powerful. They’re becoming very, very capable. That can be a wonderful thing if they’re used well. It can be less wonderful if they’re used poorly. Part of our intent is to make sure that they’re not being attacked, and they’re not being tricked into things that they shouldn’t be doing.

[00:08:52] DB: So I was wondering if you’re going to talk a bit about – before you dive into the detail on the technology itself, I know in our earlier discussions, you’ve described an interest in broadening your community or engagement. Can you describe what an ideal ecosystem would look like? What kind of stakeholders are looking to join the process? We can kind of help frame this for some of our listeners.

[00:09:15] BD: We are interested in sort of creating two communities here. First thing I want to say is that all the research done under GARD is open source, public, available to everyone. What we’re really trying to do is create two sets of tool sets that are available to the broader community. One is designed for developers based around a tool set known as ART, the Adversarial Robustness Toolkit. The idea there is to give the people who are building AI systems a set of tools, give them access to the most current up-to-date defenses, give them access to all the standard attacks so that they can test their system and give them all the tools to build a system that will be as robust as possible. That’s one part.

The other part is a tool called Armory. Armory is targeting not the developers, but the T&E folks. The idea behind Armory, is we want the people who are testing and evaluating AI systems, whether they were developed in house, or whether they were purchased from another source. Nonetheless, most large projects will have a T&E group. That’s a different set of tools. We want to build tools that will get the T&E group, allow them to test how well defended a system is or conversely, how vulnerable might it be. So we’ve got these two sets of tools, one based on ART, which are targeting developers, one based around Armory, which are targeting the T&E folks.

[00:10:40] SM: So T&E is testing and evaluation group?

[00:10:44] BD: Yes.

[00:10:45] SM: So you’re saying that all these tools are released with an open-source license and they’re publicly available? How do you deal with international collaboration? What kind of collaboration do you see happening?

[00:10:56] BD: We view this as an open-source project open to everyone. In fact, one of the key developers at ART is IBM, including their team based in Ireland. Our hope is to make tools widely available and we’re not trying – this is not a uniquely American project. This is supposed to be an open-source project. We want everyone to have tools that are as safe as possible. It’s a very interconnected world, and we’re all buying software from each other, and transferring software among each other. If the US software is safe, but the, I don’t know, Canadian software has holes, what good is that when we put them all together? We have to make these tools available for everyone.

[00:11:35] SM: For sure. I got to say that, the only thing was a lot simpler a couple of years ago, and at least in my poor man’s mind, and between Europe and United States, everybody’s friend. We’re living in a global world that seems that has been challenged more recently. Do you see any of that friction happening in your world?

[00:11:56] BD: I think that friction impacts everywhere. I think our model is – certainly my model is that, what we want is as open a world as possible that empowers all the individual developers and all the individual people. Because in an open society, we do well if all the individuals have power. I think there are some societies that now that prefer a much more closed, centralized model. The reason we’re going with an open-source model is to support a model that does well in the free world.

[00:12:27] DB: I’m agreeing with a thesis that an open-source model is a great way to develop collaboration. The problems that the GARD program is addressing are ubiquitous. It is a global problem along with the authority pieces, but that I think it’s a great approach. I appreciate sitting on in today’s interview, my personal history includes working as a government practitioner, and over the years, I’ve worked with a lot of federal agencies. It isn’t until most recently, I think that the general public has become aware of how prevalent open sources in the federal government, they’ve been doing it for decades. But with a high challenge we’re experiencing, especially in cybersecurity, we’ve heard directly from public agencies in hearings, that it’s not just a matter of security and defense. It’s also a matter of innovation. So it’s an interesting time to see projects like this that are critical, using this particular model. That was actually how it captured my imagination, and about 2000. I thought the model itself was really more important than the software. The way the product was produced add as much value. I could see that in the government environment, so I have a lot of respect for the project that Bruce is running today.

[00:13:37] SM: I want to go back a little bit to the technology first. You said that every summer of AI has been a little longer and the winter is shorter. What do you think is the key element that triggered this summer to be longer?

[00:13:52] BD: Well, obviously, the onset of deep neural networks, which have since been expanded to also include things like transformer networks, diffusion models. All of this work was a coming together of what had been a lot of very basic mathematical research with the GPUs and other processors have finally allowed all that work to be done at scale. Also, frankly, the internet which made sufficient amounts of data available. This has just led to an AI summer that has impacted everything from computer vision to language, to planning, to reasoning. So many areas are being advanced now by this current AI summer.

[00:14:34] SM: On one hand is the basic research on mathematicians and mathematics. Hardware is another piece and third element, it will be data. Am I getting it right? That would be absolutely correct. With all these changes, math is fairly available, or at least you can study hardware and data start to get more complicated. Can you give us a little bit of an overview of who the partners are in current project? What does it take to become a contributor to a project like ours, something that is so deeply complex?

[00:15:11] BD: I think we have 15 partners in GARD. We have a lot of performers. The way you get involved with this is actually quite easy. So we have a website called gardproject.org. You can go to that project, it will take you – if you’re a T&E person to the Armory tools. If you’re a developer person, it will take you to the ART tools. We also have sections there, you can have all the tools in the world, you also need the intellectual background. So we have a section of tutorials put together by the folks at Google on defensive AI, and how to make good use of these tools and provide background. And we have a set of datasets provided by another one of our partners at MITRE to make it easier to test, and run and evaluate these tools as well. So we’re trying to make data and tutorials available, as well as these two large block sets.

Anyone can come to gardproject.org, go to our GitHub repository, start accessing these tools. I should tell you that the way we’re starting now, in terms of where most of the developer work is, most of the algorithms and algorithmic pieces that we’re providing through the art toolbox. Most of those have been developed by university partners, most academic partners. We have a few companies in there, but most of those people had been academic, where most of the work being done on the testing and evaluation side, the T&E side, most of that work has been done by companies like IBM, and MITRE and a company called Two Six, because that tends to be something that is often more of a corporate function.

But again, any researcher who wants to get involved is encouraged to get involved in either of those two communities whatever their role is. Let’s get involved, let’s all work together, let’s make the most secure systems that we can.

[00:16:57] DB: As a recovering university researcher, I feel obligated to ask this question. Does DARPA make available research grants for universities that might be interested in engaging the project or would that come from other funding strings?

[00:17:11] BD: No. DARPA, we are a funding agency. We don’t do the research in house. We fund other people to do the work. In the case of a project like guard, the majority of that work is being done at universities, under funding from DARPA. Those universities, although most of them are American universities, international universities can and do apply as well.

[00:17:32] SM: There’s one thing that strikes me as very interesting in what you’re trying to do with the garden project, is to talk about security and safety very early on in the development of these machine learning systems. Because it took me or not just me, I’m not a software developer, but it took me a long time to start to get into my head, ingrained the concept of security. I think that the general public is also not very yet still needs to get familiar with having password managers and very tiny little security related things. But for GARD, it’s really central and it looks, sounds to me like it’s really ahead of the curve, like lesson learned from the internet times where everything was open, and accessible and security was an add on. What made the AI community concerned so much that they need to invest immediately into early on?

[00:18:26] BD: Well, it’s funny. The first papers on adversarial AI occurred in the academic literature and around the 2015 timeframe. Then very, very quickly, it developed to the point that we were having actual AI attacks with commercial implications as early as 2017. There was a company called Syvance that made malware detection software. It was one of the first victims that I know of where people went in and were able to do an adversary, because they were using an AI system to determine whether a piece of software was malicious or not. Makers of malicious software, went in, did an AI attack, figured out how to fool their system, and then went in and were able to attack their customers.

It went very quickly from being something of only academic papers, to something that we saw being used in practice. We decided very quickly it was going to be necessary for us to find a way to defend against it.

[00:19:29] SM: What other scenarios keep you up at night?

[00:19:32] BD: One of the scenarios that keeps me up tonight is the self-driving car scenario. The reason that one keeps me up at night, is that right now, most people when they use an AI system, it’s not safety critical. Yes, I lead an AI system on Netflix, recommend what movie to watch. But if it recommends a bad movie, nobody does. Right? And indeed, one of the reasons why I think there has not been more work on defensive adversarial AI out of Silicon Valley, is because most of the things that AI systems today are being used for are not necessarily safety critical. But that’s going to change and the self-driving cars are simply, I think the first example of something that we’re going to see out in the public that is safety critical. If someone ruins my movie recommendation system, it’s inconvenient, but it’s not a disaster. But if someone disables the brakes on my car, that’s a completely different story.

[00:20:31] SM: This means that you are dreaming of self driving cars in the streets mixed with non-self driving cars in humans.

[00:20:40] BD: I am imagining, because we already have them, railroad trains that are pretty much completely digitally controlled. I think the reality is that AI is so good, it is so cost effective. When it’s not being attacked, it’s so safe, that we’re going to rely on AI to do more and more things for us that are important and that are safety critical. As long as we can defend them, that will be a good thing. But there is a nightmare scenario, where we all become dependent on AI systems that are vulnerable to attacks. That’s what we’re trying to avoid against.

[00:21:14] DB: Self-driving car is a great example. It’s also an industry that’s starting to embrace open-source software development models, operating systems. What’s the opportunity to be able to concurrently deploy these kinds of defensive AI systems really at the ground level, where we have companies like GM, and Daimler, who’ve publicly committed to using open source in their strategy in the car. Some of these are less mission critical. We’re talking about the entertainment systems, but I agree, I see it coming down the pipe. How do you co-develop those things concurrently? So by the time you get to market with a true self-driving vehicle that’s commercially available, you’ve also made it safe in that way.

[00:21:56] BD: This is more of the developer side, there’s the developer and the T&E side. What we want to do with the developer side is have this running art toolkit. What we hope will be happening is, there’s always a game of cat and mouse. You come up with a better defense, someone tries to come up with a better a tactic to get around it. What we’re hoping is that, if everyone sort of adopted the ART toolkit, and they’re using these tools than ever. Then when there’s a new attack that comes out, there will be a whole community of people out trying to develop a new defense against that attack. And because everyone’s hopefully using the ART interfaces, as soon as that defense is created, it can rapidly be promulgated across all the people who might be using it. I don’t think it’s possible to come up with one defense that will be perfect forever. That sort of silver bullet has never happened in cyber defense. I don’t really anticipate it happening in AI defense. But what we do want to do is make sure that all these commercial systems have the best-known current defenses on them, and that they’re tied into this ecosystem, so that when newer, better defenses become available, they can immediately be downloaded and incorporated.

[BREAK]

[00:23:07] SM: Deep Dive AI is supported by our sponsor, DataStax. DataStax is the real-time data company. With DataStax, any enterprise can mobilize real-time data and quickly build the smart, highly scalable applications required to become a data-driven business and unlock the full potential of AI. With AstraDB and Astra streaming, DataStax uniquely delivers the power of Apache Cassandra, the world’s most scalable database, with the advanced Apache pulsar streaming technology in an open data stack available on any cloud. DataStax leaves the open-source cycle of innovation every day in an emerging AI everywhere future. Learn more at datastax.com.

[00:23:47] ANNOUNCER: No sponsor had any right or opportunity to approve or disapprove the content of this podcast.

[INTERVIEW CONTINUED]

[00:23:50] SM: What do you see the role of policymakers in this field? All policymakers in Europe and the United States are perfectly aware of all the risks included in AI and machine learning systems and they’re starting to regulate the. Even though there may not be agreement in the academic groups about what needs to be done. What’s your feeling from your point of view of these policies, the draft policies that are circulating?

[00:24:20] BD: I think there are a lot of policies circulated and I don’t know that I’m well qualified to speak to the strengths or weaknesses of particular ones. But I do want to bring into this conversation is this notion of the T&E tools we’re developing. Because one of the things that we have to know in order to set any policy reasonably is what are the risks, how well defended is something. There’s always a tradeoff. You give up a little bit of accuracy to get a system that’s more robust, right? How much are we giving up? How much robustness are we getting? You can’t begin to have a sensible policy if you don’t know what the risks are in your ability to defend against those risks. Part of what we’re hoping we can do with the Armory tool is give the testing and evaluation folks some way to measure how much risk are they taking, if they take this AI system, and they use it in a particular way, how vulnerable is it to attack?

Like I say, some systems are not mission critical. It may be fine to go out with a system that is perhaps a little less robustly defended. Other systems are safety critical and need to have absolute state of the art. I think of government policymakers, I also wonder about insurance companies. If you’ve got a vulnerable self-driving car, that’s a real threat to the insurance companies. They might end up having to pay out if things go badly. So I think there are a variety of players both on the government side, and on things like the insurance side and the large company sides, who all have a vested interest in trying to make sure that these systems do have some extensive regulation and yet, don’t cripple the industry. I don’t want to get to a situation where we put so many regulations on that we can’t use AI. I don’t think that’s to our advantage either.

So I don’t know where the policy sweet spot is. I don’t even know if all the right players are in the game yet. But I want to create a set of testing and evaluation tools that will give them something that they can measure, that they can start to use to make sensible policy.

[00:26:22] DB: I have to say that would be a great contribution. We don’t know really where the most risk is. We don’t have a great inventory of what we own. There’s a lot of work ahead. I want to ask though, if you have a general sense of any gap you think that needs to be addressed today, in addition to obviously providing information to create more informed policy. Do you see an area of vulnerability that you think might be good public discussion for regulatory addressing?

[00:26:52] BD: I’m not sure. Let me instead answer a slightly different question to the one you asked, but one that I think that I can answer in a way that is more usable.

There’s an intellectual or an academic hole, which is that, we’re getting better at the practice of defending these systems and that’s what we’re trying to do with these toolboxes. We’re getting better at evaluating it. That’s what we’re trying to do with Armory. We still don’t have, however, what I would call a good deep theoretical understanding of what the threats are, and what the limits of the threats are. This is really getting back to sort of our deep understanding of these networks and the theory of adversarial AI. GARD actually has a – part of the GARD program is specifically designed to try to develop and push the theory of defensive adversarial AI.

I talked about that less, because unless you’re a PhD researcher at a university, at this point, we’re producing papers, I’m trying to advance this fundamental mathematics and there’s still a long way to go. But right now, the practice is ahead of the theory and it would be really nice to have something like what the encryption folks have, where you can talk about the length of the encryption vector, and how much security advisory. We don’t have that equivalent yet.

[00:28:05] SM: There is one thing that I noticed in AI practitioners that are really well aware of the dangers and the damages that unleashed AI can do to the world. Is this something that GARD is also looking at the dangerous uses of its own technology and machine learning models in general?

[00:28:28] BD: That’s a very broad topic. In the case of OpenAI, they were not worried about what an adversary that might do. They were worried about how their system could be used when it was operating correctly. That’s a very real – where do we want to use AI? Where do we not? What are the limits? Those are very real questions for ethics and other foreign sources, and best addressed by regulators and policy experts. We’re really looking at the question of adversaries and their ability to defeat an AI so that it doesn’t do what it’s supposed to do. We’re not concentrating in this particular program, on systems that work well. We’re trying to figure out how these systems can be broken by an adversary and how do we stop that from happening? The systems behave as advertised? There are separate policy questions as to where you want to use systems that behave as advertised.

[00:29:21] DB: So what have we not touched on, Bruce, that you think would be interesting or important for any institution, or organization, even individuals interested in participating or evaluating the GARD project?

[00:29:34] BD: First of all, we’ve touched on this briefly, but I want to invite people to the gardproject.org website. Depending on whether you’re doing T&E or you’re doing developers. Look at ART, look at Armory, look at the tools, get involved in this community. One of the things that we haven’t discussed is what DARPA does is we get in, we try to make an impact in an area and we try to create something and it will sustain itself. Then we get out and we do other things. The GARD project as a funded entity by DARPA will end next year. It will end in 2024. Our hope is by then that we have an active international open-source community that will continue this work on and allow this work to continue, even without direct DARPA support.

So that’s our goal. That’s why we think building this community is so very important. That it has to be a sort of self-sustaining. This is not something that we’re inflicting on the world. This is something that we’re hopefully trying to give to the world in the hope that people will look at it and see the value and want to build systems that behave as advertised.

[00:30:42] DB: That’s very consistent with a goal most community development is for a community to be kind of self-sustaining. Do you see DARPA having any other ongoing role or will it just be complete? In other words, will you have someone at DARPA that would continue to be a liaison or a super connector? Or do you see it being holistically moved to a new community, if things go as you hope?

[00:31:07] BD: We have some performers who are particularly IBM, for ART and Two Six for Armory, that will continue to work on the projects after 2024, and hopefully be a sort of organizing force behind the community. They’re both really expert, very good technical people, I think that they will be great. I think DARPA as always will look and see where there are problems. Our role at DARPA is to see where there’s a problem that isn’t being addressed. If the problems that this community is picking up and working on, we will let the community work on. If there’s something critical that we think is not being worked on, well, then DARPA may come in and try to address that problem.

[00:31:47] SM: From the type of partners that you would like to promote the projects to, are there any preferences? Do you need more academic contributors, or more corporations, government players or agencies in other parts of the world?

[00:32:04] BD: Well, first of all, I don’t want to discourage anybody. I want to have everybody as involved as possible. I think there are two particular groups that we look at, we look a great deal to the academic community, for the developer level work for the coming up with new algorithms, new defenses, things like that, we look more to the corporate community for the teeny level work. That’s something that tends not to happen in academia, so we’re hoping that we can get the industrial players to step up there. And we’re also hoping we may get governments to get involved and play a role at that level.

[00:32:38] SM: Are there any other projects that do something similar to what GARD is doing? Competitors, so to speak.

[00:32:45] BD: There are a number of smaller projects, particularly in the academic world, but also within government. As far as I know, GARD is the largest project and that’s why we’re trying to push the open source of it. There’s the old joke that, if you have one standard, it’s useful. If you’ve got 20, it’s not. What we’re really hoping to do, is to sort of build this around these two tools. Because at the moment, they have the largest uptick. For example, ART was recently recognized by the Linux Open AI Foundation, as one of its graduated projects for its degree of activity, and the number of people starting to use it in their work.

So that’s great. We want to sort of encourage that. Again, also, work through things like the Linux Open AI Foundation, right? Work through these other organizations that exist within the open-source world to make sure that we have an ongoing and viable community.

[00:33:42] SM: How did you get involved into this project? What caught your interest into going in research for adversarial AI?

[00:33:52] BD: Well, I came out of the computer vision and machine learning community. So he’d done a lot of work on the intersection between machine learning, and computer vision when I was an academic for many, many years at Colorado State University. That’s the point where these first adversarial papers started to come out from. There are these famous examples where if you added a little bit of noise to a picture of a panda, the system all of a sudden thought it was a gibbon, or you put a sticker on a stop sign, and your self-driving car thought it was a speed limit sign instead. I happened to be working in the general area where the sort of first attacks came out of. So then, when I got to DARPA, and the question was, how do we help the nation? How do we help the free world? What needs defending? I said, “Okay. This is an area.” And it turned out that another PM had just started this project up and had to leave DARPA unexpectedly. So I stepped in shortly before the project started, and have just enjoyed working with this community tremendously.

[00:34:54] SM: What’s next for you?

[00:34:56] BD: Well, what is next for me for the next half year is continuing to run these programs, and also continuing to make sure that that smooth handoff happened. For those listeners who don’t know, one of the ways that DARPA stays fresh is no one’s allowed to be a DARPA Program Manager for more than five years. So we all come in, we do a tour of duty, we try to have as big as impact as we can in a short period of time, and then we go back and return to wherever we came from. In my case, that’s the academic world. So I will be returning back to the academic world in half a year’s time or so, but there will be other DARPA PMs to come on and continue this type of work.

[00:35:35] DB: If you would have known what’s happening today, 10 years ago, would you have expected this evolution? I was involved in supercomputing early on, and AI was just something that was interesting to talk about, but it was – it was in its winter, I think. What do you think has been the most significant change in both its opportunity, its promise and also its threat?

[00:36:00] BD: It’s really funny, because for so many years, the question in AI was, could we make anything work? Or could we make anything work in a way that was reliable enough that you would ever let it out of the laboratory. Now, we’re in this stage of, “Oh my, it works.” That’s why issues like defending Ai have suddenly become important. Defending AI was moot 20 years ago. It didn’t do anything that was worth attacking. Now, I say we spent the first 25 years trying to figure out how to see.

Now, the question is what to look at. It’s the same sort of thing. When AI wasn’t powerful, when AI could only do very niche things, we didn’t have to worry so much about defending it. Bad actors or the financial business models, there are lots of things that didn’t matter. It was just a miracle when something worked. Now that we have AI systems that really are remarkably powerful, and that are jumping from domain to domain with remarkable success. Now, we need to worry about policy. Now, we need to worry about defending them. Now, we need to worry about all the problems that success brings.

[00:37:16] DB: Well, I’ll be watching with interest the science fiction literature field, because this is all the stuff that science fiction, the best was made of. Machines would take over and the things that we do. So now we all know that they can do that. What’s next, then?

 

[00:37:31] BD: They can also be wonderful partners. They can also be – most of the work that I do at DARPA is using AI to make people better, stronger, smarter, more capable. I think there’s an awful lot of us who are really interested in using AI not to replace people, but to improve them, and make them more capable, and to empower individuals. That’s certainly my interest. So, there are risks involved, but there are also just wonderful opportunities.

[00:37:57] SM: Like many other new technologies involved, striking that balance between what it can achieve and what damage it can do.

[00:38:04] BD: I encourage people, gardproject.org. It’s like everything else in open-source software. The more people that are involved, the more brains we get on the project and the more eyes to make sure these systems are being used properly, to know how secure they are or aren’t, so that we know whether or not we want to put them in a particular critical role. The more people we have involved in that, so it isn’t just one or two people making their “expert opinion.” But it’s a large community of people, all with different backgrounds and all different expertise, getting involved. That’s what we’re looking for. I think that’s what will give us the most robust AI in going forward.

[00:38:45] SM: I’m particularly grateful for this conversation, because we covered in these podcast a lot of other threats that are coming from AI machine learning, like discrimination in data sets, or other damaging uses and improper proper uses, of properly working systems like you were saying. But this adds another layer of complexity and another layer of policymaking needs to be taken care of.

[00:39:12] DB: Now, I really appreciate the insight into a very valuable topic, and exposure to your project. I wish you great success on the project. Thank you.

[00:39:21] BD: I understand, this is your 25th anniversary for us. So thank you. Congratulations. The open software movement is such an important thing to spreading technology across the world. So thank you for your work as well.

[00:39:37] SM: Thank you.

[END OF INTERVIEW]

[00:39:38] SM: Thanks for listening. Thanks to our sponsor, Google. Remember to subscribe on your podcast player for more episodes. Please review and share. It helps more people find us. Visit deepdive.opensource.org, where you find more episodes, learn about these issues, and you can donate to become a member. Members are the only reason we can do this work. If you have any feedback on this episode, or on Deep Dive AI in general, please email contact@opensource.org.

This podcast was produced by the Open Source Initiative, with the help from Nicole Martinelli, music by Jason Shaw of audionautix.com, under Creative Commons Attribution 4.0 International license. Links in the episode notes.

[00:40:20] ANNOUNCER: The views expressed in this podcast are the personal views of the speakers and are not the views of their employers, the organizations they are affiliated with, their clients or their customers. The information provided is not legal advice. No sponsor had any right or opportunity to approve or disapprove the content of this podcast.

[END]