APPLE | GOOGLE | SPOTIFY | OTHERS

The strategic and security implications of AI

Helen Toner on the TDS podcast

To select chapters, visit the Youtube video here.

Editor’s note: This episode is part of our podcast series on emerging problems in data science and machine learning, hosted by Jeremie Harris. Apart from hosting the podcast, Jeremie helps run a data science mentorship startup called SharpestMinds. You can listen to the podcast below:

Listen on Apple, Google, Spotify

With every new technology comes the potential for abuse. And while AI is clearly starting to deliver an awful lot of value, it’s also creating new systemic vulnerabilities that governments now have to worry about and address. Self-driving cars can be hacked. Speech synthesis can make traditional ways of verifying someone’s identity less reliable. AI can be used to build weapons systems that are less predictable.

As AI technology continues to develop and become more powerful, we’ll have to worry more about safety and security. But competitive pressures risk encouraging companies and countries to focus on capabilities research rather than responsible AI development. Solving this problem will be a big challenge, and it’s probably going to require new national AI policies, and international norms and standards that don’t currently exist.

Helen Toner is Director of Strategy at the Center for Security and Emerging Technology (CSET), a US policy think tank that connects policymakers to experts on the security implications of new technologies like AI. Her work spans national security and technology policy, and international AI competition, and she’s become an expert on AI in China, in particular. Helen joined me for a special AI policy-themed episode of the podcast.

Here were some of my favourite take-homes from the conversation:

• AI is poised to have transformative effects in a lot of strategic areas. But Helen also thinks that AI might not impact others as much as many think. As an example, she mentions Deepfakes, which is often cited as a potentially dangerous disinformation tool with wide-ranging implications for the integrity of the democratic process, among other things. Helen points out that Deepfakes still require significant resources to deploy, and there are lower hanging fruit for bad actors interested in influencing public opinion or elections, including the use of simple tools like Photoshop, or hiring low-wage workers to put out massive quantities of tweets, for example. As she sees it, there’s an opportunity cost to using tools like Deepfakes that needs to be accounted for before we decide how worried we should be about their potential impact.
• My assumption has generally been that governments move slowly, and that they might therefore struggle to keep policy up-to-date as the pace of development of AI technology continues to increase. While Helen agrees that the typically slow pace of government is something worth keeping in mind, she also points out that there are different tools available to policymakers and government bodies to move faster when a pressing need presents itself. She cites the massive COVID-19 relief package as an example of significant government action that came together on a surprisingly short timeline.
• Helen discussed some common misconceptions about the state of AI competition internationally. One in particular had to do with China’s supposed “data advantage”: the idea that due to its large population, and near-ubiquitous application of AI technology, China has access to more data than its international rivals, and therefore, an important edge in AI. Helen doesn’t consider this to be a compelling argument, for two reasons.
• First, data aren’t fungible: you can’t currently use facial recognition data to help train a chatbot, or text data to train a chess-playing RL agent. AI applications are still narrow, so the fact that China may have more “bulk data” doesn’t translate into a clear advantage where it counts (e.g. security and strategic AI applications). And second, the “China is big” argument tends to ignore the multinational character of many US tech firms: Facebook and Google aren’t just serving the US public — they’re used by billions of people worldwide. As a result viewing data availability through a simple “US vs China” lens isn’t particularly helpful or informative.
• Helen points out that talent is a critical input to AI development initiatives, both in countries and in companies. Historically, the US has had a huge talent advantage, owing to its being a nice place to live, and to its being a hub of technical talent. Helen is concerned that this may change, particularly as restrictions on skilled immigration have made it harder for AI developers and tech workers more generally to move to the States.

You can follow Helen on Twitter here, or follow me on Twitter here.

Links referenced during the podcast:

• The website for the Center for Security and Emerging Technology (CSET).

Chapters:

• 0:00 Intro1:19 What is CSET?
• 2:41 CSET’s current challenges with AI policy
• 4:03 Consistent concepts within AI
• 6:46 Human bias
• 11:20 Rival nation-states and companies
• 15:00 Promising strategies for trustworthy AI development
• 17:39 Government involvement
• 21:09 The idea of consensus
• 23:53 China’s AI capabilities
• 30:21 COVID’s impact
• 34:38 Identity indicators within AI
• 40:02 The alignment problem
• 42:53 Wrap-up

Please find the transcript below:

Jeremie Harris (00:00):

Hey everyone. Welcome back to the podcast. Today we’re talking to Helen Toner, who is the director of strategy for the Center for Security and Emerging Technologies at Georgetown University. Now Helen’s work is focused on helping policy makers navigate the strategic and security implications of new technologies like artificial intelligence and machine learning. That means taking on an awful lot of issues, ranging from privacy to international competition, to safe AI development. So we’ll be talking about all those areas quite broadly, but we’ll also be zeroing in on one area in particular where Helen has an awful lot of experience and expertise. And that’s the question of China and their AI strategy. Helen’s actually spent an awful lot of time studying China’s AI ecosystem through her work at GovAI, the Future of Humanity Institute, where she worked as a research affiliate while she lived in Beijing. So she has a lot of experience, a lot of great insights to share, and I’m really looking forward to diving into the conversation. I hope you enjoy it as well.

Jeremie Harris (00:50):

Hi Helen. Thanks so much for joining me for the podcast.

Helen Toner (00:53):

Thanks. It’s a pleasure to be here.

Jeremie Harris (00:55):

Well, it’s a pleasure to have you, you’re doing some really interesting work actually, and I think you’re the first person that I’ve gotten to speak to who’s working full-time on the AI policy side of things. You’re working at CSET C-S-E-T the Center for Security and Emerging Technologies. Could you start maybe by introducing what CSET is and why the organization is focusing so much on the policy challenges that arise from AI specifically?

Helen Toner (01:19):

Yeah, for sure. So CSET is a policy research and analysis organization based at Georgetown University, something roughly like a think tank. We were set up around two years ago, early 2019, and really one of the biggest things that motivated CSET being founded was really this idea that there are a range of emerging technologies that are affecting national security and affecting sort of international relations issues. The current setup in the US government and in DC is really not set up to combine expertise on these technologies with also expertise on how they will affect these particular issues. So there’s not that much infrastructure set up to really think about the intersection of these different topics.

Helen Toner (02:05):

And in particular, and to answer your question about why focusing so much on AI, artificial intelligence and especially machine learning, deep learning is one technology where the potential impacts on national security have grown so rapidly over the past, even the past 10 years and the sort of policy expertise to keep up has not been there. And so we wanted to set up a center that was able to really work at this intersection and to understand the technical issues well enough to speak to those and the policy issues well enough to speak to those and to really think about the ways that they connect and the implications of that has for US policy.

Jeremie Harris (02:41):

And specifically with respect to AI policy, are their current challenges that you have your eye on most, or is it just sort of a broader mandate?

Helen Toner (02:47):

It’s a pretty broad mandate. We’re also probably going to expand it into other technologies in the relatively near future to give a brief rundown of some of the things CSET works on without getting too exhaustive. We do some work on sort of the foundations of AI is how we think about that or the sort of inputs to AI. So that includes talent, it includes hardware. So chips, semiconductors, includes data, funding, things like that. We do some work on potential applications of AI. So this includes applications of AI for defense but also includes things like how AI interacts with cyber operations, for example. So we have a whole project called the cyber AI project that is thinking about the ways in which machine learning will and also won’t affect that. Because it’s one can always be interesting, not just to point out sort of important ways that machine learning might change things, but also to debunk claims that are not actually true.

Helen Toner (03:38):

And then we also think about some other tools and levers. So we do some work on AI’s potential intersections with kind of diplomacy, thinking as well about trade policy, industrial policy, potentially. And then also thinking about one area that we’re really interested in having published too much work on yet is thinking about sort of standards and testing for AI in the national security context as well.

Jeremie Harris (04:03):

Interesting. So you mentioned there this idea of some things that remain constant in the context of AI, things that might not change. Can you share a couple of those things? What are some common misconceptions in terms of areas that we might sort of broadly expect shifts, but where we might expect that incorrectly?

Helen Toner (04:17):

Yeah, I mean, I think AI brings this really interesting new things to the space and some of those new things can be sort of very shiny and exciting and seem like they might be revolutionary. The one that’s coming to mind for me right now would be deep fakes for example. So this ability to swap in, for example, someone’s face onto someone else’s face and make a fake video of a political figure or whoever it might be saying something that they didn’t actually say. That’s one area where I think there’s a lot of interest. I think it is a super interesting area as part of the broader kind of synthetic media. So fake videos, fake voices, fake texts or non-human generated texts, I guess I should say. But we’re a little bit skeptical that deep fakes for example, are going to have impacts that will be as revolutionary as some people have said.

Helen Toner (05:04):

So there’s been some commentary on deep fakes, for example, saying that this is going to be the end of truth, that this is going to be an enormous tool for disinformation campaigns. For example, if Russia wants to try to interfere in our election, that they’re going to be able to create some video of a candidate and just totally work the results by effecting people’s perceptions.

Helen Toner (05:26):

And we have a paper by a colleague of mine called Tim Wong looking into not saying that that’s definitely not going to happen, but really looking into with these tools as with any tools, there’s going to be a cost benefit trade off here. And so if you’re the internet research agency, the IRA, which was heavily involved in the 2016 elections on behalf of the Russians. You’re employing many, many people to work on these topics that if you can employ someone for a very low, hourly wage and they can be making tweets at an incredibly high rate per hour. And that’s going to have some return on investment and you compare that with setting up all the infrastructure that you would need to create a convincing deep fake, and that’s going to have some return on investment.

Helen Toner (06:08):

You’re just going to be thinking about what the better return on investment is. And it’s really not clear that deep fakes in that setting are actually going to be more effective than perfectly traditional. I say traditional in the sense of traditional in the social media era. So in the last 10 or 20 years, not before that. But there are plenty of non-AI related ways that bad actors can do these things. And it’s not at all obvious that defects will have that big of an effect. So that’s sort of one example where there can be a lot of interest in something and sort of some expectations of how it might go. And we just want to step into that conversation and say, “Hey, let’s think about, how are they actually going to go, are these assumptions right? What might be different?”

Jeremie Harris (06:46):

It’s interesting, especially thinking of the context of the kinds of ways that humans tend to be biased to receive and perceive information. I’m thinking of my own reaction to deep fakes. It’s just so easy to imagine deep fakes being deployed in some nefarious way, and because it’s so easy to picture, your mind kind of gravitates towards that and starts to get a little bit worried. Does that maybe point to a broader problem or rather challenge when it comes to advocating for security with these kinds of technologies that you’re really bringing forward a lot of abstract concerns. You’re trying to raise things that have to do with trends, with growth, with exponentials, with how the future may or may not look. Has that been a challenge when you’re talking to policy makers trying to ground this advice in relatively concrete terms when there’s so much uncertainty floating around?

Helen Toner (07:32):

Yeah, I think that is a challenge. I think it’s also a challenge to know how to think about the balance between concerns that we can point to today, concretely that are already happening out there in the world. And concerns that foreseeably seem very likely to arise in the near future, but that we don’t yet have concrete examples of. And to sort of make that case and to figure out also how to balance our concerns between those things, because it’s actually not clear what is the better thing to tackle by whom in which ways? And so if we’re talking with, for example, members of Congress, one of their biggest concerns is of course going to be, how does this affect my constituents? Is this already happening? What can I do to better serve my constituents on this? Or cynically, what’s going to be better for getting me reelected?

Helen Toner (08:21):

And that’s very different from, for example, talking with someone in say a department of defense office that is specifically tasked with thinking about future technologies, they’re going to be much less worried about what do we already have evidence of? And going to be much more interested in, what do we have reason to believe will exist in the future? And so, in some ways it’s just a question of who you’re talking to and what they’re interested in. Though in other ways it certainly is also a question of uncertainty about what actually is the most impactful or important things to be working on. So it’s definitely a challenge.

Jeremie Harris (08:50):

And do you expect that current concerns are going to smoothly turn into future concerns? In other words, are the deep fake problems of today going to… Do they exist on a smooth continuum that we can gradually deploy new policies to address? Or are we going to see stepwise changes where things that aren’t possible today will be possible five years from now, and then that will have big policy implication?

Helen Toner (09:14):

I wish I knew the answer to that. It would be really great to have a clear answer to that. The best answer I have is probably some of both. I think there are, for example, challenges we can see today in just really simple things relating to AI systems around how robust and resilient they are and also how interpretable they are. I’m sure that you’ve had previous conversations about this, where clearly we’re at a point where we can use AI for things like identifying what the photos on your phone are. So on my Android phone, I can go to my photos app and type in noodles and find the last few times I cooked noodles and look at whether they got more beautiful or not, which is a very important question for me.

Helen Toner (09:52):

And that’s the point, my point is, that is a very low impact context. And so we are at a point where AI is being used all around us, but it’s mostly being used in ways where if it goes wrong, it’s not a big deal. And that’s the reason that is the case is because we really don’t have ways to be confident that AI is going to function robustly and reliably across a range of different settings. And we also don’t have good ways of understanding why did a given AI system make the decision that it made in sort of a human interpretable way? And I think those are examples of concerns that I think are pretty likely to sort of smoothly transfer into the future. As these systems get more capable, there’ll be more potential incentive to use them in higher stakes settings, because maybe they’ll save more money or there’ll be faster, or they’ll be sort of more effective. But at the same time, the potential downside increases and the importance of being able to build these systems reliably and interpretably and so on, certainly increases.

Helen Toner (10:47):

I think for other things it’s harder to say. I mean, certainly in a national security context, something that it’s impossible to escape is thinking about other countries and ways in which they might use AI. And so that’s something where you could imagine that if a US adversary, like I guess Iran or North Korea suddenly feel that some system that would be a step change and a surprise potentially or same thing for countries that are usually classed as competitors like Russia or China if they do something unexpected, maybe that’s step change in what you have to worry about.

Jeremie Harris (11:20):

And is there, I mean, there’s a lot of talk about the risk of a race to the bottom on safety or on responsible development deployment. And you’ve got rival nation states, even rival companies within a country. Is that a problem class that you’re exploring at CSET? And is that something that you’re optimistic about or just broadly uncertain about? I mean, I imagine there’s a lot of uncertainty, but.

Helen Toner (11:43):

Yeah, definitely a ton of uncertainty. And this really relates to what I was just talking about with kind of robustness interpretability and related issues sometimes grouped together under AI as a category. This is definitely something that we think about. And I think a really useful concept here is the idea of a security dilemma, which comes out of sort of international relations as a field. And the idea of the security dilemma is you could have two actors, maybe two countries and one country wants to defend itself, are reasonable of course it does. And so it takes some measures to defend itself. And the problem comes in if it’s not clear, whether those things that it’s doing are purely defensive or if they might look potentially offensive. So if you think about, for example various defensive weapons, systems could potentially also be used offensively.

Helen Toner (12:27):

And so the security dilemma comes in where this other country that’s observing also wants to defend itself. And it’s kind of concerned about whether the first country is being defensive or being offensive and the sort of preparations that it’s making. And so maybe you end up with both of them increasingly building defensive and offensive capabilities, because they’re worried about what the other one is doing, even though neither of them actually have the intention to do anything offensive necessarily. And you get a very similar situation with the potential use of AI systems, where the department of defense, for example, understands very well that current machine learning systems are not robust. They’re not interpretable, they could easily, an adversary could cause them to fail using adversarial inputs or other things, or they might just fail on their own. And so the department of defense obviously doesn’t want to field systems like that, but there’s a concern that competitors or other countries might have lower standards in terms of what they’re willing to field.

Helen Toner (13:21):

And so if that’s the case and these other countries are developing systems that might somehow give them an edge over the United States, then perhaps the US should be developing those systems in parallel so that we have the option, even if we’re not sure. And then of course that incentivizes the other countries to develop in a similar way. So that’s sort of the way that I think about what sometimes gets called a race to the bottom on safety is, how are kind of countries that are really trying to be sort of thoughtful and rational about how they’re developing these technologies, but they’re also really valuing their own defense and their own security. Are they going to feel able to be cautious and to take the time to build in safety and robustness and reliability and interpretability into their systems, or are they going to feel the need to go faster?

Helen Toner (14:07):

And so I think one thing that can help with this is just acknowledging that this is the case in AI, that these systems do have these problems that they have, and sort of talking about that pretty publicly. And we’ve seen some discussion of that among defense officials in the United States. I would love to see more potentially you could look at for other ways of building understanding. There’s a paper, there’s some good work being done on this by people like Paul Scharre at the Center for a New American Security. Mike Horowitz, who’s a professor at the University of Pennsylvania. They’re looking into things like confidence building measures. Are there ways that you could, even, if you’re in a sort of competitive relationship with another country, are there ways that you could reduce the chances of this? And I think I feel slightly optimistic, but maybe mostly uncertain about the prospects of that overall.

Jeremie Harris (15:00):

And I guess it depends greatly on the game theory of the extent to which you think other nations are going to play nice and be forthcoming with their data, which kind of points to another issue. Something that I know you’ve done some thinking and research about, which is this idea of trustworthy AI development, the ability to monitor what kind of AI work is being done. So if you have a country or a company or a party that says, “Hey we’re abiding by all the norms and standards, everything’s fine. Everything’s great.” But then they’re hiding somewhere a research program that’s violating some of those rules. What strategies do you find most promising in terms of sort of next steps towards that idea of trustworthy AI development?

Helen Toner (15:38):

Yeah, it’s a great question. I think it’s quite a broad question and therefore difficult to answer it because I think there are lots of different things that get put under this category. In some ways I sometimes hear trustworthy AI used to describe just sort of safe AI in general, or AI that you can trust, which I guess is a separate question from the idea of a development process that is trustworthy or being able to make verifiable assertions about what types of AI you are or are not developing. I don’t know that I have great answers on this. I think it’s very difficult. One paradigm you might come to this with would be a sort of non-proliferation paradigm where, for example, the go-to example would be nuclear power or nuclear weaponry. Or we have this whole international regime set up that’s designed to detect and prevent the spread of nuclear materials and thereby prevent nuclear research and the development of nuclear weapons programs. And a really key factor there among others is the fact that you need to be physically moving around nuclear fissile material.

Helen Toner (16:47):

There’s other factors such as can you observe these things from overhead imagery, facilities from overhead imagery or so on. So the nuclear system is relatively you could say it’s available in that sense. It’s a really big challenge for AI or for these kinds approaches to thinking about AI development is that software is very difficult to surveil. It’s potentially a more promising area to be looking at the kind of hardware side of things. So perhaps you can observe data centers in overhead imagery or so on. But I think a lot of the work that’s been done on this so far is quite speculative. And it’s also hard to know what exact situations you would be in where this would become relevant and what the sort of prevailing institutions would look like at that time and what the players would be. And I feel like all of those would be pretty important pieces. So I don’t feel like I have good answers at this point, unfortunately.

Jeremie Harris (17:39):

Yeah. I guess that’s why it’s so important to be working in this space to begin in with. Yeah. One area or one dimension that comes up a lot, at least when I start thinking about the interaction of a government policy and then a very, very rapidly moving technological landscape is time. So when I think of the things that move very, very rapidly, like AI technology, like those developments. I don’t think of government policy response is moving at necessarily the same pace to the extent that we’re on an exponential course with AI technology and AI capabilities. What are your feelings in terms of the prospect of government keeping up with the policy requirements of this complex space?

Helen Toner (18:18):

Yeah. I mean, I’ll start by saying, I’m not sure that an exponential curve is the right way to characterize the AI development. I think it’s not clear that that’s the case and it probably differs a lot between different sorts of applications and different ways of using AI. I do think that the underlying point though, of things moving very quickly and governments not being, especially designed to move quickly is a really important one. And I’ve definitely been interested in the time that I’ve been in DC, I’ve only been in DC for a couple of years. But it’s been really interesting during my time here so far to be learning in a more sort of up close and personal way, how these government processes look, what different agencies look like, what different people working there are doing day to day and what they can do, what they can’t do, what they have time for, what they don’t have time for.

Helen Toner (19:05):

Again, I wish I had better answers. I think the underlying observation that this is going to be a challenge is right. I will say that governments, the US government, US is obviously a large country and correspondingly the government is very large. But that also means that it has very different components within it. So the way that something like the department of defense works, for example, is very different. Even from another executive branch agency, like the department of state, it’s going to work extremely differently, let alone something like the Congress. So House of Representatives or the Senate, those work differently again. And I think in each case, there are types of things that those institutions can do more or less quickly. So in some ways, Congress, when it really wants to be quick, it can actually be quite quick.

Helen Toner (19:49):

And I think that COVID really from earlier this year is an interesting example of that, where obviously we’re now mired in lack of negotiations about a second round, but the first round by government standards really went out quite quickly, given the enormity of that decision. I guess what I’m saying is that in principle, it is possible for them to move quickly on decisions like this. And likewise within executive branch agencies or so on, it’s just really going to depend on the agency, on the office, on the person involved in terms of what authorities they have and what types of things they’re able to move. One thing that I think we’ve seen over history is that often in these big bureaucracies, if you have one person come into a relatively senior position who really has one thing that they want to get done, they’re often able to make some amount of progress on that thing.

Helen Toner (20:33):

And sometimes quite a lot of progress on that thing in ways that even if others have been struggling to get traction on it for a while, if they really make it their top priority and they really just drive it through. And so, yeah, I guess I’m trying to agree with your underlying concern while also saying that I think there is a little bit more complexity to it. And it does depend quite a bit, I think on the specifics of a given issue on, for example, how much consensus there is external to government on what needs to be done, on the individuals involved and especially on the very specific mechanics, whichever part of government is involved and what they’re able to do in which ways,

Jeremie Harris (21:09):

Speaking of consensus, actually, I’m curious about the consensus within government, within policy circles, to the extent that you can speak about this. Because I understand some of this stuff might be sensitive, but what is the general view when it comes to risks from AI, the amount of attention that should be paid to this area? Is this something that a lot of policymakers are paying attention to already? Or is that something you expect to happen more down the road?

Helen Toner (21:32):

Yeah, again, the US government and the policy space in general are very large. So I don’t know that there’s a ton of consensus at that level of generality. I think many people are interested in AI expect that it’s going to be a big deal, are thinking about it in terms of increasing use of big data more generally, the role of the internet, social media, sort of thinking about it in this sort of broader context of ways in which society is changing more generally, in there are so many different subject areas we could talk about. If we’re outside of the national security area I think there’s definitely a lot of concern in the use of AI, for example, in the justice system or things like this, I’m thinking about bias, thinking about ways in which systems might be being deployed without a full understanding of how they’re going to affect the people that are involved on the ground.

Helen Toner (22:21):

And there’s obviously various the justice system, but there’s plenty of other areas where that’s also true. If we’re talking within the national security establishment, I do think that such a big underlying concern there is kind of great power competition, which is I guess a whole other topic on its own. But essentially the national security establishment in the US over the past four or five years, not just during Donald Trump’s presidency, but also before that has been moving away from a shift on mostly counter-terrorism and moving towards more of a focus on great power competition. And so that sort of the underlying sort of the ocean in which all of this is happening. And so AI fits in there and it’s very natural to then be looking at countries like China and thinking about their AI capabilities, not because not as a sort of isolated topic on its own, but more as part of this larger question of what’s sort of on people’s minds all the time.

Jeremie Harris (23:14):

And the sort of a race to the bottom game theory we’ve been talking about. Does that tend to factor into that great powers rivalry thinking?

Helen Toner (23:22):

I think it’s all related. I think that the sort of concern the underlying attention to and concern about great power competition and concern about sort of the role of the United States in the world and role of China in the world and other countries is part of the reason why, when we’re thinking about AI development and thinking about what kinds of AI systems might be used by what kinds of actors? That the idea that China might be developing something that the US decides not to develop, because it is not sure that it’s safe, that’s sort of the assumption that’s underlying that concern, if that makes sense.

Jeremie Harris (23:53):

Yeah, absolutely. Actually, in speaking of China, that’s an area you’ve done a lot of thinking and work in. I think I read a piece of yours might’ve been on Twitter, but you were talking about some of the common misconceptions that people have with respect to China’s AI capabilities and their AI advantage. Would you mind kind of unpacking that a little bit? What are some of the things that people might erroneously believe about China’s capabilities at this stage?

Helen Toner (24:16):

Yeah, so I can mostly speak to the kinds of misconceptions that I think I see sort of around me in my day-to-day work. I’m sure there’s different perceptions everywhere. I think one big one that comes to mind immediately is this idea that data is sort of the ‘new oil’ thinking of data as this really important strategic resource. And also thinking of China as being a country that sort of has ‘more data’ and the reasoning usually given for that is that the Chinese government has the ability to surveil its citizens and collect it’s app usage data or something like that. Or sometimes just people talk about the fact that Chinese internet users are often using this one app called WeChat for a huge amount of their online activity.

Helen Toner (24:59):

And so the owners of WeChat have access to a richer data set on Chinese consumers and so on. And I think that this argument that first, that China has more data and second, that that is an inherent advantage, misses, a few different things. One thing it misses is the multinational character of American technology firms. Oh, sorry. I forgot a key piece. One reason that you might think that Chinese internet companies have a ton of data is because China has more people. It has 1.4 billion people. And so one area where I think that that sort of gets it a little bit wrong is in terms of how multinational companies like Facebook and Google and Microsoft are that they are really serving the majority of the world and not just the United States. And so their user base is in fact really large. That’s sort of a simple one.

Helen Toner (25:45):

Maybe more important one is this perception that and I’m sure any of your listeners who are machine learning researchers or engineers themselves are way ahead of me on this. But this perception that one type of data is going to help you in sort of AI development in general, I think is just pretty clearly mistaken on the whole, obviously this isn’t you have approaches like transfer learning or ways to use data in creative ways, but on the whole one type of data is going to help you solve problems that are related to that type of data. And then another misconception is just also, or another way that this data point is a misconception is also about how useful more data is in a given situation.

Helen Toner (26:23):

So my understanding, again, I’m not a machine learning engineer myself, but my impression from my friends who are, is that certainly if you can give them 10 times or 100 times more data, maybe that makes a difference on their problem, but if you give them sort of three times more data or five times more data, that’s really not sort of not going to be the decisive factor. But this is often sort of this, this concept that data is an enormous strategic resource for AI, has really taken hold and has really sort of become an underlying assumption that people will just reference offhand. Data is really important and China has more data than us and therefore, and then they go onto their sort of next point. And that’s one that I sort of tried to challenge when I get the opportunity, for sure.

Helen Toner (27:02):

I guess another misconception that occurs to me about China is the Chinese government is doing some pretty horrific things with surveillance technology and Xinjiang imprison a million or maybe more of their Muslim Weger population. But I think the misconception is that the reason they can do that is because they have this super advanced AI technology that’s simply not how they’re doing it. They’re doing it with some technology, the surveillance cameras in some cases with facial recognition being used. But a lot of it is checkpoints on the street with security forces who take your phone and look at your phone or with much more traditional methods that are really not premise, even if they are using technology, they’re really not premised on sort of using the most high-tech technology. So I think thinking of China as a country that is using all the tools available to it for some pretty concerning ends is accurate, but thinking of China as some super sophisticated techno-surveillance state is inaccurate.

Jeremie Harris (28:04):

And I guess the interesting I guess another ingredient too that comes to mind is also talent or another resource that plays into this to the extent that you have a sense of the talent distribution, the talent pool in China versus the US. Can you sort of compare those two and strategically how that might sort of factor into the geopolitical competition?

Helen Toner (28:21):

For sure. Yeah. I think the main way that this is sort of something like a misconception in Washington at least, is that I think people don’t think about it enough. So talent, I think historically through at least for the last 50 to 80 years for the United States has been an enormous source of strength and strategic advantage. And by talent, I especially mean international flows of talent to the US because the US has been such a hub for the best and brightest in all kinds of disciplines from all around the world. And so when it comes to AI, that is actually a really pretty important factor that people want to come to the US to work in US universities or to study in US universities, to work in US companies. Both because of the strength of the kind of AI ecosystem here and also for other reasons the US is just a nice place to live in a lot of ways most of the time.

Helen Toner (29:13):

And so that’s really interesting from the perspective of sort of overall talent flows, because although China is obviously if we’re talking sort of raw population significantly larger than the United States. If you’re thinking about sort of an elite research population, or even a less elite but still very highly educated population of engineers and others who are working on AI. You might initially think that the larger country is better off, but if you have this selective movement of the best talent to the United States, that provides a huge offset.

Helen Toner (29:43):

And so I think the kind of status quo and certainly for the past decades the situation has been that the US is in a really excellent situation when it comes to talent. Unfortunately, over the past few years, the federal government has been doing a range of different things that have been different levels of insane to make the immigration system here more difficult to navigate or impossible to navigate for some people. So I think this is definitely a big historical advantage for the United States that it’s already risking damaging and will continue to potentially damage going forward if it isn’t able to reverse some of those.

Jeremie Harris (30:21):

Yeah. And I guess related to that too, is going to be COVID and the accompanying breakdown of Sino-American relations, but also just the reduction in mobility. I guess those are two different issues, but just speaking of the first one, how has COVID affected this? Have you discerned a difference between the policy landscape or let’s say what policies seem feasible today versus the policies that seem feasible back in 2019?

Helen Toner (30:46):

I’m not sure I’ve noticed a huge difference when it comes to… I’m not sure that COVID has been sort of the biggest thing that has made a difference in recent years when it comes to sort of the US China relationship on technology. I think certainly the ways that the US China relationship has been affected by COVID have not been in friendly directions. And so that has made the underlying or the background of the relationship worse, but I think more specifically various sort of particular actions by the US government and also by the Chinese government over the past couple of years have been sort of much more consequential. So I guess I see this as more, a part of the sort of the, I guess I called it the ocean before of rising US China tensions rising concern in the US about China’s role in the world and so on.

Jeremie Harris (31:37):

Interesting. Yeah, I guess as well from an international cooperation standpoint, one of the key things I imagine it’s going to be China’s own awareness of the robustness interpretability problem itself. I feel like I have an intuition for the level of sensibility the average machine learning engineer in the US has for these issues. It seems baseline pretty high and plausible that it might keep going up as technology improves. But I guess I have that sort of same question over in China where I just have no idea how people are approaching this problem or thinking about it. Do you have a better sense maybe of how Chinese thinking evolves on that, or?

Helen Toner (32:13):

Yeah, I unfortunately don’t have an amazing sense. I think I know that there are some work being done in China on sort of related issues and not necessarily always coming at it from this angle. I think an important thing here that I’m really uncertain about in China is also going to be the kind of environment that these technologies are being deployed in. So what are the incentives that are being set? For example, if you’re an engineer in some company, are you being incentivized to just deploy anything and to make sure that you have something done, or do you realize that something really bad is going to happen if your product breaks or if your autonomous vehicle crashes or whatever it might be? I would guess that those incentives are on the whole going to be relatively similar in the United States and China. But you could certainly imagine there being settings where maybe there’s more pressure placed on some particular program or something and that, that could lead to worse outcomes. But yeah, unfortunately I don’t have a great sort of first-person sense.

Jeremie Harris (33:07):

Yeah. I guess that in itself is fascinating just how much uncertainty there is across the board, not just in terms of time, but also in space. We just don’t know what the mentality is in different jurisdictions. It’s such an interesting problem.

Helen Toner (33:18):

Yeah. And I think it relates as well to this question of sort of talent mobility. And I guess this is also a potentially a result of COVID if there is going to be much less movement of people between different countries. Definitely an additional reason above the sort of value to the United States of having smart people working here and living here. I also do think that it’s really valuable for people to be moving across borders and working with people from other countries and thinking of themselves as part of more of a global scientific community that is all working on similar problems rather than feeling like they’re just working with people of their own nationality and having that more sort of narrow view.

Helen Toner (33:59):

So I think, yeah, you’re absolutely right that the potential differences in perspectives are one really interesting point. And then another really interesting point is how much stronger relationships internationally say, someone moves to United States to study and then goes back to their home country to work or something like that. Do you have those relationships persisting? And does that affect how the different countries are thinking about the possibilities of collaboration or cooperation or even just how that affects the extent to which we feel like we understand what is going on in a foreign place versus it feeling much more like a sort of North Korean black box, for example.

Helen Toner (34:34):

It’s an example of where we obviously have very little sense because there’s so little mobility back and forth.

Jeremie Harris (34:38):

Yeah. I guess we can only hope that things are stable there at least on the AI side. And so actually one last area I did want to really make sure I got your thoughts on, I’m thinking lately about personal identity a fair bit. I know you’ve written about this, especially with respect to this idea of privacy protections or protecting individual identity. There’s I guess another side to the coin as well. And I’m not sure how intelligent these concerns of mine are, but as I project out into the future, I imagine technology AI technology in particular, improving to the point where the typical markers of identity that we’ve relied on until this point, seeing somebody’s face, hearing their voice are no longer going to be as reliable. We’ll have deep fakes, of course, your faces, voice emphasis, and speech synthesis for speech patterns and the sound of voice. Are these issues that you’re looking at right now, do you see these being issues that we should be concerned about in the future?

Helen Toner (35:38):

It’s not something that I spend a ton of time working on, so I’m not sure that I’ll have real kind of expertise to share here. I do think it’s interesting to think about what it might mean as you say, if for example, it’s no longer possible to verify something via telephone or via video. One thing that I think about here is I think we’ve actually already started to see this as well, is the potential for a targeted attacks to make use of this. So as we were discussing earlier, right now this kind of synthesis is expensive and difficult and takes a fair bit of effort, but it’s potentially could be worth it if you are working on some particular operation to deceive some particular people for some strong reason. So in cybersecurity, there’s this distinction, for example, between phishing attacks and spear phishing attacks.

Helen Toner (36:27):

And so a phishing attack is typically going to be low sophistication and it’s going to be sent out to a wide range of people. So that might be even something as simple as like as a spam email that you receive. Whereas a spear phishing attack, is where you’re sort of really targeting one individual or a small group of individuals. You’re going to put much more work into that. And I believe that we’ve already seen basically the equivalent of that with fake voices where you have people who have been, I think I remember hearing about a case where someone at a company was sent an email that they should transfer this money to a certain account. And it seemed to come from their boss.

Helen Toner (37:02):

And then they called up the number in the email. Maybe it wasn’t their boss because they would have their bosses number, but they called up the number in the email and heard what they thought was a familiar voice and then went ahead and made the transaction. I could be just confusing that with a hypothetical that someone was raising, I feel like I’ve really heard a true example of that.

Jeremie Harris (37:18):

It sounds plausible based on what are the technologies that, I mean, eventually you’d expect something like that to be feasible.

Helen Toner (37:23):

I mean, I kind of tend to be skeptical of the prospect of things like this being really radically transformational for society in part, because we have already seen things like Photoshop just become completely ingrained and we’ve just been able to adapt to it pretty naturally. I mean, Photoshop certainly deceive some people some part of the time. And I’m sure that it’s I guess maybe you could make a case that has been really impactful via deceiving our uncles on Facebook or whatever it might be. But again, if I were to devil’s advocate that I would say, but it can be just as impactful to take a real photo and put a fake caption underneath it. And similarly here, I think if it did become the case that you could no longer trust that someone on video was there themselves or that their voice was them, I think we would just find other ways to get around that.

Helen Toner (38:12):

I could imagine that in 10 or 20 years, we have a different way of relating to that. Relating to the likelihood that video or audio is authentic, but I have a hard time imagining it really sort of uprooting some of our fundamental assumptions. I’m curious what you think of what seems like maybe you have a different view.

Jeremie Harris (38:26):

I guess my view on this is probably biased from my world, which is startups. I just remember being at there’s a Starbucks alternative program where we were in called Y Combinator. And every, every year, I think, yeah, every year they do a thing called camp YC where a whole bunch of startups come in and they have the founders of, otherwise the companies big ones, like one was Twitch in particular. And the founder of Twitch was talking about the future of identity in tech. And he was basically just soliciting opinions from the room, from all these scrappy startup founders saying, “Hey, we’re really concerned about the future of identity, the reliability of identity when it comes to just even maintaining a reputation on a social media platform like Twitch.” Just because of this kind of thing, they’re starting to worry a little bit about essentially the anonymization of everything as everybody is able to have an anonymous crypto account, an anonymous Twitch profile and manufacture people out of full cloth.

Jeremie Harris (39:23):

But as you say, I mean, imagine there’s a world where things just kind of evolve smoothly and just like our uncles got used to it and our aunts got used to Photoshop, we’ll get used to deep fakes and voice synthesis and all that. So one final question here, just the big picture, because you’re working on so many different problems. I think this will be a tough question to answer, but hopefully it’ll lead to some interesting back and forth. If you had to pick one area where you’re most concerned and the one area where you’re most optimistic, it’s like a current problem that people are concerned about that you think is going to get solved. I’m curious about which of those might be.

Helen Toner (40:02):

Yeah. I mean, I think one problem that I don’t work a ton with day to day, but that I think is just really going to be huge for the future of how humans work together with computer systems is sometimes it gets called the alignment problem. Is essentially this question of if we’re going to be outsourcing more and more about thinking more and more about our activity to computers, how do we convey what is truly important to us and what is kind of deeply important to us in a really high bandwidth, reliable way to those systems? I think there are a lot of different ways of thinking about this, some sort of focus on if we build an AGI, will it have the same values as us? But I think there are also ways of even thinking about the Facebook newsfeed, for example, as in some sense Facebook is trying to do something, it’s trying to build a product that people want to use.

Helen Toner (40:55):

You could think of that as a nice goal, but the way that they’ve operationalized that specifically is in terms of this simple metric of engagement, essentially. And you see these really undesirable side effects or unintended effects of using a metric like engagement where really what you want or really what we would want in an ideal case is something like happiness or flourishing or whatever, what have you. I guess it’s not Facebook’s mission to produce flourishing, but I hope you see what I mean, the simple, the example of a newsfeed trying to optimize engagement and find things that you ‘want to see’ in terms of what you actually look at in practice. And that it turns out that there’s actually a difference between what we really deeply want to see and how we really deeply want to spend our time and what that metric optimizes for.

Helen Toner (41:40):

I mean, I think it’s really fascinating and a big problem for Facebook as a company is going to be a whole other discussion. People think of scrolling through Facebook this term doom scrolling has become embedded. I think it’s pretty fascinating that people have this relationship to this company that they’re using the app and they don’t like that they use the app and they wish that they use the app less. But I think this is a relatively low stakes present day example of the difficulty of embedding what we care about into non-human systems into silicon, into lines of code. And that it’s a really big challenge to, as we again, as more and more economic activity is happening by our machines as more and more of our things that we used to do ourselves is being handed over to machines.

Helen Toner (42:26):

I think the question of how able are we to continue to steer the future in a direction that is actually what humans would really want and is good for society as a whole, is going to be a really big challenge. And so I think this idea of the alignment problem is one kind of somewhat technical way of framing that question. And I guess there’s this whole other more social political angle to that question, which I also think is really important and fascinating and very much unsolved.

Jeremie Harris (42:53):

Well, I look forward to finding out what the solution to the alignment problem is too. Yeah. A bunch of problems to tackle in parallel, but it’s great that people like you are working on the policy side and thanks so much for sharing your insights and your thoughts on that whole space here.

Helen Toner (43:07):

Yeah. Thanks so much for having me. This was fun.