-: Hi, I'm Joe Ito, director of the media lab,
and I'm really happy to have my favorite
science fiction writer, Daniel Suarez,
joining me today.
Today's gonna be a little bit shorter than usual.
It's one hour, so we're gonna talk
for about half an hour and then
try to involve you guys if you have questions.
We're streaming this on the web, and
if you're watching this, you probably already know
where it is, I'm not gonna tell you where it is,
but the hashtag is
#MLTalks and will be hopefully showing Twitter
up on the screen,
but we'll be taking questions
if anybody has any questions or comments.
Just tweet us at that hashtag.
So, Daniel, wanna come up?
(audience applause)
-: Thank you.
-: So, thanks for joining us.
-: Oh, my pleasure.
-: And sorry we had to book this in the morning.
-: It's all right.
-: But, without spoiling the two books,
I mean this might to,
the two books that I remember sort of most vividly
are Daemon and your most recent, Kill Decision,
and, well, maybe, first off, talk a little bit about you,
and then we can talk about the books.
I think Daniel's, and you can correct me
if any of this is wrong, but Daniel's a peculiar
and special kind of science fiction writer.
Peculiar in a good way. -: Very peculiar, yeah.
-: Because when you read the books,
they're extremely believable.
And especially for sort of nerdy people,
it's important because there's a kind of
fundamental reality behind all of the technology
that comes out, and you sort of learn later
that you actually were a software,
enterprise software developer?
-: Systems analyst. -: Systems analyst,
and then a lot of the work, if you go to your website,
you'll find links to the papers and the research,
and so a lot of the stuff,
you do a lot of real tech research
to make the stories very plausible,
and I think that the impact of your books
is that you're addressing real-world problems
that have very contemporary technical possibilities.
Is that accurate?
-: Yeah, I would say that it's fair.
What I try to do is take extent technologies,
technologies that are either just now happening,
and combine them in new ways
with the idea of creating surprising results.
I like to think of it as looking just over
the horizon in a way.
And by the way, just as a bit of trivia,
MIT bookstore was the first bookstore in the world
to carry my books.
-: Really?
-: This is, I'm coming full circle here.
So that was even when it was a self-published book
written by some weirdo named Leinad Zeraus,
'cause that was my pen name at the time.
And somebody there read it, and we were trying to get it
in all these bookstores, and that was pretty much
the only bookstore that said,
"Hey, this is interesting."
So thank you, whoever did that.
-: That's interesting, and I think Wired
did an article about how
virality was a key element in getting
your fist self-published book out, right?
-: Well, it's funny because it isn't just a matter
of the stories, which it is,
what I think gives some relevance to them.
It also actually helped to propel the stories
into the marketplace to get a film and book deal
and stuff.
We connected with technology people in Silicon Valley
and in LA.
People who make make video games,
people who make operating systems.
And the technical accuracy in the books really help that.
-: Yeah.
-: So that gave me a good foundation of fans
who thought the book had relevance
pretty much to the larger picture,
which is what I was going for.
-: And I think that's also the key because you bring
the technical accuracy, but you always
propose an idea that most people haven't thought of.
I don't know if you wanna talk a little
bit about Daemons. -: Yeah, let's do it.
-: Let's talk about Daemon and about Kill Decision then.
-: Okay.
-: I'll let you talk about them so I don't
get blamed for being a spoiler.
-: Right, so if you haven't read them,
stop listening for a few minutes.
No, basically, the story of Daemon is
that there is a brilliant game designer.
And again, I wrote this book in 2004.
So I self-published it then so it's a ways back.
But it's a brilliant game designer who
has brain cancer, and he leaves behind a program
that monitors the web for the appearance
of his own obituary.
And when this Daemon sees that,
it triggers a series of other programs.
And I think the different thing about it was
it wasn't just another story where
it's a computer out of control.
It's instead a series of systems
that persuade people to do things,
which I felt was interesting because again,
we have a very unified world.
And I've said it a couple places,
but I don't say this much, but
really that story, like that epiphany that I had
for that story, came out of some
role-playing software that I wrote,
and it was weather creation software.
It was a program called Weather Master.
And I did this back in 1999, maybe 1998.
So this was a software that if you played
role-playing games, you could use it to create
fictional weather for fictional worlds.
And it sounds unrelated,
but really, I distributed it over the web,
you could try it for free for 30 days and then buy it,
and it was all completely automated.
And what happened was, I got very busy
on projects for a number of months
and came back to look at the accounts
and there was all this money in there,
and it was being used to pay for the website
and the advertising, and I realized,
if I got hit by a bus or something,
this thing would just keep going.
And I started thinking, you know, you can do a lot
if you're dead in modern world.
And I started thinking just how much you could do,
and putting those pieces together,
and that's really how the book came about.
-: Oh, that's really interesting.
And again, without spoiling the book too much,
I think what was also interesting was,
and this is probably your enterprise stuff
where the system starts to do job requests,
and it actually, it's amazing how much computers
can actually build things
and cause things to be made and that if,
and I think the other thing about it
that I really like was that it wasn't an AI.
-: Yeah, exactly.
-: It wasn't this fancy AI, it was just a whole bunch of,
a list of processes and algorithms.
-: A task.
-: Yeah, and I think that's also an interesting thing
'cause a lot of science fiction writers
like to make this kind of black box AI,
but yours, you could imagine the code that was written
to get this to happen, to get this person to
be killed or whatever.
-: Well, that's the thing.
Very often, you're right, it's a general AI
in books like this.
And one of the points that I was making is that
we could very easily create a monstrous system
with a narrow AI, a macro on steroids.
And what I thought was interesting,
especially at the time, and it's a little more common now,
was the idea that this Daemon doesn't know anything.
It just knows based on the traffic
and what people are looking at and talking about
what people are interested in.
So that it knows if tons of people
are talking about something,
that that must be an important topic.
And as far as tasks that are prepared in advance,
it would simply monitor headlines in the news,
looking to see that certain tasks were achieved.
And until they appear in the headlines,
it knows that the task is still open.
So it doesn't understand that there's a task,
it's just cause and effect.
It's amazing how much you can do with just that.
-: I actually had written, I'm trying to remember
when it was, but it was kind of in the 90s, I wrote
an essay about multi-user dragons, MUDs.
And so what I found interesting about your book
is 'cause it reminded me about this.
My sister had been an anthropologist studying MUDs,
and people had missed their weddings,
people had gotten divorced,
people changed their lives
because of their addiction to MUDs.
But what's interesting about MUDs
and sort of different from World of Warcraft
is when I was playing, I could actually make my own world.
So I programmed characters to tell my jokes,
I made a world that looked like my house,
and we all were working on this world,
and I thought about the MUD as like a brain
and where are its sensors and we're teaching the brain
all these things, and then the code of the MUD,
what would happen is we had the original MUD
at Essex University, but then somebody said,
"Well, I wanna make one about Fraggle Rock."
So then the Fraggle Rock.
And so what happens is that the code
would spawn instances, and these instances
were kind of like different entities of MUDs.
And then somebody made a MUSH,
and somebody made a MOO, so that's kind of a mutation.
So you have different species of MUDs and you have
instances of the same species,
and to me, the people were actually
the MUD sensors into the real world.
Because what's happening is that you invest
10,000 hours into a MUD character,
you want to now protect that MUD from dying.
So if the server shuts down, you take it over.
And then what I thought was the MUD
was like sensing things in the real world,
but the real world people's time that's invested
in the MUD was its collateral of existence, right?
And to me, I always thought that maybe a MUD is kind of like
a consciousness, because it learns from through
all these people, and so the way
that you have this game manipulating the people
in the real world felt a lot like
that architecture.
-: That's the key, isn't it?
It's the idea that people's behavior
is altered in their pursuit to serve this system
in which they have an interest.
So in this case, the case you just described,
the people are the evolutionary agents for that MUD.
-: Right, right.
-: And they're gonna adapt it to whatever
real-life circumstances come about
to make sure that it persists.
So yes.
-: Yeah, so it is, I love that idea
of kind of like a,
and then in your next book,
you talk more about it, but it's kind of like,
we think of cyberspace as where our money is,
but from cyberspace, you could think that
we are where its assets are.
Do you know what I mean?
-: Yeah, exactly.
We're its bank account of human activity,
and in this case, in the case of the book Daemon,
to defend it against the people who might tear apart
the system to help spread its goals.
And again, you look at natural systems
that urge to replicate genetics or source code
pretty much everywhere you can.
That's essentially what it does,
so natural system.
-: And I think there's certain,
I don't remember exactly what the definition is,
but there's a definition of life, right?
There's like reproduction and certain things.
Would you consider that, 'cause it's not an AI,
but you don't have to have intelligence to be alive.
-: Exactly. -: Would you call--
-: Or even a nervous system, yeah.
-: Yeah, would you call that system that you design
in Daemon as alive?
-: To me, I would, I would.
I'm sure some people would quibble with that definition,
but I would.
Not self-aware, obviously, but definitely alive
in the sense that it seeks to replicate itself,
spread to affect its environment
to better suit its continued existence.
There's a lot of ways that I think it would
meet the definition of life.
-: And I think that book will become increasingly plausible.
I mean, in your book, you have the self-driving cars
and all these other things, but the more stuff
that gets controlled by the network,
the more that's likely to happen, right?
-: I remember how funny it was,
'cause again, this is going to back to 2004,
and when the book first came out,
and this is Daemon still,
I got emails from people saying,
"Hey, really loved the book, but, you know,
"that's gonna be decades until there's automated cars."
And then at some point, Google invited
my wife and I up to sit in one of those automated cars
in the highway, and I thought, all right.
-: You were right.
-: (laughs) It's like, I felt like emailing people back
and taking a selfie.
(Joe laughs) Yeah?
I don't think so, man.
-: But do you worry about some copycat
actually trying to do what they did in the book? (laughs)
-: You know, it's funny.
Again, how to phrase this?
A person in the capital at some point told me
that they really loved the book
because it helped funding.
It scared people, and you know.
It's like, oh, that's great.
I intend to think that we shouldn't be afraid of ideas.
I mean, I know exactly what you're saying,
but I think in the main,
that by freely exchanging ideas,
that's how we adapt and react to these
types of circumstances.
Whether it's a threat, whether it's an opportunity.
'Cause I thought about that.
I mean, there's a very big difference between,
look, here's the laundry list,
here's how you assemble the wires.
You know what I mean?
As opposed to thinking about the possibilities of things.
Because I have a few hacks in there that are accurate.
They're a little older, so you're not going to be
running into too many workable net bios attack.
That's just gonna fall apart.
-: And in a way, most of the people
who would be smart enough to do that
probably already have that in mind, you know? (laughs)
-: See, that's it, that's it.
Really, my thinking about it is that
people who wanna do bad things already know about it.
And it's kinda important that everyone know about it.
Because we wanna have a society that freely exchanges
information, we are connecting our lives to these machines
for ill or good.
So it's best that we think through all the scenarios
before it's a problem.
And plus, I wanted to make it entertaining.
-: Yup, it is.
So, in a sense, we talked about the technology.
The story is awesome too,
so if you haven't read Daemon, you should.
And maybe do you wanna talk about Kill Decision
a little bit? -: Yeah, Kill Decision.
-: And you did a TED Talk about it.
-: Yeah, I did do a TED Talk on the subject
of autonomous weapons systems.
Hopefully, I'm not too loud.
Am I echoing really loud?
-: Yeah, there's a little bit of an echo
going, but okay. -: Sorry, okay.
Yeah, I did do a TED talk on autonomous weapons systems,
and again, I did the first two books,
they were one story.
Daemon and Freedom TM.
And then Kill Decision was a completely standalone book.
And this really came about because of the rapid way
drones are starting to spread.
And of course, they're being armed.
And I saw this watershed moment coming very soon.
And again, this goes into my brand
being sort of looking just over the horizon.
It is absolutely technologically possible,
as you know, to create weapons that can kill people.
-: Mm-hmm.
-: I mean, there are automatic sniper stations
that are capable of doing it now.
And that moment is gonna be a very difficult one
to turn back from if they start to spread.
Because then it'll be a compulsion
for every nation to get them in self-defense.
-: Mm-hmm.
-: And so, again, one of the things
that I try to do with my fiction
is create a very entertaining, thrilling story.
But again, address a very real,
I don't know if I would say issue.
Development, trend.
And in this case, Kill Decision is about
drones attacking the United States.
-: Mm-hmm.
-: But not just randomly blowing things up,
but doing pinpoint-precision targeted assassinations.
-: Mm-hmm.
-: And so that's why I unified social media
and data mining and all of those things,
really to kind of show how that ecosystem might work,
how to create a system which would detect
and eliminate political opposition,
commercial opposition, whatever opposition.
-: Mm-hmm.
-: It's sort of a semi-automated war machine, you know.
And how would you build that?
And what shocked me as I started doing
the research for that was,
I'm not running into anything here that's impossible.
-: Yup, yup.
-: You know, it's like, are we there already,
that you could technically do that?
And so the major impediment is the inethcial one.
And so of course, I wanna bring that out
into the light and say, let's explore what that is.
So that if there is an incident,
we don't rush to go build this thing
in a big hurry without thinking through
the consequences, so.
-: Yeah, and I love the role that students and researchers
play in the story too because, you know,
they're doing these algorithms that they don't think
are related to assassination attempt.
Turn out to be.
So, J.J. Abrams is one of our friends
and he's a fellow here.
The other day he called and said,
"Do you develop weapons systems?"
And I said, no.
He said, "Could things at the media lab be weaponized?
"'Cause I'm trying to think of a very evil world
"for a movie that we're working on."
And I sat around and I thought about it,
I said, yeah, a lot of the stuff here
could do bad things.
-: You could duct tape a rifle to the side of this robot.
-: And I think that's, I mean, we like to talk about,
you know, think about everything at the lab
and how can it help health.
Or how can it help.
But for science fiction, for a moment,
it's actually fun to think about
what if everything in the media lab were evil?
(laughs) -: Yeah.
Well, think about it though,
it's like I write thrillers.
And I've often thought about this.
If I read a thriller that starts out
in a very happy place and goes to a happier place,
that's a really crappy thriller.
-: Mm-hmm.
-: So it's like you create conflict.
You've gotta have this great new thing,
but then it's going in this direction, so, you know.
It's kind of a requirement of the form.
-: Yeah.
-: But at the same time, it's that conflict
that I think pulls people into the story
and makes it very pressing and urgent
and all that stuff.
The stakes are raised and whatnot, right?
-: Well, and I think it's also
to your point of like getting people to think.
I think it's, I mean,
'cause I think, I don't know if you triggered it
or you were just perfect timing,
but the whole idea of drone,
the conference, I think they're doing at NYU, I think,
about drones, I think it's a--
-: BRC?
-: Yeah, it's a, and I was sitting next to a
very important military person the other day
and I said, well, have you thought about autonomous drones
and Kill Decision?
He said, "You mean the drone issue that we have?"
No, no, no, not human-controlled drones,
autonomous drones.
And he said, "No," and I said, I'll send you a book.
(laughing)
-: Well, thank you for that.
But that is exactly it.
I think people just assume that this is
much further out than it is,
but really, all of the decisions that are gonna
make that happen are being made right now.
-: Yeah.
-: So again, drag it out into the light.
Let's all explore it, let's talk about it.
And again, I think drones are really cool.
-: Yeah, so can you describe, 'cause there's a,
I love the thread in the book.
And I don't know if you got it from somewhere else
but the relationship between
weapons and the system of government.
-: Yeah, and I talked about this in my TED Talk too.
-: Yeah.
-: And again, this comes from reading a lot.
When I start to do a thriller,
I'll typically start with a kernel,
a germ of an idea, and then just start to do
a ton of research, and so I have a reading list
in the backs of my books.
And in this case, something intrigued me.
And it has a bit to do with the fact
that I had a minor in Medieval History in school.
I love history.
You know, that's a great story as well.
And I started thinking about how
technology affects, well, in particular,
how we resolve conflict in society
is what gives shape to the social landscape.
So if you look back at the Middle Ages,
I'll describe it this way.
The state of the art weapons system at that time, 1350 AD
was a mounted armored knife and a castle.
And those two weapons systems are
incredibly expensive.
I mean, a suit of armor cost about as much
as a house in equivalence.
So you've gotta be trained for years
to learn how to use it, the horse has to trained
and maintained in a very different way.
All of it's very expensive.
But with that weapons system, you can project power
all around you, and you can pretty much dominate
any number of average peasants.
And the political system reflected that.
Power was concentrated at the absolute top
and it wasn't shared.
And then exactly with the advent of gunpowder,
suddenly, you have a weapons system where you don't need
any training, it's dirt cheap,
and if you pointed at somebody, you're gonna kill them
or hurt them, whether they're wearing armor or not.
And suddenly, you start to see the beginning
of the Renaissance and power's starting to be shared,
and it becomes more a factor of how many people
you bring on the battlefield as opposed to
what type of people, whether they be lords or people on,
that doesn't matter.
It's sheer numbers.
And again, this is the part that didn't
make it into my TED Talk,
which is a graph that showed the sides of battles
going decade after decade after decade,
and it's this relentless march in terms of the size
of the armies on either side.
And what that creates is, I'm a logistics guy,
is a logistical requirement.
Because you've gotta feed these people,
move these people, and that starts to give
rise to administrative needs in the nation state
as a organizing principle to get an army like that
on the field.
And then you start to have to share power
because people aren't gonna do this,
especially when they can kill you
by pointing a tube at you.
You have to start sharing power.
Parliaments start rising.
And so pretty much you see
that as the means for resolving conflict changed,
so did society itself.
And how this related to Kill Decision
and automation is, you can see a trend
going back the other way.
If I start to believe that postulate that
the way we resolve conflict is
how we shape our society.
Democracy would look less likely
if you can have a bunch of robots enacting your will.
If I can create an effective weapons system
that can project my will
a million times and I'm just one person
and I own those, and they will obey
without question, you can see how that might
tilt the balance and we could go the other way.
And that's really what I wanted to point out.
It's not just a matter of,
oh, let's not have machines 'cause that's evil.
There's another factor there,
and that is it could radically change society
whether or not they're used.
Just that idea that chill, it could put on free speech,
civil society, all of those things.
Especially combined with the rampant data mining
that's going on now.
So you have a combination of knowing
exactly what everyone's talking about,
being able to detect opposition,
and then you have an anonymous tool
that can be used to get rid of people.
Seems like a bad combination to me, so.
-: And I also loved in your book how
because they're swarming in autonomous,
they don't have to phone home,
so if they self-destruct, you really can't tell
where they came from. -: That's correct.
-: And there's a lot of these kind of
anonymous strikes.
-: And you're right, I should say that.
They're not remotely controlled even by a computer.
There's not like a central computer sending them.
The idea is that they are like ants.
Like swarming intelligence systems.
They are individually rather dumb.
But by stigmergic propagation, they change
their environment to store information in the world
so they can operate.
-: But one thing, I think when we talked one on one,
you were worried about whether they could be regulated.
Because, with nuclear weapons, we have trouble,
but at least you can kind of control the source
of the technology 'cause it's pretty hard,
and control the source of the materials
because they're rare.
But these things, you can kind of knock out
with Radio Shack parts, right?
-: That's right, you could.
You hit a couple Radio Shacks and you've got a swarm.
-: So, how do you regulate that?
-: See, and this gets back to, I think, another theme
in Kill Decision, which is that I tend to think,
and I think the data backs me up here,
that the human race in general is a bell curve
from an ethical point of view.
Most people are right in that big center part
where they wanna raise their kids,
they wanna be part of a community,
they don't wanna be terrible.
And then on either side, you have extremes.
But I think that's our biggest defense,
embracing an open society and trusting in the fact
that most people want civilization to work.
-: Yeah.
-: And I know that sounds terrible,
but that's why you ostracize people who do it.
That's why I wanted to put a stigma on the idea
that you're gonna build automated weapons systems.
You're right, there's a very low bar to entry,
but sarin gas is easy to make as well.
It shows up occasionally, but it is remarkable
how rarely it does.
How rarely it's actually used.
If you think about it, it's just incredible.
And laser-blinding weapons, perfect example.
A laser-blinding weapon is a tremendously effective weapon,
it's just really evil, that's all.
Super easy to deploy, and yet there's a worldwide ban
on it, and we don't see it happening a lot.
And that to me is right there the greatest lesson,
that if we just think about it and think it through
and agree what we're not gonna do.
-: I think the scary part is, you can't ostracize that
which you cannot see, and I think that's the problem
with terrorism, is that if it's kind of,
if they don't lose anything, so,
and I hope I don't spoil his book,
but one of my friends, Joshua Ram
was writing a book about this,
but he was explaining, I can't remember
the name of the rule, but if you look at the weapons systems
in history, when defensive weapons become stronger
than the offensive weapons, so war dies down.
So when fortresses with walls came up,
wars, I think, lessened.
And then when you had catapults and stuff like that,
it went up.
And it's interesting how the machine gun
turned out to be a defensive weapon,
but then the tank became an offensive weapon.
And you look at history at the cost of attacking
versus the cost of defending,
and that economics and balance changes them up.
Aggression.
But these drones that you're talking about
are extremely cheap to make.
-: They are.
-: And the cost, and it's interesting, 'cause nuclear war,
it turns out, has a high cost, because very likely,
you're gonna get hit again and so the cost of--
-: That's kind of a binary.
It's like, you know, you're either having it
and it's over, or you're not having it.
-: Yeah, but these drones,
if you can launch them anonymously,
they're very cheap to make,
and of course, there is some evil to it,
but I think if you can hide,
I mean, if you're a state actor like Asad,
he does it as a cost,
but if you're not,
so, I mean, I'm not gonna try to make you commit to--
-: No, no, let's do it.
-: But it is, I mean,
I can't see a really bright,
because the parallel for this is cybersecurity right now.
So the cost of attacking somebody's computer,
whether you're a state, is so low,
and so it's kind of persistent right now
and it's gonna worse before it gets better, right?
-: It is like the Middle Ages in the sense
that it's a constant state of low-intensity warfare,
is essentially.
And of course, we'll see if that flares up
into some mega warfare.
But, so again, I think of it as an ecosystem.
What I don't see is swarms of rival drones
annihilating each other.
I tend to think, I do agree,
you can have very cheap low-cost drones
being used to do terrible things,
and that's why I think there will be a great many drones
that are just looking for other drones.
And not to fire on them.
And I talked a bit about this at the end of my TED Talk,
that idea of you create an immune system, essentially,
that helps humans see these things coming.
-: And I think that's how we're thinking about cybersecurity.
I think the future of cybersecurity
looks a lot like an immune system.
There's one guy I was talking to at Foo Camp
who was explaining that really, really good exploits
are indistinguishable from bugs now.
And so when they scan, and Markoff wrote this article
a couple years ago, but when people are trying
to own computers, when they get in,
they run Windows update
to try to eliminate all the other guys in the system.
-: Yeah, well that's the first thing
a parasite does in reality.
-: Yeah, is to protect the host, right?
-: Yeah, protect the host, get rid of other parasites
and prevent other ones from coming in.
Competitors. -: But it is scary when
the parasites are fighting each other,
more worried about each other than they are
about the host.
-: I tend to think that eventually,
we'll have buildings in cities
that sort of have that immune system
in the background, and occasionally, they'll fire up
into some very local, intense struggle,
and then it'll die down
and then you'll see there was a drone combat today,
and you know, whatever.
But it's like an immune system.
Like I said, it will handle it,
and if it ever rides us to the level
that humans really need to get involved, they will.
And how long ahead is 100 years from now?
But I think that's gonna be a constant arms race
just like it has been ever since the dawn of time.
You know, competing interest struggle,
they adapt their strategy, the other side
adapts their strategy, and it continues
and it keeps ratcheting up.
Until there's a big seat change,
and then it all starts over.
-: Yeah. -: So.
-: Interesting.
So, I guess we're about half an hour.
If people have questions, you can raise your hand,
we have mics.
And if not, I will keep talking.
Is there anything from Twitter?
Social media peeps, no?
Okay, we got over there.
Wait for a mic there.
-: Hi. -: Hey.
-: So, I was really intrigued about what you were saying,
using social media to identify the nodes
in a network that represent resistance.
In many ways, that's almost scarier
than the particular choice of physical drones
as a way of neutralizing those nodes.
-: Yeah, that's just the choice of tool you wanna use.
Change their FICO score instead if that's easier, you know.
-: Yeah, I wonder if like, people just vanish
from the network without realizing it, you know,
either through legal means or through data means,
like they're just post-op propagating to the audience
they think they're propagating to
and those kind of subtle distortions.
It almost becomes like that bug
that Joey was just describing that,
how do you know if it's an exploit or a bug in that regard?
-: Yeah, yeah.
It's funny that you say that because
pretty much very similar tools
for finding and mapping terrorist networks
are used for finding and mapping people
who use a particular product.
So advertising tools, tools for hunting terrorists,
kind of the same, you know?
Just different data going into it.
And that was a similarity that I thought was pretty amazing.
It's like, oh my God, you know?
It's almost, it's branded a little differently
and it's more expensive on the other side.
There's a more threatening EULA that you have to sign.
But I don't know, it's those types of similarities
that are just revealing to me.
How many times are things that are made
in the courts of conflict are repurposed?
I mean, it's endless, that are useful for us.
Everything from aircraft to computers,
the internet, and on and on.
So, I hope that, yeah, okay.
-: Anybody else?
-: What do you think about the personal privacy at the age
of NSA revelations and how we can empower people
to protect their privacy?
-: Yeah, okay.
The question was, what is my opinion of privacy
in the age of NSA and PRISM?
This is a really loaded issue because I,
in some ways, obviously I'm a big privacy advocate,
but at the same time, it's sort of
the horse has left the barn.
So, we all like our smartphones,
we all like to be connected.
I really think what we have to do is
start thinking of these systems as civic infrastructure,
and this is gonna sound ridiculous,
but we essentially need an internet 2.0,
some new structure that is not inherently open.
We keep the old internet,
but for our more critical interactions,
whether it be financial or whatever,
we have a system that defaults to privacy,
and I do think we need a Bill of Rights 2.0
that essentially says that you own your data
and you get to know where it goes.
Now again, people can still draw on your data,
use it and crunch it, but if your data is out there
earning money for somebody, you should
essentially get the choice about whether or not
you want to participate in that.
I basically think it needs to be rethought a bit,
because we do benefit from it.
But right now, we have absolutely no leverage in it.
We have no idea where that data goes,
we have no say in it.
Essentially, it's a big system
over which we have absolutely no rights.
And its potential political power,
as I was just talking about it,
people build automated systems.
Again, let's say it's an autocratic government
we're talking about.
And this thing on our belt,
George Orwell would just kinda be
scratching his head saying, "Let me get this straight.
"You pay every month for your tracking device.
(audience laughs)
"I didn't even have the balls to write that.
"That's just crazy, why would you do that?"
And yet, that's what we do.
Because it's a quid pro quo, we get it.
We just want, I think, a bit more control over it.
Now, I think most people would still say, "I don't care."
But at the same time, I think we have to
start thinking of it like that for this reason.
Let me see if I upset people by saying this,
but the idea that these, these commercial networks
really are rapidly becoming society, these social networks.
And right now, there's just a few companies
that really control all that.
Now, that's not the way we do our roads.
The roads are built, we make it civic infrastructure,
and then we build commerce on top of that.
And the idea being that you want to create
a level playing field.
It's never perfect.
But the idea is that you create rules of the road
that everybody has to follow.
And this was done in the opposite way
simply because it happened so quickly.
So we have some dominant players
who basically control all this data
that they're sitting on top of,
and I think we really have to puzzle through
what that means for us individually,
for our political power, both individually and as people,
as a people, a whole citizenry.
Because other than that, we're simply users.
Limited rights users in a system,
and that's definitely not the way
the founding fathers and others intended it,
so I think that privacy issue is
directly related to political power,
and I think sooner to later,
we're gonna wake up to the fact that
we need to make some decisions
about that and re-architect this
so that it suits us more fully.
Right now, I think we have limited options
in this current arrangement.
Because, I mean, let's face it, we wanna use our phones
and there are really no other good choices.
So I hope that answered your question.
-: And I think you can tie it back to your earlier point
about the immune system.
I think that privacy, security, all of this.
I think you can look at the human immune system,
or biological immune systems as a really good metaphor.
So like, for instance, if you look at,
'cause it's really a war that's going on at every level,
and if you look at how viruses attack your chromosomes
and how your genes have all this noise as basically
to try to see, and basically, if you look at
evolutionary biology,
you try to understand the algorithm of your prey,
and once you discover that, you take its free energy.
And there's this constant battle going on,
and if you look at things like smallpox viruses,
they actually attack the communication system
of the immune system.
And the immune system's smart enough
so that it's got positive feedback loops
and things like that, but when the T-cells see a pathogen,
they no longer ask.
It's really interesting, they assume
they're gonna be hacked,
so they keep growing in number
and they do not allow a stop signal
to come in from the immune system
because they know they can be hacked.
And it's this really, really sophisticated system
of deception on both sides, and it's evolved.
And then it gets stronger the more it's exposed
to different types of pathogens, 'cause it itself evolves.
And I think that computer networks,
and also when we think about privacy,
we think too much in the sense of like policy
and like rules,
but I think they need to be more autonomous,
and I think they need to be more evolutionary,
and I think that, 'cause it's sloppy, it's messy,
you're not always gonna get it,
but I think if you assume that you can control it,
then you're gonna get it wrong.
-: You'll never get it perfect.
There's no such thing as perfect security.
But constant incremental change.
I always love this.
I'm trying to remember the book that I read
that the idea that sexual reproduction
is just to address that.
That you have parasites, and once they get your number,
genetically, once they've figured out
how to get your free energy from you,
if you don't change, your whole population will crash,
because you will all be susceptible in the same way.
This is why cloning is not a good strategy,
'cause you create identical copies of an organism
and they're all susceptible in exactly the same way.
So with sexual reproduction, you mix it up,
it's constantly changing, and that way,
when a parasite comes in, it crashes a portion
of the population.
And those that don't suffer from that defect,
they fill in the space.
And it's sort of like that's all we need to do
with the civilization, essentially.
Constantly keep it changing
so that it doesn't become gamed, essentially.
Politically gamed or some other way.
And privacy would be included in that, I'd say.
-: Anybody else?
-: Don't be shy.
-: We got one there.
-: So, this is more of a comment about drones and warfare.
I think it's actually sort of throwing this back at you.
If you think about a line which is,
I guess in the left side is the most destruction possible,
and on the right side is the least amount
of destruction possible.
On the left side, if you plotted all types of
technology and warfare, far in the left,
you'd have like the Death Star.
And then like a little bit closer,
you'd have like nuclear bombs,
and then a little bit closer, you'd have chemical weapons.
And over probably on the right,
on the least destructive but still having impact
would be things like targeted killing
using drones and assassinations.
So, I guess the first comment I have is,
I think thinking about using drones
as in terms of targeted killing,
it's a way to have political impact
while not doing as much destructive damage
to the entire civilization.
So, I wouldn't necessarily say
that's just a terrible thing on its own.
-: You mean by the fact that the carnage is
reduced to its absolute minimum?
-: Yes. -: Okay.
-: But I think there's also an interesting opportunity
thinking about autonomous technology,
about maybe there's totally nonlethal ways
we can have political pressure
on a government or an organization
that we're trying to change using autonomous vehicles.
Like, you could imagine like very science fiction example,
but very practical, actually, of having drones
that are stabbing the tires of every vehicle
in an entire city.
And that would cause tremendous problems
but wouldn't actually kill anybody.
And you'd also get, you know.
So things like that, I think, are things
that are unexplored that are actually
interesting opportunities with drones,
and so this is just my counter-argument about why it's--
-: So, what you're saying is your postulate
that drones aren't necessarily this bad thing
in terms of conflict.
-: Yeah. -: Okay.
See, it might surprise you.
I agree with that to some extent in this way.
I still know that there will be conflict.
My big concern is about sort of chain of custody.
I'd say checks and balances.
So the idea of we're gonna have conflict
and we're gonna have military engagements
whether they be very surgically precise or not.
My concern about making it really easy
so that it doesn't cause a lot of damage
is then it's very tempting to constantly use it.
To your point, Joey.
If you have a nuclear bomb,
you're really not gonna use that willy-nilly.
But if you have something that nobody's
gonna be able to find out who did it,
it's gonna be completely anonymous
in addition to autonomous,
you can just make a troublesome person go wait,
you'll default to use it.
And then you start thinking on the ripple on effect to that.
Even if it's nonlethal.
And by the way, I did a lot of research
on nonlethal weapons as part of several of these books.
And it's a very interesting thing about nonlethal weapons.
It's still coercive force, and it can be used.
You think of how many people get tasered, pepper sprayed,
how disturbing that is when you see somebody
who's not really resisting or causing a problem
is being put down and pepper sprayed to stop them
from making a statement or doing something.
And really, it's starting to put a,
it can be used to put a chill on free speech
and free exchange of ideas.
So, non-lethal weapons to me aren't
necessarily completely benign.
They can be used as a tool of oppression.
At the same time, if they're used properly,
they can save lives, on and on.
So, I would say the key to that is,
if we're going to have drones,
we need to have a process where it actually is
checks and balances, just like our society's built to be,
the idea being that, if we're gonna have constant war,
very low intensity, as opposed to having a declared war
every once in a while where we set up
a special legal system where generals do whatever,
there's martial law, that's rare.
But if we're gonna have constant low-intensity war,
that's a very different animal,
and we have to make sure we don't fundamentally
change our entire society to suit that.
Because again, who is the person who gets to
decide that that one person needs to be eliminated?
What process are we gonna follow
that's in some way transparent,
or at least has checks and balances
where you have two people who might debate the merits of it?
'Cause I could thoroughly see that in rare occasions,
it could save a great many lives if one person was hit,
especially if they're carrying a bomb
or gonna release a biological agent.
But I just think that's exceedingly rare,
so let's not completely alter our society
to suit that very rare ticking time bomb scenario.
-: Yeah, making it too easy is scary.
I was just imagining it as you were saying on Facebook,
mark comment as spam.
Report user.
Hurt user.
Terminate user. (laughs)
-: That's right.
-: You have a super user account.
-: You have to read the terms of service, is all I'm saying.
(laughing)
-: Anybody else? -: Anybody else?
Don't be shy. -: Don't be shy.
So, I have a question.
So, in your book,
for some reason you stand for it now. (laughs)
-: It's nothing personal.
-: But I was curious,
the extent of which you sort of talked about research
in labs and how they were targeted
and those ideas were being used.
I mean, and I know you're not in the military complex.
-: That you know of. -: That I know of.
But, I guess, 'cause it was really interesting
how you pulled pieces,
like most of the stuff that you use,
the technology that you use
for creating these weapons
weren't being created by people who were creating weapons.
-: Correct. -: They were studying ants.
-: Correct. -: So...
-: Well, the big part of that is of course,
when you have a scientist who knows
that she's studying
ant algorithms, ACO, ant colony optimization algorithms,
this logic is a hundred million years old.
-: Yeah.
-: How could this suddenly be a danger to us?
-: Right. -: So.
-: Yeah, and I guess I'm curious, do you have a theory
or an idea about how often that happens where somebody is,
because this ties to Lessig's book Code.
He talks about how software is like law and how,
and I remember afterwards, a lot of programmers said,
"Well, I don't wanna be political."
And he said, "Tough noogies, you are."
And I think as we start to study
and think about our science, if, for instance,
somebody studying ant colonies
is actually enabling a fundamental change in warfare,
should that person be aware of that,
should that person be thinking about
the ethical consequences of it, or?
I mean, this is a kind of a,
we talk about this at the media lab a lot,
which is, should we be thinking about the systemic impact
of every scientific invention or intervention that we do?
It's hard, but it seems like that could be
a design principle, it's to think about
what the effects of this getting out are,
and I wonder if you have any--
-: It's almost part of the requirements document
for something we say, okay, what would be
the ramifications of this?
-: Yeah.
-: I see what you're saying, but at the same time,
I thought about this a lot when I was writing
about that character.
I was thinking, but what would she have done differently?
Do you not inquire about the natural world?
And the other thing is, think about how quickly
the airplane was invented after the piston engine.
I mean, what was it, like nine hours later or something?
-: Yeah.
-: In other words, it became inevitable at a certain point.
I mean, it is absolutely a fact that drones
are going to spread simply because all of the technologies
that are there make it just a, forget it,
it's a fait accompli. -: Yeah.
-: So, to have as many people thinking through it
and working on it and trying to do positive things with it,
I think is a good idea.
It should be a component to think through
what the ramifications are,
if for no other reason than to know
what to lock up in the lab,
but I would hate to think that people have to
put a chill on inquiry.
-: Yeah.
-: Because that's how we puzzle through things.
When you think about it, if people didn't use
and spread gunpowder, all sorts of mining
and other equipment, I'm sure there's
tons of positive things that have happened
as a result of that.
Well, a Nobel prize, you think about it.
He came up with dynamite, felt so bad
about it that how many people
had their research funded from that.
I wonder what the balance sheet is on that.
-: Yeah, yeah.
Yeah, and I remember I once had the opportunity
to interview Edward Teller and ask him
what he thought about inventing the H-bomb.
And he said, "Well, somebody was gonna do it
"and better us than them."
-: Exactly, exactly. -: And that was his point.
But it is interesting.
But it is also kind of, I think, interesting,
and this is why science fiction helps,
is to sort of think, does the thing I'm working on
enable something else?
And how does that play out?
Even if you don't stop the inquiry,
to think about it broadly.
-: Yeah.
And if you do it in science fiction,
you're not actually building the weapon,
so you can start to think through these scenarios.
And I think that's why I have a great many people
in the defense and in military
who find my work interesting for that reason.
It's a scenario like, I hadn't thought of that.
And then they can start to take steps,
think about what they would do if that happened.
So that we're not caught completely flat-footed.
And the other thing is, it might give rise
to a good thing as well.
A good development.
Not just defensive, but a whole new mode of inquiry.
-: Mm-hmm.
And you enjoy that too, right?
-: I gotta admit, I do.
I have a devious mind, but I don't like to
actually build bad things, and this is a good way to do it.
The other thing is, I love, actually this is true.
Very often, I like putting a character
in a super difficult situation.
I mean, really difficult situation,
and then spend a lot of time doing research
to figure out how to get them out of it
in a believable way.
I love that challenge.
And I don't know whether this is probably
a place where that happens a lot
where you set a very lofty goal
and then you try to meet it.
And you might change things later
if it turns out to be unworkable,
but I love that challenge, I really do.
Making things believable, authentic
is very important to me.
-: Yup.
I wonder if people, okay, there's...
And yeah, you'll do a tour today,
so hopefully you'll get some--
-: I'm excited.
-: So, in the book Daemon,
there's several parts where people
from different government agencies get together
and they discuss what their concerns
about how Daemon can hurt the world.
And the main concern is like it might result
in a financial catastrophe.
So do you think this is,
the result for this, is this because
our life is inevitably dependent on technology
and computers, or is this mainly because the current
economic system is shaped in a free market society?
-: So, the question being that
is the concern of the authorities in that book Daemon
primarily about the health and welfare
of people or of the systems?
-: I'm actually asking like,
what would you think if this was a concern
in the real world?
Would it be like the reason,
the main concern would derive from the fact
that our lives are mainly dependent on computers
and technology or is it,
would it be a result of the free market economy?
-: See, I--
-: That like our lives depends on basic market?
-: So are the vulnerabilities a result of the free market?
Okay, okay, I'm with you.
See, to me, even, let's say all of these computers
in the world were the result of somebody's plan
on how to design it.
I think that would be even worse than a free market.
Again, creating new generations
of technology and stitching together networks.
I mean, for God's sake, TCP/IP has been around for decades
and it's at the foundation of everything.
It's inherently insecure in many ways,
we still use it, but it still functions.
And in that story, the idea of the authorities
sitting around sort of a star chamber table saying,
"What are we going to do about this malevolent,
"narrow AI that's taking over these networks?"
I had their concern be primarily and initially
about financial concerns, simply because again,
I go back to my nearly 20 years as a logistics person.
Those are actually very valid concerns,
because for people to trust that exchanges of money
that they are not fake, that it's not fiction,
that somebody hasn't gotten in there
and completely screwed up the numbers,
that's what makes all of the systems
that feed into food, energy, everything work.
So to me, and again, it sounded like they're being cold,
but if you have a crisis,
that would be one of the first things they have to look at,
is kind of as a way of defending the networks.
Because if somebody has gotten in there,
you have to try to find out how to make sure
that all of the systems that actually interact
with the world, whether it be supplies
of money, fuel, electricity, SCADA systems,
are telling the truth.
And if they're not, then all bets are off.
Especially nowadays, because a lot of that's been automated.
I hope that answered your question.
-: Yeah. -: Did it, okay, good.
-: And also, I think in Freedom, the next book,
you talk a lot about reputation,
and I think it starts to tie to this too
because the financial markets at some level,
it's almost kind of bullshit in a way,
I mean like, banks are borrowing money
and lending money and they inherently don't have value,
but they do.
People trust banks, right?
And we have this weird trust network where governments
issue currency back by just trust.
And that creates this equity,
and you can see an attack on that,
and you talk about this distribution
of reputation and an alternative reputation network
and things like that,
and I think that that thinking about
how reputation and power are traded and,
'cause that's, in the old days, it was pretty simple.
You had a few famous people and a few big brands,
and that was, but now it's so distributed
and so complex.
-: And now you can do business personally
with somebody halfway around the world.
That wasn't terribly common back in 1910.
-: And you kind of teased out some of those ideas
in that book, but I do think that developing that
is gonna be a critical part in stabilizing this
financial system that we have
that is really a house of cards,
and I think at risk, because I think as we,
like NSA and other things, those are like
brands that we know, like the US intelligence system
gets a black eye, a lot of things change.
-: Yeah, the whole system gets a black eye
because trust in it is shaken.
And again, we talked about this before,
the idea of people using cloud services
and other things, now looking at them
saying, no, we have to have physical control
of these systems now because we don't trust
what backend is in them.
And that can put a chill on commercial development,
as well as exchange of financial data.
Business plans, on and on and on.
And that becomes a tax in a way, on the system.
And there was something else I was gonna say
in response to this.
I can't remember it now.
Losing it.
-: But I do think both in this conversation
and also in Kill Decision, it really is
the cost and the value changes
which affect society, I'll let you...
-: I know what it is, it finally came back to me.
In Daemon, the currency there was based on energy.
The idea of current energy.
Being that you couldn't falsify because
the energy is either there
and available to do work, so electrical energy.
And of course, people have written about this,
but I always thought that was interesting,
because, of course, many people don't really know
where money comes from now,
like how it is originated.
And it's really 85% of it is loan origination.
So money is just invented out of thin air.
It's fiduciary but not really tied to anything
of intrinsic value, which of course,
freaks people out.
And it's also ones and zeros.
99% of the money moving around is not cash.
It's ones and zeros, and of course,
if something gets into that very, very
efficient system and screws it up,
trust is eroded like that.
And very bad follow on effects,
so you're right.
House of cards, not an ideal situation.
-: No. -: So, we can do better.
-: Got one more. -: Sure.
-: Okay, can we get a mic?
Hold on. -: Okay.
-: Wait, wait.
-: I walked right into that one.
So the question was, what do I think about Bitcoin.
And I'm gonna be very careful as I say this.
I think it's a really great idea,
that idea of creating a distributed system
that doesn't have a central banker that can
make decisions about how many coins there are.
But at the same time, it's almost
the opposite of what I was just saying
where it represents intrinsic value.
What you're doing is you're burning through energy
in order to create something.
So it's basically like a pot latch
in the sense like I can burn through this much energy
processing power to create this many coins, Bitcoins.
And I think ultimately, that's where
I think the problem is.
Because you could see how that, let's say
it just kept spreading and spreading.
We have all of these data centers chewing through energy
to create these Bitcoins.
It almost is like to me, the better design is to say,
we have
some unit of value that represents
energy available for work,
not that we've burned up to make this Bitcoin.
That's really the only problem I have with it.
But it's a pretty significant one.
But it's a good first effort, you know?
-: But the amount of Bitcoins is actually,
the created ones are actually limited,
so burning more energy will not create more Bitcoins--
-: Agreed, agreed, and they unblock.
-: If like--
-: But you know what I'm saying, you are--
-: If you have a lot of computation power
and you put all of it in there
and you get finite amount of Bitcoins,
at some point, it will be
more beneficial to use your computation power
for other things in the world
so there will be a balance there--
-: Well, it's true, it would still persist.
-: Yeah, but I think what Bitcoin's trying to do
is use scarcity as the way that it retains value,
but I think personally, that a currency
that was backed with something that
actually had value, whether it's--
-: Exactly.
-: It is more likely, and then I think the other problem
with Bitcoin, I was like this when I was a kid,
I thought that avoiding taxes,
avoiding the government was something that might happen.
I no longer think that, and I think that
a really good currency is gonna be one where
you have the providence, it has a stamp,
that you pay tax on it,
and it just basically keep the information
connected to the instrument,
so that to audit somebody, 'cause audit's broken.
I'm on the audit committee, it's broken.
Because what we do is we take the numbers
and we add them up and add them up and add them up,
and at the end, you don't really know
whether this number actually is related to
what actually was a value.
-: The chain of custody.
-: But my friend who set up the accounting system
at Shinsei Bank, he had every piece of paper
that enters the bank gets scanned,
and it's always attached to a number.
So the balance sheet is actually a view
of a pile of paper.
So you can click through all the way,
and then you see a piece of paper.
So there's not a single number that is not connected
to its origin, and so if you can imagine
if you had a pile of digital cash,
you could just sort through it and see the bills
that haven't been paid tax on,
and then you can just pile them.
And so the idea is, if you could just audit
the books by looking at the digital cash
without having to trace it back, that would
fundamentally change the nature of accounting.
And I think what's really interesting is,
and I'm gonna stop, I mean, but like,
geeking out on accounting, I think is really important.
People don't like to do that,
and people who know math make fun of accounting,
but accounting's actually pretty hard,
and there hasn't been a lot of innovation
in accounting.
And so, like Deloitte's one of our members here,
and just sitting down and talking to the head of audit,
I think that's there's a huge innovation
that we can do in accounting,
and I think the problem with some of these e-cash people,
'cause I used to be in that space,
is that they don't understand how the financial systems
and accounting systems work.
They're thinking about it more kind of--
-: Mainstream ones.
-: Mainstream ones, they think--
-: Of course, you have these early adopters
for these things, and they're usually the people
who have a reason to do it.
It's like, I don't want this to be in the mainstream
for whatever reason. -: Right, right.
-: But a system like that also
would be able to support the network and systems
that it's a part of.
In other words, that tax part could be used
to help defray expenses and so forth to keep it running
and sustaining.
So yeah.
I hope that answers you, yeah.
-: You got time for one more if there is one.
Otherwise-- -: Anybody?
-: Yeah, Phillip, yeah. -: Oh.
-: What are some of the ideas that you have in the pipeline?
What are you currently writing about?
-: What a great last question.
-: So, can you repeat the question?
-: That's funny that you ask.
-: So the question, can you repeat?
-: The question was, what sort of ideas
do I have in the pipeline?
What am I working on next?
(Joe laughs)
What a great question.
I just so happen to have a book coming out in, no.
It's February, I have a book coming out called Influx,
which is a sci-fi thriller.
It's looking a little further out over the horizon,
but it's about technical domination, essentially.
One of the things that interested me recently is,
of course for many decades, we've had a gap
between the technological lives
of first- and third-world people.
But I'm starting to see a trend
where even people within our own society
are almost living at different technological levels,
different civilizations, whether that's
privilege or preference, but the idea is
you're starting to see this schism
where people either are following along
or can follow along or, no.
And then also, I think is that continues
people who have privilege will live longer lives.
They'll be stronger, they'll be smarter.
They'll be able to improve themselves.
And so I wanted to explore the idea that
if we have a society where we're all created equal,
how does it change if, after we're created,
we can improve ourselves significantly
and some people can't?
What does that do to the distribution of,
well, what does it do to society?
But that's really what Influx is about.
And, you know, of course, it's an action thriller.
I write action stories.
But that's really what inspired me to do it,
so basically, the postulate is,
what if we're really half a century more advanced
than we know in technology?
So some people know and some people don't,
and I wanted to take that essential conflict
and kind of run with it and see where it went.
So that's what I've worked on,
and that's coming out in February.
Thank you for asking, by the way.
-: Thank you, Daniel. -: Yeah, hey, this was fun.
Thanks, guys.
(audience applause) -: Okay.
-: Okay.