Transcript
00:00:00 The following is a conversation with Dimitri Dolgov, the CTO of Waymo, which
00:00:05 is an autonomous driving company that started as Google self driving car
00:00:09 project in 2009 and became Waymo in 2016.
00:00:13 Dimitri was there all along.
00:00:16 Waymo is currently leading in the fully autonomous vehicle space and that they
00:00:20 actually have an at scale deployment of publicly accessible autonomous vehicles
00:00:25 driving passengers around with no safety driver, with nobody in the driver’s seat.
00:00:32 This to me is an incredible accomplishment of engineering on one of
00:00:37 the most difficult and exciting artificial intelligence challenges of
00:00:41 the 21st century.
00:00:43 Quick mention of a sponsor, followed by some thoughts related to the episode.
00:00:47 Thank you to Triolabs, a company that helps businesses apply machine
00:00:51 learning to solve real world problems.
00:00:54 Blinkist, an app I use for reading through summaries of books, better
00:00:58 help, online therapy with a licensed professional, and Cash App, the app
00:01:02 I use to send money to friends.
00:01:04 Please check out the sponsors in the description to get a discount
00:01:08 at the support this podcast.
00:01:10 As a side note, let me say that autonomous and semi autonomous driving
00:01:14 was the focus of my work at MIT and as a problem space that I find
00:01:18 fascinating and full of open questions from both robotics and a human
00:01:23 psychology perspective.
00:01:25 There’s quite a bit that I could say here about my experiences in academia
00:01:29 on this topic that revealed to me, let’s say the less admirable size of human
00:01:35 beings, but I choose to focus on the positive, on solutions.
00:01:40 I’m brilliant engineers like Dimitri and the team at Waymo, who work
00:01:44 tirelessly to innovate and to build amazing technology that will define
00:01:48 our future.
00:01:48 Because of Dimitri and others like him, I’m excited for this future.
00:01:53 And who knows, perhaps I too will help contribute something of value to it.
00:01:59 If you enjoy this thing, subscribe on YouTube, review it with five stars
00:02:03 and up a podcast, follow on Spotify, support on Patreon, or connect with
00:02:07 me on Twitter at Lex Friedman.
00:02:10 And now here’s my conversation with Dimitri Dolgov.
00:02:14 When did you first fall in love with MIT?
00:02:17 When did you first fall in love with robotics or even computer
00:02:20 science more in general?
00:02:22 Computer science first at a fairly young age, then robotics happened much later.
00:02:28 I think my first interesting introduction to computers was in the late 80s when
00:02:39 we got our first computer, I think it was an IBM, I think IBM AT.
00:02:44 Those things that had like a turbo button in the front, the radio
00:02:48 precedent, you know, make, make the thing goes faster.
00:02:50 Did that already have floppy disks?
00:02:52 Yeah.
00:02:52 Yeah.
00:02:52 Yeah.
00:02:53 Yeah.
00:02:53 Like the, the 5.4 inch ones.
00:02:57 I think there was a bigger inch.
00:02:58 So good.
00:02:59 When something then five inches and three inches.
00:03:02 Yeah, I think that was the five.
00:03:03 I don’t, I maybe that was before that was the giant plates and it didn’t get that.
00:03:07 But it was definitely not the, not the three inch ones.
00:03:09 Anyway, so that, that, you know, we got that computer, I spent the first few
00:03:15 months just playing video games as you would expect, I got bored of that.
00:03:20 So I started messing around and trying to figure out how to, you know, make
00:03:25 the thing do other stuff, got into exploring programming and a couple of
00:03:33 years later, it got to a point where, I actually wrote a game, a lot of games
00:03:39 and a game developer, a Japanese game developer actually offered to buy it
00:03:43 for me for a few hundred bucks.
00:03:45 But you know, for, for a kid in Russia, that’s a big deal.
00:03:48 That’s a big deal.
00:03:49 Yeah.
00:03:49 I did not take the deal.
00:03:51 Wow.
00:03:51 Integrity.
00:03:52 Yeah.
00:03:53 I, I instead, yes, that was not the most acute financial move that I made in my
00:03:58 life, you know, looking back at it now, I, I instead put it, well, you know, I had
00:04:02 a reason I put it online, it was, what’d you call it back in the days?
00:04:07 It was a freeware thing, right?
00:04:08 It was not open source, but you could upload the binaries, you would put the
00:04:11 game online and the idea was that, you know, people like it and then they, you
00:04:14 know, contribute on the send you a little donations, right?
00:04:16 So I did my quick math of like, you know, of course, you know, thousands and
00:04:20 millions of people are going to play my game, send me a couple of bucks a piece,
00:04:22 you know, should definitely do that.
00:04:24 As I said, not, not the best.
00:04:26 You’re already playing with business models at that young age.
00:04:29 Remember what language it was?
00:04:30 What programming, it was a Pascal, which what Pascal, Pascal, and that
00:04:35 a graphical component, so it’s not text based.
00:04:37 Yeah.
00:04:37 Yeah.
00:04:37 It was, uh, like, uh, I think there are 300, 320 by 200, uh, whatever it was.
00:04:43 I think that kind of the earlier, that’s the resolution, right?
00:04:46 And I actually think the reason why this company wanted to buy it is not like the
00:04:49 fancy graphics or the implementation.
00:04:51 That was maybe the idea, uh, of my actual game, the idea of the game.
00:04:57 Well, one of the things I, it’s so funny.
00:04:59 I’m used to play this game called golden X and the simplicity of the graphics and
00:05:05 something about the simplicity of the music, like it’s still haunts me.
00:05:10 I don’t know if that’s a childhood thing.
00:05:12 I don’t know if that’s the same thing for call of duty these days for young kids,
00:05:15 but I still think that the simple one of the games are simple.
00:05:21 That simple purity makes for like allows your imagination to take over and
00:05:28 thereby creating a more magical experience.
00:05:30 Like now with better and better graphics, it feels like your
00:05:34 imagination doesn’t get to, uh, create worlds, which is kind of interesting.
00:05:38 Um, it could be just an old man on a porch, like way waving at kids
00:05:43 these days that have no respect.
00:05:44 But I still think that graphics almost get in the way of the experience.
00:05:49 I don’t know.
00:05:50 Flip a bird.
00:05:51 Yeah, I don’t know if the imagination is closed.
00:05:57 I don’t yet, but that that’s more about games that op like that’s more
00:06:01 like Tetris world where they optimally masterfully, like create a fun, short
00:06:09 term dopamine experience versus I’m more referring to like role playing
00:06:14 games where there’s like a story you can live in it for months or years.
00:06:18 Um, like, uh, there’s an elder scroll series, which is probably my favorite
00:06:23 set of games that was a magical experience.
00:06:26 And that the graphics are terrible.
00:06:28 The characters were all randomly generated, but they’re, I don’t know.
00:06:31 That’s it pulls you in.
00:06:33 There’s a story.
00:06:34 It’s like an interactive version of an elder scrolls Tolkien world.
00:06:40 And you get to live in it.
00:06:42 I don’t know.
00:06:43 I miss it.
00:06:44 It’s one of the things that suck about being an adult is there’s no, you have
00:06:49 to live in the real world as opposed to the elder scrolls world, you know, whatever
00:06:54 brings you joy, right?
00:06:54 Minecraft, right?
00:06:55 Minecraft is a great example.
00:06:56 You create, like it’s not the fancy graphics, but it’s the creation of your own worlds.
00:07:01 Yeah, that one is crazy.
00:07:02 You know, one of the pitches for being a parent that people tell me is that you
00:07:06 can like use the excuse of parenting to, to go back into the video game world.
00:07:12 And like, like that’s like, you know, father, son, father, daughter time, but
00:07:17 really you just get to play video games with your kids.
00:07:19 So anyway, at that time, did you have any ridiculously ambitious dreams of where as
00:07:27 a creator, you might go as an engineer?
00:07:29 Did you, what, what did you think of yourself as, as an engineer, as a tinker,
00:07:33 or did you want to be like an astronaut or something like that?
00:07:37 You know, I’m tempted to make something up about, you know, robots, uh, engineering
00:07:42 or, you know, mysteries of the universe, but that’s not the actual memory that
00:07:45 pops into my mind when you, when you asked me about childhood dreams.
00:07:48 So I’ll actually share the, the, the real thing, uh, when I was maybe four or five
00:07:55 years old, I, you know, as we all do, I thought about, you know, what I wanted
00:08:00 to do when I grow up and I had this dream of being a traffic control cop.
00:08:08 Uh, you know, they don’t have those today’s I think, but you know, back in
00:08:11 the eighties and in Russia, uh, you probably are familiar with that Lex.
00:08:15 They had these, uh, you know, police officers that would stand in the middle
00:08:19 of intersection all day and they would have their like stripe back, black and
00:08:21 white batons that they would use to control the flow of traffic and, you
00:08:26 know, for whatever reasons, I was strangely infatuated with this whole
00:08:29 process and like that, that was my dream.
00:08:32 Uh, that’s what I wanted to do when I grew up and, you know, my parents, uh,
00:08:37 both physics profs, by the way, I think were, you know, a little concerned, uh,
00:08:41 with that level of ambition coming from their child.
00:08:44 Uh, uh, you know, that age.
00:08:46 Well, that it’s an interesting, I don’t know if you can relate,
00:08:50 but I very much love that idea.
00:08:52 I have a OCD nature that I think lends itself very close to the engineering
00:08:57 mindset, which is you want to kind of optimize, you know, solve a problem by
00:09:05 create, creating an automated solution, like a, like a set of rules, that set
00:09:11 of rules you can follow and then thereby make it ultra efficient.
00:09:14 I don’t know if that’s, it was of that nature.
00:09:17 I certainly have that.
00:09:18 There’s like fact, like SimCity and factory building games, all those
00:09:22 kinds of things kind of speak to that engineering mindset, or
00:09:26 did you just like the uniform?
00:09:27 I think it was more of the latter.
00:09:28 I think it was the uniform and the, you know, the, the stripe baton that
00:09:33 made cars go in the right directions.
00:09:36 But I guess, you know, I, it is, I did end up, uh, I guess, uh,
00:09:40 you know, working on the transportation industry one way or another uniform.
00:09:44 No, but that’s right.
00:09:46 Maybe, maybe, maybe it was my, you know, deep inner infatuation with the,
00:09:51 you know, traffic control batons that led to this career.
00:09:55 Okay.
00:09:55 What, uh, when did you, when was the leap from programming to robotics?
00:10:00 That happened later.
00:10:01 That was after grad school, uh, after, and I actually, the most self driving
00:10:05 cars was I think my first real hands on introduction to robotics.
00:10:10 But I never really had that much hands on experience in school and training.
00:10:14 I, you know, worked on applied math and physics.
00:10:17 Then in college, I did more half, uh, abstract computer science.
00:10:23 And it was after grad school that I really got involved in robotics, which
00:10:28 was actually self driving cars.
00:10:29 And, you know, that was a big flip.
00:10:32 What, uh, what grad school?
00:10:34 So I went to grad school in Michigan, and then I did a postdoc at Stanford,
00:10:37 uh, which is, that was the postdoc where I got to play with self driving cars.
00:10:42 Yeah.
00:10:42 So we’ll return there.
00:10:43 Let’s go back to, uh, to Moscow.
00:10:46 So, uh, you know, for episode 100, I talked to my dad and also I
00:10:50 grew up with my dad, I guess.
00:10:53 Uh, so I had to put up with them for many years and, uh, he, he went to the
00:11:01 FISTIEG or MIPT, it’s weird to say in English, cause I’ve heard all this
00:11:06 in Russian, Moscow Institute of Physics and Technology.
00:11:09 And to me, that was like, I met some super interesting, as a child, I met
00:11:15 some super interesting characters.
00:11:17 It felt to me like the greatest university in the world, the most elite
00:11:21 university in the world, and just the people that I met that came out of there
00:11:26 were like, not only brilliant, but also special humans.
00:11:32 It seems like that place really tested the soul, uh, both like in terms
00:11:37 of technically and like spiritually.
00:11:40 So that could be just the romanticization of that place.
00:11:43 I’m not sure, but so maybe you can speak to it, but is it correct to
00:11:47 say that you spent some time at FISTIEG?
00:11:50 Yeah, that’s right.
00:11:50 Six years.
00:11:51 Uh, I got my bachelor’s and master’s in physics and math there.
00:11:55 And it’s actually interesting cause my, my dad, and actually both my parents,
00:11:59 uh, went there and I think all the stories that I heard, uh, like, just
00:12:03 like you, Alex, uh, growing up about the place and, you know, how interesting
00:12:07 and special and magical it was, I think that was a significant, maybe the
00:12:11 main reason, uh, I wanted to go there, uh, for college, uh, enough so that
00:12:16 I actually went back to Russia from the U S I graduated high school in the U S.
00:12:21 Um, and you went back there.
00:12:23 I went back there.
00:12:24 Yeah, that’s exactly the reaction most of my peers in college had.
00:12:28 But, you know, perhaps a little bit stronger that like, you know, point
00:12:32 me out as this crazy kid, were your parents supportive of that?
00:12:34 Yeah.
00:12:35 Yeah.
00:12:35 My games, your previous question, they, uh, they supported me and, you know,
00:12:38 letting me kind of pursue my passions and the things that I was interested in.
00:12:43 That’s a bold move.
00:12:44 Wow.
00:12:44 What was it like there?
00:12:45 It was interesting, you know, definitely fairly hardcore on the fundamentals
00:12:49 of, you know, math and physics and, uh, you know, lots of good memories,
00:12:53 uh, from, you know, from those times.
00:12:55 So, okay.
00:12:56 So Stanford.
00:12:57 How’d you get into autonomous vehicles?
00:12:59 I had the great fortune, uh, and great honor to join Stanford’s
00:13:06 DARPA urban challenge team.
00:13:07 And, uh, 2006 there, this was a third in the sequence of the DARPA challenges.
00:13:12 There were two grand challenges prior to that.
00:13:14 And then in 2007, they held the DARPA urban challenge.
00:13:19 So, you know, I was doing my, my postdoc I had, I joined the team and, uh, worked
00:13:25 on motion planning, uh, for, you know, that, that competition.
00:13:30 So, okay.
00:13:30 So for people who might not know, I know from, from certain autonomous vehicles is
00:13:35 a funny world in a certain circle of people, everybody knows everything.
00:13:39 And then the certain circle, uh, nobody knows anything in terms of general public.
00:13:45 So it’s interesting.
00:13:46 It’s, it’s a good question of what to talk about, but I do think that the urban
00:13:50 challenge is worth revisiting. It’s a fun little challenge.
00:13:56 One that, first of all, like sparked so much, so many incredible minds to focus
00:14:03 on one of the hardest problems of our time in artificial intelligence.
00:14:06 So that’s, that’s a success from a perspective of a single little challenge.
00:14:11 But can you talk about like, what did the challenge involve?
00:14:14 So were there pedestrians, were there other cars, what was the goal?
00:14:18 Uh, who was on the team?
00:14:20 How long did it take any fun, fun sort of specs?
00:14:25 Sure, sure, sure.
00:14:26 So the way the challenge was constructed and just a little bit of backgrounding,
00:14:29 as I mentioned, this was the third, uh, competition in that series.
00:14:34 The first year we’re at the grand challenge called the grand challenge.
00:14:36 The goal there was to just drive in a completely static environment.
00:14:40 You know, you had to drive in a desert, uh, that was very successful.
00:14:45 So then DARPA followed with what they called the urban challenge, where the
00:14:49 goal was to have, you know, build vehicles that could operate in more dynamic
00:14:54 environments and, you know, share them with other vehicles.
00:14:56 There were no pedestrians there, but what DARPA did is they took over
00:15:00 an abandoned air force base.
00:15:02 Uh, and it was kind of like a little fake city that they built out there.
00:15:06 And they had a bunch of, uh, robots, uh, you know, cars, uh, that were
00:15:11 autonomous, uh, in there all at the same time.
00:15:13 Uh, mixed in with other vehicles driven by professional, uh, drivers and each
00:15:19 car, uh, had a mission and so there’s a crude map that they received, uh,
00:15:24 beginning and they had a mission and go here and then there and over here.
00:15:28 Um, and they kind of all were sharing this environment at the same time.
00:15:32 They had to interact with each other.
00:15:34 They had to interact with the human drivers.
00:15:36 There’s this very first, very rudimentary, um, version of, uh,
00:15:42 self driving car that, you know, could operate, uh, and, uh, in a, in an
00:15:47 environment, you know, shared with other dynamic actors that, as you said,
00:15:50 you know, really, you know, many ways, you know, kickstarted this whole industry.
00:15:55 Okay.
00:15:55 So who was on the team and how’d you do?
00:15:58 I forget.
00:15:59 Uh, I came in second.
00:16:02 Uh, perhaps that was my contribution to the team.
00:16:05 I think the Stanford team came in first in the DARPA challenge.
00:16:07 Uh, but then I joined the team and, you know, you were the one with the
00:16:10 bug in the code, I mean, do you have sort of memories of some
00:16:13 particularly challenging things or, you know, one of the cool things,
00:16:18 it’s not, you know, this isn’t a product, this isn’t the thing that, uh, you know,
00:16:24 it there’s, you have a little bit more freedom to experiment so you can take
00:16:27 risks and there’s, uh, so you can make mistakes.
00:16:30 Uh, so is there interesting mistakes?
00:16:33 Is there interesting challenges that stand out to you as some, like, taught
00:16:36 you, um, a good technical lesson or a good philosophical lesson from that time?
00:16:42 Yeah.
00:16:43 Uh, you know, definitely, definitely a very memorable time, not really
00:16:46 challenged, but like one of the most vivid memories that I have from the time.
00:16:52 And I think that was actually one of the days that really got me hooked, uh, on
00:16:58 this whole field was, uh, the first time I got to run my software and I got to
00:17:05 software on the car and, uh, I was working on a part of our planning algorithm,
00:17:11 uh, that had to navigate in parking lots.
00:17:13 So it was something that, you know, called free space emotion planning.
00:17:16 So the very first version of that, uh, was, you know, we tried on the car, it
00:17:20 was on Stanford’s campus, uh, in the middle of the night and you had this
00:17:24 little course constructed with cones, uh, in the middle of a parking lot.
00:17:28 So we’re there in like 3 am, you know, by the time we got the code to, you
00:17:31 know, uh, uh, you know, compile and turn over, uh, and, you know, it drove, I
00:17:36 could actually did something quite reasonable and, you know, it was of
00:17:39 course very buggy at the time and had all kinds of problems, but it was pretty
00:17:46 darn magical.
00:17:48 I remember going back and, you know, later at night and trying to fall
00:17:52 asleep and just, you know, being unable to fall asleep for the rest of the
00:17:55 night, uh, just my mind was blown.
00:17:57 Just like, and that, that, that’s what I’ve been doing ever since for more
00:18:02 than a decade, uh, in terms of challenges and, uh, you know, interesting
00:18:06 memories, like on the day of the competition, uh, it was pretty nerve
00:18:09 wrecking.
00:18:10 Uh, I remember standing there with Mike Montemarillo, who was, uh, the
00:18:13 software lead and wrote most of the code.
00:18:15 I think I did one little part of the planner, Mike, you know, incredibly
00:18:19 that, you know, pretty much the rest of it, uh, with, with, you know, a bunch
00:18:22 of other incredible people, but I remember standing on the day of the
00:18:25 competition, uh, you know, watching the car, you know, with Mike and cars
00:18:29 are completely empty, right?
00:18:32 They’re all there lined up in the beginning of the race and then, you
00:18:35 know, DARPA sends them, you know, on their mission one by one.
00:18:38 So then leave and Mike, you just, they had these sirens, they all had
00:18:42 their different silence silence, right?
00:18:43 Each siren had its own personality, if you will.
00:18:46 So, you know, off they go and you don’t see them.
00:18:48 You just kind of, and then every once in a while they come a little bit
00:18:50 closer to where the audience is and you can kind of hear, you know, the
00:18:55 sound of your car and then, you know, it seems to be moving along.
00:18:57 So that, you know, gives you hope.
00:18:58 And then, you know, it goes away and you can’t hear it for too long.
00:19:01 You start getting anxious, right?
00:19:02 So it’s a little bit like, you know, sending your kids to college and like,
00:19:04 you know, kind of you invested in them.
00:19:05 You hope you, you, you, you, you, you, you build it properly, but like,
00:19:09 it’s still, uh, anxiety inducing.
00:19:11 Uh, so that was, uh, an incredibly, uh, fun, uh, few days in terms of, you
00:19:16 know, bugs, as you mentioned, you know, one that that was my bug that caused
00:19:20 us the loss of the first place, uh, is still a debate that, you know,
00:19:24 occasionally have with people on the CMU team, CMU came first, I should
00:19:27 mention, uh, that you haven’t heard of them, but yeah, it’s something, you
00:19:32 know, it’s a small school, but it’s, it’s, it’s, you know, really a glitch
00:19:35 that, you know, they happen to succeed at something robotics related.
00:19:38 Very scenic though.
00:19:39 So most people go there for the scenery.
00:19:41 Um, yeah, it’s a beautiful campus.
00:19:45 I’m like, unlike Stanford.
00:19:46 So for people, yeah, that’s true.
00:19:48 Unlike Stanford, for people who don’t know, CMU is one of the great robotics
00:19:51 and sort of artificial intelligence universities in the world, CMU, Carnegie
00:19:55 Mellon university, okay, sorry, go ahead.
00:19:58 Good, good PSA.
00:19:59 So in the part that I contributed to, which was navigating parking lots and
00:20:06 the way that part of the mission work is, uh, you in a parking lot, you
00:20:12 would get from DARPA an outline of the map.
00:20:15 You basically get this, you know, giant polygon that defined the
00:20:18 perimeter of the parking lot, uh, and there would be an entrance and, you
00:20:21 know, so maybe multiple entrances or access to it, and then you would get a
00:20:25 goal, uh, within that open space, uh, X, Y, you know, heading where the car had
00:20:32 to park and had no information about the optical, so obstacles that the car might
00:20:36 encounter there.
00:20:36 So it had to navigate a kind of completely free space, uh, from the
00:20:40 entrance to the parking lot into that parking space.
00:20:43 And then, uh, once parked there, it had to, uh, exit the parking lot, you know,
00:20:50 while of course, I’m counting and reasoning about all the obstacles that
00:20:53 it encounters in real time.
00:20:54 So, uh, Our interpretation, or at least my interpretation of the rules was that
00:21:00 you had to reverse out of the parking spot.
00:21:03 And that’s what our cars did.
00:21:04 Even if there’s no obstacle in front, that’s not what CMU’s car did.
00:21:08 And it just kind of drove right through.
00:21:10 So there’s still a debate.
00:21:12 And of course, you know, as you stop and then reverse out and go out the
00:21:14 different way that costs you some time.
00:21:16 And so there’s still a debate whether, you know, it was my poor implementation
00:21:20 that cost us extra time or whether it was, you know, CMU, uh, violating an
00:21:26 important rule of the competition.
00:21:27 And, you know, I have my own, uh, opinion here in terms of other bugs.
00:21:30 And like, uh, I, I have to apologize to Mike Montemarila, uh, for sharing this
00:21:34 on air, but it is actually, uh, one of the more memorable ones.
00:21:38 Uh, and it’s something that’s kind of become a bit of, uh, a metaphor and
00:21:42 a label in the industry, uh, since then, I think, you know, at least in some
00:21:46 circles, it’s called the victory circle or victory lap.
00:21:49 Um, and, uh, uh, our cars did that.
00:21:53 So in one of the missions in the urban challenge, in one of the courses, uh,
00:21:57 there was this big oval, right by the start and finish of the race.
00:22:02 So the ARPA had a lot of the missions would finish kind of in that same location.
00:22:05 Uh, and it was pretty cool because you could see the cars come by, you know,
00:22:08 kind of finished that part leg of the trip, that leg of the mission, and then,
00:22:11 you know, go on and finish the rest of it.
00:22:15 Uh, and other vehicles would, you know, come hit their waypoint, uh, and, you
00:22:22 know, exit the oval and off they would go.
00:22:24 Our car on the hand, which hit the checkpoint, and then it would do an extra
00:22:28 lap around the oval and only then, you know, uh, leave and go on its merry way.
00:22:31 So over the course of the full day, it accumulated, uh, uh,
00:22:34 some extra time and the problem was that we had a bug where it wouldn’t, you know,
00:22:38 start reasoning about the next waypoint and plan a route to get to that next
00:22:41 point until it hit a previous one.
00:22:42 And in that particular case, by the time you hit the, that, that one, it was too
00:22:46 late for us to consider the next one and kind of make a lane change.
00:22:49 So at every time we would do like an extra lap.
00:22:50 So, you know, and that’s the Stanford victory lap.
00:22:55 The victory lap.
00:22:57 Oh, that’s there’s, I feel like there’s something philosophically
00:22:59 profound in there somehow, but, uh, I mean, ultimately everybody is
00:23:03 a winner in that kind of competition.
00:23:06 And it led to sort of famously to the creation of, um, Google self driving
00:23:13 car project and now Waymo.
00:23:15 So can we, uh, give an overview of how is Waymo born?
00:23:20 How’s the Google self driving car project born?
00:23:23 What’s the, what is the mission?
00:23:24 What is the hope?
00:23:26 What is it is the engineering kind of, uh, set of milestones that
00:23:32 it seeks to accomplish, there’s a lot of questions in there.
00:23:35 Uh, yeah, uh, I don’t know, kind of the DARPA urban challenge and the DARPA
00:23:40 and previous DARPA grand challenges, uh, kind of led, I think to a very large
00:23:44 degree to that next step and then, you know, Larry and Sergey, um, uh, Larry
00:23:48 Page and Sergey Brin, uh, uh, Google founders course, uh, I saw that
00:23:52 competition and believed in the technology.
00:23:54 So, you know, the Google self driving car project was born, you know, at that time.
00:23:59 And we started in 2009, it was a pretty small group of us, about a dozen people,
00:24:04 uh, who came together, uh, to, to work on this project at Google.
00:24:09 At that time we saw an incredible early result in the DARPA urban challenge.
00:24:18 I think we’re all incredibly excited, uh, about where we got to and we believed
00:24:23 in the future of the technology, but we still had a very, you know,
00:24:27 very, you know, rudimentary understanding of the problem space.
00:24:31 So the first goal of this project in 2009 was to really better
00:24:37 understand what we’re up against.
00:24:39 Uh, and, you know, with that goal in mind, when we started the project, we created a
00:24:44 few milestones for ourselves, uh, that.
00:24:48 Maximized learnings.
00:24:49 Well, the two milestones were, you know, uh, one was to drive a hundred thousand
00:24:54 miles in autonomous mode, which was at that time, you know, orders of magnitude
00:24:57 that, uh, more than anybody has ever done.
00:25:01 And the second milestone was to drive 10 routes, uh, each one was a hundred miles
00:25:07 long, uh, and there were specifically chosen to become extra spicy and extra
00:25:12 complicated and sample the full complexity of the, that, that, uh, domain.
00:25:18 Um, uh, and you had to drive each one from beginning to end with no intervention,
00:25:24 no human intervention.
00:25:24 So you would get to the beginning of the course, uh, you would press the button
00:25:28 that would engage in autonomy and you had to go for a hundred miles, you know,
00:25:32 beginning to end, uh, with no interventions.
00:25:35 Um, and it sampled again, the full complexity of driving conditions.
00:25:40 Some, uh, were on freeways.
00:25:42 We had one route that went all through all the freeways and all
00:25:45 the bridges in the Bay area.
00:25:46 You know, we had, uh, some that went around Lake Tahoe and kind of mountains,
00:25:50 uh, roads.
00:25:52 We had some that drove through dense urban, um, environments like in downtown
00:25:56 Palo Alto and through San Francisco.
00:25:59 So it was incredibly, uh, interesting, uh, to work on.
00:26:04 And it, uh, it took us just under two years, uh, about a year and a half,
00:26:10 a little bit more to finish both of these milestones.
00:26:14 And in that process, uh, you know, it was an incredible amount of fun,
00:26:20 probably the most fun I had in my professional career.
00:26:22 And you’re just learning so much.
00:26:24 You are, you know, the goal here is to learn and prototype.
00:26:26 You’re not yet starting to build a production system, right?
00:26:29 So you just, you were, you know, this is when you’re kind of working 24 seven
00:26:33 and you’re hacking things together.
00:26:34 And you also don’t know how hard this is.
00:26:37 I mean, that’s the point.
00:26:38 Like, so, I mean, that’s an ambitious, if I put myself in that mindset, even
00:26:42 still, that’s a really ambitious set of goals.
00:26:46 Like just those two picking, picking 10 different, difficult, spicy challenges.
00:26:56 And then having zero interventions.
00:26:59 So like not saying gradually we’re going to like, you know, over a period of 10
00:27:05 years, we’re going to have a bunch of routes and gradually reduce the number
00:27:09 of interventions, you know, that literally says like, by as soon as
00:27:13 possible, we want to have zero and on hard roads.
00:27:17 So like, to me, if I was facing that, it’s unclear that whether that takes
00:27:23 two years or whether that takes 20 years.
00:27:26 I mean, it took us under two.
00:27:27 I guess that that speaks to a really big difference between doing something
00:27:32 once and having a prototype where you are going after, you know, learning
00:27:37 about the problem versus how you go about engineering a product that, you
00:27:42 know, where you look at, you know, you do properly do evaluation, you look
00:27:47 at metrics, you drive down and you’re confident that you can do that.
00:27:50 And I guess that’s the, you know, why it took a dozen people, you know, 16
00:27:55 months or a little bit more than that back in 2009 and 2010 with the
00:28:00 technology of, you know, the more than a decade ago that amount of time to
00:28:05 achieve that milestone of, you know, 10 routes, a hundred miles each and no
00:28:10 interventions, and, you know, it took us a little bit longer to get to, you
00:28:17 know, a full driverless product that customers use.
00:28:20 That’s another really important moment.
00:28:21 Is there some memories of technical lessons or just one, like, what did you
00:28:29 learn about the problem of driving from that experience?
00:28:32 I mean, we can, we can now talk about like what you learned from modern day
00:28:36 Waymo, but I feel like you may have learned some profound things in those
00:28:41 early days, even more so because it feels like what Waymo is now is to trying
00:28:47 to, you know, how to do scale, how to make sure you create a product, how to
00:28:51 make sure it’s like safety and all those things, which is all fascinating
00:28:54 challenges, but like you were facing the more fundamental philosophical
00:28:59 problem of driving in those early days.
00:29:02 Like what the hell is driving as an autonomous, or maybe I’m again
00:29:07 romanticizing it, but is it, is there, is there some valuable lessons you
00:29:14 picked up over there at those two years?
00:29:18 A ton.
00:29:19 The most important one is probably that we believe that it’s doable and we’ve
00:29:25 gotten far enough into the problem that, you know, we had a, I think only a
00:29:31 glimpse of the true complexity of the, that the domain, you know, it’s a
00:29:37 little bit like, you know, climbing a mountain where you kind of, you know,
00:29:39 see the next peak and you think that’s kind of the summit, but then you get
00:29:42 to that and you kind of see that, that this is just the start of the journey.
00:29:46 But we’ve tried, we’ve sampled enough of the problem space and we’ve made
00:29:50 enough rapid success, even, you know, with technology of 2009, 2010, that
00:29:57 it gave us confidence to then, you know, pursue this as a real product.
00:30:02 So, okay.
00:30:04 So the next step, you mentioned the milestones that you had in the, in those
00:30:09 two years, what are the next milestones that then led to the creation of Waymo
00:30:13 and beyond?
00:30:14 Yeah, we had a, it was a really interesting journey and, you know, Waymo
00:30:18 came a little bit later, then, you know, we completed those milestones in 2010.
00:30:25 That was the pivot when we decided to focus on actually building a product
00:30:30 using this technology.
00:30:32 The initial couple of years after that, we were focused on a freeway, you
00:30:37 know, what you would call a driver assist, maybe, you know, an L3 driver
00:30:41 assist program.
00:30:42 Then around 2013, we’ve learned enough about the space and thought more deeply
00:30:49 about, you know, the product that we wanted to build, that we pivoted, we
00:30:54 pivoted towards this vision of building a driver and deploying it fully driverless
00:31:01 vehicles without a person.
00:31:02 And that that’s the path that we’ve been on since then.
00:31:05 And very, it was exactly the right decision for us.
00:31:08 So there was a moment where you’re also considered like, what is the right
00:31:13 trajectory here?
00:31:14 What is the right role of automation in the, in the task of driving?
00:31:18 There’s still, it wasn’t from the early days, obviously you want to go fully
00:31:23 autonomous.
00:31:24 From the early days, it was not.
00:31:25 I think it was in 20, around 2013, maybe that we’ve, that became very clear and
00:31:31 we made that pivot and also became very clear and that it’s either the way you
00:31:36 go building a driver assist system is, you know, fundamentally different from
00:31:41 how you go building a fully driverless vehicle.
00:31:43 So, you know, we’ve pivoted towards the ladder and that’s what we’ve been
00:31:48 working on ever since.
00:31:50 And so that was around 2013, then there’s sequence of really meaningful for us
00:31:57 really important defining milestones since then.
00:32:00 And in 2015, we had our first, actually the world’s first fully driverless
00:32:12 trade on public roads.
00:32:15 It was in a custom built vehicle that we had.
00:32:17 I must’ve seen those.
00:32:18 We called them the Firefly, that, you know, funny looking marshmallow looking
00:32:21 thing.
00:32:22 And we put a passenger, his name was Steve Mann, you know, great friend of
00:32:30 our project from the early days, the man happens to be blind.
00:32:34 So we put them in that vehicle.
00:32:36 The car had no steering wheel, no pedals.
00:32:38 It was an uncontrolled environment.
00:32:40 You know, no, you know, lead or chase cars, no police escorts.
00:32:44 And, you know, we did that trip a few times in Austin, Texas.
00:32:47 So that was a really big milestone.
00:32:49 But that was in Austin.
00:32:50 Yeah.
00:32:51 Okay.
00:32:52 And, you know, we only, but at that time we’re only, it took a tremendous
00:32:56 amount of engineering.
00:32:57 It took a tremendous amount of validation to get to that point.
00:33:01 But, you know, we only did it a few times.
00:33:03 We only did that.
00:33:04 It was a fixed route.
00:33:05 It was not kind of a controlled environment, but it was a fixed route.
00:33:08 And we only did a few times.
00:33:10 Then in 2016, end of 2016, beginning of 2017 is when we founded Waymo, the
00:33:19 company.
00:33:20 That’s when we kind of, that was the next phase of the project where I
00:33:25 wanted, we believed in kind of the commercial vision of this technology.
00:33:30 And it made sense to create an independent entity, you know, within
00:33:33 that alphabet umbrella to pursue this product at scale.
00:33:39 Beyond that in 2017, later in 2017 was another really huge step for us.
00:33:46 Really big milestone where we started, I think it was October of 2017 where
00:33:52 when we started regular driverless operations on public roads, that first
00:33:59 day of operations, we drove in one day.
00:34:02 And that first day, a hundred miles and driverless fashion.
00:34:05 And then we’ve now the most, the most important thing about that milestone
00:34:08 was not that, you know, a hundred miles in one day, but that it was the
00:34:11 start of kind of regular ongoing driverless operations.
00:34:14 And when you say driverless, it means no driver.
00:34:19 That’s exactly right.
00:34:19 So on that first day, we actually hit a mix and in some, we didn’t want
00:34:24 to like, you know, be on YouTube and Twitter that same day.
00:34:27 So in, in many of the rides we had somebody in the driver’s seat, but
00:34:32 they could not disengage like the car, not disengage, but actually on that
00:34:36 first day, some of the miles were driven and just completely empty driver’s seat.
00:34:42 And this is the key distinction that I think people don’t realize it’s, you
00:34:46 know, that oftentimes when you talk about autonomous vehicles, you’re, there’s
00:34:53 often a driver in the seat that’s ready to to take over what’s called a safety
00:34:59 driver and then Waymo is really one of the only companies at least that I’m
00:35:05 aware of, or at least as like boldly and carefully and all, and all of that is
00:35:10 actually has cases.
00:35:12 And now we’ll talk about more and more where there’s literally no driver.
00:35:17 So that’s another, the interesting case of where the driver’s not supposed
00:35:21 to disengage, that’s like a nice middle ground, they’re still there, but
00:35:24 they’re not supposed to disengage, but really there’s the case when there’s
00:35:28 no, okay, there’s something magical about there being nobody in the driver’s seat.
00:35:34 Like, just like to me, you mentioned the first time you wrote some code for free
00:35:41 space navigation of the parking lot, that was like a magical moment to me, just
00:35:46 sort of as an observer of robots, the first magical moment is seeing an
00:35:53 autonomous vehicle turn, like make a left turn, like apply sufficient torque to
00:36:01 the steering wheel to where it, like, there’s a lot of rotation and for some
00:36:05 reason, and there’s nobody in the driver’s seat, for some reason that
00:36:10 communicates that here’s a being with power that makes a decision.
00:36:16 There’s something about like the steering wheel, cause we perhaps romanticize
00:36:19 the notion of the steering wheel, it’s so essential to our conception, our 20th
00:36:24 century conception of a car and it turning the steering wheel with nobody
00:36:28 in driver’s seat, that to me, I think maybe to others, it’s really powerful.
00:36:34 Like this thing is in control and then there’s this leap of trust that you give.
00:36:39 Like I’m going to put my life in the hands of this thing that’s in control.
00:36:42 So in that sense, when there’s no, but no driver in the driver’s seat, that’s a
00:36:47 magical moment for robots.
00:36:49 So I’m, I’ve gotten a chance to last year to take a ride in a, in a
00:36:54 way more vehicle and that, that was the magical moment. There’s like nobody in
00:36:58 the driver’s seat. It’s, it’s like the little details. You would think it
00:37:03 doesn’t matter whether there’s a driver or not, but like if there’s no driver
00:37:07 and the steering wheel is turning on its own, I don’t know. That’s magical.
00:37:13 It’s absolutely magical. I, I have taken many of these rides and like completely
00:37:17 empty car, no human in the car pulls up, you know, you call it on your cell phone.
00:37:22 It pulls up, you get in, it takes you on its way. There’s nobody in the car, but
00:37:27 you, right? That’s something called, you know, fully driverless, you know, our
00:37:31 writer only mode of operation. Yeah. It, it is magical. It is, you know,
00:37:39 transformative. This is what we hear from our writers. It kind of really
00:37:44 changes your experience. And not like that, that really is what unlocks the
00:37:48 real potential of this technology. But, you know, coming back to our journey,
00:37:53 you know, that was 2017 when we started, you know, truly driverless operations.
00:37:58 Then in 2018, we’ve launched our public commercial service that we called
00:38:05 Waymo One in Phoenix. In 2019, we started offering truly driverless writer
00:38:13 only rides to our early rider population of users. And then, you know, 2020 has
00:38:22 also been a pretty interesting year. One of the first ones, less about
00:38:26 technology, but more about the maturing and the growth of Waymo as a company.
00:38:31 We raised our first round of external financing this year, you know, we were
00:38:37 part of Alphabet. So obviously we have access to, you know, significant resources
00:38:42 but as kind of on the journey of Waymo maturing as a company, it made sense
00:38:45 for us to, you know, partially go externally in this round. So, you know,
00:38:50 we’re raised about $3.2 billion from that round. We’ve also started putting
00:38:59 our fifth generation of our driver, our hardware, that is on the new vehicle,
00:39:05 but it’s also a qualitatively different set of self driving hardware.
00:39:10 That is now on the JLR pace. So that was a very important step for us.
00:39:19 Hardware specs, fifth generation. I think it’d be fun to maybe, I apologize if
00:39:25 I’m interrupting, but maybe talk about maybe the generations with a focus on
00:39:31 what we’re talking about on the fifth generation in terms of hardware specs,
00:39:35 like what’s on this car.
00:39:36 Sure. So we separated out, you know, the actual car that we are driving from
00:39:41 the self driving hardware we put on it. Right now we have, so this is, as I
00:39:45 mentioned, the fifth generation, you know, we’ve gone through, we started,
00:39:49 you know, building our own hardware, you know, many, many years ago. And
00:39:56 that, you know, Firefly vehicle also had the hardware suite that was mostly
00:40:01 designed, engineered, and built in house. Lighters are one of the more important
00:40:07 components that we design and build from the ground up. So on the fifth
00:40:11 generation of our drivers of our self driving hardware that we’re switching
00:40:18 to right now, we have, as with previous generations, in terms of sensing,
00:40:24 we have lighters, cameras, and radars, and we have a pretty beefy computer
00:40:29 that processes all that information and makes decisions in real time on
00:40:33 board the car. So in all of the, and it’s really a qualitative jump forward
00:40:41 in terms of the capabilities and the various parameters and the specs of
00:40:45 the hardware compared to what we had before and compared to what you can
00:40:49 kind of get off the shelf in the market today.
00:40:51 Meaning from fifth to fourth or from fifth to first?
00:40:54 Definitely from first to fifth, but also from the fourth.
00:40:57 That was the world’s dumbest question.
00:40:58 Definitely from fourth to fifth, as well as the last step is a big step forward.
00:41:07 So everything’s in house. So like LIDAR is built in house and cameras are
00:41:13 built in house?
00:41:15 You know, it’s different. We work with partners and there’s some components
00:41:19 that we get from our manufacturing and supply chain partners. What exactly
00:41:26 is in house is a bit different. We do a lot of custom design on all of
00:41:34 our sensing modalities, lighters, radars, cameras, you know, exactly.
00:41:37 There’s lighters are almost exclusively in house and some of the
00:41:43 technologies that we have, some of the fundamental technologies there
00:41:45 are completely unique to Waymo. That is also largely true about radars
00:41:51 and cameras. It’s a little bit more of a mix in terms of what we do
00:41:55 ourselves versus what we get from partners.
00:41:57 Is there something super sexy about the computer that you can mention
00:42:01 that’s not top secret? Like for people who enjoy computers for, I
00:42:08 mean, there’s a lot of machine learning involved, but there’s a lot
00:42:12 of just basic compute. You have to probably do a lot of signal
00:42:17 processing on all the different sensors. You have to integrate everything
00:42:20 has to be in real time. There’s probably some kind of redundancy
00:42:23 type of situation. Is there something interesting you can say about
00:42:27 the computer for the people who love hardware? It does have all of
00:42:31 the characteristics, all the properties that you just mentioned.
00:42:34 Redundancy, very beefy compute for general processing, as well as
00:42:41 inference and ML models. It is some of the more sensitive stuff that
00:42:45 I don’t want to get into for IP reasons, but it can be shared a
00:42:49 little bit in terms of the specs of the sensors that we have on the
00:42:54 car. We actually shared some videos of what our
00:43:00 lighters see in the world. We have 29 cameras. We have five lighters.
00:43:05 We have six radars on these vehicles, and you can get a feel for
00:43:09 the amount of data that they’re producing. That all has to be
00:43:12 processed in real time to do perception, to do complex
00:43:16 reasoning. That kind of gives you some idea of how beefy those computers
00:43:19 are, but I don’t want to get into specifics of exactly how we build
00:43:22 them. Okay, well, let me try some more questions that you can get
00:43:25 into the specifics of, like GPU wise. Is that something you can get
00:43:28 into? I know that Google works with GPUs and so on. I mean, for
00:43:33 machine learning folks, it’s kind of interesting. Or is there no…
00:43:38 How do I ask it? I’ve been talking to people in the government about
00:43:43 UFOs and they don’t answer any questions. So this is how I feel
00:43:46 right now asking about GPUs. But is there something interesting that
00:43:51 you could reveal? Or is it just… Or leave it up to our
00:43:57 imagination, some of the compute. Is there any, I guess, is there any
00:44:02 fun trickery? Like I talked to Chris Latner for a second time and he
00:44:05 was a key person about GPUs, and there’s a lot of fun stuff going
00:44:09 on in Google in terms of hardware that optimizes for machine
00:44:15 learning. Is there something you can reveal in terms of how much,
00:44:19 you mentioned customization, how much customization there is for
00:44:23 hardware for machine learning purposes? I’m going to be like that
00:44:26 government person who bought UFOs. I guess I will say that it’s
00:44:34 really… Compute is really important. We have very data hungry
00:44:41 and compute hungry ML models all over our stack. And this is where
00:44:48 both being part of Alphabet, as well as designing our own sensors
00:44:52 and the entire hardware suite together, where on one hand you
00:44:55 get access to really rich raw sensor data that you can pipe
00:45:01 from your sensors into your compute platform and build like
00:45:07 build the whole pipe from sensor raw sensor data to the big
00:45:11 compute as then have the massive compute to process all that
00:45:14 data. And this is where we’re finding that having a lot of
00:45:17 control of that hardware part of the stack is really
00:45:21 advantageous. One of the fascinating magical places to me
00:45:25 again, might not be able to speak to the details, but it is
00:45:29 the other compute, which is like, we’re just talking about a
00:45:32 single car, but the driving experience is a source of a lot
00:45:39 of fascinating data. And you have a huge amount of data
00:45:42 coming in on the car and the infrastructure of storing some
00:45:47 of that data to then train or to analyze or so on. That’s a
00:45:52 fascinating piece of it that I understand a single car. I
00:45:58 don’t understand how you pull it all together in a nice way.
00:46:00 Is that something that you could speak to in terms of the
00:46:03 challenges of seeing the network of cars and then
00:46:08 bringing the data back and analyzing things that like edge
00:46:12 cases of driving, be able to learn on them to improve the
00:46:15 system to see where things went wrong, where things went right
00:46:20 and analyze all that kind of stuff. Is there something
00:46:22 interesting there from an engineering perspective?
00:46:25 Oh, there’s an incredible amount of really interesting
00:46:30 work that’s happening there, both in the real time operation
00:46:35 of the fleet of cars and the information that they exchange
00:46:38 with each other in real time to make better decisions as well
00:46:43 as on the kind of the off board component where you have to
00:46:46 deal with massive amounts of data for training your ML
00:46:50 models, evaluating the ML models for simulating the entire
00:46:54 system and for evaluating your entire system. And this is
00:46:57 where being part of Alphabet has once again been tremendously
00:47:03 advantageous because we consume an incredible amount of
00:47:06 compute for ML infrastructure. We build a lot of custom
00:47:10 frameworks to get good at data mining, finding the
00:47:16 interesting edge cases for training and for evaluation of
00:47:19 the system for both training and evaluating some components
00:47:23 and your sub parts of the system and various ML models,
00:47:27 as well as the evaluating the entire system and simulation.
00:47:31 Okay. That first piece that you mentioned that cars
00:47:33 communicating to each other, essentially, I mean, through
00:47:36 perhaps through a centralized point, but what that’s
00:47:40 fascinating too, how much does that help you? Like if you
00:47:43 imagine, you know, right now the number of way more vehicles
00:47:46 is whatever X. I don’t know if you can talk to what that
00:47:50 number is, but it’s not in the hundreds of millions yet. And
00:47:55 imagine if the whole world is way more vehicles, like that
00:47:59 changes potentially the power of connectivity. Like the more
00:48:03 cars you have, I guess, actually, if you look at
00:48:06 Phoenix, cause there’s enough vehicles, there’s enough, when
00:48:09 there’s like some level of density, you can start to
00:48:13 probably do some really interesting stuff with the fact
00:48:15 that cars can negotiate, can be, can communicate with each
00:48:21 other and thereby make decisions. Is there something
00:48:24 interesting there that you can talk to about like, how does
00:48:27 that help with the driving problem from, as compared to
00:48:31 just a single car solving the driving problem by itself?
00:48:35 Yeah, it’s a spectrum. I first and say that, you know, it’s,
00:48:40 it helps and it helps in various ways, but it’s not required
00:48:44 right now with the way we build our system, like each cars can
00:48:46 operate independently. They can operate with no connectivity.
00:48:49 So I think it is important that, you know, you have a fully
00:48:53 autonomous, fully capable driver that, you know, computerized
00:48:59 driver that each car has. Then, you know, they do share
00:49:03 information and they share information in real time. It
00:49:06 really, really helps. So the way we do this today is, you know,
00:49:11 whenever one car encounters something interesting in the
00:49:15 world, whether it might be an accident or a new construction
00:49:17 zone, that information immediately gets, you know,
00:49:21 uploaded over the air and it’s propagated to the rest of the
00:49:23 fleet. So, and that’s kind of how we think about maps as
00:49:27 priors in terms of the knowledge of our drivers, of our fleet of
00:49:32 drivers that is distributed across the fleet and it’s
00:49:38 updated in real time. So that’s one use case. And
00:49:41 you know, you can imagine as the, you know, the density of
00:49:46 these vehicles go up, that they can exchange more information
00:49:50 in terms of what they’re planning to do and start
00:49:53 influencing how they interact with each other, as well as,
00:49:56 you know, potentially sharing some observations, right, to
00:49:59 help with, you know, if you have enough density of these
00:50:01 vehicles where, you know, one car might be seeing something
00:50:04 that another is relevant to another car that is very
00:50:06 dynamic. You know, it’s not part of kind of your updating
00:50:08 your static prior of the map of the world, but it’s more of a
00:50:11 dynamic information that could be relevant to the decisions
00:50:14 that another car is making real time. So you can see them
00:50:16 exchanging that information and you can build on that. But
00:50:18 again, I see that as an advantage, but it’s not a
00:50:23 requirement. So what about the human in the loop? So when I
00:50:28 got a chance to drive with a ride in a Waymo, you know,
00:50:34 there’s customer service. So like there is somebody that’s
00:50:39 able to dynamically like tune in and help you out. What role
00:50:48 does the human play in that picture? That’s a fascinating
00:50:51 like, you know, the idea of teleoperation, be able to
00:50:53 remotely control a vehicle. So here, what we’re talking
00:50:57 about is like, like frictionless, like a human being
00:51:03 able to in a in a frictionless way, sort of help you out. I
00:51:08 don’t know if they’re able to actually control the vehicle.
00:51:10 Is that something you could talk to? Yes. Okay. To be clear,
00:51:14 we don’t do teleporation. I kind of believe in
00:51:16 teleporation for various reasons. That’s not what we
00:51:19 have in our cars. We do, as you mentioned, have, you know,
00:51:22 version of, you know, customer support. You know, we call it
00:51:24 life health. In fact, we find it that it’s very important for
00:51:28 our ride experience, especially if it’s your first trip, you’ve
00:51:32 never been in a fully driverless ride or only way more
00:51:35 vehicle you get in, there’s nobody there. And so you can
00:51:37 imagine having all kinds of, you know, questions in your head,
00:51:40 like how this thing works. So we’ve put a lot of thought into
00:51:43 kind of guiding our, our writers or customers through that
00:51:47 experience, especially for the first time they get some
00:51:49 information on the phone. If the fully driverless vehicle is
00:51:54 used to service their trip, when you get into the car, we
00:51:58 have an in car, you know, screen and audio that kind of guides
00:52:01 them and explains what to expect. They also have a button
00:52:05 that they can push that will connect them to, you know, a
00:52:09 real life human being that they can talk to, right, about this
00:52:13 whole process. So that’s one aspect of it. There is, you
00:52:16 know, I should mention that there is another function that
00:52:21 humans provide to our cars, but it’s not teleoperation. You can
00:52:24 think of it a little bit more like, you know, fleet
00:52:26 assistance, kind of like, you know, traffic control that you
00:52:29 have, where our cars, again, they’re responsible on their own
00:52:34 for making all of the decisions, all of the driving decisions
00:52:37 that don’t require connectivity. They, you know,
00:52:40 anything that is safety or latency critical is done, you
00:52:44 know, purely autonomously by onboard, our onboard system.
00:52:49 But there are situations where, you know, if connectivity is
00:52:51 available, when a car encounters a particularly challenging
00:52:53 situation, you can imagine like a super hairy scene of an
00:52:57 accident, the cars will do their best, they will recognize that
00:53:00 it’s an off nominal situation, they will do their best to come
00:53:05 up with the right interpretation, the best course
00:53:07 of action in that scenario. But if connectivity is available,
00:53:09 they can ask for confirmation from, you know, human
00:53:15 assistant to kind of confirm those actions and perhaps
00:53:19 provide a little bit of kind of contextual information and
00:53:22 guidance. So October 8th was when you’re talking about the
00:53:26 was Waymo launched the fully self, the public version of
00:53:33 its fully driverless, that’s the right term, I think, service
00:53:38 in Phoenix. Is that October 8th? That’s right. It was the
00:53:41 introduction of fully driverless, right, our only
00:53:43 vehicles into our public Waymo One service. Okay, so that’s
00:53:47 that’s amazing. So it’s like anybody can get into Waymo in
00:53:51 Phoenix. So we previously had early people in our early
00:53:57 rider program, taking fully driverless rides in Phoenix.
00:54:01 And just this a little while ago, we opened on October 8th,
00:54:06 we opened that mode of operation to the public. So I
00:54:09 can download the app and go on a ride. There’s a lot more
00:54:14 demand right now for that service. And then we have
00:54:17 capacity. So we’re kind of managing that. But that’s
00:54:20 exactly the way to describe it. Yeah, that’s interesting. So
00:54:22 there’s more demand than you can handle. Like what has been
00:54:28 reception so far? I mean, okay, so this is a product,
00:54:34 right? That’s a whole nother discussion of like how
00:54:38 compelling of a product it is. Great. But it’s also like one
00:54:41 of the most kind of transformational technologies of
00:54:43 the 21st century. So it’s also like a tourist attraction.
00:54:48 Like it’s fun to, you know, to be a part of it. So it’d be
00:54:52 interesting to see like, what do people say? What do people,
00:54:56 what have been the feedback so far? You know, still early
00:54:59 days, but so far, the feedback has been incredible, incredibly
00:55:04 positive. They, you know, we asked them for feedback during
00:55:07 the ride, we asked them for feedback after the ride as part
00:55:10 of their trip. We asked them some questions, we asked them
00:55:12 to rate the performance of our driver. Most by far, you know,
00:55:17 most of our drivers give us five stars in our app, which is
00:55:21 absolutely great to see. And you know, that’s and we’re
00:55:24 they’re also giving us feedback on you know, things we can
00:55:26 improve. And you know, that’s that’s one of the main reasons
00:55:29 we’re doing this as Phoenix and you know, over the last couple
00:55:31 of years, and every day today, we are just learning a
00:55:35 tremendous amount of new stuff from our users. There’s there’s
00:55:38 no substitute for actually doing the real thing, actually
00:55:41 having a fully driverless product out there in the field
00:55:44 with, you know, users that are actually paying us money to
00:55:48 get from point A to point B. So this is a legitimate like,
00:55:51 there’s a paid service. That’s right. And the idea is you use
00:55:56 the app to go from point A to point B. And then what what are
00:55:59 the A’s? What are the what’s the freedom of the of the starting
00:56:03 and ending places? It’s an area of geography where that
00:56:07 service is enabled. It’s a decent size of geography of
00:56:12 territory. It’s actually larger than the size of San Francisco.
00:56:16 And you know, within that, you have full freedom of, you know,
00:56:20 selecting where you want to go. You know, of course, there’s
00:56:22 some and you on your app, you get a map, you tell the car
00:56:27 where you want to be picked up, where you want the car to pull
00:56:31 over and pick you up. And then you tell it where you want to
00:56:33 be dropped off. All right. And of course, there are some
00:56:34 exclusions, right? You want to be you know, you were in terms
00:56:37 of where the car is allowed to pull over, right? So that you
00:56:40 can do. But you know, besides that, it’s amazing. It’s not
00:56:43 like a fixed just would be very I guess. I don’t know. Maybe
00:56:45 that’s what’s the question behind your question. But it’s
00:56:47 not a, you know, preset set of yes, I guess. So within the
00:56:51 geographic constraints with that within that area anywhere
00:56:54 else, it can be you can be picked up and dropped off
00:56:56 anywhere. That’s right. And you know, people use them on like
00:56:59 all kinds of trips. They we have and we have an incredible
00:57:02 spectrum of riders. We I think the youngest actually have car
00:57:05 seats them and we have, you know, people taking their kids
00:57:07 and rides. I think the youngest riders we had on cars are, you
00:57:09 know, one or two years old, you know, and the full spectrum of
00:57:12 use cases people you can take them to, you know, schools to,
00:57:17 you know, go grocery shopping, to restaurants, to bars, you
00:57:21 know, run errands, you know, go shopping, etc, etc. You can go
00:57:24 to your office, right? Like the full spectrum of use cases,
00:57:27 and people are going to use them in their daily lives to get
00:57:31 around. And we see all kinds of really interesting use cases
00:57:37 and that that that’s providing us incredibly valuable
00:57:40 experience that we then, you know, use to improve our
00:57:43 product. So as somebody who’s been on done a few long rants
00:57:50 with Joe Rogan and others about the toxicity of the internet
00:57:53 and the comments and the negativity in the comments, I’m
00:57:56 fascinated by feedback. I believe that most people are
00:58:01 good and kind and intelligent and can provide, like, even in
00:58:07 disagreement, really fascinating ideas. So on a product
00:58:11 side, it’s fascinating to me, like, how do you get the richest
00:58:14 possible user feedback, like, to improve? What’s, what are the
00:58:19 channels that you use to measure? Because, like, you’re
00:58:23 no longer, that’s one of the magical things about autonomous
00:58:28 vehicles is it’s not like it’s frictionless interaction with
00:58:32 the human. So like, you don’t get to, you know, it’s just
00:58:35 giving a ride. So like, how do you get feedback from people
00:58:39 to in order to improve?
00:58:40 Yeah, great question, various mechanisms. So as part of the
00:58:44 normal flow, we ask people for feedback, they as the car is
00:58:48 driving around, we have on the phone and in the car, and we
00:58:51 have a touchscreen in the car, you can actually click some
00:58:54 buttons and provide real time feedback on how the car is
00:58:57 doing, and how the car is handling a particular situation,
00:59:00 you know, both positive and negative. So that’s one
00:59:02 channel, we have, as we discussed, customer support or
00:59:05 life help, where, you know, if a customer wants to, has a
00:59:09 question, or he has some sort of concern, they can talk to a
00:59:13 person in real time. So that that is another mechanism that
00:59:16 gives us feedback. At the end of a trip, you know, we also ask
00:59:21 them how things went, they give us comments, and you know, star
00:59:25 rating. And you know, if it’s, we also, you know, ask them to
00:59:30 explain what you know, one, well, and you know, what could
00:59:33 be improved. And we have our writers providing very rich
00:59:40 feedback, they’re a lot, a large fraction is very passionate,
00:59:44 very excited about this technology. So we get really
00:59:45 good feedback. We also run UXR studies, right, you know,
00:59:49 specific and that are kind of more, you know, go more in
00:59:53 depth. And we will run both kind of lateral and longitudinal
00:59:56 studies, where we have deeper engagement with our customers,
01:00:01 you know, we have our user experience research team,
01:00:04 tracking over time, that’s things about longitudinal is
01:00:07 cool. That’s that’s exactly right. And you know, that’s
01:00:09 another really valuable feedback, source of feedback.
01:00:12 And we’re just covering a tremendous amount, right?
01:00:16 People go grocery shopping, and they like want to load, you
01:00:19 know, 20 bags of groceries in our cars and like that, that’s
01:00:22 one workflow that you maybe don’t think about, you know,
01:00:26 getting just right when you’re building the driverless
01:00:29 product. I have people like, you know, who bike as part of
01:00:34 their trip. So they, you know, bike somewhere, then they get
01:00:37 on our cars, they take apart their bike, they load into our
01:00:39 vehicle, then go and that’s, you know, how they, you know,
01:00:42 where we want to pull over and how that, you know, get in and
01:00:45 get out process works, provides very useful feedback in terms
01:00:51 of what makes a good pickup and drop off location, we get
01:00:55 really valuable feedback. And in fact, we had to do some really
01:01:00 interesting work with high definition maps, and thinking
01:01:05 about walking directions. And if you imagine you’re in a store,
01:01:08 right in some giant space, and then you know, you want to be
01:01:11 picked up somewhere, like if you just drop a pin at a current
01:01:14 location, which is maybe in the middle of a shopping mall, like
01:01:16 what’s the best location for the car to come pick you up? And
01:01:20 you can have simple heuristics where you’re just going to take
01:01:22 your you know, you clean in distance and find the nearest
01:01:25 spot where the car can pull over that’s closest to you. But
01:01:28 oftentimes, that’s not the most convenient one. You know, I have
01:01:30 many anecdotes where that heuristic breaks in horrible
01:01:32 ways. One example that I often mentioned is somebody wanted to
01:01:38 be, you know, dropped off in Phoenix. And you know, we got
01:01:44 car picked location that was close, the closest to there,
01:01:49 you know, where the pin was dropped on the map in terms of,
01:01:51 you know, latitude and longitude. But it happened to be
01:01:55 on the other side of a parking lot that had this row of
01:01:58 cacti. And the poor person had to like walk all around the
01:02:01 parking lot to get to where they wanted to be in 110 degree
01:02:04 heat. So that, you know, that was about so then, you know, we
01:02:06 took all take all of these, all that feedback from our users
01:02:10 and incorporate it into our system and improve it. Yeah, I
01:02:14 feel like that’s like requires AGI to solve the problem of
01:02:17 like, when you’re, which is a very common case, when you’re in
01:02:21 a big space of some kind, like apartment building, it doesn’t
01:02:24 matter, it’s some large space. And then you call the, like a
01:02:29 Waymo from there, right? Like, whatever, it doesn’t matter,
01:02:32 ride share vehicle. And like, where’s the pin supposed to
01:02:37 drop? I feel like that’s, you don’t think, I think that
01:02:41 requires AGI. I’m gonna, in order to solve. Okay, the
01:02:45 alternative, which I think the Google search engine is taught
01:02:50 is like, there’s something really valuable about the
01:02:55 perhaps slightly dumb answer, but a really powerful one,
01:02:58 which is like, what was done in the past by others? Like, what
01:03:02 was the choice made by others? That seems to be like in terms
01:03:06 of Google search, when you have like billions of searches, you
01:03:09 could, you could see which, like when they recommend what you
01:03:13 might possibly mean, they suggest based on not some machine
01:03:17 learning thing, which they also do, but like, on what was
01:03:20 successful for others in the past and finding a thing that
01:03:23 they were happy with. Is that integrated at all? Waymo, like
01:03:27 what, what pickups worked for others? It is. I think you’re
01:03:31 exactly right. So there’s a real, it’s an interesting
01:03:34 problem. Naive solutions have interesting failure modes. So
01:03:43 there’s definitely lots of things that can be done to
01:03:48 improve. And both learning from, you know, what works, but
01:03:54 doesn’t work in actual heal from getting richer data and
01:03:57 getting more information about the environment and richer
01:04:01 maps. But you’re absolutely right, that there’s something
01:04:04 like there’s some properties of solutions that in terms of the
01:04:07 effect that they have on users so much, much, much better than
01:04:10 others, right? And predictability and
01:04:11 understandability is important. So you can have maybe
01:04:14 something that is not quite as optimal, but is very natural
01:04:17 and predictable to the user and kind of works the same way all
01:04:21 the time. And that matters, that matters a lot for the user
01:04:25 experience. And but you know, to get to the basics, the pretty
01:04:30 fundamental property is that the car actually arrives where you
01:04:35 told it to, right? Like, you can always, you know, change it,
01:04:37 see it on the map, and you can move it around if you don’t
01:04:39 like it. And but like, that property that the car actually
01:04:42 shows up reliably is critical, which, you know, where compared
01:04:47 to some of the human driven analogs, I think, you know, you
01:04:52 can have more predictability. It’s actually the fact, if I
01:04:56 have a little bit of a detour here, I think the fact that
01:05:00 it’s, you know, your phone and the cars, two computers talking
01:05:03 to each other, can lead to some really interesting things we
01:05:06 can do in terms of the user interfaces, both in terms of
01:05:09 function, like the car actually shows up exactly where you told
01:05:13 it, you want it to be, but also some, you know, really
01:05:16 interesting things on the user interface, like as the car is
01:05:18 driving, as you call it, and it’s on the way to come pick
01:05:21 you up. And of course, you get the position of the car and the
01:05:23 route on the map. But and they actually follow that route, of
01:05:26 course. But it can also share some really interesting
01:05:29 information about what it’s doing. So, you know, our cars, as
01:05:34 they are coming to pick you up, if it’s come, if a car is
01:05:36 coming up to a stop sign, it will actually show you that
01:05:39 like, it’s there sitting, because it’s at a stop sign or
01:05:41 a traffic light will show you that it’s got, you know,
01:05:42 sitting at a red light. So, you know, they’re like little
01:05:44 things, right? But I find those little touches really
01:05:51 interesting, really magical. And it’s just, you know, little
01:05:54 things like that, that you can do to kind of delight your
01:05:57 users. You know, this makes me think of, there’s some products
01:06:02 that I just love. Like, there’s a there’s a company called
01:06:07 Rev, Rev.com, where I like for this podcast, for example, I
01:06:13 can drag and drop a video. And then they do all the
01:06:17 captioning. It’s humans doing the captioning, but they
01:06:21 connect, they automate everything of connecting you to
01:06:24 the humans, and they do the captioning and transcription.
01:06:27 It’s all effortless. And it like, I remember when I first
01:06:29 started using them, I was like, life’s good. Like, because it
01:06:35 was so painful to figure that out earlier. The same thing
01:06:39 with something called iZotope RX, this company I use for
01:06:43 cleaning up audio, like the sound cleanup they do. It’s
01:06:46 like drag and drop, and it just cleans everything up very
01:06:49 nicely. Another experience like that I had with Amazon
01:06:52 OneClick purchase, first time. I mean, other places do that
01:06:57 now, but just the effortlessness of purchasing,
01:07:00 making it frictionless. It kind of communicates to me, like,
01:07:04 I’m a fan of design. I’m a fan of products that you can just
01:07:08 create a really pleasant experience. The simplicity of
01:07:12 it, the elegance just makes you fall in love with it. So on
01:07:16 the, do you think about this kind of stuff? I mean, it’s
01:07:19 exactly what we’ve been talking about. It’s like the little
01:07:22 details that somehow make you fall in love with the product.
01:07:25 Is that, we went from like urban challenge days, where
01:07:30 love was not part of the conversation, probably. And to
01:07:34 this point where there’s a, where there’s human beings and
01:07:39 you want them to fall in love with the experience. Is that
01:07:42 something you’re trying to optimize for? Try to think
01:07:45 about, like, how do you create an experience that people love?
01:07:48 Absolutely. I think that’s the vision is removing any friction
01:07:55 or complexity from getting our users, our writers to where
01:08:02 they want to go. Making that as simple as possible. And then,
01:08:06 you know, beyond that, just transportation, making things
01:08:10 and goods get to their destination as seamlessly as
01:08:13 possible. I talked about a drag and drop experience where I
01:08:17 kind of express your intent and then it just magically happens.
01:08:20 And for our writers, that’s what we’re trying to get to is
01:08:23 you download an app and you click and car shows up. It’s
01:08:28 the same car. It’s very predictable. It’s a safe and
01:08:33 high quality experience. And then it gets you in a very
01:08:37 reliable, very convenient, frictionless way to where you
01:08:43 want to be. And along the journey, I think we also want to
01:08:47 do little things to delight our users. Like the ride sharing
01:08:52 companies, because they don’t control the experience, I
01:08:56 think they can’t make people fall in love necessarily with
01:09:00 the experience. Or maybe they, they haven’t put in the effort,
01:09:04 but I think if I were to speak to the ride sharing experience
01:09:08 I currently have, it’s just very, it’s just very
01:09:11 convenient, but there’s a lot of room for like falling in love
01:09:16 with it. Like we can speak to sort of car companies, car
01:09:20 companies do this. Well, you can fall in love with a car,
01:09:22 right? And be like a loyal car person, like whatever. Like I
01:09:26 like badass hot rods, I guess, 69 Corvette. And at this point,
01:09:31 you know, you can’t really, cars are so, owning a car is so
01:09:35 20th century, man. But is there something about the Waymo
01:09:41 experience where you hope that people will fall in love with
01:09:43 it? Is that part of it? Or is it part of, is it just about
01:09:48 making a convenient ride, not ride sharing, I don’t know what
01:09:52 the right term is, but just a convenient A to B autonomous
01:09:56 transport or like, do you want them to fall in love with
01:10:02 Waymo? To maybe elaborate a little bit. I mean, almost like
01:10:06 from a business perspective, I’m curious, like how, do you
01:10:11 want to be in the background invisible or do you want to be
01:10:15 like a source of joy that’s in very much in the foreground? I
01:10:20 want to provide the best, most enjoyable transportation
01:10:24 solution. And that means building it, building our
01:10:31 product and building our service in a way that people do.
01:10:34 Kind of use in a very seamless, frictionless way in their
01:10:41 day to day lives. And I think that does mean, you know, in
01:10:45 some way falling in love in that product, right, just kind of
01:10:48 becomes part of your routine. It comes down my mind to safety,
01:10:54 predictability of the experience, and privacy aspects
01:11:02 of it, right? Our cars, you get the same car, you get very
01:11:07 predictable behavior. And you get a lot of different
01:11:11 things. And that is important. And if you’re going to use it
01:11:14 in your daily life, privacy, and when you’re in a car, you
01:11:18 can do other things. You’re spending a bunch, just another
01:11:21 space where you’re spending a significant part of your life.
01:11:24 And so not having to share it with other people who you don’t
01:11:27 want to share it with, I think is a very nice property. Maybe
01:11:32 you want to take a phone call or do something else in the
01:11:34 vehicle. And, you know, safety on the quality of the driving,
01:11:40 as well as the physical safety of not having to share that
01:11:45 ride is important to a lot of people. What about the idea
01:11:52 that when there’s somebody like a human driving, and they do
01:11:56 a rolling stop on a stop sign, like sometimes like, you know,
01:12:01 you get an Uber or Lyft or whatever, like human driver,
01:12:04 and, you know, they can be a little bit aggressive as
01:12:07 drivers. It feels like there’s not all aggression is bad. Now
01:12:14 that may be a wrong, again, 20th century conception of
01:12:17 driving. Maybe it’s possible to create a driving experience.
01:12:21 Like if you’re in the back, busy doing something, maybe
01:12:24 aggression is not a good thing. It’s a very different kind of
01:12:27 experience perhaps. But it feels like in order to navigate
01:12:32 this world, you need to, how do I phrase this? You need to kind
01:12:38 of bend the rules a little bit, or at least test the rules. I
01:12:42 don’t know what language politicians use to discuss this,
01:12:44 but whatever language they use, you like flirt with the rules.
01:12:48 I don’t know. But like you sort of have a bit of an aggressive
01:12:55 way of driving that asserts your presence in this world,
01:13:00 thereby making other vehicles and people respect your
01:13:03 presence and thereby allowing you to sort of navigate
01:13:06 through intersections in a timely fashion. I don’t know if
01:13:10 any of that made sense, but like, how does that fit into the
01:13:14 experience of driving autonomously? Is that?
01:13:18 It’s a lot of thoughts. This is you’re hitting on a very
01:13:20 important point of a number of behavioral components and, you
01:13:27 know, parameters that make your driving feel assertive and
01:13:34 natural and comfortable and predictable. Our cars will
01:13:37 follow rules, right? They will do the safest thing possible in
01:13:39 all situations. Let me be clear on that. But if you think of
01:13:43 really, really good drivers, just think about
01:13:47 professional lemon drivers, right? They will follow the
01:13:49 rules. They’re very, very smooth, and yet they’re very
01:13:53 efficient. But they’re assertive. They’re comfortable
01:13:58 for the people in the vehicle. They’re predictable for the
01:14:02 other people outside the vehicle that they share the
01:14:03 environment with. And that’s the kind of driver that we want
01:14:06 to build. And you think if maybe there’s a sport analogy
01:14:11 there, right? You can do in very many sports, the true
01:14:17 professionals are very efficient in their movements,
01:14:20 right? They don’t do like, you know, hectic flailing, right?
01:14:25 They’re, you know, smooth and precise, right? And they get
01:14:29 the best results. So that’s the kind of driver that we want to
01:14:30 build. In terms of, you know, aggressiveness. Yeah, you can
01:14:33 like, you know, roll through the stop signs. You can do crazy
01:14:35 lane changes. It typically doesn’t get you to your
01:14:38 destination faster. Typically not the safest or most
01:14:40 predictable, very most comfortable thing to do. But
01:14:45 there is a way to do both. And that’s what we’re
01:14:49 doing. We’re trying to build the driver that is safe,
01:14:53 comfortable, smooth, and predictable. Yeah, that’s a
01:14:58 really interesting distinction. I think in the early days of
01:15:00 autonomous vehicles, the vehicles felt cautious as
01:15:03 opposed to efficient. And I’m still probably, but when I
01:15:08 rode in the Waymo, I mean, there was, it was, it was quite
01:15:13 assertive. It moved pretty quickly. Like, yeah, then he’s
01:15:19 one of the surprising feelings was that it actually, it went
01:15:22 fast. And it didn’t feel like, awkwardly cautious than
01:15:28 autonomous vehicle. Like, like, so I’ve also programmed
01:15:31 autonomous vehicles and everything I’ve ever built was
01:15:34 felt awkwardly, either overly aggressive. Okay, especially
01:15:39 when it was my code, or like, awkwardly cautious is the way
01:15:44 I would put it. And Waymo’s vehicle felt like, assertive
01:15:53 and I think efficient is like the right terminology here.
01:15:57 It wasn’t, and I also like the professional limo driver,
01:16:01 because we often think like, you know, an Uber driver or a
01:16:06 bus driver or a taxi. This is the funny thing is people
01:16:09 think they track taxi drivers are professionals. They, I
01:16:14 mean, it’s, it’s like, that that’s like saying, I’m a
01:16:18 professional walker, just because I’ve been walking all
01:16:20 my life. I think there’s an art to it, right? And if you take
01:16:25 it seriously as an art form, then there’s a certain way that
01:16:30 mastery looks like. It’s interesting to think about what
01:16:33 does mastery look like in driving? And perhaps what we
01:16:39 associate with like aggressiveness is unnecessary,
01:16:43 like, it’s not part of the experience of driving. It’s
01:16:46 like, unnecessary fluff, that efficiency, you can be,
01:16:54 you can create a good driving experience within the rules.
01:17:00 That’s, I mean, you’re the first person to tell me this.
01:17:03 So it’s, it’s kind of interesting. I need to think
01:17:04 about this, but that’s exactly what it felt like with Waymo.
01:17:07 I kind of had this intuition. Maybe it’s the Russian thing.
01:17:10 I don’t know that you have to break the rules in life to get
01:17:13 anywhere, but maybe, maybe it’s possible that that’s not the
01:17:19 case in driving. I have to think about that, but it
01:17:23 certainly felt that way on the streets of Phoenix when I was
01:17:25 there in Waymo, that, that, that that was a very pleasant
01:17:29 experience and it wasn’t frustrating in that like, come
01:17:32 on, move already kind of feeling. It wasn’t, that wasn’t
01:17:35 there. Yeah. I mean, that’s what, that’s what we’re going
01:17:37 after. I don’t think you have to pick one. I think truly good
01:17:41 driving. It gives you both efficiency, a certainness, but
01:17:45 also comfort and predictability and safety. And, you know, it’s,
01:17:49 that’s what fundamental improvements in the core
01:17:54 capabilities truly unlock. And you can kind of think of it as,
01:17:59 you know, a precision and recall trade off. You have certain
01:18:01 capabilities of your model. And then it’s very easy when, you
01:18:04 know, you have some curve of precision and recall, you can
01:18:06 move things around and can choose your operating point and
01:18:08 your training of precision versus recall, false positives
01:18:10 versus false negatives. Right. But then, and you know, you can
01:18:14 tune things on that curve and be kind of more cautious or more
01:18:16 aggressive, but then aggressive is bad or, you know, cautious is
01:18:19 bad, but true capabilities come from actually moving the whole
01:18:22 curve up. And then you are kind of on a very different plane of
01:18:28 those trade offs. And that, that’s what we’re trying to do
01:18:31 here is to move the whole curve up. Before I forget, let’s talk
01:18:34 about trucks a little bit. So I also got a chance to check out
01:18:39 some of the Waymo trucks. I’m not sure if we want to go too
01:18:44 much into that space, but it’s a fascinating one. So maybe we
01:18:47 can mention at least briefly, you know, Waymo is also now
01:18:51 doing autonomous trucking and how different like
01:18:56 philosophically and technically is that whole space of
01:18:58 problems. It’s one of our two big products and you know,
01:19:06 commercial applications of our driver, right? Right. Hailing
01:19:09 and deliveries. You know, we have Waymo One and Waymo Via
01:19:12 moving people and moving goods. You know, trucking is an
01:19:16 example of moving goods. We’ve been working on trucking since
01:19:21 2017. It is a very interesting space. And your question of
01:19:31 how different is it? It has this really nice property that
01:19:35 the first order challenges, like the science, the hard
01:19:38 engineering, whether it’s, you know, hardware or, you know,
01:19:42 onboard software or off board software, all of the, you know,
01:19:45 systems that you build for, you know, training your ML models
01:19:48 for, you know, evaluating your time system. Like those
01:19:51 fundamentals carry over. Like the true challenges of, you
01:19:56 know, driving perception, semantic understanding,
01:20:00 prediction, decision making, planning, evaluation, the
01:20:04 simulator, ML infrastructure, those carry over. Like the data
01:20:08 and the application and kind of the domains might be
01:20:12 different, but the most difficult problems, all of that
01:20:16 carries over between the domains. So that’s very nice.
01:20:19 So that’s how we approach it. We’re kind of build investing
01:20:22 in the core, the technical core. And then there’s
01:20:26 specialization of that core technology to different
01:20:30 product lines, to different commercial applications. So on
01:20:34 just to tease it apart a little bit on trucks. So starting with
01:20:38 the hardware, the configuration of the sensors is different.
01:20:42 They’re different physically, geometrically, you know, different
01:20:46 vehicles. So for example, we have two of our main laser on
01:20:50 the trucks on both sides so that we have, you know, not have the
01:20:54 blind spots. Whereas on the JLR eye pace, we have, you know, one
01:20:59 of it sitting at the very top, but the actual sensors are
01:21:02 almost the same. Now we’re largely the same. So all of the
01:21:06 investment that over the years we’ve put into building our
01:21:11 custom lighters, custom radars, pulling the whole system
01:21:13 together, that carries over very nicely. Then, you know, on the
01:21:16 perception side, the like the fundamental challenges of
01:21:20 seeing, understanding the world, whether it’s, you know, object
01:21:22 detection, classification, you know, tracking, semantic
01:21:25 understanding, all that carries over. You know, yes, there’s
01:21:28 some specialization when you’re driving on freeways, you know,
01:21:31 range becomes more important. The domain is a little bit
01:21:33 different. But again, the fundamentals carry over very,
01:21:36 very nicely. Same, and you guess you get into prediction or
01:21:41 decision making, right, the fundamentals of what it takes to
01:21:45 predict what other people are going to do to find the long
01:21:49 tail to improve your system in that long tail of behavior
01:21:53 prediction and response that carries over right and so on and
01:21:56 so on. So I mean, that’s pretty exciting. By the way, does
01:22:00 Waymo via include using the smaller vehicles for
01:22:05 transportation of goods? That’s an interesting distinction. So
01:22:07 I would say there’s three interesting modes of operation.
01:22:13 So one is moving humans, one is moving goods, and one is like
01:22:16 moving nothing, zero occupancy, meaning like you’re going to
01:22:21 the destination, your empty vehicle. I mean, it’s the third
01:22:27 is the less of it. If that’s the entirety of it, it’s the less,
01:22:29 you know, exciting from the commercial perspective.
01:22:31 Well, I mean, in terms of like, if you think about what’s
01:22:38 inside a vehicle as it’s moving, because it does, you
01:22:42 know, some significant fraction of the vehicle’s movement has
01:22:45 to be empty. I mean, it’s kind of fascinating. Maybe just on
01:22:50 that small point, is there different control and like
01:22:57 policies that are applied for zero occupancy vehicle? So
01:23:01 vehicle with nothing in it, or is it just move as if there is
01:23:04 a person inside? What was with some subtle differences?
01:23:09 As a first order approximation, there are no differences. And
01:23:13 if you think about, you know, safety and comfort and quality
01:23:17 of driving, only part of it has to do with the people or the
01:23:26 goods inside of the vehicle. But you don’t want to be, you
01:23:29 know, you want to drive smoothly, as we discussed, not
01:23:31 for the purely for the benefit of whatever you have inside the
01:23:34 car, right? It’s also for the benefit of the people outside
01:23:38 kind of fitting naturally and predictably into that whole
01:23:41 environment, right? So, you know, yes, there are some
01:23:43 second order things you can do, you can change your route, and
01:23:47 you optimize maybe kind of your fleet, things at the fleet
01:23:50 scale. And you would take into account whether some of your
01:23:54 you know, some of your cars are actually, you know, serving a
01:23:58 useful trip, whether with people or with goods, whereas, you
01:24:01 know, other cars are, you know, driving completely empty to that
01:24:05 next valuable trip that they’re going to provide. But that those
01:24:09 are mostly second order effects. Okay, cool. So Phoenix
01:24:14 is, is an incredible place. And what you’ve announced in
01:24:18 Phoenix is, it’s kind of amazing. But, you know, that’s
01:24:23 just like one city. How do you take over the world? I mean,
01:24:30 I’m asking for a friend. One step at a time.
01:24:35 Is that a cartoon pinky in the brain? Yeah. Okay. But, you
01:24:40 know, gradually is a true answer. So I think the heart of
01:24:44 your question is, can you ask a better question than I asked?
01:24:48 You’re asking a great question. Answer that one. I’m just
01:24:52 gonna, you know, phrase it in the terms that I want to
01:24:56 answer. Exactly right. Brilliant. Please. You know,
01:25:01 where are we today? And, you know, what happens next? And
01:25:04 what does it take to go beyond Phoenix? And what does it
01:25:08 take to get this technology to more places and more people
01:25:13 around the world, right? So our next big area of focus is
01:25:23 exactly that. Larger scale commercialization and just,
01:25:26 you know, scaling up. If I think about, you know, the
01:25:35 main, and, you know, Phoenix gives us that platform and
01:25:39 gives us that foundation of upon which we can build. And
01:25:44 it’s, there are few really challenging aspects of this
01:25:51 whole problem that you have to pull together in order to build
01:25:56 the technology in order to deploy it into the field to go
01:26:03 from a driverless car to a fleet of cars that are providing a
01:26:09 service, and then all the way to commercialization. So, and
01:26:14 then, you know, this is what we have in Phoenix. We’ve taken
01:26:15 the technology from a proof point to an actual deployment
01:26:21 and have taken our driver from, you know, one car to a fleet
01:26:25 that can provide a service. Beyond that, if I think about
01:26:29 what it will take to scale up and, you know, deploy in, you
01:26:35 know, more places with more customers, I tend to think about
01:26:41 three main dimensions, three main axes of scale. One is the
01:26:48 core technology, you know, the hardware and software core
01:26:51 capabilities of our driver. The second dimension is
01:26:56 evaluation and deployment. And the third one is the, you know,
01:27:01 product, commercial, and operational excellence. So you
01:27:06 can talk a bit about where we are along, you know, each one of
01:27:09 those three dimensions about where we are today and, you
01:27:11 know, what has, what will happen next. On, you know, the core
01:27:16 technology, you know, the hardware and software, you
01:27:19 know, together comprise a driver, we, you know, obviously
01:27:25 have that foundation that is providing fully driverless
01:27:30 trips to our customers as we speak, in fact. And we’ve
01:27:34 learned a tremendous amount from that. So now what we’re
01:27:39 doing is we are incorporating all those lessons into some
01:27:44 pretty fundamental improvements in our core technology, both on
01:27:47 the hardware side and on the software side to build a more
01:27:51 general, more robust solution that then will enable us to
01:27:54 massively scale beyond Phoenix. So on the hardware side, all of
01:28:00 those lessons are now incorporated into this fifth
01:28:05 generation hardware platform that is, you know, being
01:28:09 deployed right now. And that’s the platform, the fourth
01:28:13 generation, the thing that we have right now driving in
01:28:14 Phoenix, it’s good enough to operate fully driverlessly,
01:28:18 you know, night and day, you know, various speeds and
01:28:21 various conditions, but the fifth generation is the platform
01:28:25 upon which we want to go to massive scale. We, in turn,
01:28:30 we’ve really made qualitative improvements in terms of the
01:28:32 capability of the system, the simplicity of the architecture,
01:28:35 the reliability of the redundancy. It is designed to be
01:28:39 manufacturable at very large scale and, you know, provides
01:28:42 the right unit economics. So that’s the next big step for us
01:28:46 on the hardware side. That’s already there for scale,
01:28:49 the version five. That’s right. And is that coincidence or
01:28:53 should we look into a conspiracy theory that it’s the
01:28:55 same version as the pixel phone? Is that what’s the
01:28:59 hardware? They neither confirm nor deny. All right, cool. So,
01:29:04 sorry. So that’s the, okay, that’s that axis. What else?
01:29:08 So similarly, you know, hardware is a very discreet
01:29:11 jump, but, you know, similar to how we’re making that change
01:29:14 from the fourth generation hardware to the fifth, we’re
01:29:16 making similar improvements on the software side to make it
01:29:19 more, you know, robust and more general and allow us to kind of
01:29:22 quickly scale beyond Phoenix. So that’s the first dimension of
01:29:25 core technology. The second dimension is evaluation and
01:29:27 deployment. How do you measure your system? How do you
01:29:34 evaluate it? How do you build a release and deployment process
01:29:37 where, you know, with confidence, you can, you know,
01:29:40 regularly release new versions of your driver into a fleet?
01:29:45 How do you get good at it so that it is not, you know, a
01:29:49 huge tax on your researchers and engineers that, you know, so
01:29:52 you can, how do you build all these, you know, processes, the
01:29:55 frameworks, the simulation, the evaluation, the data science,
01:29:58 the validation, so that, you know, people can focus on
01:30:01 improving the system and kind of the releases just go out the
01:30:04 door and get deployed across the fleet. So we’ve gotten really
01:30:07 good at that in Phoenix. That’s been a tremendously difficult
01:30:11 problem, but that’s what we have in Phoenix right now that gives
01:30:15 us that foundation. And now we’re working on kind of
01:30:17 incorporating all the lessons that we’ve learned to make it
01:30:20 more efficient, to go to new places, you know, and scale up
01:30:22 and just kind of, you know, stamp things out. So that’s that
01:30:25 second dimension of evaluation and deployment. And the third
01:30:28 dimension is product, commercial, and operational
01:30:33 excellence, right? And again, Phoenix there is providing an
01:30:38 incredibly valuable platform. You know, that’s why we’re doing
01:30:40 things end to end in Phoenix. We’re learning, as you know, we
01:30:43 discussed a little earlier today, tremendous amount of
01:30:47 really valuable lessons from our users getting really
01:30:50 incredible feedback. And we’ll continue to iterate on that and
01:30:54 incorporate all those lessons into making our product, you
01:30:59 know, even better and more convenient for our users.
01:31:01 So you’re converting this whole process in Phoenix into
01:31:06 something that could be copy and pasted elsewhere. So like,
01:31:11 perhaps you didn’t think of it that way when you were doing
01:31:13 the experimentation in Phoenix, but so how long did you
01:31:17 basically, and you can correct me, but you’ve, I mean, it’s
01:31:22 still early days, but you’ve taken the full journey in
01:31:24 Phoenix, right? As you were saying of like what it takes to
01:31:29 basically automate. I mean, it’s not the entirety of Phoenix,
01:31:31 right? But I imagine it can encompass the entirety of
01:31:36 Phoenix. That’s some near term date, but that’s not even
01:31:41 perhaps important. Like as long as it’s a large enough
01:31:43 geographic area. So what, how copy pastable is that process
01:31:51 currently and how like, you know, like when you copy and
01:31:58 paste in Google docs, I think now in, or in word, you can
01:32:05 like apply source formatting or apply destination formatting.
01:32:09 So how, when you copy and paste the Phoenix into like, say
01:32:14 Boston, how do you apply the destination formatting? Like
01:32:20 how much of the core of the entire process of bringing an
01:32:25 actual public transportation, autonomous transportation
01:32:30 service to a city is there in Phoenix that you understand
01:32:35 enough to copy and paste into Boston or wherever? So we’re
01:32:39 not quite there yet. We’re not at a point where we’re kind of
01:32:41 massively copy and pasting all over the place. But Phoenix,
01:32:47 what we did in Phoenix, and we very intentionally have chosen
01:32:50 Phoenix as our first full deployment area, you know,
01:32:56 exactly for that reason to kind of tease the problem apart,
01:32:59 look at each dimension and focus on the fundamentals of
01:33:03 complexity and de risking those dimensions, and then bringing
01:33:06 the entire thing together to get all the way and force
01:33:09 ourselves to learn all those hard lessons on technology,
01:33:12 hardware and software, on the evaluation deployment, on
01:33:15 operating a service, operating a business using actually
01:33:20 serving our customers all the way so that we’re fully
01:33:22 informed about the most difficult, most important
01:33:27 challenges to get us to that next step of massive copy and
01:33:31 pasting as you said. And that’s what we’re doing right now.
01:33:38 We’re incorporating all those things that we learned into
01:33:41 that next system that then will allow us to kind of copy and
01:33:44 paste all over the place and to massively scale to, you know,
01:33:47 more users and more locations. I mean, you know, just talk a
01:33:50 little bit about, you know, what does that mean along those
01:33:52 different dimensions? So on the hardware side, for example,
01:33:55 again, it’s that switch from the fourth to the fifth
01:33:57 generation. And the fifth generation is designed to kind
01:34:00 of have that property. Can you say what other cities you’re
01:34:04 thinking about? Like, I’m thinking about, sorry, we’re in
01:34:09 San Francisco now. I thought I want to move to San Francisco,
01:34:12 but I’m thinking about moving to Austin. I don’t know why
01:34:16 people are not being very nice about San Francisco currently,
01:34:19 but maybe it’s a small, maybe it’s in vogue right now.
01:34:23 But Austin seems, I visited there and it was, I was in a
01:34:28 Walmart. It’s funny, these moments like turn your life.
01:34:32 There’s this very nice woman with kind eyes, just like stopped
01:34:38 and said, he looks so handsome in that tie, honey, to me. This
01:34:44 has never happened to me in my life, but just the sweetness of
01:34:47 this woman is something I’ve never experienced, certainly on
01:34:49 the streets of Boston, but even in San Francisco where people
01:34:53 wouldn’t, that’s just not how they speak or think. I don’t
01:34:57 know. There’s a warmth to Austin that love. And since
01:35:00 Waymo does have a little bit of a history there, is that a
01:35:04 possibility? Is this your version of asking the question
01:35:07 of like, you know, Dimitri, I know you can’t share your
01:35:09 commercial and deployment roadmap, but I’m thinking about
01:35:12 moving to San Francisco, Austin, like, you know, blink twice if
01:35:16 you think I should move to it. That’s true. That’s true. You
01:35:19 got me. You know, we’ve been testing all over the place. I
01:35:23 think we’ve been testing more than 25 cities. We drive
01:35:26 in San Francisco. We drive in, you know, Michigan for snow.
01:35:31 We are doing significant amount of testing in the Bay Area,
01:35:34 including San Francisco, which is not like, because we’re
01:35:37 talking about the very different thing, which is like a
01:35:40 full on large geographic area, public service. You can’t share
01:35:46 and you, okay. What about Moscow? When is that happening?
01:35:54 Take on Yandex. I’m not paying attention to those folks.
01:35:58 They’re doing, you know, there’s a lot of fun. I mean,
01:36:02 maybe as a way of a question, you didn’t speak to sort of like
01:36:10 policy or like, is there tricky things with government and so
01:36:15 on? Like, is there other friction that you’ve
01:36:20 encountered except sort of technological friction of
01:36:25 solving this very difficult problem? Is there other stuff
01:36:28 that you have to overcome when deploying a public service in
01:36:33 a city? That’s interesting. It’s very important. So we
01:36:38 put significant effort in creating those partnerships and
01:36:44 you know, those relationships with governments at all levels,
01:36:48 local governments, municipalities, state level,
01:36:50 federal level. We’ve been engaged in very deep
01:36:53 conversations from the earliest days of our projects.
01:36:57 Whenever at all of these levels, whenever we go
01:37:01 to test or operate in a new area, we always lead
01:37:07 with a conversation with the local officials.
01:37:10 But the result of that investment is that no,
01:37:13 it’s not challenges we have to overcome, but it is very
01:37:16 important that we continue to have this conversation.
01:37:19 Oh, yeah. I love politicians too. Okay, so Mr. Elon Musk said that
01:37:27 LiDAR is a crutch. What are your thoughts?
01:37:32 I wouldn’t characterize it exactly that way. I know I think LiDAR is
01:37:36 very important. It is a key sensor that we use just like
01:37:42 other modalities, right? As we discussed, our cars use cameras, LiDAR
01:37:46 and radars. They are all very important. They are
01:37:52 at the kind of the physical level. They are very different. They have very
01:37:57 different, you know, physical characteristics.
01:38:00 Cameras are passive. LiDARs and radars are active.
01:38:03 Use different wavelengths. So that means they complement each other
01:38:07 very nicely and together combined, they can be used to
01:38:14 build a much safer and much more capable system.
01:38:20 So, you know, to me it’s more of a question,
01:38:25 you know, why the heck would you handicap yourself and not use one
01:38:28 or more of those sensing modalities when they, you know, undoubtedly just make your
01:38:32 system more capable and safer. Now,
01:38:39 it, you know, what might make sense for one product or
01:38:45 one business might not make sense for another one.
01:38:48 So if you’re talking about driver assist technologies, you make certain design
01:38:51 decisions and you make certain trade offs and make different ones if you are
01:38:55 building a driver that you deploy in fully driverless
01:38:59 vehicles. And, you know, in LiDAR specifically, when this question comes up,
01:39:04 I, you know, typically the criticisms that I hear or, you know, the
01:39:11 counterpoints is that cost and aesthetics.
01:39:16 And I don’t find either of those, honestly, very compelling.
01:39:20 So on the cost side, there’s nothing fundamentally prohibitive
01:39:24 about, you know, the cost of LiDARs. You know, radars used to be very expensive
01:39:28 before people started, you know, before people made certain advances in
01:39:32 technology and, you know, started to manufacture them at massive scale and
01:39:35 deploy them in vehicles, right? You know, similar with LiDARs. And this is
01:39:39 where the LiDARs that we have on our cars, especially the fifth generation,
01:39:43 you know, we’ve been able to make some pretty qualitative discontinuous
01:39:48 jumps in terms of the fundamental technology that allow us to
01:39:51 manufacture those things at very significant scale and at a fraction
01:39:56 of the cost of both our previous generation
01:40:00 as well as a fraction of the cost of, you know, what might be available
01:40:03 on the market, you know, off the shelf right now. And, you know, that improvement
01:40:07 will continue. So I think, you know, cost is not a
01:40:10 real issue. Second one is, you know, aesthetics.
01:40:14 You know, I don’t think that’s, you know, a real issue either.
01:40:18 Beauty is in the eye of the beholder. Yeah. You can make LiDAR sexy again.
01:40:22 I think you’re exactly right. I think it is sexy. Like, honestly, I think form
01:40:25 all of function. Well, okay. You know, I was actually, somebody brought this up to
01:40:30 me. I mean, all forms of LiDAR, even
01:40:34 like the ones that are like big, you can make
01:40:37 look, I mean, you can make look beautiful.
01:40:40 There’s no sense in which you can’t integrate it into design.
01:40:44 Like, there’s all kinds of awesome designs. I don’t think
01:40:47 small and humble is beautiful. It could be
01:40:51 like, you know, brutalism or like, it could be
01:40:55 like harsh corners. I mean, like I said, like hot rods. Like, I don’t like, I don’t
01:40:59 necessarily like, like, oh man, I’m going to start so much
01:41:02 controversy with this. I don’t like Porsches. Okay.
01:41:07 The Porsche 911, like everyone says it’s the most beautiful.
01:41:10 No, no. It’s like, it’s like a baby car. It doesn’t make any sense.
01:41:15 But everyone, it’s beauty is in the eye of the beholder. You’re already looking at
01:41:18 me like, what is this kid talking about? I’m happy to talk about. You’re digging your
01:41:24 own hole. The form and function and my take on the
01:41:27 beauty of the hardware that we put on our vehicles,
01:41:30 you know, I will not comment on your Porsche monologue.
01:41:34 Okay. All right. So, but aesthetics, fine. But there’s an underlying, like,
01:41:39 philosophical question behind the kind of lighter question is
01:41:43 like, how much of the problem can be solved
01:41:48 with computer vision, with machine learning?
01:41:51 So I think without sort of disagreements and so on,
01:41:58 it’s nice to put it on the spectrum because Waymo is doing a lot of machine
01:42:03 learning as well. It’s interesting to think how much of
01:42:06 driving, if we look at five years, 10 years, 50 years down the road,
01:42:11 what can be learned in almost more and more and more
01:42:15 end to end way. If we look at what Tesla is doing
01:42:19 with, as a machine learning problem, they’re doing a multitask learning
01:42:24 thing where it’s just, they break up driving into a bunch of learning tasks
01:42:27 and they have one single neural network and they’re just collecting huge amounts
01:42:30 of data that’s training that. I’ve recently hung out with George
01:42:33 Hotz. I don’t know if you know George.
01:42:37 I love him so much. He’s just an entertaining human being.
01:42:41 We were off mic talking about Hunter S. Thompson. He’s the Hunter S. Thompson
01:42:45 of autonomous driving. Okay. So he, I didn’t realize this with comma
01:42:49 AI, but they’re like really trying to end to end.
01:42:53 They’re the machine, like looking at the machine learning problem, they’re
01:42:58 really not doing multitask learning, but it’s
01:43:01 computing the drivable area as a machine learning task
01:43:05 and hoping that like down the line, this level two system, this driver
01:43:11 assistance will eventually lead to
01:43:15 allowing you to have a fully autonomous vehicle. Okay. There’s an underlying
01:43:19 deep philosophical question there, technical question
01:43:22 of how much of driving can be learned. So LiDAR is an effective tool today
01:43:29 for actually deploying a successful service in Phoenix, right? That’s safe,
01:43:33 that’s reliable, et cetera, et cetera. But the question,
01:43:39 and I’m not saying you can’t do machine learning on LiDAR, but the question is
01:43:43 that like how much of driving can be learned eventually.
01:43:47 Can we do fully autonomous? That’s learned.
01:43:49 Yeah. You know, learning is all over the place
01:43:53 and plays a key role in every part of our system.
01:43:56 As you said, I would, you know, decouple the sensing modalities
01:44:01 from the, you know, ML and the software parts of it.
01:44:05 LiDAR, radar, cameras, like it’s all machine learning.
01:44:09 All of the object detection classification, of course, like that’s
01:44:12 what, you know, these modern deep nets and count nets are very
01:44:15 good at. You feed them raw data, massive amounts of raw data,
01:44:19 and that’s actually what our custom build LiDARs and radars are really good
01:44:23 at. And radars, they don’t just give you point
01:44:25 estimates of, you know, objects in space, they give you raw,
01:44:28 like, physical observations. And then you take all of that raw information,
01:44:31 you know, there’s colors of the pixels, whether it’s, you know, LiDARs returns
01:44:34 and some auxiliary information. It’s not just distance,
01:44:36 right? And, you know, angle and distance is much richer information that you get
01:44:39 from those returns, plus really rich information from the
01:44:42 radars. You fuse it all together and you feed it into those massive
01:44:45 ML models that then, you know, lead to the best results in terms of, you
01:44:51 know, object detection, classification, state estimation.
01:44:55 So there’s a side to interop, but there is a fusion. I mean, that’s something
01:44:59 that people didn’t do for a very long time,
01:45:01 which is like at the sensor fusion level, I guess,
01:45:04 like early on fusing the information together, whether
01:45:07 so that the the sensory information that the vehicle receives from the different
01:45:11 modalities or even from different cameras is
01:45:15 combined before it is fed into the machine learning models.
01:45:19 Yeah, so I think this is one of the trends you’re seeing more of that you
01:45:21 mentioned end to end. There’s different interpretation of end to end. There is
01:45:24 kind of the purest interpretation of I’m going to
01:45:27 like have one model that goes from raw sensor data to like,
01:45:32 you know, steering torque and, you know, gas breaks. That, you know,
01:45:35 that’s too much. I don’t think that’s the right way to do it.
01:45:37 There’s more, you know, smaller versions of end to end
01:45:40 where you’re kind of doing more end to end learning or core training or
01:45:45 depropagation of kind of signals back and forth across
01:45:48 the different stages of your system. There’s, you know, really good ways it
01:45:51 gets into some fairly complex design choices where on one
01:45:55 hand you want modularity and decomposability,
01:45:57 decomposability of your system. But on the other hand,
01:46:01 you don’t want to create interfaces that are too narrow or too brittle
01:46:05 to engineered where you’re giving up on the generality of the solution or you’re
01:46:08 unable to properly propagate signal, you know, reach signal forward and losses
01:46:12 and, you know, back so you can optimize the whole system jointly.
01:46:17 So I would decouple and I guess what you’re seeing in terms of the fusion
01:46:21 of the sensing data from different modalities as well as kind of fusion
01:46:25 at in the temporal level going more from, you know, frame by frame
01:46:30 where, you know, you would have one net that would do frame by frame detection
01:46:32 and camera and then, you know, something that does frame by frame and
01:46:35 lighter and then radar and then you fuse it, you know, in a weaker engineered way
01:46:39 later. Like the field over the last, you know,
01:46:41 decade has been evolving in more kind of joint fusion, more end to end models that
01:46:45 are, you know, solving some of these tasks, you know, jointly and there’s
01:46:48 tremendous power in that and, you know, that’s the
01:46:50 progression that kind of our technology, our stack has been on as well.
01:46:54 Now to your, you know, that so I would decouple the kind of sensing and how
01:46:57 that information is fused from the role of ML and the entire stack.
01:47:01 And, you know, I guess it’s, there’s trade offs and, you know, modularity and
01:47:06 how do you inject inductive bias into your system?
01:47:11 All right, this is, there’s tremendous power
01:47:15 in being able to do that. So, you know, we have, there’s no
01:47:19 part of our system that is not heavily, that does not heavily, you know, leverage
01:47:25 data driven development or state of the art ML.
01:47:29 But there’s mapping, there’s a simulator, there’s perception, you know, object
01:47:33 level, you know, perception, whether it’s
01:47:34 semantic understanding, prediction, decision making, you know, so forth and
01:47:38 so on.
01:47:42 It’s, you know, of course, object detection and classification, like you’re
01:47:45 finding pedestrians and cars and cyclists and, you know, cones and signs
01:47:48 and vegetation and being very good at estimating
01:47:51 kind of detection, classification, and state estimation. There’s just stable
01:47:54 stakes, like that’s step zero of this whole stack. You can be
01:47:57 incredibly good at that, whether you use cameras or light as a
01:48:00 radar, but that’s just, you know, that’s stable stakes, that’s just step zero.
01:48:03 Beyond that, you get into the really interesting challenges of semantic
01:48:06 understanding at the perception level, you get into scene level reasoning, you
01:48:10 get into very deep problems that have to do with prediction and joint
01:48:13 prediction and interaction, so the interaction
01:48:16 between all the actors in the environment, pedestrians, cyclists, other
01:48:19 cars, and you get into decision making, right? So, how do you build a lot of
01:48:22 systems? So, we leverage ML very heavily in all of
01:48:26 these components. I do believe that the best results you
01:48:30 achieve by kind of using a hybrid approach and
01:48:33 having different types of ML, having
01:48:38 different models with different degrees of inductive bias
01:48:41 that you can have, and combining kind of model,
01:48:45 you know, free approaches with some model based approaches and some
01:48:49 rule based, physics based systems. So, you know, one example I can give
01:48:54 you is traffic lights. There’s a problem of the detection of
01:48:58 traffic light state, and obviously that’s a great problem for, you know, computer
01:49:02 vision confidence, or, you know, that’s their bread and
01:49:05 butter, right? That’s how you build that. But then the
01:49:08 interpretation of, you know, of a traffic light, that you’re
01:49:11 gonna need to learn that, right? You don’t need to build some,
01:49:15 you know, complex ML model that, you know, infers
01:49:18 with some, you know, precision and recall that red means stop.
01:49:22 Like, it’s a very clear engineered signal
01:49:25 with very clear semantics, right? So you want to induce that bias, like how you
01:49:29 induce that bias, and that whether, you know, it’s a
01:49:31 constraint or a cost, you know, function in your stack, but like
01:49:36 it is important to be able to inject that, like, clear semantic
01:49:40 signal into your stack. And, you know, that’s what we do.
01:49:44 And, but then the question of, like, and that’s when you
01:49:47 apply it to yourself, when you are making decisions whether you want to stop
01:49:50 for a red light, you know, or not.
01:49:54 But if you think about how other people treat traffic lights,
01:49:57 we’re back to the ML version of that. You know they’re supposed to stop
01:50:01 for a red light, but that doesn’t mean they will.
01:50:02 So then you’re back in the, like, very heavy
01:50:07 ML domain where you’re picking up on, like, very subtle cues about,
01:50:11 you know, they have to do with the behavior of objects, pedestrians, cyclists,
01:50:15 cars, and the whole, you know, entire configuration of the scene
01:50:19 that allow you to make accurate predictions on whether they will, in
01:50:22 fact, stop or run a red light. So it sounds like already for Waymo,
01:50:27 like, machine learning is a huge part of the stack.
01:50:29 So it’s a huge part of, like, not just, so obviously the first, the level
01:50:36 zero, or whatever you said, which is, like,
01:50:38 just the object detection of things that, you know, with no other machine learning
01:50:42 can do, but also starting to do prediction behavior and so on to
01:50:46 model the, what other, what the other parties in the
01:50:49 scene, entities in the scene are going to do.
01:50:51 So machine learning is more and more playing a role in that
01:50:55 as well. Of course. Oh, absolutely. I think we’ve been
01:50:59 going back to the, you know, earliest days, like, you know, DARPA,
01:51:02 the DARPA Grand Challenge, our team was leveraging, you know, machine
01:51:05 learning. It was, like, pre, you know, ImageNet, and it was a very
01:51:08 different type of ML, but, and I think actually it was before
01:51:11 my time, but the Stanford team during the Grand Challenge had a very
01:51:15 interesting machine learned system that would, you know, use
01:51:18 LiDAR and camera. We’ve been driving in the
01:51:21 desert, and it, we had built the model where it would kind of
01:51:26 extend the range of free space reasoning. We get a
01:51:29 clear signal from LiDAR, and then it had a model that said, hey, like,
01:51:33 this stuff on camera kind of sort of looks like this stuff in LiDAR, and I
01:51:35 know this stuff that I’m seeing in LiDAR, I’m very confident it’s free space,
01:51:38 so let me extend that free space zone into the camera range that would allow
01:51:43 the vehicle to drive faster. And then we’ve been building on top of
01:51:45 that and kind of staying and pushing the state of the art in ML,
01:51:48 in all kinds of different ML over the years. And in fact,
01:51:52 from the early days, I think, you know, 2010 is probably the year
01:51:56 where Google, maybe 2011 probably, got pretty heavily involved in
01:52:03 machine learning, kind of deep nuts, and at that time it was probably the only
01:52:07 company that was very heavily investing in kind of state of the art ML and
01:52:11 self driving cars. And they go hand in hand.
01:52:16 And we’ve been on that journey ever since. We’re doing, pushing
01:52:19 a lot of these areas in terms of research at Waymo, and we
01:52:24 collaborate very heavily with the researchers in
01:52:26 Alphabet, and all kinds of ML, supervised ML,
01:52:30 unsupervised ML, published some
01:52:34 interesting research papers in the space,
01:52:37 especially recently. It’s just a super active learning as well.
01:52:41 Yeah, so super, super active. Of course, there’s, you know, kind of the more
01:52:45 mature stuff, like, you know, ConvNets for, you know, object detection.
01:52:48 But there’s some really interesting, really active work that’s happening
01:52:52 in kind of more, you know, in bigger models and, you know,
01:52:58 models that have more structure to them,
01:53:02 you know, not just, you know, large bitmaps and reason about temporal sequences.
01:53:06 And some of the interesting breakthroughs that you’ve, you know, we’ve seen
01:53:10 in language models, right? You know, transformers,
01:53:14 you know, GPT3 inference. There’s some really interesting applications of some
01:53:19 of the core breakthroughs to those problems
01:53:21 of, you know, behavior prediction, as well as, you know, decision making and
01:53:24 planning, right? You can think about it, kind of the the behavior,
01:53:27 how, you know, the path, the trajectories, the how people drive.
01:53:31 They have kind of a share, a lot of the fundamental structure,
01:53:34 you know, this problem. There’s, you know, sequential,
01:53:38 you know, nature. There’s a lot of structure in this representation.
01:53:41 There is a strong locality, kind of like in sentences, you know, words that follow
01:53:45 each other. They’re strongly connected, but there’s
01:53:48 also kind of larger context that doesn’t have that locality, and you also see that
01:53:51 in driving, right? What, you know, is happening in the scene
01:53:53 as a whole has very strong implications on,
01:53:57 you know, the kind of the next step in that sequence where
01:54:00 whether you’re, you know, predicting what other people are going to do, whether
01:54:03 you’re making your own decisions, or whether in the simulator you’re
01:54:07 building generative models of, you know, humans walking, cyclists
01:54:10 riding, and other cars driving. That’s all really fascinating, like how
01:54:14 it’s fascinating to think that transformer models and all this,
01:54:17 all the breakthroughs in language and NLP that might be applicable to like
01:54:21 driving at the higher level, at the behavioral level, that’s kind of
01:54:24 fascinating. Let me ask about pesky little creatures
01:54:27 called pedestrians and cyclists. They seem, so humans are a problem. If we
01:54:32 can get rid of them, I would. But unfortunately, they’re all sort of
01:54:36 a source of joy and love and beauty, so let’s keep them around.
01:54:39 They’re also our customers. For your perspective, yes, yes,
01:54:43 for sure. They’re a source of money, very good.
01:54:46 But I don’t even know where I was going. Oh yes,
01:54:52 pedestrians and cyclists, you know,
01:54:57 they’re a fascinating injection into the system of
01:55:00 uncertainty of like a game theoretic dance of what to do. And also
01:55:09 they have perceptions of their own, and they can tweet
01:55:13 about your product, so you don’t want to run them over
01:55:17 from that perspective. I mean, I don’t know, I’m joking a lot, but
01:55:21 I think in seriousness, like, you know, pedestrians are a complicated
01:55:27 computer vision problem, a complicated behavioral problem. Is there something
01:55:31 interesting you could say about what you’ve learned
01:55:34 from a machine learning perspective, from also an autonomous vehicle,
01:55:38 and a product perspective about just interacting with the humans in this
01:55:42 world? Yeah, just to state on record, we care
01:55:45 deeply about the safety of pedestrians, you know, even the ones that don’t have
01:55:48 Twitter accounts. Thank you. All right, cool.
01:55:52 Not me. But yes, I’m glad, I’m glad somebody does.
01:55:57 Okay. But you know, in all seriousness, safety
01:56:01 of vulnerable road users, pedestrians or cyclists, is one of our
01:56:07 highest priorities. We do a tremendous amount of testing
01:56:12 and validation, and put a very significant emphasis
01:56:16 on, you know, the capabilities of our systems that have to do with safety
01:56:20 around those unprotected vulnerable road users.
01:56:23 You know, cars, just, you know, discussed earlier in Phoenix, we have completely
01:56:27 empty cars, completely driverless cars, you know, driving in this very large area,
01:56:31 and you know, some people use them to, you know, go to school, so they’ll drive
01:56:35 through school zones, right? So, kids are kind of the very special
01:56:39 class of those vulnerable user road users, right? You want to be,
01:56:42 you know, super, super safe, and super, super cautious around those. So, we take
01:56:45 it very, very, very seriously. And you know, what does it take to
01:56:50 be good at it? You know,
01:56:55 an incredible amount of performance across your whole stack. You know,
01:57:02 starts with hardware, and again, you want to use all
01:57:05 sensing modalities available to you. Imagine driving on a residential road
01:57:09 at night, and kind of making a turn, and you don’t have, you know, headlights
01:57:13 covering some part of the space, and like, you know, a kid might
01:57:16 run out. And you know, lighters are amazing at that. They
01:57:20 see just as well in complete darkness as they do during the day, right? So, just
01:57:24 again, it gives you that extra,
01:57:27 you know, margin in terms of, you know, capability, and performance, and safety,
01:57:32 and quality. And in fact, we oftentimes, in these
01:57:35 kinds of situations, we have our system detect something,
01:57:38 in some cases even earlier than our trained operators in the car might do,
01:57:42 right? Especially, you know, in conditions like, you know, very dark nights.
01:57:46 So, starts with sensing, then, you know, perception
01:57:50 has to be incredibly good. And you have to be very, very good
01:57:54 at kind of detecting pedestrians in all kinds of situations, and all kinds
01:58:00 of environments, including, you know, people in weird poses,
01:58:03 people kind of running around, and you know, being partially occluded.
01:58:09 So, you know, that’s step number one, right?
01:58:13 Then, you have to have in very high accuracy,
01:58:17 and very low latency, in terms of your reactions
01:58:21 to, you know, what, you know, these actors might do, right? And we’ve put a
01:58:27 tremendous amount of engineering, and tremendous amount of validation, in to
01:58:30 make sure our system performs properly. And, you know, oftentimes, it
01:58:35 does require a very strong reaction to do the safe thing. And, you know, we
01:58:38 actually see a lot of cases like that. That’s the long tail of really rare,
01:58:41 you know, really, you know, crazy events that contribute to the safety
01:58:48 around pedestrians. Like, one example that comes to mind, that we actually
01:58:52 happened in Phoenix, where we were driving
01:58:56 along, and I think it was a 45 mile per hour road, so you have pretty high speed
01:59:00 traffic, and there was a sidewalk next to it, and
01:59:03 there was a cyclist on the sidewalk. And as we were in the right lane,
01:59:09 right next to the side, so it was a multi lane road, so as we got close
01:59:13 to the cyclist on the sidewalk, it was a woman, you know, she tripped and fell.
01:59:17 Just, you know, fell right into the path of our vehicle, right?
01:59:20 And our, you know, car, you know, this was actually with a
01:59:25 test driver, our test drivers, did exactly the right thing.
01:59:29 They kind of reacted, and came to stop. It requires both very strong steering,
01:59:33 and, you know, strong application of the brake. And then we simulated what our
01:59:37 system would have done in that situation, and it did, you know,
01:59:39 exactly the same thing. And that speaks to, you know, all of
01:59:43 those components of really good state estimation and
01:59:46 tracking. And, like, imagine, you know, a person
01:59:49 on a bike, and they’re falling over, and they’re doing that right in front of you,
01:59:52 right? So you have to be really, like, things are changing. The appearance of
01:59:54 that whole thing is changing, right? And a person goes one way, they’re falling on
01:59:57 the road, they’re, you know, being flat on the ground in front of
02:00:00 you. You know, the bike goes flying the other direction.
02:00:03 Like, the two objects that used to be one, they’re now, you know,
02:00:06 are splitting apart, and the car has to, like, detect all of that.
02:00:09 Like, milliseconds matter, and it doesn’t, you know, it’s not good enough to just
02:00:12 brake. You have to, like, steer and brake, and there’s traffic around you.
02:00:15 So, like, it all has to come together, and it was really great
02:00:19 to see in this case, and other cases like that, that we’re actually seeing in the
02:00:22 wild, that our system is, you know, performing
02:00:25 exactly the way that we would have liked, and is able to,
02:00:28 you know, avoid collisions like this.
02:00:30 That’s such an exciting space for robotics.
02:00:32 Like, in that split second to make decisions of life and death.
02:00:37 I don’t know. The stakes are high, in a sense, but it’s also beautiful
02:00:41 that for somebody who loves artificial intelligence, the possibility that an AI
02:00:47 system might be able to save a human life.
02:00:49 That’s kind of exciting as a problem, like, to wake up.
02:00:53 It’s terrifying, probably, for an engineer to wake up,
02:00:57 and to think about, but it’s also exciting because it’s, like,
02:01:01 it’s in your hands. Let me try to ask a question that’s often brought up about
02:01:05 autonomous vehicles, and it might be fun to see if you have
02:01:09 anything interesting to say, which is about the trolley problem.
02:01:14 So, a trolley problem is an interesting philosophical construct
02:01:19 that highlights, and there’s many others like it,
02:01:23 of the difficult ethical decisions that we humans have before us in this
02:01:29 complicated world. So, specifically is the choice
02:01:34 between if you are forced to choose to kill
02:01:39 a group X of people versus a group Y of people, like
02:01:42 one person. If you did nothing, you would kill one person, but if
02:01:48 you would kill five people, and if you decide to swerve out of the way, you
02:01:51 would only kill one person. Do you do nothing, or you choose to do
02:01:55 something? You can construct all kinds of, sort of,
02:01:58 ethical experiments of this kind that, I think, at least on a positive note,
02:02:05 inspire you to think about, like, introspect
02:02:09 what are the physics of our morality, and there’s usually not
02:02:16 good answers there. I think people love it because it’s just an exciting
02:02:20 thing to think about. I think people who build autonomous
02:02:24 vehicles usually roll their eyes, because this is not,
02:02:30 this one as constructed, this, like, literally never comes up
02:02:34 in reality. You never have to choose between killing
02:02:38 one or, like, one of two groups of people,
02:02:41 but I wonder if you can speak to, is there some something interesting
02:02:48 to you as an engineer of autonomous vehicles that’s within the trolley
02:02:52 problem, or maybe more generally, are there
02:02:55 difficult ethical decisions that you find
02:02:58 that an algorithm must make? On the specific version of the trolley problem,
02:03:03 which one would you do, if you’re driving? The question itself
02:03:07 is a profound question, because we humans ourselves
02:03:11 cannot answer, and that’s the very point. I would kill both.
02:03:18 Yeah, humans, I think you’re exactly right in that, you know, humans are not
02:03:21 particularly good. I think they’re kind of phrased as, like, what would a computer do,
02:03:24 but, like, humans, you know, are not very good, and actually oftentimes
02:03:28 I think that, you know, freezing and kind of not doing anything, because,
02:03:32 like, you’ve taken a few extra milliseconds to just process, and then
02:03:35 you end up, like, doing the worst of the possible outcomes, right? So,
02:03:38 I do think that, as you’ve pointed out, it can be
02:03:42 a bit of a distraction, and it can be a bit of a kind of red herring. I think
02:03:45 it’s an interesting, you know, discussion
02:03:47 in the realm of philosophy, right? But in terms of
02:03:51 what, you know, how that affects the actual
02:03:54 engineering and deployment of self driving vehicles,
02:03:57 it’s not how you go about building a system, right? We’ve talked
02:04:02 about how you engineer a system, how you, you know, go about evaluating
02:04:06 the different components and, you know, the safety of the entire thing.
02:04:09 How do you kind of inject the, you know, various
02:04:13 model based, safety based arguments, and, like, yes, you reason at parts of the
02:04:17 system, you know, you reason about the
02:04:20 probability of a collision, the severity of that collision, right?
02:04:24 And that is incorporated, and there’s, you know, you have to properly reason
02:04:27 about the uncertainty that flows through the system, right? So,
02:04:29 you know, those, you know, factors definitely play a role in how
02:04:34 the cars then behave, but they tend to be more
02:04:36 of, like, the emergent behavior. And what you see, like, you’re absolutely right
02:04:39 that these, you know, clear theoretical problems that they, you
02:04:43 know, you don’t encounter that in the system, and really kind of being
02:04:46 back to our previous discussion of, like, what, you know, what, you
02:04:49 know, which one do you choose? Well, you know, oftentimes, like,
02:04:53 you made a mistake earlier. Like, you shouldn’t be in that situation
02:04:57 in the first place, right? And in reality, the system comes up.
02:05:00 If you build a very good, safe, and capable driver,
02:05:03 you have enough, you know, clues in the environment that you
02:05:08 drive defensively, so you don’t put yourself in that situation, right? And
02:05:11 again, you know, it has, you know, this, if you go back to that analogy of, you
02:05:14 know, precision and recoil, like, okay, you can make a, you know, very hard trade
02:05:16 off, but like, neither answer is really good.
02:05:19 But what instead you focus on is kind of moving
02:05:22 the whole curve up, and then you focus on building the right capability on the
02:05:26 right defensive driving, so that, you know, you don’t put yourself in the
02:05:28 situation like this. I don’t know if you have a good answer
02:05:32 for this, but people love it when I ask this question
02:05:35 about books. Are there books in your life that you’ve enjoyed,
02:05:42 philosophical, fiction, technical, that had a big impact on you as an engineer or
02:05:47 as a human being? You know, everything from science fiction
02:05:50 to a favorite textbook. Is there three books that stand out that
02:05:53 you can think of? Three books. So I would, you know, that
02:05:57 impacted me, I would say,
02:06:02 and this one is, you probably know it well,
02:06:06 but not generally well known, I think, in the U.S., or kind of
02:06:11 internationally, The Master and Margarita. It’s one of, actually, my
02:06:16 favorite books. It is, you know, by
02:06:20 Russian, it’s a novel by Russian author Mikhail Bulgakov, and it’s just, it’s a
02:06:26 great book. It’s one of those books that you can, like,
02:06:28 reread your entire life, and it’s very accessible. You can read it as a kid,
02:06:32 and, like, it’s, you know, the plot is interesting. It’s, you know, the
02:06:35 devil, you know, visiting the Soviet Union,
02:06:38 and, you know, but it, like, you read it, reread it
02:06:41 at different stages of your life, and you enjoy it for
02:06:46 different, very different reasons, and you keep finding, like, deeper and deeper
02:06:49 meaning, and, you know, kind of affected, you know,
02:06:52 had a, definitely had an, like, imprint on me, you know, mostly from the,
02:06:57 probably kind of the cultural, stylistic aspect. Like, it makes you think one of
02:07:00 those books that, you know, is good and makes you think, but also has,
02:07:04 like, this really, you know, silly, quirky, dark sense of, you know,
02:07:07 humor. It captures the Russian soul more than
02:07:10 many, perhaps, many other books. On that, like, slight note,
02:07:13 just out of curiosity, one of the saddest things is I’ve read that book
02:07:17 in English. Did you, by chance, read it in English or in Russian?
02:07:22 In Russian, only in Russian, and I actually, that is a question I had,
02:07:26 kind of posed to myself every once in a while, like, I wonder how well it
02:07:30 translates, if it translates at all, and there’s the
02:07:33 language aspect of it, and then there’s the cultural aspect, so
02:07:35 I, actually, I’m not sure if, you know, either of those would
02:07:39 work well in English. Now, I forget their names, but, so, when the COVID lifts a
02:07:43 little bit, I’m traveling to Paris for several reasons. One is just, I’ve
02:07:48 never been to Paris, I want to go to Paris, but
02:07:50 there’s the most famous translators of Dostoevsky, Tolstoy, of most of
02:07:57 Russian literature live there. There’s a couple, they’re famous,
02:08:00 a man and a woman, and I’m going to, sort of, have a series of conversations with
02:08:03 them, and in preparation for that, I’m starting
02:08:06 to read Dostoevsky in Russian, so I’m really embarrassed to say that I read
02:08:10 this, everything I’ve read in Russian literature of, like,
02:08:13 serious depth has been in English, even though
02:08:18 I can also read, I mean, obviously, in Russian, but
02:08:21 for some reason, it seemed,
02:08:26 in the optimization of life, it seemed the improper decision to do, to read in
02:08:31 Russian, like, you know, like, I don’t need to,
02:08:35 I need to think in English, not in Russian, but now I’m changing my mind on
02:08:38 that, and so, the question of how well I translate, it’s a
02:08:41 really fun to method one, like, even with Dostoevsky.
02:08:43 So, from what I understand, Dostoevsky translates easier,
02:08:47 others don’t as much. Obviously, the poetry doesn’t translate as well,
02:08:52 I’m also the music big fan of Vladimir Vosotsky,
02:08:57 he doesn’t obviously translate well, people have tried,
02:09:02 but mastermind, I don’t know, I don’t know about that one, I just know in
02:09:06 English, you know, as fun as hell in English, so, so, but
02:09:10 it’s a curious question, and I want to study it rigorously from both the
02:09:13 machine learning aspect, and also because I want to do a
02:09:16 couple of interviews in Russia, that
02:09:21 I’m still unsure of how to properly conduct an interview
02:09:27 across a language barrier, it’s a fascinating question
02:09:30 that ultimately communicates to an American audience. There’s a few
02:09:34 Russian people that I think are truly special human beings,
02:09:39 and I feel, like, I sometimes encounter this with some
02:09:44 incredible scientists, and maybe you encounter this
02:09:48 as well at some point in your life, that it feels like because of the language
02:09:52 barrier, their ideas are lost to history. It’s a sad thing, I think about, like,
02:09:57 Chinese scientists, or even authors that, like,
02:10:01 that we don’t, in an English speaking world, don’t get to appreciate
02:10:05 some, like, the depth of the culture because it’s lost in translation,
02:10:09 and I feel like I would love to show that to the world,
02:10:13 like, I’m just some idiot, but because I have this,
02:10:16 like, at least some semblance of skill in speaking Russian,
02:10:20 I feel like, and I know how to record stuff on a video camera,
02:10:25 I feel like I want to catch, like, Grigori Perlman, who’s a mathematician, I’m not
02:10:28 sure if you’re familiar with him, I want to talk to him, like, he’s a
02:10:31 fascinating mind, and to bring him to a wider audience in English speaking
02:10:35 will be fascinating, but that requires to be rigorous about this question
02:10:40 of how well Bulgakov translates. I mean, I know it’s a silly
02:10:46 concept, but it’s a fundamental one, because how do you translate, and
02:10:50 that’s the thing that Google Translate is also facing
02:10:54 as a more machine learning problem, but I wonder as a more
02:10:59 bigger problem for AI, how do we capture the magic
02:11:03 that’s there in the language? I think that’s a really interesting,
02:11:08 really challenging problem. If you do read it, Master and Margarita
02:11:12 in English, sorry, in Russian, I’d be curious
02:11:16 to get your opinion, and I think part of it is language, but part of it’s just,
02:11:20 you know, centuries of culture, that, you know, the cultures are
02:11:23 different, so it’s hard to connect that.
02:11:28 Okay, so that was my first one, right? You had two more. The second one I
02:11:31 would probably pick is the science fiction by the
02:11:35 Strogatsky brothers. You know, it’s up there with, you know,
02:11:38 Isaac Asimov and, you know, Ray Bradbury and, you know, company. The
02:11:43 Strogatsky brothers kind of appealed more to me. I think it made more of an
02:11:47 impression on me growing up. I apologize if I’m
02:11:53 showing my complete ignorance. I’m so weak on sci fi. What did
02:11:57 they write? Oh, Roadside Picnic,
02:12:04 Heart to Be a God,
02:12:07 Beetle in an Ant Hill, Monday Starts on Saturday. Like, it’s
02:12:14 not just science fiction. It also has very interesting, you know,
02:12:17 interpersonal and societal questions, and some of the
02:12:21 language is just completely hilarious.
02:12:27 That’s the one. Oh, interesting. Monday Starts on Saturday. So,
02:12:31 I need to read. Okay, oh boy. You put that in the category of science fiction?
02:12:36 That one is, I mean, this was more of a silly,
02:12:39 you know, humorous work. I mean, there is kind of…
02:12:43 It’s profound too, right? Science fiction, right? It’s about, you know, this
02:12:46 research institute, and it has deep parallels to
02:12:50 serious research, but the setting, of course,
02:12:53 is that they’re working on, you know, magic, right? And there’s a
02:12:56 lot of stuff. And that’s their style, right?
02:13:00 And, you know, other books are very different, right? You know,
02:13:03 Heart to Be a God, right? It’s about kind of this higher society being injected
02:13:07 into this primitive world, and how they operate there,
02:13:09 and some of the very deep ethical questions there,
02:13:13 right? And, like, they’ve got this full spectrum. Some is, you know, more about
02:13:16 kind of more adventure style. But, like, I enjoy all of
02:13:19 their books. There’s just, you know, probably a couple.
02:13:21 Actually, one I think that they consider their most important work.
02:13:24 I think it’s The Snail on a Hill. I’m not exactly sure how it
02:13:29 translates. I tried reading a couple times. I still don’t get it.
02:13:32 But everything else I fully enjoyed. And, like, for one of my birthdays as a kid, I
02:13:36 got, like, their entire collection, like, occupied a giant shelf in my room, and
02:13:40 then, like, over the holidays, I just, like,
02:13:42 you know, my parents couldn’t drag me out of the room, and I read the whole thing
02:13:44 cover to cover. And I really enjoyed it.
02:13:49 And that’s one more. For the third one, you know, maybe a little bit
02:13:52 darker, but, you know, comes to mind is Orwell’s
02:13:56 1984. And, you know, you asked what made an
02:14:01 impression on me and the books that people should read. That one, I think,
02:14:03 falls in the category of both. You know, definitely it’s one of those
02:14:06 books that you read, and you just kind of, you know, put it
02:14:11 down and you stare in space for a while. You know, that kind of work. I think
02:14:16 there’s, you know, lessons there. People should
02:14:19 not ignore. And, you know, nowadays, with, like,
02:14:24 everything that’s happening in the world, I,
02:14:26 like, can’t help it, but, you know, have my mind jump to some,
02:14:29 you know, parallels with what Orwell described. And, like, there’s this whole,
02:14:34 you know, concept of double think and ignoring logic and, you know, holding
02:14:38 completely contradictory opinions in your mind and not have that not bother
02:14:41 you and, you know, sticking to the party line
02:14:44 at all costs. Like, you know, there’s something there.
02:14:48 If anything, 2020 has taught me, and I’m a huge fan of Animal Farm, which is a
02:14:52 kind of friendly, as a friend of 1984 by Orwell.
02:14:57 It’s kind of another thought experiment of how our society
02:15:03 may go in directions that we wouldn’t like it to go.
02:15:07 But if anything that’s been kind of heartbreaking to an
02:15:14 optimist about 2020 is that
02:15:18 that society is kind of fragile. Like, we have this,
02:15:22 this is a special little experiment we have going on.
02:15:25 And not, it’s not unbreakable. Like, we should be careful to, like, preserve
02:15:32 whatever the special thing we have going on. I mean, I think 1984
02:15:36 and these books, The Brave New World, they’re
02:15:39 helpful in thinking, like, stuff can go wrong
02:15:43 in nonobvious ways. And it’s, like, it’s up to us to preserve it.
02:15:48 And it’s, like, it’s a responsibility. It’s been weighing heavy on me because, like,
02:15:51 for some reason, like, more than my mom follows me on Twitter and I
02:15:57 feel like I have, like, now somehow a
02:15:59 responsibility to
02:16:03 do this world. And it dawned on me that, like,
02:16:07 me and millions of others are, like, the little ants
02:16:12 that maintain this little colony, right? So we have a responsibility not to
02:16:17 be, I don’t know what the right analogy is, but
02:16:20 I’ll put a flamethrower to the place. We want to
02:16:23 not do that. And there’s interesting complicated ways of doing that as 1984
02:16:27 shows. It could be through bureaucracy. It could
02:16:29 be through incompetence. It could be through misinformation.
02:16:33 It could be through division and toxicity.
02:16:36 I’m a huge believer in, like, that love will be
02:16:39 the, somehow, the solution. So, love and robots. Love and robots, yeah.
02:16:46 I think you’re exactly right. Unfortunately, I think it’s less of a
02:16:49 flamethrower type of thing. It’s more of a,
02:16:51 in many cases, it’s going to be more of a slow boil. And that’s the
02:16:55 danger. Let me ask, it’s a fun thing to make
02:17:00 a world class roboticist, engineer, and leader uncomfortable with a
02:17:05 ridiculous question about life. What is the meaning of life,
02:17:09 Dimitri, from a robotics and a human perspective?
02:17:14 You only have a couple minutes, or one minute to answer, so.
02:17:19 I don’t know if that makes it more difficult or easier, actually.
02:17:23 You know, they’re very tempted to quote one of the
02:17:29 stories by Isaac Asimov, actually. Actually, titled,
02:17:36 appropriately titled, The Last Question. It’s a short story where, you know, the
02:17:39 plot is that, you know, humans build this supercomputer,
02:17:42 you know, this AI intelligence, and, you know, once it
02:17:46 gets powerful enough, they pose this question to it, you know,
02:17:49 how can the entropy in the universe be reduced, right? So the computer replies,
02:17:54 as of yet, insufficient information to give a meaningful answer,
02:17:58 right? And then, you know, thousands of years go by, and they keep posing the
02:18:00 same question, and the computer, you know, gets more and more powerful, and keeps
02:18:03 giving the same answer, you know, as of yet, insufficient
02:18:06 information to give a meaningful answer, or something along those lines,
02:18:09 right? And then, you know, it keeps, you know, happening, and
02:18:12 happening, you fast forward, like, millions of years into the future, and,
02:18:16 you know, billions of years, and, like, at some point, it’s just the only entity in
02:18:19 the universe, it’s, like, absorbed all humanity,
02:18:21 and all knowledge in the universe, and it, like, keeps posing the same question
02:18:24 to itself, and, you know, finally, it gets to the
02:18:28 point where it is able to answer that question, but, of course, at that point,
02:18:31 you know, there’s, you know, the heat death of the universe has occurred, and
02:18:34 that’s the only entity, and there’s nobody else to provide that
02:18:37 answer to, so the only thing it can do is to,
02:18:40 you know, answer it by demonstration, so, like, you know, it recreates the big bang,
02:18:43 right, and resets the clock, right?
02:18:47 But, like, you know, I can try to give kind of a
02:18:50 different version of the answer, you know, maybe
02:18:53 not on the behalf of all humanity, I think that that might be a little
02:18:56 presumptuous for me to speak about the meaning of life on the behalf of all
02:19:00 humans, but at least, you know, personally,
02:19:03 it changes, right? I think if you think about kind of what
02:19:06 gives, you know, you and your life meaning and purpose, and kind of
02:19:13 what drives you, it seems to
02:19:18 change over time, right, and that lifespan
02:19:22 of, you know, kind of your existence, you know, when
02:19:25 just when you just enter this world, right, it’s all about kind of new
02:19:27 experiences, right? You get, like, new smells, new sounds, new emotions, right,
02:19:33 and, like, that’s what’s driving you, right? You’re experiencing
02:19:36 new amazing things, right, and that’s magical, right? That’s pretty
02:19:40 pretty awesome, right? That gives you kind of meaning.
02:19:43 Then, you know, you get a little bit older, you start more intentionally
02:19:47 learning about things, right? I guess, actually, before you start intentionally
02:19:51 learning, it’s probably fun. Fun is a thing that gives you kind of
02:19:53 meaning and purpose and purpose and the thing you optimize for, right?
02:19:56 And, like, fun is good. Then you get, you know, start learning, and I guess that
02:20:01 this joy of comprehension
02:20:05 and discovery is another thing that, you know, gives you
02:20:09 meaning and purpose and drives you, right? Then, you know, you
02:20:12 learn enough stuff and you want to give some of it back, right? And so
02:20:17 impact and contributions back to, you know, technology or society,
02:20:20 you know, people, you know, local or more globally
02:20:24 becomes a new thing that, you know, drives a lot of kind of your behavior
02:20:28 and is something that gives you purpose and
02:20:31 that you derive, you know, positive feedback from, right?
02:20:35 You know, then you go and so on and so forth. You go through various stages of
02:20:38 life. If you have kids,
02:20:43 like, that definitely changes your perspective on things. You know, I have
02:20:46 three that definitely flips some bits in your
02:20:48 head in terms of, you know, what you care about and what you
02:20:52 optimize for and, you know, what matters, what doesn’t matter, right?
02:20:54 So, you know, and so on and so forth, right? And I,
02:20:58 it seems to me that, you know, it’s all of those things and as
02:21:02 kind of you go through life, you know,
02:21:06 you want these to be additive, right? New experiences,
02:21:10 fun, learning, impact. Like, you want to, you know, be accumulating.
02:21:14 I don’t want to, you know, stop having fun or, you know, experiencing new things and
02:21:17 I think it’s important that, you know, it just kind of becomes
02:21:20 additive as opposed to a replacement or subtraction.
02:21:23 But, you know, those fewest problems as far as I got, but, you know, ask me in a
02:21:27 few years, I might have one or two more to add to the list.
02:21:30 And before you know it, time is up, just like it is for this conversation,
02:21:34 but hopefully it was a fun ride. It was a huge honor to meet you.
02:21:38 As you know, I’ve been a fan of yours and a fan of Google Self Driving Car and
02:21:43 Waymo for a long time. I can’t wait. I mean, it’s one of the
02:21:47 most exciting, if we look back in the 21st century, I
02:21:50 truly believe it’ll be one of the most exciting things we
02:21:53 descendants of apes have created on this earth. So,
02:21:57 I’m a huge fan and I can’t wait to see what you do
02:22:00 next. Thanks so much for talking to me. Thanks, thanks for having me and it’s a
02:22:04 also a huge fan doing work, honestly, and I really
02:22:08 enjoyed it. Thank you. Thanks for listening to this
02:22:11 conversation with Dmitry Dolgov and thank you to our sponsors,
02:22:14 Triolabs, a company that helps businesses apply machine learning to
02:22:19 solve real world problems, Blinkist, an app I use for reading
02:22:23 through summaries of books, BetterHelp, online therapy with a licensed
02:22:27 professional, and CashApp, the app I use to send money to
02:22:30 friends. Please check out these sponsors in the
02:22:33 description to get a discount and to support this podcast. If you
02:22:37 enjoy this thing, subscribe on YouTube, review it with Five Stars
02:22:40 and Upper Podcast, follow on Spotify, support on Patreon,
02:22:44 or connect with me on Twitter at Lex Friedman. And now,
02:22:47 let me leave you with some words from Isaac Asimov.
02:22:51 Science can amuse and fascinate us all, but it is engineering
02:22:55 that changes the world. Thank you for listening and hope to see you
02:22:59 next time.