Transcript
00:00:00 The following is a conversation with Sirtesh Karaman, a professor at MIT, co founder of
00:00:05 the autonomous vehicle company, Optimus Ride, and is one of the top roboticists in the world,
00:00:11 including robots that drive and robots that fly.
00:00:14 To me personally, he has been a mentor, a colleague and a friend.
00:00:19 He’s one of the smartest, most generous people I know.
00:00:22 So it was a pleasure and honor to finally sit down with him for this recorded conversation.
00:00:27 This is the Artificial Intelligence Podcast.
00:00:30 If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcast, support
00:00:34 on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F R I D M A N.
00:00:41 As usual, I’ll do a few minutes of ads now and never any ads in the middle that can break
00:00:45 the flow of the conversation.
00:00:47 I hope that works for you and doesn’t hurt the listening experience.
00:00:51 This show is presented by Cash App, the number one finance app in the App Store.
00:00:55 When you get it, use the code LEX PODCAST.
00:00:59 Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with
00:01:03 as little as $1.
00:01:05 Since Cash App allows you to send and receive money digitally, let me mention a surprising
00:01:10 fact about physical money.
00:01:12 It costs 2.4 cents to produce a single penny.
00:01:16 In fact, I think it costs $85 million annually to produce them.
00:01:22 That’s a crazy little fact about physical money.
00:01:25 So again, if you get Cash App from the App Store, Google Play, and use the code LEX PODCAST,
00:01:30 you get $10, and Cash App will also donate $10 to FIRST, an organization that is helping
00:01:35 to advance robotics and STEM education for young people around the world.
00:01:40 And now, here’s my conversation with Sirtesh Karaman.
00:01:44 Since you have worked extensively on both, what is the more difficult task?
00:01:49 Autonomous flying or autonomous driving?
00:01:51 That’s a good question.
00:01:52 I think that autonomous flying, just doing it for consumer drones and so on, the kinds
00:01:58 of applications that we’re looking at right now, is probably easier.
00:02:02 And so I think that that’s maybe one of the reasons why it took off literally a little
00:02:06 earlier than the autonomous cars.
00:02:09 But I think if you look ahead, I would think that the real benefits of autonomous flying,
00:02:14 unleashing them in transportation, logistics, and so on, I think it’s a lot harder than
00:02:18 autonomous driving.
00:02:19 So I think my guess is that we’ve seen a few kind of machines fly here and there, but we
00:02:24 really haven’t yet seen any kind of machine, like at massive scale, large scale being deployed
00:02:32 and flown and so on.
00:02:33 And I think that’s going to be after we kind of resolve some of the large scale deployments
00:02:38 of autonomous driving.
00:02:39 So what’s the hard part?
00:02:40 What’s your intuition behind why at scale, when consumer facing drones are tough?
00:02:47 So I think in general, at scale is tough.
00:02:51 Like for example, when you think about it, we have actually deployed a lot of robots
00:02:57 in the, let’s say the past 50 years.
00:03:00 We as academics or we business entrepreneurs?
00:03:02 I think we as humanity.
00:03:03 Humanity?
00:03:04 A lot of people working on it.
00:03:05 So we humans deployed a lot of robots.
00:03:09 And I think that, well, when you think about it, you know, robots, they’re autonomous.
00:03:14 They work and they work on their own, but they are either like in isolated environments
00:03:19 or they are in sort of, you know, they may be at scale, but they’re really confined to
00:03:27 a certain environment that they don’t interact so much with humans.
00:03:30 And so, you know, they work in, I don’t know, factory floors, warehouses, they work on Mars,
00:03:35 you know, they are fully autonomous over there.
00:03:38 But I think that the real challenge of our time is to take these vehicles and put them
00:03:44 into places where humans are present.
00:03:47 So now I know that there’s a lot of like human robot interaction type of things that need
00:03:51 to be done.
00:03:52 And so that’s one thing, but even just from the fundamental algorithms and systems and
00:03:58 the business cases, or maybe the business models, even like architecture, planning,
00:04:03 societal issues, legal issues, there’s a whole bunch of pack of things that are related to
00:04:08 us putting robotic vehicles into human present environments.
00:04:12 And as humans, you know, they will not potentially be even trained to interact with them.
00:04:18 They may not even be using the services that are provided by these vehicles.
00:04:21 They may not even know that they’re autonomous.
00:04:23 They’re just doing their thing, living in environments that are designed for humans,
00:04:27 not for robots.
00:04:28 And that I think is one of the biggest challenges, I think, of our time to put vehicles there.
00:04:35 And you know, to go back to your question, I think doing that at scale, meaning, you
00:04:40 know, you go out in a city and you have, you know, like thousands or tens of thousands
00:04:46 of autonomous vehicles that are going around.
00:04:48 It is so dense to the point where if you see one of them, you look around, you see another
00:04:53 one.
00:04:54 It is that dense.
00:04:55 And that density, we’ve never done anything like that before.
00:04:59 And I would bet that that kind of density will first happen with autonomous cars because
00:05:05 I think, you know, we can bend the environment a little bit.
00:05:08 We can, especially kind of making them safe is a lot easier when they’re like on the ground.
00:05:15 When they’re in the air, it’s a little bit more complicated.
00:05:19 But I don’t see that there’s going to be a big separation.
00:05:21 I think that, you know, there will come a time that we’re going to quickly see these
00:05:24 things unfold.
00:05:25 Do you think there will be a time where there’s tens of thousands of delivery drones that
00:05:30 fill the sky?
00:05:31 You know, I think, I think it’s possible to be honest.
00:05:34 Delivery drones is one thing, but you know, you can imagine for transportation, like an
00:05:38 important use case is, you know, we’re in Boston, you want to go from Boston to New
00:05:42 York and you want to do it from the top of this building to the top of another building
00:05:47 in Manhattan.
00:05:48 And you’re going to do it in one and a half hours.
00:05:50 And that’s, that’s a big opportunity, I think.
00:05:53 Personal transport.
00:05:54 So like you and me be a friend, like almost like an Uber.
00:05:58 So like four people, six people, eight people.
00:06:01 In our work in autonomous vehicles, I see that.
00:06:04 So there’s kind of like a bit of a need for, you know, one person transport, but also like,
00:06:08 like a few people.
00:06:09 So you and I could take that trip together.
00:06:10 We could have lunch, you know, I think kind of sounds crazy, maybe even sounds a bit cheesy,
00:06:16 but I think that those kinds of things are some of the real opportunities.
00:06:20 And I think, you know it’s not like the typical airplane and the airport would disappear very
00:06:26 quickly, but I would think that, you know many people would feel like they would spend
00:06:31 an extra hundred dollars on doing that and cutting that four hour travel down to one
00:06:36 and a half hours.
00:06:37 So how feasible are flying cars has been the dream.
00:06:41 That’s like when people imagine the future for 50 plus years, they think flying cars,
00:06:46 it’s a, it’s like all technologies.
00:06:49 It’s cheesy to think about now because it seems so far away, but overnight it can change.
00:06:54 But just technically speaking in your view, how feasible is it to make that happen?
00:06:59 I’ll get to that question, but just one thing is that I think, you know, sometimes we think
00:07:03 about what’s going to happen in the next 50 years.
00:07:07 It’s just really hard to guess, right?
00:07:08 Next 50 years.
00:07:09 I don’t know.
00:07:10 I mean, we could get what’s going to happen in transportation in the next 50, we could
00:07:13 get flying saucers.
00:07:14 I could bet on that.
00:07:16 I think there’s a 50, 50 chance that, you know, like you can build machines that can
00:07:19 ionize the air around them and push it down with magnets and they would fly like a flying
00:07:23 saucer that is possible.
00:07:26 And it might happen in the next 50 years.
00:07:27 So it’s a bit hard to guess like when you think about 50 years before, but I would think
00:07:32 that, you know, there’s this, this, this kind of a notion where there’s a certain type of
00:07:38 airspace that we call the agile airspace.
00:07:41 And there’s, there’s good amount of opportunities in that airspace.
00:07:44 So that would be the space that is kind of a little bit higher than the place where you
00:07:49 can throw a stone because that’s a tough thing when you think about it, you know, it takes
00:07:53 a kid on a stone to take an aircraft down and then what happens.
00:07:59 But you know, imagine the airspace that’s high enough so that you cannot throw the stone,
00:08:05 but it is low enough that you’re not interacting with the, with the very large aircraft that
00:08:11 are, you know, flying several thousand feet above.
00:08:15 And that airspace is underutilized or it’s actually kind of not utilized at all.
00:08:20 Yeah, that’s right.
00:08:21 You know, there’s like recreational people kind of fly every now and then, but it’s very
00:08:24 few.
00:08:25 Like if you look up in the sky, you may not see any of them at any given time, every now
00:08:30 and then you’ll see one airplane kind of utilizing that space and you’ll be surprised.
00:08:34 And the moment you’re outside of an airport a little bit, like it just kind of flies off
00:08:38 and then it goes out.
00:08:39 And I think utilizing that airspace, the technical challenges there is, you know, building an
00:08:47 autonomy and ensuring that that kind of autonomy is safe.
00:08:51 Ultimately, I think it is going to be building in complex software or complicated so that
00:08:59 it’s maybe a few orders of magnitude more complicated than what we have on aircraft
00:09:03 today.
00:09:05 And at the same time, ensuring just like we ensure on aircraft, ensuring that it’s safe.
00:09:10 And so that becomes like building that kind of complicated hardware and software becomes
00:09:15 a challenge, especially when, you know, you build that hardware, I mean, you build that
00:09:20 software with data.
00:09:22 And so, you know, it’s, of course there’s some rule based software in there that kind
00:09:28 of do a certain set of things, but then, you know, there’s a lot of training there.
00:09:32 Do you think machine learning will be key to these kinds of, to delivering safe vehicles
00:09:37 in the future, especially flight?
00:09:40 Not maybe the safe part, but I think the intelligent part.
00:09:43 I mean, there are certain things that we do it with machine learning and it’s just, there’s
00:09:48 like right now, no other way.
00:09:50 And I don’t know how else they could be done.
00:09:53 And you know, there’s always this conundrum, I mean, we could like, could we like, we could
00:10:00 maybe gather billions of programmers, humans who program perception algorithms that detect
00:10:09 things in the sky and whatever, or, you know, we, I don’t know, we maybe even have robots
00:10:14 like learn in a simulation environment and transfer.
00:10:17 And they might be learning a lot better in a simulation environment than a billion humans
00:10:22 put their brains together and try to program.
00:10:25 Humans pretty limited.
00:10:26 So what’s, what’s the role of simulations with drones?
00:10:30 You’ve done quite a bit of work there.
00:10:32 How promising, just the very thing you said just now, how promising is the possibility
00:10:36 of training and developing a safe flying robot in simulation and deploying it and having
00:10:45 that work pretty well in the real world?
00:10:48 I think that, you know, a lot of people, when they hear simulation, they will focus on training
00:10:53 immediately.
00:10:54 But I think one thing that you said, which was interesting, it’s developing.
00:10:57 I think simulation environments are actually could be key and great for development.
00:11:01 And that’s not new.
00:11:03 Like for example, you know, there’s people in the automotive industry have been using
00:11:09 dynamic simulation for like decades now.
00:11:12 And it’s pretty standard that, you know, you would build and you would simulate.
00:11:16 If you want to build an embedded controller, you plug that kind of embedded computer into
00:11:20 another computer, that other computer would simulate dynamic and so on.
00:11:24 And I think, you know, fast forward these things, you can create pretty crazy simulation
00:11:28 environments.
00:11:29 Like for instance, one of the things that has happened recently and that, you know,
00:11:34 we can do now is that we can simulate cameras a lot better than we used to simulate them.
00:11:39 We were able to simulate them before.
00:11:41 And that’s, I think we just hit the elbow on that kind of improvement.
00:11:45 I would imagine that with improvements in hardware, especially, and with improvements
00:11:50 in machine learning, I think that we would get to a point where we can simulate cameras
00:11:55 very, very well.
00:11:57 Simulate cameras means simulate how a real camera would see the real world.
00:12:03 Therefore you can explore the limitations of that.
00:12:07 You can train perception algorithms on that in simulation, all that kind of stuff.
00:12:13 Exactly.
00:12:14 So, you know, it’s, it’s, it has been easier to simulate what we would call introspective
00:12:18 sensors like internal sensors.
00:12:20 So for example, inertial sensing has been easy to simulate.
00:12:24 It has also been easy to simulate dynamics, like physics that are governed by ordinary
00:12:29 differential equations.
00:12:30 I mean, like how a car goes around, maybe how it rolls on the road, how it interacts
00:12:35 with the road, or even an aircraft flying around, like the dynamic physics of that.
00:12:40 What has been really hard has been to simulate extra septive sensors, sensors that kind of
00:12:45 like look out from the vehicle.
00:12:48 And that’s a new thing that’s coming like laser range finders that are a little bit
00:12:52 easier.
00:12:53 Because radars are a little bit tougher.
00:12:56 I think once we nail that down, the next challenge I think in simulation will be to simulate
00:13:02 human behavior.
00:13:03 That’s also extremely hard.
00:13:05 Even when you imagine like how a human driven car would act around, even that is hard.
00:13:12 But imagine trying to simulate, you know, a model of a human just doing a bunch of gestures
00:13:17 and so on.
00:13:18 And you know, it’s, it’s actually simulated.
00:13:20 It’s not captured like with motion capture, but it is simulated.
00:13:23 That’s very hard.
00:13:24 In fact, today I get involved a lot with like sort of this kind of very high end rendering
00:13:29 projects and I have like this test that I pass it to my friends or my mom, you know,
00:13:34 I send like two photos, two kind of pictures and I say rendered, which one is rendered,
00:13:39 which one is real.
00:13:40 And it’s pretty hard to distinguish, except I realized, except when we put humans in there,
00:13:45 it’s possible that our brains are trained in a way that we recognize humans extremely
00:13:50 well.
00:13:51 We don’t so much recognize the built environments because built environments sort of came after
00:13:55 per se we evolved into sort of being humans, but humans were always there.
00:14:00 Same thing happens, for example, you look at like monkeys and you can’t distinguish one
00:14:04 from another, but they sort of do.
00:14:06 And it’s very possible that they look at humans.
00:14:08 It’s kind of pretty hard to distinguish one from another, but we do.
00:14:12 And so our eyes are pretty well trained to look at humans and understand if something
00:14:15 is off, we will get it.
00:14:18 We may not be able to pinpoint it.
00:14:19 So in my typical friend test or mom test, what would happen is that we’d put like a
00:14:23 human walking in anything and they say, you know, this is not right.
00:14:29 Something is off in this video.
00:14:31 I don’t know what, but I can tell you it’s the human.
00:14:34 I can take the human and I can show you like inside of a building or like an apartment
00:14:38 and it will look like if we had time to render it, it will look great.
00:14:42 And this should be no surprise.
00:14:43 A lot of movies that people are watching, it’s all computer generated.
00:14:47 You know, even nowadays, even you watch a drama movie and like, there’s nothing going
00:14:51 on action wise, but it turns out it’s kind of like cheaper, I guess, to render the background.
00:14:55 And so they would.
00:14:57 But how do we get there?
00:14:59 How do we get a human that’s would pass the mom slash friend test, a simulation of a human
00:15:08 walking?
00:15:09 So do you think that’s something we can creep up to by just doing kind of a comparison learning
00:15:17 where you have humans annotate what’s more realistic and not just by watching, like what’s
00:15:23 the path?
00:15:24 Cause it seems totally mysterious how we simulate human behavior.
00:15:29 It’s hard because a lot of the other things that I mentioned to you, including simulating
00:15:34 cameras, right?
00:15:35 It is, the thing there is that, you know, we know the physics, we know how it works
00:15:41 like in the real world and we can write some rules and we can do that.
00:15:46 Like for example, simulating cameras, there’s this thing called ray tracing.
00:15:49 I mean, you literally just kind of imagine it’s very similar to, it’s not exactly the
00:15:54 same, but it’s very similar to tracing photon by photon.
00:15:57 They’re going around, bouncing on things and come into your eye, but human behavior, developing
00:16:03 a dynamic, like a model of that, that is mathematical so that you can put it into a processor that
00:16:11 would go through that, that’s going to be hard.
00:16:13 And so what else do you got?
00:16:15 You can collect data, right?
00:16:17 And you can try to match the data.
00:16:20 Or another thing that you can do is that, you know, you can show the friend test, you
00:16:23 know, you can say this or that and this or that, and that will be labeling.
00:16:27 Anything that requires human labeling, ultimately we’re limited by the number of humans that,
00:16:31 you know, we have available at our disposal and the things that they can do, you know,
00:16:35 they have to do a lot of other things than also labeling this data.
00:16:39 So that modeling human behavior part is, is I think going, we’re going to realize it’s
00:16:44 very tough.
00:16:45 And I think that also affects, you know, our development of autonomous vehicles.
00:16:50 I see them in self driving as well.
00:16:52 Like you want to use, so you’re building self driving, you know, at the first time, like
00:16:57 right after urban challenge, I think everybody focused on localization, mapping and localization,
00:17:03 you know, slam algorithms came in, Google was just doing that.
00:17:06 And so building these HD maps, basically that’s about knowing where you are.
00:17:11 And then five years later in 2012, 2013 came the kind of coding code AI revolution.
00:17:16 And that started telling us where everybody else is, but we’re still missing what everybody
00:17:21 else is going to do next.
00:17:23 And so you want to know where you are.
00:17:24 You want to know what everybody else is.
00:17:26 Hopefully you know that what you’re going to do next, and then you want to predict what
00:17:29 other people are going to do.
00:17:30 And that last bit has, has been a real, real challenge.
00:17:35 What do you think is the role, your own of your, of your, the ego vehicle, the robot,
00:17:42 the you, the robotic you in controlling and having some control of how the future unrolls
00:17:49 of what’s going to happen in the future.
00:17:51 That seems to be a little bit ignored in trying to predict the future is how you yourself
00:17:57 can affect that future by being either aggressive or less aggressive or signaling in some kind
00:18:05 of way.
00:18:06 So this kind of game theoretic dance seems to be ignored for the moment.
00:18:10 It’s yeah, it’s, it’s totally ignored.
00:18:12 I mean, it’s, it’s quite interesting actually, like how we how we interact with things versus
00:18:19 we interact with humans.
00:18:21 Like so if, if you see a vehicle that’s completely empty and it’s trying to do something, all
00:18:27 of a sudden it becomes a thing.
00:18:29 So interacted with like you interact with this table and so you can throw your backpack
00:18:34 or you can kick your, kick it, put your feet on it and things like that.
00:18:38 But when it’s a human, there’s all kinds of ways of interacting with a human.
00:18:42 So if you know, like you and I are face to face, we’re very civil.
00:18:45 You know, we talk, we understand each other for the most part.
00:18:48 We’ll see you just, you never know what’s going to happen.
00:18:52 But the thing is that like, for example, you and I might interact through YouTube comments
00:18:56 and, you know, the conversation may go at a totally different angle.
00:19:01 And so I think people kind of abusing as autonomous vehicles is a real issue in some sense.
00:19:08 And so when you’re an ego vehicle, you’re trying to, you know, coordinate your way,
00:19:12 make your way, it’s actually kind of harder than being a human.
00:19:16 You know, it’s like, it’s you, you, you not only need to be as smart as, as kind of humans
00:19:20 are, but you also, you’re a thing.
00:19:22 So they’re going to abuse you a little bit.
00:19:23 So you need to make sure that you can get around and do something.
00:19:28 So I, in general, believe in that sort of game theoretic aspects.
00:19:34 I’ve actually personally have done, you know, quite a few papers, both on that kind of game
00:19:39 theory and also like this, this kind of understanding people’s social value orientation, for example,
00:19:45 you know, some people are aggressive, some people not so much.
00:19:48 And, and, you know, like a robot could understand that by just looking at how people drive.
00:19:54 And as they kind of come in approach, you can actually understand, like if someone is
00:19:58 going to be aggressive or, or not as a robot and you can make certain decisions.
00:20:02 Well, in terms of predicting what they’re going to do, the hard question is you as a
00:20:07 robot, should you be aggressive or not when faced with an aggressive robot?
00:20:13 Right now it seems like aggressive is a very dangerous thing to do because it’s costly
00:20:19 from a societal perspective, how you’re perceived.
00:20:22 People are not very accepting of aggressive robots in modern society.
00:20:27 I think that’s accurate.
00:20:28 So that is really is.
00:20:31 And so I’m not entirely sure like how to have to go about, but I know, I know for a fact
00:20:36 that how these robots interact with other people in there is going to be, and then interaction
00:20:41 is always going to be there.
00:20:42 I mean, you could be interacting with other vehicles or other just people kind of like
00:20:46 walking around.
00:20:48 And like I said, the moment there’s like nobody in the seat, it’s like an empty thing just
00:20:52 rolling off the street.
00:20:54 It becomes like no different than like any other thing that’s not human.
00:20:59 And so people, and maybe abuse is the wrong word, but people maybe rightfully even they
00:21:05 feel like this is a human present environment designed for humans to be, and they kind of
00:21:11 they want to own it.
00:21:13 And then the robots, they would need to understand it and they would need to respond in a certain
00:21:18 way.
00:21:19 And I think that this actually opens up like quite a few interesting societal questions
00:21:23 for us as we deploy, like we talk robots at large scale.
00:21:26 So what would happen when we try to deploy robots at large scale, I think is that we
00:21:30 can design systems in a way that they’re very efficient or we can design them that they’re
00:21:35 very sustainable, but ultimately the sustainability efficiency trade offs, like they’re going
00:21:40 to be right in there and we’re going to have to make some choices.
00:21:44 Like we’re not going to be able to just kind of put it aside.
00:21:47 So for example, we can be very aggressive and we can reduce transportation delays, increase
00:21:52 capacity of transportation, or we can be a lot nicer and allow other people to kind of
00:21:58 quote unquote own the environment and live in a nice place and then efficiency will drop.
00:22:04 So when you think about it, I think sustainability gets attached to energy consumption or environmental
00:22:10 impact immediately.
00:22:11 And those are there, but like livability is another sustainability impact.
00:22:15 So you create an environment that people want to live in.
00:22:19 And if, if, if robots are going around being aggressive and you don’t want to live in that
00:22:23 environment, maybe, however, you should note that if you’re not being aggressive, then,
00:22:27 you know, you’re probably taking up some, some delays in transportation and this and
00:22:31 that.
00:22:32 So you’re always balancing that.
00:22:34 And I think this, this choice has always been there in transportation, but I think the more
00:22:38 autonomy comes in, the more explicit the choice becomes.
00:22:42 Yeah.
00:22:43 And when it becomes explicit, then we can start to optimize it and then we’ll get to
00:22:47 ask the very difficult societal questions of what do we value more, efficiency or sustainability?
00:22:53 It’s kind of interesting.
00:22:56 I think we’re going to have to like, I think that the interesting thing about like the
00:23:00 whole autonomous vehicles question, I think is also kind of, um, I think a lot of times,
00:23:06 you know, we have, we have focused on technology development, like hundreds of years and you
00:23:12 know, the products somehow followed and then, you know, we got to make these choices and
00:23:15 things like that.
00:23:16 So this is, this is a good time that, you know, we even think about, you know, autonomous
00:23:20 taxi type of deployments and the systems that would evolve from there.
00:23:25 And you realize the business models are different.
00:23:28 The impact on architecture is different, urban planning, you get into like regulations, um,
00:23:35 and then you get into like these issues that you didn’t think about before, but like sustainability
00:23:39 and ethics is like right in the middle of it.
00:23:41 I mean, even testing autonomous vehicles, like think about it, you’re testing autonomous
00:23:45 vehicles in human present environments.
00:23:47 I mean, uh, the risk may be very small, but still, you know, it’s, it’s a, it’s a, it’s,
00:23:52 it’s a, you know, strictly greater than zero risk that you’re putting people into.
00:23:56 And so then you have that innovation, you know, risk trade off that you’re, you’re in
00:24:01 that somewhere.
00:24:02 Um, and we, we understand that pretty now that pretty well now is that if we don’t test
00:24:08 the, at least the, the development will be slower.
00:24:12 I mean, it doesn’t mean that we’re not going to be able to develop.
00:24:15 I think it’s going to be pretty hard actually.
00:24:17 Maybe we can, we don’t, we don’t, I don’t know.
00:24:18 But the thing is that those kinds of trade offs we already are making and as these systems
00:24:24 become more ubiquitous, I think those trade offs will just really hit.
00:24:30 So you are one of the founders of Optimus Ride and autonomous vehicle company.
00:24:34 We’ll talk about it, but let me on that point ask maybe a good examples, keeping Optimus
00:24:43 Ride out, out of this question, uh, sort of exemplars of different strategies on the spectrum
00:24:51 of innovation and safety or caution.
00:24:56 So like Waymo, Google self driving car Waymo represents maybe a more cautious approach.
00:25:03 And then you have Tesla on the other side headed by Elon Musk that represents a more,
00:25:10 however, which adjective you want to use, aggressive, innovative, I don’t know.
00:25:14 But uh, what, what do you think about the difference in the two strategies in your view?
00:25:21 What’s more likely, what’s needed and is more likely to succeed in the short term and in
00:25:27 the long term?
00:25:30 Definitely some sort of a balance is, is kind of the right way to go.
00:25:33 But I do think that the thing that is the most important is actually like an informed
00:25:38 public.
00:25:39 So I don’t, I don’t mind, you know, I personally, like if I were in some place, I wouldn’t mind
00:25:45 so much like taking a certain amount of risk, um, some other people might.
00:25:52 And so I think the key is for people to be informed and so that they can, ideally they
00:25:57 can make a choice.
00:25:59 In some cases, that kind of choice, um, making that unanimously is of course very hard.
00:26:06 But I don’t think it’s actually that hard to inform people.
00:26:10 So I think in, in, in one case, like for example, even the Tesla approach, um, I don’t know,
00:26:17 it’s hard to judge how informed it is, but it is somewhat informed.
00:26:20 I mean, you know, things kind of come out.
00:26:21 I think people know what they’re taking and things like that and so on.
00:26:25 But I think the, the underlying, um, I do think that these two companies are a little
00:26:30 bit kind of representing like the, of course they, you know, one of them seems a bit safer
00:26:36 or the other one, or, you know, um, whatever the objective for that is, and the other one
00:26:40 seems more aggressive or whatever the objective for that is.
00:26:43 But, but I think, you know, when you turn the tables, they’re actually, there are two
00:26:47 other orthogonal dimensions that these two are focusing on.
00:26:50 On the one hand for Waymo, I can see that, you know, they’re, I mean, um, they, I think
00:26:55 they a little bit see it as research as well.
00:26:57 So they kind of, they don’t, I’m not sure if they’re like really interested in like
00:27:00 an immediate, um, product, um, you know, they, they talk about it.
00:27:05 Um, sometimes there’s some pressure to talk about it.
00:27:08 So they, they kind of go for it, but I think, um, I think that they’re thinking, um, maybe
00:27:13 in the back of their minds, maybe they don’t put it this way, but I think they, they realize
00:27:17 that we’re building like a new engine.
00:27:20 It’s kind of like call it the AI engine or whatever that is.
00:27:23 And you know, an autonomous vehicles is a very interesting embodiment of that engine
00:27:27 that allows you to understand where the ego vehicle is, the ego thing is where everything
00:27:32 else is, what everything else is going to do and how do you react, how do you actually,
00:27:36 you know, interact with humans the right way?
00:27:38 How do you build these systems?
00:27:39 And I think, uh, they, they want to know that they want to understand that.
00:27:43 And so they keep going and doing that.
00:27:45 And so on the other dimension, Tesla is doing something interesting.
00:27:48 I mean, I think that they have a good product.
00:27:50 People use it.
00:27:51 I think that, you know, like it’s, it’s not for me, um, but I can totally see people,
00:27:55 people like it and, and people, I think they have a good product outside of automation,
00:27:59 but I was just referring to the, the, the automation itself.
00:28:02 I mean, you know, like it, it kind of drives itself.
00:28:05 You still have to be kind of, um, you still have to pay attention to it, right?
00:28:09 Well, you know, um, people seem to use it.
00:28:12 So it works for something.
00:28:14 And so people, I think people are willing to pay for it.
00:28:16 People are willing to buy it.
00:28:17 I think it, uh, it’s, it’s one of the other reasons why people buy a Tesla car.
00:28:22 Maybe one of those reasons is Elon Musk is the CEO and you know, he seems like a visionary
00:28:26 person.
00:28:27 That’s what people think.
00:28:28 He’s a great person.
00:28:29 And so that adds like 5k to the value of the car and then maybe another 5k is the autopilot
00:28:34 and, and you know, it’s, it’s useful.
00:28:35 I mean, it’s, um, useful in the sense that like people are using it.
00:28:40 And so I can see Tesla and sure, of course they want to be visionary.
00:28:45 They want to kind of put out a certain approach and they may actually get there.
00:28:48 Um, but I think that there’s also a primary benefit of doing all these updates and rolling
00:28:54 it out because, you know, people pay for it and it’s, it’s, you know, it’s basic, you
00:28:59 know, demand, supply market and people like it.
00:29:03 They’re happy to pay another 5k, 10k for that novelty or whatever that is, um, they, and
00:29:09 they use it.
00:29:10 It’s not like they get it and they try it a couple of times as a novelty, but they use
00:29:14 it a lot of the time.
00:29:15 And so I think that’s what Tesla is doing.
00:29:17 It’s actually pretty different.
00:29:18 Like they, they are on pretty orthogonal dimensions of what kind of things that they’re building.
00:29:23 They are using the same AI engine.
00:29:25 So it’s very possible that, you know, they’re both going to be, um, sort of one day, um,
00:29:31 kind of using a similar, almost like an internal internal combustion engine.
00:29:34 It’s a very bad metaphor, but similar internal combustion engine, and maybe one of them is
00:29:39 building like a car.
00:29:41 The other one is building a truck or something.
00:29:42 So ultimately the use case is very different.
00:29:45 So you, like I said, are one of the founders of Optimus, right?
00:29:48 Let’s take a step back.
00:29:49 That’s one of the success stories in the autonomous vehicle space.
00:29:54 It’s a great autonomous vehicle company.
00:29:56 Let’s go from the very beginning.
00:29:58 What does it take to start an autonomous vehicle company?
00:30:02 How do you go from idea to deploying vehicles like you are in a few, a bunch of places,
00:30:06 including New York?
00:30:08 I would say that I think that, you know, what happened to us is it was, was the following.
00:30:12 I think, um, we realized a lot of kind of talk in the autonomous vehicle industry back
00:30:18 in like 2014, even when we wanted to kind of get started.
00:30:22 Um, and, and I don’t know, like I, I kind of, I would hear things like fully autonomous
00:30:29 vehicles, two years from now, three years from now, I kind of never bought it.
00:30:33 Um, you know, I was a part of, um, MIT’s urban challenge entry.
00:30:37 Um, it kind of like, it has an interesting history.
00:30:40 So, um, I did in, in, in college and in high school, sort of a lot of mathematically oriented
00:30:46 work.
00:30:47 I mean, I kind of, you know, at some point, uh, it kind of hit me.
00:30:50 I wanted to build something.
00:30:52 And so I came to MIT’s mechanical engineering program and I now realize, I think my advisor
00:30:57 hired me because I could do like really good math, but I told him that, no, no, no, I want
00:31:02 to work on that urban challenge car.
00:31:04 I want to build the autonomous car.
00:31:06 And I think that was, that was kind of like a process where we really learned, I mean,
00:31:10 what the challenges are and what kind of limitations are we up against, you know, like having the
00:31:16 limitations of computers or understanding human behavior, there’s so many of these things.
00:31:21 And I think it just kind of didn’t.
00:31:23 And so, so we said, Hey, you know, like, why don’t we take a more like a market based approach?
00:31:29 So we focus on a certain kind of market and we build a system for that.
00:31:35 What we’re building is not so much of like an autonomous vehicle only, I would say.
00:31:38 So we build full autonomy into the vehicles.
00:31:41 But, you know, the way we kind of see it is that we think that the approach should actually
00:31:47 involve humans operating them, not just, just not sitting in the vehicle.
00:31:52 And I think today, what we have is today, we have one person operate one vehicle, no
00:31:58 matter what that vehicle, it could be a forklift, it could be a truck, it could be a car, whatever
00:32:03 that is.
00:32:04 And we want to go from that to 10 people operate 50 vehicles.
00:32:09 How do we do that?
00:32:10 If you’re referring to a world of maybe perhaps teleoperation, so can you just say what it
00:32:16 means for 10?
00:32:17 It might be confusing for people listening.
00:32:19 What does it mean for 10 people to control 50 vehicles?
00:32:23 That’s a good point.
00:32:24 So I think it’s, I very deliberately didn’t call it teleoperation because what people
00:32:28 think then is that people think, away from the vehicle sits a person, sees like maybe
00:32:35 puts on goggles or something, VR and drives the car.
00:32:38 So that’s not at all what we mean, but we mean the kind of intelligence whereby humans
00:32:44 are in control, except in certain places, the vehicles can execute on their own.
00:32:49 And so imagine like, like a room where people can see what the other vehicles are doing
00:32:54 and everything.
00:32:56 And you know, there will be some people who are more like, more like air traffic controllers,
00:33:01 call them like AV controllers.
00:33:04 And so these AV controllers would actually see kind of like a whole map and they would
00:33:09 understand where vehicles are really confident and where they kind of need a little bit more
00:33:15 help.
00:33:16 And the help shouldn’t be for safety.
00:33:19 Help should be for efficiency.
00:33:21 Vehicles should be safe no matter what.
00:33:22 If you had zero people, they could be very safe, but they’d be going five miles an hour.
00:33:27 And so if you want them to go around 25 miles an hour, then you need people to come in and,
00:33:32 and for example, you know, the vehicle come to an intersection and the vehicle can say,
00:33:38 you know, I can wait.
00:33:39 I can inch forward a little bit, show my intent, or I can turn left.
00:33:45 And right now it’s clear I can turn, I know that, but before you give me the go, I won’t.
00:33:50 And so that’s one example.
00:33:51 This doesn’t mean necessarily we’re doing that actually.
00:33:53 I think, I think if you go down all the, all that much detail that every intersection you’re
00:33:59 kind of expecting a person to press a button, then I don’t think you’ll get the efficiency
00:34:03 benefits you want.
00:34:04 You need to be able to kind of go around and be able to do these things.
00:34:07 But, but I think you need people to be able to set high level behavior to vehicles.
00:34:12 That’s the other thing with autonomous vehicles, you know, I think a lot of people kind of
00:34:15 think about it as follows.
00:34:16 I mean, this happens with technology a lot.
00:34:18 You know, you think, all right, so I know about cars and I heard robots.
00:34:23 So I think how this is going to work out is that I’m going to buy a car, press a button
00:34:28 and it’s going to drive itself.
00:34:29 And when is that going to happen?
00:34:31 You know, and people kind of tend to think about it that way, but when you think about
00:34:34 what really happens is that something comes in in a way that you didn’t even expect.
00:34:40 If asked, you might have said, I don’t think I need that, or I don’t think it should be
00:34:43 that and so on.
00:34:45 And then, and then that, that becomes the next big thing, coding code.
00:34:49 And so I think that this kind of different ways of humans operating vehicles could be
00:34:54 really powerful.
00:34:55 I think that sooner than later, we might open our eyes up to a world in which you go around
00:35:01 walk in a mall and there’s a bunch of security robots that are exactly operated in this way.
00:35:06 You go into a factory or a warehouse, there’s a whole bunch of robots that are playing exactly
00:35:10 in this way.
00:35:11 You go to a, you go to the Brooklyn Navy Yard, you see a whole bunch of autonomous vehicles,
00:35:17 Optimus Ride, and they’re operated maybe in this way.
00:35:21 But I think people kind of don’t see that.
00:35:22 I sincerely think that there’s a possibility that we may almost see like a whole mushrooming
00:35:28 of this technology in all kinds of places that we didn’t expect before.
00:35:33 And that may be the real surprise.
00:35:35 And then one day when your car actually drives itself, it may not be all that much of a surprise
00:35:40 at all because you see it all the time.
00:35:42 You interact with them, you take the Optimus Ride, hopefully that’s your choice.
00:35:47 And then you hear a bunch of things, you go around, you interact with them.
00:35:52 I don’t know.
00:35:53 Like you have a little delivery vehicle that goes around the sidewalks and delivers you
00:35:56 things and then you take it, it says thank you.
00:35:59 And then you get used to that and one day your car actually drives itself and the regulation
00:36:04 goes by and you can hit the button of sleep and it wouldn’t be a surprise at all.
00:36:08 I think that may be the real reality.
00:36:10 So there’s going to be a bunch of applications that pop up around autonomous vehicles, some
00:36:17 of which, maybe many of which we don’t expect at all.
00:36:20 So if we look at Optimus Ride, what do you think, you know, the viral application, the
00:36:27 one that like really works for people in mobility, what do you think Optimus Ride will connect
00:36:33 with in the near future first?
00:36:36 I think that the first places that I like to target honestly is like these places where
00:36:42 transportation is required within an environment, like people typically call it geofence.
00:36:46 So you can imagine like roughly two mile by two mile could be bigger, could be smaller
00:36:51 type of an environment.
00:36:53 And there’s a lot of these kinds of environments that are typically transportation deprived.
00:36:57 The Brooklyn Navy Yard that, you know, we’re in today, we’re in a few different places,
00:37:01 but that was the one that was last publicized and that’s a good example.
00:37:06 So there’s not a lot of transportation there and you wouldn’t expect like, I don’t know,
00:37:11 I think maybe operating an Uber there ends up being sort of a little too expensive or
00:37:15 when you compare it with operating Uber elsewhere, elsewhere becomes the priority and these places
00:37:23 become totally transportation deprived.
00:37:26 And then what happens is that, you know, people drive into these places and to go from point
00:37:29 A to point B inside this place within that day, they use their cars.
00:37:35 And so we end up building more parking for them to, for example, take their cars and
00:37:40 go to the lunch place.
00:37:43 And I think that one of the things that can be done is that, you know, you can put in
00:37:46 efficient, safe, sustainable transportation systems into these types of places first.
00:37:53 And I think that, you know, you could deliver mobility in an affordable way, affordable,
00:37:59 accessible, you know, sustainable way.
00:38:03 But I think what also enables is that this kind of effort, money, area, land that we
00:38:08 spend on parking, you could reclaim some of that.
00:38:12 And that is on the order of like, even for a small environment like two mile by two mile,
00:38:17 it doesn’t have to be smack in the middle of New York.
00:38:19 I mean, anywhere else you’re talking tens of millions of dollars.
00:38:23 If you’re smack in the middle of New York, you’re looking at billions of dollars of savings
00:38:26 just by doing that.
00:38:28 And that’s the economic part of it.
00:38:29 And there’s a societal part, right?
00:38:31 I mean, just look around.
00:38:32 I mean the places that we live are like built for cars.
00:38:38 It didn’t look like this just like a hundred years ago, like today, no one walks in the
00:38:42 middle of the street.
00:38:44 It’s for cars.
00:38:45 No one tells you that growing up, but you grow into that reality.
00:38:49 And so sometimes they close the road.
00:38:51 It happens here, you know, like the celebration, they close the road.
00:38:54 Still people don’t walk in the middle of the road, like just walk in the middle and people
00:38:57 don’t.
00:38:58 But I think it has so much impact, the car in the space that we have.
00:39:04 And I think we talked about sustainability, livability.
00:39:07 I mean, ultimately these kinds of places that parking spots at the very least could change
00:39:12 into something more useful or maybe just like park areas, recreational.
00:39:16 And so I think that’s the first thing that we’re targeting.
00:39:19 And I think that we’re getting like a really good response, both from an economic societal
00:39:23 point of view, especially places that are a little bit forward looking.
00:39:27 And like, for example, Brooklyn Navy Yard, they have tenants.
00:39:31 There’s distinct direct call like new lab.
00:39:33 It’s kind of like an innovation center.
00:39:35 There’s a bunch of startups there.
00:39:36 And so, you know, you get those kinds of people and, you know, they’re really interested
00:39:40 in sort of making that environment more livable.
00:39:44 And these kinds of solutions that Optimus Ride provides almost kind of comes in and
00:39:49 becomes that.
00:39:50 And many of these places that are transportation deprived, you know, they have, they actually
00:39:56 rent shuttles.
00:39:57 And so, you know, you can ask anybody, the shuttle experience is like terrible.
00:40:03 People hate shuttles.
00:40:05 And I can tell you why.
00:40:06 Because, you know, like the driver is very expensive in a shuttle business.
00:40:11 So what makes sense is to attach 20, 30 seats to a driver.
00:40:15 And a lot of people have this misconception.
00:40:17 They think that shuttles should be big.
00:40:19 Sometimes we get that at Optimus Ride.
00:40:20 We tell them, we’re going to give you like four seaters, six seaters.
00:40:23 And we get asked like, how about like 20 seaters?
00:40:25 I’m like, you know, you don’t need 20 seaters.
00:40:27 You want to split up those seats so that they can travel faster and the transportation delays
00:40:32 would go down.
00:40:33 That’s what you want.
00:40:34 If you make it big, not only you will get delays in transportation, but you won’t have
00:40:39 an agile vehicle.
00:40:40 It will take a long time to speed up, slow down and so on.
00:40:44 You need to climb up to the thing.
00:40:45 So it’s kind of like really hard to interact with.
00:40:48 And scheduling too, perhaps when you have more smaller vehicles, it becomes closer to
00:40:53 Uber where you can actually get a personal, I mean, just the logistics of getting the
00:40:58 vehicle to you becomes easier when you have a giant shuttle.
00:41:02 There’s fewer of them and it probably goes on a route, a specific route that is supposed
00:41:07 to hit.
00:41:08 And when you go on a specific route and all seats travel together versus, you know, you
00:41:13 have a whole bunch of them.
00:41:14 You can imagine the route you can still have, but you can imagine you split up the seats
00:41:19 and instead of, you know, them traveling, like, I don’t know, a mile apart, they could
00:41:24 be like, you know, half a mile apart if you split them into two.
00:41:28 That basically would mean that your delays, when you go out, you won’t wait for them for
00:41:34 a long time.
00:41:35 And that’s one of the main reasons, or you don’t have to climb up.
00:41:37 The other thing is that I think if you split them up in a nice way, and if you can actually
00:41:41 know where people are going to be somehow, you don’t even need the app.
00:41:46 A lot of people ask us the app, we say, why don’t you just walk into the vehicle?
00:41:50 How about you just walk into the vehicle, it recognizes who you are and it gives you
00:41:54 a bunch of options of places that you go and you just kind of go there.
00:41:57 I mean, people kind of also internalize the apps.
00:42:01 Everybody needs an app.
00:42:02 It’s like, you don’t need an app.
00:42:03 You just walk into the thing.
00:42:05 But I think one of the things that, you know, we really try to do is to take that shuttle
00:42:10 experience that no one likes and tilt it into something that everybody loves.
00:42:14 And so I think that’s another important thing.
00:42:17 I would like to say that carefully, just like teleoperation, like we don’t do shuttles.
00:42:21 You know, we’re really kind of thinking of this as a system or a network that we’re designing.
00:42:28 But ultimately, we go to places that would normally rent a shuttle service that people
00:42:33 wouldn’t like as much and we want to tilt it into something that people love.
00:42:37 So you’ve mentioned this earlier, but how many Optimus ride vehicles do you think would
00:42:42 be needed for any person in Boston or New York, if they step outside, there will be,
00:42:50 this is like a mathematical question, there’ll be two Optimus ride vehicles within line of
00:42:55 sight.
00:42:56 Is that the right number to, well, at least one.
00:42:58 For example, that’s the density.
00:43:01 So meaning that if you see one vehicle, you look around, you see another one too.
00:43:07 Imagine like, you know, Tesla would tell you they collect a lot of data.
00:43:11 Do you see that with Tesla?
00:43:12 Like you just walk around and you look around, you see Tesla?
00:43:16 Probably not.
00:43:17 Very specific areas of California, maybe.
00:43:19 You’re right.
00:43:21 Like there’s a couple of zip codes that, you know, but I think that’s kind of important
00:43:25 because you know, like maybe the couple of zip codes, the one thing that we kind of depend
00:43:29 on and I’ll get to your question in a second, but now like we’re taking a lot of tensions
00:43:33 today.
00:43:34 And so I think that this is actually important.
00:43:38 People call this data density or data velocity.
00:43:41 So it’s very good to collect data in a way that, you know, you see the same place so
00:43:46 many times.
00:43:47 Like you can drive 10,000 miles around the country or you drive 10,000 miles in a confined
00:43:53 environment.
00:43:54 You’ll see the same intersection hundreds of times.
00:43:56 And when it comes to predicting what people are going to do in that specific intersection,
00:44:01 you become really good at it versus if you draw in like 10,000 miles around the country,
00:44:05 you’ve seen that only once.
00:44:06 And so trying to predict what people do becomes hard.
00:44:10 And I think that, you know, you said what is needed, it’s tens of thousands of vehicles.
00:44:14 You know, you really need to be like a specific fractional vehicle.
00:44:17 Like for example, in good times in Singapore, you can go and you can just grab a cab and
00:44:23 they are like, you know, 10%, 20% of traffic, those taxis.
00:44:29 Ultimately that’s where you need to get to.
00:44:31 So that, you know, you get to a certain place where you really, the benefits really kick
00:44:36 off in like orders of magnitude type of a point.
00:44:40 But once you get there, you actually get the benefits.
00:44:43 And you can certainly carry people.
00:44:44 I think that’s one of the things people really don’t like to wait for themselves.
00:44:51 But for example, they can wait a lot more for the goods if they order something.
00:44:55 Like you’re sitting at home and you want to wait half an hour.
00:44:57 That sounds great.
00:44:58 People will say it’s great.
00:44:59 You want to, you’re going to take a cab, you’re waiting half an hour.
00:45:02 Like that’s crazy.
00:45:03 You don’t want to wait that much.
00:45:06 But I think, you know, you can, I think really get to a point where the system at peak times
00:45:11 really focuses on kind of transporting humans around.
00:45:14 And then it’s really, it’s a good fraction of the traffic to the point where, you know,
00:45:18 you go, you look around and there’s something there and you just kind of basically get in
00:45:23 there and it’s already waiting for you or something like that.
00:45:27 And then you take it.
00:45:28 If you do it at that scale, like today, for instance, Uber, if you talk to a driver, right?
00:45:35 I mean, Uber takes a certain cut.
00:45:37 It’s a small cut.
00:45:39 Or drivers would argue that it’s a large cut, but you know, it’s when you look at the grand
00:45:44 scheme of things, most of that money that you pay Uber kind of goes to the driver.
00:45:50 And if you talk to the driver, the driver will claim that most of it is their time.
00:45:54 You know, it’s not spent on gas.
00:45:56 They think it’s not spent on the car per se as much.
00:46:01 It’s like their time.
00:46:02 And if you didn’t have a person driving, or if you’re in a scenario where, you know, like
00:46:07 0.1 person is driving the car, a fraction of a person is kind of operating the car because
00:46:14 you know, you want to operate several.
00:46:17 If you’re in that situation, you realize that the internal combustion engine type of cars
00:46:21 are very inefficient.
00:46:23 You know, we build them to go on highways, they pass crash tests.
00:46:26 They’re like really heavy.
00:46:27 They really don’t need to be like 25 times the weight of its passengers or, you know,
00:46:32 like area wise and so on.
00:46:35 But if you get through those inefficiencies and if you really build like urban cars and
00:46:39 things like that, I think the economics really starts to check out.
00:46:43 Like to the point where, I mean, I don’t know, you may be able to get into a car and it may
00:46:47 be less than a dollar to go from A to B. As long as you don’t change your destination,
00:46:52 you just pay 99 cents and go there.
00:46:55 If you share it, if you take another stop somewhere, it becomes a lot better.
00:47:00 You know, these kinds of things, at least for models, at least for mathematics and theory,
00:47:05 they start to really check out.
00:47:07 So I think it’s really exciting what Optimus Ride is doing in terms of it feels the most
00:47:12 reachable, like it’ll actually be here and have an impact.
00:47:15 Yeah, that is the idea.
00:47:17 And if we contrast that, again, we’ll go back to our old friends, Waymo and Tesla.
00:47:23 So Waymo seems to have sort of technically similar approaches as Optimus Ride, but a
00:47:34 different, they’re not as interested as having impact today.
00:47:41 They have a longer term sort of investments, almost more of a research project still, meaning
00:47:47 they’re trying to solve, as far as I understand, maybe you can differentiate, but they seem
00:47:53 to want to do more unrestricted movement, meaning move from A to B where A to B is all
00:48:00 over the place versus Optimus Ride is really nicely geofenced and really sort of established
00:48:07 mobility in a particular environment before you expand it.
00:48:11 And then Tesla is like the complete opposite, which is, you know, the entirety of the world
00:48:17 actually is going to be automated.
00:48:21 Highway driving, urban driving, every kind of driving, you know, you kind of creep up
00:48:26 to it by incrementally improving the capabilities of the autopilot system.
00:48:33 So when you contrast all of these, and on top of that, let me throw a question that
00:48:37 nobody likes, but is a timeline.
00:48:42 When do you think each of these approaches, loosely speaking, nobody can predict the future,
00:48:47 will see mass deployment?
00:48:49 So Elon Musk predicts the craziest approach is, I’ve heard figures like at the end of
00:48:56 this year, right?
00:48:58 So that’s probably wildly inaccurate, but how wildly inaccurate is it?
00:49:06 I mean, first thing to lay out, like everybody else, it’s really hard to guess.
00:49:11 I mean, I don’t know where Tesla can look at or Elon Musk can look at and say, hey,
00:49:18 you know, it’s the end of this year.
00:49:19 I mean, I don’t know what you can look at.
00:49:22 You know, even the data that, I mean, if you look at the data, even kind of trying to extrapolate
00:49:30 the end state without knowing what exactly is going to go, especially for like a machine
00:49:34 learning approach.
00:49:35 I mean, it’s just kind of very hard to predict.
00:49:39 But I do think the following does happen.
00:49:41 I think a lot of people, you know, what they do is that there’s something that I called
00:49:46 a couple times time dilation in technology prediction happens.
00:49:51 Let me try to describe a little bit.
00:49:53 There’s a lot of things that are so far ahead, people think they’re close.
00:49:57 And there’s a lot of things that are actually close.
00:50:00 People think it’s far ahead.
00:50:02 People try to kind of look at a whole landscape of technology development, admittedly, it’s
00:50:07 chaos.
00:50:08 Anything can happen in any order at any time.
00:50:10 And there’s a whole bunch of things in there.
00:50:12 People take it, clamp it, and put it into the next three years.
00:50:17 And so then what happens is that there’s some things that maybe can happen by the end of
00:50:21 the year or next year and so on.
00:50:23 And they push that into like few years ahead, because it’s just hard to explain.
00:50:28 And there are things that are like, we’re looking at 20 years more, maybe, you know,
00:50:33 hopefully in my lifetime type of things, because, you know, we don’t know.
00:50:37 I mean, we don’t know how hard it is even.
00:50:40 Like that’s a problem.
00:50:41 We don’t know like if some of these problems are actually AI complete, like, we have no
00:50:45 idea what’s going on.
00:50:48 And you know, we take all of that and then we clump it.
00:50:51 And then we say three years from now.
00:50:55 And then some of us are more optimistic.
00:50:57 So they’re shooting at the end of the year and some of us are more realistic.
00:51:00 They say like five years, but you know, we all, I think it’s just hard to know.
00:51:06 And I think trying to predict like products ahead two, three years, it’s hard to know
00:51:12 in the following sense.
00:51:14 You know, like we typically say, okay, this is a technology company, but sometimes, sometimes
00:51:19 really you’re trying to build something where the technology does, like there’s a technology
00:51:22 gap, you know, like, and Tesla had that with electric vehicles, you know, like when they
00:51:29 first started, they would look at a chart much like a Moore’s law type of chart.
00:51:33 And they would just kind of extrapolate that out and they’d say, we want to be here.
00:51:37 What’s the technology to get that?
00:51:38 We don’t know.
00:51:39 It goes like this.
00:51:40 We’re just going to, you know, keep going with AI that goes into the cars.
00:51:46 We don’t even have that.
00:51:47 Like we don’t, we can’t, I mean, what can you quantify, like what kind of chart are
00:51:51 you looking at?
00:51:52 You know?
00:51:53 But so, but so I think when there’s that technology gap, it’s just kind of really hard to predict.
00:51:58 So now I realize I talked like five minutes and avoid your question.
00:52:01 I didn’t tell you anything about that and it was very skillfully done.
00:52:05 That was very well done.
00:52:07 And I don’t think you, I think you’ve actually argued that it’s not a use, even any answer
00:52:10 you provide now is not that useful.
00:52:12 It’s going to be very hard.
00:52:13 There’s one thing that I really believe in and, um, and you know, this is not my idea
00:52:17 and it’s been, you know, discussed several times, but, but this, um, this, this kind
00:52:22 of like something like a startup, um, or, or a kind of an innovative company, um, including
00:52:29 definitely may one, may Waymo, Tesla, maybe even some of the other big companies that
00:52:33 are kind of trying things.
00:52:34 This kind of like iterated learning is very important.
00:52:38 The fact that we’re over there and we’re trying things and so on, I think that’s, um, that
00:52:43 that’s important.
00:52:44 We try to understand.
00:52:45 And, and I think that, you know, the code in code Silicon Valley has done that with
00:52:49 business models pretty well.
00:52:52 And now I think we’re trying to get to do it, but there’s a literal technology gap.
00:52:56 I mean, before, like, you know, you’re trying to build, I’m not trying to, you know, I think
00:53:01 these companies are building great technology to, for example, enable internet search to
00:53:06 do it so quickly.
00:53:07 And that kind of didn’t, didn’t, wasn’t there so much, but at least like it was a kind of
00:53:11 a technology that you could predict to some degree and so on.
00:53:14 And now we’re just kind of trying to build, you know, things that it’s kind of hard to
00:53:18 quantify what kind of a metric are we looking at?
00:53:21 So psychologically as a sort of a, as a leader of graduate students and at Optimus ride a
00:53:28 bunch of brilliant engineers, just curiosity, psychologically, do you think it’s good to
00:53:35 think that, you know, whatever technology gap we’re talking about can be closed by the
00:53:42 end of the year or do you, you know, cause we don’t know.
00:53:46 So the way, do you want to say that everything is going to improve exponentially to yourself
00:53:54 and to others around you as a leader, or do you want to be more sort of maybe not cynical,
00:54:01 but I don’t want to use realistic cause it’s hard to predict, but yeah, maybe more cynical,
00:54:07 pessimistic about the ability to close that gap.
00:54:11 Yeah.
00:54:12 I think that, you know, going back, I think that iterated learning is like key that, you
00:54:16 know, you’re out there, you’re running experiments to learn.
00:54:19 And that doesn’t mean sort of like, you know, like, like your Optimus ride, you’re kind
00:54:22 of doing something, but like in an environment, but like what Tesla is doing, I think is also
00:54:28 kind of like this, this kind of notion.
00:54:30 And, and, you know, people can go around and say like, you know, this year, next year,
00:54:34 the other year and so on.
00:54:35 But, but I think that the nice thing about it is that they’re out there, they’re pushing
00:54:39 this technology in.
00:54:40 I think what they should do more of, I think that kind of informed people about what kind
00:54:45 of technology that they’re providing, you know, the good and the bad.
00:54:48 And then, you know, not just sort of, you know, it works very well, but I think, you
00:54:52 know, I’m not saying they’re not doing bad and informing, I think they’re, they’re kind
00:54:56 of trying, they, you know, they put up certain things or at the very least YouTube videos
00:55:00 comes out on, on how the summon function works every now and then, and, and, you know, people
00:55:04 get informed and so that, that kind of cycle continues, but I, you know, I, I admire it.
00:55:10 I think they’re kind of go out there and they, they do great things.
00:55:13 They do their own kind of experiment.
00:55:14 I think we do our own and I think we’re closing some similar technology gaps, but some also
00:55:20 some are orthogonal as well.
00:55:22 You know, I think like, like we talked about, you know, people being remote, like it’s something
00:55:27 or in the kind of environments that we’re in or think about a Tesla car, maybe, maybe
00:55:31 you can enable it one day.
00:55:32 Like there’s, you know, low traffic, like you’re kind of the stop on go motion, you
00:55:36 just hit the button and the, you can release, or maybe there’s another lane that you can
00:55:41 pass into, you go in that.
00:55:42 I think they can enable these kinds of, I believe it.
00:55:45 And so I think that that part, that is really important and that is really key.
00:55:51 And beyond that, I think, you know, when is it exactly going to happen and, and, and so
00:55:57 on.
00:55:58 I mean it’s like I said, it’s very hard to predict.
00:56:02 And I would, I would imagine that it would be good to do some sort of like a, like a
00:56:07 one or two year plan when it’s a little bit more predictable that, you know, the technology
00:56:12 gaps you close and, and the, and the kind of sort of product that would ensue.
00:56:18 So I know that from Optimus ride or, you know, other companies that I get involved in.
00:56:22 I mean, at some point you find yourself in a situation where you’re trying to build a
00:56:27 product and, and people are investing in that, in that, you know, building effort and those
00:56:35 investors that they do want to know as they compare the investments they want to make,
00:56:39 they do want to know what happens in the next one or two years.
00:56:42 And I think that’s good to communicate that.
00:56:44 But I think beyond that, it becomes, it becomes a vision that we want to get to someday and
00:56:48 saying five years, 10 years, I don’t think it means anything.
00:56:52 But iterative learning is key to do and learn.
00:56:56 I think that is key.
00:56:57 You know, I got to sort of throw back right at you criticism in terms of, you know, like
00:57:03 Tesla or somebody communicating, you know, how someone works and so on.
00:57:07 I got a chance to visit Optimus ride and you guys are doing some awesome stuff and yet
00:57:12 the internet doesn’t know about it.
00:57:14 So you should also communicate more showing off, you know, showing off some of the awesome
00:57:20 stuff, the stuff that works and stuff that doesn’t work.
00:57:22 I mean, it’s just the stuff I saw with the tracking of different objects and pedestrians.
00:57:27 So I mean, incredible stuff going on there.
00:57:30 Maybe it’s just the nerd in me, but I think the world would love to see that kind of stuff.
00:57:34 Yeah.
00:57:35 That’s, that’s well taken.
00:57:36 Um, you know, I, I should say that it’s not like, you know, we, we, we weren’t able to,
00:57:41 I think we made a decision at some point, um, that decision did involve me quite a bit
00:57:46 on kind of, um, uh, sort of doing this in kind of coding code stealth mode for a bit.
00:57:53 Um, but I think that, you know, we’ll, we’ll open it up quite a lot more.
00:57:56 And I think that we are also at Optimus ride kind of hitting, um, when you have new era,
00:58:02 um, you know, we’re, we’re, we’re big now, we’re doing a lot of interesting things and
00:58:06 I think, you know, some of the deployments that we’ve kind of announced were some of
00:58:10 the first bits, bits of, um, information that we kind of put out into the world.
00:58:16 We’ll also put out our technology, a lot of the things that we’ve been developing is really
00:58:20 amazing.
00:58:21 And then, you know, we’re, we’re gonna, we’re gonna start putting that out now.
00:58:24 We’re especially interested in sort of like, um, being able to work with the best people.
00:58:28 And I think, and I think it’s, it’s good to not just kind of show them when they come
00:58:32 to our office for an interview, but just put it out there in terms of like, you know, get
00:58:36 people excited about what we’re doing.
00:58:39 So on the autonomous vehicle space, let me ask one last question.
00:58:43 So Elon Musk famously said that lighter is a crutch.
00:58:47 So I’ve talked to a bunch of people about it, got to ask you, you use that crutch quite
00:58:52 a bit in the DARPA days.
00:58:55 So, uh, uh, you know, and his, his idea in general, sort of, you know, more provocative
00:59:01 and fun, I think than a technical discussion, but the idea is that camera based, primarily
00:59:08 camera based systems is going to be what defines the future of autonomous vehicles.
00:59:14 So what do you think of this idea?
00:59:16 Lighter is a crutch versus primarily, uh, camera based systems.
00:59:21 First things first, I think, you know, I’m a big believer in just camera based autonomous
00:59:27 vehicle systems.
00:59:28 Um, I think that, you know, you can put in a lot of autonomy and, and you can do great
00:59:33 things.
00:59:34 And, and it’s, it’s, it’s very possible that at the time scales, like I said, we can’t
00:59:37 predict 20 years from now, like you may be able to do, do things that we’re doing today
00:59:43 only with LIDAR and then you may be able to do them just with cameras.
00:59:48 And I think that, um, you know, you, you can just, um, I, I, I think that I will put my
00:59:53 name on it too.
00:59:54 You know, there will be a time when you can only use cameras and you’ll be fine.
01:00:00 Um, at that time though, it’s very possible that, you know, you find the LIDAR system
01:00:06 as another robustifier or, or it’s so affordable that it’s stupid not to, you know, just kind
01:00:13 of put it there.
01:00:15 And I think, um, and I think we may be looking at a future like that.
01:00:20 You think we’re over relying on LIDAR right now, because we understand the better it’s
01:00:25 more reliable in many ways in terms of, from a safety perspective.
01:00:28 It’s easier to build with.
01:00:29 That’s the other, that’s the other thing.
01:00:31 I think to be very frank with you, I mean, um, you know, we’ve seen a lot of sort of
01:00:36 autonomous vehicles companies come and go and the approach has been, you know, you slap
01:00:41 a LIDAR on a car and it’s kind of easy to build with when you have a LIDAR, you know,
01:00:46 you just kind of code it up and, and you hit the button and you do a demo.
01:00:52 So I think there’s admittedly, there’s a lot of people, they focus on the LIDAR cause it’s
01:00:55 easier to build with.
01:00:57 That doesn’t mean that, you know, without the camera, just cameras, you can, uh, you
01:01:02 cannot do what they’re doing, but it’s just kind of a lot harder.
01:01:05 And so you need to have certain kinds of expertise to exploit that.
01:01:08 What we believe in and, you know, you may be seeing some of it is that, um, we believe
01:01:13 in computer vision.
01:01:14 We certainly work on computer vision and Optimus ride, uh, by a lot, like, um, and, and we’ve
01:01:19 been doing that from day one.
01:01:21 And we also believe in sensor fusion.
01:01:23 So, you know, we, we do, we have a relatively minimal use of LIDARs, but, but we do use
01:01:28 them.
01:01:29 And I think, you know, in the future, I really believe that the following sequence of events
01:01:33 may happen.
01:01:35 First things first, number one, there may be a future in which, you know, there’s like
01:01:39 cars with LIDARs and everything and the cameras, but you know, this in this 50 year ahead future,
01:01:45 they can just drive with cameras as well.
01:01:47 Especially in some isolated environments and cameras, they go and they do the thing in
01:01:52 the same future.
01:01:53 It’s very possible that, you know, the LIDARs are so cheap and frankly make the software
01:01:57 maybe, um, a little less compute intensive, uh, at the very least, or maybe less complicated
01:02:04 so that they can be certified or, or insured, they’re of their safety and things like that,
01:02:09 that it’s kind of stupid not to put the LIDAR, like, imagine this, you either put, pay money
01:02:15 for the LIDAR or you pay money for the compute.
01:02:18 And if you don’t put the LIDAR, it’s a more expensive system because you have to put in
01:02:22 a lot of compute.
01:02:23 Like, this is another possibility.
01:02:25 Um, I do think that a lot of the, um, sort of initial deployments of self driving vehicles,
01:02:30 I think they will involve LIDARs and especially either low range or short, um, either short
01:02:37 range or low resolution LIDARs are actually not that hard to build in solid state.
01:02:42 Uh, they’re still scanning, but like MEMS type of scanning LIDARs and things like that,
01:02:47 they’re like, they’re actually not that hard.
01:02:48 I think they will maybe kind of playing with the spectrum and the phase arrays that they’re
01:02:52 a little bit harder, but, but I think, um, like, you know, putting a MEMS mirror in there
01:02:57 that kind of scans the environment, it’s not hard.
01:03:00 The only thing is that, you know, you, just like with a lot of the things that we do nowadays
01:03:04 in developing technology, you hit fundamental limits of the universe, um, the speed of light
01:03:09 becomes a problem in when you’re trying to scan the environment.
01:03:12 So you don’t get either good resolution or you don’t get range.
01:03:15 Um, but, but you know, it’s still, it’s something that you can put in there affordably.
01:03:20 So let me jump back to, uh, drones.
01:03:24 You’ve, uh, you have a role in the Lockheed Martin Alpha Pilot Innovation Challenge.
01:03:30 Where, uh, teams compete in drone racing and super cool, super intense, interesting application
01:03:37 of AI.
01:03:38 So can you tell me about the very basics of the challenge and where you fit in, what your
01:03:44 thoughts are on this problem?
01:03:46 And it’s sort of echoes of the early DARPA challenge in the, through the desert that
01:03:51 we’re seeing now, now with drone racing.
01:03:53 Yeah.
01:03:54 I mean, one interesting thing about it is that, you know, people, the drone racing exists
01:03:59 as an eSport.
01:04:01 And so it’s much like you’re playing a game, but there’s a real drone going in an environment.
01:04:06 A human being is controlling it with goggles on.
01:04:08 So there’s no, it is a robot, but there’s no AI.
01:04:13 There’s no AI.
01:04:14 Yeah.
01:04:15 Human being is controlling it.
01:04:16 And so that’s already there.
01:04:17 And, um, and I’ve been interested in this problem for quite a while, actually, um, from
01:04:22 a roboticist point of view.
01:04:23 And that’s what’s happening in Alpha Pilot, which, which problem of aggressive flight
01:04:27 of aggressive flight, fully autonomous, aggressive flight.
01:04:30 Um, the problem that I’m interested, I mean, you asked about Alpha Pilot and I’ll, I’ll
01:04:34 get there in a second, but the problem that I’m interested in, I’d love to build autonomous
01:04:38 vehicles like, like drones that can go far faster than any human possibly can.
01:04:45 I think we should recognize that we as humans have, you know, limitations in how fast we
01:04:50 can process information.
01:04:52 And those are some biological limitations.
01:04:54 Like we think about this AI this way too.
01:04:56 I mean, this has been discussed a lot and this is not sort of my idea per se, but a
01:05:00 lot of people kind of think about human level AI and they think that, you know, AI is not
01:05:05 human level.
01:05:06 One day it’ll be human level and humans and AI’s, they kind of interact.
01:05:09 Um, versus I think that the situation really is that humans are at a certain place and
01:05:14 AI keeps improving and at some point it just crosses off and then, you know, it gets smarter
01:05:19 and smarter and smarter.
01:05:21 And so drone racing, the same issue.
01:05:24 Just play this game and you know, you have to like react in milliseconds and there’s
01:05:29 really, you know, you see something with your eyes and then that information just flows
01:05:34 through your brain, into your hands so that you can command it.
01:05:37 And there’s some also delays on, you know, getting information back and forth, but suppose
01:05:40 those delays didn’t exist.
01:05:41 You just, just the delay between your eye and your fingers is a delay that a robot doesn’t
01:05:49 have to have.
01:05:51 Um, so we end up building in my research group, like systems that, you know, see things at
01:05:57 a kilohertz, like a human eye would barely hit a hundred Hertz.
01:06:00 So imagine things that see stuff in slow motion, like 10 X slow motion.
01:06:07 Um, it will be very useful.
01:06:08 Like we talked a lot about autonomous cars.
01:06:10 So, um, you know, we don’t get to see it, but a hundred lives are lost every day, just
01:06:17 in the United States on traffic accidents.
01:06:19 And many of them are like known cases, you know, like the, uh, you’re coming through
01:06:24 like, uh, like a ramp going into a highway, you hit somebody and you’re off, or, you know,
01:06:29 like you kind of get confused.
01:06:30 You try to like swerve into the next lane, you go off the road and you crash, whatever.
01:06:35 And um, I think if you had enough compute in a car and a very fast camera right at the
01:06:41 time of an accident, you could use all compute you have, like you could shut down the infotainment
01:06:46 system and use that kind of computing resources instead of rendering, you use it for the kind
01:06:53 of artificial intelligence that goes in there, the autonomy.
01:06:56 And you can, you can either take control of the car and bring it to a full stop.
01:07:00 But even, even if you can’t do that, you can deliver what the human is trying to do.
01:07:04 Human is trying to change the lane, but goes off the road, not being able to do that with
01:07:08 motor skills and the eyes.
01:07:10 And you know, you can get in there and I was, there’s so many other things that you can
01:07:14 enable with what I would call high throughput computing.
01:07:17 You know, data is coming in extremely fast and in real time you have to process it.
01:07:24 And the current CPUs, however fast you clock it are typically not enough.
01:07:30 You need to build those computers from the ground up so that they can ingest all that
01:07:34 data that I’m really interested in.
01:07:36 Just on that point, just really quick is the currently what’s the bottom, like you mentioned
01:07:42 the delays in humans, is it the hardware?
01:07:45 So you work a lot with Nvidia hardware.
01:07:47 Is it the hardware or is it the software?
01:07:50 I think it’s both.
01:07:51 I think it’s both.
01:07:52 In fact, they need to be co developed I think in the future.
01:07:54 I mean, that’s a little bit what Nvidia does sort of like they almost like build the hardware
01:07:59 and then they build the neural networks and then they build the hardware back and the
01:08:02 neural networks back and it goes back and forth, but it’s that co design.
01:08:06 And I think that, you know, like we try to way back, we try to build a fast drone that
01:08:11 could use a camera image to like track what’s moving in order to find where it is in the
01:08:16 world.
01:08:17 This typical sort of, you know, visual inertial state estimation problems that we would solve.
01:08:22 And you know, we just kind of realized that we’re at the limit sometimes of, you know,
01:08:25 doing simple tasks.
01:08:26 We’re at the limit of the camera frame rate because you know, if you really want to track
01:08:30 things, you want the camera image to be 90% kind of like, or some somewhat the same from
01:08:36 one frame to the next.
01:08:39 And why are we at the limit of the camera frame rate?
01:08:42 It’s because camera captures data.
01:08:44 It puts it into some serial connection.
01:08:47 It could be USB or like there’s something called camera serial interface that we use
01:08:51 a lot.
01:08:52 It puts into some serial connection and copper wires can only transmit so much data.
01:08:58 And you hit the channel limit on copper wires and you know, you, you hit yet another kind
01:09:02 of universal limit that you can transfer the data.
01:09:06 So you have to be much more intelligent on how you capture those pixels.
01:09:11 You can take compute and put it right next to the pixels.
01:09:16 People are building those.
01:09:17 How hard is it to do?
01:09:18 How hard is it to get past the bottleneck of the copper wire?
01:09:23 Yeah, you need to, you need to do a lot of parallel processing, as you can imagine.
01:09:27 The same thing happens in the GPUs, you know, like the data is transferred in parallel somehow.
01:09:31 It gets into some parallel processing.
01:09:33 I think that, you know, like now we’re really kind of diverted off into so many different
01:09:38 dimensions, but.
01:09:39 Great.
01:09:40 So it’s aggressive flight.
01:09:41 How do we make drones see many more frames a second in order to enable aggressive flight?
01:09:46 That’s a super interesting problem.
01:09:48 That’s an interesting problem.
01:09:49 So, but like, think about it.
01:09:50 You have, you have CPUs.
01:09:52 You clock them at, you know, several gigahertz.
01:09:57 We don’t clock them faster, largely because, you know, we run into some heating issues
01:10:00 and things like that.
01:10:01 But the whole thing is that three gigahertz clock light travels kind of like on the order
01:10:07 of a few inches or an inch.
01:10:09 That’s the size of a chip.
01:10:11 And so you pass a clock cycle and as the clock signal is going around in the chip, you pass
01:10:17 another one.
01:10:19 And so trying to coordinate that, the design of the complexity of the chip becomes so hard.
01:10:23 I mean, we have hit the fundamental limits of the universe in so many things that we’re
01:10:29 designing.
01:10:30 I don’t know if people realize that.
01:10:31 Like, we can’t make transistors smaller because like quantum effects, the electrons start
01:10:35 to tunnel around.
01:10:36 We can’t clock it faster.
01:10:38 One of the reasons why is because like information doesn’t travel faster in the universe and
01:10:45 we’re limited by that.
01:10:46 Same thing with the laser scanner.
01:10:48 But so then it becomes clear that, you know, the way you organize the chip into a CPU or
01:10:54 even a GPU, you now need to look at how to redesign that.
01:10:59 If you’re going to stick with Silicon, you could go do other things too.
01:11:02 I mean, there’s that too, but you really almost need to take those transistors, put them in
01:11:06 a different way so that the information travels on those transistors in a different way, in
01:11:12 a much more way that is specific to the high speed cameras coming in.
01:11:16 And so that’s one of the things that we talk about quite a bit.
01:11:20 So drone racing kind of really makes that embodies that and that’s why it’s exciting.
01:11:27 It’s exciting for people, you know, students like it.
01:11:30 It embodies all those problems.
01:11:32 But going back, we’re building, quote, unquote, another engine.
01:11:36 And that engine, I hope one day will be just like how impactful seat belts were in driving.
01:11:43 I hope so.
01:11:45 Or it could enable, you know, next generation autonomous air taxis and things like that.
01:11:49 I mean, it sounds crazy, but one day we may need to perch land these things.
01:11:53 If you really want to go from Boston to New York in more than a half hours, you may want
01:11:58 to fix wing aircraft.
01:12:00 Most of these companies that are kind of doing quote unquote flying cars, they’re focusing
01:12:03 on that.
01:12:04 But then how do you land it on top of a building?
01:12:06 You may need to pull off like kind of fast maneuvers for a robot, like perch land.
01:12:10 It’s going to go perch into a building.
01:12:14 If you want to do that, like you need these kinds of systems.
01:12:17 And so drone racing, you know, it’s being able to go way faster than any human can comprehend.
01:12:25 Take an aircraft, forget the quadcopter, you take your fixed wing, while you’re at it,
01:12:30 you might as well put some like rocket engines in the back and you just light it.
01:12:34 You go through the gate and a human looks at it and just said, what just happened?
01:12:39 And they would say, it’s impossible for me to do that.
01:12:41 And that’s closing the same technology gap that would, you know, one day steer cars out
01:12:47 of accidents.
01:12:48 So but then let’s get back to the practical, which is sort of just getting the thing to
01:12:55 to work in a race environment, which is kind of what the is another kind of exciting thing,
01:13:01 which the DARPA challenge to the desert did, you know, theoretically, we had autonomous
01:13:05 vehicles, but making them successfully finish a race, first of all, which nobody finished
01:13:11 the first year, and then the second year just to get, you know, to finish and go at a reasonable
01:13:16 time is really difficult engineering, practically speaking challenge.
01:13:21 So that let me ask about the the the Alpha pilot challenge is a, I guess, a big prize
01:13:27 potentially associated with it.
01:13:29 But let me ask, reminiscent of the DARPA days, predictions, you think anybody will finish?
01:13:36 Well, not, not soon.
01:13:39 I think that depends on how you set up the race course.
01:13:42 And so if the race course is a solo course, I think people will kind of do it.
01:13:46 But can you set up some course, like literally some core, you get to design it is the algorithm
01:13:53 developer, can you set up some course, so that you can be the best human?
01:13:58 When is that going to happen?
01:14:00 Like that’s not very easy, even just setting up some course, if you let the human that
01:14:05 you’re competing with set up the course, it becomes a lot easier, a lot harder.
01:14:10 So how many in the space of all possible courses are, would humans win and would machines win?
01:14:18 Great question.
01:14:19 Let’s get to that.
01:14:20 I want to answer your other question, which is like, the DARPA challenge days, right?
01:14:24 What was really hard?
01:14:25 I think, I think we understand, we understood what we wanted to build, but still building
01:14:30 things, that experimentation, that iterated learning, that takes up a lot of time actually.
01:14:36 And so in my group, for example, in order for us to be able to develop fast, we build
01:14:41 like VR environments, we’ll take an aircraft, we’ll put it in a motion capture room, big,
01:14:46 huge motion capture room, and we’ll fly it in real time, we’ll render other images and
01:14:52 beam it back to the drone.
01:14:54 That sounds kind of notionally simple, but it’s actually hard because now you’re trying
01:14:58 to fit all that data through the air into the drone.
01:15:02 And so you need to do a few crazy things to make that happen.
01:15:05 But once you do that, then at least you can try things.
01:15:09 If you crash into something, you didn’t actually crash.
01:15:12 So it’s like the whole drone is in VR.
01:15:14 We can do augmented reality and so on.
01:15:17 And so I think at some point testing becomes very important.
01:15:20 One of the nice things about Alpha Pilot is that they built the drone and they build a
01:15:24 lot of drones and it’s okay to crash.
01:15:28 In fact, I think maybe the viewers may kind of like to see things that crash.
01:15:34 That potentially could be the most exciting part.
01:15:36 It could be the exciting part.
01:15:38 And I think as an engineer, it’s a very different situation to be in.
01:15:42 Like in academia, a lot of my colleagues who are actually in this race and they’re really
01:15:46 great researchers, but I’ve seen them trying to do similar things whereby they built this
01:15:51 one drone and somebody with like a face mask and a gloves are going right behind the drone.
01:15:58 They’re trying to hold it.
01:15:59 If it falls down, imagine you don’t have to do that.
01:16:02 I think that’s one of the nice things about Alpha Pilot Challenge where we have these
01:16:06 drones and we’re going to design the courses in a way that we’ll keep pushing people up
01:16:11 until the crashes start to happen.
01:16:14 And we’ll hopefully sort of, I don’t think you want to tell people crashing is okay.
01:16:19 Like we want to be careful here, but because we don’t want people to crash a lot, but certainly
01:16:24 we want them to push it so that everybody crashes once or twice and they’re really pushing
01:16:30 it to their limits.
01:16:32 That’s where iterated learning comes in, because every crash is a lesson.
01:16:36 Is a lesson.
01:16:37 Exactly.
01:16:38 So in terms of the space of possible courses, how do you think about it in the war of humans
01:16:44 versus machines, where do machines win?
01:16:47 We look at that quite a bit.
01:16:48 I mean, I think that you will see quickly that you can design a course and in certain
01:16:56 courses like in the middle somewhere, if you kind of run through the course once, the machine
01:17:03 gets beaten pretty much consistently by slightly.
01:17:07 But if you go through the course like 10 times, humans get beaten very slightly, but consistently.
01:17:13 So humans at some point, you get confused, you get tired and things like that versus
01:17:17 this machine is just executing the same line of code tirelessly, just going back to the
01:17:23 beginning and doing the same thing exactly.
01:17:26 I think that kind of thing happens and I realized sort of as humans, there’s the classical things
01:17:34 that everybody has realized.
01:17:36 Like if you put in some sort of like strategic thinking, that’s a little bit harder for machines
01:17:41 that I think sort of comprehend.
01:17:45 Machine is easy to do, so that’s what they excel in.
01:17:48 And also sort of repeatability is easy to do.
01:17:53 That’s what they excel in.
01:17:55 You can build machines that excel in strategy as well and beat humans that way too, but
01:17:59 that’s a lot harder to build.
01:18:00 I have a million more questions, but in the interest of time, last question.
01:18:06 What is the most beautiful idea you’ve come across in robotics?
01:18:10 Is it a simple equation, experiment, a demo, a simulation, a piece of software?
01:18:15 What just gives you pause?
01:18:19 That’s an interesting question.
01:18:21 I have done a lot of work myself in decision making, so I’ve been interested in that area.
01:18:26 So you know, in robotics, somehow the field has split into like, you know, there’s people
01:18:32 who would work on like perception, how robots perceive the environment, then how do you
01:18:37 actually make like decisions and there’s people also like how do you interact, people interact
01:18:41 with robots, there’s a whole bunch of different fields.
01:18:44 And you know, I have admittedly worked a lot on the more control and decision making than
01:18:49 the others.
01:18:52 And I think that, you know, the one equation that has always kind of baffled me is Bellman’s
01:18:57 equation.
01:18:59 And so it’s this person who have realized like way back, you know, more than half a
01:19:04 century ago on like, how do you actually sit down?
01:19:10 And if you have several variables that you’re kind of jointly trying to determine, how do
01:19:15 you determine that?
01:19:17 And there’s one beautiful equation that, you know, like today people do reinforcement
01:19:22 and we still use it.
01:19:24 And it’s baffling to me because it both kind of tells you the simplicity, because it’s
01:19:31 a single equation that anyone can write down.
01:19:33 You can teach it in the first course on decision making.
01:19:37 At the same time, it tells you how computationally, how hard the problem is.
01:19:41 I feel like my, like a lot of the things that I’ve done at MIT for research has been kind
01:19:45 of just this fight against computational efficiency things.
01:19:48 Like how can we get it faster to the point where we now got to like, let’s just redesign
01:19:54 this chip.
01:19:55 Like maybe that’s the way, but I think it talks about how computationally hard certain
01:20:01 problems can be by nowadays what people call curse of dimensionality.
01:20:07 And so as the number of variables kind of grow, the number of decisions you can make
01:20:13 grows rapidly.
01:20:16 Like if you have, you know, a hundred variables, each one of them take 10 values, all possible
01:20:21 assignments is more than the number of atoms in the universe.
01:20:24 It’s just crazy.
01:20:26 And that kind of thinking is just embodied in that one equation that I really like.
01:20:31 And the beautiful balance between it being theoretically optimal and somehow practically
01:20:38 speaking, given the curse of dimensionality, nevertheless in practice works among, you
01:20:45 know, despite all those challenges, which is quite incredible.
01:20:48 Which is quite incredible.
01:20:49 So, you know, I would say that it’s kind of like quite baffling actually, you know, in
01:20:53 a lot of fields that we think about how little we know, you know, like, and so I think here
01:21:00 too.
01:21:01 We know that in the worst case, things are pretty hard, but you know, in practice, generally
01:21:06 things work.
01:21:07 So it’s just kind of, it’s kind of baffling decision making, how little we know.
01:21:12 Just like how little we know about the beginning of time, how little we know about, you know,
01:21:17 our own future.
01:21:19 Like if you actually go into like from Bellman’s equation all the way down, I mean, there’s
01:21:23 also how little we know about like mathematics.
01:21:26 I mean, we don’t even know if the axioms are like consistent.
01:21:28 It’s just crazy.
01:21:29 I think a good lesson there, just like as you said, we tend to focus on the worst case
01:21:35 or the boundaries of everything we’re studying and then the average case seems to somehow
01:21:40 work out.
01:21:41 If you think about life in general, we mess it up a bunch.
01:21:45 You know, we freak out about a bunch of the traumatic stuff, but in the end it seems to
01:21:49 work out okay.
01:21:50 Yeah.
01:21:51 It seems like a good metaphor.
01:21:52 So Tashi, thank you so much for being a friend, a colleague, a mentor.
01:21:57 I really appreciate it.
01:21:58 It’s an honor to talk to you.
01:21:59 Thank you so much for your advice.
01:22:00 Thank you Lex.
01:22:01 Thanks for listening to this conversation with Sertaj Karaman and thank you to our presenting
01:22:05 sponsor Cash App.
01:22:07 Please consider supporting the podcast by downloading Cash App and using code LexPodcast.
01:22:11 If you enjoy this podcast, subscribe on YouTube, review it with five stars on Apple Podcast,
01:22:18 support it on Patreon, or simply connect with me on Twitter at Lex Friedman.
01:22:23 And now let me leave you with some words from Hal9000 from the movie 2001 A Space Odyssey.
01:22:30 I’m putting myself to the fullest possible use, which is all I think that any conscious
01:22:36 entity can ever hope to do.
01:22:39 Thank you for listening and hope to see you next time.