Chris Lattner: The Future of Computing and Programming Languages #131

Transcript

00:00:00 The following is a conversation with Chris Latner,

00:00:02 his second time on the podcast.

00:00:04 He’s one of the most brilliant engineers

00:00:06 in modern computing, having created

00:00:08 LLVM compiler infrastructure project,

00:00:11 the Clang compiler, the Swift programming language,

00:00:14 a lot of key contributions to TensorFlow and TPUs

00:00:17 as part of Google.

00:00:19 He served as vice president of autopilot software at Tesla,

00:00:23 was a software innovator and leader at Apple,

00:00:26 and now is at SciFive as senior vice president

00:00:29 of platform engineering, looking to revolutionize

00:00:32 chip design to make it faster, better, and cheaper.

00:00:36 Quick mention of each sponsor, followed by some thoughts

00:00:39 related to the episode.

00:00:40 First sponsor is Blinkist, an app that summarizes

00:00:43 key ideas from thousands of books.

00:00:45 I use it almost every day to learn new things

00:00:48 or to pick which books I want to read or listen to next.

00:00:52 Second is Neuro, the maker of functional sugar free gum

00:00:55 and mints that I use to supercharge my mind

00:00:58 with caffeine, altheanine, and B vitamins.

00:01:01 Third is Masterclass, online courses from the best people

00:01:05 in the world on each of the topics covered,

00:01:08 from rockets, to game design, to poker,

00:01:11 to writing, and to guitar.

00:01:13 And finally, Cash App, the app I use to send money

00:01:16 to friends for food, drinks, and unfortunately, lost bets.

00:01:21 Please check out the sponsors in the description

00:01:23 to get a discount and to support this podcast.

00:01:27 As a side note, let me say that Chris has been

00:01:29 an inspiration to me on a human level

00:01:32 because he is so damn good as an engineer

00:01:35 and leader of engineers, and yet he’s able to stay humble,

00:01:38 especially humble enough to hear the voices

00:01:41 of disagreement and to learn from them.

00:01:43 He was supportive of me and this podcast

00:01:46 from the early days, and for that, I’m forever grateful.

00:01:49 To be honest, most of my life, no one really believed

00:01:52 that I would amount to much.

00:01:53 So when another human being looks at me,

00:01:56 it makes me feel like I might be someone special,

00:01:58 it can be truly inspiring.

00:02:00 That’s a lesson for educators.

00:02:02 The weird kid in the corner with a dream

00:02:05 is someone who might need your love and support

00:02:08 in order for that dream to flourish.

00:02:10 If you enjoy this thing, subscribe on YouTube,

00:02:13 review it with five stars on Apple Podcast,

00:02:15 follow on Spotify, support on Patreon,

00:02:17 or connect with me on Twitter at Lex Friedman.

00:02:21 And now, here’s my conversation with Chris Latner.

00:02:24 What are the strongest qualities of Steve Jobs,

00:02:28 Elon Musk, and the great and powerful Jeff Dean

00:02:32 since you’ve gotten the chance to work with each?

00:02:36 You’re starting with an easy question there.

00:02:38 These are three very different people.

00:02:40 I guess you could do maybe a pairwise comparison

00:02:43 between them instead of a group comparison.

00:02:45 So if you look at Steve Jobs and Elon,

00:02:48 I worked a lot more with Elon than I did with Steve.

00:02:51 They have a lot of commonality.

00:02:52 They’re both visionary in their own way.

00:02:55 They’re both very demanding in their own way.

00:02:58 My sense is Steve is much more human factor focused

00:03:02 where Elon is more technology focused.

00:03:04 What does human factor mean?

00:03:05 Steve’s trying to build things that feel good,

00:03:08 that people love, that affect people’s lives, how they live.

00:03:11 He’s looking into the future a little bit

00:03:14 in terms of what people want.

00:03:17 Where I think that Elon focuses more on

00:03:20 learning how exponentials work and predicting

00:03:22 the development of those.

00:03:24 Steve worked with a lot of engineers.

00:03:26 That was one of the things that are reading the biography.

00:03:29 How can a designer essentially talk to engineers

00:03:33 and get their respect?

00:03:35 I think, so I did not work very closely with Steve.

00:03:37 I’m not an expert at all.

00:03:38 My sense is that he pushed people really hard,

00:03:41 but then when he got an explanation that made sense to him,

00:03:44 then he would let go.

00:03:45 And he did actually have a lot of respect for engineering,

00:03:49 but he also knew when to push.

00:03:51 And when you can read people well,

00:03:54 you can know when they’re holding back

00:03:56 and when you can get a little bit more out of them.

00:03:58 And I think he was very good at that.

00:04:01 I mean, if you compare the other folks,

00:04:03 so Jeff Dean, right?

00:04:05 Jeff Dean’s an amazing guy.

00:04:06 He’s super smart, as are the other guys.

00:04:10 Jeff is a really, really, really nice guy, well meaning.

00:04:13 He’s a classic Googler.

00:04:15 He wants people to be happy.

00:04:17 He combines it with brilliance

00:04:19 so he can pull people together in a really great way.

00:04:22 He’s definitely not a CEO type.

00:04:24 I don’t think he would even want to be that.

00:04:28 Do you know if he still programs?

00:04:29 Oh yeah, he definitely programs.

00:04:30 Jeff is an amazing engineer today, right?

00:04:32 And that has never changed.

00:04:34 So it’s really hard to compare Jeff to either of those two.

00:04:40 I think that Jeff leads through technology

00:04:43 and building it himself and then pulling people in

00:04:45 and inspiring them.

00:04:46 And so I think that that’s one of the amazing things

00:04:50 about Jeff.

00:04:50 But each of these people, with their pros and cons,

00:04:53 all are really inspirational

00:04:55 and have achieved amazing things.

00:04:56 So I’ve been very fortunate to get to work with these guys.

00:05:00 For yourself, you’ve led large teams,

00:05:03 you’ve done so many incredible,

00:05:06 difficult technical challenges.

00:05:08 Is there something you’ve picked up from them

00:05:10 about how to lead?

00:05:12 Yeah, so I mean, I think leadership is really hard.

00:05:14 It really depends on what you’re looking for there.

00:05:17 I think you really need to know what you’re talking about.

00:05:20 So being grounded on the product, on the technology,

00:05:23 on the business, on the mission is really important.

00:05:28 Understanding what people are looking for,

00:05:29 why they’re there.

00:05:30 One of the most amazing things about Tesla

00:05:32 is the unifying vision, right?

00:05:34 People are there because they believe in clean energy

00:05:37 and electrification, all these kinds of things.

00:05:39 The other is to understand what really motivates people,

00:05:42 how to get the best people,

00:05:43 how to build a plan that actually can be executed, right?

00:05:46 There’s so many different aspects of leadership

00:05:48 and it really depends on the time, the place, the problems.

00:05:52 There’s a lot of issues that don’t need to be solved.

00:05:54 And so if you focus on the right things and prioritize well,

00:05:57 that can really help move things.

00:05:59 Two interesting things you mentioned.

00:06:01 One is you really have to know what you’re talking about.

00:06:03 How you’ve worked on your business,

00:06:08 you’ve worked on a lot of very challenging technical things.

00:06:12 So I kind of assume you were born technically savvy,

00:06:18 but assuming that’s not the case,

00:06:20 how did you develop technical expertise?

00:06:24 Like even at Google you worked on,

00:06:27 I don’t know how many projects,

00:06:28 but really challenging, very varied.

00:06:32 Compilers, TPUs, hardware, cloud stuff,

00:06:34 bunch of different things.

00:06:36 The thing that I’ve become comfortable

00:06:37 as I’ve more comfortable with as I’ve gained experience

00:06:42 is being okay with not knowing.

00:06:45 And so a major part of leadership is actually,

00:06:49 it’s not about having the right answer,

00:06:50 it’s about getting the right answer.

00:06:52 And so if you’re working in a team of amazing people, right?

00:06:56 And many of these places, many of these companies

00:06:58 all have amazing people.

00:07:00 It’s the question of how do you get people together?

00:07:02 How do you build trust?

00:07:04 How do you get people to open up?

00:07:05 How do you get people to be vulnerable sometimes

00:07:10 with an idea that maybe isn’t good enough,

00:07:11 but it’s the start of something beautiful?

00:07:13 How do you provide an environment

00:07:17 where you’re not just like top down,

00:07:18 thou shalt do the thing that I tell you to do, right?

00:07:21 But you’re encouraging people to be part of the solution

00:07:23 and providing a safe space

00:07:26 where if you’re not doing the right thing,

00:07:27 they’re willing to tell you about it, right?

00:07:29 So you’re asking dumb questions?

00:07:31 Yeah, dumb questions are my specialty, yeah.

00:07:33 Well, so I’ve been in the hardware realm recently

00:07:35 and I don’t know much at all about how chips are designed.

00:07:39 I know a lot about using them.

00:07:40 I know some of the principles

00:07:41 and the art’s technical level of this,

00:07:43 but it turns out that if you ask a lot of dumb questions,

00:07:47 you get smarter really, really quick.

00:07:48 And when you’re surrounded by people that want to teach

00:07:51 and learn themselves, it can be a beautiful thing.

00:07:56 So let’s talk about programming languages, if it’s okay.

00:07:58 Sure, sure.

00:07:59 At the highest absurd philosophical level,

00:08:01 because I…

00:08:02 Don’t get romantic on me, Lex.

00:08:03 I will forever get romantic and torture you, I apologize.

00:08:09 Why do programming languages even matter?

00:08:14 Okay, well, thank you very much.

00:08:15 You’re saying why should you care

00:08:17 about any one programming language

00:08:18 or why do we care about programming computers or?

00:08:20 No, why do we care about programming language design,

00:08:25 creating effective programming languages,

00:08:30 choosing one programming languages

00:08:32 such as another programming language,

00:08:34 why we keep struggling and improving

00:08:37 through the evolution of these programming languages.

00:08:39 Sure, sure, sure.

00:08:40 Okay, so I mean, I think you have to come back

00:08:42 to what are we trying to do here, right?

00:08:43 So we have these beasts called computers

00:08:47 that are very good at specific kinds of things

00:08:48 and we think it’s useful to have them do it for us, right?

00:08:52 Now you have this question of how best to express that

00:08:55 because you have a human brain still

00:08:57 that has an idea in its head

00:08:58 and you want to achieve something, right?

00:09:00 So, well, there’s lots of ways of doing this.

00:09:03 You can go directly to the machine

00:09:04 and speak assembly language

00:09:06 and then you can express directly

00:09:07 what the computer understands, that’s fine.

00:09:10 You can then have higher and higher and higher levels

00:09:12 of abstraction up until machine learning

00:09:14 and you’re designing a neural net to do the work for you.

00:09:18 The question is where along this way do you want to stop

00:09:21 and what benefits do you get out of doing so?

00:09:23 And so programming languages in general,

00:09:25 you have C, you have Fortran, Java and Ada, Pascal, Swift,

00:09:31 you have lots of different things.

00:09:33 They’ll have different trade offs

00:09:34 and they’re tackling different parts of the problems.

00:09:36 Now, one of the things that most programming languages do

00:09:39 is they’re trying to make it

00:09:40 so that you have pretty basic things

00:09:42 like portability across different hardware.

00:09:45 So you’ve got, I’m gonna run on an Intel PC,

00:09:47 I’m gonna run on a RISC 5 PC,

00:09:49 I’m gonna run on a ARM phone or something like that, fine.

00:09:53 I wanna write one program and have it portable.

00:09:55 And this is something that assembly doesn’t do.

00:09:57 Now, when you start looking

00:09:59 at the space of programming languages,

00:10:00 this is where I think it’s fun

00:10:02 because programming languages all have trade offs

00:10:06 and most people will walk up to them

00:10:07 and they look at the surface level of syntax and say,

00:10:11 oh, I like curly braces or I like tabs

00:10:13 or I like semi colons or not or whatever, right?

00:10:17 Subjective, fairly subjective, very shallow things.

00:10:21 But programming languages when done right

00:10:23 can actually be very powerful.

00:10:24 And the benefit they bring is expression.

00:10:30 Okay, and if you look at programming languages,

00:10:32 there’s really kind of two different levels to them.

00:10:34 One is the down in the dirt, nuts and bolts

00:10:37 of how do you get the computer to be efficient,

00:10:39 stuff like that, how they work,

00:10:40 type systems, compiler stuff, things like that.

00:10:43 The other is the UI.

00:10:45 And the UI for programming language

00:10:47 is really a design problem

00:10:48 and a lot of people don’t think about it that way.

00:10:50 And the UI, you mean all that stuff with the braces and?

00:10:53 Yeah, all that stuff’s the UI and what it is

00:10:55 and UI means user interface.

00:10:58 And so what’s really going on is

00:11:00 it’s the interface between the guts and the human.

00:11:04 And humans are hard, right?

00:11:05 Humans have feelings, they have things they like,

00:11:09 they have things they don’t like.

00:11:10 And a lot of people treat programming languages

00:11:12 as though humans are just kind of abstract creatures

00:11:16 that cannot be predicted.

00:11:17 But it turns out that actually there is better and worse.

00:11:21 Like people can tell when a programming language is good

00:11:24 or when it was an accident, right?

00:11:26 And one of the things with Swift in particular

00:11:29 is that a tremendous amount of time

00:11:30 by a tremendous number of people

00:11:33 have been put into really polishing and making it feel good.

00:11:36 But it also has really good nuts and bolts underneath it.

00:11:39 You said that Swift makes a lot of people feel good.

00:11:42 How do you get to that point?

00:11:45 So how do you predict that tens of thousands,

00:11:51 hundreds of thousands of people are going to enjoy

00:11:53 using this user experience of this programming language?

00:11:57 Well, you can look at it in terms of better and worse, right?

00:11:59 So if you have to write lots of boilerplate

00:12:01 or something like that, you will feel unproductive.

00:12:03 And so that’s a bad thing.

00:12:05 You can look at it in terms of safety.

00:12:06 If like C for example,

00:12:08 is what’s called a memory unsafe language.

00:12:10 And so you get dangling pointers

00:12:11 and you get all these kinds of bugs

00:12:13 that then you have spent tons of time debugging

00:12:15 and it’s a real pain in the butt and you feel unproductive.

00:12:17 And so by subtracting these things from the experience,

00:12:19 you get happier people.

00:12:22 But again, keep interrupting.

00:12:25 I’m sorry, but so hard to deal with.

00:12:29 If you look at the people that are most productive

00:12:31 on Stack Overflow, they have a set of priorities

00:12:37 that may not always correlate perfectly

00:12:39 with the experience of the majority of users.

00:12:43 If you look at the most upvoted,

00:12:46 quote unquote, correct answer on Stack Overflow,

00:12:49 it usually really sort of prioritizes

00:12:55 like safe code, proper code, stable code,

00:13:00 you know, that kind of stuff.

00:13:01 As opposed to like,

00:13:02 if I wanna use go to statements in my basic, right?

00:13:08 I wanna use go to statements.

00:13:09 Like what if 99% of people wanna use go to statements?

00:13:12 So you use completely improper, you know, unsafe syntax.

00:13:16 I don’t think that people actually,

00:13:17 like if you boil it down and you get below

00:13:19 the surface level, people don’t actually care

00:13:21 about go tos or if statements or things like this.

00:13:24 They care about achieving a goal, right?

00:13:26 So the real question is I wanna set up a web server

00:13:30 and I wanna do a thing, whatever.

00:13:32 Like how quickly can I achieve that, right?

00:13:34 And so from a programming language perspective,

00:13:36 there’s really two things that matter there.

00:13:39 One is what libraries exist

00:13:41 and then how quickly can you put it together

00:13:44 and what are the tools around that look like, right?

00:13:47 And when you wanna build a library that’s missing,

00:13:49 what do you do?

00:13:50 Okay, now this is where you see huge divergence

00:13:53 in the force between worlds, okay?

00:13:55 And so you look at Python, for example.

00:13:57 Python is really good at assembling things,

00:13:59 but it’s not so great at building all the libraries.

00:14:02 And so what you get because of performance reasons,

00:14:04 other things like this,

00:14:05 is you get Python layered on top of C, for example,

00:14:09 and that means that doing certain kinds of things

00:14:11 well, it doesn’t really make sense to do in Python.

00:14:13 Instead you do it in C and then you wrap it

00:14:15 and then you have, you’re living in two worlds

00:14:17 and two worlds never is really great

00:14:19 because tooling and the debugger doesn’t work right

00:14:21 and like all these kinds of things.

00:14:23 Can you clarify a little bit what you mean

00:14:25 by Python is not good at building libraries,

00:14:28 meaning it doesn’t make it conducive.

00:14:30 Certain kinds of libraries.

00:14:31 No, but just the actual meaning of the sentence,

00:14:35 meaning like it’s not conducive to developers

00:14:38 to come in and add libraries

00:14:40 or is it the duality of the,

00:14:44 it’s a dance between Python and C and…

00:14:48 Well, so Python’s amazing.

00:14:49 Python’s a great language.

00:14:50 I did not mean to say that Python is bad for libraries.

00:14:53 What I meant to say is there are libraries

00:14:56 that Python’s really good at that you can write in Python,

00:15:00 but there are other things,

00:15:01 like if you wanna build a machine learning framework,

00:15:03 you’re not gonna build a machine learning framework

00:15:05 in Python because of performance, for example,

00:15:07 or you want GPU acceleration or things like this.

00:15:10 Instead, what you do is you write a bunch of C

00:15:13 or C++ code or something like that,

00:15:15 and then you talk to it from Python, right?

00:15:18 And so this is because of decisions

00:15:21 that were made in the Python design

00:15:23 and those decisions have other counterbalancing forces.

00:15:27 But the trick when you start looking at this

00:15:29 from a programming language perspective,

00:15:31 you start to say, okay, cool.

00:15:33 How do I build this catalog of libraries

00:15:36 that are really powerful?

00:15:37 And how do I make it so that then they can be assembled

00:15:40 into ways that feel good

00:15:42 and they generally work the first time?

00:15:44 Because when you’re talking about building a thing,

00:15:46 you have to include the debugging, the fixing,

00:15:50 the turnaround cycle, the development cycle,

00:15:51 all that kind of stuff

00:15:53 into the process of building the thing.

00:15:56 It’s not just about pounding out the code.

00:15:58 And so this is where things like catching bugs

00:16:01 at compile time is valuable, for example.

00:16:04 But if you dive into the details in this,

00:16:07 Swift, for example, has certain things like value semantics,

00:16:10 which is this fancy way of saying

00:16:11 that when you treat a variable like a value,

00:16:18 it acts like a mathematical object would.

00:16:21 Okay, so you have used PyTorch a little bit.

00:16:25 In PyTorch, you have tensors.

00:16:26 Tensors are n dimensional grid of numbers, very simple.

00:16:31 You can do plus and other operators on them.

00:16:34 It’s all totally fine.

00:16:35 But why do you need to clone a tensor sometimes?

00:16:39 Have you ever run into that?

00:16:40 Yeah.

00:16:41 Okay, and so why is that?

00:16:42 Why do you need to clone a tensor?

00:16:43 It’s the usual object thing that’s in Python.

00:16:46 So in Python, and just like with Java

00:16:49 and many other languages, this isn’t unique to Python.

00:16:51 In Python, it has a thing called reference semantics,

00:16:53 which is the nerdy way of explaining this.

00:16:55 And what that means is you actually have a pointer

00:16:58 do a thing instead of the thing, okay?

00:17:01 Now, this is due to a bunch of implementation details

00:17:05 that you don’t want to go into.

00:17:06 But in Swift, you have this thing called value semantics.

00:17:09 And so when you have a tensor in Swift, it is a value.

00:17:12 If you copy it, it looks like you have a unique copy.

00:17:15 And if you go change one of those copies,

00:17:16 then it doesn’t update the other one

00:17:19 because you just made a copy of this thing, right?

00:17:21 So that’s like highly error prone

00:17:24 in at least computer science, math centric disciplines

00:17:29 about Python, that like the thing you would expect

00:17:33 to behave like math.

00:17:35 Like math, it doesn’t behave like math.

00:17:38 And in fact, quietly it doesn’t behave like math

00:17:41 and then can ruin the entirety of your math thing.

00:17:43 Exactly.

00:17:44 Well, and then it puts you in debugging land again.

00:17:45 Yeah.

00:17:46 Right now, you just want to get something done

00:17:48 and you’re like, wait a second, where do I need to put clone?

00:17:51 And what level of the stack, which is very complicated,

00:17:54 which I thought I was reusing somebody’s library

00:17:56 and now I need to understand it

00:17:57 to know where to clone a thing, right?

00:17:59 And hard to debug, by the way.

00:18:01 Exactly, right.

00:18:02 And so this is where programming languages really matter.

00:18:04 Right, and so in Swift having value semantics

00:18:06 so that both you get the benefit of math,

00:18:10 working like math, right?

00:18:12 But also the efficiency that comes with certain advantages

00:18:15 there, certain implementation details there

00:18:17 really benefit you as a programmer, right?

00:18:18 Can you clarify the value semantics?

00:18:20 Like how do you know that a thing should be treated

00:18:22 like a value?

00:18:23 Yeah, so Swift has a pretty strong culture

00:18:27 and good language support for defining values.

00:18:30 And so if you have an array,

00:18:31 so tensors are one example that the machine learning folks

00:18:34 are very used to.

00:18:36 Just think about arrays, same thing,

00:18:38 where you have an array, you create an array,

00:18:41 you put two or three or four things into it,

00:18:43 and then you pass it off to another function.

00:18:46 What happens if that function adds some more things to it?

00:18:51 Well, you’ll see it on the side that you pass it in, right?

00:18:54 This is called reference semantics.

00:18:56 Now, what if you pass an array off to a function,

00:19:01 it scrolls it away in some dictionary

00:19:02 or some other data structure somewhere, right?

00:19:04 Well, it thought that you just handed it that array,

00:19:07 then you return back and that reference to that array

00:19:10 still exists in the caller,

00:19:12 and they go and put more stuff in it, right?

00:19:15 The person you handed it off to

00:19:17 may have thought they had the only reference to that,

00:19:20 and so they didn’t know that this was gonna change

00:19:22 underneath the covers.

00:19:23 And so this is where you end up having to do clone.

00:19:26 So like I was passed a thing,

00:19:27 I’m not sure if I have the only version of it,

00:19:30 so now I have to clone it.

00:19:32 So what value semantics does is it allows you to say,

00:19:34 hey, I have a, so in Swift, it defaults to value semantics.

00:19:38 Oh, so it defaults to value semantics,

00:19:40 and then because most things

00:19:42 should end up being like values,

00:19:44 then it makes sense for that to be the default.

00:19:46 And one of the important things about that

00:19:47 is that arrays and dictionaries

00:19:48 and all these other collections

00:19:49 that are aggregations of other things

00:19:51 also have value semantics.

00:19:53 And so when you pass this around

00:19:55 to different parts of your program,

00:19:56 you don’t have to do these defensive copies.

00:19:59 And so this is great for two sides, right?

00:20:01 It’s great because you define away the bug,

00:20:04 which is a big deal for productivity,

00:20:05 the number one thing most people care about,

00:20:08 but it’s also good for performance

00:20:09 because when you’re doing a clone,

00:20:11 so you pass the array down to the thing,

00:20:13 it’s like, I don’t know if anybody else has it,

00:20:15 I have to clone it.

00:20:16 Well, you just did a copy of a bunch of data.

00:20:18 It could be big.

00:20:19 And then it could be that the thing that called you

00:20:21 is not keeping track of the old thing.

00:20:24 So you just made a copy of it,

00:20:25 and you may not have had to.

00:20:27 And so the way the value semantics work in Swift

00:20:30 is it uses this thing called copy on write,

00:20:32 which means that you get the benefit of safety

00:20:35 and performance.

00:20:36 And it has another special trick

00:20:38 because if you think certain languages like Java,

00:20:41 for example, they have immutable strings.

00:20:43 And so what they’re trying to do

00:20:44 is they provide value semantics

00:20:46 by having pure immutability.

00:20:48 Functional languages have pure immutability

00:20:51 in lots of different places,

00:20:52 and this provides a much safer model

00:20:53 and it provides value semantics.

00:20:56 The problem with this is if you have immutability,

00:20:58 everything is expensive.

00:20:59 Everything requires a copy.

00:21:02 For example, in Java, if you have a string X

00:21:05 and a string Y, you append them together,

00:21:07 we have to allocate a new string to hold X, Y.

00:21:12 If they’re immutable.

00:21:13 Well, strings in Java are immutable.

00:21:16 And if there’s optimizations for short ones,

00:21:19 it’s complicated, but generally think about them

00:21:22 as a separate allocation.

00:21:24 And so when you append them together,

00:21:26 you have to go allocate a third thing

00:21:28 because somebody might have a pointer

00:21:29 to either of the other ones, right?

00:21:31 And you can’t go change them.

00:21:32 So you have to go allocate a third thing.

00:21:34 Because of the beauty of how the Swift value semantics

00:21:36 system works out, if you have a string in Swift

00:21:38 and you say, hey, put in X, right?

00:21:40 And they say, append on Y, Z, W,

00:21:44 it knows that there’s only one reference to that.

00:21:47 And so it can do an in place update.

00:21:50 And so you’re not allocating tons of stuff on the side.

00:21:53 You don’t have all those problems.

00:21:54 When you pass it off,

00:21:56 you can know you have the only reference.

00:21:57 If you pass it off to multiple different people,

00:21:59 but nobody changes it, they can all share the same thing.

00:22:02 So you get a lot of the benefit of purely immutable design.

00:22:05 And so you get a really nice sweet spot

00:22:07 that I haven’t seen in other languages.

00:22:09 Yeah, that’s interesting.

00:22:10 I thought there was going to be a philosophical narrative

00:22:15 here that you’re gonna have to pay a cost for it.

00:22:19 Cause it sounds like, I think value semantics

00:22:24 is beneficial for easing of debugging

00:22:27 or minimizing the risk of errors,

00:22:30 like bringing the errors closer to the source,

00:22:35 bringing the symptom of the error closer

00:22:38 to the source of the error, however you say that.

00:22:40 But you’re saying there’s not a performance cost either

00:22:44 if you implement it correctly.

00:22:46 Well, so there’s trade offs with everything.

00:22:48 And so if you are doing very low level stuff,

00:22:51 then sometimes you can notice a cost,

00:22:53 but then what you’re doing is you’re saying,

00:22:54 what is the right default?

00:22:56 So coming back to user interface,

00:22:59 when you talk about programming languages,

00:23:00 one of the major things that Swift does

00:23:03 that makes people love it,

00:23:04 that is not obvious when it comes to designing a language

00:23:08 is this UI principle of progressive disclosure

00:23:11 of complexity.

00:23:12 Okay, so Swift, like many languages is very powerful.

00:23:16 The question is, when do you have to learn

00:23:18 the power as a user?

00:23:20 So Swift, like Python, allows you to start with like,

00:23:22 print hello world, right?

00:23:24 Certain other languages start with like,

00:23:26 public static void main class,

00:23:30 like all the ceremony, right?

00:23:32 And so you go to teach a new person,

00:23:34 hey, welcome to this new thing.

00:23:36 Let’s talk about public access control classes.

00:23:40 Wait, what’s that?

00:23:41 String system.out.println, like packages,

00:23:44 like, God, right?

00:23:46 And so instead, if you take this and you say,

00:23:48 hey, we need packages, modules,

00:23:51 we need powerful things like classes,

00:23:54 we need data structures, we need like all these things.

00:23:57 The question is, how do you factor the complexity?

00:23:59 And how do you make it so that the normal case scenario

00:24:02 is you’re dealing with things that work the right way

00:24:05 in the right way, give you good performance

00:24:07 by default, but then as a power user,

00:24:11 if you want to dive down to it,

00:24:12 you have full C performance, full control

00:24:15 over low level pointers.

00:24:15 You can call malloc if you want to call malloc.

00:24:18 This is not recommended on the first page of every tutorial,

00:24:20 but it’s actually really important

00:24:22 when you want to get work done, right?

00:24:23 And so being able to have that is really the design

00:24:27 in programming language design,

00:24:28 and design is really, really hard.

00:24:31 It’s something that I think a lot of people kind of,

00:24:34 outside of UI, again, a lot of people just think

00:24:37 is subjective, like there’s nothing,

00:24:40 you know, it’s just like curly braces or whatever.

00:24:43 It’s just like somebody’s preference,

00:24:45 but actually good design is something that you can feel.

00:24:48 And how many people are involved with good design?

00:24:52 So if we looked at Swift, but look at historically,

00:24:54 I mean, this might touch like,

00:24:57 it’s almost like a Steve Jobs question too,

00:24:59 like how much dictatorial decision making is required

00:25:04 versus collaborative, and we’ll talk about

00:25:09 how all that can go wrong or right, but.

00:25:11 Yeah, well, Swift, so I can’t speak to in general,

00:25:14 all design everywhere.

00:25:15 So the way it works with Swift is that there’s a core team,

00:25:19 and so a core team is six or seven people ish,

00:25:22 something like that, that is people that have been working

00:25:25 with Swift since very early days, and so.

00:25:27 And by early days is not that long ago.

00:25:30 Okay, yeah, so it became public in 2014,

00:25:33 so it’s been six years public now,

00:25:35 but so that’s enough time that there’s a story arc there.

00:25:38 Okay, yeah, and there’s mistakes have been made

00:25:41 that then get fixed, and you learn something,

00:25:43 and then you, you know, and so what the core team does

00:25:46 is it provides continuity, and so you wanna have a,

00:25:50 okay, well, there’s a big hole that we wanna fill.

00:25:54 We know we wanna fill it, so don’t do other things

00:25:56 that invade that space until we fill the hole, right?

00:25:59 There’s a boulder that’s missing here,

00:26:01 we wanna do, we will do that boulder,

00:26:03 even though it’s not today, keep out of that space.

00:26:06 And the whole team remembers the myth of the boulder

00:26:10 that’s there.

00:26:11 Yeah, yeah, there’s a general sense

00:26:12 of what the future looks like in broad strokes,

00:26:14 and a shared understanding of that,

00:26:16 combined with a shared understanding of what has happened

00:26:18 in the past that worked out well and didn’t work out well.

00:26:22 The next level out is you have the,

00:26:24 what’s called the Swift evolution community,

00:26:25 and you’ve got, in that case, hundreds of people

00:26:27 that really care passionately about the way Swift evolves,

00:26:30 and that’s like an amazing thing to, again,

00:26:33 the core team doesn’t necessarily need to come up

00:26:35 with all the good ideas.

00:26:36 You got hundreds of people out there

00:26:38 that care about something,

00:26:39 and they come up with really good ideas too,

00:26:41 and that provides this rock tumbler for ideas.

00:26:45 And so the evolution process is,

00:26:48 a lot of people in a discourse forum,

00:26:50 they’re like hashing it out and trying to talk about,

00:26:52 okay, well, should we go left or right,

00:26:54 or if we did this, what would be good?

00:26:55 And here you’re talking about hundreds of people,

00:26:57 so you’re not gonna get consensus, necessarily,

00:27:00 not obvious consensus, and so there’s a proposal process

00:27:04 that then allows the core team and the community

00:27:07 to work this out, and what the core team does

00:27:10 is it aims to get consensus out of the community

00:27:12 and provide guardrails, but also provide long term,

00:27:17 make sure we’re going the right direction kind of things.

00:27:20 So does that group represent like the,

00:27:23 how much people will love the user interface?

00:27:27 Like, do you think they’re able to capture that?

00:27:29 Well, I mean, it’s something we talk about a lot,

00:27:31 it’s something we care about.

00:27:32 How well we do that’s up for debate,

00:27:34 but I think that we’ve done pretty well so far.

00:27:36 Is the beginner in mind?

00:27:38 Yeah. Like, because you said

00:27:39 the progressive disclosure complexity.

00:27:40 Yeah, so we care a lot about that,

00:27:44 a lot about power, a lot about efficiency,

00:27:46 a lot about, there are many factors to good design,

00:27:48 and you have to figure out a way

00:27:50 to kind of work your way through that, and.

00:27:53 So if you think about, like the language I love is Lisp,

00:27:57 probably still because I use Emacs,

00:27:59 but I haven’t done anything, any serious work in Lisp,

00:28:02 but it has a ridiculous amount of parentheses.

00:28:05 Yeah.

00:28:06 I’ve also, you know, with Java and C++, the braces,

00:28:14 you know, I like, I enjoyed the comfort

00:28:17 of being between braces, you know?

00:28:20 Yeah, yeah, well, let’s talk.

00:28:21 And then Python is, sorry to interrupt,

00:28:23 just like, and last thing to me, as a designer,

00:28:25 if I was a language designer, God forbid,

00:28:28 is I would be very surprised that Python with no braces

00:28:34 would nevertheless somehow be comforting also.

00:28:38 So like, I could see arguments for all of this.

00:28:40 But look at this, this is evidence

00:28:41 that it’s not about braces versus tabs.

00:28:44 Right, exactly, you’re good, that’s a good point.

00:28:47 Right, so like, you know, there’s evidence that.

00:28:50 But see, like, it’s one of the most argued about things.

00:28:52 Oh yeah, of course, just like tabs and spaces,

00:28:54 which it doesn’t, I mean, there’s one obvious right answer,

00:28:57 but it doesn’t actually matter.

00:28:59 What’s that?

00:28:59 Let’s not, like, come on, we’re friends.

00:29:01 Like, come on, what are you trying to do to me here?

00:29:03 People are gonna, yeah, half the people are gonna tune out.

00:29:05 Yeah, so these, so you’re able to identify things

00:29:09 that don’t really matter for the experience.

00:29:12 Well, no, no, no, it’s always a really hard,

00:29:14 so the easy decisions are easy, right?

00:29:16 I mean, fine, those are not the interesting ones.

00:29:19 The hard ones are the ones that are most interesting, right?

00:29:21 The hard ones are the places where,

00:29:23 hey, we wanna do a thing, everybody agrees we should do it,

00:29:27 there’s one proposal on the table,

00:29:28 but it has all these bad things associated with it.

00:29:31 Well, okay, what are we gonna do about that?

00:29:33 Do we just take it?

00:29:34 Do we delay it?

00:29:36 Do we say, hey, well, maybe there’s this other feature

00:29:38 that if we do that first, this will work out better.

00:29:41 How does this, if we do this,

00:29:44 are we paying ourselves into a corner, right?

00:29:46 And so this is where, again,

00:29:47 you’re having that core team of people

00:29:48 that has some continuity and has perspective,

00:29:51 has some of the historical understanding,

00:29:53 is really valuable because you get,

00:29:56 it’s not just like one brain,

00:29:57 you get the power of multiple people coming together

00:29:59 to make good decisions,

00:30:00 and then you get the best out of all these people,

00:30:02 and you also can harness the community around it.

00:30:06 And what about the decision of whether in Python

00:30:09 having one type or having strict typing?

00:30:14 Yeah, okay.

00:30:15 Yeah, let’s talk about this.

00:30:16 So I like how you put that, by the way.

00:30:19 So many people would say that Python doesn’t have types.

00:30:21 Doesn’t have types, yeah.

00:30:22 But you’re right.

00:30:23 I haven’t listened to you enough to where,

00:30:26 I’m a fan of yours and I’ve listened to way too many

00:30:29 podcasts and videos of you talking about this stuff.

00:30:32 Oh yeah, so I would argue that Python has one type.

00:30:34 And so like when you import Python into Swift,

00:30:38 which by the way works really well,

00:30:39 you have everything comes in as a Python object.

00:30:41 Now here are their trade offs because

00:30:46 it depends on what you’re optimizing for.

00:30:47 And Python is a super successful language

00:30:49 for a really good reason.

00:30:51 Because it has one type,

00:30:52 you get duck typing for free and things like this.

00:30:55 But also you’re pushing,

00:30:56 you’re making it very easy to pound out code on one hand,

00:31:00 but you’re also making it very easy to introduce

00:31:03 complicated bugs that you have to debug.

00:31:05 And you pass a string into something that expects an integer

00:31:08 and it doesn’t immediately die.

00:31:10 It goes all the way down the stack trace

00:31:12 and you find yourself in the middle of some code

00:31:13 that you really didn’t want to know anything about.

00:31:14 And it blows up and you’re just saying,

00:31:16 well, what did I do wrong, right?

00:31:18 And so types are good and bad and they have trade offs.

00:31:21 They’re good for performance and certain other things

00:31:23 depending on where you’re coming from,

00:31:24 but it’s all about trade offs.

00:31:26 And so this is what design is, right?

00:31:28 Design is about weighing trade offs

00:31:30 and trying to understand the ramifications

00:31:32 of the things that you’re weighing,

00:31:34 like types or not, or one type or many types.

00:31:38 But also within many types,

00:31:39 how powerful do you make that type system

00:31:41 is another very complicated question

00:31:44 with lots of trade offs.

00:31:45 It’s very interesting by the way,

00:31:47 but that’s like one dimension and there’s a bunch

00:31:52 of other dimensions, JIT compiled versus static compiled,

00:31:55 garbage collected versus reference counted

00:31:57 versus manual memory management versus,

00:32:00 like in like all these different trade offs

00:32:02 and how you balance them

00:32:03 are what make a program language good.

00:32:05 Concurrency.

00:32:06 Yeah.

00:32:07 So in all those things, I guess,

00:32:09 when you’re designing the language,

00:32:11 you also have to think of how that’s gonna get

00:32:13 all compiled down to.

00:32:15 If you care about performance, yeah.

00:32:17 Well, and go back to Lisp, right?

00:32:18 So Lisp also, I would say JavaScript

00:32:20 is another example of a very simple language, right?

00:32:24 And so one of the, so I also love Lisp.

00:32:27 I don’t use it as much as maybe you do or you did.

00:32:29 No, I think we’re both, everyone who loves Lisp,

00:32:32 it’s like, you love, it’s like, I don’t know,

00:32:35 I love Frank Sinatra,

00:32:36 but like how often do I seriously listen to Frank Sinatra?

00:32:39 Sure, sure.

00:32:40 But you look at that or you look at JavaScript,

00:32:42 which is another very different,

00:32:44 but relatively simple language.

00:32:45 And there are certain things that don’t exist

00:32:47 in the language,

00:32:49 but there is inherent complexity to the problems

00:32:51 that we’re trying to model.

00:32:53 And so what happens to the complexity?

00:32:54 In the case of both of them, for example,

00:32:57 you say, well, what about large scale software development?

00:33:00 Okay, well, you need something like packages.

00:33:02 Neither language has a language affordance for packages.

00:33:05 And so what you get is patterns.

00:33:07 You get things like NPN.

00:33:08 You get things like these ecosystems that get built around.

00:33:12 And I’m a believer that if you don’t model

00:33:15 at least the most important inherent complexity

00:33:17 in the language,

00:33:18 then what ends up happening

00:33:19 is that complexity gets pushed elsewhere.

00:33:22 And when it gets pushed elsewhere,

00:33:24 sometimes that’s great because often building things

00:33:26 as libraries is very flexible and very powerful

00:33:28 and allows you to evolve and things like that.

00:33:30 But often it leads to a lot of unnecessary divergence

00:33:34 in the force and fragmentation.

00:33:35 And when that happens, you just get kind of a mess.

00:33:39 And so the question is, how do you balance that?

00:33:42 Don’t put too much stuff in the language

00:33:44 because that’s really expensive

00:33:45 and it makes things complicated.

00:33:46 But how do you model enough of the inherent complexity

00:33:49 of the problem that you provide the framework

00:33:52 and the structure for people to think about?

00:33:54 Well, so the key thing to think about

00:33:57 with programming languages,

00:33:59 and you think about what a programming language is therefore,

00:34:01 is it’s about making a human more productive, right?

00:34:04 And so there’s an old, I think it’s Steve Jobs quote

00:34:07 about it’s a bicycle for the mind, right?

00:34:10 You can definitely walk,

00:34:13 but you’ll get there a lot faster

00:34:15 if you can bicycle on your way.

00:34:17 And…

00:34:18 A programming language is a bicycle for the mind?

00:34:20 Yeah.

00:34:20 Is it basically, wow,

00:34:22 that’s really interesting way to think about it.

00:34:23 By raising the level of abstraction,

00:34:25 now you can fit more things in your head.

00:34:27 By being able to just directly leverage somebody’s library,

00:34:30 you can now get something done quickly.

00:34:33 In the case of Swift, Swift UI is this new framework

00:34:36 that Apple has released recently for doing UI programming.

00:34:39 And it has this declarative programming model,

00:34:42 which defines away entire classes of bugs.

00:34:45 It builds on value semantics and many other nice Swift things.

00:34:48 And what this does is it allows you to just get way more done

00:34:51 with way less code.

00:34:53 And now your productivity as a developer is much higher,

00:34:56 right?

00:34:57 And so that’s really what programming languages

00:34:59 should be about,

00:35:00 is it’s not about tabs versus spaces

00:35:01 or curly braces or whatever.

00:35:03 It’s about how productive do you make the person.

00:35:05 And you can only see that when you have libraries

00:35:08 that were built with the right intention

00:35:11 that the language was designed for.

00:35:13 And with Swift, I think we’re still a little bit early,

00:35:16 but Swift UI and many other things that are coming out now

00:35:19 are really showing that.

00:35:20 And I think that they’re opening people’s eyes.

00:35:22 It’s kind of interesting to think about like how

00:35:26 that the knowledge of something,

00:35:29 of how good the bicycle is,

00:35:31 how people learn about that.

00:35:33 So I’ve used C++,

00:35:36 now this is not going to be a trash talking session

00:35:38 about C++, but I used C++ for a really long time.

00:35:41 You can go there if you want, I have the scars.

00:35:45 I feel like I spent many years without realizing

00:35:49 like there’s languages that could,

00:35:51 for my particular lifestyle, brain style, thinking style,

00:35:56 there’s languages that could make me a lot more productive

00:36:00 in the debugging stage, in just the development stage

00:36:04 and thinking like the bicycle for the mind

00:36:05 that I can fit more stuff into my…

00:36:07 Python’s a great example of that, right?

00:36:09 I mean, a machine learning framework in Python

00:36:10 is a great example of that.

00:36:12 It’s just very high abstraction level.

00:36:14 And so you can be thinking about things

00:36:15 on a like very high level algorithmic level

00:36:19 instead of thinking about, okay, well,

00:36:20 am I copying this tensor to a GPU or not, right?

00:36:23 It’s not what you want to be thinking about.

00:36:25 And as I was telling you, I mean,

00:36:26 I guess the question I had is,

00:36:29 how does a person like me or in general people

00:36:31 discover more productive languages?

00:36:36 Like how I was, as I’ve been telling you offline,

00:36:39 I’ve been looking for like a project to work on in Swift

00:36:43 so I can really try it out.

00:36:45 I mean, my intuition was like doing a hello world

00:36:48 is not going to get me there

00:36:50 to get me to experience the power of language.

00:36:54 You need a few weeks to change your metabolism.

00:36:55 Exactly, beautifully put.

00:36:59 That’s one of the problems with people with diets,

00:37:01 like I’m actually currently, to go in parallel,

00:37:05 but in a small tangent is I’ve been recently

00:37:07 eating only meat, okay?

00:37:10 And okay, so most people are like,

00:37:14 they think that’s horribly unhealthy or whatever,

00:37:16 you have like a million, whatever the science is,

00:37:20 it just doesn’t sound right.

00:37:22 Well, so back when I was in college,

00:37:24 we did the Atkins diet, that was a thing.

00:37:26 Similar, but you have to always give these things a chance.

00:37:30 I mean, with dieting, I was not dieting,

00:37:33 but it’s just the things that you like.

00:37:35 If I eat, personally, if I eat meat,

00:37:38 just everything, I can be super focused

00:37:40 or more focused than usual.

00:37:42 I just feel great.

00:37:43 I mean, I’ve been running a lot,

00:37:46 doing pushups and pull ups and so on.

00:37:47 I mean, Python is similar in that sense for me.

00:37:50 Where are you going with this?

00:37:53 I mean, literally, I just felt I had like a stupid smile

00:37:57 on my face when I first started using Python.

00:38:00 I could code up really quick things.

00:38:02 Like I would see the world,

00:38:05 I would be empowered to write a script

00:38:07 to do some basic data processing,

00:38:11 to rename files on my computer.

00:38:14 Like Perl didn’t do that for me,

00:38:18 a little bit.

00:38:19 And again, none of these are about which is best

00:38:22 or something like that,

00:38:23 but there’s definitely better and worse here.

00:38:25 But it clicks, right?

00:38:26 Well, yeah.

00:38:27 If you look at Perl, for example,

00:38:29 you get bogged down in scalars versus arrays

00:38:32 versus hashes versus type globs

00:38:34 and like all that kind of stuff.

00:38:35 And Python’s like, yeah, let’s not do this.

00:38:38 And some of it is debugging.

00:38:39 Like everyone has different priorities.

00:38:41 But for me, it’s, can I create systems for myself

00:38:44 that empower me to debug quickly?

00:38:47 Like I’ve always been a big fan,

00:38:50 even just crude like asserts,

00:38:52 like always stating things that should be true,

00:38:57 which in Python, I found in myself doing more

00:38:59 because of type, all these kinds of stuff.

00:39:02 Well, you could think of types in a programming language

00:39:04 as being kind of assert.

00:39:05 Yeah.

00:39:06 They could check the compile time, right?

00:39:08 So how do you learn a new thing?

00:39:10 Well, so this, or how do people learn new things, right?

00:39:13 This is hard.

00:39:15 People don’t like to change.

00:39:17 People generally don’t like change around them either.

00:39:19 And so we’re all very slow to adapt and change.

00:39:22 And usually there’s a catalyst that’s required

00:39:25 to force yourself over this.

00:39:27 So for learning a programming language,

00:39:29 it really comes down to finding an excuse,

00:39:32 like build a thing that the language is actually good for,

00:39:36 that the ecosystem’s ready for.

00:39:38 And so if you were to write an iOS app, for example,

00:39:42 that’d be the easy case.

00:39:44 Obviously you would use Swift for that, right?

00:39:46 There are other…

00:39:47 Android.

00:39:48 So Swift runs on Android.

00:39:50 Oh, does it?

00:39:51 Oh yeah.

00:39:52 Yeah, Swift runs in lots of places.

00:39:53 How does that work?

00:39:54 So…

00:39:55 Okay, so Swift is built on top of LLVM.

00:39:58 LLVM runs everywhere.

00:40:00 LLVM, for example, builds the Android kernel.

00:40:03 Oh, okay.

00:40:04 So yeah.

00:40:05 I didn’t realize this.

00:40:06 Yeah, so Swift is very portable, runs on Windows.

00:40:09 There’s, it runs on lots of different things.

00:40:12 And Swift, sorry to interrupt, Swift UI,

00:40:15 and then there’s a thing called UI Kit.

00:40:17 So can I build an app with Swift?

00:40:21 Well, so that’s the thing,

00:40:22 is the ecosystem is what matters there.

00:40:23 So Swift UI and UI Kit are Apple technologies.

00:40:27 Okay, got it.

00:40:27 And so they happen to,

00:40:28 like Swift UI happens to be written in Swift,

00:40:30 but it’s an Apple proprietary framework

00:40:32 that Apple loves and wants to keep on its platform,

00:40:35 which makes total sense.

00:40:36 You go to Android and you don’t have that library, right?

00:40:39 And so Android has a different ecosystem of things

00:40:42 that hasn’t been built out

00:40:44 and doesn’t work as well with Swift.

00:40:45 And so you can totally use Swift to do like arithmetic

00:40:48 and things like this,

00:40:49 but building UI with Swift on Android

00:40:51 is not a great experience right now.

00:40:54 So if I wanted to learn Swift, what’s the,

00:40:58 I mean, the one practical different version of that

00:41:01 is Swift for TensorFlow, for example.

00:41:05 And one of the inspiring things for me

00:41:08 with both TensorFlow and PyTorch

00:41:10 is how quickly the community can like switch

00:41:13 from different libraries, like you could see

00:41:16 some of the communities switching to PyTorch now,

00:41:19 but it’s very easy to see.

00:41:21 And then TensorFlow is really stepping up its game.

00:41:24 And then there’s no reason why,

00:41:26 I think the way it works is basically

00:41:27 it has to be one GitHub repo,

00:41:29 like one paper steps up.

00:41:31 It gets people excited.

00:41:32 It gets people excited and they’re like,

00:41:33 ah, I have to learn this Swift for,

00:41:37 what’s Swift again?

00:41:39 And then they learn and they fall in love with it.

00:41:41 I mean, that’s what happened, PyTorch has it.

00:41:43 There has to be a reason, a catalyst.

00:41:44 Yeah.

00:41:45 And so, and there, I mean, people don’t like change,

00:41:48 but it turns out that once you’ve worked

00:41:50 with one or two programming languages,

00:41:52 the basics are pretty similar.

00:41:54 And so one of the fun things

00:41:55 about learning programming languages,

00:41:57 even maybe Lisp, I don’t know if you agree with this,

00:41:59 is that when you start doing that,

00:42:01 you start learning new things.

00:42:04 Cause you have a new way to do things

00:42:05 and you’re forced to do them.

00:42:06 And that forces you to explore

00:42:09 and it puts you in learning mode.

00:42:10 And when you get in learning mode,

00:42:11 your mind kind of opens a little bit

00:42:12 and you can see things in a new way,

00:42:15 even when you go back to the old place.

00:42:17 Right.

00:42:17 Yeah, it’s totally, well Lisp is functional.

00:42:19 Yeah.

00:42:21 But yeah, I wish there was a kind of window,

00:42:23 maybe you can tell me if there is, there you go.

00:42:26 This is a question to ask,

00:42:28 what is the most beautiful feature

00:42:29 in a programming language?

00:42:30 Before I ask it, let me say like with Python,

00:42:33 I remember I saw Lisp comprehensions.

00:42:36 Yeah.

00:42:37 Was like, when I like really took it in.

00:42:40 Yeah.

00:42:41 I don’t know, I just loved it.

00:42:43 It was like fun to do, like it was fun to do that kind of,

00:42:49 something about it to be able to filter through a list

00:42:52 and to create a new list all in a single line was elegant.

00:42:56 I could all get into my head

00:42:58 and it just made me fall in love with the language.

00:43:01 Yeah.

00:43:02 So is there, let me ask you a question.

00:43:04 Is there, what do you use the most beautiful feature

00:43:07 in a programming languages that you’ve ever encountered

00:43:11 in Swift maybe and then outside of Swift?

00:43:15 I think the thing that I like the most

00:43:17 from a programming language.

00:43:18 So I think the thing you have to think about

00:43:21 with the programming language, again, what is the goal?

00:43:23 You’re trying to get people to get things done quickly.

00:43:27 And so you need libraries, you need high quality libraries

00:43:30 and then you need a user base around them

00:43:32 that can assemble them and do cool things with them, right?

00:43:35 And so to me, the question is

00:43:36 what enables high quality libraries?

00:43:39 Okay.

00:43:40 Yeah.

00:43:41 And there’s a huge divide in the world

00:43:43 between libraries who enable high quality libraries

00:43:48 versus the ones that put special stuff in the language.

00:43:52 So programming languages that enable high quality libraries?

00:43:56 Got it.

00:43:57 So, and what I mean by that is expressive libraries

00:44:00 that then feel like a natural integrated part

00:44:03 of the language itself.

00:44:05 So an example of this in Swift is that int and float

00:44:09 and also array and string, things like this,

00:44:12 these are all part of the library.

00:44:13 Like int is not hard coded into Swift.

00:44:17 And so what that means is that

00:44:19 because int is just a library thing

00:44:21 defined in the standard library,

00:44:22 along with strings and arrays and all the other things

00:44:24 that come with the standard library.

00:44:27 Well, hopefully you do like int,

00:44:29 but anything that any language features

00:44:31 that you needed to define int,

00:44:33 you can also use in your own types.

00:44:36 So if you wanted to find a quaternion

00:44:39 or something like this, right?

00:44:41 Well, it doesn’t come in the standard library.

00:44:43 There’s a very special set of people

00:44:45 that care a lot about this,

00:44:47 but those people are also important.

00:44:49 It’s not about classism, right?

00:44:51 It’s not about the people who care about ints and floats

00:44:53 are more important than the people who care about quaternions.

00:44:55 And so to me, the beautiful things

00:44:56 about programming languages is when you allow

00:44:58 those communities to build high quality libraries,

00:45:02 they feel native.

00:45:03 They feel like they’re built into the compiler

00:45:05 without having to be.

00:45:08 What does it mean for the int to be part

00:45:11 of not hard coded in?

00:45:13 So is it like how, so what is an int?

00:45:18 Okay, int is just a integer.

00:45:20 In this case, it’s like a 64 bit integer

00:45:23 or something like this.

00:45:24 But so like the 64 bit is hard coded or no?

00:45:28 No, none of that’s hard coded.

00:45:29 So int, if you go look at how it’s implemented,

00:45:32 it’s just a struct in Swift.

00:45:34 And so it’s a struct.

00:45:35 And then how do you add two structs?

00:45:37 Well, you define plus.

00:45:39 And so you can define plus on int.

00:45:41 Well, you can define plus on your thing too.

00:45:43 You can define, int is an odd method

00:45:46 or something like that on it.

00:45:47 And so yeah, you can add methods on the things.

00:45:50 Yeah.

00:45:51 So you can define operators, like how it behaves.

00:45:55 That’s just beautiful when there’s something

00:45:57 about the language which enables others

00:46:00 to create libraries which are not hacky.

00:46:05 Yeah, they feel native.

00:46:07 And so one of the best examples of this is Lisp, right?

00:46:10 Because in Lisp, like all the libraries

00:46:13 are basically part of the language, right?

00:46:15 You write, turn, rewrite systems and things like this.

00:46:17 And so.

00:46:18 Can you as a counter example provide

00:46:20 what makes it difficult to write a library that’s native?

00:46:23 Is it the Python C?

00:46:25 Well, so one example, I’ll give you two examples.

00:46:29 Java and C++, there’s Java and C.

00:46:33 They both allow you to define your own types,

00:46:35 but int is hard code in the language.

00:46:38 Okay, well, why?

00:46:39 Well, in Java, for example, coming back

00:46:41 to this whole reference semantic value semantic thing,

00:46:45 int gets passed around by value.

00:46:48 Yeah.

00:46:49 But if you make like a pair or something like that,

00:46:53 a complex number, right, it’s a class in Java.

00:46:56 And now it gets passed around by reference, by pointer.

00:46:59 And so now you lose value semantics, right?

00:47:02 You lost math, okay.

00:47:04 Well, that’s not great, right?

00:47:06 If you can do something with int,

00:47:08 why can’t I do it with my type, right?

00:47:10 So that’s the negative side of the thing I find beautiful

00:47:15 is when you can solve that,

00:47:17 when you can have full expressivity,

00:47:19 where you as a user of the language

00:47:21 have as much or almost as much power

00:47:24 as the people who implemented

00:47:25 all the standard built in stuff,

00:47:27 because what that enables

00:47:28 is that enables truly beautiful libraries.

00:47:31 You know, it’s kind of weird

00:47:32 because I’ve gotten used to that.

00:47:36 That’s one, I guess, other aspect

00:47:37 of program language design.

00:47:39 You have to think, you know,

00:47:41 the old first principles thinking,

00:47:43 like, why are we doing it this way?

00:47:45 By the way, I mean, I remember,

00:47:47 because I was thinking about the walrus operator

00:47:50 and I’ll ask you about it later,

00:47:53 but it hit me that like the equal sign for assignment.

00:47:57 Yeah.

00:47:58 Like, why are we using the equal sign for assignment?

00:48:00 It’s wrong, yeah.

00:48:02 And that’s not the only solution, right?

00:48:04 So if you look at Pascal,

00:48:05 they use colon equals for assignment

00:48:07 and equals for equality.

00:48:11 And they use like less than greater than

00:48:12 instead of the not equal thing.

00:48:14 Yeah.

00:48:15 Like, there are other answers here.

00:48:16 So, but like, and yeah, like I ask you all,

00:48:19 but how do you then decide to break convention

00:48:24 to say, you know what, everybody’s doing it wrong.

00:48:29 We’re gonna do it right.

00:48:30 Yeah.

00:48:31 So it’s like an ROI,

00:48:33 like return on investment trade off, right?

00:48:35 So if you do something weird,

00:48:37 let’s just say like not like colon equal

00:48:39 instead of equal for assignment,

00:48:40 that would be weird with today’s aesthetic, right?

00:48:44 And so you’d say, cool, this is theoretically better,

00:48:47 but is it better in which ways?

00:48:49 Like, what do I get out of that?

00:48:50 Do I define away class of bugs?

00:48:52 Well, one of the class of bugs that C has

00:48:54 is that you can use like, you know,

00:48:55 if X equals without equals equals X equals Y, right?

00:49:01 Well, turns out you can solve that problem in lots of ways.

00:49:05 Clang, for example, GCC, all these compilers

00:49:07 will detect that as a likely bug, produce a warning.

00:49:10 Do they?

00:49:11 Yeah.

00:49:12 I feel like they didn’t.

00:49:13 Oh, Clang does.

00:49:14 They didn’t.

00:49:14 GCC didn’t.

00:49:15 It’s like one of the important things

00:49:17 about programming language design is like,

00:49:19 you’re literally creating suffering in the world.

00:49:22 Okay.

00:49:23 Like, I feel like, I mean, one way to see it

00:49:27 is the bicycle for the mind,

00:49:29 but the other way is to like minimizing suffering.

00:49:32 Well, you have to decide if it’s worth it, right?

00:49:33 And so let’s come back to that.

00:49:35 Okay.

00:49:36 But if you look at this, and again,

00:49:38 this is where there’s a lot of detail

00:49:40 that goes into each of these things.

00:49:42 Equal and C returns a value.

00:49:46 Yep.

00:49:47 Is it messed up?

00:49:48 That allows you to say X equals Y equals Z,

00:49:51 like that works in C.

00:49:52 Yeah.

00:49:53 Is it messed up?

00:49:54 You know, most people think it’s messed up, I think.

00:49:57 It is very, by messed up, what I mean is

00:50:00 it is very rarely used for good,

00:50:03 and it’s often used for bugs.

00:50:05 Yeah.

00:50:06 Right, and so.

00:50:07 That’s a good definition of messed up, yeah.

00:50:09 You could use, you know, in hindsight,

00:50:12 this was not such a great idea, right?

00:50:13 No.

00:50:14 One of the things with Swift that is really powerful

00:50:16 and one of the reasons it’s actually good

00:50:18 versus it being full of good ideas

00:50:20 is that when we launched Swift 1,

00:50:23 we announced that it was public,

00:50:26 people could use it, people could build apps,

00:50:27 but it was gonna change and break, okay?

00:50:30 When Swift 2 came out, we said, hey, it’s open source,

00:50:33 and there’s this open process

00:50:34 which people can help evolve and direct the language.

00:50:37 So the community at large, like Swift users,

00:50:40 can now help shape the language as it is.

00:50:43 And what happened as part of that process is

00:50:46 a lot of really bad mistakes got taken out.

00:50:49 So for example, Swift used to have the C style plus plus

00:50:53 and minus minus operators.

00:50:55 Like, what does it mean when you put it before

00:50:56 versus after, right?

00:50:59 Well, that got cargo culted from C into Swift early on.

00:51:02 What’s cargo culted?

00:51:03 Cargo culted means brought forward

00:51:05 without really considering it.

00:51:07 Okay.

00:51:08 This is maybe not the most PC term, but.

00:51:12 You have to look it up in Urban Dictionary, yeah.

00:51:13 Yeah, so it got pulled into C without,

00:51:17 or it got pulled into Swift

00:51:18 without very good consideration.

00:51:20 And we went through this process,

00:51:22 and one of the first things got ripped out

00:51:23 was plus plus and minus minus,

00:51:25 because they lead to confusion.

00:51:27 They have very low value over saying x plus equals one,

00:51:31 and x plus equals one is way more clear.

00:51:34 And so when you’re optimizing for teachability and clarity

00:51:36 and bugs and this multidimensional space

00:51:39 that you’re looking at,

00:51:40 things like that really matter.

00:51:42 And so being first principles on where you’re coming from

00:51:45 and what you’re trying to achieve

00:51:46 and being anchored on the objective is really important.

00:51:50 Well, let me ask you about the most,

00:51:54 sort of this podcast isn’t about information,

00:51:58 it’s about drama.

00:51:59 Okay.

00:52:00 Let me talk to you about some drama.

00:52:01 So you mentioned Pascal and colon equals,

00:52:06 there’s something that’s called the Walrus operator.

00:52:08 Okay.

00:52:09 And Python in Python 3.8 added the Walrus operator.

00:52:15 And the reason I think it’s interesting

00:52:19 is not just because of the feature,

00:52:21 it has the same kind of expression feature

00:52:23 you can mention to see that it returns

00:52:25 the value of the assignment.

00:52:27 And then maybe you can comment on that in general,

00:52:29 but on the other side of it,

00:52:31 it’s also the thing that toppled the dictator.

00:52:36 So it finally drove Guido

00:52:39 to step down from BDFL, the toxicity of the community.

00:52:42 So maybe what do you think about the Walrus operator

00:52:46 in Python?

00:52:47 Is there an equivalent thing in Swift

00:52:50 that really stress tested the community?

00:52:54 And then on the flip side,

00:52:56 what do you think about Guido stepping down over it?

00:52:58 Yeah, well, if I look past the details

00:53:01 of the Walrus operator,

00:53:02 one of the things that makes it most polarizing

00:53:04 is that it’s syntactic sugar.

00:53:06 Okay.

00:53:07 What do you mean by syntactic sugar?

00:53:09 It means you can take something

00:53:10 that already exists in the language

00:53:11 and you can express it in a more concise way.

00:53:14 So, okay, I’m going to play dollars advocate.

00:53:15 So this is great.

00:53:18 Is that a objective or subjective statement?

00:53:21 Like, can you argue that basically anything

00:53:24 isn’t syntactic sugar or not?

00:53:27 No, not everything is syntactic sugar.

00:53:30 So for example, the type system,

00:53:32 like can you have classes versus,

00:53:35 like, do you have types or not, right?

00:53:37 So one type versus many types

00:53:40 is not something that affects syntactic sugar.

00:53:42 And so if you say,

00:53:43 I want to have the ability to define types,

00:53:46 I have to have all this like language mechanics

00:53:47 to define classes.

00:53:49 And oh, now I have to have inheritance.

00:53:51 And I have like, I have all this stuff

00:53:52 that’s just making the language more complicated.

00:53:54 That’s not about sugaring it.

00:53:58 Swift has the sugar.

00:54:00 So like Swift has this thing called if let,

00:54:02 and it has a lot of different types

00:54:04 and it has various operators

00:54:06 that are used to concisify specific use cases.

00:54:10 So the problem with syntactic sugar,

00:54:12 when you’re talking about,

00:54:14 hey, I have a thing that takes a lot to write

00:54:16 and I have a new way to write it.

00:54:17 You have this like horrible trade off,

00:54:19 which becomes almost completely subjective,

00:54:22 which is how often does this happen and does it matter?

00:54:26 And one of the things that is true about human psychology,

00:54:28 particularly when you’re talking about introducing

00:54:29 a new thing is that people overestimate

00:54:34 the burden of learning something.

00:54:36 And so it looks foreign when you haven’t gotten used to it.

00:54:38 But if it was there from the beginning,

00:54:40 of course it’s just part of Python.

00:54:42 Like unquestionably, like this is just the thing I know.

00:54:45 And it’s not a new thing that you’re worried about learning.

00:54:47 It’s just part of the deal.

00:54:49 Now with Guido, I don’t know Guido well.

00:54:55 Yeah, have you passed cross much?

00:54:56 Yeah, I’ve met him a couple of times,

00:54:58 but I don’t know Guido well.

00:55:00 But the sense that I got out of that whole dynamic

00:55:03 was that he had put the,

00:55:04 not just the decision maker weight on his shoulders,

00:55:08 but it was so tied to his personal identity

00:55:11 that he took it personally and he felt the need

00:55:15 and he kind of put himself in the situation

00:55:16 of being the person,

00:55:18 instead of building a base of support around him.

00:55:20 I mean, this is probably not quite literally true,

00:55:23 but by too much concentrated on him, right?

00:55:29 And that can wear you down.

00:55:31 Well, yeah, particularly because people then say,

00:55:33 Guido, you’re a horrible person.

00:55:35 I hate this thing, blah, blah, blah, blah, blah, blah, blah.

00:55:37 And sure, it’s like maybe 1% of the community

00:55:40 that’s doing that, but Python’s got a big community.

00:55:43 And 1% of millions of people is a lot of hate mail.

00:55:46 And that just from human factor will just wear on you.

00:55:49 Well, to clarify, it looked from just what I saw

00:55:52 in the messaging for the,

00:55:53 let’s not look at the million Python users,

00:55:55 but at the Python core developers,

00:55:58 it feels like the majority, the big majority

00:56:01 on a vote were opposed to it.

00:56:03 Okay, I’m not that close to it, so I don’t know.

00:56:06 Okay, so the situation is like literally,

00:56:10 yeah, I mean, the majority of the core developers

00:56:13 are against it.

00:56:13 Were opposed to it.

00:56:14 So, and they weren’t even like against it.

00:56:20 It was, there was a few, well, they were against it,

00:56:23 but the against it wasn’t like, this is a bad idea.

00:56:27 They were more like, we don’t see why this is a good idea.

00:56:31 And what that results in is there’s a stalling feeling,

00:56:35 like you just slow things down.

00:56:37 Now, from my perspective, that you could argue this,

00:56:41 and I think it’s very interesting

00:56:44 if we look at politics today and the way Congress works,

00:56:47 it’s slowed down everything.

00:56:49 It’s a dampener.

00:56:50 Yeah, it’s a dampener, but like,

00:56:51 that’s a dangerous thing too,

00:56:53 because if it dampens things like, you know,

00:56:57 if the dampening results.

00:56:58 What are you talking about?

00:56:59 Like, it’s a low pass filter,

00:57:00 but if you need billions of dollars

00:57:02 injected into the economy or trillions of dollars,

00:57:05 then suddenly stuff happens, right?

00:57:06 And so.

00:57:07 For sure.

00:57:09 So you’re talking about.

00:57:10 I’m not defending our political situation,

00:57:11 just to be clear.

00:57:13 But you’re talking about like a global pandemic.

00:57:16 Well.

00:57:17 I was hoping we could fix like the healthcare system

00:57:20 and the education system, like, you know.

00:57:22 I’m not a politics person.

00:57:24 I don’t know.

00:57:26 When it comes to languages,

00:57:28 the community’s kind of right in terms of,

00:57:30 it’s a very high burden to add something to a language.

00:57:33 So as soon as you add something,

00:57:34 you have a community of people building on it

00:57:35 and you can’t remove it, okay?

00:57:38 And if there’s a community of people

00:57:39 that feel really uncomfortable with it,

00:57:41 then taking it slow, I think, is an important thing to do.

00:57:45 And there’s no rush, particularly if it was something

00:57:48 that’s 25 years old and is very established.

00:57:50 And, you know, it’s not like coming into its own.

00:57:54 What about features?

00:57:55 Well, so I think that the issue with Guido

00:57:58 is that maybe this is a case

00:58:00 where he realized it had outgrown him

00:58:03 and it went from being the language.

00:58:06 So Python, I mean, Guido’s amazing,

00:58:09 but Python isn’t about Guido anymore.

00:58:12 It’s about the users.

00:58:13 And to a certain extent, the users own it.

00:58:15 And, you know, Guido spent years of his life,

00:58:19 a significant fraction of his career on Python.

00:58:22 And from his perspective, I imagine he’s like,

00:58:24 well, this is my thing.

00:58:25 I should be able to do the thing I think is right.

00:58:28 But you can also understand the users

00:58:30 where they feel like, you know, this is my thing.

00:58:33 I use this, like, and I don’t know, it’s a hard thing.

00:58:38 But what, if we could talk about leadership in this,

00:58:41 because it’s so interesting to me.

00:58:42 I’m gonna make, I’m gonna work.

00:58:44 Hopefully somebody makes it.

00:58:45 If not, I’ll make it a Walrus Operator shirt,

00:58:47 because I think it represents, to me,

00:58:50 maybe it’s my Russian roots or something.

00:58:52 But, you know, it’s the burden of leadership.

00:58:56 Like, I feel like to push back,

00:59:01 I feel like progress can only,

00:59:02 like most difficult decisions, just like you said,

00:59:06 there’ll be a lot of divisiveness over,

00:59:09 especially in a passionate community.

00:59:12 It just feels like leaders need to take

00:59:14 those risky decisions that if you like listen,

00:59:19 that with some nonzero probability,

00:59:23 maybe even a high probability would be the wrong decision.

00:59:26 But they have to use their gut and make that decision.

00:59:29 Well, this is like one of the things

00:59:30 where you see amazing founders.

00:59:34 The founders understand exactly what’s happened

00:59:36 and how the company got there and are willing to say,

00:59:39 we have been doing thing X the last 20 years,

00:59:42 but today we’re gonna do thing Y.

00:59:45 And they make a major pivot for the whole company.

00:59:47 The company lines up behind them,

00:59:48 they move and it’s the right thing.

00:59:50 But then when the founder dies,

00:59:52 the successor doesn’t always feel that agency

00:59:57 to be able to make those kinds of decisions.

00:59:59 Even though they’re a CEO,

01:00:00 they could theoretically do whatever.

01:00:02 There’s two reasons for that, in my opinion,

01:00:04 or in many cases, it’s always different.

01:00:07 But one of which is they weren’t there

01:00:09 for all the decisions that were made.

01:00:11 And so they don’t know the principles

01:00:13 in which those decisions were made.

01:00:15 And once the principles change,

01:00:17 you should be obligated to change what you’re doing

01:00:20 and change direction, right?

01:00:22 And so if you don’t know how you got to where you are,

01:00:25 it just seems like gospel

01:00:27 and you’re not gonna question it.

01:00:29 You may not understand

01:00:30 that it really is the right thing to do,

01:00:32 so you just may not see it.

01:00:33 That’s so brilliant.

01:00:34 I never thought of it that way.

01:00:35 Like it’s so much higher burden

01:00:38 when as a leader you step into a thing

01:00:40 that’s already worked for a long time.

01:00:41 Yeah, yeah.

01:00:42 Well, and if you change it and it doesn’t work out,

01:00:44 now you’re the person who screwed it up.

01:00:46 People always second guess it.

01:00:47 Yeah.

01:00:48 And the second thing is that

01:00:49 even if you decide to make a change,

01:00:51 even if you’re theoretically in charge,

01:00:53 you’re just a person that thinks they’re in charge.

01:00:57 Meanwhile, you have to motivate the troops.

01:00:58 You have to explain it to them

01:00:59 in terms they’ll understand.

01:01:00 You have to get them to buy into it and believe in it,

01:01:02 because if they don’t,

01:01:03 then they’re not gonna be able to make the turn

01:01:05 even if you tell them their bonuses are gonna be curtailed.

01:01:08 They’re just not gonna like buy into it, you know?

01:01:10 And so there’s only so much power you have as a leader,

01:01:12 and you have to understand what those limitations are.

01:01:16 Are you still BDFL?

01:01:18 You’ve been a BDFL of some stuff.

01:01:21 You’re very heavy on the B,

01:01:24 the benevolent, benevolent dictator for life.

01:01:27 I guess LLVM?

01:01:29 Yeah, so I still lead the LLVM world.

01:01:32 I mean, what’s the role of,

01:01:35 so then on Swift you said that there’s a group of people.

01:01:38 Yeah, so if you contrast Python with Swift, right,

01:01:41 one of the reasons,

01:01:43 so everybody on the core team takes the role

01:01:45 really seriously, and I think we all really care

01:01:47 about where Swift goes,

01:01:49 but you’re almost delegating the final decision making

01:01:52 to the wisdom of the group,

01:01:54 and so it doesn’t become personal.

01:01:57 And also, when you’re talking with the community,

01:01:59 so yeah, some people are very annoyed

01:02:02 as certain decisions get made.

01:02:04 There’s a certain faith in the process,

01:02:06 because it’s a very transparent process,

01:02:08 and when a decision gets made,

01:02:09 a full rationale is provided, things like this.

01:02:12 These are almost defense mechanisms

01:02:14 to help both guide future discussions

01:02:16 and provide case law, kind of like Supreme Court does

01:02:18 about this decision was made for this reason,

01:02:20 and here’s the rationale

01:02:21 and what we want to see more of or less of.

01:02:25 But it’s also a way to provide a defense mechanism,

01:02:27 so that when somebody’s griping about it,

01:02:28 they’re not saying that person did the wrong thing.

01:02:31 They’re saying, well, this thing sucks,

01:02:33 and later they move on and they get over it.

01:02:38 Yeah, the analogy of the Supreme Court,

01:02:40 I think, is really good.

01:02:42 But then, okay, not to get personal on the SWIFT team,

01:02:45 but it just seems like it’s impossible

01:02:50 for division not to emerge.

01:02:52 Well, each of the humans on the SWIFT Core Team,

01:02:55 for example, are different,

01:02:56 and the membership of the SWIFT Core Team

01:02:58 changes slowly over time, which is, I think, a healthy thing.

01:03:02 And so each of these different humans

01:03:04 have different opinions.

01:03:05 Trust me, it’s not a singular consciousness

01:03:09 by any stretch of the imagination.

01:03:11 You’ve got three major organizations,

01:03:12 including Apple, Google, and SciFive,

01:03:14 all kind of working together.

01:03:16 And it’s a small group of people, but you need high trust.

01:03:20 You need, again, it comes back to the principles

01:03:21 of what you’re trying to achieve

01:03:23 and understanding what you’re optimizing for.

01:03:27 And I think that starting with strong principles

01:03:30 and working towards decisions

01:03:32 is always a good way to both make wise decisions in general

01:03:36 but then be able to communicate them to people

01:03:37 so that they can buy into them.

01:03:39 And that is hard.

01:03:41 And so you mentioned LLVM.

01:03:42 LLVM is gonna be 20 years old this December,

01:03:46 so it’s showing its own age.

01:03:49 Do you have like a dragon cake plan?

01:03:53 No, I should definitely do that.

01:03:54 Yeah, if we can have a pandemic cake.

01:03:57 Pandemic cake.

01:03:58 Everybody gets a slice of cake

01:04:00 and it gets sent through email.

01:04:04 But LLVM has had tons of its own challenges

01:04:08 over time too, right?

01:04:09 And one of the challenges that the LLVM community has,

01:04:12 in my opinion, is that it has a whole bunch of people

01:04:15 that have been working on LLVM for 10 years, right?

01:04:19 Because this happens somehow.

01:04:20 And LLVM has always been one way,

01:04:22 but it needs to be a different way, right?

01:04:25 And they’ve worked on it for like 10 years.

01:04:26 It’s a long time to work on something.

01:04:28 And you suddenly can’t see the faults

01:04:32 in the thing that you’re working on.

01:04:33 And LLVM has lots of problems and we need to address them

01:04:35 and we need to make it better.

01:04:36 And if we don’t make it better,

01:04:37 then somebody else will come up with a better idea, right?

01:04:40 And so it’s just kind of of that age

01:04:42 where the community is like in danger

01:04:45 of getting too calcified.

01:04:46 And so I’m happy to see new projects joining

01:04:50 and new things mixing it up.

01:04:51 Fortran is now a new thing in the LLVM community,

01:04:54 which is hilarious and good.

01:04:56 I’ve been trying to find, on a little tangent,

01:04:58 find people who program in Cobalt or Fortran,

01:05:02 Fortran especially, to talk to, they’re hard to find.

01:05:06 Yeah, look to the scientific community.

01:05:09 They still use Fortran quite a bit.

01:05:11 Well, interesting thing you kind of mentioned with LLVM,

01:05:14 or just in general, that as something evolves,

01:05:16 you’re not able to see the faults.

01:05:19 So do you fall in love with the thing over time?

01:05:23 Or do you start hating everything

01:05:24 about the thing over time?

01:05:26 Well, so my personal folly is that I see,

01:05:31 maybe not all, but many of the faults,

01:05:33 and they grate on me, and I don’t have time to go fix them.

01:05:35 Yeah, and they get magnified over time.

01:05:37 Well, and they may not get magnified,

01:05:38 but they never get fixed.

01:05:39 And it’s like sand underneath,

01:05:41 you know, it’s just like grating against you.

01:05:43 And it’s like sand underneath your fingernails or something.

01:05:45 It’s just like, you know it’s there,

01:05:46 you can’t get rid of it.

01:05:49 And so the problem is that if other people don’t see it,

01:05:52 like I don’t have time to go write the code

01:05:56 and fix it anymore,

01:05:58 but then people are resistant to change.

01:06:01 And so you say, hey, we should go fix this thing.

01:06:03 They’re like, oh yeah, that sounds risky.

01:06:05 It’s like, well, is it the right thing or not?

01:06:07 Are the challenges the group dynamics,

01:06:10 or is it also just technical?

01:06:11 I mean, some of these features like,

01:06:14 I think as an observer, it’s almost like a fan

01:06:17 in the, you know, as a spectator of the whole thing,

01:06:21 I don’t often think about, you know,

01:06:23 some things might actually be

01:06:24 technically difficult to implement.

01:06:27 An example of this is we built this new compiler framework

01:06:30 called MLIR.

01:06:31 Yes.

01:06:32 MLIR is a whole new framework.

01:06:34 It’s not, many people think it’s about machine learning.

01:06:37 The ML stands for multi level

01:06:39 because compiler people can’t name things very well,

01:06:41 I guess.

01:06:42 Do we dig into what MLIR is?

01:06:45 Yeah, so when you look at compilers,

01:06:47 compilers have historically been solutions for a given space.

01:06:51 So LLVM is a, it’s really good for dealing with CPUs,

01:06:56 let’s just say, at a high level.

01:06:58 You look at Java, Java has a JVM.

01:07:01 The JVM is very good for garbage collected languages

01:07:04 that need dynamic compilation,

01:07:05 and it’s very optimized for a specific space.

01:07:08 And so hotspot is one of the compilers

01:07:09 that gets used in that space,

01:07:11 and that compiler is really good at that kind of stuff.

01:07:14 Usually when you build these domain specific compilers,

01:07:16 you end up building the whole thing from scratch

01:07:19 for each domain.

01:07:22 What’s a domain?

01:07:23 So what’s the scope of a domain?

01:07:26 Well, so here I would say, like, if you look at Swift,

01:07:29 there’s several different parts to the Swift compiler,

01:07:31 one of which is covered by the LLVM part of it.

01:07:36 There’s also a high level piece that’s specific to Swift,

01:07:39 and there’s a huge amount of redundancy

01:07:41 between those two different infrastructures

01:07:44 and a lot of re implemented stuff

01:07:46 that is similar but different.

01:07:48 What does LLVM define?

01:07:49 LLVM is effectively an infrastructure.

01:07:53 So you can mix and match it in different ways.

01:07:55 It’s built out of libraries.

01:07:56 You can use it for different things,

01:07:57 but it’s really good at CPUs and GPUs.

01:07:59 CPUs and like the tip of the iceberg on GPUs.

01:08:02 It’s not really great at GPUs.

01:08:04 Okay.

01:08:05 But it turns out. A bunch of languages that.

01:08:07 That then use it to talk to CPUs.

01:08:10 Got it.

01:08:11 And so it turns out there’s a lot of hardware out there

01:08:13 that is custom accelerators.

01:08:14 So machine learning, for example.

01:08:16 There are a lot of matrix multiply accelerators

01:08:18 and things like this.

01:08:20 There’s a whole world of hardware synthesis.

01:08:22 So we’re using MLIR to build circuits.

01:08:26 Okay.

01:08:27 And so you’re compiling for a domain of transistors.

01:08:30 And so what MLIR does is it provides

01:08:32 a tremendous amount of compiler infrastructure

01:08:34 that allows you to build these domain specific compilers

01:08:37 in a much faster way and have the result be good.

01:08:41 If we’re thinking about the future,

01:08:44 now we’re talking about like ASICs.

01:08:45 So anything.

01:08:46 Yeah, yeah.

01:08:47 So if we project into the future,

01:08:50 it’s very possible that the number of these kinds of ASICs,

01:08:54 very specific infrastructure architecture things

01:09:02 like multiplies exponentially.

01:09:05 I hope so.

01:09:06 So that’s MLIR.

01:09:08 So what MLIR does is it allows you

01:09:10 to build these compilers very efficiently.

01:09:13 Right now, one of the things that coming back

01:09:15 to the LLVM thing, and then we’ll go to hardware,

01:09:17 is LLVM is a specific compiler for a specific domain.

01:09:23 MLIR is now this very general, very flexible thing

01:09:26 that can solve lots of different kinds of problems.

01:09:29 So LLVM is a subset of what MLIR does.

01:09:32 So MLIR is, I mean, it’s an ambitious project then.

01:09:35 Yeah, it’s a very ambitious project, yeah.

01:09:36 And so to make it even more confusing,

01:09:39 MLIR has joined the LLVM Umbrella Project.

01:09:42 So it’s part of the LLVM family.

01:09:44 Right.

01:09:45 But where this comes full circle is now folks

01:09:47 that work on the LLVM part,

01:09:49 the classic part that’s 20 years old,

01:09:51 aren’t aware of all the cool new things

01:09:54 that have been done in the new thing,

01:09:56 that MLIR was built by me and many other people

01:09:59 that knew a lot about LLVM,

01:10:01 and so we fixed a lot of the mistakes that lived in LLVM.

01:10:05 And so now you have this community dynamic

01:10:07 where it’s like, well, there’s this new thing,

01:10:08 but it’s not familiar, nobody knows it,

01:10:10 it feels like it’s new, and so let’s not trust it.

01:10:12 And so it’s just really interesting

01:10:13 to see the cultural social dynamic that comes out of that.

01:10:16 And I think it’s super healthy

01:10:19 because we’re seeing the ideas percolate

01:10:21 and we’re seeing the technology diffusion happen

01:10:24 as people get more comfortable with it,

01:10:25 they start to understand things in their own terms.

01:10:27 And this just gets to the,

01:10:28 it takes a while for ideas to propagate,

01:10:31 even though they may be very different

01:10:33 than what people are used to.

01:10:35 So maybe let’s talk about that a little bit,

01:10:37 the world of Asics.

01:10:38 Yeah.

01:10:39 Actually, you have a new role at SciFive.

01:10:45 What’s that place about?

01:10:47 What is the vision for their vision

01:10:50 for, I would say, the future of computer?

01:10:52 Yeah, so I lead the engineering and product teams at SciFive.

01:10:55 SciFive is a company who was founded

01:10:59 with this architecture called RISC5.

01:11:02 RISC5 is a new instruction set.

01:11:04 Instruction sets are the things inside of your computer

01:11:06 that tell it how to run things.

01:11:08 X86 from Intel and ARM from the ARM company

01:11:12 and things like this are other instruction sets.

01:11:13 I’ve talked to, sorry to interrupt,

01:11:15 I’ve talked to Dave Patterson,

01:11:15 who’s super excited about RISC5.

01:11:17 Dave is awesome.

01:11:18 Yeah, he’s brilliant, yeah.

01:11:20 The RISC5 is distinguished by not being proprietary.

01:11:24 And so X86 can only be made by Intel and AMD.

01:11:28 ARM can only be made by ARM.

01:11:30 They sell licenses to build ARM chips to other companies,

01:11:33 things like this.

01:11:34 MIPS is another instruction set

01:11:35 that is owned by the MIPS company, now Wave.

01:11:38 And then it gets licensed out, things like that.

01:11:40 And so RISC5 is an open standard

01:11:43 that anybody can build chips for.

01:11:45 And so SciFive was founded by three of the founders

01:11:48 of RISC5 that designed and built it in Berkeley,

01:11:51 working with Dave.

01:11:52 And so that was the genesis of the company.

01:11:56 SciFive today has some of the world’s best RISC5 cores

01:11:59 and we’re selling them and that’s really great.

01:12:01 They’re going to tons of products, it’s very exciting.

01:12:04 So they’re taking this thing that’s open source

01:12:06 and just trying to be or are the best in the world

01:12:09 at building these things.

01:12:10 Yeah, so here it’s the specifications open source.

01:12:13 It’s like saying TCP IP is an open standard

01:12:15 or C is an open standard,

01:12:18 but then you have to build an implementation

01:12:19 of the standard.

01:12:20 And so SciFive, on the one hand, pushes forward

01:12:23 and defined and pushes forward the standard.

01:12:26 On the other hand, we have implementations

01:12:28 that are best in class for different points in the space,

01:12:30 depending on if you want a really tiny CPU

01:12:33 or if you want a really big, beefy one that is faster,

01:12:36 but it uses more area and things like this.

01:12:38 What about the actual manufacturer chips?

01:12:41 So like, where does that all fit?

01:12:43 I’m going to ask a bunch of dumb questions.

01:12:45 That’s okay, this is how we learn, right?

01:12:48 And so the way this works is that there’s generally

01:12:52 a separation of the people who designed the circuits

01:12:55 and then people who manufacture them.

01:12:56 And so you’ll hear about fabs like TSMC and Samsung

01:13:00 and things like this that actually produce the chips,

01:13:03 but they take a design coming in

01:13:05 and that design specifies how the,

01:13:09 you turn code for the chip into little rectangles

01:13:16 that then use photolithography to make mask sets

01:13:20 and then burn transistors onto a chip

01:13:22 or onto a, onto silicon rather.

01:13:24 So, and we’re talking about mass manufacturing, so.

01:13:28 Yeah, they’re talking about making hundreds of millions

01:13:29 of parts and things like that, yeah.

01:13:31 And so the fab handles the volume production,

01:13:33 things like that.

01:13:34 But when you look at this problem,

01:13:37 the interesting thing about the space when you look at it

01:13:39 is that these, the steps that you go from designing a chip

01:13:44 and writing the quote unquote code for it

01:13:46 and things like Verilog and languages like that,

01:13:49 down to what you hand off to the fab

01:13:51 is a really well studied, really old problem, okay?

01:13:56 Tons of people have worked on it.

01:13:57 Lots of smart people have built systems and tools.

01:14:00 These tools then have generally gone through acquisitions.

01:14:03 And so they’ve ended up at three different major companies

01:14:06 that build and sell these tools.

01:14:07 They’re called the EDA tools

01:14:08 like for electronic design automation.

01:14:11 The problem with this is you have huge amounts

01:14:13 of fragmentation, you have loose standards

01:14:17 and the tools don’t really work together.

01:14:20 So you have tons of duct tape

01:14:21 and you have tons of loss productivity.

01:14:24 Now these are, these are tools for designing.

01:14:26 So the RISC 5 is a instruction.

01:14:30 Like what is RISC 5?

01:14:32 Like how deep does it go?

01:14:33 How much does it touch the hardware?

01:14:35 How much does it define how much of the hardware is?

01:14:38 Yeah, so RISC 5 is all about given a CPU.

01:14:41 So the processor and your computer,

01:14:44 how does the compiler like Swift compiler,

01:14:47 the C compiler, things like this, how does it make it work?

01:14:50 So it’s, what is the assembly code?

01:14:52 And so you write RISC 5 assembly

01:14:54 instead of XA6 assembly, for example.

01:14:57 But it’s a set of instructions

01:14:58 as opposed to instructions.

01:15:00 Why do you say it tells you how the compiler works?

01:15:03 Sorry, it’s what the compiler talks to.

01:15:05 Okay. Yeah.

01:15:06 And then the tooling you mentioned

01:15:08 that the disparate tools are for what?

01:15:10 For when you’re building a specific chip.

01:15:13 So RISC 5. In hardware.

01:15:14 In hardware, yeah.

01:15:15 So RISC 5, you can buy a RISC 5 core from SciFive

01:15:19 and say, hey, I want to have a certain number of,

01:15:21 run a certain number of gigahertz.

01:15:23 I want it to be this big.

01:15:24 I want it to be, have these features.

01:15:26 I want to have like, I want floating point or not,

01:15:29 for example.

01:15:31 And then what you get is you get a description

01:15:34 of a CPU with those characteristics.

01:15:36 Now, if you want to make a chip,

01:15:38 you want to build like an iPhone chip

01:15:39 or something like that, right?

01:15:41 You have to take both the CPU,

01:15:42 but then you have to talk to memory.

01:15:44 You have to have timers, IOs, a GPU, other components.

01:15:49 And so you need to pull all those things together

01:15:51 into what’s called an ASIC,

01:15:53 an Application Specific Integrated Circuit.

01:15:55 So a custom chip.

01:15:56 And then you take that design

01:15:58 and then you have to transform it into something

01:16:00 that the fabs, like TSMC, for example,

01:16:03 know how to take to production.

01:16:06 Got it.

01:16:07 So, but yeah, okay.

01:16:08 And so that process, I will,

01:16:11 I can’t help but see it as, is a big compiler.

01:16:15 Yeah, yeah.

01:16:16 It’s a whole bunch of compilers written

01:16:18 without thinking about it through that lens.

01:16:21 Isn’t the universe a compiler?

01:16:23 Yeah, compilers do two things.

01:16:26 They represent things and transform them.

01:16:29 And so there’s a lot of things that end up being compilers.

01:16:31 But this is a space where we’re talking about design

01:16:34 and usability and the way you think about things,

01:16:37 the way things compose correctly, it matters a lot.

01:16:40 And so SciFi is investing a lot into that space.

01:16:43 And we think that there’s a lot of benefit

01:16:45 that can be made by allowing people to design chips faster,

01:16:48 get them to market quicker and scale out

01:16:52 because at the alleged end of Moore’s law,

01:16:56 you’ve got this problem of you’re not getting

01:16:59 free performance just by waiting another year

01:17:01 for a faster CPU.

01:17:03 And so you have to find performance in other ways.

01:17:06 And one of the ways to do that is with custom accelerators

01:17:09 and other things and hardware.

01:17:11 And so, well, we’ll talk a little more about ASICs,

01:17:17 but do you see that a lot of people,

01:17:21 a lot of companies will try to have

01:17:25 different sets of requirements

01:17:26 that this whole process to go for?

01:17:28 So like almost different car companies might use different

01:17:32 and like different PC manufacturers.

01:17:35 So is RISC 5 in this whole process,

01:17:40 is it potentially the future of all computing devices?

01:17:44 Yeah, I think that, so if you look at RISC 5

01:17:47 and step back from the Silicon side of things,

01:17:49 RISC 5 is an open standard.

01:17:51 And one of the things that has happened

01:17:53 over the course of decades,

01:17:55 if you look over the long arc of computing,

01:17:57 somehow became decades old.

01:17:59 Yeah.

01:18:00 Is that you have companies that come and go

01:18:02 and you have instruction sets that come and go.

01:18:04 Like one example of this out of many is Sun with Spark.

01:18:09 Yeah, it’s on one way.

01:18:11 Spark still lives on at Fujitsu,

01:18:12 but we have HP had this instruction set called PA RISC.

01:18:18 So PA RISC was this big server business

01:18:21 and had tons of customers.

01:18:22 They decided to move to this architecture

01:18:25 called Itanium from Intel.

01:18:27 Yeah.

01:18:27 This didn’t work out so well.

01:18:29 Yeah.

01:18:30 Right, and so you have this issue of

01:18:32 you’re making many billion dollar investments

01:18:35 on instruction sets that are owned by a company.

01:18:38 And even companies as big as Intel

01:18:39 don’t always execute as well as they could.

01:18:42 They even have their own issues.

01:18:44 HP, for example, decided that it wasn’t

01:18:46 in their best interest to continue investing in the space

01:18:48 because it was very expensive.

01:18:49 And so they make technology decisions

01:18:52 or they make their own business decisions.

01:18:54 And this means that as a customer, what do you do?

01:18:57 You’ve sunk all this time, all this engineering,

01:18:59 all this software work, all these,

01:19:01 you’ve built other products around them

01:19:02 and now you’re stuck, right?

01:19:05 What RISC 5 does is provide you more optionality

01:19:07 in the space because if you buy an implementation

01:19:10 of RISC 5 from SciFive, and you should,

01:19:13 they’re the best ones.

01:19:14 Yeah.

01:19:16 But if something bad happens to SciFive in 20 years, right?

01:19:19 Well, great, you can turn around

01:19:21 and buy a RISC 5 core from somebody else.

01:19:23 And there’s an ecosystem of people

01:19:25 that are all making different RISC 5 cores

01:19:26 with different trade offs, which means that

01:19:29 if you have more than one requirement,

01:19:30 if you have a family of products,

01:19:31 you can probably find something in the RISC 5 space

01:19:34 that fits your needs.

01:19:36 Whereas with, if you’re talking about XA6, for example,

01:19:39 it’s Intel’s only gonna bother

01:19:41 to make certain classes of devices, right?

01:19:45 I see, so maybe a weird question,

01:19:47 but like if SciFive is like infinitely successful

01:19:54 in the next 20, 30 years, what does the world look like?

01:19:58 So like how does the world of computing change?

01:20:01 So too much diversity in hardware instruction sets,

01:20:05 I think is bad.

01:20:06 Like we have a lot of people that are using

01:20:09 lots of different instruction sets,

01:20:10 particularly in the embedded,

01:20:12 the like very tiny microcontroller space,

01:20:14 the thing in your toaster that are just weird

01:20:19 and different for historical reasons.

01:20:21 And so the compilers and the tool chains

01:20:23 and the languages on top of them aren’t there.

01:20:27 And so the developers for that software

01:20:29 have to use really weird tools

01:20:31 because the ecosystem that supports is not big enough.

01:20:34 So I expect that will change, right?

01:20:35 People will have better tools and better languages,

01:20:38 better features everywhere

01:20:39 that then can serve as many different points in the space.

01:20:43 And I think RISC5 will progressively

01:20:46 eat more of the ecosystem because it can scale up,

01:20:49 it can scale down, sideways, left, right.

01:20:51 It’s very flexible and very well considered

01:20:53 and well designed instruction set.

01:20:56 I think when you look at SciFive tackling silicon

01:20:58 and how people build chips,

01:21:00 which is a very different space,

01:21:03 that’s where you say,

01:21:05 I think we’ll see a lot more custom chips.

01:21:07 And that means that you get much more battery life,

01:21:09 you get better tuned solutions for your IoT thingy.

01:21:16 You get people that move faster,

01:21:18 you get the ability to have faster time to market,

01:21:20 for example.

01:21:21 So how many custom…

01:21:22 So first of all, on the IoT side of things,

01:21:25 do you see the number of smart toasters

01:21:29 increasing exponentially?

01:21:30 So, and if you do,

01:21:35 like how much customization per toaster is there?

01:21:38 Do all toasters in the world run the same silicon,

01:21:42 like the same design,

01:21:44 or is it different companies have different design?

01:21:46 Like how much customization is possible here?

01:21:49 Well, a lot of it comes down to cost, right?

01:21:52 And so the way that chips work is you end up paying by the…

01:21:56 One of the factors is the size of the chip.

01:21:58 And so what ends up happening

01:22:01 just from an economic perspective is

01:22:03 there’s only so many chips that get made in a year

01:22:06 of a given design.

01:22:07 And so often what customers end up having to do

01:22:10 is they end up having to pick up a chip that exists

01:22:12 that was built for somebody else

01:22:14 so that they can then ship their product.

01:22:16 And the reason for that

01:22:17 is they don’t have the volume of the iPhone.

01:22:19 They can’t afford to build a custom chip.

01:22:21 However, what that means is they’re now buying

01:22:23 an off the shelf chip that isn’t really good,

01:22:26 isn’t a perfect fit for their needs.

01:22:28 And so they’re paying a lot of money for it

01:22:30 because they’re buying silicon that they’re not using.

01:22:33 Well, if you now reduce the cost of designing the chip,

01:22:36 now you get a lot more chips.

01:22:37 And the more you reduce it,

01:22:39 the easier it is to design chips.

01:22:42 The more the world keeps evolving

01:22:44 and we get more AI accelerators,

01:22:45 we get more other things,

01:22:46 we get more standards to talk to,

01:22:48 we get 6G, right?

01:22:50 You get changes in the world

01:22:53 that you wanna be able to talk to these different things.

01:22:54 There’s more diversity in the cross product of features

01:22:57 that people want.

01:22:58 And that drives differentiated chips

01:23:02 in another direction.

01:23:03 And so nobody really knows what the future looks like,

01:23:05 but I think that there’s a lot of silicon in the future.

01:23:09 Speaking of the future,

01:23:11 you said Moore’s law allegedly is dead.

01:23:13 So do you agree with Dave Patterson and many folks

01:23:20 that Moore’s law is dead?

01:23:22 Or do you agree with Jim Keller,

01:23:23 who’s standing at the helm of the pirate ship

01:23:28 saying it’s still alive?

01:23:31 Yeah.

01:23:32 Well, so I agree with what they’re saying

01:23:35 and different people are interpreting

01:23:37 the end of Moore’s law in different ways.

01:23:39 Yeah.

01:23:40 So Jim would say,

01:23:41 there’s another thousand X left in physics

01:23:44 and we can continue to squeeze the stone

01:23:46 and make it faster and smaller and smaller geometries

01:23:50 and all that kind of stuff.

01:23:52 He’s right.

01:23:53 So Jim is absolutely right

01:23:55 that there’s a ton of progress left

01:23:57 and we’re not at the limit of physics yet.

01:24:01 That’s not really what Moore’s law is though.

01:24:04 If you look at what Moore’s law is,

01:24:06 is that it’s a very simple evaluation of,

01:24:10 okay, well you look at the cost per,

01:24:13 I think it was cost per area

01:24:14 and the most economic point in that space.

01:24:17 And if you go look at the now quite old paper

01:24:20 that describes this,

01:24:21 Moore’s law has a specific economic aspect to it

01:24:25 and I think this is something

01:24:26 that Dave and others often point out.

01:24:28 And so on a technicality, that’s right.

01:24:31 I look at it from,

01:24:33 so I can acknowledge both of those viewpoints.

01:24:34 They’re both right.

01:24:35 They’re both right.

01:24:36 I’ll give you a third wrong viewpoint

01:24:39 that may be right in its own way,

01:24:40 which is single threaded performance

01:24:44 doesn’t improve like it used to.

01:24:46 And it used to be back when you got a,

01:24:48 you know, a Pentium 66 or something

01:24:50 and the year before you had a Pentium 33

01:24:53 and now it’s twice as fast, right?

01:24:56 Well, it was twice as fast at doing exactly the same thing.

01:25:00 Okay, like literally the same program ran twice as fast.

01:25:03 You just wrote a check and waited a year, year and a half.

01:25:07 Well, so that’s what a lot of people think about Moore’s law

01:25:10 and I think that is dead.

01:25:11 And so what we’re seeing instead is we’re pushing,

01:25:15 we’re pushing people to write software in different ways.

01:25:17 And so we’re pushing people to write CUDA

01:25:19 so they can get GPU compute

01:25:20 and the thousands of cores on GPU.

01:25:23 We’re talking about C programmers having to use P threads

01:25:26 because they now have, you know,

01:25:27 a hundred threads or 50 cores in a machine

01:25:30 or something like that.

01:25:31 You’re now talking about machine learning accelerators

01:25:33 that are now domain specific.

01:25:35 And when you look at these kinds of use cases,

01:25:38 you can still get performance

01:25:40 and Jim will come up with cool things

01:25:42 that utilize the silicon in new ways for sure,

01:25:45 but you’re also gonna change the programming model.

01:25:48 Right.

01:25:49 And now when you start talking about changing

01:25:50 the programming model,

01:25:50 that’s when you come back to languages

01:25:53 and things like this too,

01:25:54 because often what you see is like you take

01:25:58 the C programming language, right?

01:25:59 The C programming language is designed for CPUs.

01:26:03 And so if you want to talk to a GPU,

01:26:04 now you’re talking to its cousin CUDA, okay?

01:26:08 CUDA is a different thing with a different set of tools,

01:26:11 a different world, a different way of thinking.

01:26:14 And we don’t have one world that scales.

01:26:16 And I think that we can get there.

01:26:18 We can have one world that scales in a much better way.

01:26:21 And a small tangent then,

01:26:22 I think most programming languages are designed for CPUs,

01:26:25 for single core, even just in their spirit,

01:26:28 even if they allow for parallelization.

01:26:30 So what does it look like for a programming language

01:26:34 to have parallelization or massive parallelization

01:26:38 as it’s like first principle?

01:26:41 So the canonical example of this

01:26:43 is the hardware design world.

01:26:46 So Verilog, VHDL, these kinds of languages,

01:26:50 they’re what’s called a high level synthesis language.

01:26:53 This is the thing people design chips in.

01:26:56 And when you’re designing a chip,

01:26:58 it’s kind of like a brain where you have infinite parallelism.

01:27:02 Like you’re like laying down transistors.

01:27:05 Transistors are always running, okay?

01:27:08 And so you’re not saying run this transistor,

01:27:10 then this transistor, then this transistor.

01:27:12 It’s like your brain,

01:27:13 like your neurons are always just doing something.

01:27:15 They’re not clocked, right?

01:27:16 They’re just doing their thing.

01:27:20 And so when you design a chip or when you design a CPU,

01:27:23 when you design a GPU, when you design,

01:27:25 when you’re laying down the transistors,

01:27:27 similarly, you’re talking about,

01:27:28 well, okay, well, how do these things communicate?

01:27:31 And so these languages exist.

01:27:32 Verilog is a kind of mixed example of that.

01:27:36 None of these languages are really great.

01:27:37 You have a very low level, yeah.

01:27:39 Yeah, they’re very low level

01:27:40 and abstraction is necessary here.

01:27:42 And there’s different approaches with that.

01:27:44 And it’s itself a very complicated world,

01:27:47 but it’s implicitly parallel.

01:27:50 And so having that as the domain that you program towards

01:27:56 makes it so that by default, you get parallel systems.

01:27:59 If you look at CUDA,

01:28:00 CUDA is a point halfway in the space where in CUDA,

01:28:03 when you write a CUDA kernel for your GPU,

01:28:05 it feels like you’re writing a scalar program.

01:28:08 So you’re like, you have ifs, you have for loops,

01:28:10 stuff like this, you’re just writing normal code.

01:28:12 But what happens outside of that in your driver

01:28:14 is that it actually is running you

01:28:16 on like a thousand things at once, right?

01:28:18 And so it’s parallel,

01:28:20 but it has pulled it out of the programming model.

01:28:23 And so now you as a programmer are working in a simpler world

01:28:28 and it’s solved that for you, right?

01:28:31 How do you take the language like Swift?

01:28:36 If we think about GPUs, but also ASICs,

01:28:39 maybe if we can dance back and forth

01:28:40 between hardware and software.

01:28:42 How do you design for these features

01:28:46 to be able to program and get a first class citizen

01:28:50 to be able to do like Swift for TensorFlow

01:28:53 to be able to do machine learning on current hardware,

01:28:56 but also future hardware like TPUs

01:28:59 and all kinds of ASICs

01:29:00 that I’m sure will be popping up more and more.

01:29:02 Yeah, well, so a lot of this comes down

01:29:04 to this whole idea of having the nuts and bolts

01:29:06 underneath the covers that work really well.

01:29:08 So you need, if you’re talking to TPUs,

01:29:10 you need MLIR or XLA or one of these compilers

01:29:13 that talks to TPUs to build on top of, okay?

01:29:17 And if you’re talking to circuits,

01:29:19 you need to figure out how to lay down the transistors

01:29:21 and how to organize it and how to set up clocking

01:29:23 and like all the domain problems

01:29:24 that you get with circuits.

01:29:27 Then you have to decide how to explain it to a human.

01:29:29 What is ZY, right?

01:29:31 And if you do it right, that’s a library problem,

01:29:34 not a language problem.

01:29:36 And that works if you have a library or a language

01:29:39 which allows your library to write things

01:29:42 that feel native in the language by implementing libraries,

01:29:45 because then you can innovate in programming models

01:29:49 without having to change your syntax again.

01:29:51 Like you have to invent new code formatting tools

01:29:54 and like all the other things that languages come with.

01:29:57 And this gets really interesting.

01:29:59 And so if you look at the space,

01:30:02 the interesting thing once you separate out syntax

01:30:05 becomes what is that programming model?

01:30:07 And so do you want the CUDA style?

01:30:10 I write one program and it runs many places.

01:30:14 Do you want the implicitly parallel model?

01:30:16 How do you reason about that?

01:30:17 How do you give developers, chip architects,

01:30:20 the ability to express their intent?

01:30:24 And that comes into this whole design question

01:30:26 of how do you detect bugs quickly?

01:30:29 So you don’t have to tape out a chip

01:30:30 to find out it’s wrong, ideally, right?

01:30:32 How do you, and this is a spectrum,

01:30:35 how do you make it so that people feel productive?

01:30:38 So their turnaround time is very quick.

01:30:40 All these things are really hard problems.

01:30:42 And in this world, I think that not a lot of effort

01:30:46 has been put into that design problem

01:30:48 and thinking about the layering in other pieces.

01:30:51 Well, you’ve, on the topic of concurrency,

01:30:53 you’ve written the Swift concurrency manifest.

01:30:55 I think it’s kind of interesting.

01:30:57 Anything that has the word manifest on it

01:31:00 is very interesting.

01:31:02 Can you summarize the key ideas of each of the five parts

01:31:06 you’ve written about?

01:31:07 So what is a manifesto?

01:31:08 Yes.

01:31:09 How about, we start there.

01:31:11 So in the Swift community, we have this problem,

01:31:15 which is on the one hand,

01:31:16 you wanna have relatively small proposals

01:31:19 that you can kind of fit in your head,

01:31:21 you can understand the details at a very fine grain level

01:31:24 that move the world forward.

01:31:26 But then you also have these big arcs, okay?

01:31:28 And often when you’re working on something

01:31:30 that is a big arc, but you’re tackling it in small pieces,

01:31:34 you have this question of,

01:31:35 how do I know I’m not doing a random walk?

01:31:37 Where are we going?

01:31:38 How does this add up?

01:31:39 Furthermore, when you start the first small step,

01:31:43 what terminology do you use?

01:31:45 How do we think about it?

01:31:46 What is better and worse in the space?

01:31:47 What are the principles?

01:31:48 What are we trying to achieve?

01:31:50 And so what a manifesto in the Swift community does

01:31:52 is it starts to say,

01:31:53 hey, well, let’s step back from the details of everything.

01:31:56 Let’s paint a broad picture to talk about

01:31:58 what we’re trying to achieve.

01:32:01 Let’s give an example design point.

01:32:02 Let’s try to paint the big picture

01:32:05 so that then we can zero in on the individual steps

01:32:07 and make sure that we’re making good progress.

01:32:09 And so the Swift concurrency manifesto

01:32:11 is something I wrote three years ago.

01:32:13 It’s been a while, maybe more.

01:32:16 Trying to do that for Swift and concurrency.

01:32:19 It starts with some fairly simple things

01:32:22 like making the observation that

01:32:25 when you have multiple different computers

01:32:26 and multiple different threads that are communicating,

01:32:28 it’s best for them to be asynchronous.

01:32:32 And so you need things to be able to run separately

01:32:34 and then communicate with each other.

01:32:35 And this means asynchrony.

01:32:37 And this means that you need a way

01:32:38 to modeling asynchronous communication.

01:32:41 Many languages have features like this.

01:32:43 Async await is a popular one.

01:32:45 And so that’s what I think is very likely in Swift.

01:32:49 But as you start building this tower of abstractions,

01:32:51 it’s not just about how do you write this,

01:32:53 you then reach into the how do you get memory safety

01:32:57 because you want correctness,

01:32:58 you want debuggability and sanity for developers.

01:33:01 And how do you get that memory safety into the language?

01:33:06 So if you take a language like Go or C

01:33:09 or any of these languages,

01:33:10 you get what’s called a race condition

01:33:11 when two different threads or Go routines or whatever

01:33:14 touch the same point in memory, right?

01:33:17 This is a huge like maddening problem to debug

01:33:21 because it’s not reproducible generally.

01:33:24 And so there’s tools,

01:33:25 there’s a whole ecosystem of solutions

01:33:27 that built up around this,

01:33:28 but it’s a huge problem

01:33:29 when you’re writing concurrent code.

01:33:31 And so with Swift,

01:33:33 this whole value semantics thing is really powerful there

01:33:35 because it turns out that math and copies actually work

01:33:39 even in concurrent worlds.

01:33:40 And so you get a lot of safety just out of the box,

01:33:43 but there are also some hard problems.

01:33:44 And it talks about some of that.

01:33:47 When you start building up to the next level up

01:33:48 and you start talking beyond memory safety,

01:33:50 you have to talk about what is the programmer model?

01:33:53 How does a human think about this?

01:33:54 So a developer that’s trying to build a program

01:33:56 think about this,

01:33:57 and it proposes a really old model with a new spin

01:34:01 called Actors.

01:34:02 Actors are about saying,

01:34:03 we have islands of single threadedness logically.

01:34:08 So you write something that feels like

01:34:09 it’s one program running in a unit,

01:34:13 and then it communicates asynchronously with other things.

01:34:16 And so making that expressive and natural feel good

01:34:19 be the first thing you reach for and being safe by default

01:34:22 is a big part of the design of that proposal.

01:34:25 When you start going beyond that,

01:34:26 now you start to say, cool,

01:34:27 well, these things that communicate asynchronously,

01:34:30 they don’t have to share memory.

01:34:32 Well, if they don’t have to share memory

01:34:33 and they’re sending messages to each other,

01:34:35 why do they have to be in the same process?

01:34:38 These things should be able to be in different processes

01:34:40 on your machine.

01:34:41 And why just processes?

01:34:43 Well, why not different machines?

01:34:44 And so now you have a very nice gradual transition

01:34:47 towards distributed programming.

01:34:49 And of course, when you start talking about the big future,

01:34:53 the manifesto doesn’t go into it,

01:34:55 but accelerators are things you talk to asynchronously

01:35:00 by sending messages to them.

01:35:01 And how do you program those?

01:35:03 Well, that gets very interesting.

01:35:05 That’s not in the proposal.

01:35:07 And how much do you wanna make that explicit

01:35:12 like the control of that whole process

01:35:14 explicit to the program?

01:35:15 Yeah, good question.

01:35:16 So when you’re designing any of these kinds of features

01:35:20 or language features or even libraries,

01:35:22 you have this really hard trade off you have to make,

01:35:25 which is how much is it magic

01:35:27 or how much is it in the human’s control?

01:35:29 How much can they predict and control it?

01:35:32 What do you do when the default case is the wrong case?

01:35:37 And so when you’re designing a system,

01:35:39 and so when you’re designing a system, I won’t name names,

01:35:45 but there are systems where it’s really easy to get started

01:35:50 and then you jump.

01:35:52 So let’s pick like logo.

01:35:54 Okay, so something like this.

01:35:55 So it’s really easy to get started.

01:35:57 It’s really designed for teaching kids,

01:35:59 but as you get into it, you hit a ceiling

01:36:02 and then you can’t go any higher.

01:36:03 And then what do you do?

01:36:04 Well, you have to go switch to a different world

01:36:05 and rewrite all your code.

01:36:07 And this logo is a silly example here.

01:36:09 This exists in many other languages.

01:36:11 With Python, you would say like concurrency, right?

01:36:15 So Python has the global interpreter block.

01:36:17 So threading is challenging in Python.

01:36:19 And so if you start writing a large scale application

01:36:22 in Python, and then suddenly you need concurrency,

01:36:25 you’re kind of stuck with a series of bad trade offs, right?

01:36:30 There’s other ways to go where you say like,

01:36:32 foist all the complexity on the user all at once, right?

01:36:37 And that’s also bad in a different way.

01:36:38 And so what I prefer is building a simple model

01:36:43 that you can explain that then has an escape hatch.

01:36:46 So you get in, you have guardrails,

01:36:50 memory safety works like this in Swift,

01:36:52 where you can start with, like by default,

01:36:54 if you use all the standard things, it’s memory safe,

01:36:57 you’re not gonna shoot your foot off.

01:36:58 But if you wanna get a C level pointer to something,

01:37:02 you can explicitly do that.

01:37:04 But by default, there’s guardrails.

01:37:07 There’s guardrails.

01:37:08 Okay, so but like, whose job is it to figure out

01:37:14 which part of the code is parallelizable?

01:37:17 So in the case of the proposal, it is the human’s job.

01:37:21 So they decide how to architect their application.

01:37:24 And then the runtime in the compiler is very predictable.

01:37:29 And so this is in contrast to like,

01:37:31 there’s a long body of work, including on Fortran

01:37:34 for auto parallelizing compilers.

01:37:36 And this is an example of a bad thing in my,

01:37:39 so as a compiler person, I can drag on compiler people.

01:37:42 Often compiler people will say,

01:37:45 cool, since I can’t change the code,

01:37:46 I’m gonna write my compiler that then takes

01:37:48 this unmodified code and makes go way faster on this machine.

01:37:52 Okay, application, and so it does pattern matching.

01:37:55 It does like really deep analysis.

01:37:57 Compiler people are really smart.

01:37:58 And so they like wanna like do something

01:38:00 really clever and tricky.

01:38:01 And you get like 10X speed up by taking

01:38:04 like an array of structures and turn it

01:38:06 into a structure of arrays or something,

01:38:08 because it’s so much better for memory.

01:38:09 Like there’s bodies, like tons of tricks.

01:38:12 They love optimization.

01:38:13 Yeah, you love optimization.

01:38:14 Everyone loves optimization.

01:38:15 Everyone loves it.

01:38:16 Well, and it’s this promise of build with my compiler

01:38:19 and your thing goes fast, right?

01:38:20 But here’s the problem, Lex, you write a program,

01:38:24 you run it with my compiler, it goes fast.

01:38:26 You’re very happy.

01:38:27 Wow, it’s so much faster than the other compiler.

01:38:29 Then you go and you add a feature to your program

01:38:31 or you refactor some code.

01:38:32 And suddenly you got a 10X loss in performance.

01:38:35 Well, why?

01:38:36 What just happened there?

01:38:37 What just happened there is the heuristic,

01:38:39 the pattern matching, the compiler,

01:38:41 whatever analysis it was doing just got defeated

01:38:43 because you didn’t inline a function or something, right?

01:38:48 As a user, you don’t know, you don’t wanna know.

01:38:50 That was the whole point.

01:38:51 You don’t wanna know how the compiler works.

01:38:52 You don’t wanna know how the memory hierarchy works.

01:38:54 You don’t wanna know how it got parallelized

01:38:56 across all these things.

01:38:57 You wanted that abstracted away from you,

01:38:59 but then the magic is lost as soon as you did something

01:39:02 and you fall off a performance cliff.

01:39:05 And now you’re in this funny position

01:39:06 where what do I do?

01:39:08 I don’t change my code.

01:39:09 I don’t fix that bug.

01:39:10 It costs 10X performance.

01:39:12 Now what do I do?

01:39:13 Well, this is the problem with unpredictable performance.

01:39:16 If you care about performance,

01:39:17 predictability is a very important thing.

01:39:19 And so what the proposal does is it provides

01:39:23 architectural patterns for being able to lay out your code,

01:39:26 gives you full control over that,

01:39:28 makes it really simple so you can explain it.

01:39:30 And then if you wanna scale out in different ways,

01:39:34 you have full control over that.

01:39:36 So in your sense, the intuition is for a compiler,

01:39:39 it’s too hard to do automated parallelization.

01:39:43 Cause the compilers do stuff automatically

01:39:47 that’s incredibly impressive for other things,

01:39:50 but for parallelization, we’re not close to there.

01:39:54 Well, it depends on the programming model.

01:39:56 So there’s many different kinds of compilers.

01:39:58 And so if you talk about like a C compiler

01:40:00 or Swift compiler or something like that,

01:40:01 where you’re writing imperative code,

01:40:04 parallelizing that and reasoning about all the pointers

01:40:07 and stuff like that is a very difficult problem.

01:40:10 Now, if you switch domains,

01:40:12 so there’s this cool thing called machine learning, right?

01:40:15 So machine learning nerds among other endearing things

01:40:18 like solving cat detectors and other things like that

01:40:23 have done this amazing breakthrough

01:40:25 of producing a programming model,

01:40:27 operations that you compose together

01:40:30 that has raised levels of abstraction high enough

01:40:33 that suddenly you can have auto parallelizing compilers.

01:40:36 You can write a model using a TensorFlow

01:40:39 and have it run on 1024 nodes of a TPU.

01:40:43 Yeah, that’s true.

01:40:44 I didn’t even think about like,

01:40:46 cause there’s so much flexibility

01:40:48 in the design of architectures that ultimately boil down

01:40:50 to a graph that’s parallelized for you.

01:40:54 And if you think about it, that’s pretty cool.

01:40:56 That’s pretty cool, yeah.

01:40:57 And you think about batching, for example,

01:40:59 as a way of being able to exploit more parallelism.

01:41:02 Like that’s a very simple thing that now is very powerful.

01:41:05 That didn’t come out of the programming language nerds,

01:41:08 those people, like that came out of people

01:41:10 that are just looking to solve a problem

01:41:11 and use a few GPUs and organically developed

01:41:14 by the community of people focusing on machine learning.

01:41:16 And it’s an incredibly powerful abstraction layer

01:41:19 that enables the compiler people to go and exploit that.

01:41:22 And now you can drive supercomputers from Python.

01:41:26 Well, that’s pretty cool.

01:41:27 That’s amazing.

01:41:28 So just to pause on that,

01:41:30 cause I’m not sufficiently low level,

01:41:32 I forget to admire the beauty and power of that,

01:41:35 but maybe just to linger on it,

01:41:38 like what does it take to run a neural network fast?

01:41:42 Like how hard is that compilation?

01:41:44 It’s really hard.

01:41:45 So we just skipped,

01:41:46 you said like, it’s amazing that that’s a thing,

01:41:49 but yeah, how hard is that of a thing?

01:41:51 It’s hard and I would say that not all of the systems

01:41:55 are really great, including the ones I helped build.

01:41:58 So there’s a lot of work left to be done there.

01:42:00 Is it the compiler nerds working on that

01:42:02 or is it a whole new group of people?

01:42:04 Well, it’s a full stack problem,

01:42:05 including compiler people, including APIs,

01:42:09 so like Keras and the module API and PyTorch and Jax.

01:42:14 And there’s a bunch of people pushing

01:42:15 on all the different parts of these things,

01:42:17 because when you look at it as it’s both,

01:42:19 how do I express the computation?

01:42:21 Do I stack up layers?

01:42:22 Well, cool, like setting up a linear sequence of layers

01:42:25 is great for the simple case,

01:42:26 but how do I do the hard case?

01:42:28 How do I do reinforcement learning?

01:42:29 Well, now I need to integrate my application logic in this.

01:42:32 Then it’s the next level down of,

01:42:34 how do you represent that for the runtime?

01:42:36 How do you get hardware abstraction?

01:42:39 And then you get to the next level down of saying like,

01:42:40 forget about abstraction,

01:42:41 how do I get the peak performance out of my TPU

01:42:44 or my iPhone accelerator or whatever, right?

01:42:47 And all these different things.

01:42:48 And so this is a layered problem

01:42:50 with a lot of really interesting design and work

01:42:53 going on in the space

01:42:54 and a lot of really smart people working on it.

01:42:56 Machine learning is a very well funded area

01:42:59 of investment right now.

01:43:00 And so there’s a lot of progress being made.

01:43:02 So how much innovation is there on the lower level,

01:43:05 so closer to the ASIC,

01:43:08 so redesigning the hardware

01:43:09 or redesigning concurrently compilers with that hardware?

01:43:13 Is that like, if you were to predict the biggest,

01:43:16 the equivalent of Moore’s law improvements

01:43:20 in the inference and the training of neural networks

01:43:24 and just all of that,

01:43:25 where is that gonna come from, you think?

01:43:26 Sure, you get scalability of different things.

01:43:28 And so you get Jim Keller shrinking process technology,

01:43:33 you get three nanometer instead of five or seven or 10

01:43:36 or 28 or whatever.

01:43:38 And so that marches forward and that provides improvements.

01:43:41 You get architectural level performance.

01:43:44 And so the TPU with a matrix multiply unit

01:43:47 and a systolic array is much more efficient

01:43:49 than having a scalar core doing multiplies and adds

01:43:53 and things like that.

01:43:54 You then get system level improvements.

01:43:58 So how you talk to memory,

01:43:59 how you talk across a cluster of machines,

01:44:02 how you scale out,

01:44:03 how you have fast interconnects between machines.

01:44:06 You then get system level programming models.

01:44:08 So now that you have all this hardware, how to utilize it.

01:44:11 You then have algorithmic breakthroughs where you say,

01:44:13 hey, wow, cool.

01:44:14 Instead of training in a resonant 50 in a week,

01:44:18 I’m now training it in 25 seconds.

01:44:21 And it’s a combination of new optimizers

01:44:26 and new just training regimens

01:44:29 and different approaches to train.

01:44:32 And all of these things come together

01:44:33 to push the world forward.

01:44:36 That was a beautiful exposition.

01:44:39 But if you were to force to bet all your money

01:44:42 on one of these.

01:44:45 Why do we have to?

01:44:48 Unfortunately, we have people working on all this.

01:44:50 It’s an exciting time, right?

01:44:52 So, I mean, OpenAI did this little paper

01:44:56 showing the algorithmic improvement you can get.

01:44:58 It’s been improving exponentially.

01:45:00 I haven’t quite seen the same kind of analysis

01:45:04 on other layers of the stack.

01:45:06 I’m sure it’s also improving significantly.

01:45:09 I just, it’s a nice intuition builder.

01:45:12 I mean, there’s a reason why Moore’s Law,

01:45:16 that’s the beauty of Moore’s Law is

01:45:18 somebody writes a paper that makes a ridiculous prediction.

01:45:22 And it becomes reality in a sense.

01:45:27 There’s something about these narratives

01:45:28 when you, when Chris Ladner on a silly little podcast

01:45:33 makes, bets all his money on a particular thing,

01:45:37 somehow it can have a ripple effect

01:45:39 of actually becoming real.

01:45:40 That’s an interesting aspect of it.

01:45:43 Cause like it might’ve been,

01:45:45 we focus with Moore’s Law,

01:45:47 most of the computing industry

01:45:49 really, really focused on the hardware.

01:45:53 I mean, software innovation,

01:45:55 I don’t know how much software innovation

01:45:56 there was in terms of efficient.

01:45:57 What Intel Giveth Bill takes away, right?

01:46:00 Yeah, I mean, compilers improved significantly also.

01:46:04 Well, not really.

01:46:04 So actually, I mean, so I’m joking

01:46:06 about how software has gotten slower

01:46:08 pretty much as fast as hardware got better,

01:46:11 at least through the nineties.

01:46:13 There’s another joke, another law in compilers,

01:46:15 which is called, I think it’s called Probstine’s Law,

01:46:18 which is compilers double the performance

01:46:21 of any given code every 18 years.

01:46:26 So they move slowly.

01:46:27 Yeah, also.

01:46:28 Well, yeah, it’s exponential also.

01:46:30 Yeah, you’re making progress,

01:46:32 but there again, it’s not about,

01:46:35 the power of compilers is not just about

01:46:37 how do you make the same thing go faster?

01:46:39 It’s how do you unlock the new hardware?

01:46:41 A new chip came out, how do you utilize it?

01:46:43 You say, oh, the programming model,

01:46:45 how do we make people more productive?

01:46:46 How do we have better error messages?

01:46:51 Even such mundane things like how do I generate

01:46:55 a very specific error message about your code

01:46:58 actually makes people happy

01:46:59 because then they know how to fix it, right?

01:47:01 And it comes back to how do you help people

01:47:03 get their job done.

01:47:04 Yeah, and yeah, and then in this world

01:47:06 of exponentially increasing smart toasters,

01:47:10 how do you expand computing to all these kinds of devices?

01:47:16 Do you see this world where just everything

01:47:18 is a computing surface?

01:47:20 You see that possibility?

01:47:22 Just everything is a computer?

01:47:23 Yeah, I don’t see any reason

01:47:25 that that couldn’t be achieved.

01:47:26 It turns out that sand goes into glass

01:47:30 and glass is pretty useful too.

01:47:32 And why not?

01:47:35 Why not?

01:47:35 So very important question then,

01:47:39 if we’re living in a simulation

01:47:44 and the simulation is running a computer,

01:47:47 like what’s the architecture of that computer, do you think?

01:47:51 So you’re saying is it a quantum system?

01:47:54 Yeah, like this whole quantum discussion, is it needed?

01:47:57 Or can we run it with a RISC 5 architecture,

01:48:03 a bunch of CPUs?

01:48:05 I think it comes down to the right tool for the job.

01:48:07 Yeah, and so.

01:48:08 And what’s the compiler?

01:48:10 Yeah, exactly, that’s my question.

01:48:12 Did I get that job?

01:48:13 Feed the universe compiler.

01:48:16 And so there, as far as we know,

01:48:19 quantum systems are the bottom of the pile of turtles

01:48:23 so far.

01:48:24 Yeah.

01:48:25 And so we don’t know efficient ways

01:48:28 to implement quantum systems without using quantum computers.

01:48:32 Yeah, and that’s totally outside

01:48:33 of everything we’ve talked about.

01:48:35 But who runs that quantum computer?

01:48:37 Yeah.

01:48:37 Right, so if we really are living in a simulation,

01:48:41 then is it bigger quantum computers?

01:48:44 Is it different ones?

01:48:45 Like how does that work out?

01:48:46 How does that scale?

01:48:47 Well, it’s the same size.

01:48:49 It’s the same size.

01:48:50 But then the thought of the simulation

01:48:52 is that you don’t have to run the whole thing,

01:48:54 that we humans are cognitively very limited.

01:48:56 We do checkpoints.

01:48:57 We do checkpoints, yeah.

01:48:59 And if we, the point at which we human,

01:49:03 so you basically do minimal amount of,

01:49:06 what is it, Swift does on right, copy on right.

01:49:12 So you only adjust the simulation.

01:49:15 Parallel universe theories, right?

01:49:17 And so every time a decision’s made,

01:49:20 somebody opens the short end of your box,

01:49:22 then there’s a fork.

01:49:23 And then this could happen.

01:49:24 And then, thank you for considering the possibility.

01:49:29 But yeah, so it may not require the entirety

01:49:32 of the universe to simulate it.

01:49:34 But it’s interesting to think about

01:49:38 as we create this higher and higher fidelity systems.

01:49:42 But I do wanna ask on the quantum computer side,

01:49:46 because everything we’ve talked about,

01:49:47 whether you work with SciFive, with compilers,

01:49:51 none of that includes quantum computers, right?

01:49:54 That’s true.

01:49:55 So have you ever thought about this whole

01:50:03 serious engineering work of quantum computers

01:50:06 looks like of compilers, of architectures,

01:50:09 all of that kind of stuff?

01:50:10 So I’ve looked at it a little bit.

01:50:11 I know almost nothing about it,

01:50:14 which means that at some point,

01:50:15 I will have to find an excuse to get involved,

01:50:17 because that’s how it works.

01:50:18 But do you think that’s a thing to be,

01:50:21 like with your little tingly senses of the timing

01:50:24 of when to be involved, is it not yet?

01:50:26 Well, so the thing I do really well

01:50:28 is I jump into messy systems

01:50:31 and figure out how to make them,

01:50:33 figure out what the truth in the situation is,

01:50:35 try to figure out what the unifying theory is,

01:50:39 how to like factor the complexity,

01:50:40 how to find a beautiful answer to a problem

01:50:42 that has been well studied

01:50:44 and lots of people have bashed their heads against it.

01:50:47 I don’t know that quantum computers are mature enough

01:50:49 and accessible enough to be figured out yet, right?

01:50:53 And I think the open question with quantum computers is,

01:50:58 is there a useful problem

01:51:00 that gets solved with a quantum computer

01:51:02 that makes it worth the economic cost

01:51:05 of like having one of these things

01:51:06 and having legions of people that set it up?

01:51:11 You go back to the fifties, right?

01:51:12 And there’s the projections

01:51:13 of the world will only need seven computers, right?

01:51:18 Well, and part of that was that people hadn’t figured out

01:51:20 what they’re useful for.

01:51:21 What are the algorithms we wanna run?

01:51:23 What are the problems that get solved?

01:51:24 And this comes back to how do we make the world better,

01:51:27 either economically or making somebody’s life better

01:51:29 or like solving a problem that wasn’t solved before,

01:51:31 things like this.

01:51:33 And I think that just we’re a little bit too early

01:51:36 in that development cycle

01:51:36 because it’s still like literally a science project,

01:51:39 not a negative connotation, right?

01:51:41 It’s literally a science project

01:51:42 and the progress there is amazing.

01:51:45 And so I don’t know if it’s 10 years away,

01:51:48 if it’s two years away,

01:51:50 exactly where that breakthrough happens,

01:51:51 but you look at machine learning,

01:51:56 we went through a few winners

01:51:58 before the AlexNet transition

01:52:00 and then suddenly it had its breakout moment.

01:52:02 And that was the catalyst

01:52:04 that then drove the talent flocking into it.

01:52:07 That’s what drove the economic applications of it.

01:52:10 That’s what drove the technology to go faster

01:52:13 because you now have more minds thrown at the problem.

01:52:15 This is what caused like a serious knee in deep learning

01:52:20 and the algorithms that we’re using.

01:52:21 And so I think that’s what quantum needs to go through.

01:52:25 And so right now it’s in that formidable finding itself,

01:52:28 getting the like literally the physics figured out.

01:52:32 And then it has to figure out the application

01:52:36 that makes this useful.

01:52:37 Yeah, but I’m not skeptical that I think that will happen.

01:52:40 I think it’s just 10 years away, something like that.

01:52:43 I forgot to ask,

01:52:44 what programming language do you think

01:52:46 the simulation is written in?

01:52:48 Ooh, probably Lisp.

01:52:50 So not Swift.

01:52:52 Like if you’re a Tibet, I’ll just leave it at that.

01:52:58 So, I mean, we’ve mentioned that you worked

01:53:00 with all these companies,

01:53:01 we’ve talked about all these projects.

01:53:03 It’s kind of like if we just step back and zoom out

01:53:07 about the way you did that work.

01:53:10 And we look at COVID times,

01:53:12 this pandemic we’re living through that may,

01:53:14 if I look at the way Silicon Valley folks

01:53:17 are talking about it, the way MIT is talking about it,

01:53:19 this might last for a long time.

01:53:23 Not just the virus, but the remote nature.

01:53:28 The economic impact.

01:53:29 I mean, all of it, yeah.

01:53:30 Yeah, it’s gonna be a mess.

01:53:32 Do you think, what’s your prediction?

01:53:34 I mean, from sci fi to Google,

01:53:36 to just all the places you worked in,

01:53:42 just Silicon Valley, you’re in the middle of it.

01:53:44 What do you think is,

01:53:45 how is this whole place gonna change?

01:53:46 Yeah, so, I mean, I really can only speak

01:53:49 to the tech perspective.

01:53:50 I am in that bubble.

01:53:54 I think it’s gonna be really interesting

01:53:55 because the Zoom culture of being remote

01:53:58 and on video chat all the time

01:54:00 has really interesting effects on people.

01:54:01 So on the one hand, it’s a great normalizer.

01:54:05 It’s a normalizer that I think will help communities

01:54:09 of people that have traditionally been underrepresented

01:54:12 because now you’re taking, in some cases, a face off

01:54:16 because you don’t have to have a camera going, right?

01:54:18 And so you can have conversations

01:54:20 without physical appearance being part of the dynamic,

01:54:22 which is pretty powerful.

01:54:24 You’re taking remote employees

01:54:25 that have already been remote,

01:54:27 and you’re saying you’re now on the same level

01:54:29 and footing as everybody else.

01:54:31 Nobody gets whiteboards.

01:54:33 You’re not gonna be the one person

01:54:34 that doesn’t get to be participating

01:54:36 in the whiteboard conversation,

01:54:37 and that’s pretty powerful.

01:54:39 You’ve got, you’re forcing people to think asynchronously

01:54:44 in some cases because it’s harder to just get people

01:54:47 physically together, and the bumping into each other

01:54:49 forces people to find new ways to solve those problems.

01:54:52 And I think that that leads to more inclusive behavior,

01:54:55 which is good.

01:54:56 On the other hand, it’s also, it just sucks, right?

01:55:00 And so the actual communication just sucks

01:55:05 being not with people on a daily basis

01:55:09 and collaborating with them.

01:55:11 Yeah, all of that, right?

01:55:13 I mean, everything, this whole situation is terrible.

01:55:15 What I meant primarily was the,

01:55:19 I think that most humans

01:55:20 like working physically with humans.

01:55:22 I think this is something that not everybody,

01:55:24 but many people are programmed to do.

01:55:27 And I think that we get something out of that

01:55:29 that is very hard to express, at least for me.

01:55:31 And so maybe this isn’t true of everybody.

01:55:33 But, and so the question to me is,

01:55:36 when you get through that time of adaptation,

01:55:40 you get out of March and April,

01:55:41 and you get into December,

01:55:43 you get into next March, if it’s not changed, right?

01:55:46 It’s already terrifying.

01:55:47 Well, you think about that,

01:55:49 and you think about what is the nature of work?

01:55:51 And how do we adapt?

01:55:52 And humans are very adaptable species, right?

01:55:55 We can learn things when we’re forced to,

01:55:58 and there’s a catalyst to make that happen.

01:56:00 And so what is it that comes out of this,

01:56:02 and are we better or worse off?

01:56:04 I think that you look at the Bay Area,

01:56:07 housing prices are insane.

01:56:08 Well, why?

01:56:09 Well, there’s a high incentive to be physically located,

01:56:12 because if you don’t have proximity,

01:56:14 you end up paying for it and commute, right?

01:56:18 And there has been huge social pressure

01:56:21 in terms of you will be there for the meeting, right?

01:56:24 Or whatever scenario it is.

01:56:26 And I think that’s gonna be way better.

01:56:28 I think it’s gonna be much more the norm

01:56:30 to have remote employees,

01:56:31 and I think this is gonna be really great.

01:56:33 Do you have friends, or do you hear of people moving?

01:56:37 I know one family friend that moved.

01:56:40 They moved back to Michigan,

01:56:41 and they were a family with three kids

01:56:45 living in a small apartment,

01:56:46 and we’re going insane, right?

01:56:50 And they’re in tech, husband works for Google.

01:56:54 So first of all, friends of mine

01:56:57 are in the process of, or have already lost the business.

01:57:00 The thing that represents their passion, their dream,

01:57:03 it could be small entrepreneurial projects,

01:57:05 but it could be large businesses,

01:57:06 like people that run gyms.

01:57:08 Restaurants, tons of things, yeah.

01:57:10 But also, people look at themselves in the mirror

01:57:14 and ask the question of, what do I wanna do in life?

01:57:17 For some reason, they haven’t done it until COVID.

01:57:20 They really ask that question,

01:57:22 and that results often in moving or leaving the company

01:57:26 with starting your own business

01:57:28 or transitioning to a different company.

01:57:30 Do you think we’re gonna see that a lot?

01:57:33 Well, I can’t speak to that.

01:57:36 I mean, we’re definitely gonna see it

01:57:37 at a higher frequency than we did before,

01:57:39 just because I think what you’re trying to say

01:57:41 is there are decisions that you make yourself,

01:57:45 big life decisions that you make yourself,

01:57:47 and I’m gonna quit my job and start a new thing.

01:57:49 There’s also decisions that get made for you.

01:57:52 I got fired from my job, what am I gonna do, right?

01:57:55 And that’s not a decision that you think about,

01:57:58 but you’re forced to act, okay?

01:58:00 And so I think that those, you’re forced to act

01:58:02 kind of moments where global pandemic

01:58:05 comes and wipes out the economy,

01:58:06 and now your business doesn’t exist.

01:58:09 I think that does lead to more reflection, right?

01:58:11 Because you’re less anchored on what you have,

01:58:14 and it’s not a, what do I have to lose

01:58:17 versus what do I have to gain, A, B, comparison.

01:58:19 It’s more of a fresh slate.

01:58:21 Cool, I could do anything now.

01:58:23 Do I wanna do the same thing I was doing?

01:58:26 Did that make me happy?

01:58:27 Is this now time to go back to college

01:58:29 and take a class and learn a new skill?

01:58:33 Is this a time to spend time with family

01:58:36 if you can afford to do that?

01:58:37 Is this time to literally move in with parents, right?

01:58:40 I mean, all these things that were not normative before

01:58:43 suddenly become, I think, very, the value systems change.

01:58:49 And I think that’s actually a good thing

01:58:50 in the short term, at least, because it leads to,

01:58:56 there’s kind of been an overoptimization

01:58:58 along one set of priorities for the world,

01:59:01 and now maybe we’ll get to a more balanced

01:59:03 and more interesting world

01:59:05 where people are doing different things.

01:59:06 I think it could be good.

01:59:07 I think there could be more innovation

01:59:08 that comes out of it, for example.

01:59:10 What do you think about all the social chaos

01:59:12 we’re in the middle of?

01:59:13 It sucks.

01:59:14 You think it’s, let me ask you a whole,

01:59:18 you think it’s all gonna be okay?

01:59:21 Well, I think humanity will survive.

01:59:23 The, from an existential,

01:59:25 like we’re not all gonna kill, yeah, well.

01:59:27 Yeah, I don’t think the virus is gonna kill all the humans.

01:59:30 I don’t think all the humans are gonna kill all the humans.

01:59:31 I think that’s unlikely.

01:59:32 But I look at it as progress requires a catalyst, right?

01:59:42 So you need a reason for people to be willing

01:59:45 to do things that are uncomfortable.

01:59:47 I think that the US, at least,

01:59:50 but I think the world in general

01:59:51 is a pretty unoptimal place to live in for a lot of people.

01:59:56 And I think that what we’re seeing right now

01:59:58 is we’re seeing a lot of unhappiness.

02:00:00 And because of all the pressure,

02:00:03 because of all the badness in the world

02:00:05 that’s coming together,

02:00:06 it’s really kind of igniting some of that debate

02:00:07 that should have happened a long time ago, right?

02:00:10 I mean, I think that we’ll see more progress.

02:00:11 You’re asking about, offline you’re asking about politics

02:00:14 and wouldn’t it be great if politics moved faster

02:00:15 because there’s all these problems in the world

02:00:16 and we can move it.

02:00:18 Well, people are intentionally, are inherently conservative.

02:00:22 And so if you’re talking about conservative people,

02:00:25 particularly if they have heavy burdens on their shoulders

02:00:27 because they represent literally thousands of people,

02:00:31 it makes sense to be conservative.

02:00:33 But on the other hand, when you need change,

02:00:35 how do you get it?

02:00:37 The global pandemic will probably lead to some change.

02:00:40 And it’s not a directed, it’s not a directed plan,

02:00:44 but I think that it leads to people

02:00:45 asking really interesting questions.

02:00:47 And some of those questions

02:00:48 should have been asked a long time ago.

02:00:50 Well, let me know if you’ve observed this as well.

02:00:53 Something that’s bothered me in the machine learning

02:00:55 community, I’m guessing it might be prevalent

02:00:58 in other places, is something that feels like in 2020

02:01:02 increase the level of toxicity.

02:01:05 Like people are just quicker to pile on,

02:01:09 to just be, they’re just harsh on each other,

02:01:13 to like mob, pick a person that screwed up

02:01:19 and like make it a big thing.

02:01:22 And is there something that we can like,

02:01:26 yeah, have you observed that in other places?

02:01:28 Is there some way out of this?

02:01:30 I think there’s an inherent thing in humanity

02:01:32 that’s kind of an us versus them thing,

02:01:34 which is that you wanna succeed and how do you succeed?

02:01:37 Well, it’s relative to somebody else.

02:01:39 And so what’s happening in, at least in some part

02:01:43 is that with the internet and with online communication,

02:01:47 the world’s getting smaller, right?

02:01:49 And so we’re having some of the social ties

02:01:53 of like my town versus your town’s football team, right?

02:01:57 Turn into much larger and yet shallower problems.

02:02:02 And people don’t have time, the incentives,

02:02:06 the clickbait and like all these things

02:02:08 kind of really feed into this machine.

02:02:10 And I don’t know where that goes.

02:02:12 Yeah, I mean, the reason I think about that,

02:02:15 I mentioned to you this offline a little bit,

02:02:17 but I have a few difficult conversations scheduled,

02:02:23 some of them political related,

02:02:25 some of them within the community,

02:02:28 difficult personalities that went through some stuff.

02:02:30 I mean, one of them I’ve talked before,

02:02:32 I will talk again is Yann LeCun.

02:02:34 He got a little bit of crap on Twitter

02:02:38 for talking about a particular paper

02:02:40 and the bias within a data set.

02:02:42 And then there’s been a huge, in my view,

02:02:45 and I’m willing, comfortable saying it,

02:02:49 irrational, over exaggerated pile on his comments

02:02:54 because he made pretty basic comments about the fact that

02:02:58 if there’s bias in the data,

02:02:59 there’s going to be bias in the results.

02:03:02 So we should not have bias in the data,

02:03:04 but people piled on to him because he said

02:03:07 he trivialized the problem of bias.

02:03:10 Like it’s a lot more than just bias in the data,

02:03:13 but like, yes, that’s a very good point,

02:03:16 but that’s not what he was saying.

02:03:18 That’s not what he was saying.

02:03:19 And the response, like the implied response

02:03:23 that he’s basically sexist and racist

02:03:27 is something that completely drives away

02:03:30 the possibility of nuanced discussion.

02:03:32 One nice thing about like a pocket long form of conversation

02:03:37 is you can talk it out.

02:03:40 You can lay your reasoning out.

02:03:42 And even if you’re wrong,

02:03:44 you can still show that you’re a good human being

02:03:47 underneath it.

02:03:48 You know, your point about

02:03:49 you can’t have a productive discussion.

02:03:50 Well, how do you get to the point where people can turn?

02:03:53 They can learn, they can listen, they can think,

02:03:56 they can engage versus just being a shallow like,

02:04:00 and then keep moving, right?

02:04:02 And I don’t think that progress really comes from that,

02:04:06 right?

02:04:07 And I don’t think that one should expect that.

02:04:09 I think that you’d see that as reinforcing

02:04:12 individual circles and the us versus them thing.

02:04:14 And I think that’s fairly divisive.

02:04:17 Yeah, I think there’s a big role in,

02:04:20 like the people that bother me most on Twitter

02:04:24 when I observe things is not the people

02:04:26 who get very emotional, angry, like over the top.

02:04:30 It’s the people who like prop them up.

02:04:33 It’s all the, it’s this,

02:04:36 I think what should be the,

02:04:37 we should teach each other is to be sort of empathetic.

02:04:42 The thing that it’s really easy to forget,

02:04:44 particularly on like Twitter or the internet or an email,

02:04:47 is that sometimes people just have a bad day, right?

02:04:50 You have a bad day or you’re like,

02:04:53 I’ve been in the situation where it’s like between meetings,

02:04:55 like fire off a quick response to an email

02:04:57 because I want to like help get something unblocked,

02:05:00 phrase it really objectively wrong.

02:05:03 I screwed up.

02:05:04 And suddenly this is now something that sticks with people.

02:05:08 And it’s not because they’re bad.

02:05:10 It’s not because you’re bad.

02:05:11 Just psychology of like, you said a thing,

02:05:15 it sticks with you.

02:05:16 You didn’t mean it that way,

02:05:16 but it really impacted somebody

02:05:18 because the way they interpret it.

02:05:20 And this is just an aspect of working together as humans.

02:05:23 And I have a lot of optimism in the long term,

02:05:26 the very long term about what we as humanity can do.

02:05:29 But I think that’s going to be,

02:05:29 it’s just always a rough ride.

02:05:31 And you came into this by saying like,

02:05:33 what does COVID and all the social strife

02:05:36 that’s happening right now mean?

02:05:38 And I think that it’s really bad in the short term,

02:05:40 but I think it’ll lead to progress.

02:05:42 And for that, I’m very thankful.

02:05:45 Yeah, painful in the short term though.

02:05:47 Well, yeah.

02:05:48 I mean, people are out of jobs.

02:05:49 Like some people can’t eat.

02:05:50 Like it’s horrible.

02:05:52 And, but you know, it’s progress.

02:05:56 So we’ll see what happens.

02:05:58 I mean, the real question is when you look back 10 years,

02:06:01 20 years, a hundred years from now,

02:06:03 how do we evaluate the decisions are being made right now?

02:06:06 I think that’s really the way you can frame that

02:06:09 and look at it.

02:06:10 And you say, you know,

02:06:11 you integrate across all the short term horribleness

02:06:14 that’s happening and you look at what that means

02:06:17 and is the improvement across the world

02:06:19 or the regression across the world significant enough

02:06:22 to make it a good or a bad thing?

02:06:24 I think that’s the question.

02:06:26 Yeah.

02:06:27 And for that, it’s good to study history.

02:06:29 I mean, one of the big problems for me right now

02:06:32 is I’m reading The Rise and Fall of the Third Reich.

02:06:36 Light reading?

02:06:37 So it’s everything is just,

02:06:39 I just see parallels and it means it’s,

02:06:42 you have to be really careful not to overstep it,

02:06:45 but just the thing that worries me the most

02:06:48 is the pain that people feel when a few things combine,

02:06:54 which is like economic depression,

02:06:55 which is quite possible in this country.

02:06:57 And then just being disrespected in some kind of way,

02:07:02 which the German people were really disrespected

02:07:05 by most of the world, like in a way that’s over the top,

02:07:10 that something can build up

02:07:12 and then all you need is a charismatic leader

02:07:15 to go either positive or negative and both work

02:07:19 as long as they’re charismatic.

02:07:21 And there’s…

02:07:22 It’s taking advantage of, again,

02:07:23 that inflection point that the world’s in

02:07:26 and what they do with it could be good or bad.

02:07:29 And so it’s a good way to think about times now,

02:07:32 like on an individual level,

02:07:34 what we decide to do is when history is written,

02:07:38 30 years from now, what happened in 2020,

02:07:40 probably history is gonna remember 2020.

02:07:43 Yeah, I think so.

02:07:43 Either for good or bad,

02:07:46 and it’s up to us to write it so it’s good.

02:07:49 Well, one of the things I’ve observed

02:07:50 that I find fascinating is most people act

02:07:54 as though the world doesn’t change.

02:07:56 You make decision knowingly, right?

02:07:59 You make a decision where you’re predicting the future

02:08:02 based on what you’ve seen in the recent past.

02:08:04 And so if something’s always been,

02:08:06 it’s rained every single day,

02:08:07 then of course you expect it to rain today too, right?

02:08:10 On the other hand, the world changes all the time.

02:08:13 Yeah.

02:08:14 Constantly, like for better and for worse.

02:08:16 And so the question is,

02:08:17 if you’re interested in something that’s not right,

02:08:20 what is the inflection point that led to a change?

02:08:22 And you can look to history for this.

02:08:24 Like what is the catalyst that led to that explosion

02:08:27 that led to that bill that led to the,

02:08:30 like you can kind of work your way backwards from that.

02:08:33 And maybe if you pull together the right people

02:08:35 and you get the right ideas together,

02:08:36 you can actually start driving that change

02:08:38 and doing it in a way that’s productive

02:08:40 and hurts fewer people.

02:08:41 Yeah, like a single person,

02:08:43 a single event can turn all of history.

02:08:44 Absolutely, everything starts somewhere.

02:08:46 And often it’s a combination of multiple factors,

02:08:48 but yeah, these things can be engineered.

02:08:52 That’s actually the optimistic view that.

02:08:54 I’m a longterm optimist on pretty much everything

02:08:57 and human nature.

02:08:59 We can look to all the negative things

02:09:00 that humanity has, all the pettiness

02:09:03 and all the like self servingness

02:09:05 and the just the cruelty, right?

02:09:09 The biases, the just humans can be very horrible.

02:09:13 But on the other hand, we’re capable of amazing things.

02:09:17 And the progress across 100 year chunks

02:09:21 is striking.

02:09:23 And even across decades, we’ve come a long ways

02:09:26 and there’s still a long ways to go,

02:09:27 but that doesn’t mean that we’ve stopped.

02:09:29 Yeah, the kind of stuff we’ve done in the last 100 years

02:09:33 is unbelievable.

02:09:34 It’s kind of scary to think what’s gonna happen

02:09:36 in this 100 years.

02:09:37 It’s scary, like exciting.

02:09:39 Like scary in a sense that it’s kind of sad

02:09:41 that the kind of technology is gonna come out

02:09:43 in 10, 20, 30 years.

02:09:45 We’re probably too old to really appreciate

02:09:47 if you don’t grow up with it.

02:09:49 It’ll be like kids these days with their virtual reality

02:09:51 and their TikToks and stuff like this.

02:09:54 Like, how does this thing and like,

02:09:56 come on, give me my static photo.

02:10:00 My Commodore 64.

02:10:02 Yeah, exactly.

02:10:03 Okay, sorry, we kind of skipped over.

02:10:05 Let me ask on, the machine learning world

02:10:11 has been kind of inspired, their imagination captivated

02:10:15 with GPT3 and these language models.

02:10:18 I thought it’d be cool to get your opinion on it.

02:10:21 What’s your thoughts on this exciting world of,

02:10:27 it connects to computation actually,

02:10:29 is of language models that are huge

02:10:33 and take many, many computers, not just to train,

02:10:37 but to also do inference on.

02:10:39 Sure.

02:10:40 Well, I mean, it depends on what you’re speaking to there.

02:10:43 But I mean, I think that there’s been

02:10:45 a pretty well understood maximum in deep learning

02:10:48 that if you make the model bigger

02:10:49 and you shove more data into it,

02:10:51 assuming you train it right

02:10:52 and you have a good model architecture,

02:10:54 that you’ll get a better model out.

02:10:55 And so on one hand, GPT3 was not that surprising.

02:10:59 On the other hand, a tremendous amount of engineering

02:11:02 went into making it possible.

02:11:04 The implications of it are pretty huge.

02:11:07 I think that when GPT2 came out,

02:11:08 there was a very provocative blog post from OpenAI

02:11:11 talking about, we’re not gonna release it

02:11:13 because of the social damage it could cause

02:11:15 if it’s misused.

02:11:18 I think that’s still a concern.

02:11:20 I think that we need to look at how technology is applied

02:11:23 and well meaning tools can be applied in very horrible ways

02:11:26 and they can have very profound impact on that.

02:11:30 I think that GPT3 is a huge technical achievement.

02:11:34 And what will GPT4 be?

02:11:35 Well, it’ll probably be bigger, more expensive to train.

02:11:38 Really cool architectural tricks.

02:11:42 What do you think, is there,

02:11:43 I don’t know how much thought you’ve done

02:11:46 on distributed computing.

02:11:48 Is there some technical challenges that are interesting

02:11:52 that you’re hopeful about exploring

02:11:54 in terms of a system that,

02:11:57 like a piece of code that with GPT4 that might have,

02:12:05 I don’t know, hundreds of trillions of parameters

02:12:09 which have to run on thousands of computers.

02:12:11 Is there some hope that we can make that happen?

02:12:15 Yeah, well, I mean, today you can write a check

02:12:18 and get access to a thousand TPU cores

02:12:21 and do really interesting large scale training

02:12:23 and inference and things like that in Google Cloud,

02:12:26 for example, right?

02:12:27 And so I don’t think it’s a question about scale,

02:12:31 it’s a question about utility.

02:12:33 And when I look at the transformer series of architectures

02:12:36 that the GPT series is based on,

02:12:38 it’s really interesting to look at that

02:12:39 because they’re actually very simple designs.

02:12:42 They’re not recurrent.

02:12:44 The training regimens are pretty simple.

02:12:47 And so they don’t really reflect like human brains, right?

02:12:51 But they’re really good at learning language models

02:12:54 and they’re unrolled enough

02:12:55 that you can simulate some recurrence, right?

02:12:59 And so the question I think about is,

02:13:02 where does this take us?

02:13:03 Like, so we can just keep scaling it,

02:13:05 have more parameters, more data, more things,

02:13:07 we’ll get a better result for sure.

02:13:09 But are there architectural techniques

02:13:11 that can lead to progress at a faster pace, right?

02:13:15 This is when, you know, how do you get,

02:13:17 instead of just like making it a constant time bigger,

02:13:20 how do you get like an algorithmic improvement

02:13:23 out of this, right?

02:13:24 And whether it be a new training regimen,

02:13:25 if it becomes sparse networks, for example,

02:13:30 the human brain is sparse, all these networks are dense.

02:13:33 The connectivity patterns can be very different.

02:13:36 I think this is where I get very interested

02:13:38 and I’m way out of my league

02:13:39 on the deep learning side of this.

02:13:41 But I think that could lead to big breakthroughs.

02:13:43 When you talk about large scale networks,

02:13:46 one of the things that Jeff Dean likes to talk about

02:13:47 and he’s given a few talks on

02:13:50 is this idea of having a sparsely gated mixture of experts

02:13:54 kind of a model where you have, you know,

02:13:57 different nets that are trained

02:13:59 and are really good at certain kinds of tasks.

02:14:02 And so you have this distributor across a cluster

02:14:04 and so you have a lot of different computers

02:14:06 that end up being kind of locally specialized

02:14:08 in different demands.

02:14:09 And then when a query comes in,

02:14:11 you gate it and you use learn techniques

02:14:13 to route to different parts of the network.

02:14:15 And then you utilize the compute resources

02:14:18 of the entire cluster by having specialization within it.

02:14:20 And I don’t know where that goes

02:14:23 or when it starts to work,

02:14:25 but I think things like that

02:14:26 could be really interesting as well.

02:14:28 And on the data side too,

02:14:30 if you can think of data selection

02:14:33 as a kind of programming.

02:14:35 Yeah.

02:14:36 I mean, essentially, if you look at it,

02:14:37 like Karpathy talked about software 2.0,

02:14:40 I mean, in a sense, data is the program.

02:14:44 Yeah, yeah.

02:14:44 So let me try to summarize Andre’s position really quick

02:14:48 before I disagree with it.

02:14:50 Yeah.

02:14:51 So Andre Karpathy is amazing.

02:14:53 So this is nothing personal with him.

02:14:55 He’s an amazing engineer.

02:14:57 And also a good blog post writer.

02:14:59 Yeah, well, he’s a great communicator.

02:15:01 You know, he’s just an amazing person.

02:15:02 He’s also really sweet.

02:15:03 So his basic premise is that software is suboptimal.

02:15:09 I think we can all agree to that.

02:15:11 He also points out that deep learning

02:15:14 and other learning based techniques are really great

02:15:16 because you can solve problems

02:15:17 in more structured ways with less like ad hoc code

02:15:22 that people write out and don’t write test cases for

02:15:24 in some cases.

02:15:25 And so they don’t even know if it works in the first place.

02:15:27 And so if you start replacing systems of imperative code

02:15:32 with deep learning models, then you get a better result.

02:15:37 Okay.

02:15:38 And I think that he argues that software 2.0

02:15:40 is a pervasively learned set of models

02:15:44 and you get away from writing code.

02:15:45 And he’s given talks where he talks about, you know,

02:15:49 swapping over more and more and more parts of the code

02:15:50 to being learned and driven that way.

02:15:54 I think that works.

02:15:56 And if you’re predisposed to liking machine learning,

02:15:59 then I think that that’s definitely a good thing.

02:16:01 I think this is also good for accessibility in many ways

02:16:04 because certain people are not gonna write C code

02:16:06 or something.

02:16:07 And so having a data driven approach to do

02:16:10 this kind of stuff, I think can be very valuable.

02:16:12 On the other hand, there are huge trade offs.

02:16:14 It’s not clear to me that software 2.0 is the answer.

02:16:19 And probably Andre wouldn’t argue that it’s the answer

02:16:21 for every problem either.

02:16:22 But I look at machine learning as not a replacement

02:16:26 for software 1.0.

02:16:27 I look at it as a new programming paradigm.

02:16:30 And so programming paradigms, when you look across demands,

02:16:35 is structured programming where you go from go tos

02:16:38 to if, then, else, or functional programming from Lisp.

02:16:42 And you start talking about higher order functions

02:16:44 and values and things like this.

02:16:45 Or you talk about object oriented programming.

02:16:48 You’re talking about encapsulation,

02:16:49 subclassing, inheritance.

02:16:50 You start talking about generic programming

02:16:52 where you start talking about code reuse

02:16:54 through specialization and different type instantiations.

02:16:59 When you start talking about differentiable programming,

02:17:01 something that I am very excited about

02:17:04 in the context of machine learning,

02:17:05 talking about taking functions and generating variants,

02:17:09 like the derivative of another function.

02:17:11 Like that’s a programming paradigm that’s very useful

02:17:13 for solving certain classes of problems.

02:17:16 Machine learning is amazing

02:17:17 at solving certain classes of problems.

02:17:19 Like you’re not gonna write a cat detector

02:17:21 or even a language translation system by writing C code.

02:17:25 That’s not a very productive way to do things anymore.

02:17:28 And so machine learning is absolutely

02:17:31 the right way to do that.

02:17:31 In fact, I would say that learned models

02:17:34 are really one of the best ways to work

02:17:35 with the human world in general.

02:17:38 And so anytime you’re talking about sensory input

02:17:40 of different modalities,

02:17:41 anytime that you’re talking about generating things

02:17:44 in a way that makes sense to a human,

02:17:45 I think that learned models are really, really useful.

02:17:48 And that’s because humans are very difficult

02:17:50 to characterize, okay?

02:17:52 And so this is a very powerful paradigm

02:17:55 for solving classes of problems.

02:17:57 But on the other hand, imperative code is too.

02:17:59 You’re not gonna write a bootloader for your computer

02:18:03 with a deep learning model.

02:18:04 Deep learning models are very hardware intensive.

02:18:07 They’re very energy intensive

02:18:08 because you have a lot of parameters

02:18:11 and you can provably implement any function

02:18:14 with a learned model, like this has been shown,

02:18:17 but that doesn’t make it efficient.

02:18:20 And so if you’re talking about caring about a few orders

02:18:22 of magnitudes worth of energy usage,

02:18:24 then it’s useful to have other tools in the toolbox.

02:18:26 There’s also robustness too.

02:18:28 I mean, as a…

02:18:29 Yeah, exactly.

02:18:30 All the problems of dealing with data and bias in data,

02:18:32 all the problems of software 2.0.

02:18:35 And one of the great things that Andre is arguing towards,

02:18:39 which I completely agree with him,

02:18:40 is that when you start implementing things

02:18:43 with deep learning, you need to learn from software 1.0

02:18:46 in terms of testing, continuous integration,

02:18:49 how you deploy, how do you validate,

02:18:51 all these things and building systems around that

02:18:53 so that you’re not just saying like,

02:18:55 oh, it seems like it’s good, ship it, right?

02:18:58 Well, what happens when I regress something?

02:18:59 What happens when I make a classification that’s wrong

02:19:02 and now I hurt somebody, right?

02:19:05 I mean, all these things you have to reason about.

02:19:07 Yeah, but at the same time,

02:19:08 the bootloader that works for us humans

02:19:12 looks awfully a lot like a neural network, right?

02:19:15 Yeah.

02:19:16 It’s messy and you can cut out

02:19:18 different parts of the brain.

02:19:19 There’s a lot of this neuroplasticity work

02:19:22 that shows that it’s gonna adjust.

02:19:24 It’s a really interesting question,

02:19:26 how much of the world’s programming

02:19:29 could be replaced by software 2.0?

02:19:31 Like with…

02:19:32 Oh, well, I mean, it’s provably true

02:19:35 that you could replace all of it.

02:19:37 Right, so then it’s a question of the trade offs.

02:19:39 Anything that’s a function, you can.

02:19:40 So it’s not a question about if.

02:19:42 I think it’s a economic question.

02:19:44 It’s a, what kind of talent can you get?

02:19:47 What kind of trade offs in terms of maintenance, right?

02:19:50 Those kinds of questions, I think.

02:19:51 What kind of data can you collect?

02:19:53 I think one of the reasons that I’m most interested

02:19:55 in machine learning as a programming paradigm

02:19:58 is that one of the things that we’ve seen

02:20:00 across computing in general is that

02:20:02 being laser focused on one paradigm

02:20:04 often puts you in a box that’s not super great.

02:20:07 And so you look at object oriented programming,

02:20:10 like it was all the rage in the early 80s

02:20:12 and like everything has to be objects.

02:20:13 And people forgot about functional programming

02:20:15 even though it came first.

02:20:17 And then people rediscovered that,

02:20:19 hey, if you mix functional and object oriented

02:20:21 in structure, like you mix these things together,

02:20:24 you can provide very interesting tools

02:20:25 that are good at solving different problems.

02:20:28 And so the question there is how do you get

02:20:30 the best way to solve the problems?

02:20:32 It’s not about whose tribe should win, right?

02:20:36 It’s not about, you know, that shouldn’t be the question.

02:20:38 The question is how do you make it

02:20:40 so that people can solve those problems the fastest

02:20:42 and they have the right tools in their box

02:20:44 to build good libraries and they can solve these problems.

02:20:47 And when you look at that, that’s like, you know,

02:20:49 you look at reinforcement learning

02:20:50 as one really interesting subdomain of this.

02:20:52 Reinforcement learning, often you have to have

02:20:55 the integration of a learned model

02:20:57 combined with your Atari or whatever the other scenario

02:21:00 it is that you’re working in.

02:21:02 You have to combine that thing with the robot control

02:21:05 for the arm, right?

02:21:07 And so now it’s not just about that one paradigm,

02:21:11 it’s about integrating that with all the other systems

02:21:14 that you have, including often legacy systems

02:21:17 and things like this, right?

02:21:18 And so to me, I think that the interesting thing to say

02:21:21 is like, how do you get the best out of this domain

02:21:23 and how do you enable people to achieve things

02:21:25 that they otherwise couldn’t do

02:21:27 without excluding all the good things

02:21:29 we already know how to do?

02:21:31 Right, but okay, this is a crazy question,

02:21:35 but we talked a little bit about GPT3,

02:21:38 but do you think it’s possible that these language models

02:21:42 that in essence, in the language domain,

02:21:47 Software 2.0 could replace some aspect of compilation,

02:21:51 for example, or do program synthesis,

02:21:54 replace some aspect of programming?

02:21:56 Yeah, absolutely.

02:21:57 So I think that learned models in general

02:22:00 are extremely powerful,

02:22:01 and I think that people underestimate them.

02:22:04 Maybe you can suggest what I should do.

02:22:07 So if I have access to the GPT3 API,

02:22:11 would I be able to generate Swift code, for example?

02:22:14 Do you think that could do something interesting

02:22:16 and would work?

02:22:17 So GPT3 is probably not trained on the right corpus,

02:22:21 so it probably has the ability to generate some Swift.

02:22:23 I bet it does.

02:22:25 It’s probably not gonna generate a large enough body

02:22:27 of Swift to be useful,

02:22:28 but take it a next step further.

02:22:30 Like if you had the goal of training something like GPT3

02:22:33 and you wanted to train it to generate source code, right?

02:22:38 It could definitely do that.

02:22:39 Now the question is, how do you express the intent

02:22:42 of what you want filled in?

02:22:44 You can definitely write scaffolding of code

02:22:47 and say, fill in the hole,

02:22:48 and sort of put in some for loops,

02:22:50 or put in some classes or whatever.

02:22:51 And the power of these models is impressive,

02:22:53 but there’s an unsolved question, at least unsolved to me,

02:22:56 which is, how do I express the intent of what to fill in?

02:22:59 Right?

02:23:01 And kind of what you’d really want to have,

02:23:03 and I don’t know that these models are up to the task,

02:23:06 is you wanna be able to say,

02:23:08 here’s the scaffolding,

02:23:09 and here are the assertions at the end.

02:23:12 And the assertions always pass.

02:23:14 And so you want a generative model on the one hand, yes.

02:23:16 Oh, that’s fascinating, yeah.

02:23:17 Right, but you also want some loop back,

02:23:20 some reinforcement learning system or something,

02:23:23 where you’re actually saying like,

02:23:24 I need to hill climb towards something

02:23:26 that is more correct.

02:23:28 And I don’t know that we have that.

02:23:29 So it would generate not only a bunch of the code,

02:23:33 but like the checks that do the testing.

02:23:35 It would generate the tests.

02:23:37 I think the humans would generate the tests, right?

02:23:38 Oh, okay.

02:23:39 But it would be fascinating if…

02:23:41 Well, the tests are the requirements.

02:23:43 Yes, but the, okay, so…

02:23:44 Because you have to express to the model

02:23:45 what you want to…

02:23:47 You don’t just want gibberish code.

02:23:49 Look at how compelling this code looks.

02:23:51 You want a story about four horned unicorns or something.

02:23:54 Well, okay, so exactly.

02:23:55 But that’s human requirements.

02:23:57 But then I thought it’s a compelling idea

02:24:00 that the GPT4 model could generate checks

02:24:06 that are more high fidelity that check for correctness.

02:24:11 Because the code it generates,

02:24:14 like say I ask it to generate a function

02:24:18 that gives me the Fibonacci sequence.

02:24:21 Sure.

02:24:22 I don’t like…

02:24:24 So decompose the problem, right?

02:24:25 So you have two things.

02:24:26 You have, you need the ability to generate

02:24:29 syntactically correct Swift code that’s interesting, right?

02:24:33 I think GPT series of model architectures can do that.

02:24:37 But then you need the ability to add the requirements.

02:24:41 So generate Fibonacci.

02:24:43 The human needs to express that goal.

02:24:46 We don’t have that language that I know of.

02:24:49 No, I mean, it can generate stuff.

02:24:50 Have you seen what GPT3 can generate?

02:24:52 You can say, I mean, there’s a interface stuff

02:24:55 like it can generate HTML.

02:24:58 It can generate basic for loops that give you like…

02:25:02 Right, but pick HTML.

02:25:02 How do I say I want google.com?

02:25:06 Well, no, you could say…

02:25:07 Or not literally google.com.

02:25:09 How do I say I want a webpage

02:25:10 that’s got a shopping cart and this and that?

02:25:13 It does that.

02:25:14 I mean, so, okay.

02:25:14 So just, I don’t know if you’ve seen these demonstrations

02:25:17 but you type in, I want a red button

02:25:20 with the text that says hello.

02:25:22 And you type that in natural language

02:25:24 and it generates the correct HTML.

02:25:26 I’ve done this demo.

02:25:27 It’s kind of compelling.

02:25:29 So you have to prompt it with similar kinds of mappings.

02:25:33 Of course, it’s probably handpicked.

02:25:35 I got to experiment that probably,

02:25:37 but the fact that you can do that once

02:25:39 even out of like 20 is quite impressive.

02:25:43 Again, that’s very basic.

02:25:45 Like the HTML is kind of messy and bad.

02:25:48 But yes, the intent is…

02:25:49 The idea is the intent is specified in natural language.

02:25:53 Yeah, so I have not seen that.

02:25:54 That’s really cool.

02:25:55 Yeah.

02:25:56 Yeah, but the question is the correctness of that.

02:25:59 Like visually you can check, oh, the button is red,

02:26:02 but for more complicated functions

02:26:10 where the intent is harder to check.

02:26:12 This goes into like NP completeness kind of things.

02:26:15 Like I want to know that this code is correct

02:26:18 and generates a giant thing

02:26:20 that does some kind of calculation.

02:26:23 It seems to be working.

02:26:25 It’s interesting to think like,

02:26:27 should the system also try to generate checks

02:26:30 for itself for correctness?

02:26:32 Yeah, I don’t know.

02:26:33 And this is way beyond my experience.

02:26:35 The thing that I think about is that

02:26:39 there doesn’t seem to be a lot of

02:26:41 equational reasoning going on.

02:26:43 There’s a lot of pattern matching and filling in

02:26:45 and kind of propagating patterns

02:26:47 that have been seen before into the future

02:26:49 and into the generator result.

02:26:50 And so if you want to get correctness,

02:26:53 you kind of need theorem proving kind of things

02:26:55 and like higher level logic.

02:26:57 And I don’t know that…

02:26:58 You could talk to Jan about that

02:27:00 and see what the bright minds

02:27:03 are thinking about right now,

02:27:04 but I don’t think the GPT is in that vein.

02:27:08 It’s still really cool.

02:27:09 Yeah, and surprisingly, who knows,

02:27:11 maybe reasoning is…

02:27:13 Is overrated.

02:27:14 Yeah, is overrated.

02:27:15 Right, I mean, do we reason?

02:27:17 Yeah.

02:27:18 How do you tell, right?

02:27:18 Are we just pattern matching based on what we have

02:27:20 and then reverse justifying to ourselves?

02:27:22 Exactly, the reverse.

02:27:24 So like I think what the neural networks are missing

02:27:26 and I think GPT4 might have

02:27:29 is to be able to tell stories to itself about what it did.

02:27:33 Well, that’s what humans do, right?

02:27:34 I mean, you talk about like network explainability, right?

02:27:38 And we give, no, that’s a hard time about this,

02:27:40 but humans don’t know why we make decisions.

02:27:42 We have this thing called intuition

02:27:43 and then we try to like say,

02:27:45 this feels like the right thing, but why, right?

02:27:47 And you wrestle with that

02:27:49 when you’re making hard decisions

02:27:50 and is that science?

02:27:52 Not really.

02:27:54 Let me ask you about a few high level questions, I guess.

02:27:57 Because you’ve done a million things in your life

02:28:02 and been very successful.

02:28:04 A bunch of young folks listen to this,

02:28:07 ask for advice from successful people like you.

02:28:11 If you were to give advice to somebody,

02:28:15 you know, another graduate student

02:28:17 or some high school student

02:28:19 about pursuing a career in computing

02:28:23 or just advice about life in general,

02:28:25 is there some words of wisdom you can give them?

02:28:28 So I think you come back to change

02:28:30 and profound leaps happen

02:28:34 because people are willing to believe

02:28:35 that change is possible

02:28:36 and that the world does change

02:28:39 and are willing to do the hard thing

02:28:41 that it takes to make change happen.

02:28:42 And whether it be implementing a new programming language

02:28:45 or employing a new system

02:28:47 or employing a new research paper,

02:28:49 designing a new thing,

02:28:50 moving the world forward in science

02:28:51 and philosophy, whatever,

02:28:53 it really comes down to somebody

02:28:54 who’s willing to put in the work, right?

02:28:57 And you have, the work is hard

02:29:00 for a whole bunch of different reasons.

02:29:01 One of which is, it’s work, right?

02:29:06 And so you have to have the space in your life

02:29:08 in which you can do that work,

02:29:09 which is why going to grad school

02:29:11 can be a beautiful thing for certain people.

02:29:14 But also there’s a self doubt that happens.

02:29:16 Like you’re two years into a project,

02:29:18 is it going anywhere, right?

02:29:20 Well, what do you do?

02:29:21 Do you just give up because it’s hard?

02:29:23 No, no, I mean, some people like suffering.

02:29:26 And so you plow through it.

02:29:29 The secret to me is that you have to love what you’re doing

02:29:31 and follow that passion

02:29:35 because when you get to the hard times,

02:29:37 that’s when, if you love what you’re doing,

02:29:40 you’re willing to kind of push through.

02:29:41 And this is really hard

02:29:45 because it’s hard to know what you will love doing

02:29:48 until you start doing a lot of things.

02:29:50 And so that’s why I think that,

02:29:51 particularly early in your career,

02:29:53 it’s good to experiment.

02:29:54 Do a little bit of everything.

02:29:55 Go take the survey class on

02:29:59 the first half of every class in your upper division lessons

02:30:03 and just get exposure to things

02:30:05 because certain things will resonate with you

02:30:07 and you’ll find out, wow, I’m really good at this.

02:30:08 I’m really smart at this.

02:30:10 Well, it’s just because it works with the way your brain.

02:30:13 And when something jumps out,

02:30:14 I mean, that’s one of the things

02:30:15 that people often ask about is like,

02:30:19 well, I think there’s a bunch of cool stuff out there.

02:30:21 Like how do I pick the thing?

02:30:23 Like how do you hook, in your life,

02:30:27 how did you just hook yourself in and stuck with it?

02:30:30 Well, I got lucky, right?

02:30:31 I mean, I think that many people forget

02:30:34 that a huge amount of it or most of it is luck, right?

02:30:38 So let’s not forget that.

02:30:41 So for me, I fell in love with computers early on

02:30:44 because they spoke to me, I guess.

02:30:49 What language did they speak?

02:30:50 Basic.

02:30:51 Basic, yeah.

02:30:53 But then it was just kind of following

02:30:56 a set of logical progressions,

02:30:58 but also deciding that something that was hard

02:31:01 was worth doing and a lot of fun, right?

02:31:04 And so I think that that is also something

02:31:06 that’s true for many other domains,

02:31:08 which is if you find something that you love doing

02:31:10 that’s also hard, if you invest yourself in it

02:31:13 and add value to the world,

02:31:14 then it will mean something generally, right?

02:31:17 And again, that can be a research paper,

02:31:19 that can be a software system,

02:31:20 that can be a new robot,

02:31:22 that can be, there’s many things that that can be,

02:31:24 but a lot of it is like real value

02:31:27 comes from doing things that are hard.

02:31:29 And that doesn’t mean you have to suffer, but.

02:31:33 It’s hard.

02:31:34 I mean, you don’t often hear that message.

02:31:36 We talked about it last time a little bit,

02:31:38 but it’s one of my, not enough people talk about this.

02:31:43 It’s beautiful to hear a successful person.

02:31:47 Well, and self doubt and imposter syndrome,

02:31:49 these are all things that successful people

02:31:52 suffer with as well,

02:31:53 particularly when they put themselves

02:31:55 in a point of being uncomfortable,

02:31:56 which I like to do now and then

02:31:59 just because it puts you in learning mode.

02:32:02 Like if you wanna grow as a person,

02:32:04 put yourself in a room with a bunch of people

02:32:07 that know way more about whatever you’re talking about

02:32:09 than you do and ask dumb questions.

02:32:11 And guess what?

02:32:13 Smart people love to teach often,

02:32:15 not always, but often.

02:32:16 And if you listen, if you’re prepared to listen,

02:32:18 if you’re prepared to grow,

02:32:19 if you’re prepared to make connections,

02:32:20 you can do some really interesting things.

02:32:22 And I think a lot of progress is made by people

02:32:25 who kind of hop between domains now and then,

02:32:28 because they bring a perspective into a field

02:32:32 that nobody else has,

02:32:34 if people have only been working in that field themselves.

02:32:38 We mentioned that the universe is kind of like a compiler.

02:32:43 The entirety of it, the whole evolution

02:32:44 is kind of a kind of compilation.

02:32:46 Maybe us human beings are kind of compilers.

02:32:51 Let me ask the old sort of question

02:32:53 that I didn’t ask you last time,

02:32:54 which is what’s the meaning of it all?

02:32:57 Is there a meaning?

02:32:58 Like if you asked a compiler why,

02:33:01 what would a compiler say?

02:33:03 What’s the meaning of life?

02:33:04 What’s the meaning of life?

02:33:06 I’m prepared for it not to mean anything.

02:33:09 Here we are, all biological things programmed to survive

02:33:14 and propagate our DNA.

02:33:19 And maybe the universe is just a computer

02:33:21 and you just go until entropy takes over the universe

02:33:25 and then you’re done.

02:33:27 I don’t think that’s a very productive way

02:33:29 to live your life, if so.

02:33:33 And so I prefer to bias towards the other way,

02:33:34 which is saying the universe has a lot of value.

02:33:37 And I take happiness out of other people.

02:33:41 And a lot of times part of that’s having kids,

02:33:43 but also the relationships you build with other people.

02:33:46 And so the way I try to live my life is like,

02:33:49 what can I do that has value?

02:33:51 How can I move the world forward?

02:33:52 How can I take what I’m good at

02:33:54 and bring it into the world?

02:33:57 And I’m one of these people that likes to work really hard

02:34:00 and be very focused on the things that I do.

02:34:03 And so if I’m gonna do that,

02:34:05 how can it be in a domain that actually will matter?

02:34:08 Because a lot of things that we do,

02:34:10 we find ourselves in the cycle of like,

02:34:11 okay, I’m doing a thing.

02:34:12 I’m very familiar with it.

02:34:13 I’ve done it for a long time.

02:34:15 I’ve never done anything else,

02:34:16 but I’m not really learning, right?

02:34:19 I’m not really, I’m keeping things going,

02:34:21 but there’s a younger generation

02:34:23 that can do the same thing,

02:34:24 maybe even better than me, right?

02:34:26 Maybe if I actually step out of this

02:34:28 and jump into something I’m less comfortable with,

02:34:31 it’s scary.

02:34:32 But on the other hand,

02:34:33 it gives somebody else a new opportunity.

02:34:34 It also then puts you back in learning mode,

02:34:37 and that can be really interesting.

02:34:38 And one of the things I’ve learned is that

02:34:41 when you go through that,

02:34:42 that first you’re deep into imposter syndrome,

02:34:45 but when you start working your way out,

02:34:46 you start to realize,

02:34:47 hey, well, there’s actually a method to this.

02:34:49 And now I’m able to add new things

02:34:53 because I bring different perspective.

02:34:54 And this is one of the good things

02:34:57 about bringing different kinds of people together.

02:34:59 Diversity of thought is really important.

02:35:01 And if you can pull together people

02:35:04 that are coming at things from different directions,

02:35:06 you often get innovation.

02:35:07 And I love to see that, that aha moment

02:35:10 where you’re like, oh, we’ve really cracked this.

02:35:12 This is something nobody’s ever done before.

02:35:15 And then if you can do it in the context

02:35:16 where it adds value, other people can build on it,

02:35:18 it helps move the world,

02:35:20 then that’s what really excites me.

02:35:22 So that kind of description

02:35:24 of the magic of the human experience,

02:35:26 do you think we’ll ever create that in an AGI system?

02:35:29 Do you think we’ll be able to create,

02:35:34 give AI systems a sense of meaning

02:35:38 where they operate in this kind of world

02:35:39 exactly in the way you’ve described,

02:35:41 which is they interact with each other,

02:35:43 they interact with us humans.

02:35:44 Sure, sure.

02:35:45 Well, so, I mean, why are you being so a speciest, right?

02:35:50 All right, so AGI versus Bionets,

02:35:54 or something like that versus biology, right?

02:35:57 You know, what are we but machines, right?

02:36:00 We’re just programmed to run our,

02:36:02 we have our objective function

02:36:03 that we were optimized for, right?

02:36:06 And so we’re doing our thing,

02:36:07 we think we have purpose, but do we really, right?

02:36:10 I’m not prepared to say that those newfangled AGI’s

02:36:13 have no soul just because we don’t understand them, right?

02:36:16 And I think that would be, when they exist,

02:36:20 that would be very premature to look at a new thing

02:36:24 through your own lens without fully understanding it.

02:36:28 You might be just saying that

02:36:29 because AI systems in the future

02:36:31 will be listening to this and then.

02:36:33 Oh yeah, exactly.

02:36:34 You don’t wanna say anything.

02:36:34 Please be nice to me, you know,

02:36:36 when Skynet kills everybody, please spare me.

02:36:39 So wise look ahead thinking.

02:36:42 Yeah, but I mean, I think that people will spend

02:36:44 a lot of time worrying about this kind of stuff,

02:36:46 and I think that what we should be worrying about

02:36:48 is how do we make the world better?

02:36:49 And the thing that I’m most scared about with AGI’s

02:36:52 is not that necessarily the Skynet

02:36:57 will start shooting everybody with lasers

02:36:58 and stuff like that to use us for our calories.

02:37:03 The thing that I’m worried about is that

02:37:05 humanity, I think, needs a challenge.

02:37:08 And if we get into a mode of not having a personal challenge,

02:37:11 not having a personal contribution,

02:37:13 whether that be like, you know, your kids

02:37:15 and seeing what they grow into and helping guide them,

02:37:18 whether it be your community that you’re engaged in,

02:37:21 you’re driving forward, whether it be your work

02:37:23 and the things that you’re doing

02:37:25 and the people you’re working with

02:37:25 and the products you’re building and the contribution there,

02:37:28 if people don’t have a objective,

02:37:31 I’m afraid what that means.

02:37:33 And I think that this would lead to a rise

02:37:37 of the worst part of people, right?

02:37:39 Instead of people striving together

02:37:42 and trying to make the world better,

02:37:45 it could degrade into a very unpleasant world.

02:37:49 But I don’t know.

02:37:51 I mean, we hopefully have a long ways to go

02:37:53 before we discover that.

02:37:54 And fortunately, we have pretty on the ground problems

02:37:57 with the pandemic right now,

02:37:58 and so I think we should be focused on that as well.

02:38:01 Yeah, ultimately, just as you said, you’re optimistic.

02:38:04 I think it helps for us to be optimistic.

02:38:07 So that’s fake it until you make it.

02:38:10 Yeah, well, and why not?

02:38:11 What’s the other side, right?

02:38:12 So, I mean, I’m not personally a very religious person,

02:38:17 but I’ve heard people say like,

02:38:19 oh yeah, of course I believe in God.

02:38:20 Of course I go to church, because if God’s real,

02:38:23 you know, I wanna be on the right side of that.

02:38:25 If it’s not real, it doesn’t matter.

02:38:27 Yeah, it doesn’t matter.

02:38:27 And so, you know, that’s a fair way to do it.

02:38:32 Yeah, I mean, the same thing with nuclear deterrence,

02:38:35 all, you know, global warming, all these things,

02:38:38 all these threats, natural engineer pandemics,

02:38:41 all these threats we face.

02:38:42 I think it’s paralyzing to be terrified

02:38:49 of all the possible ways we could destroy ourselves.

02:38:52 I think it’s much better or at least productive

02:38:56 to be hopeful and to engineer defenses

02:38:59 against these things, to engineer a future

02:39:03 where like, you know, see like a positive future

02:39:06 and engineer that future.

02:39:07 Yeah, well, and I think that’s another thing

02:39:10 to think about as, you know, a human,

02:39:12 particularly if you’re young and trying to figure out

02:39:14 what it is that you wanna be when you grow up, like I am.

02:39:18 I’m always looking for that.

02:39:19 The question then is, how do you wanna spend your time?

02:39:24 And right now there seems to be a norm

02:39:26 of being a consumption culture.

02:39:28 Like I’m gonna watch the news and revel

02:39:31 in how horrible everything is right now.

02:39:33 I’m going to go find out about the latest atrocity

02:39:36 and find out all the details of like the terrible thing

02:39:38 that happened and be outraged by it.

02:39:41 You can spend a lot of time watching TV

02:39:44 and watching the news at home or whatever

02:39:46 people watch these days, I don’t know.

02:39:49 But that’s a lot of hours, right?

02:39:51 And those are hours that if you’re turned

02:39:53 to being productive, learning, growing,

02:39:56 experiencing, you know, when the pandemic’s over,

02:39:59 going exploring, right, it leads to more growth.

02:40:03 And I think it leads to more optimism and happiness

02:40:06 because you’re building, right?

02:40:08 You’re building yourself, you’re building your capabilities,

02:40:10 you’re building your viewpoints,

02:40:12 you’re building your perspective.

02:40:13 And I think that a lot of the consuming

02:40:18 of other people’s messages leads

02:40:19 to kind of a negative viewpoint,

02:40:21 which you need to be aware of what’s happening

02:40:24 because that’s also important,

02:40:25 but there’s a balance that I think focusing

02:40:28 on creation is a very valuable thing to do.