Transcript
00:00:00 The following is a conversation with Daphne Koller,
00:00:03 a professor of computer science at Stanford University,
00:00:06 a cofounder of Coursera with Andrew Ng,
00:00:08 and founder and CEO of Incitro,
00:00:11 a company at the intersection
00:00:13 of machine learning and biomedicine.
00:00:15 We’re now in the exciting early days
00:00:17 of using the data driven methods of machine learning
00:00:20 to help discover and develop new drugs
00:00:22 and treatments at scale.
00:00:24 Daphne and Incitro are leading the way on this
00:00:27 with breakthroughs that may ripple
00:00:29 through all fields of medicine,
00:00:31 including ones most critical for helping
00:00:34 with the current coronavirus pandemic.
00:00:37 This conversation was recorded
00:00:38 before the COVID 19 outbreak.
00:00:41 For everyone feeling the medical, psychological,
00:00:43 and financial burden of this crisis,
00:00:45 I’m sending love your way.
00:00:47 Stay strong, we’re in this together, we’ll beat this thing.
00:00:51 This is the Artificial Intelligence Podcast.
00:00:54 If you enjoy it, subscribe on YouTube,
00:00:56 review it with five stars on Apple Podcast,
00:00:58 support it on Patreon,
00:01:00 or simply connect with me on Twitter
00:01:02 at Lex Friedman, spelled F R I D M A N.
00:01:05 As usual, I’ll do a few minutes of ads now
00:01:08 and never any ads in the middle
00:01:09 that can break the flow of this conversation.
00:01:11 I hope that works for you
00:01:13 and doesn’t hurt the listening experience.
00:01:15 This show is presented by Cash App,
00:01:17 the number one finance app in the app store.
00:01:20 When you get it, use code LEXPODCAST.
00:01:23 Cash App lets you send money to friends,
00:01:25 buy Bitcoin, and invest in the stock market
00:01:27 with as little as one dollar.
00:01:30 Since Cash App allows you to send
00:01:31 and receive money digitally,
00:01:33 peer to peer, and security in all digital transactions
00:01:36 is very important,
00:01:38 let me mention the PCI data security standard
00:01:41 that Cash App is compliant with.
00:01:43 I’m a big fan of standards for safety and security.
00:01:46 PCI DSS is a good example of that,
00:01:49 where a bunch of competitors got together
00:01:51 and agreed that there needs to be a global standard
00:01:53 around the security of transactions.
00:01:56 Now we just need to do the same for autonomous vehicles
00:01:58 and AI systems in general.
00:02:00 So again, if you get Cash App from the App Store
00:02:03 or Google Play and use the code LEXPODCAST,
00:02:07 you get $10 and Cash App will also donate $10 to FIRST,
00:02:11 an organization that is helping to advance robotics
00:02:14 and STEM education for young people around the world.
00:02:17 And now here’s my conversation with Daphne Koller.
00:02:22 So you cofounded Coursera and made a huge impact
00:02:25 in the global education of AI.
00:02:26 And after five years in August, 2016,
00:02:29 wrote a blog post saying that you’re stepping away
00:02:33 and wrote, quote,
00:02:34 it is time for me to turn to another critical challenge,
00:02:37 the development of machine learning
00:02:38 and its applications to improving human health.
00:02:41 So let me ask two far out philosophical questions.
00:02:45 One, do you think we’ll one day find cures
00:02:48 for all major diseases known today?
00:02:50 And two, do you think we’ll one day figure out
00:02:53 a way to extend the human lifespan,
00:02:55 perhaps to the point of immortality?
00:02:59 So one day is a very long time
00:03:01 and I don’t like to make predictions
00:03:04 of the type we will never be able to do X
00:03:07 because I think that’s a smacks of hubris.
00:03:12 It seems that never in the entire eternity
00:03:16 of human existence will we be able to solve a problem.
00:03:19 That being said, curing disease is very hard
00:03:24 because oftentimes by the time you discover the disease,
00:03:28 a lot of damage has already been done.
00:03:30 And so to assume that we would be able to cure disease
00:03:34 at that stage assumes that we would come up with ways
00:03:37 of basically regenerating entire parts of the human body
00:03:41 in the way that actually returns it to its original state.
00:03:45 And that’s a very challenging problem.
00:03:47 We have cured very few diseases.
00:03:49 We’ve been able to provide treatment
00:03:51 for an increasingly large number,
00:03:52 but the number of things that you could actually define
00:03:54 to be cures is actually not that large.
00:03:59 So I think that there’s a lot of work
00:04:02 that would need to happen before one could legitimately say
00:04:05 that we have cured even a reasonable number,
00:04:08 far less all diseases.
00:04:10 On the scale of zero to 100,
00:04:12 where are we in understanding the fundamental mechanisms
00:04:15 of all of major diseases?
00:04:18 What’s your sense?
00:04:19 So from the computer science perspective
00:04:21 that you’ve entered the world of health,
00:04:24 how far along are we?
00:04:26 I think it depends on which disease.
00:04:29 I mean, there are ones where I would say
00:04:31 we’re maybe not quite at a hundred
00:04:33 because biology is really complicated
00:04:35 and there’s always new things that we uncover
00:04:38 that people didn’t even realize existed.
00:04:43 But I would say there’s diseases
00:04:44 where we might be in the 70s or 80s,
00:04:48 and then there’s diseases in which I would say
00:04:51 with probably the majority where we’re really close to zero.
00:04:55 Would Alzheimer’s and schizophrenia
00:04:57 and type two diabetes fall closer to zero or to the 80?
00:05:04 I think Alzheimer’s is probably closer to zero than to 80.
00:05:11 There are hypotheses,
00:05:12 but I don’t think those hypotheses have as of yet
00:05:17 been sufficiently validated that we believe them to be true.
00:05:21 And there is an increasing number of people
00:05:23 who believe that the traditional hypotheses
00:05:25 might not really explain what’s going on.
00:05:28 I would also say that Alzheimer’s and schizophrenia
00:05:31 and even type two diabetes are not really one disease.
00:05:35 They’re almost certainly a heterogeneous collection
00:05:39 of mechanisms that manifest in clinically similar ways.
00:05:43 So in the same way that we now understand
00:05:46 that breast cancer is really not one disease,
00:05:48 it is multitude of cellular mechanisms,
00:05:53 all of which ultimately translate
00:05:55 to uncontrolled proliferation, but it’s not one disease.
00:05:59 The same is almost undoubtedly true
00:06:01 for those other diseases as well.
00:06:02 And that understanding that needs to precede
00:06:05 any understanding of the specific mechanisms
00:06:08 of any of those other diseases.
00:06:10 Now, in schizophrenia, I would say
00:06:11 we’re almost certainly closer to zero than to anything else.
00:06:15 Type two diabetes is a bit of a mix.
00:06:18 There are clear mechanisms that are implicated
00:06:21 that I think have been validated
00:06:22 that have to do with insulin resistance and such,
00:06:25 but there’s almost certainly there as well
00:06:28 many mechanisms that we have not yet understood.
00:06:31 You’ve also thought and worked a little bit
00:06:34 on the longevity side.
00:06:35 Do you see the disease and longevity as overlapping
00:06:40 completely, partially, or not at all as efforts?
00:06:45 Those mechanisms are certainly overlapping.
00:06:48 There’s a well known phenomenon that says
00:06:51 that for most diseases, other than childhood diseases,
00:06:56 the risk for contracting that disease
00:07:01 increases exponentially year on year,
00:07:03 every year from the time you’re about 40.
00:07:05 So obviously there’s a connection between those two things.
00:07:10 That’s not to say that they’re identical.
00:07:12 There’s clearly aging that happens
00:07:14 that is not really associated with any specific disease.
00:07:18 And there’s also diseases and mechanisms of disease
00:07:22 that are not specifically related to aging.
00:07:25 So I think overlap is where we’re at.
00:07:29 Okay.
00:07:30 It is a little unfortunate that we get older
00:07:32 and it seems that there’s some correlation
00:07:34 with the occurrence of diseases
00:07:39 or the fact that we get older.
00:07:40 And both are quite sad.
00:07:43 I mean, there’s processes that happen as cells age
00:07:46 that I think are contributing to disease.
00:07:49 Some of those have to do with DNA damage
00:07:52 that accumulates as cells divide
00:07:54 where the repair mechanisms don’t fully correct for those.
00:07:59 There are accumulations of proteins
00:08:03 that are misfolded and potentially aggregate
00:08:06 and those too contribute to disease
00:08:08 and will contribute to inflammation.
00:08:10 There’s a multitude of mechanisms that have been uncovered
00:08:14 that are sort of wear and tear at the cellular level
00:08:17 that contribute to disease processes
00:08:21 and I’m sure there’s many that we don’t yet understand.
00:08:24 On a small tangent and perhaps philosophical,
00:08:30 the fact that things get older
00:08:32 and the fact that things die is a very powerful feature
00:08:36 for the growth of new things.
00:08:38 It’s a learning, it’s a kind of learning mechanism.
00:08:41 So it’s both tragic and beautiful.
00:08:44 So do you, so in trying to fight disease
00:08:52 and trying to fight aging,
00:08:55 do you think about sort of the useful fact of our mortality
00:08:58 or would you, like if you were, could be immortal,
00:09:02 would you choose to be immortal?
00:09:07 Again, I think immortal is a very long time
00:09:10 and I don’t know that that would necessarily be something
00:09:16 that I would want to aspire to
00:09:17 but I think all of us aspire to an increased health span,
00:09:24 I would say, which is an increased amount of time
00:09:27 where you’re healthy and active
00:09:29 and feel as you did when you were 20
00:09:33 and we’re nowhere close to that.
00:09:36 People deteriorate physically and mentally over time
00:09:41 and that is a very sad phenomenon.
00:09:43 So I think a wonderful aspiration would be
00:09:47 if we could all live to the biblical 120 maybe
00:09:52 in perfect health.
00:09:53 In high quality of life.
00:09:54 High quality of life.
00:09:55 I think that would be an amazing goal
00:09:57 for us to achieve as a society
00:09:59 now is the right age 120 or 100 or 150.
00:10:03 I think that’s up for debate
00:10:05 but I think an increased health span
00:10:07 is a really worthy goal.
00:10:10 And anyway, in a grand time of the age of the universe,
00:10:14 it’s all pretty short.
00:10:16 So from the perspective,
00:10:18 you’ve done obviously a lot of incredible work
00:10:20 in machine learning.
00:10:22 So what role do you think data and machine learning
00:10:25 play in this goal of trying to understand diseases
00:10:29 and trying to eradicate diseases?
00:10:32 Up until now, I don’t think it’s played
00:10:35 very much of a significant role
00:10:37 because largely the data sets that one really needed
00:10:42 to enable a powerful machine learning methods,
00:10:47 those data sets haven’t really existed.
00:10:49 There’s been dribs and drabs
00:10:50 and some interesting machine learning
00:10:53 that has been applied, I would say machine learning
00:10:55 slash data science,
00:10:57 but the last few years are starting to change that.
00:11:00 So we now see an increase in some large data sets
00:11:06 but equally importantly, an increase in technologies
00:11:11 that are able to produce data at scale.
00:11:14 It’s not typically the case that people have deliberately
00:11:19 proactively used those tools
00:11:21 for the purpose of generating data for machine learning.
00:11:24 They, to the extent that those techniques
00:11:26 have been used for data production,
00:11:28 they’ve been used for data production
00:11:29 to drive scientific discovery
00:11:31 and the machine learning came as a sort of byproduct
00:11:34 second stage of, oh, you know, now we have a data set,
00:11:36 let’s do machine learning on that
00:11:38 rather than a more simplistic data analysis method.
00:11:41 But what we are doing in Citro
00:11:44 is actually flipping that around and saying,
00:11:46 here’s this incredible repertoire of methods
00:11:50 that bioengineers, cell biologists have come up with,
00:11:54 let’s see if we can put them together in brand new ways
00:11:57 with the goal of creating data sets
00:12:00 that machine learning can really be applied on productively
00:12:03 to create powerful predictive models
00:12:06 that can help us address fundamental problems
00:12:08 in human health.
00:12:09 So really focus to get, make data the primary focus
00:12:14 and the primary goal and find,
00:12:16 use the mechanisms of biology and chemistry
00:12:18 to create the kinds of data set
00:12:23 that could allow machine learning to benefit the most.
00:12:25 I wouldn’t put it in those terms
00:12:27 because that says that data is the end goal.
00:12:30 Data is the means.
00:12:32 So for us, the end goal is helping address challenges
00:12:35 in human health and the method that we’ve elected to do that
00:12:39 is to apply machine learning to build predictive models
00:12:44 and machine learning, in my opinion,
00:12:45 can only be really successfully applied
00:12:48 especially the more powerful models
00:12:50 if you give it data that is of sufficient scale
00:12:53 and sufficient quality.
00:12:54 So how do you create those data sets
00:12:58 so as to drive the ability to generate predictive models
00:13:03 which subsequently help improve human health?
00:13:05 So before we dive into the details of that,
00:13:08 let me take a step back and ask when and where
00:13:13 was your interest in human health born?
00:13:16 Are there moments, events, perhaps if I may ask,
00:13:19 tragedies in your own life that catalyzes passion
00:13:23 or was it the broader desire to help humankind?
00:13:26 So I would say it’s a bit of both.
00:13:29 So on, I mean, my interest in human health
00:13:32 actually dates back to the early 2000s
00:13:37 when a lot of my peers in machine learning
00:13:43 and I were using data sets
00:13:45 that frankly were not very inspiring.
00:13:47 Some of us old timers still remember
00:13:49 the quote unquote 20 news groups data set
00:13:52 where this was literally a bunch of texts
00:13:55 from 20 news groups,
00:13:57 a concept that doesn’t really even exist anymore.
00:13:59 And the question was, can you classify
00:14:01 which news group a particular bag of words came from?
00:14:06 And it wasn’t very interesting.
00:14:08 The data sets at the time on the biology side
00:14:12 were much more interesting,
00:14:14 both from a technical and also from
00:14:15 an aspirational perspective.
00:14:17 They were still pretty small,
00:14:18 but they were better than 20 news groups.
00:14:20 And so I started out, I think just by wanting
00:14:25 to do something that was more, I don’t know,
00:14:27 societally useful and technically interesting.
00:14:30 And then over time became more and more interested
00:14:34 in the biology and the human health aspects for themselves
00:14:40 and began to work even sometimes on papers
00:14:43 that were just in biology
00:14:45 without having a significant machine learning component.
00:14:48 I think my interest in drug discovery
00:14:52 is partly due to an incident I had with
00:14:58 when my father sadly passed away about 12 years ago.
00:15:02 He had an autoimmune disease that settled in his lungs
00:15:08 and the doctors basically said,
00:15:11 well, there’s only one thing that we could do,
00:15:13 which is give him prednisone.
00:15:15 At some point, I remember a doctor even came and said,
00:15:17 hey, let’s do a lung biopsy to figure out
00:15:19 which autoimmune disease he has.
00:15:20 And I said, would that be helpful?
00:15:23 Would that change treatment?
00:15:24 He said, no, there’s only prednisone.
00:15:25 That’s the only thing we can give him.
00:15:27 And I had friends who were rheumatologists who said
00:15:29 the FDA would never approve prednisone today
00:15:32 because the ratio of side effects to benefit
00:15:37 is probably not large enough.
00:15:39 Today, we’re in a state where there’s probably four or five,
00:15:44 maybe even more, well, it depends for which autoimmune disease,
00:15:48 but there are multiple drugs that can help people
00:15:52 with autoimmune disease,
00:15:53 many of which didn’t exist 12 years ago.
00:15:56 And I think we’re at a golden time in some ways
00:16:00 in drug discovery where there’s the ability to create drugs
00:16:05 that are much more safe and much more effective
00:16:10 than we’ve ever been able to before.
00:16:13 And what’s lacking is enough understanding
00:16:16 of biology and mechanism to know where to aim that engine.
00:16:22 And I think that’s where machine learning can help.
00:16:25 So in 2018, you started and now lead a company in Citro,
00:16:29 which is, like you mentioned,
00:16:32 perhaps the focus is drug discovery
00:16:34 and the utilization of machine learning for drug discovery.
00:16:38 So you mentioned that, quote,
00:16:40 we’re really interested in creating
00:16:42 what you might call a disease in a dish model,
00:16:45 disease in a dish models,
00:16:47 places where diseases are complex,
00:16:49 where we really haven’t had a good model system,
00:16:52 where typical animal models that have been used for years,
00:16:55 including testing on mice, just aren’t very effective.
00:16:58 So can you try to describe what is an animal model
00:17:02 and what is a disease in a dish model?
00:17:05 Sure.
00:17:06 So an animal model for disease
00:17:09 is where you create effectively,
00:17:13 it’s what it sounds like.
00:17:14 It’s oftentimes a mouse where we have introduced
00:17:19 some external perturbation that creates the disease
00:17:22 and then we cure that disease.
00:17:26 And the hope is that by doing that,
00:17:28 we will cure a similar disease in the human.
00:17:31 The problem is that oftentimes
00:17:33 the way in which we generate the disease in the animal
00:17:36 has nothing to do with how that disease
00:17:38 actually comes about in a human.
00:17:40 It’s what you might think of as a copy of the phenotype,
00:17:44 a copy of the clinical outcome,
00:17:46 but the mechanisms are quite different.
00:17:48 And so curing the disease in the animal,
00:17:52 which in most cases doesn’t happen naturally,
00:17:54 mice don’t get Alzheimer’s, they don’t get diabetes,
00:17:57 they don’t get atherosclerosis,
00:17:58 they don’t get autism or schizophrenia.
00:18:02 Those cures don’t translate over
00:18:05 to what happens in the human.
00:18:08 And that’s where most drugs fails
00:18:10 just because the findings that we had in the mouse
00:18:13 don’t translate to a human.
00:18:16 The disease in the dish models is a fairly new approach.
00:18:20 It’s been enabled by technologies
00:18:24 that have not existed for more than five to 10 years.
00:18:28 So for instance, the ability for us to take a cell
00:18:32 from any one of us, you or me,
00:18:35 revert that say skin cell to what’s called stem cell status,
00:18:39 which is what’s called the pluripotent cell
00:18:44 that can then be differentiated
00:18:46 into different types of cells.
00:18:47 So from that pluripotent cell,
00:18:49 one can create a Lex neuron or a Lex cardiomyocyte
00:18:54 or a Lex hepatocyte that has your genetics,
00:18:57 but that right cell type.
00:19:00 And so if there’s a genetic burden of disease
00:19:04 that would manifest in that particular cell type,
00:19:07 you might be able to see it by looking at those cells
00:19:10 and saying, oh, that’s what potentially sick cells
00:19:13 look like versus healthy cells
00:19:15 and then explore what kind of interventions
00:19:20 might revert the unhealthy looking cell to a healthy cell.
00:19:24 Now, of course, curing cells is not the same
00:19:27 as curing people.
00:19:29 And so there’s still potentially a translatability gap,
00:19:33 but at least for diseases that are driven,
00:19:38 say by human genetics and where the human genetics
00:19:41 is what drives the cellular phenotype,
00:19:43 there is some reason to hope that if we revert those cells
00:19:47 in which the disease begins
00:19:49 and where the disease is driven by genetics
00:19:52 and we can revert that cell back to a healthy state,
00:19:55 maybe that will help also revert
00:19:58 the more global clinical phenotype.
00:20:00 So that’s really what we’re hoping to do.
00:20:02 That step, that backward step, I was reading about it,
00:20:06 the Yamanaka factor.
00:20:08 Yes.
00:20:09 So it’s like that reverse step back to stem cells.
00:20:12 Yes.
00:20:13 Seems like magic.
00:20:14 It is.
00:20:15 Honestly, before that happened,
00:20:17 I think very few people would have predicted
00:20:20 that to be possible.
00:20:21 It’s amazing.
00:20:22 Can you maybe elaborate, is it actually possible?
00:20:25 Like where, like how stable?
00:20:27 So this result was maybe like,
00:20:29 I don’t know how many years ago,
00:20:30 maybe 10 years ago was first demonstrated,
00:20:32 something like that.
00:20:33 Is this, how hard is this?
00:20:35 Like how noisy is this backward step?
00:20:37 It seems quite incredible and cool.
00:20:39 It is, it is incredible and cool.
00:20:42 It was much more, I think finicky and bespoke
00:20:46 at the early stages when the discovery was first made.
00:20:49 But at this point, it’s become almost industrialized.
00:20:54 There are what’s called contract research organizations,
00:20:59 vendors that will take a sample from a human
00:21:02 and revert it back to stem cell status.
00:21:04 And it works a very good fraction of the time.
00:21:07 Now there are people who will ask,
00:21:10 I think good questions.
00:21:12 Is this really truly a stem cell or does it remember
00:21:15 certain aspects of what,
00:21:17 of changes that were made in the human beyond the genetics?
00:21:22 It’s passed as a skin cell, yeah.
00:21:24 It’s passed as a skin cell or it’s passed
00:21:26 in terms of exposures to different environmental factors
00:21:29 and so on.
00:21:30 So I think the consensus right now
00:21:33 is that these are not always perfect
00:21:36 and there is little bits and pieces of memory sometimes,
00:21:40 but by and large, these are actually pretty good.
00:21:44 So one of the key things,
00:21:47 well, maybe you can correct me,
00:21:48 but one of the useful things for machine learning
00:21:50 is size, scale of data.
00:21:54 How easy it is to do these kinds of reversals to stem cells
00:21:59 and then disease in a dish models at scale.
00:22:02 Is that a huge challenge or not?
00:22:06 So the reversal is not as of this point
00:22:11 something that can be done at the scale
00:22:14 of tens of thousands or hundreds of thousands.
00:22:18 I think total number of stem cells or IPS cells
00:22:22 that are what’s called induced pluripotent stem cells
00:22:25 in the world I think is somewhere between five and 10,000
00:22:30 last I looked.
00:22:31 Now again, that might not count things that exist
00:22:34 in this or that academic center
00:22:36 and they may add up to a bit more,
00:22:37 but that’s about the range.
00:22:40 So it’s not something that you could at this point
00:22:42 generate IPS cells from a million people,
00:22:45 but maybe you don’t need to
00:22:47 because maybe that background is enough
00:22:51 because it can also be now perturbed in different ways.
00:22:56 And some people have done really interesting experiments
00:23:00 in for instance, taking cells from a healthy human
00:23:05 and then introducing a mutation into it
00:23:08 using one of the other miracle technologies
00:23:11 that’s emerged in the last decade
00:23:13 which is CRISPR gene editing
00:23:16 and introduced a mutation that is known to be pathogenic.
00:23:19 And so you can now look at the healthy cells
00:23:22 and the unhealthy cells, the one with the mutation
00:23:24 and do a one on one comparison
00:23:26 where everything else is held constant.
00:23:28 And so you could really start to understand specifically
00:23:31 what the mutation does at the cellular level.
00:23:34 So the IPS cells are a great starting point
00:23:37 and obviously more diversity is better
00:23:39 because you also wanna capture ethnic background
00:23:42 and how that affects things,
00:23:43 but maybe you don’t need one from every single patient
00:23:46 with every single type of disease
00:23:48 because we have other tools at our disposal.
00:23:50 Well, how much difference is there between people
00:23:52 I mentioned ethnic background in terms of IPS cells?
00:23:54 So we’re all like, it seems like these magical cells
00:23:59 that can do to create anything
00:24:01 between different populations, different people.
00:24:04 Is there a lot of variability between cell cells?
00:24:07 Well, first of all, there’s the variability,
00:24:09 that’s driven simply by the fact
00:24:10 that genetically we’re different.
00:24:13 So a stem cell that’s derived from my genotype
00:24:15 is gonna be different from a stem cell
00:24:18 that’s derived from your genotype.
00:24:20 There’s also some differences that have more to do with
00:24:23 for whatever reason, some people’s stem cells
00:24:27 differentiate better than other people’s stem cells.
00:24:29 We don’t entirely understand why.
00:24:31 So there’s certainly some differences there as well,
00:24:34 but the fundamental difference
00:24:35 and the one that we really care about and is a positive
00:24:38 is that the fact that the genetics are different
00:24:43 and therefore recapitulate my disease burden
00:24:45 versus your disease burden.
00:24:47 What’s a disease burden?
00:24:49 Well, a disease burden is just if you think,
00:24:52 I mean, it’s not a well defined mathematical term,
00:24:55 although there are mathematical formulations of it.
00:24:58 If you think about the fact that some of us are more likely
00:25:01 to get a certain disease than others
00:25:03 because we have more variations in our genome
00:25:07 that are causative of the disease,
00:25:09 maybe fewer that are protective of the disease.
00:25:12 People have quantified that
00:25:14 using what are called polygenic risk scores,
00:25:17 which look at all of the variations
00:25:20 in an individual person’s genome
00:25:23 and add them all up in terms of how much risk they confer
00:25:26 for a particular disease.
00:25:27 And then they’ve put people on a spectrum
00:25:30 of their disease risk.
00:25:32 And for certain diseases where we’ve been sufficiently
00:25:36 powered to really understand the connection
00:25:38 between the many, many small variations
00:25:41 that give rise to an increased disease risk,
00:25:44 there’s some pretty significant differences
00:25:47 in terms of the risk between the people,
00:25:49 say at the highest decile of this polygenic risk score
00:25:52 and the people at the lowest decile.
00:25:53 Sometimes those differences are factor of 10 or 12 higher.
00:25:58 So there’s definitely a lot that our genetics
00:26:03 contributes to disease risk, even if it’s not
00:26:07 by any stretch the full explanation.
00:26:09 And from a machine learning perspective,
00:26:10 there’s signal there.
00:26:12 There is definitely signal in the genetics
00:26:14 and there’s even more signal, we believe,
00:26:19 in looking at the cells that are derived
00:26:21 from those different genetics because in principle,
00:26:25 you could say all the signal is there at the genetics level.
00:26:28 So we don’t need to look at the cells,
00:26:30 but our understanding of the biology is so limited at this
00:26:34 point than seeing what actually happens at the cellular
00:26:37 level is a heck of a lot closer to the human clinical outcome
00:26:41 than looking at the genetics directly.
00:26:44 And so we can learn a lot more from it
00:26:47 than we could by looking at genetics alone.
00:26:49 So just to get a sense, I don’t know if it’s easy to do,
00:26:51 but what kind of data is useful
00:26:54 in this disease in a dish model?
00:26:56 Like what’s the source of raw data information?
00:26:59 And also from my outsider’s perspective,
00:27:03 so biology and cells are squishy things.
00:27:08 And then how do you connect the computer to that?
00:27:15 Which sensory mechanisms, I guess.
00:27:17 So that’s another one of those revolutions
00:27:20 that have happened in the last 10 years
00:27:22 in that our ability to measure cells very quantitatively
00:27:27 has also dramatically increased.
00:27:30 So back when I started doing biology in the late 90s,
00:27:35 early 2000s, that was the initial era
00:27:40 where we started to measure biology
00:27:42 in really quantitative ways using things like microarrays,
00:27:46 where you would measure in a single experiment
00:27:50 the activity level, what’s called expression level
00:27:53 of every gene in the genome in that sample.
00:27:56 And that ability is what actually allowed us
00:28:00 to even understand that there are molecular subtypes
00:28:04 of diseases like cancer, where up until that point,
00:28:06 it’s like, oh, you have breast cancer.
00:28:09 But then when we looked at the molecular data,
00:28:13 it was clear that there’s different subtypes
00:28:14 of breast cancer that at the level of gene activity
00:28:17 look completely different to each other.
00:28:20 So that was the beginning of this process.
00:28:23 Now we have the ability to measure individual cells
00:28:26 in terms of their gene activity
00:28:28 using what’s called single cell RNA sequencing,
00:28:31 which basically sequences the RNA,
00:28:35 which is that activity level of different genes
00:28:39 for every gene in the genome.
00:28:40 And you could do that at single cell level.
00:28:42 So that’s an incredibly powerful way of measuring cells.
00:28:45 I mean, you literally count the number of transcripts.
00:28:47 So it really turns that squishy thing
00:28:50 into something that’s digital.
00:28:51 Another tremendous data source that’s emerged
00:28:55 in the last few years is microscopy
00:28:57 and specifically even super resolution microscopy,
00:29:00 where you could use digital reconstruction
00:29:03 to look at subcellular structures,
00:29:06 sometimes even things that are below
00:29:08 the diffraction limit of light
00:29:10 by doing a sophisticated reconstruction.
00:29:13 And again, that gives you a tremendous amount of information
00:29:16 at the subcellular level.
00:29:18 There’s now more and more ways that amazing scientists
00:29:22 out there are developing for getting new types
00:29:27 of information from even single cells.
00:29:30 And so that is a way of turning those squishy things
00:29:35 into digital data.
00:29:37 Into beautiful data sets.
00:29:38 But so that data set then with machine learning tools
00:29:42 allows you to maybe understand the developmental,
00:29:45 like the mechanism of a particular disease.
00:29:49 And if it’s possible to sort of at a high level describe,
00:29:54 how does that help lead to a drug discovery
00:30:01 that can help prevent, reverse that mechanism?
00:30:05 So I think there’s different ways in which this data
00:30:08 could potentially be used.
00:30:10 Some people use it for scientific discovery
00:30:13 and say, oh, look, we see this phenotype
00:30:17 at the cellular level.
00:30:20 So let’s try and work our way backwards
00:30:22 and think which genes might be involved in pathways
00:30:26 that give rise to that.
00:30:27 So that’s a very sort of analytical method
00:30:32 to sort of work our way backwards
00:30:35 using our understanding of known biology.
00:30:38 Some people use it in a somewhat more,
00:30:44 sort of forward, if that was a backward,
00:30:46 this would be forward, which is to say,
00:30:48 okay, if I can perturb this gene,
00:30:51 does it show a phenotype that is similar
00:30:54 to what I see in disease patients?
00:30:56 And so maybe that gene is actually causal of the disease.
00:30:58 So that’s a different way.
00:31:00 And then there’s what we do,
00:31:01 which is basically to take that very large collection
00:31:06 of data and use machine learning to uncover the patterns
00:31:10 that emerge from it.
00:31:12 So for instance, what are those subtypes
00:31:14 that might be similar at the human clinical outcome,
00:31:18 but quite distinct when you look at the molecular data?
00:31:21 And then if we can identify such a subtype,
00:31:25 are there interventions that if I apply it
00:31:27 to cells that come from this subtype of the disease
00:31:32 and you apply that intervention,
00:31:34 it could be a drug or it could be a CRISPR gene intervention,
00:31:38 does it revert the disease state
00:31:41 to something that looks more like normal,
00:31:42 happy, healthy cells?
00:31:44 And so hopefully if you see that,
00:31:46 that gives you a certain hope
00:31:50 that that intervention will also have
00:31:53 a meaningful clinical benefit to people.
00:31:55 And there’s obviously a bunch of things
00:31:56 that you would wanna do after that to validate that,
00:31:58 but it’s a very different and much less hypothesis driven way
00:32:03 of uncovering new potential interventions
00:32:06 and might give rise to things that are not the same things
00:32:10 that everyone else is already looking at.
00:32:12 That’s, I don’t know, I’m just like to psychoanalyze
00:32:16 my own feeling about our discussion currently.
00:32:18 It’s so exciting to talk about sort of a machine,
00:32:21 fundamentally, well, something that’s been turned
00:32:23 into a machine learning problem
00:32:25 and that says can have so much real world impact.
00:32:29 That’s how I feel too.
00:32:30 That’s kind of exciting because I’m so,
00:32:32 most of my day is spent with data sets
00:32:35 that I guess closer to the news groups.
00:32:39 So this is a kind of, it just feels good to talk about.
00:32:41 In fact, I almost don’t wanna talk about machine learning.
00:32:45 I wanna talk about the fundamentals of the data set,
00:32:47 which is an exciting place to be.
00:32:50 I agree with you.
00:32:51 It’s what gets me up in the morning.
00:32:53 It’s also what attracts a lot of the people
00:32:57 who work at InCetro to InCetro
00:32:59 because I think all of the,
00:33:01 certainly all of our machine learning people
00:33:03 are outstanding and could go get a job selling ads online
00:33:08 or doing eCommerce or even self driving cars.
00:33:12 But I think they would want, they come to us
00:33:17 because they want to work on something
00:33:20 that has more of an aspirational nature
00:33:22 and can really benefit humanity.
00:33:24 What, with these approaches, what do you hope,
00:33:28 what kind of diseases can be helped?
00:33:31 We mentioned Alzheimer’s, schizophrenia, type 2 diabetes.
00:33:33 Can you just describe the various kinds of diseases
00:33:36 that this approach can help?
00:33:38 Well, we don’t know.
00:33:39 And I try and be very cautious about making promises
00:33:43 about some things that, oh, we will cure X.
00:33:46 People make that promise.
00:33:48 And I think it’s, I tried to first deliver and then promise
00:33:52 as opposed to the other way around.
00:33:54 There are characteristics of a disease
00:33:57 that make it more likely that this type of approach
00:34:00 can potentially be helpful.
00:34:02 So for instance, diseases that have
00:34:04 a very strong genetic basis are ones
00:34:08 that are more likely to manifest
00:34:10 in a stem cell derived model.
00:34:13 We would want the cellular models
00:34:16 to be relatively reproducible and robust
00:34:19 so that you could actually get enough of those cells
00:34:25 and in a way that isn’t very highly variable and noisy.
00:34:30 You would want the disease to be relatively contained
00:34:34 in one or a small number of cell types
00:34:36 that you could actually create in an in vitro,
00:34:40 in a dish setting.
00:34:40 Whereas if it’s something that’s really broad and systemic
00:34:43 and involves multiple cells
00:34:45 that are in very distal parts of your body,
00:34:48 putting that all in the dish is really challenging.
00:34:50 So we want to focus on the ones
00:34:53 that are most likely to be successful today
00:34:56 with the hope, I think, that really smart bioengineers
00:35:01 out there are developing better and better systems
00:35:04 all the time so that diseases that might not be tractable
00:35:07 today might be tractable in three years.
00:35:11 So for instance, five years ago,
00:35:14 these stem cell derived models didn’t really exist.
00:35:16 People were doing most of the work in cancer cells
00:35:18 and cancer cells are very, very poor models
00:35:21 of most human biology because they’re,
00:35:24 A, they were cancer to begin with
00:35:25 and B, as you passage them and they proliferate in a dish,
00:35:30 they become, because of the genomic instability,
00:35:32 even less similar to human biology.
00:35:35 Now we have these stem cell derived models.
00:35:39 We have the capability to reasonably robustly,
00:35:42 not quite at the right scale yet, but close,
00:35:45 to derive what’s called organoids,
00:35:47 which are these teeny little sort of multicellular organ,
00:35:54 sort of models of an organ system.
00:35:56 So there’s cerebral organoids and liver organoids
00:35:59 and kidney organoids and.
00:36:01 Yeah, brain organoids.
00:36:03 That’s organoids.
00:36:04 It’s possibly the coolest thing I’ve ever seen.
00:36:05 Is that not like the coolest thing?
00:36:07 Yeah.
00:36:08 And then I think on the horizon,
00:36:09 we’re starting to see things like connecting
00:36:11 these organoids to each other
00:36:13 so that you could actually start,
00:36:15 and there’s some really cool papers that start to do that
00:36:17 where you can actually start to say,
00:36:19 okay, can we do multi organ system stuff?
00:36:22 There’s many challenges to that.
00:36:23 It’s not easy by any stretch, but it might,
00:36:27 I’m sure people will figure it out.
00:36:29 And in three years or five years,
00:36:31 there will be disease models that we could make
00:36:34 for things that we can’t make today.
00:36:35 Yeah, and this conversation would seem almost outdated
00:36:38 with the kind of scale that could be achieved
00:36:40 in like three years.
00:36:41 I hope so.
00:36:42 That’s the hope.
00:36:42 That would be so cool.
00:36:43 So you’ve cofounded Coursera with Andrew Ng
00:36:48 and were part of the whole MOOC revolution.
00:36:51 So to jump topics a little bit,
00:36:53 can you maybe tell the origin story of the history,
00:36:57 the origin story of MOOCs, of Coursera,
00:37:00 and in general, your teaching to huge audiences
00:37:07 on a very sort of impactful topic of AI in general?
00:37:12 So I think the origin story of MOOCs
00:37:15 emanates from a number of efforts
00:37:17 that occurred at Stanford University
00:37:20 around the late 2000s
00:37:25 where different individuals within Stanford,
00:37:28 myself included, were getting really excited
00:37:31 about the opportunities of using online technologies
00:37:35 as a way of achieving both improved quality of teaching
00:37:38 and also improved scale.
00:37:40 And so Andrew, for instance,
00:37:44 led the Stanford Engineering Everywhere,
00:37:48 which was sort of an attempt to take 10 Stanford courses
00:37:51 and put them online just as video lectures.
00:37:55 I led an effort within Stanford to take some of the courses
00:38:00 and really create a very different teaching model
00:38:04 that broke those up into smaller units
00:38:07 and had some of those embedded interactions and so on,
00:38:11 which got a lot of support from university leaders
00:38:14 because they felt like it was potentially a way
00:38:17 of improving the quality of instruction at Stanford
00:38:19 by moving to what’s now called the flipped classroom model.
00:38:22 And so those efforts eventually sort of started
00:38:26 to interplay with each other
00:38:28 and created a tremendous sense of excitement and energy
00:38:30 within the Stanford community
00:38:32 about the potential of online teaching
00:38:36 and led in the fall of 2011
00:38:39 to the launch of the first Stanford MOOCs.
00:38:43 By the way, MOOCs, it’s probably impossible
00:38:46 that people don’t know, but it’s, I guess, massive.
00:38:49 Open online courses. Open online courses.
00:38:51 We did not come up with the acronym.
00:38:54 I’m not particularly fond of the acronym,
00:38:57 but it is what it is. It is what it is.
00:38:58 Big bang is not a great term for the start of the universe,
00:39:01 but it is what it is. Probably so.
00:39:05 So anyway, so those courses launched in the fall of 2011,
00:39:10 and there were, within a matter of weeks,
00:39:13 with no real publicity campaign, just a New York Times article
00:39:17 that went viral, about 100,000 students or more
00:39:22 in each of those courses.
00:39:24 And I remember this conversation that Andrew and I had.
00:39:29 We were just like, wow, there’s this real need here.
00:39:33 And I think we both felt like, sure,
00:39:36 we were accomplished academics and we could go back
00:39:39 and go back to our labs, write more papers.
00:39:42 But if we did that, then this wouldn’t happen.
00:39:45 And it seemed too important not to happen.
00:39:48 And so we spent a fair bit of time debating,
00:39:51 do we wanna do this as a Stanford effort,
00:39:55 kind of building on what we’d started?
00:39:56 Do we wanna do this as a for profit company?
00:39:59 Do we wanna do this as a nonprofit?
00:40:00 And we decided ultimately to do it as we did with Coursera.
00:40:04 And so, you know, we started really operating
00:40:09 as a company at the beginning of 2012.
00:40:13 And the rest is history.
00:40:15 But how did you, was that really surprising to you?
00:40:19 How did you at that time and at this time
00:40:23 make sense of this need for sort of global education
00:40:27 you mentioned that you felt that, wow,
00:40:29 the popularity indicates that there’s a hunger
00:40:33 for sort of globalization of learning.
00:40:37 I think there is a hunger for learning that,
00:40:43 you know, globalization is part of it,
00:40:45 but I think it’s just a hunger for learning.
00:40:47 The world has changed in the last 50 years.
00:40:50 It used to be that you finished college, you got a job,
00:40:54 by and large, the skills that you learned in college
00:40:57 were pretty much what got you through
00:40:59 the rest of your job history.
00:41:01 And yeah, you learn some stuff,
00:41:02 but it wasn’t a dramatic change.
00:41:05 Today, we’re in a world where the skills that you need
00:41:09 for a lot of jobs, they didn’t even exist
00:41:11 when you went to college.
00:41:12 And the jobs, and many of the jobs that existed
00:41:14 when you went to college don’t even exist today or are dying.
00:41:18 So part of that is due to AI, but not only.
00:41:22 And we need to find a way of keeping people,
00:41:27 giving people access to the skills that they need today.
00:41:29 And I think that’s really what’s driving
00:41:32 a lot of this hunger.
00:41:33 So I think if we even take a step back,
00:41:37 for you, all of this started in trying to think
00:41:39 of new ways to teach or to,
00:41:43 new ways to sort of organize the material
00:41:47 and present the material in a way
00:41:48 that would help the education process, the pedagogy, yeah.
00:41:51 So what have you learned about effective education
00:41:56 from this process of playing,
00:41:57 of experimenting with different ideas?
00:42:00 So we learned a number of things.
00:42:03 Some of which I think could translate back
00:42:06 and have translated back effectively
00:42:08 to how people teach on campus.
00:42:09 And some of which I think are more specific
00:42:11 to people who learn online,
00:42:13 more sort of people who learn as part of their daily life.
00:42:18 So we learned, for instance, very quickly
00:42:20 that short is better.
00:42:23 So people who are especially in the workforce
00:42:26 can’t do a 15 week semester long course.
00:42:30 They just can’t fit that into their lives.
00:42:32 Sure, can you describe the shortness of what?
00:42:35 The entirety, so every aspect,
00:42:39 so the little lecture, the lecture’s short,
00:42:41 the course is short.
00:42:43 Both.
00:42:43 We started out, the first online education efforts
00:42:47 were actually MIT’s OpenCourseWare initiatives.
00:42:50 And that was recording of classroom lectures and,
00:42:55 Hour and a half or something like that, yeah.
00:42:57 And that didn’t really work very well.
00:43:00 I mean, some people benefit.
00:43:01 I mean, of course they did,
00:43:03 but it’s not really a very palatable experience
00:43:06 for someone who has a job and three kids
00:43:11 and they need to run errands and such.
00:43:13 They can’t fit 15 weeks into their life
00:43:17 and the hour and a half is really hard.
00:43:20 So we learned very quickly.
00:43:22 I mean, we started out with short video modules
00:43:26 and over time we made them shorter
00:43:28 because we realized that 15 minutes was still too long.
00:43:31 If you wanna fit in when you’re waiting in line
00:43:33 for your kid’s doctor’s appointment,
00:43:35 it’s better if it’s five to seven.
00:43:38 We learned that 15 week courses don’t work
00:43:42 and you really wanna break this up into shorter units
00:43:44 so that there is a natural completion point,
00:43:46 gives people a sense of they’re really close
00:43:48 to finishing something meaningful.
00:43:50 They can always come back and take part two and part three.
00:43:53 We also learned that compressing the content works
00:43:56 really well because if some people that pace works well
00:44:00 and for others, they can always rewind and watch again.
00:44:03 And so people have the ability
00:44:05 to then learn at their own pace.
00:44:06 And so that flexibility, the brevity and the flexibility
00:44:11 are both things that we found to be very important.
00:44:15 We learned that engagement during the content is important
00:44:18 and the quicker you give people feedback,
00:44:20 the more likely they are to be engaged.
00:44:22 Hence the introduction of these,
00:44:24 which we actually was an intuition that I had going in
00:44:27 and was then validated using data
00:44:30 that introducing some of these sort of little micro quizzes
00:44:34 into the lectures really helps.
00:44:36 Self graded as automatically graded assessments
00:44:39 really helped too because it gives people feedback.
00:44:41 See, there you are.
00:44:43 So all of these are valuable.
00:44:45 And then we learned a bunch of other things too.
00:44:47 We did some really interesting experiments, for instance,
00:44:49 on gender bias and how having a female role model
00:44:54 as an instructor can change the balance of men to women
00:44:59 in terms of, especially in STEM courses.
00:45:02 And you could do that online by doing AB testing
00:45:04 in ways that would be really difficult to go on campus.
00:45:07 Oh, that’s exciting.
00:45:09 But so the shortness, the compression,
00:45:11 I mean, that’s actually, so that probably is true
00:45:15 for all good editing is always just compressing the content,
00:45:20 making it shorter.
00:45:21 So that puts a lot of burden on the creator of the,
00:45:24 the instructor and the creator of the educational content.
00:45:28 Probably most lectures at MIT or Stanford
00:45:31 could be five times shorter
00:45:34 if the preparation was put enough.
00:45:37 So maybe people might disagree with that,
00:45:41 but like the Christmas, the clarity that a lot of the,
00:45:45 like Coursera delivers is, how much effort does that take?
00:45:50 So first of all, let me say that it’s not clear
00:45:54 that that crispness would work as effectively
00:45:57 in a face to face setting
00:45:58 because people need time to absorb the material.
00:46:02 And so you need to at least pause
00:46:04 and give people a chance to reflect and maybe practice.
00:46:07 And that’s what MOOCs do is that they give you
00:46:09 these chunks of content and then ask you
00:46:11 to practice with it.
00:46:13 And that’s where I think some of the newer pedagogy
00:46:16 that people are adopting in face to face teaching
00:46:19 that have to do with interactive learning and such
00:46:21 can be really helpful.
00:46:23 But both those approaches,
00:46:26 whether you’re doing that type of methodology
00:46:29 in online teaching or in that flipped classroom,
00:46:32 interactive teaching.
00:46:34 What’s that, sorry to pause, what’s flipped classroom?
00:46:37 Flipped classroom is a way in which online content
00:46:41 is used to supplement face to face teaching
00:46:45 where people watch the videos perhaps
00:46:47 and do some of the exercises before coming to class.
00:46:49 And then when they come to class,
00:46:51 it’s actually to do much deeper problem solving
00:46:53 oftentimes in a group.
00:46:56 But any one of those different pedagogies
00:47:00 that are beyond just standing there and droning on
00:47:03 in front of the classroom for an hour and 15 minutes
00:47:06 require a heck of a lot more preparation.
00:47:09 And so it’s one of the challenges I think that people have
00:47:13 that we had when trying to convince instructors
00:47:15 to teach on Coursera.
00:47:16 And it’s part of the challenges that pedagogy experts
00:47:20 on campus have in trying to get faculty
00:47:22 to teach differently is that it’s actually harder
00:47:23 to teach that way than it is to stand there and drone.
00:47:27 Do you think MOOCs will replace in person education
00:47:32 or become the majority of in person of education
00:47:37 of the way people learn in the future?
00:47:41 Again, the future could be very far away,
00:47:43 but where’s the trend going do you think?
00:47:46 So I think it’s a nuanced and complicated answer.
00:47:50 I don’t think MOOCs will replace face to face teaching.
00:47:55 I think learning is in many cases a social experience.
00:48:00 And even at Coursera, we had people who naturally formed
00:48:05 study groups, even when they didn’t have to,
00:48:07 to just come and talk to each other.
00:48:10 And we found that that actually benefited their learning
00:48:14 in very important ways.
00:48:15 So there was more success among learners
00:48:19 who had those study groups than among ones who didn’t.
00:48:22 So I don’t think it’s just gonna,
00:48:23 oh, we’re all gonna just suddenly learn online
00:48:26 with a computer and no one else in the same way
00:48:28 that recorded music has not replaced live concerts.
00:48:33 But I do think that especially when you are thinking
00:48:38 about continuing education, the stuff that people get
00:48:42 when they’re traditional,
00:48:44 whatever high school, college education is done,
00:48:47 and they yet have to maintain their level of expertise
00:48:52 and skills in a rapidly changing world,
00:48:54 I think people will consume more and more educational content
00:48:58 in this online format because going back to school
00:49:01 for formal education is not an option for most people.
00:49:04 Briefly, it might be a difficult question to ask,
00:49:07 but there’s a lot of people fascinated
00:49:09 by artificial intelligence, by machine learning,
00:49:12 by deep learning.
00:49:13 Is there a recommendation for the next year
00:49:18 or for a lifelong journey of somebody interested in this?
00:49:21 How do they begin?
00:49:23 How do they enter that learning journey?
00:49:27 I think the important thing is first to just get started.
00:49:30 And there’s plenty of online content that one can get
00:49:36 for both the core foundations of mathematics
00:49:40 and statistics and programming.
00:49:42 And then from there to machine learning,
00:49:44 I would encourage people not to skip
00:49:47 to quickly pass the foundations
00:49:48 because I find that there’s a lot of people
00:49:51 who learn machine learning, whether it’s online
00:49:53 or on campus without getting those foundations.
00:49:56 And they basically just turn the crank on existing models
00:50:00 in ways that A, don’t allow for a lot of innovation
00:50:03 and an adjustment to the problem at hand,
00:50:07 but also B, are sometimes just wrong
00:50:09 and they don’t even realize that their application is wrong
00:50:12 because there’s artifacts that they haven’t fully understood.
00:50:15 So I think the foundations,
00:50:17 machine learning is an important step.
00:50:19 And then actually start solving problems,
00:50:24 try and find someone to solve them with
00:50:27 because especially at the beginning,
00:50:28 it’s useful to have someone to bounce ideas off
00:50:31 and fix mistakes that you make
00:50:33 and you can fix mistakes that they make,
00:50:35 but then just find practical problems,
00:50:40 whether it’s in your workplace or if you don’t have that,
00:50:43 Kaggle competitions or such are a really great place
00:50:46 to find interesting problems and just practice.
00:50:50 Practice.
00:50:52 Perhaps a bit of a romanticized question,
00:50:54 but what idea in deep learning do you find,
00:50:59 have you found in your journey the most beautiful
00:51:02 or surprising or interesting?
00:51:07 Perhaps not just deep learning,
00:51:09 but AI in general, statistics.
00:51:14 I’m gonna answer with two things.
00:51:19 One would be the foundational concept of end to end training,
00:51:23 which is that you start from the raw data
00:51:26 and you train something that is not like a single piece,
00:51:32 but rather towards the actual goal that you’re looking to.
00:51:38 From the raw data to the outcome,
00:51:40 like no details in between.
00:51:43 Well, not no details, but the fact that you,
00:51:45 I mean, you could certainly introduce building blocks
00:51:47 that were trained towards other tasks.
00:51:50 I’m actually coming to that in my second half of the answer,
00:51:53 but it doesn’t have to be like a single monolithic blob
00:51:57 in the middle.
00:51:58 Actually, I think that’s not ideal,
00:52:00 but rather the fact that at the end of the day,
00:52:02 you can actually train something that goes all the way
00:52:04 from the beginning to the end.
00:52:06 And the other one that I find really compelling
00:52:09 is the notion of learning a representation
00:52:13 that in its turn, even if it was trained to another task,
00:52:18 can potentially be used as a much more rapid starting point
00:52:24 to solving a different task.
00:52:26 And that’s, I think, reminiscent
00:52:29 of what makes people successful learners.
00:52:32 It’s something that is relatively new
00:52:35 in the machine learning space.
00:52:36 I think it’s underutilized even relative
00:52:38 to today’s capabilities, but more and more
00:52:41 of how do we learn sort of reusable representation?
00:52:45 And so end to end and transfer learning.
00:52:49 Yeah.
00:52:51 Is it surprising to you that neural networks
00:52:53 are able to, in many cases, do these things?
00:52:56 Is it maybe taken back to when you first would dive deep
00:53:02 into neural networks or in general, even today,
00:53:05 is it surprising that neural networks work at all
00:53:07 and work wonderfully to do this kind of raw end to end
00:53:12 and end to end learning and even transfer learning?
00:53:16 I think I was surprised by how well
00:53:22 when you have large enough amounts of data,
00:53:26 it’s possible to find a meaningful representation
00:53:32 in what is an exceedingly high dimensional space.
00:53:36 And so I find that to be really exciting
00:53:39 and people are still working on the math for that.
00:53:41 There’s more papers on that every year.
00:53:43 And I think it would be really cool
00:53:46 if we figured that out, but that to me was a surprise
00:53:52 because in the early days when I was starting my way
00:53:55 in machine learning and the data sets were rather small,
00:53:58 I think we believed, I believed that you needed
00:54:02 to have a much more constrained
00:54:05 and knowledge rich search space
00:54:08 to really make, to really get to a meaningful answer.
00:54:11 And I think it was true at the time.
00:54:13 What I think is still a question
00:54:18 is will a completely knowledge free approach
00:54:23 where there’s no prior knowledge going
00:54:26 into the construction of the model,
00:54:28 is that gonna be the solution or not?
00:54:31 It’s not actually the solution today
00:54:34 in the sense that the architecture of a convolutional
00:54:38 neural network that’s used for images
00:54:41 is actually quite different
00:54:43 to the type of network that’s used for language
00:54:46 and yet different from the one that’s used for speech
00:54:50 or biology or any other application.
00:54:52 There’s still some insight that goes
00:54:55 into the structure of the network
00:54:58 to get the right performance.
00:55:00 Will you be able to come up
00:55:01 with a universal learning machine?
00:55:03 I don’t know.
00:55:05 I wonder if there’s always has to be some insight
00:55:07 injected somewhere or whether it can converge.
00:55:10 So you’ve done a lot of interesting work
00:55:13 with probabilistic graphical models in general,
00:55:16 Bayesian deep learning and so on.
00:55:18 Can you maybe speak high level,
00:55:21 how can learning systems deal with uncertainty?
00:55:25 One of the limitations I think of a lot
00:55:28 of machine learning models is that
00:55:33 they come up with an answer
00:55:35 and you don’t know how much you can believe that answer.
00:55:40 And oftentimes the answer is actually
00:55:47 quite poorly calibrated relative to its uncertainties.
00:55:50 Even if you look at where the confidence
00:55:55 that comes out of say the neural network at the end,
00:55:58 and you ask how much more likely
00:56:01 is an answer of 0.8 versus 0.9,
00:56:04 it’s not really in any way calibrated
00:56:07 to the actual reliability of that network
00:56:12 and how true it is.
00:56:13 And the further away you move from the training data,
00:56:16 the more, not only the more wrong the network is,
00:56:20 often it’s more wrong and more confident
00:56:22 in its wrong answer.
00:56:24 And that is a serious issue in a lot of application areas.
00:56:29 So when you think for instance,
00:56:30 about medical diagnosis as being maybe an epitome
00:56:33 of how problematic this can be,
00:56:35 if you were training your network
00:56:37 on a certain set of patients
00:56:40 and a certain patient population,
00:56:41 and I have a patient that is an outlier
00:56:44 and there’s no human that looks at this,
00:56:46 and that patient is put into a neural network
00:56:49 and your network not only gives
00:56:50 a completely incorrect diagnosis,
00:56:51 but is supremely confident
00:56:53 in its wrong answer, you could kill people.
00:56:56 So I think creating more of an understanding
00:57:01 of how do you produce networks
00:57:05 that are calibrated in their uncertainty
00:57:09 and can also say, you know what, I give up.
00:57:10 I don’t know what to say about this particular data instance
00:57:14 because I’ve never seen something
00:57:16 that’s sufficiently like it before.
00:57:18 I think it’s going to be really important
00:57:20 in mission critical applications,
00:57:23 especially ones where human life is at stake
00:57:25 and that includes medical applications,
00:57:28 but it also includes automated driving
00:57:31 because you’d want the network to be able to say,
00:57:33 you know what, I have no idea what this blob is
00:57:36 that I’m seeing in the middle of the road.
00:57:37 So I’m just going to stop
00:57:38 because I don’t want to potentially run over a pedestrian
00:57:41 that I don’t recognize.
00:57:42 Is there good mechanisms, ideas of how to allow
00:57:47 learning systems to provide that uncertainty
00:57:52 along with their predictions?
00:57:54 Certainly people have come up with mechanisms
00:57:57 that involve Bayesian deep learning,
00:58:00 deep learning that involves Gaussian processes.
00:58:04 I mean, there’s a slew of different approaches
00:58:07 that people have come up with.
00:58:09 There’s methods that use ensembles of networks
00:58:13 trained with different subsets of data
00:58:15 or different random starting points.
00:58:17 Those are actually sometimes surprisingly good
00:58:20 at creating a sort of set of how confident
00:58:24 or not you are in your answer.
00:58:26 It’s very much an area of open research.
00:58:30 Let’s cautiously venture back into the land of philosophy
00:58:33 and speaking of AI systems providing uncertainty,
00:58:37 somebody like Stuart Russell believes
00:58:41 that as we create more and more intelligence systems,
00:58:43 it’s really important for them to be full of self doubt
00:58:46 because if they’re given more and more power,
00:58:51 we want the way to maintain human control
00:58:54 over AI systems or human supervision, which is true.
00:58:57 Like you just mentioned with autonomous vehicles,
00:58:59 it’s really important to get human supervision
00:59:02 when the car is not sure because if it’s really confident
00:59:05 in cases when it can get in trouble,
00:59:07 it’s gonna be really problematic.
00:59:09 So let me ask about sort of the questions of AGI
00:59:12 and human level intelligence.
00:59:14 I mean, we’ve talked about curing diseases,
00:59:18 which is sort of fundamental thing
00:59:20 we can have an impact today,
00:59:21 but AI people also dream of both understanding
00:59:26 and creating intelligence.
00:59:29 Is that something you think about?
00:59:30 Is that something you dream about?
00:59:32 Is that something you think is within our reach
00:59:36 to be thinking about as computer scientists?
00:59:39 Well, boy, let me tease apart different parts
00:59:43 of that question.
00:59:45 The worst question.
00:59:46 Yeah, it’s a multi part question.
00:59:50 So let me start with the feasibility of AGI.
00:59:57 Then I’ll talk about the timelines a little bit
01:00:01 and then talk about, well, what controls does one need
01:00:05 when thinking about protections in the AI space?
01:00:10 So, I think AGI obviously is a longstanding dream
01:00:17 that even our early pioneers in the space had,
01:00:21 the Turing test and so on
01:00:23 are the earliest discussions of that.
01:00:27 We’re obviously closer than we were 70 or so years ago,
01:00:32 but I think it’s still very far away.
01:00:37 I think machine learning algorithms today
01:00:40 are really exquisitely good pattern recognizers
01:00:46 in very specific problem domains
01:00:49 where they have seen enough training data
01:00:51 to make good predictions.
01:00:53 You take a machine learning algorithm
01:00:57 and you move it to a slightly different version
01:01:00 of even that same problem, far less one that’s different
01:01:03 and it will just completely choke.
01:01:06 So I think we’re nowhere close to the versatility
01:01:11 and flexibility of even a human toddler
01:01:15 in terms of their ability to context switch
01:01:19 and solve different problems
01:01:20 using a single knowledge base, single brain.
01:01:24 So am I desperately worried about
01:01:28 the machines taking over the universe
01:01:33 and starting to kill people
01:01:35 because they want to have more power?
01:01:37 I don’t think so.
01:01:38 Well, so to pause on that,
01:01:40 so you kind of intuited that super intelligence
01:01:43 is a very difficult thing to achieve.
01:01:46 Even intelligence.
01:01:47 Intelligence, intelligence.
01:01:48 Super intelligence, we’re not even close to intelligence.
01:01:50 Even just the greater abilities of generalization
01:01:53 of our current systems.
01:01:55 But we haven’t answered all the parts
01:01:59 and we’ll take another.
01:02:00 I’m getting to the second part.
01:02:00 Okay, but maybe another tangent you can also pick up
01:02:04 is can we get in trouble with much dumber systems?
01:02:08 Yes, and that is exactly where I was going.
01:02:11 So just to wrap up on the threats of AGI,
01:02:16 I think that it seems to me a little early today
01:02:21 to figure out protections against a human level
01:02:26 or superhuman level intelligence
01:02:28 where we don’t even see the skeleton
01:02:31 of what that would look like.
01:02:33 So it seems that it’s very speculative
01:02:35 on how to protect against that.
01:02:39 But we can definitely and have gotten into trouble
01:02:43 on much dumber systems.
01:02:45 And a lot of that has to do with the fact
01:02:48 that the systems that we’re building are increasingly
01:02:52 complex, increasingly poorly understood.
01:02:57 And there’s ripple effects that are unpredictable
01:03:01 in changing little things that can have dramatic consequences
01:03:06 on the outcome.
01:03:08 And by the way, that’s not unique to artificial intelligence.
01:03:11 I think artificial intelligence exacerbates that,
01:03:13 brings it to a new level.
01:03:15 But heck, our electric grid is really complicated.
01:03:18 The software that runs our financial markets
01:03:20 is really complicated.
01:03:22 And we’ve seen those ripple effects translate
01:03:25 to dramatic negative consequences,
01:03:28 like for instance, financial crashes that have to do
01:03:32 with feedback loops that we didn’t anticipate.
01:03:35 So I think that’s an issue that we need to be thoughtful
01:03:38 about in many places,
01:03:41 artificial intelligence being one of them.
01:03:44 And I think it’s really important that people are thinking
01:03:49 about ways in which we can have better interpretability
01:03:54 of systems, better tests for, for instance,
01:03:59 measuring the extent to which a machine learning system
01:04:01 that was trained in one set of circumstances,
01:04:04 how well does it actually work
01:04:07 in a very different set of circumstances
01:04:09 where you might say, for instance,
01:04:12 well, I’m not gonna be able to test my automated vehicle
01:04:14 in every possible city, village,
01:04:18 weather condition and so on.
01:04:20 But if you trained it on this set of conditions
01:04:23 and then tested it on 50 or a hundred others
01:04:27 that were quite different from the ones
01:04:29 that you trained it on and it worked,
01:04:31 then that gives you confidence that the next 50
01:04:34 that you didn’t test it on might also work.
01:04:36 So effectively it’s testing for generalizability.
01:04:39 So I think there’s ways that we should be
01:04:41 constantly thinking about to validate the robustness
01:04:45 of our systems.
01:04:47 I think it’s very different from the let’s make sure
01:04:50 robots don’t take over the world.
01:04:53 And then the other place where I think we have a threat,
01:04:57 which is also important for us to think about
01:04:59 is the extent to which technology can be abused.
01:05:03 So like any really powerful technology,
01:05:06 machine learning can be very much used badly
01:05:10 as well as to good.
01:05:12 And that goes back to many other technologies
01:05:15 that have come up with when people invented
01:05:19 projectile missiles and it turned into guns
01:05:22 and people invented nuclear power
01:05:24 and it turned into nuclear bombs.
01:05:26 And I think honestly, I would say that to me,
01:05:30 gene editing and CRISPR is at least as dangerous
01:05:33 as technology if used badly than as machine learning.
01:05:39 You could create really nasty viruses and such
01:05:43 using gene editing that you would be really careful about.
01:05:51 So anyway, that’s something that we need
01:05:56 to be really thoughtful about whenever we have
01:05:59 any really powerful new technology.
01:06:02 Yeah, and in the case of machine learning
01:06:04 is adversarial machine learning.
01:06:06 So all the kinds of attacks like security almost threats
01:06:09 and there’s a social engineering
01:06:10 with machine learning algorithms.
01:06:12 And there’s face recognition and big brother is watching you
01:06:15 and there’s the killer drones that can potentially go
01:06:20 and targeted execution of people in a different country.
01:06:27 One can argue that bombs are not necessarily
01:06:29 that much better, but people wanna kill someone,
01:06:34 they’ll find a way to do it.
01:06:35 So in general, if you look at trends in the data,
01:06:39 there’s less wars, there’s less violence,
01:06:41 there’s more human rights.
01:06:42 So we’ve been doing overall quite good as a human species.
01:06:48 Are you optimistic?
01:06:49 Surprisingly sometimes.
01:06:50 Are you optimistic?
01:06:52 Maybe another way to ask is do you think most people
01:06:55 are good and fundamentally we tend towards a better world,
01:07:03 which is underlying the question,
01:07:05 will machine learning with gene editing
01:07:09 ultimately land us somewhere good?
01:07:12 Are you optimistic?
01:07:15 I think by and large, I’m optimistic.
01:07:19 I think that most people mean well,
01:07:24 that doesn’t mean that most people are altruistic do gooders,
01:07:28 but I think most people mean well,
01:07:31 but I think it’s also really important for us as a society
01:07:34 to create social norms where doing good
01:07:40 and being perceived well by our peers
01:07:47 are positively correlated.
01:07:49 I mean, it’s very easy to create dysfunctional norms
01:07:54 in emotional societies.
01:07:55 There’s certainly multiple psychological experiments
01:07:58 as well as sadly real world events
01:08:02 where people have devolved to a world
01:08:05 where being perceived well by your peers
01:08:09 is correlated with really atrocious,
01:08:14 often genocidal behaviors.
01:08:17 So we really want to make sure
01:08:19 that we maintain a set of social norms
01:08:21 where people know that to be a successful member of society,
01:08:25 you want to be doing good.
01:08:27 And one of the things that I sometimes worry about
01:08:31 is that some societies don’t seem to necessarily
01:08:35 be moving in the forward direction in that regard
01:08:38 where it’s not necessarily the case
01:08:43 that being a good person
01:08:45 is what makes you be perceived well by your peers.
01:08:47 And I think that’s a really important thing
01:08:49 for us as a society to remember.
01:08:51 It’s really easy to degenerate back into a universe
01:08:55 where it’s okay to do really bad stuff
01:09:00 and still have your peers think you’re amazing.
01:09:04 It’s fun to ask a world class computer scientist
01:09:08 and engineer a ridiculously philosophical question
01:09:11 like what is the meaning of life?
01:09:13 Let me ask, what gives your life meaning?
01:09:17 Or what is the source of fulfillment, happiness,
01:09:22 joy, purpose?
01:09:26 When we were starting Coursera in the fall of 2011,
01:09:32 that was right around the time that Steve Jobs passed away.
01:09:37 And so the media was full of various famous quotes
01:09:41 that he uttered and one of them that really stuck with me
01:09:45 because it resonated with stuff that I’d been feeling
01:09:48 for even years before that is that our goal in life
01:09:52 should be to make a dent in the universe.
01:09:55 So I think that to me, what gives my life meaning
01:10:00 is that I would hope that when I am lying there
01:10:05 on my deathbed and looking at what I’d done in my life
01:10:09 that I can point to ways in which I have left the world
01:10:15 a better place than it was when I entered it.
01:10:20 This is something I tell my kids all the time
01:10:23 because I also think that the burden of that
01:10:27 is much greater for those of us who were born to privilege.
01:10:31 And in some ways I was, I mean, I wasn’t born super wealthy
01:10:34 or anything like that, but I grew up in an educated family
01:10:37 with parents who loved me and took care of me
01:10:40 and I had a chance at a great education
01:10:43 and I always had enough to eat.
01:10:46 So I was in many ways born to privilege
01:10:48 more than the vast majority of humanity.
01:10:51 And my kids I think are even more so born to privilege
01:10:55 than I was fortunate enough to be.
01:10:57 And I think it’s really important that especially
01:11:01 for those of us who have that opportunity
01:11:03 that we use our lives to make the world a better place.
01:11:07 I don’t think there’s a better way to end it.
01:11:09 Daphne, it was an honor to talk to you.
01:11:11 Thank you so much for talking today.
01:11:12 Thank you.
01:11:14 Thanks for listening to this conversation
01:11:15 with Daphne Koller and thank you
01:11:17 to our presenting sponsor, Cash App.
01:11:19 Please consider supporting the podcast
01:11:21 by downloading Cash App and using code LEXPodcast.
01:11:26 If you enjoy this podcast, subscribe on YouTube,
01:11:28 review it with five stars on Apple Podcast,
01:11:31 support it on Patreon, or simply connect with me
01:11:33 on Twitter at LEXFREEDMAN.
01:11:36 And now let me leave you with some words from Hippocrates,
01:11:39 a physician from ancient Greece
01:11:41 who’s considered to be the father of medicine.
01:11:45 Wherever the art of medicine is loved,
01:11:48 there’s also a love of humanity.
01:11:50 Thank you for listening and hope to see you next time.