WEBVTT

1
00:00:00.000 --> 00:00:05.249
FI CLASSROOM 03-14: everyone's backgrounds are. So, if you're, like, an interp person, could you raise your hand?

2
00:00:06.010 --> 00:00:07.709
FI CLASSROOM 03-14: Okay, and then biology?

3
00:00:08.200 --> 00:00:14.150
FI CLASSROOM 03-14: Okay, so I guess everyone else is more general, like, theory and ML, or… Mathematics.

4
00:00:14.470 --> 00:00:18.409
FI CLASSROOM 03-14: Oh, they're all on Zoom? Okay, that sounds good. Okay.

5
00:00:18.550 --> 00:00:25.309
FI CLASSROOM 03-14: So yeah, I guess I'll just start from a problem statement, which is really, like, why are we using AI protein folding?

6
00:00:25.410 --> 00:00:26.470
FI CLASSROOM 03-14: So

7
00:00:27.520 --> 00:00:47.049
FI CLASSROOM 03-14: Okay, so I'm just gonna… so this is, like, the drug discovery problem, where we're really interested in finding early-stage binders. So let's say I have my target, here it's HER2, which is a protein receptor that drives tumor growth for breast cancer. So our goal is to design some binder that will block the receptor signaling. So we'll…

8
00:00:47.050 --> 00:00:50.320
FI CLASSROOM 03-14: The two proteins will bind, and,

9
00:00:50.680 --> 00:01:01.440
FI CLASSROOM 03-14: we'll get a successful drug. So how do we do this? Well, something that we can do is just generate a bunch of sequences, and then we'll have to go to the wet lab and express the proteins.

10
00:01:01.480 --> 00:01:11.240
FI CLASSROOM 03-14: And then we'll have to do some binding assays. And so this is a lot of physical work in order to check whether our two proteins are actually going to bind. And in terms of cost, this is…

11
00:01:11.340 --> 00:01:22.400
FI CLASSROOM 03-14: orders of magnitude more expensive than you'd normally think. So 1,000 proteins could be $50,000 to $100,000 in weeks of time. And this is only 1,000 proteins. The search space for sequences is huge.

12
00:01:22.840 --> 00:01:24.150
FI CLASSROOM 03-14: So here's a problem.

13
00:01:24.280 --> 00:01:38.290
FI CLASSROOM 03-14: And this is where AI protein folding comes into play. So, the way I see it is, it's reducing the search space. We want a way to have cheap and fast virtual screening of our proteins before we actually go through the whole process of putting them in the wet lab.

14
00:01:38.300 --> 00:01:44.640
FI CLASSROOM 03-14: So, here we have the two amino acid sequences of our target and our potential protein binder.

15
00:01:44.670 --> 00:01:56.589
FI CLASSROOM 03-14: So, just as a side note for non-biologists, so proteins are chains of molecules called amino acids, and so we can usually represent them with letters. Here there are 20 common amino acid types.

16
00:01:56.590 --> 00:02:08.070
FI CLASSROOM 03-14: And when amino acids actually come together to form a protein, they lose atoms and are then called residues. So I'll interchange the two terms, but just for your information. So here, instead of needing to

17
00:02:08.280 --> 00:02:23.610
FI CLASSROOM 03-14: fully express them and go through the whole wet lab process. Something that we can do is put the two sequences into AlphaFold, and it'll generate what it thinks is the protein complex. Here we have a huge speedup now, where we can virtually screen our proteins in order to

18
00:02:23.830 --> 00:02:26.850
FI CLASSROOM 03-14: Reduce the amount of proteins we actually need to put into the wet lab.

19
00:02:28.200 --> 00:02:32.840
FI CLASSROOM 03-14: But there's this question of, like, why do I want to interpret this protein folding model at all?

20
00:02:32.890 --> 00:02:49.889
FI CLASSROOM 03-14: Or why do this for scientific models? So on the way here, I was reading some papers on Flatiron, and I saw this physics steering model that our brochure worked on, where they really were curious about, yeah, how does this other type of physical simulation model work? I think there are really two lessons that we can try and draw from interpretability.

21
00:02:49.960 --> 00:03:08.009
FI CLASSROOM 03-14: So the first question that we're trying to understand is, well, how does the model do these internal computations? And I think understanding this is helpful for building trust in the model, also for model debugging, and there's a lot of scientific curiosity, too. And I've seen some people do this type of work where they interpret the model, and then this helps inform maybe the inference setup.

22
00:03:08.720 --> 00:03:12.770
FI CLASSROOM 03-14: A second question is, can we, after we understand the model internals.

23
00:03:12.940 --> 00:03:20.350
FI CLASSROOM 03-14: begin to control its outputs, and even extend capabilities. So, these are the two questions that I'll be asking for protein models as a whole.

24
00:03:21.910 --> 00:03:31.480
FI CLASSROOM 03-14: So when it comes to folding an actual protein, I'm just going to walk through the setup, and we're going to start with a protein sequence. So here we're just going to fold a single structure.

25
00:03:31.590 --> 00:03:42.329
FI CLASSROOM 03-14: So we're going to pass it into this protein language model, which produces a sequence representation where each token in our sequence receives some embedding.

26
00:03:43.210 --> 00:03:48.109
FI CLASSROOM 03-14: What happens next is we take this sequence representation and pass it into something called the folding trunk.

27
00:03:48.240 --> 00:03:57.690
FI CLASSROOM 03-14: Now, at this stage, we also initialize a pairwise representation, and so this is pretty unique to protein folding. It's not a normal representation you find in transformers.

28
00:03:57.690 --> 00:04:09.930
FI CLASSROOM 03-14: But all it is, is it's meant to represent relationships between residues. And so it's just a 2D matrix, and here we just initialize it with positional embeddings. All the folding trunk does is it just updates these representations.

29
00:04:10.330 --> 00:04:21.280
FI CLASSROOM 03-14: And then we pass it into a third module called the structure module, and it just renders our 3D protein structure. So it just outputs 3D coordinates of different atoms.

30
00:04:21.690 --> 00:04:23.390
FI CLASSROOM 03-14: So, from a whole.

31
00:04:23.570 --> 00:04:30.029
FI CLASSROOM 03-14: We're looking at ESMFold, it has three modules, and it maintains these two representations, the pairwise and the sequence.

32
00:04:30.710 --> 00:04:40.940
FI CLASSROOM 03-14: Now, the question that we want to ask is, how does the model actually fold proteins? And so we choose to look at a specific case where we're looking at secondary… oh, yeah.

33
00:04:41.070 --> 00:04:44.129
FI CLASSROOM 03-14: Hi, so,

34
00:04:44.280 --> 00:04:56.429
FI CLASSROOM 03-14: Basically, you keep… every intermediate step keep… keeps having some sequence. Start with a sequence, then you have a sequence again, then you do something, sequence again, yeah? Yep.

35
00:04:56.580 --> 00:04:58.450
FI CLASSROOM 03-14: So…

36
00:04:59.690 --> 00:05:10.169
FI CLASSROOM 03-14: is just the intuition, because, like, at the end of the day, you just want to use LLMs. Like, why would you compress everything into a segments? Why don't you keep, like.

37
00:05:11.370 --> 00:05:17.899
FI CLASSROOM 03-14: some embeddings? I don't know if that makes sense. I… is the question about the…

38
00:05:18.720 --> 00:05:38.439
FI CLASSROOM 03-14: Yeah, I guess… Every step spits out a sequence. Oh, I see. So you keep compressing, you do something complicated, spit out a sequence, do something like… Oh, I see. So, instead, you could have just had some, like, sequence is compression, right? So you could, like, live in some embedding space, I don't know.

39
00:05:38.530 --> 00:05:41.780
FI CLASSROOM 03-14: Yeah. I guess in this case, so,

40
00:05:42.390 --> 00:05:45.549
FI CLASSROOM 03-14: The sequence representation here is always the embeddings, right?

41
00:05:45.770 --> 00:06:00.320
FI CLASSROOM 03-14: So we actually… yeah, yeah, yeah, sorry, sorry, so that's… that's what the, yeah, that makes sense, that makes sense. So yeah, here, when I say sequence representation, it is going to be an embedding per, token that was originally in the amino acid sequence.

42
00:06:00.990 --> 00:06:05.559
FI CLASSROOM 03-14: Okay, yeah, that's a good question. Are there any other questions about this folding process?

43
00:06:06.910 --> 00:06:10.700
FI CLASSROOM 03-14: Okay, so, yeah, we're going to ask now how…

44
00:06:10.890 --> 00:06:30.039
FI CLASSROOM 03-14: the model actually folds a specific type of protein. Here it's called the beta hairpin. So I highlighted it in purple, but basically it's just one strand connected by a loop, and then another strand. And so it's a very simple, but very, very common structure in many, many proteins. And what we're interested in understanding is, well.

45
00:06:30.540 --> 00:06:37.320
FI CLASSROOM 03-14: How does the model use the sequence representation and the pairwise representation to come up with the 3D coordinates for that hairpin?

46
00:06:38.050 --> 00:06:41.330
FI CLASSROOM 03-14: So, something that we can do as a first intuition is

47
00:06:41.580 --> 00:06:47.500
FI CLASSROOM 03-14: see, oh, are the hair prints actually present in the folding trunk representations? And this is actually…

48
00:06:47.620 --> 00:06:57.270
FI CLASSROOM 03-14: Okay, so what we can do is we take our beta hairpin, and we can throw it through the model, and we're mainly interested in the folding trunk, because that is actually the component that

49
00:06:57.510 --> 00:07:12.380
FI CLASSROOM 03-14: the folding really happens. It's actually… it shows up in many different modules, but basically, the pairwise… this is… the folding trunk is where the pairwise representation is built up, and that is actually what's used for folding later… later on. So, we're going to be looking at the representations there.

50
00:07:12.400 --> 00:07:17.839
FI CLASSROOM 03-14: Yeah, so the beta… so beta hairpin is this purple thing. Yep. I think I missed…

51
00:07:18.430 --> 00:07:21.210
FI CLASSROOM 03-14: So you're trying to see what causes these things to, like.

52
00:07:21.370 --> 00:07:33.650
FI CLASSROOM 03-14: Yeah, we're… I think right now we're just looking for, okay, are there linear representations of the hairpin that we can identify within the sequence or the pairwise representations?

53
00:07:33.810 --> 00:07:38.449
FI CLASSROOM 03-14: when you write these arrows and stuff, this is, like, you have a 3D geometry.

54
00:07:38.610 --> 00:07:56.669
FI CLASSROOM 03-14: And then some software figures out that this is a hairpin? Or is it, like, there's a tag in the model output it tells you? So in this case, it's… yeah, there's some software that just identifies patterns, because these hairpins will form structures that, like, are basically… have defined distances, so you can just look at that.

55
00:07:57.490 --> 00:08:17.470
FI CLASSROOM 03-14: But yeah, so… and when we're looking for these representations, what we're really interested in is, oh, does the model represent the hairpin in a way that we can access? And can we now take that representation and change the way that the model actually folds proteins? So, for instance, we can take a different protein that has no hairpins at all, so here it's just made up of alpha helices.

56
00:08:17.470 --> 00:08:29.689
FI CLASSROOM 03-14: And we can once again look at its sequence and the pairwise representations in the folding trunk. Now, there's this interesting question now of, well, what happens if I just take the sequence representation of my hairpin.

57
00:08:29.900 --> 00:08:35.750
FI CLASSROOM 03-14: And I overwrite the sequence representation of this helix-loop-heelix region.

58
00:08:36.270 --> 00:08:38.950
FI CLASSROOM 03-14: that's going to happen, and I only need to do this at a single block.

59
00:08:39.409 --> 00:08:44.590
FI CLASSROOM 03-14: Similarly, what happens if I just replace the pairwise representation at a single block?

60
00:08:45.130 --> 00:08:47.179
FI CLASSROOM 03-14: Well, what we observe is actually that

61
00:08:47.540 --> 00:08:59.989
FI CLASSROOM 03-14: The model produces a protein where instead of an alpha helix, it now produces hairpins. And so this is a really interesting observation, because we only need to patch one type of representation at a single block.

62
00:09:00.220 --> 00:09:02.039
FI CLASSROOM 03-14: And this… and so…

63
00:09:02.230 --> 00:09:12.770
FI CLASSROOM 03-14: what we can do now is try and understand, well, when does the patching work, and how… that, I guess, reveals information about how the model is actually performing the folding computation.

64
00:09:13.190 --> 00:09:14.429
FI CLASSROOM 03-14: Question? Yeah.

65
00:09:14.530 --> 00:09:23.179
FI CLASSROOM 03-14: You were first. Okay, so can you explain again, what is this pairwise patching? So, it's just,

66
00:09:23.640 --> 00:09:40.049
FI CLASSROOM 03-14: It's just a 2D matrix that relates, items in the sequence? Yep. That's it? Yep. And then, what is… what I also observed there is some other dimension, what is this? Oh, if per sequence… Oh, yeah, here, the multiple blocks, because… so the full-link trunk

67
00:09:40.050 --> 00:09:44.989
FI CLASSROOM 03-14: It basically has 48 layers, and they call it 48 blocks, so that's what the stacking is meant to represent.

68
00:09:45.190 --> 00:09:49.450
FI CLASSROOM 03-14: So… No, it's not general dimension.

69
00:09:50.420 --> 00:09:59.470
FI CLASSROOM 03-14: Oh, it's… that's just meant to represent, how many blocks there are, or how many layers, in the folding trim theory there are.

70
00:09:59.910 --> 00:10:12.030
FI CLASSROOM 03-14: Yeah, yeah, it's all sequential. I don't understand. Is it, like, a channel? Like, internal, kind of, like, multiple filters, and yeah, for each filter? It just runs in sequence, instead of having multiple channels.

71
00:10:12.400 --> 00:10:32.060
FI CLASSROOM 03-14: I'm talking about the one in depth. Yeah, it's like multiple layers, like, imagine… Yeah, yeah, it's like multiple layers. Your network has 10 layers, then each layer, he's extracting the pairwise, represent… the patching representation. And then the structure model looks… depends on…

72
00:10:32.120 --> 00:10:34.489
FI CLASSROOM 03-14: On the final one, yeah, sorry.

73
00:10:34.590 --> 00:10:47.329
FI CLASSROOM 03-14: Oh, on the final layer. Yeah, yeah, sorry, so after, it'll… it'll do all… it'll go through all the layers, and then just produce a final output, and then the structure module just… All of these are irrelevant. You only, in fact, need just the final layer, yeah?

74
00:10:47.330 --> 00:10:57.619
FI CLASSROOM 03-14: final sequence and final matrix. In terms of the folding process, it only looks at the final one. But we're interested in understanding, well, how does the intermediate computation happen?

75
00:10:58.030 --> 00:11:00.600
Siavash Golkar: So, can I? Yep.

76
00:11:00.860 --> 00:11:14.379
Siavash Golkar: Can I ask a quick question? I'm wondering if this task is too simple, because there are models that can figure out, like, the secondary structure, like, you can just spit out the DSSP, and then you can just look at, you know, try to parse

77
00:11:14.800 --> 00:11:24.270
Siavash Golkar: I guess, beta strand, loop, beta strand, and just try to figure it out. You will miss a lot of things, but it's fairly good as a first approximation, so I wonder…

78
00:11:24.480 --> 00:11:26.630
Siavash Golkar: What's your thoughts on that?

79
00:11:27.370 --> 00:11:33.869
FI CLASSROOM 03-14: Could you, I guess… I think I understand, where you're going at. So I guess, what is the problem that,

80
00:11:34.200 --> 00:11:42.550
Siavash Golkar: What I'm saying is that this is a fairly simple problem in terms of, like, figuring out, like, where beta hairpins are for a majority of them.

81
00:11:42.550 --> 00:11:52.469
FI CLASSROOM 03-14: Yeah, I see what you're saying. So, and I guess, like, our… the problem is less so, like, oh, I want to identify, like, yeah, when… where beta hairpins are in the protein, and more so about…

82
00:11:52.570 --> 00:11:57.020
FI CLASSROOM 03-14: Well, when I patch, when I patch a single representation.

83
00:11:57.330 --> 00:12:05.220
FI CLASSROOM 03-14: how do I… what does… how does that change the downstream computation? And so, I guess we're using… we're… it's a very simple test, setup.

84
00:12:05.410 --> 00:12:10.700
FI CLASSROOM 03-14: More for the interpretability side than for, like, trying to do some application to protein design.

85
00:12:11.440 --> 00:12:13.430
FI CLASSROOM 03-14: Does that answer the question?

86
00:12:13.690 --> 00:12:21.460
Siavash Golkar: Sure, yeah, I think that's… that's good, but I think you can also try to do that on the simpler models that just spit out, like, where these things are.

87
00:12:22.430 --> 00:12:23.010
Siavash Golkar: And.

88
00:12:23.010 --> 00:12:23.440
FI CLASSROOM 03-14: Oh.

89
00:12:24.150 --> 00:12:25.579
Siavash Golkar: But okay, no, I think that's, that's.

90
00:12:25.580 --> 00:12:27.610
FI CLASSROOM 03-14: I see, I see what you're saying, yeah, that makes sense, okay.

91
00:12:28.910 --> 00:12:30.200
FI CLASSROOM 03-14: Ask about the…

92
00:12:30.500 --> 00:12:41.020
FI CLASSROOM 03-14: So here you're… you have a bunch of data… a bunch of sequences with the hairpin and a bunch of… with the alpha helix? Yeah. Or… and these are potentially very different. Yes. And then you take an average vector.

93
00:12:41.160 --> 00:12:48.800
FI CLASSROOM 03-14: Like, how do you do the patching? I see, yeah, so it'll just be… I'll have a single alpha helix, and I'll just sample

94
00:12:49.220 --> 00:13:06.860
FI CLASSROOM 03-14: the sequence representation from a single hairpin. Yeah, from a single one. Yeah, I test over many of them. So we collect a bunch of… a bunch of these alpha helices, and then for a given alpha helix, I just find some random hairpin, and I just patch that representation in. Yes, yes.

95
00:13:08.490 --> 00:13:17.389
FI CLASSROOM 03-14: Yeah, any other questions? Just a question, please. So, when you do the sequence patching, it's only on the last layer of the folding trunk? So…

96
00:13:17.550 --> 00:13:36.760
FI CLASSROOM 03-14: Yeah, that's a good question. Could you, like, do it, like, in one layer in the middle, and then it starts, like, it understands that you change something… Right, right. That's the, that's actually the whole reason why we're doing this type of patching, is because we're interested in understanding, like, at which layers are these representations useful?

97
00:13:37.500 --> 00:13:38.240
FI CLASSROOM 03-14: Yeah.

98
00:13:40.330 --> 00:13:47.819
FI CLASSROOM 03-14: Is there pillow as metrics actually, like, encoding some kind of, like, chemical or physical interaction between different, components in the sequence?

99
00:13:47.820 --> 00:14:07.219
FI CLASSROOM 03-14: What was the first part? You were asking about a metric? I'm asking about, like, whether the pairwise metrics, that is the output of this, like, folding trunk, whether it encodes, like, some kind of, like, physical or chemical interaction between, like, different components, etc. Yeah, I think that's a great question. I think it does, and, like, that's something that we look at here as well.

100
00:14:07.440 --> 00:14:22.419
FI CLASSROOM 03-14: sort of maybe a follow-up question. So, like, is it, like, the intervention that causes the model to suddenly, like, output a new structure, is it kind of, like, chemically, like, fateful in the sense that if you see this kind of, like, interaction that you expect to see this kind of structure?

101
00:14:22.550 --> 00:14:38.220
FI CLASSROOM 03-14: Oh, I see. Yeah, and that's something that we end up looking at downstream, but we see some… we see evidence of, like, yeah, that if we do some chemical intervention, then we'll see faithful downstream outputs, even though… although the outputs sometimes are not, like.

102
00:14:38.790 --> 00:14:41.450
FI CLASSROOM 03-14: Perfectly realistic. Yeah, let's go ahead.

103
00:14:42.220 --> 00:14:43.020
FI CLASSROOM 03-14: Okay.

104
00:14:43.260 --> 00:14:48.160
FI CLASSROOM 03-14: Yeah, so what we end up doing is we get a bunch of these alpha helices, and then we just…

105
00:14:48.310 --> 00:14:57.420
FI CLASSROOM 03-14: For a given alpha helix, I'll just find some random hairpin, patch the representation, add a single block, and then we're… and then we'll just see if the output has a hairpin.

106
00:14:57.860 --> 00:15:01.030
FI CLASSROOM 03-14: So, this was actually… what happens is…

107
00:15:01.950 --> 00:15:12.039
FI CLASSROOM 03-14: we notice that sequence patching only works in the first 7 blocks. And then we see a second success regime where pairwise patching begins to work only in the second half of the model.

108
00:15:12.280 --> 00:15:28.200
FI CLASSROOM 03-14: And this was actually, like, the critical observation that informed, like, the rest of our experiments. Because we see such a clear two-stage setup, it kind of helped us develop some hypotheses about maybe what's happening within the model. So, when we saw this.

109
00:15:28.600 --> 00:15:45.940
FI CLASSROOM 03-14: We thought, okay, well, maybe what's happening in this first stage is the model is taking information from the sequence, and then it's using that to update the pairwise representation so that it has, like, a… it has a faithful chemical representation of everything that's happening in the protein.

110
00:15:46.340 --> 00:15:51.830
FI CLASSROOM 03-14: And then maybe what's happening in the second stage is, okay, this is where the model is really beginning to decide

111
00:15:51.880 --> 00:16:06.639
FI CLASSROOM 03-14: the spatial features of my protein, and, like, really just understand what the structure should look like. And so this kind of makes sense because, okay, if stage 2 is where the spatial features are determined, then if I just patch the spatial representation of my hairpin.

112
00:16:06.950 --> 00:16:11.259
FI CLASSROOM 03-14: at this part, at this stage, then I'm just injecting the structure into my protein.

113
00:16:13.060 --> 00:16:17.899
FI CLASSROOM 03-14: So, once again, I'm just gonna look at the model as a whole, and…

114
00:16:18.270 --> 00:16:28.750
FI CLASSROOM 03-14: Specifically, we're looking at this folding trunk, and I think something that's important to understand is how the folding trunk operates. So, the folding trunk actually is composed of 48 of these blocks.

115
00:16:28.750 --> 00:16:39.690
FI CLASSROOM 03-14: And so they happen in sequence. And what's most important to understand is, well, one, all that's happening in the folding trunk is we perform self-attention over the sequence representation.

116
00:16:39.690 --> 00:16:56.669
FI CLASSROOM 03-14: And then something called the triangular update over the pairwise representation. And so that's just, like, we're doing attention over the rows and the columns, and there's some other operations. But the most important piece is to understand that the pairwise and the sequence representations communicate with each other during the folding process. And something that

117
00:16:56.900 --> 00:17:05.460
FI CLASSROOM 03-14: And so that kind of informs why we think, okay, well, the sequence representation is probably talking to the pairwise in the early blocks, and then something else could be happening later.

118
00:17:09.470 --> 00:17:12.380
FI CLASSROOM 03-14: Well, no, it's asymmetric, yeah.

119
00:17:15.540 --> 00:17:17.019
FI CLASSROOM 03-14: Why is it asymmetric?

120
00:17:17.560 --> 00:17:24.130
FI CLASSROOM 03-14: Yeah, I mean, it just tells you about some of the slides, but… Oh, I see.

121
00:17:24.849 --> 00:17:30.370
FI CLASSROOM 03-14: Well, I guess, yeah, similarity is, like, maybe… Yeah, I…

122
00:17:31.060 --> 00:17:41.549
FI CLASSROOM 03-14: So, there's no, like, enforcing of symmetric… like, of having a symmetric 2D matrix here. I think, like, that's probably just a design choice from the people who made the model.

123
00:17:41.860 --> 00:17:46.840
FI CLASSROOM 03-14: Well, you can just… are you asking about symmetriness of the pair representation?

124
00:17:47.490 --> 00:17:53.900
FI CLASSROOM 03-14: then I can just imply some order, right? So IJ does not need to be equal to JI.

125
00:17:54.740 --> 00:17:57.659
FI CLASSROOM 03-14: He just instantly doesn't make sense to have special education.

126
00:17:58.620 --> 00:18:01.999
FI CLASSROOM 03-14: Yeah, and I guess maybe… yeah.

127
00:18:02.350 --> 00:18:07.079
FI CLASSROOM 03-14: The order matters, right? It's a sequence. Yeah.

128
00:18:07.870 --> 00:18:13.109
FI CLASSROOM 03-14: I mean, in terms of encoding the sequence of the… the input

129
00:18:15.190 --> 00:18:29.930
FI CLASSROOM 03-14: Yeah, that's a really interesting point. We also looked at that, and there's a lot of choices in the model that are somewhat ad hoc, but that's, like, something that shows up. Quick question. So, how is the triangular update defined? It's…

130
00:18:30.670 --> 00:18:39.539
FI CLASSROOM 03-14: So, the triangular update, like, the main motivation there is… They, they perform attention operations.

131
00:18:39.850 --> 00:18:41.020
FI CLASSROOM 03-14: Yeah, over pairs.

132
00:18:41.150 --> 00:18:52.389
FI CLASSROOM 03-14: So, let's see, so I will compute the attention spore between, say, IJ and LK, but… Yeah, I think it's… they'll do… yeah…

133
00:18:52.690 --> 00:19:03.160
FI CLASSROOM 03-14: It's saying triangular. Yeah, it's triangular because, like, any update that is performed to, like, IJ and then JK, they'll take into account for the update to KI.

134
00:19:03.440 --> 00:19:08.369
FI CLASSROOM 03-14: Or IK. It's like… yeah, yeah. They consider triplins, yeah.

135
00:19:08.760 --> 00:19:12.709
FI CLASSROOM 03-14: So then it will produce a… Yeah, 3D, yeah.

136
00:19:14.460 --> 00:19:21.360
FI CLASSROOM 03-14: Yeah, but for this case, we don't look at the triangular updates or self-attention that much, so… but that's just, like, an interesting thing to understand.

137
00:19:22.210 --> 00:19:33.589
FI CLASSROOM 03-14: Okay, so now we're going to start looking at the mechanisms. So here we suggested that in the early blocks, there's likely information… biochemical signal flow from the sequence into the pairwise representation.

138
00:19:33.640 --> 00:19:49.719
FI CLASSROOM 03-14: And when we look at the folding block, well, this is definitely happening through this seek-to-pair update. And all that is, is really, we take our sequence representation, we basically perform an outer product dish operation so that it becomes a 2D matrix, and then we just add that to the pairwise representation.

139
00:19:50.300 --> 00:19:55.379
FI CLASSROOM 03-14: So, there is this question of, like, well, what type of biochemical information do we expect is flowing?

140
00:19:55.380 --> 00:20:12.689
FI CLASSROOM 03-14: And when it comes to protein folding, something that we'd expect is, like, the electrostatic charge, because if I have one residue that's positively charged, another residue that's negatively charged, and they're close together, they'll probably come and reduce the distance between them. Or we'll have repulsion between two residues that are of the same charge.

141
00:20:12.690 --> 00:20:22.479
FI CLASSROOM 03-14: So here, when it comes to amino acids, there's 3 canonically positively charged amino acids and 2 negatively charged ones. So, just as, like, a first observation, well.

142
00:20:22.480 --> 00:20:34.699
FI CLASSROOM 03-14: we can probably just look for this in terms of using linear probing. So what that really means is we just train a linear classifier on the seek-to-pair representation and the pairwise representation, just to see, well, can we actually

143
00:20:34.780 --> 00:20:37.020
FI CLASSROOM 03-14: extract charge information.

144
00:20:37.260 --> 00:20:44.800
FI CLASSROOM 03-14: For pairs of residues. So all we find, really, is that, yeah, in the seek… basically, the seek-to-pair update

145
00:20:44.940 --> 00:20:51.380
FI CLASSROOM 03-14: includes information about the charge, and what's most interesting is that plot on the right, where we have

146
00:20:51.530 --> 00:21:06.690
FI CLASSROOM 03-14: The test accuracy jumps the most after the first 7 blocks, which is also when we found the sequence patching to work. And so this is just giving us some more motivation to think, okay, charges would probably be something that is propagating from the sequence into the pairwise representation in this early stage.

147
00:21:07.270 --> 00:21:08.810
FI CLASSROOM 03-14: And this is to be expected.

148
00:21:09.040 --> 00:21:17.239
FI CLASSROOM 03-14: But what we're more interested in is, well, let's look for linear representations of charge in our latent space, and then can we use that to steer the model outputs?

149
00:21:17.240 --> 00:21:35.240
FI CLASSROOM 03-14: So, here's something that we can do. Well, we have this sequence, and we can put it into our model, and we can just extract the sequence representations of positively charged and negatively charged residues. And what we can end up doing is we can look at the mean representation of all of my positively charged

150
00:21:35.240 --> 00:21:52.690
FI CLASSROOM 03-14: and the mean representation of all my negatively charged residues in latent space. And if we just draw a vector between them, well, this is just a hyperplane that separates the two concepts. And what we would expect is, well, now if I take a new protein, and I apply this vector to the sequence representations.

151
00:21:52.730 --> 00:21:53.790
FI CLASSROOM 03-14: then…

152
00:21:54.050 --> 00:22:05.779
FI CLASSROOM 03-14: If I add it, it'll, like, it'll make the residue look more positively charged, and if I subtract it, it'll make the residue look negatively charged. So, now we can… now, the question is, does the model actually use charge? So…

153
00:22:05.860 --> 00:22:17.830
FI CLASSROOM 03-14: Something that we want to do in mechanistic interpretability is always confirm that our representations are causally meaningful, so that the model actually uses… looks at this representation. So here what we can do is we can look at

154
00:22:17.960 --> 00:22:21.219
FI CLASSROOM 03-14: Helix-loop helix motifs in our proteins.

155
00:22:21.820 --> 00:22:40.530
FI CLASSROOM 03-14: And we can just make all the residues in this first helix positive, and all the residues in the second helix negative. And so what we would expect is that if this is a charge representation, and if the model actually uses this, then the two helices should come close together and form what looks more like a beta hairpin.

156
00:22:41.310 --> 00:22:42.400
FI CLASSROOM 03-14: And so…

157
00:22:42.770 --> 00:22:50.540
FI CLASSROOM 03-14: And the way that we can measure this is just by looking to see if the two strands form what's called hydrogen bonds, which are those bonds… those bars in purple.

158
00:22:51.730 --> 00:22:57.399
FI CLASSROOM 03-14: So this is question, and now we see… we apply the intervention, and we find that the success rate

159
00:22:57.940 --> 00:23:06.139
FI CLASSROOM 03-14: peaks within the early blocks, and we apply the intervention across with a sliding window, but this kind of matches our expectation. So if we apply this

160
00:23:06.740 --> 00:23:10.919
FI CLASSROOM 03-14: Vector to sequence representations, In the early blocks.

161
00:23:11.450 --> 00:23:26.010
FI CLASSROOM 03-14: it looks… it produces outputs that look like a hairpin. But something that we can also do is sort of an opposite experiment. So what happens if I take a beta hairpin, and then I make both of the strands look like they're positively charged? Well, we would expect that the two strands would begin to repel.

162
00:23:26.250 --> 00:23:37.060
FI CLASSROOM 03-14: And so what we find is that this is actually the case. So here's a plot where the red is making both of them negative, and we find that if we apply the same vector and make them… make residues look different.

163
00:23:37.170 --> 00:23:42.530
FI CLASSROOM 03-14: both strands look negative, the two strands will repel, as we hope to see. And so I think the most…

164
00:23:43.150 --> 00:23:50.370
FI CLASSROOM 03-14: The important piece is that this effect is focused in the early blocks, which aligns with what we propose the early block mechanism would work.

165
00:23:50.850 --> 00:24:04.920
FI CLASSROOM 03-14: And, yeah. So I don't know about this hydrogen bonding, so is that something… just geometrically, you can measure if there's hydrogen bonding? Yeah, yeah. So it's like, there's a hydrogen atom in the middle or something? There… the way that it's, commonly measured here is by taking

166
00:24:05.280 --> 00:24:16.669
FI CLASSROOM 03-14: Is by measuring the distance between the nitrogen and the oxygen atom, which is, like, which are always present in all of the… in all amino acids, and then that's where the hydrogen bond will flow.

167
00:24:17.820 --> 00:24:29.320
FI CLASSROOM 03-14: Yeah. Can you compel the plot with what you would expect to see based on some, like, simulation? Yeah, that's a good question. We thought about doing that. I guess the…

168
00:24:30.730 --> 00:24:35.570
FI CLASSROOM 03-14: We didn't end up going down that route, mainly because we weren't sure, like, how to…

169
00:24:36.410 --> 00:24:39.310
FI CLASSROOM 03-14: calibrate that in terms of, like, oh, I don't know what…

170
00:24:39.750 --> 00:24:51.239
FI CLASSROOM 03-14: as I steer, like, what the charge, like, value will look like, but that was something that we thought about doing, because, well, let's say we find that this plot correlates very well with a standard

171
00:24:51.570 --> 00:25:00.209
FI CLASSROOM 03-14: like, what the physics simulation suggests, then that's suggesting that, well, the model not only has learned charge, but also the physical properties of charge. So yeah, that's a good question.

172
00:25:00.760 --> 00:25:11.420
Minhuan Li: I have a question here, like, I remember, like, all the folding modes actually does not really include any hydrogen atoms, right? And, like, what do you mean by hydrogen bond here? You only have.

173
00:25:11.420 --> 00:25:11.780
FI CLASSROOM 03-14: Yeah.

174
00:25:11.780 --> 00:25:13.220
Minhuan Li: I'm seeing the models, right?

175
00:25:13.750 --> 00:25:18.379
FI CLASSROOM 03-14: Yeah, yeah, that's a good point. So, the… I guess the hydrogen bonding is…

176
00:25:18.890 --> 00:25:25.550
FI CLASSROOM 03-14: And, like, the way that we measure it is not necessarily, like, oh, we look for hydrogens here. And more so just, like, that canonically.

177
00:25:25.970 --> 00:25:26.810
FI CLASSROOM 03-14: If…

178
00:25:26.940 --> 00:25:35.160
FI CLASSROOM 03-14: If our protein… okay, I'm just gonna skip that, sorry. So if the protein ends up making a hydrogen bond, then

179
00:25:35.270 --> 00:25:36.540
FI CLASSROOM 03-14: the distance…

180
00:25:36.760 --> 00:25:43.929
FI CLASSROOM 03-14: Between the two… between the nitrogen atom of one residue and the oxygen atom of the other residue is…

181
00:25:44.010 --> 00:25:56.789
FI CLASSROOM 03-14: going to be, like, 3.5 angstroms or less. So that's just, like, the way we measure it. But I'm also going to be honest and say, like, I'm not super… like, the biology and, like, the chemistry behind it is not my strongest suit. But yeah.

182
00:25:56.790 --> 00:26:05.219
Minhuan Li: I see. But I guess, like, my major question is about, like, during the whole training of the models, like, the model has not really accessed any of the hydrogen bond information during.

183
00:26:05.220 --> 00:26:05.700
FI CLASSROOM 03-14: Oh, my God.

184
00:26:05.700 --> 00:26:17.129
Minhuan Li: How do you imagine, like, to learn something about the hydrogen bonding stuff? Because, like, hydrogen bonding is not only about distance, it's all about, like, the orientation of the bonds, right? It's actually very, like,

185
00:26:17.410 --> 00:26:19.640
Minhuan Li: Like, angle-related stuff.

186
00:26:19.980 --> 00:26:20.530
FI CLASSROOM 03-14: Yeah.

187
00:26:20.530 --> 00:26:26.189
Minhuan Li: Do you think I could learn this kind of information from only Abby Adams from the… from training datasets, or…

188
00:26:26.300 --> 00:26:31.669
FI CLASSROOM 03-14: Yeah, I mean, I guess my… my intuition was always, like, yeah, we're… even though we're only looking at

189
00:26:32.090 --> 00:26:35.979
FI CLASSROOM 03-14: like, these PDB structures, which are pretty coarse.

190
00:26:36.440 --> 00:26:50.289
FI CLASSROOM 03-14: I guess, like, the… there's… the pattern is, like, implicit within all the structures, and so the model just picks up on things like this. Yeah, it's pretty… I guess it's, like, a note to how powerful these deep learning approaches can be.

191
00:26:51.760 --> 00:26:52.740
Minhuan Li: Okay, thanks.

192
00:26:54.640 --> 00:27:00.860
FI CLASSROOM 03-14: What's the final representation of the geometry? Is it, like, angles? It's…

193
00:27:00.980 --> 00:27:06.530
FI CLASSROOM 03-14: It's like XYZ coordinates of, all the atoms in the,

194
00:27:07.240 --> 00:27:16.860
FI CLASSROOM 03-14: And then, maybe I missed this, but the… in each layer, the representation's kind of the same, right? Like, it's the same format, you have the… so can you…

195
00:27:17.180 --> 00:27:33.109
FI CLASSROOM 03-14: just, like, dump out the coordinates in the middle of the… Yeah, yeah. So we've… the intermediate decoding is something that we've looked at. Do you have a follow-up question for that, or is it… oh, yeah. But something that we do notice is, like, sometimes it's out of distribution. Even… even if we, like, do some…

196
00:27:33.110 --> 00:27:41.129
FI CLASSROOM 03-14: clever, like, layer norming. The structure renderer usually only works in, like, the second half of the model, but… yep.

197
00:27:43.850 --> 00:27:45.479
FI CLASSROOM 03-14: Just plug it into the middle, and it'll work

198
00:27:45.880 --> 00:27:50.260
FI CLASSROOM 03-14: Yeah, it'll, like, produce… sometimes it'll produce, like, Decent structures, yeah.

199
00:27:50.490 --> 00:28:10.409
FI CLASSROOM 03-14: And that's something that we also… that's… we look at something very similar, for the next experiment. But yeah, here's just, like, some more verification of the mechanism. So this was something that we… just, like, we'd hope to see, oh, if we apply this intervention to make all the residues in strand 1 positive, then hopefully my linear probe agrees with this, and so that's the case that we see here.

200
00:28:10.600 --> 00:28:17.709
FI CLASSROOM 03-14: And something else that we also checked was, well, what if it's… what if we just apply the intervention to the sequence, and then we don't let

201
00:28:17.810 --> 00:28:28.940
FI CLASSROOM 03-14: the sequence talk to the pairwise representation. Like, if the output still follows what… still, like, forms hairpin-like structures, then this mechanism is not what we expect.

202
00:28:28.940 --> 00:28:38.219
FI CLASSROOM 03-14: But we actually see, okay, in the early blocks, if we just don't prevent… if we prevent the sequence from talking to the pairwise representation, then we don't see the expected hairpin formation.

203
00:28:39.260 --> 00:28:49.719
FI CLASSROOM 03-14: But this is just going to be a visualization of this charge mechanism now. So here we have a toy protein, which is just a single hairpin, and then I'm going to apply the intervention

204
00:28:49.760 --> 00:29:06.619
FI CLASSROOM 03-14: from block 0 to 15, and we're just going to slowly increase the magnitude of the vector that we're adding. And so you can see that, like, as I make both the strands more positive, they begin to repel, and the distance between them increases. So it's like a pretty nice visualization of what's happening within our proteins.

205
00:29:08.140 --> 00:29:22.190
FI CLASSROOM 03-14: Sorry, so each arrow is composed of multiple atoms? Like, in this case, 4? Yeah, that's, 4 residues, and then each residue has, like, maybe 15 to 20 atoms in it, but yeah.

206
00:29:22.210 --> 00:29:28.769
FI CLASSROOM 03-14: And you just visualize them as arrows? Yeah, that's like the canonical, like,

207
00:29:28.830 --> 00:29:34.339
FI CLASSROOM 03-14: This is called a cartoon representation, but yeah, it's helpful to see this as opposed to, like, the actual atoms themselves.

208
00:29:36.330 --> 00:29:41.079
FI CLASSROOM 03-14: And I guess, like, technically, because the atoms are, like, fields, so, like, it would be blobby and weird, but yeah.

209
00:29:42.550 --> 00:29:44.860
FI CLASSROOM 03-14: This is only for beta? Beta?

210
00:29:45.010 --> 00:30:01.150
FI CLASSROOM 03-14: Like, when you have arrows, you already have this beta thing, or it could also be alpha? Yeah, I guess in… yeah, you would just assume, like, this is, beta content, and that's when the arrows show up. Yeah. So it's like, you're already taking something that would fold together, and then you're trying to…

211
00:30:01.430 --> 00:30:07.879
FI CLASSROOM 03-14: break it apart, but it will not change the nature. Yeah, I think technically this visualization is, like,

212
00:30:08.160 --> 00:30:11.120
FI CLASSROOM 03-14: Not entirely accurate, but…

213
00:30:11.530 --> 00:30:17.129
FI CLASSROOM 03-14: Yeah, it's like a, like… it actually shouldn't be arrows anymore, but it's just there so you can see it.

214
00:30:18.430 --> 00:30:19.290
FI CLASSROOM 03-14: Okay.

215
00:30:19.550 --> 00:30:22.140
FI CLASSROOM 03-14: So yeah, we have this mechanism now.

216
00:30:22.610 --> 00:30:35.090
FI CLASSROOM 03-14: Now, something else that we looked at, well, if we looked at this first stage, then what's really going on in the second stage? And so, here we're assuming… we're hypothesizing that this is where the structural decisions are being made. So…

217
00:30:35.200 --> 00:30:43.290
FI CLASSROOM 03-14: Something that we can do, as suggested, like, the intermediate decoding, is, well, then we can just basically say, well, what if we just look for

218
00:30:43.640 --> 00:30:51.890
FI CLASSROOM 03-14: distance… pairwise distances in the pairwise representation. And so we just train a linear probe, where it's just a simple linear regression, and what we find is

219
00:30:52.300 --> 00:30:56.419
FI CLASSROOM 03-14: Sort of what we expect is that, while the accuracy of this probe increases.

220
00:30:56.690 --> 00:31:02.609
FI CLASSROOM 03-14: As we go down the layers and the blocks of the, of the folding trunk,

221
00:31:03.080 --> 00:31:19.960
FI CLASSROOM 03-14: Which is definitely what we would expect. But the question that I'm more interested in is, well, if I perform an intervention with this, does this help us understand when the structure is decided? Yeah. I'm sorry, what's D heightens? Oh, sorry. So, this is the distance between residue I and J.

222
00:31:20.110 --> 00:31:24.580
FI CLASSROOM 03-14: And then we… all we take is the pairwise representation between I and J. Yeah.

223
00:31:25.030 --> 00:31:28.530
FI CLASSROOM 03-14: Just to further clarify, so for each…

224
00:31:28.640 --> 00:31:37.839
FI CLASSROOM 03-14: block. You're training a different linear code, and so it's like, assuming you're training the best one, and then you're comparing which one gives you the best tests or scaring.

225
00:31:40.120 --> 00:31:51.189
Minhuan Li: I have a question here, like, because basically, like, the pair representation is not symmetric, as you just mentioned, right? But D matrix is actually symmetric. So how do you, like, do that here?

226
00:31:51.290 --> 00:31:55.289
FI CLASSROOM 03-14: Yeah, that's a pretty good question. I mean, here we just take…

227
00:31:55.690 --> 00:32:01.309
FI CLASSROOM 03-14: Well, I guess, is the question, like, how do we account to make sure, like, there's no… Yeah.

228
00:32:01.530 --> 00:32:02.640
Minhuan Li: I… I would say…

229
00:32:02.950 --> 00:32:16.700
Minhuan Li: because, like, in your Pell representation matrix, like, ZIJ does not equal to ZJI, right? Like, in the real distance matrix, DIJ actually have to equal to DJI, because, like, distance matrix is kind of symmetric. So how do you do that in practice?

230
00:32:17.140 --> 00:32:25.170
FI CLASSROOM 03-14: I guess we… we didn't take anything special into account, like, I'm not sure what the.

231
00:32:27.690 --> 00:32:40.249
Minhuan Li: I guess, like, my general question would be, like, because, like, to compute the distance DIJ, you have to, Z, you can use, you can use DIJ or use DJI, because, like, at the end, they should correspond to the same…

232
00:32:40.440 --> 00:32:42.810
Minhuan Li: Do it the average, or you choose.

233
00:32:42.810 --> 00:32:49.890
FI CLASSROOM 03-14: Yeah, I think we… Oh, oh, I see, in terms of, like, the performance? Like, we… yeah, I think we just take the average.

234
00:32:50.400 --> 00:32:57.689
FI CLASSROOM 03-14: Oh, you take the average of ZIJ? Oh, so we… the probe take… we'll, like, we'll have one case where the probe will take ZIJ, and then…

235
00:32:58.780 --> 00:33:15.910
FI CLASSROOM 03-14: Oh, I actually remember now, sorry. Both directions encode the same distance. Yeah, both… But they might have other asymmetric stuff, but… Yeah, I think in this case, so what we actually end up doing is we only… we basically just look at, like, I think the lower triangle here, where it's… we'll only pass in, like, ZIJ, and then we just ignore ZJI.

236
00:33:17.160 --> 00:33:18.030
FI CLASSROOM 03-14: Yep.

237
00:33:19.100 --> 00:33:21.300
FI CLASSROOM 03-14: Okay. Maybe the other, the other site doesn't have to…

238
00:33:21.770 --> 00:33:27.770
FI CLASSROOM 03-14: I mean, we could just, like, redo it and check. I guess I'm pretty sure, like, it would just follow the same pattern, yeah.

239
00:33:28.400 --> 00:33:29.860
FI CLASSROOM 03-14: Okay.

240
00:33:30.710 --> 00:33:37.269
FI CLASSROOM 03-14: So, yeah, now we're curious, like, when does the model decide structure? And the way that we could do this is…

241
00:33:37.990 --> 00:33:43.039
FI CLASSROOM 03-14: Well, we have this probe, which predicts pairwise distances, and so now we can look at

242
00:33:43.280 --> 00:33:59.829
FI CLASSROOM 03-14: once again, the alpha helix… the helix-loop helix motif that shows up in a lot of our target proteins, and we can just say, like, well, what if we just use the probe to decrease the distance between the two proteins? So here, on the left, we're visualizing the cross-strand distances, so basically, like.

243
00:33:59.830 --> 00:34:06.239
FI CLASSROOM 03-14: EG is the distance between, like, atom E and atom G. And so here we can just see, like, okay, they're all pretty far apart.

244
00:34:06.540 --> 00:34:24.310
FI CLASSROOM 03-14: Here's, like, 16 angstroms. So the goal… the question is, well, can we just, like, force the two strands to come together? And here we can just use… essentially, it's gradient steering, but because our model is a linear arrestor, then the gradient is just the weight space. So we're just going to just subtract the weights from my pairwise representation.

245
00:34:25.400 --> 00:34:34.660
FI CLASSROOM 03-14: And the goal is to sort of force them closer together to form sort of, like, a hairpin-like structure again. So… and we can see, like, now the… now EG

246
00:34:35.030 --> 00:34:39.909
FI CLASSROOM 03-14: it's supposed to become a lot closer together, and so that's why it's dark here. So that's what we're expecting to see.

247
00:34:40.060 --> 00:34:46.039
FI CLASSROOM 03-14: And once again, we measure the hairpinness, using the hydrogen bonds again. Yeah.

248
00:34:46.469 --> 00:34:52.939
FI CLASSROOM 03-14: I'm not sure, like, I understood what you just did, like, could you re-explain, please? Yeah, so…

249
00:34:53.600 --> 00:35:02.579
FI CLASSROOM 03-14: we're going to do an intervention on the pairwise representation, and we just trained linear probes to predict distance from that pairwise space again. So…

250
00:35:03.340 --> 00:35:21.130
FI CLASSROOM 03-14: now what we want to do is… our goal is to decrease the distance, and then eventually we want to see, like, okay, we're going to apply this with a sliding window, so we want to see, like, when does this intervention occur, like, most effectively? That's, like, the end goal. So, in order to apply, like, the actual intervention, what we're going to do is…

251
00:35:21.910 --> 00:35:27.750
FI CLASSROOM 03-14: during the forward pass, all I'm gonna do is, for any… for any given ZIJ, for, like, these.

252
00:35:27.990 --> 00:35:37.519
FI CLASSROOM 03-14: like, any given representation where it's, like, relating residues in the first helix to the… to residues in the second helix, I'm just going to subtract

253
00:35:37.600 --> 00:35:50.370
FI CLASSROOM 03-14: the weight of the linear probe from that representation. And, like, in theory, that should just decrease the distances, in the same way that, like, gradient steering would work. And so, does that make sense?

254
00:35:50.500 --> 00:35:52.849
FI CLASSROOM 03-14: Okay, and so we just… we…

255
00:35:55.310 --> 00:35:59.610
FI CLASSROOM 03-14: Right, and so I guess, like, we've been thinking about how to…

256
00:35:59.790 --> 00:36:03.830
FI CLASSROOM 03-14: like, we've been thinking about how does the model form hairpins? And…

257
00:36:04.050 --> 00:36:17.849
FI CLASSROOM 03-14: So I guess here, like, the previous experiment with the charge, we were trying to force the model to make a hairpin by just, like, editing the chemical information, right? And so here, we're just going to physically just push the distances together to see if we can make a hairpin again.

258
00:36:22.150 --> 00:36:25.960
FI CLASSROOM 03-14: Do you know anything about how the charge stuff affects this doubly direction?

259
00:36:27.230 --> 00:36:35.010
FI CLASSROOM 03-14: Maybe that's more like this circuit… Oh, I see, yeah, we didn't look at the charge. But yeah, that's… I think that's an interesting circuit to look for, which is, like.

260
00:36:35.090 --> 00:36:51.249
FI CLASSROOM 03-14: you… what are… what are heads that are attending to information that's related to charge? Or, like, if I edit the charge of one residue, will a head begin to attend to it more now? Or does it change the activate… like, the embeddings in this W direction? Yeah, oh, that's… I see what you're saying, yeah, that would be pretty interesting to look for.

261
00:36:51.360 --> 00:36:54.409
FI CLASSROOM 03-14: But yeah, so we applied this intervention.

262
00:36:54.910 --> 00:36:56.940
FI CLASSROOM 03-14: And… Following…

263
00:36:57.100 --> 00:37:11.919
FI CLASSROOM 03-14: the pattern, we noticed that, like, this intervention, when we apply it with a window of 15, is most effective, once again, in, like, the second stage, actually right where the pairwise patching was occurring. And so this is, yet again, another case where, okay, it's very likely, then, that this is where

264
00:37:11.930 --> 00:37:21.059
FI CLASSROOM 03-14: the structure is being decided, at least in the pairwise space. And so, if we edit the structure in this window, then the output structure is going to follow what we hoped for.

265
00:37:23.040 --> 00:37:24.450
FI CLASSROOM 03-14: So, yeah.

266
00:37:25.000 --> 00:37:28.719
FI CLASSROOM 03-14: And so this is just, like, a visualization of what that,

267
00:37:29.410 --> 00:37:32.299
FI CLASSROOM 03-14: what that process sort of looks like. So here, we're going to…

268
00:37:32.670 --> 00:37:42.610
FI CLASSROOM 03-14: take the alpha helix, and then I'm just going to force the residues to be closer together. And it doesn't actually form, like, the arrows, but it just, like, looks sort of hairpinny. So that's, like, the…

269
00:37:42.990 --> 00:37:55.430
FI CLASSROOM 03-14: That was the goal. So… and you can look at the… those are the cross-strand distances, so we're hoping to see dark diagonals, which will represent the cross-trend distances decreasing. So that's what we expect, and that's what we end up seeing.

270
00:37:57.020 --> 00:37:57.920
FI CLASSROOM 03-14: Yeah.

271
00:37:58.090 --> 00:38:03.490
FI CLASSROOM 03-14: Okay, so… Yeah, just like as… in… sort of like a…

272
00:38:03.820 --> 00:38:16.809
FI CLASSROOM 03-14: high-level overview of what we just looked at. So, we started out with trying to ask, like, when and where is hairpin information computed? And so what we… what we saw is, yeah, in the early blocks, there's some sort of

273
00:38:16.820 --> 00:38:31.680
FI CLASSROOM 03-14: hairpin information in the sequence, and in the linear box, we see that there's spatial features. And then we took this step of, well, can we actually confirm that this is the mechanism? And so, we looked at mechanism one, which is a biochemical signal flow, and we find that it aligns with

274
00:38:31.790 --> 00:38:33.400
FI CLASSROOM 03-14: Our sequence patching results.

275
00:38:33.620 --> 00:38:50.580
FI CLASSROOM 03-14: And then we look at mechanism two, which was structural, spatial feature or structural decisions, and that, once again, lines up with our pairwise patching results. So I think it's pretty, pretty, yeah. So can you explain again the first picture?

276
00:38:50.760 --> 00:38:59.629
FI CLASSROOM 03-14: So, basically, when you intervene, At this sequence representation,

277
00:38:59.810 --> 00:39:06.739
FI CLASSROOM 03-14: Everywhere, like, after level 10, then, you don't get the hairpins.

278
00:39:07.460 --> 00:39:10.599
FI CLASSROOM 03-14: And then, how did you jump into biochemical signals?

279
00:39:11.130 --> 00:39:20.850
FI CLASSROOM 03-14: Oh, I see. You're saying, like, when we saw this, like, why did we just assume that this was, like, that was the case? Yeah, and I think that was… the intuition for that came from looking at the…

280
00:39:21.550 --> 00:39:39.549
FI CLASSROOM 03-14: Basically, the structure of the folding block, along with the fact that the pairwise representation is initialized, basically is empty. So, like, the only way that the pairwise information, or the pairwise representation is populated at the beginning, is most likely due to the fact that the sequence will add some updates.

281
00:39:39.670 --> 00:39:42.649
FI CLASSROOM 03-14: And so… because otherwise, like, the… how will the pairwise…

282
00:39:42.840 --> 00:39:45.849
FI CLASSROOM 03-14: Operations Understand anything about the protein?

283
00:39:46.350 --> 00:39:58.429
FI CLASSROOM 03-14: And so that was… I guess that was, like, our intuition. And then the second half was, like, okay, it's very likely just going to be, biochemical signals along with some other stuff that's going to be flowing into the pairwise representation at the beginning.

284
00:39:58.690 --> 00:39:59.410
FI CLASSROOM 03-14: Yeah.

285
00:40:00.850 --> 00:40:14.009
FI CLASSROOM 03-14: So does the ESM embedding have a lot… probably has a lot more than just charge? Yeah, yeah, yeah, so it's just one direction that you could look for. Do you feel like that other stuff might not be as powerful as the… as this charge stuff, or…

286
00:40:14.250 --> 00:40:23.010
FI CLASSROOM 03-14: I think… so, part of the reason why we went for charge was because it was a lot more obvious to, like, how to actually isolate that signal. Like, something else we were looking

287
00:40:23.070 --> 00:40:40.800
FI CLASSROOM 03-14: potentially looking at was, like, oh, the hydrophobicity is very important to protein folding, so that's something we could have looked at. But we just, yeah, we just chose to look at charge, because it's very intuitive, too. Like, if I make one residue, like, hydrophobic, then, like, it is not quite clear, like, how the protein will end up changing.

288
00:40:42.550 --> 00:40:47.810
FI CLASSROOM 03-14: So, yeah, this is just, like, an overview of the mechanisms that we just looked at.

289
00:40:48.100 --> 00:40:56.539
FI CLASSROOM 03-14: And so, yeah, I guess, thinking about what's next. So, like, this is basically, like, some more, like, bio-y experiments that we're thinking about doing, so…

290
00:40:56.770 --> 00:41:14.520
FI CLASSROOM 03-14: Something that happens with these proteins is, like, they're not actually static, so they take on many conformations, and so there's this potential, like, oh, when we take a… when we do a forward pass of a protein that has many conformations, or even disordered regions, like, can we… can we somehow predict that from the representations?

291
00:41:14.520 --> 00:41:31.160
FI CLASSROOM 03-14: like, the pairwise representations, and can we even, like, do some interventions to explore other possibilities for protein structures? Which is already being done using, like, MD trajectories, but this is just, like, one potential example. Another idea is, well, we have this interpretable human charge representation.

292
00:41:31.380 --> 00:41:48.559
FI CLASSROOM 03-14: well, what if we just use this to simulate environmental conditions now for protein folding? So, proteins will change depending on maybe if they're in solution and it has a different pH, so we could, like, apply the charge vector to all these different residues to now extend the model capabilities, because the model was never trained to

293
00:41:48.560 --> 00:41:54.839
FI CLASSROOM 03-14: account for environmental conditions. But we can extend that capability by using this interpretable representation.

294
00:41:55.640 --> 00:42:12.809
FI CLASSROOM 03-14: But I think something else that's important is we talked about these two questions of, like, what will interpretability give us here? Which was, well, one, understanding the model internals, and two, being able to steer. But I think the third piece that is really important when it comes to AI for science is

295
00:42:12.900 --> 00:42:22.929
FI CLASSROOM 03-14: can we discover new science, or even discover new mechanisms of biology? I think here, in terms of the protein space, it's less clear, but something else that I'd love to explore

296
00:42:23.140 --> 00:42:32.069
FI CLASSROOM 03-14: We'll probably be looking at, like, these virtual cell components, so something like gene… like, looking for… looking at single-cell perturbation models, and, like.

297
00:42:32.070 --> 00:42:47.489
FI CLASSROOM 03-14: finding new gene perturbation networks, stuff like that. And there's always a question of, can we do this in an unsupervised fashion? Because this project relied on a lot of human intuition, and there's this question of, well, if you keep applying human intuition to a model, how will you ever just, like.

298
00:42:47.620 --> 00:42:58.560
FI CLASSROOM 03-14: find something new. But yeah, that's just where this project is at right now. So, check out our website at the Battle Lab, and yeah, thank you guys for your time.

299
00:43:04.020 --> 00:43:21.960
FI CLASSROOM 03-14: Any questions? Yeah. Hi, thanks for a great talk. Can you please explain? So, for that matrix, so it would make sense to make it symmetric, right? Because it's the same sequence, it's not, like, keys and queries, right? Yep.

300
00:43:22.080 --> 00:43:32.400
FI CLASSROOM 03-14: Okay, so why don't you just take an average of itself, and it transpose and, like, work with it? Yeah, and I, I think… so there is one…

301
00:43:33.840 --> 00:43:37.929
FI CLASSROOM 03-14: if I was to, like, make ESM fold, I do think…

302
00:43:38.450 --> 00:43:57.229
FI CLASSROOM 03-14: they had a statement about why they actually wanted the asymmetry, and I guess their idea was, like, oh, maybe, like, this asymmetry will allow the model to capture, like, a richer representation than, like, if you force symmetry. Like, I think symmetry would be, like, a nice-to-have, but if we're hoping for the model to capture, like.

303
00:43:57.360 --> 00:44:07.670
FI CLASSROOM 03-14: new… new relationships that maybe we don't understand, like, maybe having more representation space is helpful. But honestly, like, from a geometric deep learning perspective, like, they probably should have just made it symmetric.

304
00:44:07.670 --> 00:44:17.329
Siavash Golkar: But the whole thing is not symmetric, so if you have, like, an alpha helix, this is a right-handed helix that has a handedness. There's a directionality to it.

305
00:44:17.330 --> 00:44:17.920
FI CLASSROOM 03-14: That's true.

306
00:44:17.920 --> 00:44:22.199
Siavash Golkar: Residue to residue, if you invert the whole thing, it's not the same thing.

307
00:44:22.350 --> 00:44:25.190
Siavash Golkar: So I think, like, it's good to be able to capture that.

308
00:44:25.590 --> 00:44:26.470
FI CLASSROOM 03-14: Yeah.

309
00:44:27.450 --> 00:44:35.430
FI CLASSROOM 03-14: Yeah, and I guess, like, even the way that the proteins fold, like, in… I don't understand what is that matrix doing, like…

310
00:44:35.540 --> 00:44:43.710
FI CLASSROOM 03-14: what is this direction, IJ to JI? Yeah. I thought it's, like, some kind of notion of distance or similarity between

311
00:44:44.370 --> 00:45:04.030
FI CLASSROOM 03-14: those positions. Yeah. I think their direction. So maybe I think you can… you can think about it as just, like, a feature encoding, right? Rather than encoding just individual, atoms? I'm not a bio person, but then now you're just encoding pairs of them.

312
00:45:04.080 --> 00:45:08.350
FI CLASSROOM 03-14: And the payer does not have to…

313
00:45:09.800 --> 00:45:29.530
FI CLASSROOM 03-14: So it's like, when we model language, right, we can encode each word individually as a token, but we can encode phrases, pairs of words, that's called, like, bigrams and trigrams. Yeah, then there is… there is… there is meaning, right? Because they just point out it's different.

314
00:45:31.390 --> 00:45:34.050
Siavash Golkar: I guess you can think of this as a directed graph.

315
00:45:34.560 --> 00:45:50.970
Siavash Golkar: And, like, chemically, like, you have, like, I guess elements, I guess atoms that blend electrons and atoms that take them. So that's… there's definitely directionality here. And to have… thinking about this as a directed graph is a… is a good way of thinking about it.

316
00:45:51.180 --> 00:45:56.129
Siavash Golkar: And, this is the representation of the edges of that directed graph.

317
00:45:58.770 --> 00:46:04.659
FI CLASSROOM 03-14: There's more than distance, probably going… Oh yeah, there's definitely more than distance.

318
00:46:04.860 --> 00:46:12.270
FI CLASSROOM 03-14: Did you try cyclic proteins, and then you should seek surgery in the… That's…

319
00:46:12.730 --> 00:46:21.060
FI CLASSROOM 03-14: We didn't look at… yeah, we only looked at, like, secondary structures, like… but I think looking at cyclic proteins might make a lot of sense.

320
00:46:22.890 --> 00:46:39.049
FI CLASSROOM 03-14: Also, how are you initializing the… the pairwise matrix? Is it just, like, identity or random, or… It's just initialized with some positional embeddings. So it's, like, basically, like, if I have, like, residue IJ, then, like, I have an embedding that

321
00:46:39.330 --> 00:46:42.539
FI CLASSROOM 03-14: Tells you how far apart they are, actually, in, like, in the sequence.

322
00:46:43.800 --> 00:46:44.850
FI CLASSROOM 03-14: Yeah.

323
00:46:50.150 --> 00:46:54.229
FI CLASSROOM 03-14: Do you know how well TNNs perform on that type of task?

324
00:46:54.290 --> 00:47:12.189
FI CLASSROOM 03-14: Or if it's any, like, if it's used at all? Yeah, I… I know for, like, DNNs are used a lot for sequence design, and I think for… right now, the main paradigm is still just, like, we have this… this folding module, and then, like, the fusion afterwards. But yeah, there was a… there was a… there was a phase.

325
00:47:14.050 --> 00:47:23.480
FI CLASSROOM 03-14: Yeah, protein MPMN is, graph-based. Yeah, a lot of, like, the protein representation is often… lends itself to graph representation, so…

326
00:47:24.290 --> 00:47:26.530
FI CLASSROOM 03-14: For sequence design, people usually use graphs.

327
00:47:28.940 --> 00:47:31.439
FI CLASSROOM 03-14: All right, okay, thanks again, Kevin.

328
00:47:33.250 --> 00:47:36.549
FI CLASSROOM 03-14: And we'll go upstairs for lunch if people want to join, feel free to join.

329
00:47:36.700 --> 00:47:37.673
FI CLASSROOM 03-14: Thank you.