Main<\/h2>\n\nEvolution searches across an immense space of possible sequences for rare mutations that improve fitness1<\/a>,2<\/a><\/sup>. In nature, this search is based on simple processes of random mutation and recombination1<\/a><\/sup>, but using the same approach for directed evolution of proteins in the laboratory3<\/a><\/sup> imposes a considerable experimental burden. Artificial evolution based on random guessing or brute force search typically devotes substantial effort to interrogate weakly active or non-functional proteins, requiring high experimental throughput to identify variants with improved fitness4<\/a>,5<\/a><\/sup>.<\/p>\nAlthough evolutionary fitness is determined, in part, by specific selection pressures, there are also properties that apply more generally across a protein family or are prerequisites for fitness and function across most proteins; for example, some mutations maintain or improve stability or evolvability6<\/a>,7<\/a><\/sup>, whereas others are structurally destabilizing7<\/a><\/sup> or induce incompetent, misfolded states8<\/a><\/sup>. One approach to improving the efficiency of evolution is to ensure that mutations adhere to these general properties, which we refer to as evolutionary plausibility. Identifying plausible mutations could help guide evolution away from invalid regimes9<\/a><\/sup>, thereby indirectly improving evolutionary efficiency without requiring any explicit knowledge of the function of interest. However, this strategy is also challenging because, first, protein sequences are governed by complex rules, and, second, even if we restrict search to evolutionarily plausible mutations, those that also improve a specific definition of fitness might still be rare beyond practical utility (Fig. 1a<\/a>). More broadly, a major open question10<\/a><\/sup> is whether general evolutionary information (for example, learning patterns from sequence variation across past evolution) is sufficient to enable efficient evolution under specific selection pressures (for example, higher binding affinity to a specific antigen).<\/p>\n\nFig. 1: Guiding evolution with protein language models.<\/b><\/figcaption>\n![\"Science](\"http:\/\/media.springernature.com\/lw685\/springer-static\/image\/art%3A10.1038%2Fs41587-023-01763-2\/MediaObjects\/41587_2023_1763_Fig1_HTML.png\")
<\/picture><\/a><\/div>\na<\/b>,b<\/b>, Two possible models for relating the space of mutations with high evolutionary plausibility (for example, mutations seen in antibodies) to the space with high fitness under specific selection pressures (for example, mutations that result in high binding affinity to a specific antigen). Both models assume that mutations with high fitness make up a rare subset of the full mutational space and that, in general, high-fitness mutations are also evolutionarily plausible. Under the first model (a<\/b>), mutations with high fitness are rare within the subset of mutations that are evolutionarily plausible. Under the second model (b<\/b>), when restricted to the regime of plausible mutations, improvements to fitness become much more common. c<\/b>, Protein language models, trained on millions of natural protein sequences learn amino acid patterns that are likely to be seen in nature. We hypothesized that most mutations with high language model likelihood would also be evolutionarily plausible. Assuming that this is true, and if the second model (b<\/b>) better describes nature, then a language model with no information about specific selection pressures can still efficiently guide evolution.<\/p>\n<\/div>\n
Full size image<\/span><\/a><\/p>\n<\/figure>\n<\/div>\nHere we show that evolutionary information alone can lead to improved fitness under specific selection pressures with high efficiency (Fig. 1b<\/a>). For our main experimental test case, we focused on affinity maturation of human antibodies in which our specific selection pressure is defined as stronger binding affinity to a particular antigen. In nature, a process known as somatic hypermutation evolves or \u2018matures\u2019 an antibody lineage to have higher affinity for an antigen via repeated mutagenesis11<\/a>,12<\/a>,13<\/a><\/sup>. In the laboratory, affinity maturation is a major application of directed evolution due to the therapeutic potential of antibodies with high affinity for disease targets14<\/a><\/sup>.<\/p>\nTo select evolutionarily plausible mutations, we used algorithms known as language models (Fig. 1c<\/a>) to learn patterns that are likely to occur in natural proteins15<\/a>,16<\/a>,17<\/a>,18<\/a>,19<\/a>,20<\/a>,21<\/a>,22<\/a><\/sup>. Because we used general language models19<\/a>,20<\/a><\/sup>, trained on non-redundant sequence datasets that are meant to represent variation across all natural proteins23<\/a><\/sup>, these models can only learn more general evolutionary rules than could a model trained specifically on antibody sequences24<\/a>,25<\/a>,26<\/a>,27<\/a><\/sup> or a model directly supervised with binding affinity28<\/a><\/sup>. Given a single starting sequence, we used these language models to recommend plausible amino acid substitutions that we then experimentally screened for improved fitness. To the end user, the algorithm requires only a single wild-type sequence, without any initial binding affinity data, knowledge of the antigen, task-specific supervision, evolutionary homologs or protein structure information.<\/p>\nWe evolved seven human immunoglobulin G (IgG) antibodies that bind to antigens from coronavirus, ebolavirus and influenza A virus. We focused on viral antigens given the importance of antibody therapeutics for viral diseases29<\/a>,30<\/a>,31<\/a>,32<\/a><\/sup>. We improved the affinity of all antibodies after measuring only 20 or fewer new variants of each antibody across just two rounds of evolution, which, to our knowledge, represents unprecedented efficiency for machine-learning-guided evolution33<\/a>,34<\/a><\/sup>. We also demonstrate that the same<\/i> general protein language models that we used to affinity mature antibodies can also enrich for high-fitness substitutions to diverse proteins beyond antibodies.<\/p>\n<\/div>\n<\/div>\n\nResults<\/h2>\n\nEfficient affinity maturation with protein language models<\/h3>\n
Recent work has demonstrated that language models can predict natural evolution despite having no knowledge of specific selection pressures10<\/a><\/sup>. However, this prior work only predicted the direction of evolution retrospectively when given full knowledge of the evolutionary trajectory. We hypothesized that the predictive capabilities of protein language models might enable a researcher to provide only a single, wild-type antibody sequence to the algorithm and receive a small, manageable set (~101<\/sup>) of high-likelihood variants to experimentally measure for desirable properties. This is a very general setting that does not assume knowledge of protein structure or task-specific training data. A major question, however, is if higher evolutionary likelihood would efficiently translate to higher fitness.<\/p>\nWe tested our hypothesis by conducting evolutionary campaigns, guided by language model likelihood, to affinity mature seven antibodies representing diverse antigens and degrees of maturity (Supplementary Table 1<\/a>):<\/p>\n\n- \n
MEDI8852: a broadly neutralizing antibody (bnAb) that binds influenza A hemagglutinin (HA) across variants of both major phylogenetic groups (group 1 and group 2) and that reached phase 2 clinical trials; this antibody is highly matured, with its parent being isolated from a human, followed by substantial artificial evolution29<\/a><\/sup><\/p>\n<\/li>\n- \n
MEDI8852 unmutated common ancestor (UCA): the unmatured, inferred germline sequence of MEDI8852, which only neutralizes viruses with group 1 HAs29<\/a><\/sup><\/p>\n<\/li>\n- \n
mAb114: a patient-derived antibody that neutralizes ebolavirus by binding to its glycoprotein (GP)30<\/a><\/sup> and has been approved for clinical use by the US Food and Drug Administration (FDA)<\/p>\n<\/li>\n- \n
mAb114 UCA: the unmatured, inferred germline sequence of mAb114 with weak binding to ebolavirus GP30<\/a><\/sup><\/p>\n<\/li>\n- \n
S309: a patient-derived antibody that cross-neutralizes the sarbecoviruses severe acute respiratory syndrome coronavirus 1 (SARS-CoV-1) and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) by binding to the spike glycoprotein (Spike)31<\/a><\/sup> and is the parent antibody of sotrovimab35<\/a><\/sup>, which had an FDA emergency use authorization (EUA) for treatment of Coronavirus Disease 2019 (COVID-19) caused by earlier variants of SARS-CoV-2 (refs. 36<\/a>,37<\/a><\/sup>)<\/p>\n<\/li>\n- \n
REGN10987: a patient-derived antibody that binds early variants of SARS-CoV-2 Spike32<\/a><\/sup> and that had an FDA EUA for use against these variants<\/p>\n<\/li>\n- \n
C143: an unmatured, patient-derived antibody that binds the SARS-CoV-2 Wuhan-Hu-1 Spike but was isolated before extensive in vivo somatic hypermutation38<\/a>,39<\/a><\/sup><\/p>\n<\/li>\n<\/ul>\nWe performed evolution with the ESM-1b language model and the ESM-1v ensemble of five language models (six language models in total)19<\/a>,20<\/a><\/sup>. ESM-1b and ESM-1v were trained on UniRef50 and UniRef90, respectively, which are protein sequence datasets that represent variation across millions of observed natural proteins (UniRef90 contains ~98 million total sequences) and that include only a few thousand antibody-related sequences23<\/a><\/sup>. These datasets are also constructed such that no two sequences have more than 50% (UniRef50) or 90% (UniRef90) sequence similarity with each other to avoid biological redundancy. Additionally, both datasets precede the discovery of the SARS-CoV-2 antibodies considered in the study as well as the evolution of all SARS-CoV-2 variants of concern. Therefore, to evolve these antibodies, the language models cannot use disease-specific biases in the training data and must, instead, learn more general evolutionary patterns.<\/p>\nWe used these language models to compute likelihoods of all single-residue substitutions to the antibody variable regions of either the heavy chain (VH) or the light chain (VL). We selected substitutions with higher evolutionary likelihood than wild-type across a consensus of six language models (Methods<\/a> and Extended Data Fig. 1<\/a>). In the first round of evolution, we measured the antigen interaction strength by biolayer interferometry (BLI) of variants that contain only a single-residue substitution from wild-type. In the second round, we measured variants containing combinations of substitutions, where we selected substitutions that corresponded to preserved or improved binding based on the results of the first round. We performed these two rounds for all seven antibodies, measuring 8\u201314 variants per antibody in round one and 1\u201311 variants per antibody in round two (Fig. 2<\/a> and Supplementary Table 1<\/a>). Variants of the clinically relevant antibodies, which have very low or undetectable dissociation as IgGs, were screened by measuring the dissociation constant (K<\/i>d<\/sub>) of the monovalent fragment antigen-binding (Fab) region; variants of the unmatured antibodies were screened by measuring the apparent K<\/i>d<\/sub> of the bivalent IgG followed by also measuring the K<\/i>d<\/sub> values of the Fab fragments of the highest-avidity variants (Methods<\/a>).<\/p>\n\nFig. 2: Language-model-guided affinity maturation of seven human antibodies.<\/b><\/figcaption>\n![\"Science](\"http:\/\/media.springernature.com\/lw685\/springer-static\/image\/art%3A10.1038%2Fs41587-023-01763-2\/MediaObjects\/41587_2023_1763_Fig2_HTML.png\")
<\/picture><\/a><\/div>\na<\/b>, Strip plots visualizing the two rounds of directed evolution conducted for each antibody. Each point represents an IgG or Fab variant plotted according to the fold change in K<\/i>d<\/sub> from wild-type on the y<\/i> axis and jitter on the x<\/i> axis; a gray, dashed line is drawn at a fold change of 1, and the wild-type point is colored gray. MEDI8852 variants were screened against HA H4 Hubei; MEDI8852 UCA variants against HA H1 Solomon; mAb114 and mAb114 UCA variants against ebolavirus GP; S309 variants against Wuhan-Hu-1 S-6P; and REGN10987 and C143 variants against Beta S-6P. b<\/b>, Phylogenetic trees illustrating the evolutionary trajectories from wild-type to the highest-affinity variant(s) of each antibody. Nodes are annotated with the K<\/i>d<\/sub> values for different antigens and the T<\/i>m<\/sub> of the Fab; all K<\/i>d<\/sub> values are for the monovalent Fab versions except those of C143, which are apparent K<\/i>d<\/sub> values for the bivalent IgGs. B, Beta; H1 Solo., H1 Solomon; ML variant, machine-learning-guided variant; O, Omicron; W1, Wuhan-Hu-1. c<\/b>, We obtained avidity and affinity measurements via BLI of IgGs and Fabs at the indicated concentrations binding to the indicated antigen. Selected BLI traces of the highest-affinity variants for the respective antigens are plotted alongside those of the wild-type variants.<\/p>\n<\/div>\nFull size image<\/span><\/a><\/p>\n<\/figure>\n<\/div>\nWe could successfully express all but one of 122 variants across our seven evolutionary trajectories. Across all seven antibodies, we found that 71\u2013100% of the first-round Fab variants (containing a single-residue substitution) retained sub-micromolar binding to the antigen, and 14\u201371% percent of first-round variants led to improved binding affinity (defined as a 1.1-fold or higher improvement in K<\/i>d<\/sub> compared to wild-type) (Supplementary Table 1<\/a>). Most of the second-round variants (containing a combination of substitutions) also have improved binding (Supplementary Tables 1<\/a>\u20139<\/a>). For all antibodies except for REGN10987, we also obtained variants with at least a two-fold improvement in K<\/i>d<\/sub>. Thirty-six out of all 76 language-model-recommended, single-residue substitutions (and 18 out of 32 substitutions that lead to improved affinity) occur in framework regions (Supplementary Tables 2<\/a>\u20139<\/a>), which are generally less mutated during conventional affinity maturation compared to the complementarity-determining regions (CDRs)12<\/a><\/sup>.<\/p>\nWe were able to improve the binding affinities for all clinically relevant antibodies tested, despite these antibodies being already highly evolved (starting at low nanomolar or picomolar affinity). MEDI8852 is a potent binder with a sub-picomolar Fab K<\/i>d<\/sub> across many HAs and picomolar or nanomolar binding to HAs from subtypes H4 and H7. Although we explicitly screened variants using an HA H4 antigen, the best design also improves binding across a broad set of HAs (Supplementary Tables 2<\/a> and 3<\/a>), including a sevenfold improvement (from 0.21\u2009nM to 0.03\u2009nM) for HA H7 HK17 (A\/Hong Kong\/125\/2017(H7N9)). The best variant of mAb114, a clinically approved drug, achieves a 3.4-fold improvement in Fab K<\/i>d<\/sub> for ebolavirus GP (Supplementary Table 5<\/a>). For REGN10987, the highest-affinity variant has a 1.3-fold improvement against Beta-variant Spike with six stabilizing proline substitutions (S-6P)40<\/a><\/sup> (the antigen used in screening), and another of our designs has a 5.1-fold improvement for the Omicron BA.1 receptor-binding domain (RBD) (Supplementary Table 8<\/a>). For S309, we compared our designs to wild-type and to a variant with the N55Q substitution in the VH introduced after a small-scale, rational evolutionary screen35<\/a><\/sup>; the S309 Fab with the VH N55Q substitution forms the Fab of the therapeutic antibody sotrovimab. Our best variant of S309 has higher affinity than sotrovimab, including a 1.3-fold improvement in Fab K<\/i>d<\/sub> compared to wild-type S309 (versus 1.1-fold for sotrovimab) for SARS-CoV-2 Wuhan-Hu-1 S-6P (the antigen used in screening); a 1.7-fold improvement (versus 1.3-fold for sotrovimab) for Beta S-6P; and a 0.93-fold change (versus 0.82-fold for sotrovimab) for Omicron RBD (Supplementary Table 7<\/a>).<\/p>\nWe were also able to improve affinities for all three unmatured antibodies, often involving much higher fold changes than when evolving the matured antibodies, indicating easier evolvability with respect to affinity. For MEDI8852 UCA, the best Fab design achieves a 2.6-fold improvement in K<\/i>d<\/sub> against HA H1 Solomon (A\/Solomon Islands\/3\/2006(H1N1)), the antigen used in screening. Our best designs also acquire breadth of binding to some group 2 HAs, including a 23-fold improvement for HA H4 Hubei (A\/swine\/Hubei\/06\/2009(H4N1)) and a 5.4-fold improvement for HA H7 HK17 (Supplementary Table 4<\/a>). For mAb114 UCA, our best Fab design achieves a 160-fold improvement in K<\/i>d<\/sub> for ebolavirus GP (Supplementary Table 6<\/a>). Although the algorithm recommends amino acid substitutions to both of these UCA antibodies that are also observed in the matured antibody, other affinity-enhancing substitutions to the UCA antibodies are not found in the matured versions: excluding any substitutions or modified sites found in the matured antibody, our UCA variants achieve up to a sevenfold improvement for HA H4 Hubei (variant VH P75R\/VL G95P; Supplementary Table 4<\/a>) and a 33-fold improvement for ebolavirus GP (variant VH G88E\/VL V43A; Supplementary Table 6<\/a>), demonstrating that our algorithm successfully explores alternative evolutionary routes. For C143, a patient-derived antibody isolated before extensive affinity maturation38<\/a>,39<\/a><\/sup>, our best design achieves a 13-fold improvement for Beta S-6P and a 3.8-fold improvement for Omicron RBD (Supplementary Table 9<\/a>). Results from our directed evolution campaigns are further summarized in Fig. 2<\/a>, Supplementary Tables 2<\/a>\u20139<\/a> and Supplementary Data 1<\/a>.<\/p>\nAdditional characterization of evolved antibodies<\/h3>\n
Although we explicitly selected for improved binders, we also tested these variants for improved stability (Methods<\/a>). We found that Fabs for 21 out of the 31 language-model-recommended, affinity-enhancing variants that we tested had a higher melting temperature (T<\/i>m<\/sub>) than wild-type, and all variants maintained thermostability (T<\/i>m<\/sub>\u2009>\u200970\u2009\u00b0C). When evolving S309 to have higher affinity, our best design has a T<\/i>m<\/sub> of 72.8\u2009\u00b0C compared to 72.5\u2009\u00b0C for wild-type, whereas the VH N55Q substitution introduced in sotrovimab decreases the T<\/i>m<\/sub> to 69.6\u2009\u00b0C (Fig. 2<\/a>). Our evolved variants for mAb114, mAb114 UCA, REGN10987 and C143 also preserve or improve T<\/i>m<\/sub>; the highest change that we observed was an increase from 74.5\u2009\u00b0C to 82.5\u2009\u00b0C when evolving mAb114 UCA. Improved thermostability does not completely explain our affinity maturation results, however, as we observed somewhat decreased T<\/i>m<\/sub> for our affinity-matured variants of MEDI8852 and its UCA, although these Fabs are still thermostable (Fig. 2<\/a>).<\/p>\nAdditionally, we tested our affinity-matured designs for polyspecific binding, because binding unintended targets could lead to undesirable side effects in therapeutic settings. For each of the seven antibodies, we tested the wild-type alongside three affinity-matured variants using a polyspecificity assay that assesses non-specific binding to soluble membrane proteins (Methods<\/a>)41<\/a>,42<\/a><\/sup>. We observed no substantial changes in polyspecificity for any variants of all seven antibodies, and all tested antibodies have polyspecificity values within a therapeutically viable range (Fig. 3a<\/a> and Supplementary Data 2<\/a>).<\/p>\n\nFig. 3: Specificity and improved neutralization potency of affinity-matured variants.<\/b><\/figcaption>\n![\"Science](\"http:\/\/media.springernature.com\/lw685\/springer-static\/image\/art%3A10.1038%2Fs41587-023-01763-2\/MediaObjects\/41587_2023_1763_Fig3_HTML.png\")
<\/picture><\/a><\/div>\na<\/b>, Polyspecificity of antibody wild-types and variants was quantified using an assay42<\/a><\/sup> that measures non-specific binding to soluble membrane proteins via flow cytometry, where higher MFI values correspond to more non-specific binding (Methods<\/a>). Control antibodies42<\/a><\/sup> are elotuzumab (a clinical antibody with low polyspecificity), ixekizumab (a clinical antibody with high polyspecificity) and 4E10 (a research antibody with high polyspecificity beyond a therapeutically viable level)62<\/a><\/sup>. Bar height indicates the mean across n<\/i>\u2009=\u20093 replicate wells; black dots indicate independent measurements. b<\/b>, Variants of the antibody C143, obtained from our language-model-guided affinity maturation campaign, demonstrate improved neutralization activity in a pseudovirus assay. For Beta pseudovirus, out of the three higher-affinity variants that we also screened for neutralization activity, the best improvement is the 32-fold improvement of VL G53V; for D614G pseudovirus, the best improvement is the 19-fold improvement of VL T33N-G53V (Supplementary Table 9<\/a>). Also see Extended Data Fig. 2<\/a>. Points indicate the mean; error bars indicate the s.d.; n<\/i>\u2009=\u20094 independent experiments. c<\/b>, Fold change in K<\/i>d<\/sub> correlates well with fold change in IC50<\/sub> (Spearman r<\/i>\u2009=\u20090.82, n<\/i>\u2009=\u200915 antibody variants) across all designs tested, consistent with higher binding affinity contributing to improved viral neutralization activity. WT, wild-type.<\/p>\n<\/div>\nFull size image<\/span><\/a><\/p>\n<\/figure>\n<\/div>\nAnother therapeutic consideration is immunogenicity. Although computational prediction of immunogenicity remains a challenge, especially involving recognition of discontinuous epitopes, the immunogenicity of linear peptides is better understood43<\/a><\/sup>. We observed that our affinity-matured variants have no significant increase (one-sided binomial P<\/i>\u2009>\u20090.05) in the number of computationally predicted peptide binders to both human leukocyte antigen (HLA) class I and class II (exact P<\/i> values and sample sizes for these experiments are provided in Supplementary Data 2<\/a>), which underlies T-cell-mediated immunogenicity.<\/p>\nWe also wanted to determine if our affinity-matured variants have better viral neutralization activity. We tested affinity-enhancing variants of four antibodies using pseudovirus neutralization assays (Methods<\/a>) and, in all cases, observed variants with half-maximal inhibitory concentration (IC50<\/sub>) values that are significantly improved (Bonferroni-corrected, one-sided t<\/i>-test P<\/i>\u2009<\u20090.05, n<\/i>\u2009=\u20094 independent experiments), including a 1.5-fold improvement for the best mAb114 variant against Ebola pseudovirus; a twofold improvement for the best REGN10987 variant against SARS-CoV-2 Beta pseudovirus; and a 32-fold improvement for the best C143 variant against Beta pseudovirus (Fig. 3b<\/a>, Extended Data Fig. 2<\/a> and Supplementary Tables 5<\/a>, 8<\/a> and 9<\/a>). Additionally, the affinity-matured variants of mAb114 UCA demonstrate detectable neutralization at a >100-fold lower concentration compared to wild-type (Extended Data Fig. 2a<\/a>). In general, change in binding affinity corelates well with change in neutralization (Spearman r<\/i>\u2009=\u20090.82, two-sided t<\/i>-distribution P<\/i>\u2009=\u20091.9\u2009\u00d7\u200910\u22124<\/sup>, n<\/i>\u2009=\u200915 antibody variants) (Fig. 3c<\/a> and Extended Data Fig. 2b<\/a>).<\/p>\nOriginality of affinity-enhancing substitutions<\/h3>\n
Although the ability to find any improvement in affinity is itself useful for engineering applications, we were also interested in whether some of the changes recommended by our algorithm demonstrate \u2018originality\u2019. We quantified originality by computing the frequency that a given residue is observed in nature (Methods<\/a>). Although many affinity-enhancing substitutions are indeed observed at high frequency both in the model\u2019s training data23<\/a><\/sup> and in a database of antibody sequences44<\/a><\/sup>, other substitutions demonstrate greater originality. For example, in the MEDI8852 UCA trajectory, the VL G95P framework substitution (Fig. 2<\/a> and Supplementary Table 4<\/a>) involves changing a glycine observed in 99% of natural antibody sequences to a proline observed in less than 1% of natural sequences. Overall, five out of 32 affinity-enhancing substitutions (~16%) involve changing the wild-type residue to a rare or uncommon residue (Supplementary Table 10<\/a>) and that are also rare when considering only natural variation of antibodies derived from the same germline genes (Supplementary Table 11<\/a>). These results indicate that the language models learn both the \u2018easy\u2019 evolutionary rules involving high-frequency residues and more complex rules that are not captured by a multiple sequence alignment or conventional antibody evolution. Conceptually, these low-frequency, affinity-enhancing substitutions are analogous to examples in other disciplines where an artificial intelligence program occasionally makes unusual but advantageous choices (for example, unintuitive game-playing decisions45<\/a><\/sup>) and likewise may be worth further study.<\/p>\nComparison to other sequence-based methods<\/h3>\n
We also sought to compare general language models to other methods for selecting plausible mutations based on sequence information alone. To assess the contribution of epistatic information learned by the language model, we considered two site-independent models of mutational frequencies: (1) abYsis sequence annotation, which uses extensively curated antibody sequence alignments, and (2) frequencies based on sequence alignments to the UniRef90 dataset, which was used to train ESM-1v (Methods<\/a>). To assess the impact of using language models not trained on antibody-specific sequence variation, we also compared to two antibody language models: (1) AbLang24<\/a><\/sup>, trained on ~10
Evolution searches across an immense space of possible sequences for rare mutations that improve fitness1<\/a>,2<\/a><\/sup>. In nature, this search is based on simple processes of random mutation and recombination1<\/a><\/sup>, but using the same approach for directed evolution of proteins in the laboratory3<\/a><\/sup> imposes a considerable experimental burden. Artificial evolution based on random guessing or brute force search typically devotes substantial effort to interrogate weakly active or non-functional proteins, requiring high experimental throughput to identify variants with improved fitness4<\/a>,5<\/a><\/sup>.<\/p>\n Although evolutionary fitness is determined, in part, by specific selection pressures, there are also properties that apply more generally across a protein family or are prerequisites for fitness and function across most proteins; for example, some mutations maintain or improve stability or evolvability6<\/a>,7<\/a><\/sup>, whereas others are structurally destabilizing7<\/a><\/sup> or induce incompetent, misfolded states8<\/a><\/sup>. One approach to improving the efficiency of evolution is to ensure that mutations adhere to these general properties, which we refer to as evolutionary plausibility. Identifying plausible mutations could help guide evolution away from invalid regimes9<\/a><\/sup>, thereby indirectly improving evolutionary efficiency without requiring any explicit knowledge of the function of interest. However, this strategy is also challenging because, first, protein sequences are governed by complex rules, and, second, even if we restrict search to evolutionarily plausible mutations, those that also improve a specific definition of fitness might still be rare beyond practical utility (Fig. 1a<\/a>). More broadly, a major open question10<\/a><\/sup> is whether general evolutionary information (for example, learning patterns from sequence variation across past evolution) is sufficient to enable efficient evolution under specific selection pressures (for example, higher binding affinity to a specific antigen).<\/p>\n a<\/b>,b<\/b>, Two possible models for relating the space of mutations with high evolutionary plausibility (for example, mutations seen in antibodies) to the space with high fitness under specific selection pressures (for example, mutations that result in high binding affinity to a specific antigen). Both models assume that mutations with high fitness make up a rare subset of the full mutational space and that, in general, high-fitness mutations are also evolutionarily plausible. Under the first model (a<\/b>), mutations with high fitness are rare within the subset of mutations that are evolutionarily plausible. Under the second model (b<\/b>), when restricted to the regime of plausible mutations, improvements to fitness become much more common. c<\/b>, Protein language models, trained on millions of natural protein sequences learn amino acid patterns that are likely to be seen in nature. We hypothesized that most mutations with high language model likelihood would also be evolutionarily plausible. Assuming that this is true, and if the second model (b<\/b>) better describes nature, then a language model with no information about specific selection pressures can still efficiently guide evolution.<\/p>\n<\/div>\n Full size image<\/span><\/a><\/p>\n<\/figure>\n<\/div>\n Here we show that evolutionary information alone can lead to improved fitness under specific selection pressures with high efficiency (Fig. 1b<\/a>). For our main experimental test case, we focused on affinity maturation of human antibodies in which our specific selection pressure is defined as stronger binding affinity to a particular antigen. In nature, a process known as somatic hypermutation evolves or \u2018matures\u2019 an antibody lineage to have higher affinity for an antigen via repeated mutagenesis11<\/a>,12<\/a>,13<\/a><\/sup>. In the laboratory, affinity maturation is a major application of directed evolution due to the therapeutic potential of antibodies with high affinity for disease targets14<\/a><\/sup>.<\/p>\n To select evolutionarily plausible mutations, we used algorithms known as language models (Fig. 1c<\/a>) to learn patterns that are likely to occur in natural proteins15<\/a>,16<\/a>,17<\/a>,18<\/a>,19<\/a>,20<\/a>,21<\/a>,22<\/a><\/sup>. Because we used general language models19<\/a>,20<\/a><\/sup>, trained on non-redundant sequence datasets that are meant to represent variation across all natural proteins23<\/a><\/sup>, these models can only learn more general evolutionary rules than could a model trained specifically on antibody sequences24<\/a>,25<\/a>,26<\/a>,27<\/a><\/sup> or a model directly supervised with binding affinity28<\/a><\/sup>. Given a single starting sequence, we used these language models to recommend plausible amino acid substitutions that we then experimentally screened for improved fitness. To the end user, the algorithm requires only a single wild-type sequence, without any initial binding affinity data, knowledge of the antigen, task-specific supervision, evolutionary homologs or protein structure information.<\/p>\n We evolved seven human immunoglobulin G (IgG) antibodies that bind to antigens from coronavirus, ebolavirus and influenza A virus. We focused on viral antigens given the importance of antibody therapeutics for viral diseases29<\/a>,30<\/a>,31<\/a>,32<\/a><\/sup>. We improved the affinity of all antibodies after measuring only 20 or fewer new variants of each antibody across just two rounds of evolution, which, to our knowledge, represents unprecedented efficiency for machine-learning-guided evolution33<\/a>,34<\/a><\/sup>. We also demonstrate that the same<\/i> general protein language models that we used to affinity mature antibodies can also enrich for high-fitness substitutions to diverse proteins beyond antibodies.<\/p>\n<\/div>\n<\/div>\n Recent work has demonstrated that language models can predict natural evolution despite having no knowledge of specific selection pressures10<\/a><\/sup>. However, this prior work only predicted the direction of evolution retrospectively when given full knowledge of the evolutionary trajectory. We hypothesized that the predictive capabilities of protein language models might enable a researcher to provide only a single, wild-type antibody sequence to the algorithm and receive a small, manageable set (~101<\/sup>) of high-likelihood variants to experimentally measure for desirable properties. This is a very general setting that does not assume knowledge of protein structure or task-specific training data. A major question, however, is if higher evolutionary likelihood would efficiently translate to higher fitness.<\/p>\n We tested our hypothesis by conducting evolutionary campaigns, guided by language model likelihood, to affinity mature seven antibodies representing diverse antigens and degrees of maturity (Supplementary Table 1<\/a>):<\/p>\n MEDI8852: a broadly neutralizing antibody (bnAb) that binds influenza A hemagglutinin (HA) across variants of both major phylogenetic groups (group 1 and group 2) and that reached phase 2 clinical trials; this antibody is highly matured, with its parent being isolated from a human, followed by substantial artificial evolution29<\/a><\/sup><\/p>\n<\/li>\n MEDI8852 unmutated common ancestor (UCA): the unmatured, inferred germline sequence of MEDI8852, which only neutralizes viruses with group 1 HAs29<\/a><\/sup><\/p>\n<\/li>\n mAb114: a patient-derived antibody that neutralizes ebolavirus by binding to its glycoprotein (GP)30<\/a><\/sup> and has been approved for clinical use by the US Food and Drug Administration (FDA)<\/p>\n<\/li>\n mAb114 UCA: the unmatured, inferred germline sequence of mAb114 with weak binding to ebolavirus GP30<\/a><\/sup><\/p>\n<\/li>\n S309: a patient-derived antibody that cross-neutralizes the sarbecoviruses severe acute respiratory syndrome coronavirus 1 (SARS-CoV-1) and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) by binding to the spike glycoprotein (Spike)31<\/a><\/sup> and is the parent antibody of sotrovimab35<\/a><\/sup>, which had an FDA emergency use authorization (EUA) for treatment of Coronavirus Disease 2019 (COVID-19) caused by earlier variants of SARS-CoV-2 (refs. 36<\/a>,37<\/a><\/sup>)<\/p>\n<\/li>\n REGN10987: a patient-derived antibody that binds early variants of SARS-CoV-2 Spike32<\/a><\/sup> and that had an FDA EUA for use against these variants<\/p>\n<\/li>\n C143: an unmatured, patient-derived antibody that binds the SARS-CoV-2 Wuhan-Hu-1 Spike but was isolated before extensive in vivo somatic hypermutation38<\/a>,39<\/a><\/sup><\/p>\n<\/li>\n<\/ul>\n We performed evolution with the ESM-1b language model and the ESM-1v ensemble of five language models (six language models in total)19<\/a>,20<\/a><\/sup>. ESM-1b and ESM-1v were trained on UniRef50 and UniRef90, respectively, which are protein sequence datasets that represent variation across millions of observed natural proteins (UniRef90 contains ~98 million total sequences) and that include only a few thousand antibody-related sequences23<\/a><\/sup>. These datasets are also constructed such that no two sequences have more than 50% (UniRef50) or 90% (UniRef90) sequence similarity with each other to avoid biological redundancy. Additionally, both datasets precede the discovery of the SARS-CoV-2 antibodies considered in the study as well as the evolution of all SARS-CoV-2 variants of concern. Therefore, to evolve these antibodies, the language models cannot use disease-specific biases in the training data and must, instead, learn more general evolutionary patterns.<\/p>\n We used these language models to compute likelihoods of all single-residue substitutions to the antibody variable regions of either the heavy chain (VH) or the light chain (VL). We selected substitutions with higher evolutionary likelihood than wild-type across a consensus of six language models (Methods<\/a> and Extended Data Fig. 1<\/a>). In the first round of evolution, we measured the antigen interaction strength by biolayer interferometry (BLI) of variants that contain only a single-residue substitution from wild-type. In the second round, we measured variants containing combinations of substitutions, where we selected substitutions that corresponded to preserved or improved binding based on the results of the first round. We performed these two rounds for all seven antibodies, measuring 8\u201314 variants per antibody in round one and 1\u201311 variants per antibody in round two (Fig. 2<\/a> and Supplementary Table 1<\/a>). Variants of the clinically relevant antibodies, which have very low or undetectable dissociation as IgGs, were screened by measuring the dissociation constant (K<\/i>d<\/sub>) of the monovalent fragment antigen-binding (Fab) region; variants of the unmatured antibodies were screened by measuring the apparent K<\/i>d<\/sub> of the bivalent IgG followed by also measuring the K<\/i>d<\/sub> values of the Fab fragments of the highest-avidity variants (Methods<\/a>).<\/p>\n a<\/b>, Strip plots visualizing the two rounds of directed evolution conducted for each antibody. Each point represents an IgG or Fab variant plotted according to the fold change in K<\/i>d<\/sub> from wild-type on the y<\/i> axis and jitter on the x<\/i> axis; a gray, dashed line is drawn at a fold change of 1, and the wild-type point is colored gray. MEDI8852 variants were screened against HA H4 Hubei; MEDI8852 UCA variants against HA H1 Solomon; mAb114 and mAb114 UCA variants against ebolavirus GP; S309 variants against Wuhan-Hu-1 S-6P; and REGN10987 and C143 variants against Beta S-6P. b<\/b>, Phylogenetic trees illustrating the evolutionary trajectories from wild-type to the highest-affinity variant(s) of each antibody. Nodes are annotated with the K<\/i>d<\/sub> values for different antigens and the T<\/i>m<\/sub> of the Fab; all K<\/i>d<\/sub> values are for the monovalent Fab versions except those of C143, which are apparent K<\/i>d<\/sub> values for the bivalent IgGs. B, Beta; H1 Solo., H1 Solomon; ML variant, machine-learning-guided variant; O, Omicron; W1, Wuhan-Hu-1. c<\/b>, We obtained avidity and affinity measurements via BLI of IgGs and Fabs at the indicated concentrations binding to the indicated antigen. Selected BLI traces of the highest-affinity variants for the respective antigens are plotted alongside those of the wild-type variants.<\/p>\n<\/div>\n Full size image<\/span><\/a><\/p>\n<\/figure>\n<\/div>\n We could successfully express all but one of 122 variants across our seven evolutionary trajectories. Across all seven antibodies, we found that 71\u2013100% of the first-round Fab variants (containing a single-residue substitution) retained sub-micromolar binding to the antigen, and 14\u201371% percent of first-round variants led to improved binding affinity (defined as a 1.1-fold or higher improvement in K<\/i>d<\/sub> compared to wild-type) (Supplementary Table 1<\/a>). Most of the second-round variants (containing a combination of substitutions) also have improved binding (Supplementary Tables 1<\/a>\u20139<\/a>). For all antibodies except for REGN10987, we also obtained variants with at least a two-fold improvement in K<\/i>d<\/sub>. Thirty-six out of all 76 language-model-recommended, single-residue substitutions (and 18 out of 32 substitutions that lead to improved affinity) occur in framework regions (Supplementary Tables 2<\/a>\u20139<\/a>), which are generally less mutated during conventional affinity maturation compared to the complementarity-determining regions (CDRs)12<\/a><\/sup>.<\/p>\n We were able to improve the binding affinities for all clinically relevant antibodies tested, despite these antibodies being already highly evolved (starting at low nanomolar or picomolar affinity). MEDI8852 is a potent binder with a sub-picomolar Fab K<\/i>d<\/sub> across many HAs and picomolar or nanomolar binding to HAs from subtypes H4 and H7. Although we explicitly screened variants using an HA H4 antigen, the best design also improves binding across a broad set of HAs (Supplementary Tables 2<\/a> and 3<\/a>), including a sevenfold improvement (from 0.21\u2009nM to 0.03\u2009nM) for HA H7 HK17 (A\/Hong Kong\/125\/2017(H7N9)). The best variant of mAb114, a clinically approved drug, achieves a 3.4-fold improvement in Fab K<\/i>d<\/sub> for ebolavirus GP (Supplementary Table 5<\/a>). For REGN10987, the highest-affinity variant has a 1.3-fold improvement against Beta-variant Spike with six stabilizing proline substitutions (S-6P)40<\/a><\/sup> (the antigen used in screening), and another of our designs has a 5.1-fold improvement for the Omicron BA.1 receptor-binding domain (RBD) (Supplementary Table 8<\/a>). For S309, we compared our designs to wild-type and to a variant with the N55Q substitution in the VH introduced after a small-scale, rational evolutionary screen35<\/a><\/sup>; the S309 Fab with the VH N55Q substitution forms the Fab of the therapeutic antibody sotrovimab. Our best variant of S309 has higher affinity than sotrovimab, including a 1.3-fold improvement in Fab K<\/i>d<\/sub> compared to wild-type S309 (versus 1.1-fold for sotrovimab) for SARS-CoV-2 Wuhan-Hu-1 S-6P (the antigen used in screening); a 1.7-fold improvement (versus 1.3-fold for sotrovimab) for Beta S-6P; and a 0.93-fold change (versus 0.82-fold for sotrovimab) for Omicron RBD (Supplementary Table 7<\/a>).<\/p>\n We were also able to improve affinities for all three unmatured antibodies, often involving much higher fold changes than when evolving the matured antibodies, indicating easier evolvability with respect to affinity. For MEDI8852 UCA, the best Fab design achieves a 2.6-fold improvement in K<\/i>d<\/sub> against HA H1 Solomon (A\/Solomon Islands\/3\/2006(H1N1)), the antigen used in screening. Our best designs also acquire breadth of binding to some group 2 HAs, including a 23-fold improvement for HA H4 Hubei (A\/swine\/Hubei\/06\/2009(H4N1)) and a 5.4-fold improvement for HA H7 HK17 (Supplementary Table 4<\/a>). For mAb114 UCA, our best Fab design achieves a 160-fold improvement in K<\/i>d<\/sub> for ebolavirus GP (Supplementary Table 6<\/a>). Although the algorithm recommends amino acid substitutions to both of these UCA antibodies that are also observed in the matured antibody, other affinity-enhancing substitutions to the UCA antibodies are not found in the matured versions: excluding any substitutions or modified sites found in the matured antibody, our UCA variants achieve up to a sevenfold improvement for HA H4 Hubei (variant VH P75R\/VL G95P; Supplementary Table 4<\/a>) and a 33-fold improvement for ebolavirus GP (variant VH G88E\/VL V43A; Supplementary Table 6<\/a>), demonstrating that our algorithm successfully explores alternative evolutionary routes. For C143, a patient-derived antibody isolated before extensive affinity maturation38<\/a>,39<\/a><\/sup>, our best design achieves a 13-fold improvement for Beta S-6P and a 3.8-fold improvement for Omicron RBD (Supplementary Table 9<\/a>). Results from our directed evolution campaigns are further summarized in Fig. 2<\/a>, Supplementary Tables 2<\/a>\u20139<\/a> and Supplementary Data 1<\/a>.<\/p>\n Although we explicitly selected for improved binders, we also tested these variants for improved stability (Methods<\/a>). We found that Fabs for 21 out of the 31 language-model-recommended, affinity-enhancing variants that we tested had a higher melting temperature (T<\/i>m<\/sub>) than wild-type, and all variants maintained thermostability (T<\/i>m<\/sub>\u2009>\u200970\u2009\u00b0C). When evolving S309 to have higher affinity, our best design has a T<\/i>m<\/sub> of 72.8\u2009\u00b0C compared to 72.5\u2009\u00b0C for wild-type, whereas the VH N55Q substitution introduced in sotrovimab decreases the T<\/i>m<\/sub> to 69.6\u2009\u00b0C (Fig. 2<\/a>). Our evolved variants for mAb114, mAb114 UCA, REGN10987 and C143 also preserve or improve T<\/i>m<\/sub>; the highest change that we observed was an increase from 74.5\u2009\u00b0C to 82.5\u2009\u00b0C when evolving mAb114 UCA. Improved thermostability does not completely explain our affinity maturation results, however, as we observed somewhat decreased T<\/i>m<\/sub> for our affinity-matured variants of MEDI8852 and its UCA, although these Fabs are still thermostable (Fig. 2<\/a>).<\/p>\n Additionally, we tested our affinity-matured designs for polyspecific binding, because binding unintended targets could lead to undesirable side effects in therapeutic settings. For each of the seven antibodies, we tested the wild-type alongside three affinity-matured variants using a polyspecificity assay that assesses non-specific binding to soluble membrane proteins (Methods<\/a>)41<\/a>,42<\/a><\/sup>. We observed no substantial changes in polyspecificity for any variants of all seven antibodies, and all tested antibodies have polyspecificity values within a therapeutically viable range (Fig. 3a<\/a> and Supplementary Data 2<\/a>).<\/p>\n a<\/b>, Polyspecificity of antibody wild-types and variants was quantified using an assay42<\/a><\/sup> that measures non-specific binding to soluble membrane proteins via flow cytometry, where higher MFI values correspond to more non-specific binding (Methods<\/a>). Control antibodies42<\/a><\/sup> are elotuzumab (a clinical antibody with low polyspecificity), ixekizumab (a clinical antibody with high polyspecificity) and 4E10 (a research antibody with high polyspecificity beyond a therapeutically viable level)62<\/a><\/sup>. Bar height indicates the mean across n<\/i>\u2009=\u20093 replicate wells; black dots indicate independent measurements. b<\/b>, Variants of the antibody C143, obtained from our language-model-guided affinity maturation campaign, demonstrate improved neutralization activity in a pseudovirus assay. For Beta pseudovirus, out of the three higher-affinity variants that we also screened for neutralization activity, the best improvement is the 32-fold improvement of VL G53V; for D614G pseudovirus, the best improvement is the 19-fold improvement of VL T33N-G53V (Supplementary Table 9<\/a>). Also see Extended Data Fig. 2<\/a>. Points indicate the mean; error bars indicate the s.d.; n<\/i>\u2009=\u20094 independent experiments. c<\/b>, Fold change in K<\/i>d<\/sub> correlates well with fold change in IC50<\/sub> (Spearman r<\/i>\u2009=\u20090.82, n<\/i>\u2009=\u200915 antibody variants) across all designs tested, consistent with higher binding affinity contributing to improved viral neutralization activity. WT, wild-type.<\/p>\n<\/div>\n Full size image<\/span><\/a><\/p>\n<\/figure>\n<\/div>\n Another therapeutic consideration is immunogenicity. Although computational prediction of immunogenicity remains a challenge, especially involving recognition of discontinuous epitopes, the immunogenicity of linear peptides is better understood43<\/a><\/sup>. We observed that our affinity-matured variants have no significant increase (one-sided binomial P<\/i>\u2009>\u20090.05) in the number of computationally predicted peptide binders to both human leukocyte antigen (HLA) class I and class II (exact P<\/i> values and sample sizes for these experiments are provided in Supplementary Data 2<\/a>), which underlies T-cell-mediated immunogenicity.<\/p>\n We also wanted to determine if our affinity-matured variants have better viral neutralization activity. We tested affinity-enhancing variants of four antibodies using pseudovirus neutralization assays (Methods<\/a>) and, in all cases, observed variants with half-maximal inhibitory concentration (IC50<\/sub>) values that are significantly improved (Bonferroni-corrected, one-sided t<\/i>-test P<\/i>\u2009<\u20090.05, n<\/i>\u2009=\u20094 independent experiments), including a 1.5-fold improvement for the best mAb114 variant against Ebola pseudovirus; a twofold improvement for the best REGN10987 variant against SARS-CoV-2 Beta pseudovirus; and a 32-fold improvement for the best C143 variant against Beta pseudovirus (Fig. 3b<\/a>, Extended Data Fig. 2<\/a> and Supplementary Tables 5<\/a>, 8<\/a> and 9<\/a>). Additionally, the affinity-matured variants of mAb114 UCA demonstrate detectable neutralization at a >100-fold lower concentration compared to wild-type (Extended Data Fig. 2a<\/a>). In general, change in binding affinity corelates well with change in neutralization (Spearman r<\/i>\u2009=\u20090.82, two-sided t<\/i>-distribution P<\/i>\u2009=\u20091.9\u2009\u00d7\u200910\u22124<\/sup>, n<\/i>\u2009=\u200915 antibody variants) (Fig. 3c<\/a> and Extended Data Fig. 2b<\/a>).<\/p>\n Although the ability to find any improvement in affinity is itself useful for engineering applications, we were also interested in whether some of the changes recommended by our algorithm demonstrate \u2018originality\u2019. We quantified originality by computing the frequency that a given residue is observed in nature (Methods<\/a>). Although many affinity-enhancing substitutions are indeed observed at high frequency both in the model\u2019s training data23<\/a><\/sup> and in a database of antibody sequences44<\/a><\/sup>, other substitutions demonstrate greater originality. For example, in the MEDI8852 UCA trajectory, the VL G95P framework substitution (Fig. 2<\/a> and Supplementary Table 4<\/a>) involves changing a glycine observed in 99% of natural antibody sequences to a proline observed in less than 1% of natural sequences. Overall, five out of 32 affinity-enhancing substitutions (~16%) involve changing the wild-type residue to a rare or uncommon residue (Supplementary Table 10<\/a>) and that are also rare when considering only natural variation of antibodies derived from the same germline genes (Supplementary Table 11<\/a>). These results indicate that the language models learn both the \u2018easy\u2019 evolutionary rules involving high-frequency residues and more complex rules that are not captured by a multiple sequence alignment or conventional antibody evolution. Conceptually, these low-frequency, affinity-enhancing substitutions are analogous to examples in other disciplines where an artificial intelligence program occasionally makes unusual but advantageous choices (for example, unintuitive game-playing decisions45<\/a><\/sup>) and likewise may be worth further study.<\/p>\n We also sought to compare general language models to other methods for selecting plausible mutations based on sequence information alone. To assess the contribution of epistatic information learned by the language model, we considered two site-independent models of mutational frequencies: (1) abYsis sequence annotation, which uses extensively curated antibody sequence alignments, and (2) frequencies based on sequence alignments to the UniRef90 dataset, which was used to train ESM-1v (Methods<\/a>). To assess the impact of using language models not trained on antibody-specific sequence variation, we also compared to two antibody language models: (1) AbLang24<\/a><\/sup>, trained on ~10<\/picture><\/a><\/div>\nResults<\/h2>\n
Efficient affinity maturation with protein language models<\/h3>\n
\n
<\/picture><\/a><\/div>\nAdditional characterization of evolved antibodies<\/h3>\n
<\/picture><\/a><\/div>\nOriginality of affinity-enhancing substitutions<\/h3>\n
Comparison to other sequence-based methods<\/h3>\n