MSA Transformer¶

A 100M-parameter protein language model that operates on Multiple Sequence Alignments (MSAs) using axial attention with tied row attention, producing evolutionary-aware embeddings and unsupervised contact predictions.

License: MIT · Molecules: protein · Tasks: embedding

Actions: encode · Variants: 1

At a glance¶

Use it when

You have precomputed MSAs available and want embeddings that explicitly encode evolutionary covariation patterns for downstream ML tasks
You need unsupervised contact predictions for quick structural assessment without running a full structure prediction pipeline
You are working on protein family analysis where evolutionary relationships across homologs are central to the biological question
You need MSA-derived features for a structural prediction pipeline (e.g., secondary structure, solvent accessibility, disorder prediction) and want a neural alternative to traditional PSSM/HMM profiles
You want to validate MSA quality by checking whether predicted contacts match known or predicted structures
You are studying evolutionary covariation and need attention-based co-evolutionary coupling scores that are faster than traditional DCA/PSICOV methods

Strengths

Explicitly captures evolutionary covariation — tied row attention and column attention jointly process MSA columns, detecting co-evolving residue pairs that single-sequence models (ESM2, ESMC) cannot directly access
Unsupervised contact prediction from attention maps — symmetrized attention weights with APC correction produce contact maps without any supervised training on known structures, rivaling dedicated covariation methods
MSA-aware embeddings encode both within-sequence context and between-sequence evolutionary context, providing richer representations for proteins with deep alignments than any single-sequence model
Relatively compact (100M parameters) — runs efficiently on T4 GPU (16GB VRAM), lighter than most other protein language models in this group
Published applications in 3D structure modeling (A-Prot), secondary structure prediction (S-Pred), and multi-feature structural prediction (MFTrans) demonstrate utility as a feature extractor for diverse structural tasks
MIT license with no commercial restrictions, suitable for any deployment scenario
Validated on 26 million MSAs during training, providing strong generalization across protein families

Limitations

Requires precomputed multiple sequence alignments (MSAs) as input — MSA construction via JackHMMER or HHblits takes minutes to hours per query, creating a major throughput bottleneck compared to single-sequence models
Performance degrades severely for proteins with shallow MSAs (fewer than 16 effective sequences) — orphan proteins and recently evolved proteins with few homologs produce unreliable results
Single action only (encode) — no dedicated variant scoring, masked prediction, or log-probability computation; variant analysis requires custom post-processing of embeddings or attention maps
Contact prediction, while impressive for an unsupervised method, has been surpassed by dedicated structure prediction models (AlphaFold2, ESMFold, Chai-1) that produce full atomic coordinates
Fixed MSA depth handling — the model processes up to a fixed number of sequences, potentially discarding informative homologs from very deep alignments or underperforming on very sparse ones
No structure awareness beyond what is implicitly captured by evolutionary covariation — cannot directly use 3D structural information as input unlike structure-aware models such as ProstT5
Not designed for protein design or generation — purely analytical model that cannot propose novel sequences

Reach for something else when

You do not have precomputed MSAs or the computational resources to build them — use ESM2 or ESMC for instant single-sequence embedding and scoring
You need variant effect prediction scores — use ESM1v, ESM2 log_prob, or PoET score, which are purpose-built for variant scoring
You need accurate 3D protein structure prediction — use Chai-1, RF3, or ESMFold, which produce full atomic coordinates rather than just contact maps
You are working with orphan proteins that have no detectable homologs — MSA Transformer provides no advantage over single-sequence models when no meaningful alignment can be built
You need high-throughput processing of millions of sequences — the MSA construction bottleneck makes MSA Transformer impractical for proteome-scale analyses
You need a generative model for protein design or sequence sampling — MSA Transformer is purely analytical
You need structure-conditioned analysis — use ProstT5, which can directly leverage structural (3Di) information

API & schema¶

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

`encode`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/msa-transformer/encode \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "msa": [
        ">seq1\nMKTAYIAK\n>seq2\nMKTAYIAK"
      ]
    }
  ]
}'

Request — MSATransformerEncodeRequest

Field	Type	Required	Constraints	Description
`params`	MSATransformerEncodeRequestParams	no		Optional parameters controlling this action (defaults are used when omitted).
`items`	list[MSATransformerEncodeRequestItem]	yes	items 1–4	Batch of inputs to process in a single request. Up to 4 MSAs per request.

Nested types

MSATransformerEncodeIncludeOptions

Allowed values: mean, per_residue, per_residue, row_attention, contacts

MSATransformerEncodeRequestItem

Field	Type	Required	Constraints	Description
`msa`	list[string]	yes	items 2–256	Multiple-sequence alignment for the query protein; first row is the query, all rows pre-aligned to equal length.

MSATransformerEncodeRequestParams

Field	Type	Required	Constraints	Description
`repr_layers`	list[integer]	no	default `[-1]`	Hidden layers whose representations to return (negative indexes count from the last layer).
`include`	list[MSATransformerEncodeIncludeOptions]	no	default `['mean']`	Optional outputs to compute and include in the response.

Raw JSON Schema

{
  "$defs": {
    "MSATransformerEncodeIncludeOptions": {
      "description": "Options for what to include in encode response.",
      "enum": [
        "mean",
        "per_residue",
        "per_residue",
        "row_attention",
        "contacts"
      ],
      "title": "MSATransformerEncodeIncludeOptions",
      "type": "string"
    },
    "MSATransformerEncodeRequestItem": {
      "additionalProperties": false,
      "description": "Single MSA input item.",
      "properties": {
        "msa": {
          "description": "Multiple-sequence alignment for the query protein; first row is the query, all rows pre-aligned to equal length.",
          "items": {
            "type": "string"
          },
          "maxItems": 256,
          "minItems": 2,
          "title": "Msa",
          "type": "array"
        }
      },
      "required": [
        "msa"
      ],
      "title": "MSATransformerEncodeRequestItem",
      "type": "object"
    },
    "MSATransformerEncodeRequestParams": {
      "additionalProperties": false,
      "description": "Parameters for encode request.",
      "properties": {
        "repr_layers": {
          "description": "Hidden layers whose representations to return (negative indexes count from the last layer).",
          "items": {
            "type": "integer"
          },
          "title": "Repr Layers",
          "type": "array",
          "default": [
            -1
          ]
        },
        "include": {
          "description": "Optional outputs to compute and include in the response.",
          "items": {
            "$ref": "#/$defs/MSATransformerEncodeIncludeOptions"
          },
          "title": "Include",
          "type": "array",
          "default": [
            "mean"
          ]
        }
      },
      "title": "MSATransformerEncodeRequestParams",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "description": "Request for MSA Transformer encode action.",
  "properties": {
    "params": {
      "$ref": "#/$defs/MSATransformerEncodeRequestParams",
      "description": "Optional parameters controlling this action (defaults are used when omitted)."
    },
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 4 MSAs per request.",
      "items": {
        "$ref": "#/$defs/MSATransformerEncodeRequestItem"
      },
      "maxItems": 4,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "MSATransformerEncodeRequest",
  "type": "object"
}

Response — MSATransformerEncodeResponse

Field	Type	Required	Constraints	Description
`results`	list[MSATransformerEncodeResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

LayerEmbedding

Field	Type	Required	Constraints	Description
`layer`	integer	yes		Model layer this representation was taken from.
`embedding`	list[number]	yes		Mean-pooled embedding vector for the sequence.

LayerPerTokenEmbeddings

Field	Type	Required	Constraints	Description
`layer`	integer	yes		Model layer this representation was taken from.
`embeddings`	list[list[number]]	yes		Per-token (per-residue) embedding vectors for this layer.

MSATransformerEncodeResponseResult

Field	Type	Required	Description
`sequence_index`	integer	yes	Index of the corresponding input sequence within the request batch.
`embeddings`	list[LayerEmbedding] \| null	no	Mean-pooled embeddings of the query sequence, one entry per requested layer.
`residue_embeddings`	list[LayerPerTokenEmbeddings] \| null	no	Per-residue embedding vectors.
`row_attentions`	list[list[list[number]]] \| null	no	Tied row-attention maps averaged over heads; shape [layers, seq_len, seq_len].
`contacts`	list[list[number]] \| null	no	Predicted residue–residue contact probability map.

Raw JSON Schema

{
  "$defs": {
    "LayerEmbedding": {
      "description": "Embedding for a specific layer.",
      "properties": {
        "layer": {
          "description": "Model layer this representation was taken from.",
          "title": "Layer",
          "type": "integer"
        },
        "embedding": {
          "description": "Mean-pooled embedding vector for the sequence.",
          "items": {
            "type": "number"
          },
          "title": "Embedding",
          "type": "array"
        }
      },
      "required": [
        "layer",
        "embedding"
      ],
      "title": "LayerEmbedding",
      "type": "object"
    },
    "LayerPerTokenEmbeddings": {
      "description": "Per-token embeddings for a specific layer.",
      "properties": {
        "layer": {
          "description": "Model layer this representation was taken from.",
          "title": "Layer",
          "type": "integer"
        },
        "embeddings": {
          "description": "Per-token (per-residue) embedding vectors for this layer.",
          "items": {
            "items": {
              "type": "number"
            },
            "type": "array"
          },
          "title": "Embeddings",
          "type": "array"
        }
      },
      "required": [
        "layer",
        "embeddings"
      ],
      "title": "LayerPerTokenEmbeddings",
      "type": "object"
    },
    "MSATransformerEncodeResponseResult": {
      "description": "Result for a single MSA encode.",
      "exclude_none": true,
      "exclude_unset": true,
      "properties": {
        "sequence_index": {
          "description": "Index of the corresponding input sequence within the request batch.",
          "title": "Sequence Index",
          "type": "integer"
        },
        "embeddings": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/LayerEmbedding"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Mean-pooled embeddings of the query sequence, one entry per requested layer.",
          "title": "Embeddings"
        },
        "residue_embeddings": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/LayerPerTokenEmbeddings"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-residue embedding vectors.",
          "title": "Residue Embeddings"
        },
        "row_attentions": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "items": {
                    "type": "number"
                  },
                  "type": "array"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Tied row-attention maps averaged over heads; shape [layers, seq_len, seq_len].",
          "title": "Row Attentions"
        },
        "contacts": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Predicted residue\u2013residue contact probability map.",
          "title": "Contacts"
        }
      },
      "required": [
        "sequence_index"
      ],
      "title": "MSATransformerEncodeResponseResult",
      "type": "object"
    }
  },
  "description": "Response for MSA Transformer encode action.",
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/MSATransformerEncodeResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "MSATransformerEncodeResponse",
  "type": "object"
}

Usage¶

One-line summary: A 100M-parameter protein language model that operates on Multiple Sequence Alignments (MSAs) using axial attention with tied row attention, producing evolutionary-aware embeddings and unsupervised contact predictions.

Overview¶

The MSA Transformer is a protein language model that combines the benefits of evolutionary covariation analysis with deep learning. Unlike single-sequence models (e.g., ESM-2) that process one protein at a time, the MSA Transformer takes a Multiple Sequence Alignment as input, enabling it to directly capture patterns of conservation and covariation across evolutionarily related sequences.

Developed by Meta AI / FAIR as part of the ESM suite, the MSA Transformer uses an axial attention mechanism that alternates between row attention (across positions) and column attention (across sequences). Row attention is tied across all sequences, reducing memory requirements while enabling direct extraction of contact information from attention weights.

Architecture¶

Property	Value
Architecture	Axial Transformer with tied row attention
Parameters	100M
Layers	12
Embedding dimension	768
Attention heads	12
Training data	26M MSAs from UniRef50/UniClust30 (avg 1192 sequences per MSA)
License	MIT

The model uses axial attention that alternates between row attention (across positions) and column attention (across sequences). Row attention is tied across all sequences, reducing memory from O(ML^2) to O(L^2) while enabling direct extraction of contact information.

For detailed architecture information, see MODEL.md.

Capabilities & Limitations¶

CAN be used for: - Extracting evolutionary-aware embeddings from protein family alignments - Unsupervised contact prediction from attention maps - Structure-informed representations for downstream tasks - Transfer learning with MSA-based features - Per-token embeddings for residue-level prediction tasks

CANNOT be used for: - Proteins without pre-computed MSAs (the model does not perform sequence search) - Sequence generation or design - DNA, RNA, or non-protein molecules - Multi-chain or protein complex analysis - Proteins longer than 1,024 residues

Other considerations: - Requires pre-computed MSA as input (does not perform sequence search) - Maximum sequence length: 1,024 residues - Maximum MSA depth: 256 sequences (hard limit enforced by schema validation) - Performance degrades significantly with very shallow MSAs (< 16 sequences) - Not suitable for orphan proteins without homologs - First sequence in MSA must be the query/reference sequence - All sequences must be pre-aligned to identical length - Gap character (-) and insert character (.) are supported - Contact predictions are derived from symmetrized, APC-corrected attention maps

Usage Examples¶

# Single MSA encode with contacts
from models.msa_transformer.schema import (
    MSATransformerEncodeIncludeOptions,
    MSATransformerEncodeRequest,
    MSATransformerEncodeRequestItem,
    MSATransformerEncodeRequestParams,
)

request = MSATransformerEncodeRequest(
    params=MSATransformerEncodeRequestParams(
        repr_layers=[-1],
        include=[
            MSATransformerEncodeIncludeOptions.MEAN,
            MSATransformerEncodeIncludeOptions.CONTACTS,
        ],
    ),
    items=[
        MSATransformerEncodeRequestItem(
            msa=[
                "MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVL",
                "MKAVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVL",
                "MKTVRQERLKSIIRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVL",
                "MKTVRQERLKSIVRILERSKEPVSGAQLAEE-SVSRQVIVQDIAYLRSLGYNIVATPRGYVL",
            ]
        )
    ],
)

# Per-token embeddings for residue-level analysis
request = MSATransformerEncodeRequest(
    params=MSATransformerEncodeRequestParams(
        repr_layers=[-1],
        include=[MSATransformerEncodeIncludeOptions.PER_TOKEN],
    ),
    items=[
        MSATransformerEncodeRequestItem(
            msa=[
                "MKTVRQERLKSIVRILERSKEPVSG",
                "MKAVRQERLKSIVRILERSKEPVSG",
                "MKTVRQERLKSIIRILERSKEPVSG",
            ]
        )
    ],
)

Architecture & training¶

Architecture¶

Model Type & Innovation¶

The MSA Transformer is a protein language model that operates on Multiple Sequence Alignments (MSAs) rather than single sequences. Developed by Meta AI / FAIR, it is part of the ESM (Evolutionary Scale Modeling) suite.

The key innovation is axial attention with tied row attention. Standard transformers would require O(M * L^2) memory for an MSA of M sequences and L positions. The MSA Transformer decomposes attention into: - Row attention (across positions within each sequence): Tied across all sequences in the MSA, reducing memory from O(M * L^2) to O(L^2) - Column attention (across sequences at each position): Captures evolutionary covariation between sequences

Tying row attention serves a dual purpose: it reduces computational cost and it means the attention maps can be directly interpreted as contact predictions -- residue pairs that attend strongly to each other tend to be physically close in the 3D structure.

Parameters & Layers¶

Property	Value
Architecture	Axial Transformer with tied row attention
Total parameters	100M
Layers	12
Embedding dimension	768
Attention heads	12
Pretrained model	`esm_msa1b_t12_100M_UR50S`

Training Data¶

Property	Details
Dataset	UniRef50 / UniClust30
Training set size	~26 million MSAs
Average MSA depth	~1,192 sequences per MSA
Sequence type	Protein (amino acid sequences)

Known biases: - Well-studied protein families with many homologs are over-represented - Orphan proteins and recently evolved sequences are under-represented - Prokaryotic proteins dominate due to sequencing bias

Loss Function & Objective¶

Masked language modeling (MLM): a fraction of amino acid tokens are masked, and the model learns to predict the original residue from both the sequence context (row attention) and evolutionary context (column attention).

L = -sum_i log P(x_masked_i | x_visible, MSA_context)

This objective forces the model to learn both within-sequence grammar and between-sequence covariation patterns.

Tokenization / Input Processing¶

Property	Details
Tokenizer	ESM alphabet (character-level)
Input type	Multiple Sequence Alignment (list of aligned sequences)
Amino acid alphabet	20 standard + extended AA + gap (-) + insert (.)
Special tokens	BOS, EOS
Max sequence length	1,024 residues
Max MSA depth	256 sequences
Min MSA depth	2 sequences (first is query)
Batch size	4 MSAs per request

The first sequence in each MSA is treated as the query/reference sequence. All sequences must be pre-aligned to identical length.

Performance & Benchmarks¶

Published Benchmarks¶

From Rao et al. "MSA Transformer" (ICML 2021):

Key results: - Unsupervised contact prediction achieves state-of-the-art among MSA-based methods at the time of publication - Contact maps derived from tied row attention outperform previous attention-based methods (e.g., ESM-1b attention) - Performance improves with MSA depth up to approximately 128 sequences, then plateaus

BioLM Verification Results¶

Test	Expected	Actual	Source	Status
Embedding dimension	768	768	Paper: "768 embedding size"	PASS
Number of layers	12	12	Paper: "12 layers"	PASS
Contact map symmetry	Symmetric	max_diff=1.49e-07	APC correction produces symmetric maps	PASS
Proximity bias	Short > Long	Short=0.031, Long=0.006	Expected from protein folding physics	PASS
Attention range	[0, 1]	[0.0009, 0.155]	Post-softmax values	PASS
Determinism	Identical runs	max_diff=0.0	Seeded RNG	PASS
Covariation detection	>50th percentile	89.4th percentile	Synthetic MSA with designed covariation	PASS

7/7 verification tests passed.

Comparison to Alternatives¶

Model	Type	Key Advantage	Key Disadvantage
MSA Transformer (this)	MSA-based protein LM	Captures evolutionary covariation; unsupervised contacts	Requires pre-computed MSA
ESM-2	Single-sequence protein LM	No MSA needed; fast	Misses evolutionary covariation signal
ESM-1b	Single-sequence protein LM	Predecessor; still functional	Strictly worse than ESM-2
AlphaFold2	Structure prediction	Full 3D structure output	Much heavier; requires MSA + templates

Error Bars & Confidence¶

The model is deterministic when seeds are set (torch.manual_seed(42)). Contact maps are derived from attention weights via symmetrization and APC (Average Product Correction), which are deterministic operations.

Sources of variability: - Different GPU architectures may produce floating-point differences within 1e-4 - Shallow MSAs (< 16 sequences) produce lower quality outputs

Strengths & Limitations¶

Pros¶

Captures evolutionary covariation patterns that single-sequence models miss
Unsupervised contact prediction from attention maps without separate training
Relatively lightweight (100M parameters) compared to structure prediction models
Rich output options: mean embeddings, per-token embeddings, row attentions, contacts
Efficient tied row attention reduces memory complexity

Cons¶

Requires pre-computed MSA as input (does not perform sequence search)
Performance degrades with very shallow MSAs (< 16 sequences)
Not suitable for orphan proteins without detectable homologs
Single-chain only -- no multi-chain or complex modeling
Contact predictions are approximate (not a dedicated structure predictor)
Max sequence length of 1,024 residues limits application to large proteins

Known Failure Modes¶

Shallow MSAs (< 16 sequences): Insufficient evolutionary signal leads to poor embeddings and unreliable contacts
Misaligned MSAs: Sequences not properly aligned will produce incorrect attention patterns
Very long proteins (> 1024 residues): Truncation removes C-terminal information
Orphan proteins: Proteins without detectable homologs cannot leverage the MSA-based approach
Intrinsically disordered regions: Contact predictions are unreliable for disordered regions

Implementation Details¶

Inference Pipeline¶

Request
  |-- 1. Validate MSA input:
  |     |-- All sequences same length
  |     |-- Characters in AA alphabet + gap (-) + insert (.)
  |     |-- Depth >= 2, <= 256
  |     |-- Length <= 1024
  |-- 2. Normalize repr_layers (convert negative to positive)
  |-- 3. For each MSA in batch:
  |     |-- Format as [(label, seq)] tuples
  |     |-- batch_converter -> tokens [1, M, L]
  |     |-- Forward pass on GPU:
  |     |     |-- need_head_weights if row_attention or contacts requested
  |     |     |-- return_contacts if contacts requested
  |     |-- Extract outputs per include options:
  |     |     |-- mean: query row representations averaged over positions
  |     |     |-- per_token: query row representations at each position
  |     |     |-- row_attention: tied row attention maps averaged over heads
  |     |     |-- contacts: symmetrized APC-corrected contact predictions
  |-- 4. Return MSATransformerEncodeResponse

Memory & Compute Profile¶

Property	Value
GPU	T4 (16 GB VRAM)
Memory	16 GB system RAM
CPU	4 cores
Timeout	20 minutes
Model size on disk	~400 MB

Memory scales with MSA depth and sequence length. For deep MSAs (256 sequences) of long sequences (1024 residues), GPU memory usage approaches T4 limits.

Determinism & Reproducibility¶

Setting	Value
`torch.manual_seed`	42
`torch.cuda.manual_seed_all`	42
Model mode	eval() with torch.no_grad()

All outputs are deterministic on the same hardware. Verified with max_diff=0.0 across repeated runs.

Caching Behavior¶

Response caching is handled outside the model container at the serving layer
GPU memory snapshots enabled (enable_memory_snapshot=True, enable_gpu_snapshot=True)
Cache key includes MSA sequences, params, and include options

Versions & Changelog¶

Version	Date	Changes
v1	--	Initial implementation with encode action supporting mean, per_token, row_attention, and contacts outputs

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

The MSA Transformer operates on protein Multiple Sequence Alignments (MSAs) -- collections of evolutionarily related protein sequences aligned to a common coordinate system. It is designed to capture the patterns of amino acid conservation and covariation that arise from shared evolutionary history.

The model handles: - Globular proteins: Excellent performance when deep MSAs (> 64 sequences) are available - Enzymes: Strong coverage in UniRef50; active site conservation well-captured - Membrane proteins: Functional but may have shallower MSAs due to under-representation in databases - Antibodies: Variable regions may have limited homologs; framework regions are better represented

Input requirements: - The first sequence in the MSA is the query (reference) sequence - All sequences must be pre-aligned to identical length - Gap character (-) and insert character (.) are supported - Standard and extended amino acid alphabet accepted

Cross-Applicability¶

Molecule Type	Applicability	Evidence	Caveats
Protein families (deep MSAs)	High	Core design purpose; 26M MSAs in training	Best with >= 64 sequences in MSA
Enzymes	High	Well-represented in UniRef50	Active site covariation well-captured
Antibodies	Moderate	Variable regions have limited homologs	CDRs may lack evolutionary signal; consider antibody-specific models
Orphan proteins	Low	Requires homologs for MSA construction	No benefit over single-sequence models if homologs unavailable
Peptides	Low	Very short alignments provide minimal signal	Consider peptide-specific models
DNA/RNA	Not applicable	Protein-only model	Use Evo, NT, or RNA-specific models
Protein complexes	Not supported	Single-chain MSA input only	Concatenated MSAs could work but are not validated

Biological Problems Addressed¶

Problem 1: Evolutionary Covariation Analysis¶

Why this matters: When two positions in a protein are in physical contact in the 3D structure, mutations at one position create evolutionary pressure for compensatory mutations at the other. This "covariation" signal is the basis for unsupervised protein structure prediction and has been used since the 1990s (e.g., DCA, PSICOV, Gremlin). However, traditional covariation methods require hundreds of sequences and struggle with indirect correlations.

How MSA Transformer addresses it: The tied row attention mechanism naturally learns to attend to covarying positions. By extracting and symmetrizing the attention maps, the model produces contact predictions without any supervised training on known structures. The APC (Average Product Correction) applied to the attention maps removes background correlations, revealing direct contacts.

Biological meaning: High-scoring entries in the contact map indicate residue pairs likely to be within ~8 Angstroms in the protein's 3D structure. Long-range contacts (positions far apart in sequence but close in space) are particularly informative for determining the protein's fold.

Problem 2: MSA-Aware Protein Representation Learning¶

Why this matters: Single-sequence protein language models (e.g., ESM-2) learn representations from individual sequences, capturing the "grammar" of amino acid sequences. However, the evolutionary signal encoded in MSAs -- which positions are conserved, which covary, which are free to vary -- provides additional information about protein structure and function that single-sequence models cannot directly access.

How MSA Transformer addresses it: The encode action produces embeddings that incorporate both within-sequence context (row attention) and evolutionary context (column attention). These MSA-aware embeddings capture information about: - Conservation: Positions with low variation across the MSA are likely functionally important - Covariation: Positions that change in concert across the MSA are likely structurally coupled - Insertion/deletion patterns: Gap distributions reveal loop regions and domain boundaries

Biological meaning: MSA Transformer embeddings can be more informative than single-sequence ESM-2 embeddings for tasks where evolutionary signal is important -- protein family classification, function prediction, and structure-informed applications. The tradeoff is requiring a pre-computed MSA as input.

Problem 3: Unsupervised Protein Structure Insight¶

Why this matters: Before AlphaFold2, unsupervised contact prediction from MSAs was one of the primary routes to protein structure information. While AlphaFold2 has largely superseded this for full structure prediction, fast contact maps remain useful for: - Quick structural assessment of novel proteins - Validating MSA quality (do predicted contacts match known structure?) - Identifying structurally important residue pairs for mutagenesis

How MSA Transformer addresses it: The contacts output from the encode action provides a fast, unsupervised estimate of the protein's contact map. At the time of publication (2021), this was state-of-the-art among purely unsupervised methods.

Applied Use Cases¶

Use Case 1: Unsupervised Contact Prediction (Published)¶

Source: Rao R, Liu J, Verkuil R, Meier J, Canny J, Abbeel P, Sercu T, Rives A. "MSA Transformer." ICML (2021). DOI: 10.1101/2021.02.12.430858

The paper demonstrates that tied row attention maps, after symmetrization and APC correction, produce contact predictions that outperform previous attention-based methods and approach the accuracy of dedicated covariation analysis tools.

Use Case 2: Transfer Learning with MSA Features (Anticipated)¶

Using MSA Transformer embeddings as input features for downstream classifiers -- enzyme function prediction, thermostability classification, or binding site identification -- where evolutionary context provides additional signal beyond single-sequence representations.

Complementary Models¶

ESM-2: Single-sequence protein embeddings. For proteins without deep MSAs, use ESM-2. For proteins with rich evolutionary data, MSA Transformer provides complementary evolutionary signal.
ESMFold: Uses ESM-2 embeddings for full 3D structure prediction. MSA Transformer contacts can serve as a quick structural check before running the more expensive ESMFold.
Chai-1 / RF3: Full structure prediction from sequences and MSAs. Use these when you need atomic-level 3D structures rather than contact maps.

Alternative Models¶

Alternative	Advantage over MSA Transformer	Disadvantage vs MSA Transformer
ESM-2	No MSA required; fast single-sequence inference	Cannot capture covariation; no contact prediction from attention
AlphaFold2	Full atomic 3D structure	Much heavier; requires templates; overkill for contact-level analysis
EVCouplings / DCA	Mathematically principled covariation analysis	Slower; requires very deep MSAs; no learned representations

Biological Background¶

Multiple Sequence Alignment (MSA): A collection of protein sequences from different organisms that share a common ancestor, aligned so that homologous positions are in the same column. MSAs are the primary data structure for studying protein evolution.

Key concepts relevant to MSA Transformer:

Evolutionary covariation: When two positions in a protein are structurally coupled (e.g., in physical contact), a mutation at one position often requires a compensatory mutation at the other to maintain function. This creates statistical correlations in the MSA that can be detected computationally.
Contact prediction: Predicting which residue pairs are physically close (< 8 Angstroms) in the 3D structure from sequence information alone. Long-range contacts (positions separated by > 12 residues in sequence) are the most informative for fold determination.
Axial attention: A decomposition of attention into row (across positions) and column (across sequences) components, reducing computational complexity while preserving the ability to capture both within-sequence and between-sequence relationships.
Tied row attention: Sharing attention weights across all sequences in the MSA. This constraint forces the model to learn position-level patterns that are consistent across the entire alignment, naturally encoding structural constraints.
APC (Average Product Correction): A statistical correction applied to covariation scores that removes the effect of phylogenetic bias and uneven amino acid composition, revealing direct contacts.
MSA depth: The number of sequences in an alignment. Deeper MSAs provide stronger covariation signal. Performance degrades significantly below 16 sequences.

Sources & license¶

License: MIT (text) — Part of the ESM (Evolutionary Scale Modeling) suite from Meta AI

Papers

MSA Transformer — ICML 2021 · DOI

Source repositories

github: https://github.com/facebookresearch/esm

Cite

@article{rao2021msa,
  title={MSA Transformer},
  author={Rao, Roshan and Liu, Jason and Verkuil, Robert and Meier, Joshua and
          Canny, John and Abbeel, Pieter and Sercu, Tom and Rives, Alexander},
  journal={bioRxiv},
  doi={10.1101/2021.02.12.430858},
  year={2021}
}

MSA Transformer¶

At a glance¶

API & schema¶

encode¶

Usage¶

Overview¶

Architecture¶

Capabilities & Limitations¶

Usage Examples¶

Architecture & training¶

Architecture¶

Model Type & Innovation¶

Parameters & Layers¶

Training Data¶

Loss Function & Objective¶

Tokenization / Input Processing¶

Performance & Benchmarks¶

Published Benchmarks¶

BioLM Verification Results¶

Comparison to Alternatives¶

Error Bars & Confidence¶

Strengths & Limitations¶

Pros¶

Cons¶

Known Failure Modes¶

Implementation Details¶

Inference Pipeline¶

Memory & Compute Profile¶

Determinism & Reproducibility¶

Caching Behavior¶

Versions & Changelog¶

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

Cross-Applicability¶

Biological Problems Addressed¶

Problem 1: Evolutionary Covariation Analysis¶

Problem 2: MSA-Aware Protein Representation Learning¶

Problem 3: Unsupervised Protein Structure Insight¶

Applied Use Cases¶

Use Case 1: Unsupervised Contact Prediction (Published)¶

Use Case 2: Transfer Learning with MSA Features (Anticipated)¶

Related Models¶

Complementary Models¶

Alternative Models¶

Biological Background¶

Sources & license¶

`encode`¶