ProstT5¶

Bilingual T5-based protein language model that translates between amino acid sequences and Foldseek 3Di structural alphabet tokens, enabling structure-aware embeddings and inverse folding.

License: MIT · Molecules: protein · Tasks: embedding, sequence_generation

Actions: encode, generate · Variants: 4

At a glance¶

Use it when

You need rapid structural annotation of large protein databases — ProstT5 generates 3Di tokens from sequence at three orders of magnitude faster than running AlphaFold2
You need structure-aware protein embeddings for remote homology detection, fold classification, or function annotation where structural similarity matters more than sequence similarity
You want to design novel protein sequences conditioned on target structural constraints (3Di tokens) — ProstT5's inverse folding capability generates diverse sequence candidates
You need to search structural databases (via Foldseek) for query proteins without running full 3D structure prediction — generate 3Di tokens with ProstT5 for fast Foldseek search
You are studying protein structure-sequence relationships and need a model that explicitly captures the bidirectional mapping between amino acid language and structural alphabet language
You are annotating metagenomic protein sequences at scale where structural family assignment from sequence alone provides more informative classification than sequence-based clustering

Strengths

Bilingual sequence-structure translation — uniquely provides bidirectional conversion between amino acid sequences and Foldseek 3Di structural tokens, enabling both forward folding (AA to 3Di) and inverse folding (3Di to AA)
Structure-aware embeddings for remote homology detection — achieves 92.2% accuracy on CATH superfamily classification with 9.9% F1 improvement over sequence-only ProtT5 (CATHe2, 2025)
Three orders of magnitude faster than AlphaFold2 for deriving 3Di structural representations from sequence, enabling rapid structural annotation of large protein databases
T5 encoder-decoder architecture enables generative capability — can design novel amino acid sequences conditioned on target 3Di structural constraints (inverse folding)
MIT license with no commercial restrictions, suitable for any deployment scenario
Fine-tuned from the 3-billion-parameter ProtT5-XL-U50 model (pretrained on UniRef50, ~45M sequences), inheriting strong sequence modeling while adding structural translation capability
Validated for protein-protein binding site prediction (GraphPBSP, 2024) outperforming state-of-the-art methods, demonstrating utility beyond fold classification
Two complementary actions (encode, generate) with direction control (AA2fold, fold2AA) provide flexible workflows for both analysis and design tasks

Limitations

Requires 3Di structural tokens as input for structure-to-sequence direction — users must either run Foldseek on existing structures or use the model's own AA2fold prediction, adding pipeline complexity
Generated amino acid sequences from inverse folding (fold2AA) are predictions, not experimentally validated designs — output quality varies with generation parameters and should be treated as starting points for experimental testing
Structure prediction via 3Di tokens is lower resolution than dedicated atomic-level structure predictors (AlphaFold2, Chai-1, ESMFold) — captures local geometry but not precise atomic coordinates
No dedicated variant effect scoring action — cannot directly compute fitness scores for mutations like ESM1v or PoET
Single size variant only — no scaling flexibility across different compute budgets
Performance degrades for intrinsically disordered proteins where 3Di tokens have ambiguous assignments, reducing structure-aware embeddings to near-sequence-only quality
Requires L4 GPU (24GB VRAM) — higher compute requirement than ESM2-650M (T4) or MSA Transformer (T4)
Variant axes create multiple deployments (encode x AA2fold, encode x fold2AA, generate x AA2fold, generate x fold2AA), increasing operational complexity

Reach for something else when

You need accurate atomic-level 3D protein structure prediction — use Chai-1, RF3, or ESMFold, which produce full coordinate sets rather than structural alphabet tokens
You need variant effect prediction scores — use ESM1v, ESM2 log_prob, or PoET score, which are purpose-built for quantitative mutation scoring
You need general-purpose protein embeddings without structural considerations — use ESM2 or ESMC, which are simpler, faster, and produce high-quality sequence embeddings
You need MSA-aware analysis that leverages evolutionary covariation from multiple sequence alignments — use MSA Transformer or PoET
You are working primarily with intrinsically disordered proteins where structural token assignments are unreliable
You need a single model that covers embeddings, variant scoring, and atomic structure prediction together — no single catalog model does all three; combine ESM2 or ESMC (embeddings, scoring) with a structure predictor such as ESMFold or Chai-1
You need dedicated structure-conditioned inverse folding with atomic-level structural input — use ProteinMPNN, which accepts full PDB coordinates rather than structural alphabet tokens

Alternatives

Model	Better when	Worse when
esm2	Simpler, faster sequence-only protein embeddings with five size variants and lower GPU needs; broader action set (encode/predict/log_prob/contacts); MIT.	No structure-aware embeddings and no sequence-structure translation — misses the fold-level signal ProstT5 encodes directly from sequence.
esmc	State-of-the-art protein sequence embeddings and pseudo-log-likelihood scoring with no structural input dependency; simpler for pure sequence analysis.	No structural awareness and no generative/inverse-folding capability; ProstT5 uniquely bridges the sequence-structure gap bidirectionally.

API & schema¶

Variants

Variant	Endpoint slug	GPU	CPU	Memory
encode-fold2AA	`prostt5-fold2aa-encode`	l4	4.0	16 GB
encode-AA2fold	`prostt5-aa2fold-encode`	l4	4.0	16 GB
generate-fold2AA	`prostt5-fold2aa-generate`	l4	4.0	16 GB
generate-AA2fold	`prostt5-aa2fold-generate`	l4	4.0	16 GB

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

`encode`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/prostt5-fold2aa-encode/encode \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ProstT5EncodeRequestAA

Field	Type	Required	Constraints	Description
`items`	list[ProstT5EncodeRequestItemAA]	yes	items 1–16	Batch of inputs to process in a single request. Up to 16 sequences per request.

Nested types

ProstT5EncodeRequestItemAA

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–1000	A protein sequence in single-letter amino-acid codes.

Raw JSON Schema

{
  "$defs": {
    "ProstT5EncodeRequestItemAA": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes.",
          "maxLength": 1000,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ProstT5EncodeRequestItemAA",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 16 sequences per request.",
      "items": {
        "$ref": "#/$defs/ProstT5EncodeRequestItemAA"
      },
      "maxItems": 16,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ProstT5EncodeRequestAA",
  "type": "object"
}

Response — ProstT5EncodeResponse

Field	Type	Required	Constraints	Description
`results`	list[ProstT5EncodeResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

ProstT5EncodeResponseResult

Field	Type	Required	Constraints	Description
`embeddings`	list[number]	yes		Mean-pooled 1024-dimensional embedding vector for the input sequence.

Raw JSON Schema

{
  "$defs": {
    "ProstT5EncodeResponseResult": {
      "properties": {
        "embeddings": {
          "description": "Mean-pooled 1024-dimensional embedding vector for the input sequence.",
          "items": {
            "type": "number"
          },
          "title": "Embeddings",
          "type": "array"
        }
      },
      "required": [
        "embeddings"
      ],
      "title": "ProstT5EncodeResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ProstT5EncodeResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ProstT5EncodeResponse",
  "type": "object"
}

`generate`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/prostt5-fold2aa-encode/generate \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ProstT5GenerateRequestAA

Field	Type	Required	Constraints	Description
`params`	ProstT5GenerateParamsAA	no		Optional parameters controlling this action (defaults are used when omitted).
`items`	list[ProstT5GenerateRequestItemAA]	yes	items 1–2	Batch of inputs to process in a single request. Up to 2 sequences per request.

Nested types

ProstT5GenerateParamsAA

Field	Type	Required	Constraints	Description
`temperature`	number	no	≥0.0; ≤8.0; default `1.2`	Sampling temperature; higher values increase diversity.
`top_p`	number	no	≥0.0; ≤1.0; default `0.95`	Nucleus (top-p) sampling threshold.
`top_k`	integer	no	≥1; ≤20; default `6`	Top-k sampling cutoff; only the k most likely tokens are sampled.
`repetition_penalty`	number	no	≥0.0; ≤3.0; default `1.2`	Repetition penalty applied during generation; values > 1.0 discourage repeated tokens.
`num_samples`	integer	no	≥1; ≤3; default `1`	Number of sequences to generate per input.
`num_beams`	integer	no	≥1; ≤3; default `3`	Beam search width for AA2fold translation; wider beams improve quality at higher compute cost.
`seed`	integer \| null	no		Random seed for reproducible sampling.

ProstT5GenerateRequestItemAA

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–512	A protein sequence in single-letter amino-acid codes.

Raw JSON Schema

{
  "$defs": {
    "ProstT5GenerateParamsAA": {
      "additionalProperties": false,
      "properties": {
        "temperature": {
          "default": 1.2,
          "description": "Sampling temperature; higher values increase diversity.",
          "maximum": 8.0,
          "minimum": 0.0,
          "title": "Temperature",
          "type": "number"
        },
        "top_p": {
          "default": 0.95,
          "description": "Nucleus (top-p) sampling threshold.",
          "maximum": 1.0,
          "minimum": 0.0,
          "title": "Top P",
          "type": "number"
        },
        "top_k": {
          "default": 6,
          "description": "Top-k sampling cutoff; only the k most likely tokens are sampled.",
          "maximum": 20,
          "minimum": 1,
          "title": "Top K",
          "type": "integer"
        },
        "repetition_penalty": {
          "default": 1.2,
          "description": "Repetition penalty applied during generation; values > 1.0 discourage repeated tokens.",
          "maximum": 3.0,
          "minimum": 0.0,
          "title": "Repetition Penalty",
          "type": "number"
        },
        "num_samples": {
          "default": 1,
          "description": "Number of sequences to generate per input.",
          "maximum": 3,
          "minimum": 1,
          "title": "Num Samples",
          "type": "integer"
        },
        "num_beams": {
          "default": 3,
          "description": "Beam search width for AA2fold translation; wider beams improve quality at higher compute cost.",
          "maximum": 3,
          "minimum": 1,
          "title": "Num Beams",
          "type": "integer"
        },
        "seed": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Random seed for reproducible sampling.",
          "title": "Seed"
        }
      },
      "title": "ProstT5GenerateParamsAA",
      "type": "object"
    },
    "ProstT5GenerateRequestItemAA": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes.",
          "maxLength": 512,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ProstT5GenerateRequestItemAA",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "params": {
      "$ref": "#/$defs/ProstT5GenerateParamsAA",
      "description": "Optional parameters controlling this action (defaults are used when omitted)."
    },
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 2 sequences per request.",
      "items": {
        "$ref": "#/$defs/ProstT5GenerateRequestItemAA"
      },
      "maxItems": 2,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ProstT5GenerateRequestAA",
  "type": "object"
}

Response — ProstT5GenerateResponse

Field	Type	Required	Constraints	Description
`results`	list[list[ProstT5GenerateResponseGenerated]]	yes		Per-input results, returned in the same order as the request items.

Nested types

ProstT5GenerateResponseGenerated

Field	Type	Required	Constraints	Description
`sequence`	string	yes		A generated sequence in the target alphabet (uppercase amino acids for fold2AA; lowercase 3Di tokens for AA2fold).

Raw JSON Schema

{
  "$defs": {
    "ProstT5GenerateResponseGenerated": {
      "properties": {
        "sequence": {
          "description": "A generated sequence in the target alphabet (uppercase amino acids for fold2AA; lowercase 3Di tokens for AA2fold).",
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ProstT5GenerateResponseGenerated",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "items": {
          "$ref": "#/$defs/ProstT5GenerateResponseGenerated"
        },
        "type": "array"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ProstT5GenerateResponse",
  "type": "object"
}

Usage¶

One-line summary: Bilingual T5-based protein language model that translates between amino acid sequences and Foldseek 3Di structural alphabet tokens, enabling structure-aware embeddings and inverse folding.

Overview¶

ProstT5 is a bilingual protein language model developed by Heinzinger et al. (2023) at the Rostlab (TU Munich). Built on the ProtT5-XL-U50 architecture, it is fine-tuned to translate between amino acid sequences and 3Di structural tokens (the 20-letter alphabet from Foldseek that encodes local 3D geometry). ProstT5 supports two actions -- encode (embedding extraction) and generate (sequence translation) -- in both directions (AA2fold and fold2AA).

Architecture¶

Property	Value
Architecture	T5 encoder-decoder Transformer (ProtT5-XL-U50 backbone)
Parameters	~3B
Embedding dimension	1024
AA vocabulary	Standard 20 amino acids (uppercase) + X
3Di vocabulary	20-letter structural alphabet (lowercase: acdefghiklmnpqrstvwy)
Direction tokens	`<AA2fold>`, `<fold2AA>`
HuggingFace model	Rostlab/ProstT5

Capabilities & Limitations¶

CAN be used for: - Extracting structure-aware protein embeddings (1024-dim) from amino acid sequences (AA2fold encode) - Extracting embeddings from 3Di structural token sequences (fold2AA encode) - Translating amino acid sequences to 3Di structural tokens (AA2fold generate) - Translating 3Di structural tokens to amino acid sequences -- inverse folding (fold2AA generate) - Protein fold classification and remote homology detection via embeddings - Structural annotation of large protein databases without full 3D prediction

CANNOT be used for: - Full 3D atomic structure prediction (use AlphaFold2, ESMFold, or RF3) - Nucleic acid or small molecule analysis - Multi-chain complex modeling - Per-residue confidence scores (use structure prediction models for pLDDT)

Usage Examples¶

Encode amino acid sequences (AA2fold)¶

from models.prostt5.schema import (
    ProstT5EncodeRequestAA,
    ProstT5EncodeRequestItemAA,
)

request = ProstT5EncodeRequestAA(
    items=[
        ProstT5EncodeRequestItemAA(sequence="MKLLILAVVFTVFGTAVQAAYPYDDAAQLTEEQR"),
    ]
)

Encode 3Di structural tokens (fold2AA)¶

from models.prostt5.schema import (
    ProstT5EncodeRequestFold,
    ProstT5EncodeRequestItemFold,
)

request = ProstT5EncodeRequestFold(
    items=[
        ProstT5EncodeRequestItemFold(sequence="dddahklqppddvvddddahhppllddddefgh"),
    ]
)

Generate 3Di from amino acids (AA2fold)¶

from models.prostt5.schema import (
    ProstT5GenerateRequestAA,
    ProstT5GenerateRequestItemAA,
    ProstT5GenerateParamsAA,
)

request = ProstT5GenerateRequestAA(
    params=ProstT5GenerateParamsAA(
        temperature=1.2,
        top_p=0.95,
        num_samples=2,
        seed=42,
    ),
    items=[
        ProstT5GenerateRequestItemAA(sequence="MKLLILAVVFTVFGTAVQAAYPYDDAAQLTEEQR"),
    ],
)

Inverse folding: generate amino acids from 3Di (fold2AA)¶

from models.prostt5.schema import (
    ProstT5GenerateRequestFold,
    ProstT5GenerateRequestItemFold,
    ProstT5GenerateParamsFold,
)

request = ProstT5GenerateRequestFold(
    params=ProstT5GenerateParamsFold(
        temperature=1.0,
        num_samples=3,
        seed=42,
    ),
    items=[
        ProstT5GenerateRequestItemFold(sequence="dddahklqppddvvddddahhppllddddefgh"),
    ],
)

Architecture & training¶

Architecture¶

Model Type & Innovation¶

ProstT5 is a bilingual protein language model based on the T5 (Text-to-Text Transfer Transformer) architecture that translates between two "languages": amino acid sequences and 3Di structural alphabet tokens. 3Di tokens are a 20-letter structural alphabet introduced by Foldseek that encodes the local 3D geometry around each residue into a single character.

The key innovation is framing structure-sequence relationships as a sequence-to-sequence translation problem. ProstT5 can: 1. Encode: Produce fixed-length embeddings from amino acid sequences or 3Di structure tokens 2. Generate: Translate amino acid sequences to 3Di tokens (AA2fold) or 3Di tokens to amino acid sequences (fold2AA)

This bilingual capability means the model implicitly learns protein structure through its translation objective, without requiring explicit 3D coordinates during training. The learned representations capture both sequence and structural information simultaneously.

Parameters & Layers¶

Component	Details
Architecture	T5 (encoder-decoder Transformer)
Base model	ProtT5-XL-U50 (3B parameters)
Pre-training	Masked language modeling on UniRef50
Fine-tuning	Translation between amino acid sequences and 3Di tokens
Encoder output	1024-dimensional embeddings per token
Vocabulary	Standard amino acids (uppercase) + 3Di alphabet (lowercase, 20 characters: acdefghiklmnpqrstvwy)
Direction tokens	`<AA2fold>` (sequence to structure) and `<fold2AA>` (structure to sequence)
HuggingFace model	Rostlab/ProstT5

Training Data¶

Property	Details
Pre-training	ProtT5-XL-U50 on UniRef50 (~45M sequences)
Structure annotations	3Di tokens generated by Foldseek from AlphaFold DB structures
Translation pairs	~1M protein sequences paired with their 3Di structural representations
Structural coverage	AlphaFold Database predicted structures

Loss Function & Objective¶

ProstT5 is trained with two complementary objectives: 1. Masked language modeling (pre-training): Standard T5 span corruption on amino acid sequences 2. Translation (fine-tuning): Cross-entropy loss for translating between amino acid and 3Di vocabularies in both directions (AA->3Di and 3Di->AA)

Tokenization / Input Processing¶

Amino acid sequences: Uppercase single-letter codes, space-separated (e.g., "M K L L I S")
3Di structural tokens: Lowercase 20-letter alphabet (acdefghiklmnpqrstvwy), space-separated
Direction prefix: <AA2fold> or <fold2AA> prepended to indicate translation direction
Ambiguous residues: U, Z, O, B replaced with X before tokenization
Tokenizer: T5Tokenizer from HuggingFace Transformers
Padding: Sequences padded to longest in batch

Performance & Benchmarks¶

Published Benchmarks¶

From Heinzinger et al., bioRxiv (2023):

Embedding Quality (AA2fold direction)¶

Task	ProstT5	ProtT5 (AA-only)	Notes
Fold-level classification	88.8% accuracy	85.1%	On SCOP superfamily benchmark
Secondary structure prediction	Q3 ~82%	Q3 ~81%	Comparable to AA-only
Remote homology detection	Improved	Baseline	Structure-aware embeddings capture fold similarity

Translation Quality¶

Direction	Metric	Value	Notes
AA -> 3Di	Per-token accuracy	~65-70%	Captures dominant structural elements
3Di -> AA	Sequence recovery	~35-45%	Multiple valid AA for each structure
Inverse folding (3Di->AA)	Perplexity	Lower than random	Generates structurally compatible sequences

BioLM Verification Results¶

Variant	Action	Tolerance	Status
fold2AA / encode	encode	rel_tol 1e-3, cosine distance < 0.02	PASS
AA2fold / encode	encode	rel_tol 1e-3, cosine distance < 0.02	PASS
fold2AA / generate	generate	Length match, case validation	PASS
AA2fold / generate	generate	Length match, case validation	PASS

Comparison to Alternatives¶

Model	Strength	When to prefer
ProstT5	Bilingual (seq+struct); translation capability	When structural alphabet representation is needed
ESM2	More extensively validated embeddings	General-purpose protein representations
ProtT5	AA-only; same architecture	When structural information is not needed
SaProt	Structure-aware with SA tokens	Similar bilingual approach

Strengths & Limitations¶

Pros¶

Bilingual: jointly models amino acid sequences and structural tokens
Translation capability enables sequence design from structural constraints
Embeddings capture both sequence and structural information
Based on well-validated T5/ProtT5 architecture
Multiple generation controls (temperature, top_p, top_k, repetition_penalty, beam search)
Deterministic encoding; stochastic generation with seed control

Cons¶

3Di tokens are an approximation of 3D structure (20-letter alphabet, not full atomic detail)
Translation accuracy limited (~65-70% per-token for AA->3Di)
Sequence generation from structure is highly stochastic (many valid sequences per structure)
Maximum sequence length of 1000 (encode) or 512 (generate)
Requires GPU (L4) for inference
Four separate deployments needed for all direction/action combinations

Known Failure Modes¶

Sequences > 512 residues in generate mode may produce length mismatches between input and output
Very long sequences near the 1000-residue limit may have degraded embedding quality at the edges
Non-standard amino acids (U, Z, O, B) are replaced with X, which may affect local predictions
Beam search in fold2AA direction may generate overly repetitive sequences without sufficient temperature

Implementation Details¶

Inference Pipeline¶

Encode Action¶

Request
  |-- 1. Validate sequences (AA or 3Di alphabet based on direction)
  |-- 2. Add white-space between residues
  |-- 3. Prepend direction token (<AA2fold> or <fold2AA>)
  |-- 4. Tokenize with T5Tokenizer (pad to longest)
  |-- 5. Forward pass through T5EncoderModel
  |-- 6. Extract last hidden state, trim padding and special tokens
  |-- 7. Compute mean-pool embedding (1024-dim per protein)
  |-- 8. Return embeddings

Generate Action¶

Request
  |-- 1. Validate sequences (AA or 3Di alphabet based on direction)
  |-- 2. Set random seeds (user-specified or time-based)
  |-- 3. Add white-space between residues
  |-- 4. Prepend direction token
  |-- 5. Build bad_words list (prevent generating tokens from wrong vocabulary)
  |-- 6. Tokenize and pad
  |-- 7. Call model.generate() with sampling parameters
  |-- 8. Decode output tokens, remove white-space
  |-- 9. Handle length mismatches (truncate or pad with 'd')
  |-- 10. Return generated sequences

Memory & Compute Profile¶

Resource	Value
GPU	L4
Memory	16 GB RAM
CPU	4.0 cores
Encode batch size	16
Generate batch size	2
Max encode sequence length	1000
Max generate sequence length	512
Max beam width	3

Determinism & Reproducibility¶

Setting	Value
Encode action	Deterministic (model in eval mode, no random sampling)
Generate action	Stochastic by default (temperature-based sampling)
User-specified seed	Supported for generate (controls torch, numpy, random)
Time-based seed	Used when seed is None (for diversity)

Caching Behavior¶

Encode results are fully deterministic (eval mode, no random sampling) and benefit strongly from response caching. Generate results vary by seed and are typically not cached.

Versions & Changelog¶

Version	Date	Changes
v1	2025	Initial implementation with encode and generate actions, both directions

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

ProstT5 is designed for proteins -- it operates on amino acid sequences and their corresponding Foldseek 3Di structural alphabet representations. The model is applicable to any protein that can be represented as: - A standard amino acid sequence (uppercase, 20-letter alphabet + X for ambiguous) - A 3Di structural token sequence (lowercase, 20-letter alphabet: acdefghiklmnpqrstvwy)

The model does not directly handle nucleic acids, small molecules, or multi-chain complexes. It treats each protein as an independent sequence.

Cross-Applicability¶

Molecule Type	Applicability	Evidence	Caveats
Globular proteins	High	Primary training target	Well-represented in AlphaFold DB
Enzymes	High	Well-represented in training data	3Di tokens capture active site geometry
Membrane proteins	Moderate	Included in AlphaFold DB structures	Less diverse training examples
Disordered proteins	Low	3Di alphabet assumes local structure exists	Disordered regions have ambiguous 3Di assignments
Antibodies	Moderate	General protein coverage	Specialized models (ImmuneBuilder) may be better for CDR loops
Peptides	Low	Short sequences have limited context	3Di tokens need sufficient local context
Nucleic acids	Not applicable	Not in training vocabulary	Use specialized models
Small molecules	Not applicable	Not in training vocabulary	Use RDKit or molecular docking

Biological Problems Addressed¶

Protein Structure Representation via Structural Alphabet¶

Biological context: Protein 3D structures are traditionally represented as sets of atomic coordinates, which are high-dimensional and computationally expensive to process. The Foldseek 3Di alphabet compresses local 3D geometry into a single character per residue, creating a "structural sequence" that can be processed by sequence models. This dramatically reduces the complexity of structural comparison and search.

How ProstT5 helps: ProstT5 learns the bidirectional mapping between amino acid sequences and 3Di structural tokens. The encode action in AA2fold direction produces embeddings that capture both sequence and structural information, enabling structure-aware downstream predictions (fold classification, function annotation) without requiring explicit 3D coordinates.

Output interpretation: For the encode action, the 1024-dimensional mean representation captures the global structural/sequence properties of the protein. These embeddings can be used for clustering, classification, or as features for downstream ML models.

Protein Fold Classification and Remote Homology Detection¶

Biological context: Identifying the structural fold of a protein is essential for function prediction, as proteins with similar folds often share evolutionary relationships and biochemical functions, even when sequence similarity is low (remote homology). Traditional methods rely on sequence alignment, which fails for remote homologs with <20% sequence identity.

How ProstT5 helps: Because ProstT5 embeddings encode structural information learned from the translation task, they are better at detecting remote homology than pure sequence-based embeddings. Proteins with similar folds but different sequences will have similar ProstT5 embeddings, enabling classification of proteins into structural families.

Inverse Folding (Structure-to-Sequence Translation)¶

Biological context: Given a desired protein structure (represented as 3Di tokens from Foldseek), predicting which amino acid sequences could adopt that structure is the inverse folding problem. This is central to protein design: it allows computational generation of novel sequences that fold into target structures.

How ProstT5 helps: The generate action in fold2AA direction takes a 3Di structural sequence and generates amino acid sequences predicted to adopt that structure. Multiple samples can be generated with different temperatures and sampling strategies to explore the sequence space of structurally compatible designs.

Output interpretation: Generated amino acid sequences are uppercase strings of the same length as the input 3Di sequence. Higher temperature produces more diverse sequences; lower temperature produces sequences more similar to the training distribution. Multiple samples should be generated and evaluated experimentally.

Forward Folding (Sequence-to-Structure Translation)¶

Biological context: Predicting the 3Di structural representation from an amino acid sequence is a lightweight alternative to full 3D structure prediction. While less detailed than atomic-level prediction (AlphaFold2, ESMFold), 3Di predictions enable rapid structural annotation and comparison using Foldseek.

How ProstT5 helps: The generate action in AA2fold direction translates amino acid sequences to 3Di tokens. This enables rapid structural annotation of large protein databases without running full structure prediction. The output 3Di sequences can be searched against structural databases using Foldseek.

Applied Use Cases¶

Published Applications¶

Structural database search: Generate 3Di tokens from query sequences for Foldseek structural search without running AlphaFold2
Protein function prediction: Use ProstT5 embeddings as features for function classifiers
Fold-level classification: Assign SCOP/CATH fold families using structure-aware embeddings

Anticipated Use Cases¶

Protein design: Generate diverse sequences from target 3Di structures as starting points for experimental validation
Metagenomic annotation: Rapidly annotate structural families for millions of metagenomic sequences
Evolutionary analysis: Use structural embeddings to detect remote homology in protein family studies

Published applications include: CATHe2 (Mouret & Abbass, 2025, Biology Methods and Protocols) using ProstT5 3Di embeddings for remote homology detection across ~1700 CATH superfamilies; GraphPBSP (2024, International Journal of Biological Macromolecules) for protein binding site prediction; and Phold (2026, Nucleic Acids Research) for phage genome annotation via 3Di-based Foldseek search of 1.36M phage proteins.

Predecessor Models¶

ProtT5-XL-U50 (Elnaggar et al., 2022): The amino acid-only protein language model that ProstT5 is fine-tuned from. ProtT5 provides strong sequence embeddings but lacks structural awareness.
ESM-2 (Lin et al., 2023): Alternative protein language model with similar embedding capabilities. ProstT5 differs by explicitly modeling the structure-sequence relationship via translation.

Complementary Models¶

Foldseek: The structural alignment tool that defines the 3Di alphabet ProstT5 operates on. Foldseek can convert PDB/mmCIF structures to 3Di tokens for ProstT5 input.
AlphaFold2 / ESMFold: Full 3D structure prediction models. Their predicted structures can be converted to 3Di tokens for ProstT5 encoding.
ESM2: Alternative embedding model. ProstT5 and ESM2 embeddings capture complementary information and can be combined for downstream tasks.

Alternative Models¶

Alternative	Advantage over ProstT5	Disadvantage vs ProstT5
ESM2	Larger models available (3B); wider validation	No structural alphabet translation capability
ProtT5	Simpler (AA-only); same base architecture	Lacks structure-aware embeddings
SaProt	Similar bilingual approach with SA tokens	Different structural alphabet (not 3Di)
ESMFold	Full 3D structure prediction	Heavier compute; no embedding generation mode

Biological Background¶

The 3Di Structural Alphabet¶

The 3Di alphabet, introduced by van Kempen et al. (2023) in Foldseek, encodes the local 3D geometry around each residue using a 20-letter alphabet. Each 3Di token captures: - Backbone dihedral angles (phi, psi, omega) - Relative orientations of adjacent residues - Local structural context (secondary structure, loop geometry)

The 20 3Di states roughly correspond to different local structural environments: alpha-helix interior, beta-strand, various turn types, and loop conformations. Two proteins with identical 3Di sequences share very similar local structures throughout their chains.

Protein Structure-Sequence Relationships¶

The relationship between amino acid sequence and protein structure is many-to-many: - Forward folding: One sequence typically folds into one dominant structure (with thermal fluctuations) - Inverse folding: Many different sequences can adopt the same 3D structure

ProstT5 captures this asymmetry: AA->3Di translation is relatively accurate (the structure is largely determined by the sequence), while 3Di->AA translation is inherently stochastic (many sequences are compatible with a given structure). The generation parameters (temperature, top_k, top_p) control the diversity of the generated sequences.

Applications of Structural Alphabets in Bioinformatics¶

Structural alphabets enable treating structure comparison as a sequence comparison problem, which is dramatically faster. Applications include: - Foldseek: Structural database search in seconds (vs hours for coordinate-based methods) - Fold classification: Assigning proteins to structural families using sequence alignment tools - Structural clustering: Grouping proteins by structural similarity at genome/metagenome scale - Structure-conditioned generation: Designing proteins with desired structural properties

ProstT5 extends these applications by providing a neural model that can translate between the two languages, enabling structure-aware protein understanding without explicit 3D coordinates.

Sources & license¶

License: MIT (text)

Papers

ProstT5: Bilingual Language Model for Protein Sequence and Structure — bioRxiv preprint, 2023 · arXiv
Bilingual language model for protein sequence and structure — NAR Genomics and Bioinformatics, 2024 · DOI

Source repositories

github: https://github.com/mheinzinger/ProstT5
huggingface: https://huggingface.co/Rostlab/ProstT5

Cite

@article{heinzinger2023prostt5,
  title={ProstT5: Bilingual Language Model for Protein Sequence and Structure},
  author={Heinzinger, Michael and Weissenow, Konstantin and Gomez Sanchez, Joaquin and Hartmann, Adrian and Steinegger, Martin and Rost, Burkhard},
  journal={bioRxiv},
  year={2023},
  eprint={2310.07083},
  archivePrefix={arXiv}
}

ProstT5¶

At a glance¶

API & schema¶

encode¶

generate¶

Usage¶

Overview¶

Architecture¶

Capabilities & Limitations¶

Usage Examples¶

Encode amino acid sequences (AA2fold)¶

Encode 3Di structural tokens (fold2AA)¶

Generate 3Di from amino acids (AA2fold)¶

Inverse folding: generate amino acids from 3Di (fold2AA)¶

Architecture & training¶

Architecture¶

Model Type & Innovation¶

Parameters & Layers¶

Training Data¶

Loss Function & Objective¶

Tokenization / Input Processing¶

Performance & Benchmarks¶

Published Benchmarks¶

Embedding Quality (AA2fold direction)¶

Translation Quality¶

BioLM Verification Results¶

Comparison to Alternatives¶

Strengths & Limitations¶

Pros¶

Cons¶

Known Failure Modes¶

Implementation Details¶

Inference Pipeline¶

Encode Action¶

Generate Action¶

Memory & Compute Profile¶

Determinism & Reproducibility¶

Caching Behavior¶

Versions & Changelog¶

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

Cross-Applicability¶

Biological Problems Addressed¶

Protein Structure Representation via Structural Alphabet¶

Protein Fold Classification and Remote Homology Detection¶

Inverse Folding (Structure-to-Sequence Translation)¶

Forward Folding (Sequence-to-Structure Translation)¶

Applied Use Cases¶

Published Applications¶

Anticipated Use Cases¶

Related Models¶

Predecessor Models¶

Complementary Models¶

Alternative Models¶

Biological Background¶

The 3Di Structural Alphabet¶

Protein Structure-Sequence Relationships¶

Applications of Structural Alphabets in Bioinformatics¶

Sources & license¶

`encode`¶

`generate`¶