ProGen2

Autoregressive protein language model from Salesforce Research that generates novel protein sequences and computes sequence log-likelihoods for fitness prediction.

License: BSD-3-Clause · Molecules: protein · Tasks: sequence_generation

Actions: generate · Variants: 4

At a glance

Use it when

• You need general-purpose autoregressive protein sequence generation from a context seed, with controllable diversity through temperature and top-p sampling
• You are designing antibody variable region sequences and need a model trained on the largest available antibody sequence database (the OAS variant, trained on 554M sequences clustered at 85% identity from OAS's ~1.5B raw sequences)
• You need to generate diverse protein sequence libraries for experimental screening, ranking candidates by bidirectional log-likelihood scores
• You are exploring protein sequence space around a known protein family by extending seed sequences from conserved regions
• You need protein fitness scoring via log-likelihood to rank mutant libraries or guide directed evolution campaigns without task-specific training data
• You want to generate candidate enzymes for experimental testing and plan to validate with structure prediction (ESMFold) and stability analysis (TemBERTure) downstream
• You need a commercially licensed autoregressive protein generator for production pipelines

Strengths

• Four specialized variants (OAS, medium, large, BFD90) cover distinct protein domains — antibodies (OAS), general proteins (medium/large), and metagenomic proteins (BFD90) — enabling targeted generation for different use cases
• Bidirectional log-likelihood scoring (forward + reverse average) reduces positional bias inherent in left-to-right autoregressive models, providing more robust fitness prediction
• Published in Cell Systems (2023) with extensive protein generation benchmarks, including scaling law analysis showing predictable quality improvement with model size
• Context-conditioned generation enables directed sequence extension from a seed, mimicking natural sequence diversification for directed evolution applications
• Temperature and top-p sampling controls allow fine-grained diversity tuning — from conservative (low temperature, close to natural) to exploratory (high temperature, novel sequences)
• OAS variant trained on 554M antibody sequences (clustered at 85% identity from ~1.5B raw OAS sequences) provides the most comprehensive antibody generation model available in this catalog
• BSD-3-Clause license permits commercial use with minimal restrictions
• Benchmarked in Nature Biotechnology (2024) enzyme design evaluation, demonstrating practical relevance for computational enzyme engineering workflows

Limitations

• No explicit function conditioning — cannot specify target enzymatic activity (EC number), binding target, or other functional constraints during generation; only context-conditioned via seed sequence
• Generation quality varies significantly by protein type — membrane proteins, fibrous proteins, and intrinsically disordered proteins are poorly modeled due to training data biases
• No embedding extraction endpoint — cannot produce dense vector representations for downstream ML tasks; use ESM-2 for protein embeddings
• OAS variant is antibody-only and produces poor results on non-antibody proteins; users must select the correct variant for their protein type
• Maximum useful context is limited by the model's attention window; very long proteins may lose N-terminal context during generation
• Generated sequences have no guarantee of function — statistical plausibility does not ensure folding, stability, or catalytic activity without computational and experimental validation
• No structure awareness — generates sequences without knowledge of 3D geometry, unlike inverse folding models (MPNN, ESM-IF1) that condition on backbone structure

Reach for something else when

• You need enzyme generation conditioned on a specific EC number (use zymctrl, which is specifically trained for EC-conditioned enzyme design)
• You need structure-conditioned sequence design from a target backbone (use mpnn for comprehensive inverse folding or esm_if1 for lightweight inverse folding)
• You need protein embeddings for downstream ML tasks (use esm2, which provides the most widely used protein embeddings in this catalog)
• You need masked infilling or bidirectional sequence design where specific regions are fixed (use dsm, which supports masked diffusion infilling)
• You need to score protein variants without generating new sequences (use esm2 log_prob for pseudo-log-likelihood scoring)
• You are designing membrane proteins or fibrous proteins where ProGen2's training data is underrepresented
• You need protein-protein interaction-aware generation (use dsm with the PPI variant trained on STRING interaction data)
• You need multimodal protein design combining sequence, structure, and function constraints

Alternatives

Model	Better when	Worse when
zymctrl	EC-number conditioned enzyme generation targets specific catalytic functions; trained on ~37M enzyme sequences with experimental validation of generated enzymes.	Enzyme-only and steered by an EC number rather than a seed sequence; cannot extend an arbitrary protein context or generate antibodies/general proteins the way ProGen2's four variants do.
dsm	Masked-diffusion protein LM: bidirectional generation and masked infilling (fix a scaffold, design a loop) plus encode and score in one model; PPI variant for interaction-aware design.	Newer with less independent validation; no dedicated antibody variant like ProGen2's OAS, and stronger publication track record favors ProGen2.
esm2	Best single-sequence protein embeddings and (pseudo-)log-likelihood scoring for ranking variants; five size variants and the widest downstream ecosystem.	Encoder-only — cannot generate or extend sequences; overlaps with ProGen2 only for scoring/fitness ranking, not for design.

API & schema

Variants

Variant	Endpoint slug	GPU	CPU	Memory
oas	progen2-oas	CPU	2.0	8 GB
medium	progen2-medium	t4	2.0	8 GB
large	progen2-large	t4	4.0	16 GB
bfd90	progen2-bfd90	t4	4.0	16 GB

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

generate

Call it

curl -X POST http://127.0.0.1:8000/api/v1/progen2-oas/generate \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "context": "string"
    }
  ]
}'

Request — ProGen2GenerateRequest

Field	Type	Required	Constraints	Description
params	ProGen2GenerateParams	no		Optional parameters controlling this action (defaults are used when omitted).
items	list[ProGen2GenerateRequestItem]	yes	items 1–1	Batch of inputs to process in a single request. Up to 1 sequence per request.

Nested types

ProGen2GenerateParams

Field	Type	Required	Constraints	Description
temperature	number	no	>0.0; ≤8.0; default 0.8	Sampling temperature; higher values increase diversity.
top_p	number	no	≥0.0; ≤1.0; default 0.9	Nucleus (top-p) sampling threshold.
num_samples	integer	no	≥1; ≤3; default 1	Number of sequences to generate per input.
max_length	integer	no	≥12; ≤512; default 128	Maximum length of the generated sequence.
seed	integer | null	no		Random seed for reproducible sampling.

ProGen2GenerateRequestItem

Field	Type	Required	Constraints	Description
context	string	yes	len 1–512	Amino acid seed sequence (unambiguous codes) to condition generation; the output begins with this context.

Raw JSON Schema

{
  "$defs": {
    "ProGen2GenerateParams": {
      "additionalProperties": false,
      "properties": {
        "temperature": {
          "default": 0.8,
          "description": "Sampling temperature; higher values increase diversity.",
          "exclusiveMinimum": 0.0,
          "maximum": 8.0,
          "title": "Temperature",
          "type": "number"
        },
        "top_p": {
          "default": 0.9,
          "description": "Nucleus (top-p) sampling threshold.",
          "maximum": 1.0,
          "minimum": 0.0,
          "title": "Top P",
          "type": "number"
        },
        "num_samples": {
          "default": 1,
          "description": "Number of sequences to generate per input.",
          "maximum": 3,
          "minimum": 1,
          "title": "Num Samples",
          "type": "integer"
        },
        "max_length": {
          "default": 128,
          "description": "Maximum length of the generated sequence.",
          "maximum": 512,
          "minimum": 12,
          "title": "Max Length",
          "type": "integer"
        },
        "seed": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Random seed for reproducible sampling.",
          "title": "Seed"
        }
      },
      "title": "ProGen2GenerateParams",
      "type": "object"
    },
    "ProGen2GenerateRequestItem": {
      "additionalProperties": false,
      "properties": {
        "context": {
          "description": "Amino acid seed sequence (unambiguous codes) to condition generation; the output begins with this context.",
          "maxLength": 512,
          "minLength": 1,
          "title": "Context",
          "type": "string"
        }
      },
      "required": [
        "context"
      ],
      "title": "ProGen2GenerateRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "params": {
      "$ref": "#/$defs/ProGen2GenerateParams",
      "description": "Optional parameters controlling this action (defaults are used when omitted)."
    },
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 1 sequence per request.",
      "items": {
        "$ref": "#/$defs/ProGen2GenerateRequestItem"
      },
      "maxItems": 1,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ProGen2GenerateRequest",
  "type": "object"
}

Response — ProGen2GenerateResponse

Field	Type	Required	Constraints	Description
results	list[list[ProGen2GenerateResponseGenerated]]	yes		Per-input results, returned in the same order as the request items.

Nested types

ProGen2GenerateResponseGenerated

Field	Type	Required	Constraints	Description
sequence	string	yes		Generated protein sequence (context prefix + completion), with terminal tokens stripped.
ll_sum	number	yes		Summed bidirectional log-likelihood (forward + reverse passes averaged); more negative means less likely.
ll_mean	number	yes		Mean bidirectional log-likelihood per position; more negative means less likely per residue.

Raw JSON Schema

{
  "$defs": {
    "ProGen2GenerateResponseGenerated": {
      "properties": {
        "sequence": {
          "description": "Generated protein sequence (context prefix + completion), with terminal tokens stripped.",
          "title": "Sequence",
          "type": "string"
        },
        "ll_sum": {
          "description": "Summed bidirectional log-likelihood (forward + reverse passes averaged); more negative means less likely.",
          "title": "Ll Sum",
          "type": "number"
        },
        "ll_mean": {
          "description": "Mean bidirectional log-likelihood per position; more negative means less likely per residue.",
          "title": "Ll Mean",
          "type": "number"
        }
      },
      "required": [
        "sequence",
        "ll_sum",
        "ll_mean"
      ],
      "title": "ProGen2GenerateResponseGenerated",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "items": {
          "$ref": "#/$defs/ProGen2GenerateResponseGenerated"
        },
        "type": "array"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ProGen2GenerateResponse",
  "type": "object"
}

Usage

One-line summary: Autoregressive protein language model from Salesforce Research that generates novel protein sequences and computes sequence log-likelihoods for fitness prediction.

Overview

ProGen2 is a family of autoregressive protein language models developed by Salesforce Research. Based on the GPT-J architecture, ProGen2 is trained on large protein sequence databases to learn the statistical patterns of natural proteins and generate novel, biologically plausible sequences.

ProGen2 is available in four variants in this catalog, each trained on different data: OAS (antibody-specialized), medium and large (general-purpose on UniRef90+BFD30), and BFD90 (metagenomic proteins). The model supports controllable generation via temperature and nucleus sampling parameters, and provides bidirectional log-likelihood scores for each generated sequence.

Published in Cell Systems (2023), ProGen2 demonstrates that autoregressive protein language models follow scaling laws similar to natural language models, with larger models producing more realistic protein sequences and better fitness predictions.

Architecture

Property	Value
Architecture	Transformer decoder (GPT-J-style, autoregressive)
Training objective	Causal language modeling (next-token prediction)
Training data	UniRef90, BFD90, OAS (variant-dependent)
Max sequence length	512 residues (BioLM limit; model supports 2048)
Vocabulary	32 tokens (ProGen2 custom amino-acid tokenizer)
Positional encoding	Rotary (RoPE)
License	BSD-3-Clause

Capabilities & Limitations

CAN be used for: - Generating novel protein sequences conditioned on an amino acid context (seed sequence) - Scoring protein sequences via bidirectional log-likelihood (fitness prediction) - Generating antibody variable region sequences (OAS variant) - Producing diverse sequence libraries with controllable temperature and top-p sampling - Extending partial protein sequences (context-conditioned completion)

CANNOT be used for: - Structure prediction (use ESMFold or Chai-1 instead) - Protein embeddings (use ESM-2 instead) - Non-protein molecules (DNA, RNA, small molecules) - Sequences longer than 512 residues - Batch requests with more than 1 sequence per call

Other considerations: - Generation is stochastic by default; set seed for reproducible outputs - The ll_sum and ll_mean likelihood scores average forward and reverse passes for reduced positional bias - Terminal tokens (1 for N-terminal, 2 for C-terminal) are automatically handled and stripped from output - The OAS variant should only be used for antibody sequences

Usage Examples

from models.progen2.schema import (
    ProGen2GenerateRequest,
    ProGen2GenerateRequestItem,
    ProGen2GenerateParams,
)

# Basic generation: extend a context sequence
request = ProGen2GenerateRequest(
    params=ProGen2GenerateParams(
        temperature=0.8,
        top_p=0.9,
        num_samples=2,
        max_length=128,
    ),
    items=[
        ProGen2GenerateRequestItem(context="MKTVRQERLKSIVRILERSKEPVSGAQ"),
    ],
)

# Reproducible generation with fixed seed
request_seeded = ProGen2GenerateRequest(
    params=ProGen2GenerateParams(
        temperature=0.7,
        top_p=0.95,
        num_samples=3,
        max_length=256,
        seed=42,
    ),
    items=[
        ProGen2GenerateRequestItem(context="MGSSHHHHHHSSGLVPRGSH"),
    ],
)

Architecture & training

Architecture

Model Type & Innovation

ProGen2 is an autoregressive protein language model based on the GPT-J architecture (decoder-only transformer). It is trained with a standard causal language modeling (next-token prediction) objective on large protein sequence databases, learning to generate protein sequences one amino acid at a time.

The key innovation of ProGen2 over its predecessor ProGen is systematic scaling. The authors trained models from ~151M to ~6.4B parameters across different protein datasets and demonstrated that larger models produce more realistic protein sequences and better fitness predictions. ProGen2 also introduced training on multiple dataset compositions (UniRef90, BFD90, and the OAS antibody database), allowing specialized variants for different protein families.

ProGen2 uses a GPT-J-style transformer decoder with rotary position embeddings (RoPE), GELU activation, and pre-layer normalization. The architecture supports autoregressive sampling with nucleus (top-p) filtering and temperature control for controllable protein generation.

Parameters & Layers

Variant	Parameters	Layers	Attention Heads	Rotary Dim	Hidden Dim	Context Length
progen2-oas	151M	12	16	64	1280	2048
progen2-medium	764M	27	16	96	2560	2048
progen2-large	2.7B	32	32	80	4096	2048
progen2-bfd90	2.7B	32	32	80	4096	2048

Parameter counts and layer/head configurations are confirmed from paper Table 1 (Nijkamp et al., 2023). The paper also describes PROGEN2-small (151M, same architecture as OAS), PROGEN2-base (764M, same as medium but with 2048 context), and PROGEN2-xlarge (6.4B, 32 layers, 16 heads, head dim 256) which are not included in this catalog. The BioLM OAS variant uses a 2048 context length (matching PROGEN2-base) rather than the paper's 1024 for PROGEN2-small.

Common across all variants:

Property	Value
Vocabulary size	32 tokens (ProGen2 custom amino-acid tokenizer)
Positional encoding	Rotary (RoPE), dim varies by variant (64/96/80)
Normalization	Pre-LayerNorm
Activation	GELU (gelu_new)
Context length	2048 tokens

Training Data

ProGen2 variants are trained on different protein sequence databases:

Variant	Dataset	Size	Description
progen2-oas	OAS (Observed Antibody Space)	554M sequences (after 85% identity clustering)	Unpaired antibody sequences from 80 immune repertoire sequencing studies, covering heavy and light chains from 6 species (human, mouse, rat, camel, rabbit, rhesus). Clustered at 85% identity using Linclust to reduce redundancy from the original 1.5B sequences.
progen2-medium	UniRef90 + BFD30	UniRef90 cluster representatives + BFD30 (~1/3 size of UniRef90)	UniRef90 are cluster representative sequences from UniProtKB at 90% sequence identity. BFD30 is the Big Fantastic Database clustered at 30% identity, majority from metagenomic sources.
progen2-large	UniRef90 + BFD30	Same as medium	Same training data as medium, with larger model capacity (2.7B vs 764M parameters).
progen2-bfd90	UniRef90 + BFD90	BFD90 (~2x size of UniRef90)	UniRef90 mixed with BFD90: representative sequences with at least 3 cluster members after clustering UniProtKB, Metaclust, SRC, and MERC at 90% identity.

Training data details confirmed from Nijkamp et al. (2023), Section 3.2. All sequences are provided with N-terminal (1) and C-terminal (2) tokens, and each sequence is included in both forward and reverse orientations during training.

Known biases in the training data: - OAS variant is specialized for antibodies and will underperform on non-antibody proteins - BFD90 has strong bias toward bacterial and metagenomic proteins - All variants under-represent eukaryotic membrane proteins relative to their biological importance - Training data lacks non-natural amino acids and post-translational modifications

Loss Function & Objective

Standard autoregressive language modeling with cross-entropy loss:

L = -Sum_i log P(x_i | x_1, ..., x_{i-1})

Each token is predicted conditioned on all previous tokens. No masking strategy is used -- this is purely left-to-right next-token prediction.

ProGen2 uses special terminal tokens: 1 for N-terminal (beginning of protein) and 2 for C-terminal (end of protein). During training, sequences are formatted as 1{sequence}2.

Tokenization / Input Processing

Property	Details
Tokenizer	ProGen2 custom tokenizer (32 tokens: 20 standard amino acids + special/terminal tokens)
Special tokens	1 (N-terminal/BOS), 2 (C-terminal/EOS), <\|pad\|> (padding)
N-terminal prepended	Yes (token 1)
C-terminal appended	Yes (token 2, for likelihood computation)
Maximum sequence length	512 residues (BioLM implementation limit)

The implementation prepends the 1 N-terminal token to the context sequence before sampling. For likelihood computation, both 1 (N-terminal) and 2 (C-terminal) tokens are added to frame the full sequence. The tokenizer is character-level for amino acids -- each standard amino acid maps to a single token.

Performance & Benchmarks

Published Benchmarks

Protein Fitness Prediction

ProGen2 log-likelihoods correlate with experimentally measured protein fitness:

Narrow Fitness Landscapes (Paper Table 3)

Zero-shot fitness prediction on narrow experimentally-measured fitness landscapes (primarily single-substitution DMS experiments). Average Spearman rho reported with baselines from Hesslow et al. (2022):

Model	Avg Spearman rho	Notes
PROGEN2-small (151M)	0.456	Outperforms much larger RITA-XL
PROGEN2-base (764M)	0.505	Best single PROGEN2 model on narrow landscapes
PROGEN2-large (2.7B)	0.485	Larger capacity does not always improve fitness prediction
PROGEN2-xlarge (6.4B)	0.476	Performance decreases with further scaling
PROGEN2-ensemble	0.518	Best overall ProGen2 result
RITA-XL	0.443	Order of magnitude larger than PROGEN2-small
EVE	0.511	Family-specific VAE
Tranception (no retrieval)	0.447	Autoregressive
Tranception (retrieval)	0.503	Autoregressive with MSA retrieval
MSA Transformer	0.476	Requires MSA input
ESM-1v (single)	0.475	Masked LM, single model

Key finding: performance peaks at 764M parameters (PROGEN2-base) for narrow fitness landscapes, then decreases with scale -- likely because smaller models project the data distribution onto a model class closer to the true fitness landscape.

Wide Fitness Landscapes (Paper Table 4)

Zero-shot fitness prediction on wider experimental landscapes with higher edit distances:

Dataset [Metric]	PROGEN2-small	PROGEN2-base	PROGEN2-large	PROGEN2-xlarge
AAV [AUC]	0.59	0.62	0.65	0.68
GFP [AUC]	0.51	0.64	0.84	0.84
CM [AUC]	0.68	0.72	0.66	0.64
GB1 [top100avg]	0.01	0.01	0.24	0.85

For wider landscapes, larger models show clear advantages, particularly for the GB1 low-homology epistatic landscape where the 6.4B model may exhibit emergent behavior.

Antibody-Specific Landscapes (Paper Table 5)

Property	PROGEN2-small	PROGEN2-base	PROGEN2-large	PROGEN2-xlarge	PROGEN2-OAS
Binding [avg rho]	0.44	0.41	0.42	0.40	0.37
General [avg rho]	0.61	0.73	0.73	0.74	0.66

Notably, the OAS-trained model underperforms universal models on antibody fitness prediction, suggesting that redundancy-reduced immune repertoire sequences alone do not lead to better fitness prediction for antibodies.

Scaling Laws

Perplexity on held-out test sequences (paper Table 2). Lower perplexity indicates the model better captures the distribution of observed evolutionary sequences:

Model	Parameters	Test-max90 (ppl)	Test-max50 (ppl)
PROGEN2-small	151M	12.9	15.0
PROGEN2-medium	764M	11.2	14.3
PROGEN2-large	2.7B	11.1	14.4
PROGEN2-xlarge	6.4B	9.9	13.9

Test-max90 and Test-max50 correspond to held-out clusters at 90% and 50% sequence identity respectively. Test-max50 is a harder, more out-of-distribution evaluation. Perplexity decreases consistently with model scale, confirming scaling laws hold for protein language models.

BioLM Verification Results

The BioLM implementation uses official pre-trained weights from the Salesforce ProGen repository. Verification is performed using a custom validator that checks:

Check	Criterion	Status
Sequence count	num_samples matches requested	PASS
Context preservation	Generated sequences start with input context	PASS
Length constraint	Generated sequences do not exceed max_length	PASS

Comparison to Alternatives

Model	Type	Key Advantage	Key Disadvantage
ProGen2 (this)	Autoregressive LM	True sequence generation with controllable sampling	No bidirectional context; fitness prediction less accurate than MLM models
ESM-2	Masked LM	Better embeddings and fitness prediction	Cannot generate sequences
ProtGPT2	Autoregressive LM	Simpler architecture	Smaller model, less diverse training data
ProGen (v1)	Conditional generation	Controllable generation with taxonomy tags	Superseded by ProGen2
EvoDiff	Diffusion-based	Non-autoregressive generation	Slower sampling, newer/less benchmarked

Error Bars & Confidence

ProGen2 is inherently stochastic -- sampling uses temperature and top-p nucleus filtering, producing different outputs on each call (unless a fixed seed is provided). The log-likelihood scores (ll_sum, ll_mean) are deterministic for a given sequence and model.

Sources of variability: - Sampling: Different random seeds produce different generated sequences. Typical diversity across samples is high -- sequences may share only the context prefix. - Likelihood averaging: The implementation averages forward (left-to-right) and reverse (right-to-left) log-likelihoods to reduce directional bias. This bidirectional averaging is a deliberate design choice from the original code. - GPU precision: Small numerical differences in likelihoods may occur across GPU architectures.

Strengths & Limitations

Pros

• True protein sequence generation -- can design novel proteins from scratch or extend existing sequences
• Multiple training data variants allow specialization (antibodies via OAS, general proteins via BFD90)
• Controllable generation via temperature, top-p, and seed parameters
• Log-likelihood scoring enables fitness prediction and sequence ranking
• Bidirectional likelihood averaging (forward + reverse) reduces positional bias
• BSD-3-Clause license allows commercial use

Cons

• Autoregressive generation is inherently sequential -- no parallel token generation
• Fitness prediction accuracy is generally below masked language models (ESM-2) on standard benchmarks
• No structural awareness -- generated sequences may not fold into stable structures
• Maximum 512 residues in BioLM implementation (model supports 2048 but capped for safety)
• Batch size limited to 1 item per request
• OAS variant is narrowly specialized; other variants may underperform on antibodies

Known Failure Modes

• Very short contexts (< 5 residues): Generation may produce highly diverse and potentially non-biological sequences due to insufficient conditioning
• Very low temperature (approaching 0.0): Schema requires temperature > 0.0; near-zero values can produce repetitive sequences (poly-amino acid tracts)
• High temperature (> 2.0): Generated sequences become increasingly random and biologically implausible
• Context outside training distribution: Sequences with non-standard amino acids or unusual compositions may produce poor completions
• Long generation lengths: Quality degrades as generated length increases beyond ~200-300 residues, especially without a strong context signal

Implementation Details

Inference Pipeline

Request
  |-- 1. Validate sequences (AA alphabet, length, batch size)
  |-- 2. Set random seeds (user-provided or time-based entropy)
  |-- 3. Prepend N-terminal token "1" to context
  |-- 4. Autoregressive sampling on GPU
  |     |-- Tokenize context with the ProGen2 custom amino-acid tokenizer
  |     |-- model.generate() with temperature + top-p
  |     |-- Decode token IDs back to amino acid sequences
  |-- 5. Truncate at terminal tokens ("1" or "2")
  |-- 6. Strip terminal tokens from generated sequences
  |-- 7. Compute bidirectional log-likelihoods
  |     |-- Forward: log P(1{seq}2)
  |     |-- Reverse: log P(reversed(1{seq}2))
  |     |-- Average: 0.5 * (forward + reverse)
  |-- 8. Return ProGen2GenerateResponse with sequences + likelihoods

Memory & Compute Profile

Variant	GPU	GPU Memory (approx)	Inference Time (128 tokens)
oas	None (CPU)	CPU only, ~2 GB RAM	~2-5s
medium	T4	~4 GB VRAM	~1-3s
large	T4	~10 GB VRAM	~3-8s
bfd90	T4	~10 GB VRAM	~3-8s

Autoregressive generation scales linearly with output length (O(n) forward passes, each O(n) with KV-cache).

Determinism & Reproducibility

Setting	Value
torch.manual_seed	User seed or time.time_ns() % 2^32
torch.cuda.manual_seed_all	Same as above
numpy.random.seed	Same as above
random.seed	Same as above
cuDNN deterministic	Not explicitly set
cuDNN benchmark	Not explicitly disabled

The model produces reproducible outputs when the same seed is provided. Without a seed, each call produces different sequences using time-based entropy.

Caching Behavior

Response caching is handled outside the model container by the serving infrastructure: - Cache key: Determined by the request payload (context, params, model variant) - Note: Due to stochastic generation, caching is most useful when seeds are fixed. Without a seed, identical requests will return cached results from the first call rather than fresh samples.

Versions & Changelog

Version	Date	Changes
v1	2025-01-15	Initial implementation with generate action for oas, medium, large, bfd90 variants
v1 (updated)	2026-03-14	Migrated to declarative download system and source layer setup

---

Biology

Molecule Coverage

Primary Molecule Type(s)

ProGen2 is trained on protein sequences and is designed for protein sequence generation. The model's capabilities vary by variant:

• General protein variants (medium, large, bfd90): Trained on large-scale protein databases (UniRef90, BFD90) covering proteins from all domains of life -- bacteria, archaea, eukaryota, and viruses. These variants handle globular proteins well and can generate plausible sequences for most protein families represented in the training data.
• Antibody-specific variant (oas): Trained exclusively on the Observed Antibody Space database, covering paired and unpaired antibody variable region sequences from diverse immune repertoires. This variant is specialized for antibody sequence generation and will produce poor results on non-antibody proteins.

Performance characteristics by protein type: - Globular, soluble proteins: Strong generation quality. These dominate the training data for medium/large/bfd90 variants. - Enzymes: Well-represented in training data. The model can generate plausible enzyme sequences, though it has no explicit understanding of catalytic mechanisms. - Antibodies: Best served by the OAS variant. General variants can generate antibody-like sequences but lack the specificity of the OAS-trained model. - Membrane proteins: Under-represented in training data. Generated sequences may not faithfully reproduce transmembrane topology. - Intrinsically disordered proteins: Poorly modeled -- the autoregressive objective may struggle with low-complexity, repetitive regions.

Cross-Applicability

Molecule Type	Applicability	Evidence	Caveats
Antibodies	High (OAS variant) / Moderate (general variants)	OAS variant trained on 554M antibody sequences (clustered at 85% identity from ~1.5B raw OAS sequences)	OAS variant is antibody-only; general variants lack CDR-specific knowledge
Enzymes	High	BFD90 and UniRef90 training data are rich in enzyme families	No explicit modeling of active sites, cofactor binding, or catalytic residues
Peptides	Low--Moderate	Short sequences provide limited context for autoregressive generation	Context window may be too short for meaningful conditioning; consider peptide-specific models
Therapeutic proteins (non-antibody)	Moderate	General protein training covers many therapeutic targets	No explicit optimization for developability, immunogenicity, or stability
Fibrous proteins	Low	Under-represented in training; repetitive sequences are difficult to model autoregressively	Generated sequences may degenerate into repeats

Biological Problems Addressed

Protein Sequence Generation (De Novo Design)

Problem: Designing entirely new protein sequences that fold into functional structures is a central challenge in protein engineering. Traditional approaches rely on directed evolution (iterative rounds of mutation and selection) or rational design (structure-guided mutagenesis), both of which are experimentally expensive and limited in the sequence space they can explore.

How ProGen2 helps: The generate action takes a short amino acid context (seed sequence) and autoregressively extends it, sampling from the learned distribution of natural proteins. The model has internalized statistical patterns of amino acid co-occurrence, secondary structure preferences, and long-range dependencies from millions of protein sequences. Generated sequences are biologically plausible completions of the given context.

Biological meaning: A generated sequence with a high log-likelihood (ll_mean close to 0) is one that the model considers consistent with natural protein sequences -- it follows the "grammar" of proteins. Lower likelihoods suggest the sequence deviates from natural patterns and may be less likely to fold or function. The num_samples parameter allows generating multiple candidates for experimental screening.

Practical considerations: Generated sequences should be validated computationally (e.g., structure prediction with ESMFold or Chai-1, stability estimation with ThermoMPNN) before experimental testing. ProGen2 generates sequences that look statistically natural but provides no guarantee of function.

Protein Fitness Prediction (Sequence Scoring)

Problem: Given a set of protein sequence variants (e.g., from a mutagenesis library), predicting which variants retain function, fold correctly, or have improved properties. Experimental characterization of every variant is infeasible for large libraries.

How ProGen2 helps: Each generated sequence receives a log-likelihood score (ll_sum and ll_mean) computed as the average of forward and reverse autoregressive log-probabilities. These scores correlate with experimental measures of protein fitness -- sequences with higher log-likelihood tend to be more functional, stable, and well-folded.

Biological meaning: The log-likelihood measures how "natural" a sequence appears under the model's learned distribution. Deleterious mutations that disrupt conserved patterns reduce the likelihood; neutral or beneficial mutations at tolerant positions maintain or increase it. The bidirectional averaging (forward + reverse) reduces positional bias inherent in left-to-right autoregressive models.

Directed Evolution Guidance

Problem: Directed evolution campaigns generate large mutant libraries but require efficient screening to identify improved variants. Computational pre-screening can reduce the experimental burden by orders of magnitude.

How ProGen2 helps: By generating sequences conditioned on a starting context (the wild-type or a promising variant), ProGen2 produces a focused library of plausible next-step sequences. The accompanying likelihood scores enable ranking without additional computation. Researchers can select the top-scoring candidates for experimental characterization.

Biological meaning: The context-conditioned generation mimics natural sequence diversification -- the model generates sequences that are plausible relatives of the input, much as natural evolution produces homologs through mutation and selection. The temperature parameter controls the "evolutionary distance" from the context: low temperature produces conservative variants, high temperature produces more diverse (but potentially less stable) sequences.

Applied Use Cases

ProGen2 and its predecessor ProGen have been used in several applied settings:

• Enzyme design: Generating novel enzyme sequences for industrial biocatalysis, followed by experimental validation of folding and activity

Note: This is an anticipated use case based on the model's capabilities.

• Antibody library design: Using the OAS variant to generate diverse antibody variable region sequences for therapeutic screening campaigns

Note: This is an anticipated use case based on the model's capabilities.

• Protein fitness landscapes: Scoring mutant libraries to predict the effect of mutations on protein function, complementing deep mutational scanning experiments

Note: This is an anticipated use case based on the model's capabilities.

• Sequence in-filling: Using context-conditioned generation to propose plausible sequences for protein regions with missing or ambiguous annotation

Note: This is an anticipated use case based on the model's capabilities.

Related Models

Predecessor Models

• ProGen (Madani et al., 2023): The original ProGen model, a conditional transformer trained with taxonomy and function tags. ProGen2 removes the conditioning tags and focuses on pure sequence modeling at larger scale. ProGen is not available in this catalog.

Complementary Models

ProGen2 is often used in multi-step computational workflows:

• ProGen2 + ESMFold/Chai-1: Generate sequences with ProGen2, then predict their 3D structures with a structure prediction model to filter for foldable designs
• ProGen2 + ThermoMPNN: Generate sequences, then predict stability changes (ddG) to select candidates with desirable thermal properties
• ProGen2 + ESM-2: Use ESM-2 embeddings to cluster or characterize ProGen2-generated sequences, or use ESM-2 log-probabilities as an orthogonal fitness score

Typical workflow: 1. Generate candidate sequences with ProGen2 (diverse sampling, multiple seeds) 2. Score candidates with ESM-2 pseudo-log-likelihood 3. Predict structures for top candidates with Chai-1 4. Estimate stability changes for top candidates with ThermoMPNN 5. Select final candidates for experimental validation

Alternative Models

Alternative	Advantage Over ProGen2	Disadvantage vs ProGen2
ProtGPT2	Simpler, widely available	Smaller model, trained on less diverse data
ESM-2 (masked prediction)	Better fitness prediction accuracy	Cannot generate sequences autoregressively
EvoDiff	Non-autoregressive diffusion enables guided generation	Newer, less benchmarked, slower sampling
Tranception	Better fitness prediction via retrieval augmentation	Primarily a scorer, not a generator
RFDiffusion	Structure-guided design (backbone + sequence)	Requires structural input, much more complex

When to choose ProGen2: Use ProGen2 when you need to generate novel protein sequences from a context seed, especially when you want controllable diversity (temperature, top-p) and likelihood-based scoring. It is the strongest autoregressive protein generation model available in this catalog.

When to choose alternatives: Consider ESM-2 for fitness prediction without generation; consider Chai-1 + MPNN for structure-guided sequence design; consider EvoDiff for non-autoregressive generation with potentially better diversity.

Biological Background

Proteins are linear polymers of amino acids that fold into three-dimensional structures to carry out biological functions. The sequence of amino acids -- typically 50 to 2000 residues long, drawn from a 20-letter alphabet -- determines the protein's structure and function. Natural proteins have been shaped by billions of years of evolution, and the sequences that survive in nature represent a tiny but highly structured subset of all possible amino acid combinations.

Protein generation: The goal of computational protein generation is to produce novel amino acid sequences that fold into stable, functional structures. This is central to applications in drug development (designing therapeutic proteins), industrial biotechnology (engineering enzymes for chemical synthesis), and basic research (understanding the sequence-structure-function relationship). An autoregressive language model like ProGen2 learns the statistical rules governing natural protein sequences and can sample new sequences from this learned distribution.

Autoregressive vs. masked language modeling: In autoregressive modeling, the protein sequence is treated like a sentence, and each amino acid is predicted from all preceding amino acids (left-to-right). This naturally supports generation -- the model can extend a partial sequence indefinitely. In masked language modeling (e.g., ESM-2), random positions are hidden and predicted from bidirectional context, which produces better representations but does not support generation.

Fitness landscapes: In protein engineering, a "fitness landscape" maps every possible sequence variant to an experimentally measurable property (stability, activity, binding affinity). Language model log-likelihoods provide a computational approximation of this landscape -- sequences with high likelihood under the model tend to be functional, while low-likelihood sequences are more likely to be non-functional or deleterious.

Key terminology: - Autoregressive model: A model that generates sequences one token at a time, each conditioned on all previous tokens. - Nucleus sampling (top-p): A sampling strategy that restricts token selection to the smallest set of tokens whose cumulative probability exceeds a threshold p. This balances diversity and quality. - Temperature: A parameter that scales the model's output logits before sampling. Lower temperatures produce more conservative (higher-confidence) outputs; higher temperatures increase diversity. - Log-likelihood: The logarithm of the probability assigned to a sequence by the model. Higher (less negative) values indicate the model considers the sequence more plausible. - N-terminal / C-terminal: The two ends of a protein chain. The N-terminal has a free amino group (-NH2) and is conventionally written first; the C-terminal has a free carboxyl group (-COOH). - BFD (Big Fantastic Database): A large metagenomic protein sequence database compiled from soil, ocean, and gut metagenomes, clustered at various identity thresholds. - OAS (Observed Antibody Space): A database of antibody sequences from immune repertoire sequencing studies across multiple species and disease states.

---

Sources & license

License: BSD-3-Clause (text) — Code is BSD-3-Clause; model architecture based on Apache-2.0 GPT-J

Papers

• ProGen2: Exploring the Boundaries of Protein Language Models — Cell Systems, 2023 · DOI arXiv

Source repositories

• github: https://github.com/salesforce/progen

Cite

@article{nijkamp2023progen2,
  title={ProGen2: Exploring the Boundaries of Protein Language Models},
  author={Nijkamp, Erik and Ruffolo, Jeffrey A. and Weinstein, Eli N. and Naik, Nikhil and Madani, Ali},
  journal={Cell Systems},
  year={2023},
  doi={10.1016/j.cels.2023.10.002}
}