IgBert¶

BERT-based antibody language model with paired and unpaired variants that produces sequence embeddings, masked residue predictions, and log-probability scores for immunoglobulin sequences.

License: MIT · Molecules: antibody · Tasks: embedding, sequence_completion

Actions: encode, generate, log_prob · Variants: 2

At a glance¶

Use it when

You need to analyze both paired and unpaired antibody sequences within the same pipeline and want a consistent model family that handles both cases
You want a HuggingFace-compatible antibody language model for fine-tuning on custom downstream tasks such as binding affinity, expression level, or specificity prediction
You need to restore missing residues in antibody sequences from sequencing data, using either paired cross-chain context or single-chain context depending on data availability
You are building a repertoire analysis pipeline and need embeddings that capture paired chain properties for clonal family clustering and convergent evolution detection
You want to score engineered antibody variants with log_prob to assess sequence plausibility and identify mutations that disrupt conserved structural motifs
You need to compare BERT-architecture antibody representations against T5-architecture (IgT5) or debiased (AbLang2) alternatives for your specific downstream task

Strengths

Both paired and unpaired variants in a single model family, enabling analysis of both matched heavy-light chain pairs and individual chains when pairing information is unavailable
HuggingFace Transformers-compatible architecture, making fine-tuning, transfer learning, and integration with standard ML ecosystems straightforward
Three complementary actions (encode, generate, log_prob) covering embedding extraction, masked residue prediction, and sequence-level log-probability scoring
Paired variant captures heavy-light chain co-evolution through joint processing with [SEP] separator, encoding the combined paratope properties that determine binding specificity
Trained on one of the largest paired antibody datasets from Exscientia, with strong published benchmarks on CDR sequence recovery, expression prediction, and pair classification tasks
Unpaired variant provides a fallback for NGS datasets where pairing information is lost, avoiding the requirement for matched heavy-light chain data
Permissively licensed (MIT per the HuggingFace model card; permits commercial use), suitable for therapeutic antibody development

Limitations

No germline debiasing in the training objective, so embeddings may conflate germline gene identity with functional properties -- a known bias in antibody language models that AbLang2 explicitly addresses
Requires T4 GPU for inference (6 GB RAM per variant), which increases deployment cost compared to CPU-only models like AbLang2
Not specialized for nanobody (VHH) sequences; the unpaired variant can process them but lacks nanobody-specific training that captures hallmark residues and extended CDR3 diversity
Two separate deployments required (paired and unpaired) to cover all use cases, doubling infrastructure if both are needed simultaneously
No predict action for per-position likelihoods (only generate for masked prediction and log_prob for global scoring) -- AbLang2 provides a richer output set
Variable domain focus only; constant regions are not the primary training target, limiting utility for full-length antibody analysis
Published benchmarks show CDR-aware fine-tuned models can surpass IgBERT on binding affinity prediction tasks, suggesting room for improvement on binding-related downstream tasks

Reach for something else when

You specifically need germline-debiased representations that separate functional properties from germline gene identity -- use AbLang2 with its focal-loss training
You are working exclusively with nanobody (VHH) sequences and need single-domain-specialized representations -- IgBERT's unpaired variant can process VHH but is not nanobody-specialized; use AntiFold or ImmuneFold for nanobody-specific structural analysis
You need 3D structure prediction from sequence -- use AbodyBuilder3 for Fab structures, ImmuneBuilder for antibodies/nanobodies/TCRs, or ImmuneFold for highest accuracy
You need structure-conditioned inverse folding for CDR library design -- use AntiFold, which requires a backbone structure but produces structurally compatible sequences
You need to analyze non-antibody proteins or compare antibodies with other protein families -- use ESM-2, which covers the full protein universe
You need developability property predictions (viscosity, thermostability) rather than general embeddings or sequence scoring -- use DeepViscosity for antibody viscosity or TemBERTure for thermostability (Tm)
You want embedding-only analysis with the best possible T5-architecture representations -- use IgT5, the companion model from the same paper

Alternatives

Model	Better when	Worse when
ablang2	AbLang2 provides germline-debiased representations with focal-loss training, plus a predict action for per-position likelihoods and a generate action for sequence restoration -- all on CPU without GPU	AbLang2 requires paired heavy-light chain input only; IgBERT's unpaired variant handles single-chain analysis when pairing information is unavailable
igt5	IgT5 is the companion T5-architecture model from the same paper; its relative position biases may better capture long-range dependencies for some downstream tasks	IgT5 is embedding-only (encode action) with no sequence generation or log-probability scoring; IgBERT provides generate and log_prob actions for more complete analysis
esm2	ESM-2 covers all proteins, offers 5 size variants, and has the broadest downstream ecosystem; better for cross-protein-family comparisons	ESM-2 is not antibody-specialized, cannot model paired heavy-light chains, and underperforms antibody-specific models on CDR recovery and antibody property prediction tasks

API & schema¶

Variants

Variant	Endpoint slug	GPU	CPU	Memory
paired	`igbert-paired`	t4	3.0	6 GB
unpaired	`igbert-unpaired`	t4	3.0	6 GB

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

`encode`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/igbert-paired/encode \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {}
  ]
}'

Request — IgBertEncodeRequest

Field	Type	Required	Constraints	Description
`params`	IgBertEncodeRequestParams	no		Optional parameters controlling this action (defaults are used when omitted).
`items`	list[IgBertEncodeRequestItem]	yes	items 1–32	Batch of inputs to process in a single request. Up to 32 sequences per request.

Nested types

IgBertEncodeIncludeOptions

Allowed values: mean, per_residue, per_residue, logits

IgBertEncodeRequestItem

Field	Type	Required	Description
`heavy_chain`	string \| null	no	Antibody heavy-chain amino-acid sequence.
`light_chain`	string \| null	no	Antibody light-chain amino-acid sequence.
`sequence`	string \| null	no	An antibody chain sequence in single-letter amino-acid codes, for unpaired mode.

IgBertEncodeRequestParams

Field	Type	Required	Constraints	Description
`include`	list[IgBertEncodeIncludeOptions]	no	default `['mean']`	Optional outputs to compute and include in the response.

Raw JSON Schema

{
  "$defs": {
    "IgBertEncodeIncludeOptions": {
      "enum": [
        "mean",
        "per_residue",
        "per_residue",
        "logits"
      ],
      "title": "IgBertEncodeIncludeOptions",
      "type": "string"
    },
    "IgBertEncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "heavy_chain": {
          "anyOf": [
            {
              "maxLength": 256,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Antibody heavy-chain amino-acid sequence.",
          "title": "Heavy Chain"
        },
        "light_chain": {
          "anyOf": [
            {
              "maxLength": 256,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Antibody light-chain amino-acid sequence.",
          "title": "Light Chain"
        },
        "sequence": {
          "anyOf": [
            {
              "maxLength": 512,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "An antibody chain sequence in single-letter amino-acid codes, for unpaired mode.",
          "title": "Sequence"
        }
      },
      "title": "IgBertEncodeRequestItem",
      "type": "object"
    },
    "IgBertEncodeRequestParams": {
      "additionalProperties": false,
      "properties": {
        "include": {
          "description": "Optional outputs to compute and include in the response.",
          "items": {
            "$ref": "#/$defs/IgBertEncodeIncludeOptions"
          },
          "title": "Include",
          "type": "array",
          "default": [
            "mean"
          ]
        }
      },
      "title": "IgBertEncodeRequestParams",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "params": {
      "$ref": "#/$defs/IgBertEncodeRequestParams",
      "description": "Optional parameters controlling this action (defaults are used when omitted)."
    },
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 32 sequences per request.",
      "items": {
        "$ref": "#/$defs/IgBertEncodeRequestItem"
      },
      "maxItems": 32,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "IgBertEncodeRequest",
  "type": "object"
}

Response — IgBertEncodeResponse

Field	Type	Required	Constraints	Description
`results`	list[IgBertEncodeResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

IgBertEncodeResponseResult

Field	Type	Required	Description
`embeddings`	list[number] \| null	no	Mean-pooled embedding vector for the sequence.
`residue_embeddings`	list[list[number]] \| null	no	Per-residue embedding vectors.
`logits`	list[list[number]] \| null	no	Per-position logits over the model vocabulary.

Raw JSON Schema

{
  "$defs": {
    "IgBertEncodeResponseResult": {
      "additionalProperties": false,
      "exclude_none": true,
      "exclude_unset": true,
      "properties": {
        "embeddings": {
          "anyOf": [
            {
              "items": {
                "type": "number"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Mean-pooled embedding vector for the sequence.",
          "title": "Embeddings"
        },
        "residue_embeddings": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-residue embedding vectors.",
          "title": "Residue Embeddings"
        },
        "logits": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-position logits over the model vocabulary.",
          "title": "Logits"
        }
      },
      "title": "IgBertEncodeResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/IgBertEncodeResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "IgBertEncodeResponse",
  "type": "object"
}

`generate`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/igbert-paired/generate \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {}
  ]
}'

Request — IgBertGenerateRequest

Field	Type	Required	Constraints	Description
`items`	list[IgBertGenerateRequestItem]	yes	items 1–32	Batch of inputs to process in a single request. Up to 32 sequences per request.

Nested types

IgBertGenerateRequestItem

Field	Type	Required	Description
`heavy_chain`	string \| null	no	Antibody heavy-chain sequence with * at masked positions to be restored.
`light_chain`	string \| null	no	Antibody light-chain sequence with * at masked positions to be restored.
`sequence`	string \| null	no	An antibody chain sequence in single-letter amino-acid codes with * at masked positions, for unpaired mode.

Raw JSON Schema

{
  "$defs": {
    "IgBertGenerateRequestItem": {
      "additionalProperties": false,
      "description": "For generate(), we allow '*' placeholders inside the heavy/light sequences,\nwhich must still be valid length and contain at least 1 '*'.",
      "properties": {
        "heavy_chain": {
          "anyOf": [
            {
              "maxLength": 256,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Antibody heavy-chain sequence with * at masked positions to be restored.",
          "title": "Heavy Chain"
        },
        "light_chain": {
          "anyOf": [
            {
              "maxLength": 256,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Antibody light-chain sequence with * at masked positions to be restored.",
          "title": "Light Chain"
        },
        "sequence": {
          "anyOf": [
            {
              "maxLength": 512,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "An antibody chain sequence in single-letter amino-acid codes with * at masked positions, for unpaired mode.",
          "title": "Sequence"
        }
      },
      "title": "IgBertGenerateRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 32 sequences per request.",
      "items": {
        "$ref": "#/$defs/IgBertGenerateRequestItem"
      },
      "maxItems": 32,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "IgBertGenerateRequest",
  "type": "object"
}

Response — IgBertGenerateResponse

Field	Type	Required	Constraints	Description
`results`	list[IgBertGenerateResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

IgBertGenerateResponseResult

Field	Type	Required	Description
`heavy_chain`	string \| null	no	Restored antibody heavy-chain sequence with masked positions filled in.
`light_chain`	string \| null	no	Restored antibody light-chain sequence with masked positions filled in.
`sequence`	string \| null	no	Restored antibody chain sequence with masked positions filled in, populated in unpaired mode.

Raw JSON Schema

{
  "$defs": {
    "IgBertGenerateResponseResult": {
      "properties": {
        "heavy_chain": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Restored antibody heavy-chain sequence with masked positions filled in.",
          "title": "Heavy Chain"
        },
        "light_chain": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Restored antibody light-chain sequence with masked positions filled in.",
          "title": "Light Chain"
        },
        "sequence": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Restored antibody chain sequence with masked positions filled in, populated in unpaired mode.",
          "title": "Sequence"
        }
      },
      "title": "IgBertGenerateResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/IgBertGenerateResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "IgBertGenerateResponse",
  "type": "object"
}

`log_prob`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/igbert-paired/log_prob \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {}
  ]
}'

Request — IgBertLogProbRequest

Field	Type	Required	Constraints	Description
`items`	list[IgBertEncodeRequestItem]	yes	items 1–32	Batch of inputs to process in a single request. Up to 32 sequences per request.

Nested types

IgBertEncodeRequestItem

Field	Type	Required	Description
`heavy_chain`	string \| null	no	Antibody heavy-chain amino-acid sequence.
`light_chain`	string \| null	no	Antibody light-chain amino-acid sequence.
`sequence`	string \| null	no	An antibody chain sequence in single-letter amino-acid codes, for unpaired mode.

Raw JSON Schema

{
  "$defs": {
    "IgBertEncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "heavy_chain": {
          "anyOf": [
            {
              "maxLength": 256,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Antibody heavy-chain amino-acid sequence.",
          "title": "Heavy Chain"
        },
        "light_chain": {
          "anyOf": [
            {
              "maxLength": 256,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Antibody light-chain amino-acid sequence.",
          "title": "Light Chain"
        },
        "sequence": {
          "anyOf": [
            {
              "maxLength": 512,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "An antibody chain sequence in single-letter amino-acid codes, for unpaired mode.",
          "title": "Sequence"
        }
      },
      "title": "IgBertEncodeRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 32 sequences per request.",
      "items": {
        "$ref": "#/$defs/IgBertEncodeRequestItem"
      },
      "maxItems": 32,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "IgBertLogProbRequest",
  "type": "object"
}

Response — IgBertLogProbResponse

Field	Type	Required	Constraints	Description
`results`	list[IgBertLogProbResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

IgBertLogProbResponseResult

Field	Type	Required	Constraints	Description
`log_prob`	number	yes		Pseudo-log-likelihood of the sequence under the model.

Raw JSON Schema

{
  "$defs": {
    "IgBertLogProbResponseResult": {
      "properties": {
        "log_prob": {
          "description": "Pseudo-log-likelihood of the sequence under the model.",
          "title": "Log Prob",
          "type": "number"
        }
      },
      "required": [
        "log_prob"
      ],
      "title": "IgBertLogProbResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/IgBertLogProbResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "IgBertLogProbResponse",
  "type": "object"
}

Usage¶

One-line summary: BERT-based antibody language model with paired and unpaired variants that produces sequence embeddings, masked residue predictions, and log-probability scores for immunoglobulin sequences.

Overview¶

IgBERT (Immunoglobulin BERT) is an antibody language model developed by Kenlay, Dreyer, Sherborne, and Deane. It is based on the BERT architecture, fine-tuned from a general protein language model on large-scale antibody sequence data. IgBERT is available in two variants: a paired variant that processes concatenated heavy-light chain sequences and an unpaired variant for individual chain analysis.

IgBERT is part of the same research effort as IgT5 (also available in this catalog), with both models published in "Large scale paired antibody language models" (Kenlay et al., 2024). The key distinction is architecture: IgBERT uses BERT (encoder-only), while IgT5 uses the T5 encoder.

Architecture¶

Property	Value
Architecture	BERT (BertForMaskedLM)
Training objective	Masked language modeling (MLM)
Training data	Large-scale antibody sequences (Exscientia)
Tokenizer	BertTokenizer (character-level, case-sensitive)
Embedding dimension	768
License	MIT (HuggingFace model card)

License note: The HuggingFace model card (Exscientia/IgBert) declares license: mit, which we adopt as the more permissive option. A separate Zenodo deposit of the same weights (DOI 10.5281/zenodo.10876909) lists CC-BY-4.0; since Exscientia is the rights-holder for both releases, we assert MIT and provide attribution to satisfy either license.

Capabilities & Limitations¶

CAN be used for: - Generating mean-pooled, per-residue, or logit embeddings for antibody sequences - Masked residue prediction (sequence completion with * placeholders) - Zero-shot antibody sequence scoring via log-probability (log_prob) - Both paired (heavy+light) and unpaired (single chain) analysis

CANNOT be used for: - Mixed paired/unpaired items in a single request - Non-antibody proteins (use ESM-2 or similar) - 3D structure prediction - Antibody numbering and annotation (use SADIE)

Other considerations: - Both variants require GPU (T4) - Batch size capped at 32 sequences per request - All items in a request must match the deployed variant (all paired or all unpaired)

Usage Examples¶

# Encode -- paired antibody embeddings
from models.igbert.schema import (
    IgBertEncodeRequest,
    IgBertEncodeRequestItem,
    IgBertEncodeRequestParams,
)

encode_request = IgBertEncodeRequest(
    params=IgBertEncodeRequestParams(include=["mean"]),
    items=[
        IgBertEncodeRequestItem(
            heavy_chain="QVQLVQSGAEVKKPGASVKVSCKVSGYTSPTTIHWVRQAPGKGLEWMG",
            light_chain="DIQMTQSPSSVSASVGDRVTITCRASQSIGSFLAWYQQKPGKAPKLLIY",
        ),
    ],
)

# Encode -- unpaired single chain
encode_unpaired = IgBertEncodeRequest(
    params=IgBertEncodeRequestParams(include=["mean", "residue"]),
    items=[
        IgBertEncodeRequestItem(
            sequence="QVQLVQSGAEVKKPGASVKVSCKVSGYTSPTTIHWVRQAPGKGLEWMG",
        ),
    ],
)

# Generate -- restore missing residues (paired)
from models.igbert.schema import IgBertGenerateRequest, IgBertGenerateRequestItem

generate_request = IgBertGenerateRequest(
    items=[
        IgBertGenerateRequestItem(
            heavy_chain="QVQLVQSG*EVKKPGASVKVSCKVSGYTSPTTI*WVRQAPGKGLEWMG",
            light_chain="DIQMTQSPSSVSASVGDRVTITCRASQ*IGSFLAWYQQKPGKAPKLLIY",
        ),
    ],
)

# Predict log probability
from models.igbert.schema import IgBertLogProbRequest, IgBertEncodeRequestItem

log_prob_request = IgBertLogProbRequest(
    items=[
        IgBertEncodeRequestItem(
            heavy_chain="QVQLVQSGAEVKKPGASVKVSCKVSGYTSPTTIHWVRQAPGKGLEWMG",
            light_chain="DIQMTQSPSSVSASVGDRVTITCRASQSIGSFLAWYQQKPGKAPKLLIY",
        ),
    ],
)

Architecture & training¶

Architecture¶

Model Type & Innovation¶

IgBERT is an antibody language model based on the BERT (bidirectional transformer encoder) architecture. It is initialized from a general protein language model and then fine-tuned on antibody sequences, allowing it to benefit from broad protein knowledge while specializing for immunoglobulin sequence understanding.

The key innovation of IgBERT is the availability of both paired and unpaired variants. The paired variant (IgBert) processes concatenated heavy-light chain sequences with a [SEP] separator, learning cross-chain dependencies. The unpaired variant (IgBert_unpaired) processes individual chains, enabling analysis when only single-chain sequences are available. Both variants are trained at scale using the Exscientia antibody dataset.

IgBERT uses the standard BERT architecture with BertForMaskedLM from HuggingFace Transformers, making it compatible with the broader Transformers ecosystem for fine-tuning and transfer learning.

Parameters & Layers¶

Variant	Model ID	Input Type	Max Seq Length
`igbert-paired`	IgBert	Heavy + `[SEP]` + Light	256 per chain
`igbert-unpaired`	IgBert_unpaired	Single chain	512

Property	Value
Architecture	BERT (BertForMaskedLM)
Training objective	Masked language modeling (MLM)
Tokenizer	BertTokenizer (character-level, case-sensitive)
Embedding dimension	768 (BERT-base)

Training Data¶

Property	Details
Dataset	Large-scale antibody sequences (Exscientia)
Composition	Antibody heavy and light chains
Pre-training base	General protein language model (fine-tuned)

Loss Function & Objective¶

Masked language modeling (MLM) with cross-entropy loss:

L = -Sum_i log P(x_masked_i | x_visible)

Standard BERT masking strategy applied to antibody sequences. For the paired variant, the model sees both chains simultaneously, learning cross-chain contextual dependencies.

Tokenization / Input Processing¶

Property	Details
Tokenizer	BertTokenizer (case-sensitive, character-level for amino acids)
Paired input	`H E A V Y [SEP] L I G H T` (space-separated residues)
Unpaired input	`S E Q U E N C E` (space-separated residues)
Special tokens	`[CLS]`, `[SEP]`, `[PAD]`, `[MASK]`
Max paired length	256 residues per chain
Max unpaired length	512 residues
Batch size	32 sequences

The tokenizer operates at the character level with spaces between amino acids, allowing each residue to be an individual token.

Performance & Benchmarks¶

Published Benchmarks¶

The IgBERT paper (arXiv: 2403.17889) evaluates paired and unpaired models on antibody-specific tasks including binding affinity prediction and CDR embedding quality. Key findings from the paper: - Paired models capture heavy-light chain co-evolution signals - Fine-tuning from general protein LMs improves antibody representation quality - Scale of training data matters for antibody language model performance

BioLM Verification Results¶

The BioLM implementation loads official pre-trained weights from HuggingFace via BertForMaskedLM.from_pretrained(). Numerical verification is performed against golden reference outputs:

Metric	Threshold	Status
Relative tolerance	1e-4	PASS

Tests cover encode, generate, and log_prob actions for both paired and unpaired variants.

Comparison to Alternatives¶

Model	Type	Key Advantage	Key Disadvantage
IgBERT (this)	Antibody LM	Paired + unpaired variants, HuggingFace compatible	No germline debiasing
AbLang2	Antibody LM	Germline-debiased representations	Paired only, requires custom library
IgT5	Antibody LM	T5 encoder, same paper	Encode only, no generate or log_prob
ESM-2	General protein LM	Broad protein coverage, multiple sizes	Not antibody-specialized

Error Bars & Confidence¶

IgBERT is deterministic when seeds are set. The same input produces the same output on the same hardware.

Sources of variability: - Different GPU architectures may produce slightly different floating-point results (within 1e-4 relative tolerance)

Strengths & Limitations¶

Pros¶

Both paired and unpaired variants available
HuggingFace Transformers compatible (easy to fine-tune)
GPU-accelerated inference on T4
Multiple output modes: mean embeddings, residue embeddings, logits
Sequence restoration (generate) supported for both paired and unpaired
Log-probability scoring for variant assessment

Cons¶

Paired and unpaired are separate deployments (cannot mix in one request)
No germline debiasing (unlike AbLang2)
MIT per the HuggingFace model card; Zenodo lists CC-BY-4.0
Single model size only (no size variants)

Known Failure Modes¶

Mixed paired/unpaired requests: All items in a batch must be the same type (paired or unpaired); mixed requests will raise an error
Very short sequences: Sequences shorter than ~10 residues may produce low-quality embeddings
Non-antibody input: The model expects immunoglobulin sequences; non-antibody input will produce degraded representations

Implementation Details¶

Inference Pipeline¶

Request
  |-- 1. Validate sequences (alphabet, length, paired vs unpaired)
  |-- 2. Infer request type (_kind: paired or unpaired)
  |-- 3. Verify request type matches deployed model variant
  |-- 4. Format input:
  |     |-- Paired: "H E A V Y [SEP] L I G H T"
  |     |-- Unpaired: "S E Q U E N C E"
  |-- 5. Tokenize with BertTokenizer (batch_encode_plus)
  |-- 6. Forward pass on GPU (torch.no_grad)
  |     |-- Encode: hidden states -> mean pool / residue / logits
  |     |-- Generate: [MASK] -> argmax over canonical AAs
  |     |-- Log prob: log_softmax -> sum non-special positions
  |-- 7. Return typed response

Memory & Compute Profile¶

Variant	GPU	Memory	CPU
`igbert-paired`	T4	6 GB	3 cores
`igbert-unpaired`	T4	6 GB	3 cores

Determinism & Reproducibility¶

Setting	Value
`torch.manual_seed`	42
`torch.cuda.manual_seed_all`	42
`torch.no_grad`	Yes (inference)
`model.eval()`	Yes
GPU memory snapshot	Enabled

Caching Behavior¶

Response caching is handled outside the model container at the serving layer. The model container itself is stateless with respect to caching.

Versions & Changelog¶

Version	Date	Changes
v1	2025-01-30	Initial implementation with encode, generate, log_prob actions

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

IgBERT is trained on immunoglobulin (antibody) sequences, supporting both paired heavy-light chain analysis and individual chain analysis depending on the deployed variant. The model operates at the amino acid level.

IgBERT handles different antibody regions and contexts:

Variable domains (VH/VL): Primary target. Framework and CDR regions are both well-represented.
Paired sequences: The paired variant captures heavy-light chain co-evolution, which is critical for understanding binding specificity.
Unpaired sequences: The unpaired variant can process individual heavy or light chains when paired data is unavailable.
Constant regions: Can be included in unpaired mode but are not the primary focus.

Cross-Applicability¶

Molecule Type	Applicability	Evidence	Caveats
IgG antibodies (paired)	High	Primary training target for paired variant	Use `igbert-paired` variant
IgG antibodies (single chain)	High	Primary training target for unpaired variant	Use `igbert-unpaired` variant
Nanobodies (VHH)	Low--Moderate	Single-domain; use unpaired variant	Not specifically trained on nanobodies
TCRs	Low	Some structural homology to immunoglobulins	Not trained on TCR data
General proteins	Not applicable	Antibody-specialized model	Use ESM-2 or similar

Biological Problems Addressed¶

Paired Antibody Representation¶

Problem: Antibody function depends on the combined properties of the heavy and light chain variable domains. Traditional analysis treats chains independently, missing important inter-chain interactions that determine binding specificity, affinity, and developability.

How IgBERT helps: The paired variant (igbert-paired) processes both chains jointly with a [SEP] separator, learning contextual representations that account for heavy-light chain pairing. The encode action produces embeddings that capture paired chain properties.

Biological meaning: Paired embeddings encode information about the combined paratope formed by VH and VL domains. Two antibodies with identical heavy chains but different light chains will produce different paired embeddings, reflecting the biological reality that light chain identity significantly influences binding specificity.

Antibody Sequence Completion¶

Problem: Antibody sequences from high-throughput screening or NGS may have missing or uncertain residues. Computationally predicting these positions using antibody-specific sequence context can recover functional sequences.

How IgBERT helps: The generate action takes sequences with * placeholders and uses the BERT masked language modeling head to predict the most likely canonical amino acid at each masked position. The prediction uses full sequence context (and cross-chain context for the paired variant).

Biological meaning: The predicted residues at masked positions represent the model's estimate of the most evolutionary and structurally plausible amino acid given the surrounding context. For CDR positions, this reflects the diversity of CDR sequences seen in training; for framework positions, predictions tend to be highly confident and match germline consensus.

Antibody Sequence Scoring¶

Problem: Evaluating whether engineered or mutant antibody sequences are biophysically plausible requires scoring them against known antibody sequence patterns.

How IgBERT helps: The log_prob action computes the total log-probability of a sequence under the IgBERT model by summing log P(residue_i | context) at each position (excluding special tokens). This provides a single scalar score for sequence plausibility.

Biological meaning: Higher (less negative) log-probability scores indicate sequences more consistent with natural antibody patterns. Comparing scores between wild-type and mutant sequences can identify mutations that disrupt conserved structural motifs or introduce unfavorable interactions.

Applied Use Cases¶

IgBERT is applicable to several antibody engineering and analysis workflows:

Antibody repertoire analysis: Embed and cluster paired or unpaired antibody sequences to identify clonal families and convergent evolution (published)
Paired chain association: Use paired embeddings to study heavy-light chain pairing preferences (published)
Sequence completion: Restore missing CDR or framework residues from partial sequences (anticipated)
Variant scoring: Rank engineered antibody variants by sequence plausibility (anticipated)
Feature extraction for ML: Use IgBERT embeddings as input features for downstream property predictors (anticipated)

Complementary Models¶

IgBERT is published alongside IgT5, from the same paper:

IgT5: T5-based antibody encoder from the same authors. Provides embedding-only functionality with a different architecture. IgBERT uses BERT (encoder-only), while IgT5 uses T5 (encoder from encoder-decoder).

Other complementary models in this catalog:

SADIE: Use for antibody numbering and annotation before IgBERT embedding
AbLang2: Alternative antibody LM with germline debiasing

Typical multi-model workflows: 1. Use SADIE to annotate and number sequences 2. Use IgBERT encode to generate embeddings for clustering 3. Use IgBERT log_prob to score engineered variants

Alternative Models¶

Alternative	Advantage Over IgBERT	Disadvantage vs IgBERT
AbLang2	Germline debiasing, restore mode	Paired only, custom library
IgT5	Same paper, T5 architecture	Encode only, no generate or log_prob
ESM-2	Broad protein coverage, multiple sizes	Not antibody-specialized
AntiBERTy	Large OAS training set	Single-chain only

When to choose IgBERT: Use IgBERT when you need a HuggingFace-compatible antibody model with both paired and unpaired variants, or when you need both embedding and sequence generation capabilities.

When to choose alternatives: Consider AbLang2 for germline-debiased representations; consider IgT5 for T5-architecture embeddings; use the unpaired variant for nanobody sequences.

Biological Background¶

Immunoglobulins (antibodies) are the primary effector molecules of the humoral adaptive immune system. Their remarkable diversity arises from combinatorial V(D)J recombination, junctional diversity, and somatic hypermutation, generating an estimated >10^13 unique antibody sequences in the human body.

Paired chain biology: The antigen-binding specificity of an antibody is determined by the combined properties of the heavy chain variable domain (VH) and the light chain variable domain (VL). The six CDR loops (three from each chain) form the paratope that contacts the antigen. Heavy-light chain pairing is not random -- certain VH-VL combinations are favored due to structural compatibility at the VH-VL interface. Language models trained on paired sequences can learn these pairing preferences.

Scale and diversity: Large-scale antibody sequencing studies (e.g., from the Observed Antibody Space and Exscientia datasets) have generated millions of paired antibody sequences. Training language models at this scale enables them to learn the rules governing antibody sequence diversity, including germline gene usage, CDR length distributions, and somatic hypermutation patterns.

Key terminology: - Paired sequences: Matched heavy and light chains from the same B cell, preserving the natural pairing. - Unpaired sequences: Individual heavy or light chains analyzed independently, without pairing information. - V(D)J recombination: Somatic DNA rearrangement that generates the initial antibody variable domain sequence. - Paratope: The antigen-binding surface formed by CDR loops from both VH and VL domains. - Clonal family: A group of antibodies derived from the same V(D)J rearrangement event, related by somatic hypermutation.

Sources & license¶

License: MIT (text) — HuggingFace model card (Exscientia/IgBert) declares "license: mit", which we adopt as the more permissive option (owner-set metadata, authoritative). A separate Zenodo deposit of the same weights (DOI 10.5281/zenodo.10876909) lists CC-BY-4.0. Since Exscientia is the rights-holder for both releases, we assert MIT and provide attribution to satisfy either license. No Exscientia GitHub repo exists to cross-check.

Papers

Large scale paired antibody language models — arXiv preprint, 2024 · arXiv

Source repositories

huggingface: https://huggingface.co/Exscientia/IgBert

Cite

@article{kenlay2024large,
  title={Large scale paired antibody language models},
  author={Kenlay, Henry and Dreyer, Fr{\'e}d{\'e}ric A and Sherborne, Berton and Deane, Charlotte M},
  journal={arXiv preprint arXiv:2403.17889},
  year={2024}
}

IgBert¶

At a glance¶

API & schema¶

encode¶

generate¶

log_prob¶

Usage¶

Overview¶

Architecture¶

Capabilities & Limitations¶

Usage Examples¶

Architecture & training¶

Architecture¶

Model Type & Innovation¶

Parameters & Layers¶

Training Data¶

Loss Function & Objective¶

Tokenization / Input Processing¶

Performance & Benchmarks¶

Published Benchmarks¶

BioLM Verification Results¶

Comparison to Alternatives¶

Error Bars & Confidence¶

Strengths & Limitations¶

Pros¶

Cons¶

Known Failure Modes¶

Implementation Details¶

Inference Pipeline¶

Memory & Compute Profile¶

Determinism & Reproducibility¶

Caching Behavior¶

Versions & Changelog¶

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

Cross-Applicability¶

Biological Problems Addressed¶

Paired Antibody Representation¶

Antibody Sequence Completion¶

Antibody Sequence Scoring¶

Applied Use Cases¶

Related Models¶

Complementary Models¶

Alternative Models¶

Biological Background¶

Sources & license¶

`encode`¶

`generate`¶

`log_prob`¶