Skip to content

TemBERTure

Protein thermostability prediction using adapter-tuned ProtBERT, providing both thermophilicity classification and melting temperature regression from sequence alone.

License: MIT · Molecules: protein · Tasks: embedding, sequence_classification, property_prediction

Actions: encode, predict · Variants: 2

At a glance

Use it when

  • You need a quick binary assessment of whether a protein is thermophilic or non-thermophilic for large-scale classification of protein databases or metagenomic datasets
  • You want thermostability-aware protein embeddings for use in custom downstream ML models (clustering, transfer learning, feature engineering)
  • You need both classification and quantitative Tm prediction in a unified model family with consistent embedding representations
  • You prefer a self-contained model without ESM2 dependency for simpler deployment and operation
  • You are performing initial thermostability screening where binary classification is sufficient before deeper analysis with other tools
  • You want to generate embeddings that specifically capture thermal stability features, rather than general protein representations from ESM2
Strengths
  • Self-contained model with its own ProtBERT-BFD backbone -- does not require ESM2 as a dependency, simplifying deployment and reducing inference pipeline complexity
  • Dual-mode capability: classifier variant provides binary thermophilic/non-thermophilic classification (83% accuracy on TemStaPro benchmark), regression variant predicts quantitative Tm in degrees C
  • Thermostability-aware embeddings via the encode action capture stability-relevant features learned during fine-tuning, useful as input features for custom downstream models
  • Three output types from a single model family: embeddings, classification probabilities, and scalar Tm predictions -- versatile for different analysis workflows
  • Handles proteins up to 512 residues from all domains of life, covering both thermophilic and mesophilic organisms
  • Published in Bioinformatics Advances (2024) with clear benchmarking against TemStaPro and other thermostability prediction methods
  • MIT license enables unrestricted commercial use in enzyme engineering and pharmaceutical development
Limitations
  • Requires T4 GPU with 16 GB RAM for both classifier and regression variants due to the ProtBERT-BFD backbone -- significantly more resource-intensive than lightweight CPU-based stability predictors
  • Maximum sequence length of 512 residues is more restrictive than ESM2-based alternatives (1022 residues), excluding many larger proteins
  • Classification provides only binary thermophilic/non-thermophilic distinction -- does not quantify the degree of thermostability within each class
  • Regression accuracy for quantitative Tm prediction has not been extensively benchmarked in independent studies
  • No per-mutation resolution -- predicts global protein stability properties without identifying which specific residues or mutations influence thermostability
  • ProtBERT-BFD backbone is an older protein language model compared to ESM2, potentially capturing less nuanced sequence representations
  • No support for organism growth temperature context that could improve prediction accuracy for proteins from characterized organisms
Reach for something else when
  • You need a quantitative Tm regressor whose accuracy has been extensively validated in independent studies -- TemBERTure's regression mode has limited external benchmarking to date
  • You need per-mutation ddG predictions to identify specific stabilizing mutations -- use ThermoMPNN or SPURS with a 3D structure
  • Your sequences exceed 512 residues -- use ESM2-based approaches, which handle longer sequences (up to ~1022 residues)
  • You want CPU-only inference to minimize costs -- TemBERTure requires a GPU for its ProtBERT-BFD backbone
  • You need pH-dependent or formulation-specific stability predictions -- no current model in this catalog supports this

API & schema

Variants

Variant Endpoint slug GPU CPU Memory
classifier temberture-classifier t4 4.0 16 GB
regression temberture-regression t4 4.0 16 GB

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

encode

Call it

curl -X POST http://127.0.0.1:8000/api/v1/temberture-classifier/encode \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

RequestTemBERTureEncodeRequest

Field Type Required Constraints Description
params TemBERTureEncodeRequestParams no Optional parameters controlling this action (defaults are used when omitted).
items list[TemBERTureEncodeRequestItem] yes items 1–8 Batch of inputs to process in a single request. Up to 8 sequences per request.
Nested types

TemBERTureEncodeIncludeOptions

Allowed values: mean, per_residue, cls

TemBERTureEncodeRequestItem

Field Type Required Constraints Description
sequence string yes len 1–512 A protein sequence in single-letter amino-acid codes.

TemBERTureEncodeRequestParams

Field Type Required Constraints Description
include list[TemBERTureEncodeIncludeOptions] no default ['mean'] Optional outputs to compute and include in the response.
Raw JSON Schema
{
  "$defs": {
    "TemBERTureEncodeIncludeOptions": {
      "enum": [
        "mean",
        "per_residue",
        "cls"
      ],
      "title": "TemBERTureEncodeIncludeOptions",
      "type": "string"
    },
    "TemBERTureEncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes.",
          "maxLength": 512,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "TemBERTureEncodeRequestItem",
      "type": "object"
    },
    "TemBERTureEncodeRequestParams": {
      "additionalProperties": false,
      "properties": {
        "include": {
          "description": "Optional outputs to compute and include in the response.",
          "items": {
            "$ref": "#/$defs/TemBERTureEncodeIncludeOptions"
          },
          "title": "Include",
          "type": "array",
          "default": [
            "mean"
          ]
        }
      },
      "title": "TemBERTureEncodeRequestParams",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "params": {
      "$ref": "#/$defs/TemBERTureEncodeRequestParams",
      "description": "Optional parameters controlling this action (defaults are used when omitted)."
    },
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 8 sequences per request.",
      "items": {
        "$ref": "#/$defs/TemBERTureEncodeRequestItem"
      },
      "maxItems": 8,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "TemBERTureEncodeRequest",
  "type": "object"
}

ResponseTemBERTureEncodeResponse

Field Type Required Constraints Description
results list[TemBERTureEncodeResponseResult] yes Per-input results, returned in the same order as the request items.
Nested types

TemBERTureEncodeResponseResult

Field Type Required Constraints Description
sequence_index integer yes Index of the corresponding input sequence within the request batch.
embeddings list[number] | null no Mean-pooled embedding vector for the sequence (present when 'mean' is requested).
residue_embeddings list[list[number]] | null no Per-residue embedding vectors.
cls_embeddings list[number] | null no CLS token embedding vector from the ProtBERT encoder (present when 'cls' is requested).
Raw JSON Schema
{
  "$defs": {
    "TemBERTureEncodeResponseResult": {
      "exclude_none": true,
      "exclude_unset": true,
      "properties": {
        "sequence_index": {
          "description": "Index of the corresponding input sequence within the request batch.",
          "title": "Sequence Index",
          "type": "integer"
        },
        "embeddings": {
          "anyOf": [
            {
              "items": {
                "type": "number"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Mean-pooled embedding vector for the sequence (present when 'mean' is requested).",
          "title": "Embeddings"
        },
        "residue_embeddings": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-residue embedding vectors.",
          "title": "Residue Embeddings"
        },
        "cls_embeddings": {
          "anyOf": [
            {
              "items": {
                "type": "number"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "CLS token embedding vector from the ProtBERT encoder (present when 'cls' is requested).",
          "title": "Cls Embeddings"
        }
      },
      "required": [
        "sequence_index"
      ],
      "title": "TemBERTureEncodeResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/TemBERTureEncodeResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "TemBERTureEncodeResponse",
  "type": "object"
}

predict

Call it

curl -X POST http://127.0.0.1:8000/api/v1/temberture-classifier/predict \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

RequestTemBERTurePredictRequest

Field Type Required Constraints Description
items list[TemBERTurePredictRequestItem] yes items 1–8 Batch of inputs to process in a single request. Up to 8 sequences per request.
Nested types

TemBERTurePredictRequestItem

Field Type Required Constraints Description
sequence string yes len 1–512 A protein sequence in single-letter amino-acid codes.
Raw JSON Schema
{
  "$defs": {
    "TemBERTurePredictRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes.",
          "maxLength": 512,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "TemBERTurePredictRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 8 sequences per request.",
      "items": {
        "$ref": "#/$defs/TemBERTurePredictRequestItem"
      },
      "maxItems": 8,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "TemBERTurePredictRequest",
  "type": "object"
}

ResponseTemBERTurePredictResponse

Field Type Required Constraints Description
results list[TemBERTurePredictResponseResult] yes Per-input results, returned in the same order as the request items.
Nested types

TemBERTurePredictResponseResult

Field Type Required Constraints Description
score number yes Thermophilicity probability (0-1) in classifier mode, or melting temperature in degrees Celsius in regression mode.
classification string | null no Thermophilicity label ('Thermophilic' or 'Non-thermophilic'); present only for the classifier variant.
Raw JSON Schema
{
  "$defs": {
    "TemBERTurePredictResponseResult": {
      "properties": {
        "score": {
          "description": "Thermophilicity probability (0-1) in classifier mode, or melting temperature in degrees Celsius in regression mode.",
          "title": "Score",
          "type": "number"
        },
        "classification": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Thermophilicity label ('Thermophilic' or 'Non-thermophilic'); present only for the classifier variant.",
          "title": "Classification"
        }
      },
      "required": [
        "score"
      ],
      "title": "TemBERTurePredictResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/TemBERTurePredictResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "TemBERTurePredictResponse",
  "type": "object"
}

Usage

One-line summary: Protein thermostability prediction using adapter-tuned ProtBERT, providing both thermophilicity classification and melting temperature regression from sequence alone.

Overview

TemBERTure is a deep learning model for protein thermostability prediction developed by Rodella et al. (2024) at the University of Bern. It fine-tunes ProtBERT-BFD with adapter modules to predict whether a protein is thermophilic or non-thermophilic (classifier) and to estimate melting temperature in degrees Celsius (regression). The model operates on amino acid sequences alone, requiring no structural information.

Architecture

Property Value
Architecture BERT encoder + adapter layers
Base model ProtBERT-BFD
Hidden dimensions 1024
Max sequence length 512 residues
Batch size 8
Adapter type Bottleneck adapters (AdapterBERT)

Capabilities & Limitations

CAN be used for: - Classifying proteins as thermophilic or non-thermophilic (classifier variant) - Predicting melting temperature (Tm) in degrees Celsius (regression variant) - Extracting protein embeddings (mean, per-residue, CLS) from the fine-tuned model - Screening protein libraries for thermostable candidates

CANNOT be used for: - Sequences longer than 512 residues (rejected with a validation error, not truncated) - Structure-based stability prediction (use ThermoMPNN instead) - Per-mutation ddG prediction (use ThermoMPNN instead) - Non-standard amino acids beyond the extended alphabet plus gap character

Other considerations: - The classifier variant returns a probability and label ("Thermophilic" or "Non-thermophilic") - The regression variant returns a raw Tm value in degrees C - Both variants share the same ProtBERT base model but use different adapter heads

Usage Examples

from models.temberture.schema import (
    TemBERTureEncodeRequest,
    TemBERTureEncodeRequestItem,
    TemBERTureEncodeRequestParams,
    TemBERTureEncodeIncludeOptions,
    TemBERTurePredictRequest,
    TemBERTurePredictRequestItem,
)

# Encode request
encode_request = TemBERTureEncodeRequest(
    params=TemBERTureEncodeRequestParams(
        include=[
            TemBERTureEncodeIncludeOptions.MEAN,
            TemBERTureEncodeIncludeOptions.CLS,
        ]
    ),
    items=[
        TemBERTureEncodeRequestItem(
            sequence="MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVLAGG"
        ),
    ],
)

# Predict request
predict_request = TemBERTurePredictRequest(
    items=[
        TemBERTurePredictRequestItem(
            sequence="MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVLAGG"
        ),
    ],
)

Architecture & training

Architecture

Model Type & Innovation

TemBERTure is a fine-tuned ProtBERT model that predicts protein thermostability. It uses adapter modules on top of a pre-trained BERT model (ProtBERT-BFD) to classify proteins as thermophilic or non-thermophilic (classifier variant) and to predict melting temperature (Tm) in degrees Celsius (regression variant). The key innovation is the use of lightweight adapter layers for transfer learning, allowing efficient fine-tuning without modifying the full ProtBERT backbone.

Parameters & Layers

Component Details
Architecture BERT encoder with adapter layers
Base model ProtBERT-BFD (Rostlab/prot_bert_bfd)
Hidden dimensions 1024
Adapter type AdapterBERT (bottleneck adapters)
Tokenization Character-level (space-separated amino acids)
Max sequence length 512 residues

Training Data

The model is trained on a curated dataset of protein sequences labeled with thermostability properties (thermophilic/non-thermophilic classification and melting temperature values). Full training dataset details are available in Rodella et al. (2024), Bioinformatics Advances, DOI 10.1093/bioadv/vbae103.

Loss Function & Objective

  • Classifier variant: Binary cross-entropy loss for thermophilic vs non-thermophilic classification
  • Regression variant: Mean squared error (MSE) loss for melting temperature (Tm) prediction in degrees Celsius

Tokenization / Input Processing

ProtBERT uses character-level tokenization where each amino acid is treated as a separate token, with spaces inserted between residues. Special tokens include [CLS] (classification), [SEP] (separator), and [PAD] (padding). Sequences are truncated to the maximum length of 512 tokens.

Performance & Benchmarks

Published Benchmarks

The TemBERTure classifier achieves 83% accuracy on the TemStaPro benchmark dataset. Additional benchmark results are reported in Rodella et al. (2024), Bioinformatics Advances.

BioLM Verification Results

Both classifier and regression variants have been verified through integration tests with golden output comparison (relative tolerance 1e-4, cosine distance threshold 0.02 for embeddings).

Comparison to Alternatives

Model Task Input Advantage
TemBERTure Thermophilicity + Tm Sequence only Dual prediction (classification + regression)
ThermoMPNN ddG prediction Structure (PDB) Structure-aware, per-mutation predictions

Strengths & Limitations

Pros

  • Dual-mode: classification (thermophilic/non-thermophilic) and regression (Tm in degrees C)
  • Sequence-only input -- no structure required
  • Lightweight adapter approach enables efficient fine-tuning
  • Provides embeddings for downstream tasks (mean, per-residue, CLS)

Cons

  • Limited to 512 residues maximum
  • Based on older ProtBERT architecture (not ESM2)
  • Regression predictions may have limited accuracy for proteins far from training distribution

Known Failure Modes

  • Proteins longer than 512 residues are rejected at the API boundary by the request-schema length limit (they are not silently truncated), so they must be shortened or split before submission
  • Non-standard amino acids beyond extended alphabet may cause tokenization issues

Implementation Details

Inference Pipeline

Request --> Validate sequences --> Space-separate AAs --> Tokenize (ProtBERT)
  --> [GPU] Forward pass through BERT + adapter --> Post-process --> Response

For encode: extracts hidden states from last layer, computes mean/per-residue/CLS embeddings. For predict: passes through classification/regression head, applies sigmoid (classifier) or returns raw value (regression).

Memory & Compute Profile

Resource Value
GPU T4
Memory 16 GB
CPU 4 cores

Determinism & Reproducibility

  • Torch manual seed: Yes (42)
  • CUDA manual seed: Yes (42)
  • Model set to eval mode: Yes
  • Inference under torch.no_grad(): Yes

Caching Behavior

Response caching is handled by the serving layer, not by the model container.

Versions & Changelog

Version Date Changes
v1 2024 Initial implementation with classifier and regression variants

Biology

Molecule Coverage

Primary Molecule Type(s)

TemBERTure is designed for protein sequences from all domains of life. It handles single-chain protein sequences up to 512 residues. The model was trained on proteins with known thermostability properties, covering both thermophilic organisms (optimal growth above 45 degrees C) and mesophilic organisms.

Cross-Applicability

Molecule Type Applicability Evidence Caveats
Globular proteins High Primary training target Best for proteins with known structural context
Enzymes High Well-represented in training data Industrial enzyme engineering use case
Nanobodies Moderate Single-domain antibody fragments are proteins Not specialized for nanobody-specific Tm prediction
Peptides Low Short sequences may lack sufficient context Minimum meaningful length depends on the fold
Membrane proteins Unknown May be under-represented in training Use predictions with caution

Biological Problems Addressed

Thermostability Classification

Protein thermostability is a critical property for biotechnology and industrial enzyme applications. Thermostable proteins maintain their structure and function at elevated temperatures, making them valuable for applications such as industrial catalysis, therapeutic protein development, and food processing. Traditional experimental methods for measuring thermostability (differential scanning calorimetry, thermal shift assays) are time-consuming and require purified protein.

TemBERTure's classifier variant predicts whether a protein is thermophilic (stable at high temperatures) or non-thermophilic from sequence alone. This enables rapid screening of large protein libraries for thermostable candidates without experimental measurement.

Melting Temperature Prediction

The melting temperature (Tm) is the temperature at which 50% of a protein population is unfolded. It is a quantitative measure of protein thermal stability. Knowing the Tm helps in:

  • Protein engineering: designing mutations that increase stability
  • Drug formulation: selecting storage conditions for therapeutic proteins
  • Industrial applications: choosing enzymes that function at process temperatures

TemBERTure's regression variant predicts the Tm in degrees Celsius from sequence alone, providing a quantitative stability estimate.

Applied Use Cases

  • TemStaPro (Pudžiuvelytė et al., 2024) — Concurrent thermostability prediction method using protein language model embeddings; TemBERTure achieves 83% accuracy on the TemStaPro benchmark dataset, providing a direct performance comparison. DOI: 10.1093/bioinformatics/btae157.

  • ESMStabP (Ramos et al., 2025) — Incorporates TemBERTure thermophilic classifications as an input feature alongside ESM2 embeddings for melting temperature regression (R² = 0.95, PCC = 0.97); directly validates TemBERTure's classification outputs as informative features for quantitative Tm prediction. DOI: 10.1101/2025.02.18.638450.

  • Computation of Protein Thermostability and Epistasis (2025) — Comprehensive WIREs review covering computational strategies for protein stability prediction including TemBERTure and other deep learning approaches, contextualising their strengths and limitations for protein engineering. DOI: 10.1002/wcms.70045.

  • iCASE strategy for industrial enzyme engineering (2025) — Applies machine learning-based strategies for industrial enzyme thermostability engineering, representing the practical downstream application domain that TemBERTure predictions support. DOI: 10.1038/s41467-025-55944-5.

  • Click, Compute, Create — enzyme engineering web tools review (Tripp et al., 2024) — Comprehensive review of web-based enzyme engineering tools including TemBERTure, cataloguing it as a key thermostability prediction resource for computational enzyme design workflows. DOI: 10.1002/cbic.202400092.

Predecessor Models

TemBERTure builds on ProtBERT-BFD (Elnaggar et al., 2022), a BERT model pre-trained on the BFD protein database. ProtBERT provides the base representations; TemBERTure adds task-specific adapter layers for thermostability prediction.

Complementary Models

  • ThermoMPNN / ThermoMPNN-D: Structure-based stability prediction (ddG for mutations). Use when you have a 3D structure and want to evaluate specific mutations.
  • ESM2: General-purpose protein language model. Use ESM2 embeddings for broader protein property prediction tasks.

Alternative Models

Alternative Advantage over TemBERTure Disadvantage
ThermoMPNN Structure-aware, per-mutation ddG Requires PDB structure input

Biological Background

Proteins are chains of amino acids that fold into three-dimensional structures to perform biological functions. Thermal stability -- the ability of a protein to resist unfolding at elevated temperatures -- is determined by a complex interplay of intramolecular interactions including hydrogen bonds, hydrophobic packing, salt bridges, and disulfide bonds.

Organisms that thrive in extreme temperatures (thermophiles, growing above 45 degrees C; hyperthermophiles, above 80 degrees C) have evolved proteins with enhanced thermal stability. These adaptations include increased hydrophobic core packing, additional salt bridges, shorter surface loops, and higher proline content. Understanding and predicting these stability determinants is valuable for protein engineering, where enhancing thermostability can improve protein shelf life, process efficiency, and therapeutic applicability.


Sources & license

License: MIT (text) — Copyright (c) 2024 Institute of Biochemistry and Molecular Medicine. Base model (ProtBERT-BFD, Rostlab/prot_bert_bfd) has no explicit license on HuggingFace; verify redistribution terms before production use.

Papers

  • TemBERTure: advancing protein thermostability prediction with deep learning and attention mechanisms — Bioinformatics Advances, 2024 · DOI

Source repositories

Cite

@article{rodella2024temberture,
  title={TemBERTure: advancing protein thermostability prediction with deep learning and attention mechanisms},
  author={Rodella, Chiara and Lazaridi, Symela and Lemmin, Thomas},
  journal={Bioinformatics Advances},
  year={2024},
  doi={10.1093/bioadv/vbae103}
}