Skip to content

ChemBERTa

A ~92M-parameter RoBERTa masked language model over SMILES strings that produces fixed-length small-molecule embeddings and a pseudo-log-likelihood plausibility score — opening the small-molecule modality for the catalog.

License: MIT · Molecules: ligand · Tasks: embedding, property_prediction

Actions: encode, log_prob · Variants: 1

At a glance

Use it when

  • You need fast, cheap fixed-length embeddings for small molecules from SMILES to feed a downstream classifier or regressor (QSAR / molecular property prediction)
  • You are building a small-molecule pipeline and need a permissively licensed (MIT) SMILES featurizer that runs on CPU
  • You want a pretrained chemical language model baseline to compare against ECFP fingerprints or graph neural networks on MoleculeNet-style tasks
  • You need a pseudo-log-likelihood 'plausibility' score to rank or flag unusual SMILES (e.g., filtering generative-model outputs) rather than a supervised property value
  • You are clustering, deduplicating, or performing similarity search over a compound library using learned molecular representations
  • You need a chemical embedding to concatenate with protein embeddings (e.g., ESM-2) for a protein–ligand interaction model
Strengths
  • Opens the small-molecule (SMILES) modality for the catalog — a chemical language model where every other sequence model targets protein, DNA, or RNA
  • Lightweight at ~92M parameters and runs comfortably on CPU, making SMILES embedding and scoring cheap for high-throughput virtual screening
  • MIT license (declared via the HuggingFace card) permits unrestricted commercial and academic use, unlike many non-commercial cheminformatics assets
  • Standard RobertaForMaskedLM + byte-level BPE tokenizer — loads without trust_remote_code, so it is robust across transformers versions and easy to audit
  • Byte-level BPE over SMILES tokenizes arbitrary molecules without a hand-curated atom vocabulary, learning multi-character chemical motifs (rings, functional groups) as single tokens
  • Pretrained self-supervised on a large 100M-molecule subset of ZINC20, covering broad drug-like / purchasable chemical space
  • Dual-action API (encode + log_prob) covers both fixed-length molecular embeddings and a pseudo-log-likelihood plausibility score from one deployment
Limitations
  • Encoder-only masked LM: it cannot generate or design novel molecules — it only embeds and scores existing SMILES
  • This checkpoint uses the MLM objective; the ChemBERTa-2 paper reports that the multi-task-regression (MTR) variant often outperforms MLM on supervised property prediction, so MLM embeddings may be suboptimal for that use
  • Independent benchmarking (Praski et al., 2025) finds pretrained molecular embeddings frequently give negligible improvement over the classical ECFP fingerprint baseline — validate lift on your task before adopting
  • Consumes 1D SMILES only — no explicit 3D conformer or molecular-graph information, and results can depend on SMILES canonicalization/kekulization of the input
  • Input validation is a lightweight character/bracket check (no rdkit): it does not guarantee chemical validity or a canonical form, so garbage-but-well-formed SMILES are accepted
  • The log_prob output is a pseudo-log-likelihood summed over byte-level BPE tokens (not per-atom), so it is not directly comparable across molecules of very different SMILES length
  • 512-position context and a ZINC20-drug-like training distribution mean very large molecules (long peptides, polymers) are truncated and exotic chemotypes are out-of-distribution
Reach for something else when
  • You need to generate, optimize, or design novel molecules (ChemBERTa is encoder-only; use a generative chemical model, which the catalog does not yet host)
  • You need a 3D structure or binding pose for a molecule or a protein–ligand complex (use chai1 or boltzgen, which consume SMILES ligands and predict structure)
  • You require guaranteed chemical validity or canonicalization of inputs (run rdkit upstream; this model's validator is a lightweight format check only)
  • You are working with proteins, DNA, or RNA rather than small molecules (use esm2/esmc for protein, dnabert2/evo for DNA)
  • Your task already has a strong ECFP-fingerprint or graph-neural-network baseline that a pretrained SMILES embedding has not been shown to beat
  • You need the best available supervised property predictor and can fine-tune — a task-specific fine-tuned model (or the ChemBERTa-2 MTR variant) will typically outperform frozen MLM embeddings

API & schema

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

encode

Call it

curl -X POST http://127.0.0.1:8000/api/v1/chemberta/encode \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "smiles": "CC(=O)Oc1ccccc1C(=O)O"
    }
  ]
}'

RequestChemBERTaEncodeRequest

Field Type Required Constraints Description
items list[ChemBERTaEncodeRequestItem] yes items 1–16 Batch of inputs to process in a single request. Up to 16 molecules per request.
Nested types

ChemBERTaEncodeRequestItem

Field Type Required Constraints Description
smiles string yes len 1–512 A small molecule represented as a SMILES string.
Raw JSON Schema
{
  "$defs": {
    "ChemBERTaEncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "smiles": {
          "description": "A small molecule represented as a SMILES string.",
          "maxLength": 512,
          "minLength": 1,
          "title": "Smiles",
          "type": "string"
        }
      },
      "required": [
        "smiles"
      ],
      "title": "ChemBERTaEncodeRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 16 molecules per request.",
      "items": {
        "$ref": "#/$defs/ChemBERTaEncodeRequestItem"
      },
      "maxItems": 16,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ChemBERTaEncodeRequest",
  "type": "object"
}

ResponseChemBERTaEncodeResponse

Field Type Required Constraints Description
results list[ChemBERTaEncodeResponseResult] yes Per-input results, returned in the same order as the request items.
Nested types

ChemBERTaEncodeResponseResult

Field Type Required Constraints Description
embedding list[number] yes Mean-pooled embedding vector for the molecule (768 dimensions).
Raw JSON Schema
{
  "$defs": {
    "ChemBERTaEncodeResponseResult": {
      "properties": {
        "embedding": {
          "description": "Mean-pooled embedding vector for the molecule (768 dimensions).",
          "items": {
            "type": "number"
          },
          "title": "Embedding",
          "type": "array"
        }
      },
      "required": [
        "embedding"
      ],
      "title": "ChemBERTaEncodeResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ChemBERTaEncodeResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ChemBERTaEncodeResponse",
  "type": "object"
}

log_prob

Call it

curl -X POST http://127.0.0.1:8000/api/v1/chemberta/log_prob \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "smiles": "CC(=O)Oc1ccccc1C(=O)O"
    }
  ]
}'

RequestChemBERTaLogProbRequest

Field Type Required Constraints Description
items list[ChemBERTaLogProbRequestItem] yes items 1–16 Batch of inputs to process in a single request. Up to 16 molecules per request.
Nested types

ChemBERTaLogProbRequestItem

Field Type Required Constraints Description
smiles string yes len 1–512 A small molecule represented as a SMILES string.
Raw JSON Schema
{
  "$defs": {
    "ChemBERTaLogProbRequestItem": {
      "additionalProperties": false,
      "properties": {
        "smiles": {
          "description": "A small molecule represented as a SMILES string.",
          "maxLength": 512,
          "minLength": 1,
          "title": "Smiles",
          "type": "string"
        }
      },
      "required": [
        "smiles"
      ],
      "title": "ChemBERTaLogProbRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 16 molecules per request.",
      "items": {
        "$ref": "#/$defs/ChemBERTaLogProbRequestItem"
      },
      "maxItems": 16,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ChemBERTaLogProbRequest",
  "type": "object"
}

ResponseChemBERTaLogProbResponse

Field Type Required Constraints Description
results list[ChemBERTaLogProbResponseResult] yes Per-input results, returned in the same order as the request items.
Nested types

ChemBERTaLogProbResponseResult

Field Type Required Constraints Description
log_prob number yes Pseudo-log-likelihood of the sequence under the model.
Raw JSON Schema
{
  "$defs": {
    "ChemBERTaLogProbResponseResult": {
      "properties": {
        "log_prob": {
          "description": "Pseudo-log-likelihood of the sequence under the model.",
          "title": "Log Prob",
          "type": "number"
        }
      },
      "required": [
        "log_prob"
      ],
      "title": "ChemBERTaLogProbResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ChemBERTaLogProbResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ChemBERTaLogProbResponse",
  "type": "object"
}

Usage

One-line summary: A ~92M-parameter RoBERTa masked language model over SMILES strings that produces fixed-length small-molecule embeddings and a pseudo-log-likelihood plausibility score — opening the small-molecule modality for the catalog.

Overview

ChemBERTa is a chemical language model developed by the DeepChem team. It applies the RoBERTa masked-language-modeling (MLM) paradigm to molecules represented as SMILES strings, producing dense molecular embeddings and pseudo-log-likelihood scores useful for property prediction, virtual screening, similarity search, and flagging unusual chemistry.

The checkpoint served here, DeepChem/ChemBERTa-100M-MLM, was pretrained with the MLM objective on a 100M-molecule subset of the ZINC20 database. It is the direct descendant of the ChemBERTa (Chithrananda et al., 2020) and ChemBERTa-2 (Ahmad et al., 2022) line of work, which showed that transformer models over SMILES scale with pretraining set size and are competitive with prior art on MoleculeNet. ChemBERTa is the first small-molecule model in the catalog: where the other sequence models target proteins, DNA, or RNA, ChemBERTa reads chemistry.

Architecture

Property Value
Architecture Transformer encoder (RoBERTa / RobertaForMaskedLM)
Parameters ~92.1M
Hidden dimensions 768
Attention heads 12
Layers 12
Tokenization Byte-level BPE over SMILES (RobertaTokenizer), vocab 7,924
Max positions 512 (usable input ~510 tokens after the RoBERTa position offset)
Training data 100M-molecule subset of ZINC20 (MLM objective)
Precision float32

See MODEL.md for a full architecture and training deep-dive.

Capabilities & Limitations

CAN be used for: - Generating a fixed-length (768-dim) mean-pooled embedding for a small molecule from its SMILES (encode) - Scoring a molecule's pseudo-log-likelihood under the model to rank or flag unusual/out-of-distribution SMILES (log_prob) - Producing molecular features for downstream QSAR / property-prediction classifiers and regressors - Similarity search, clustering, and deduplication over compound libraries

CANNOT be used for: - Generating, optimizing, or designing novel molecules (encoder-only masked LM — no sampling head exposed) - Predicting 3D structure or protein–ligand binding poses (use chai1 or boltzgen, which consume SMILES ligands) - Guaranteeing chemical validity or canonicalization of inputs (validation is a lightweight format check, not rdkit) - Proteins, DNA, or RNA (use esm2/esmc for protein, dnabert2/evo for DNA)

Other considerations: - Results can depend on the exact SMILES writing (canonicalization / kekulization) of the input; canonicalize upstream for consistency. - The log_prob value is a pseudo-log-likelihood summed over byte-level BPE tokens (not per-atom), so it is not directly comparable across molecules of very different SMILES length. - Very long SMILES are truncated to ~510 tokens; the training distribution is drug-like/purchasable ZINC20 chemistry, so exotic chemotypes are out-of-distribution.

Usage Examples

from models.chemberta.schema import (
    ChemBERTaEncodeRequest,
    ChemBERTaLogProbRequest,
)

# encode: SMILES -> 768-dim embedding
encode_request = ChemBERTaEncodeRequest.model_validate(
    {"items": [{"smiles": "CC(=O)Oc1ccccc1C(=O)O"}]}  # aspirin
)

# log_prob: pseudo-log-likelihood of a molecule
logprob_request = ChemBERTaLogProbRequest.model_validate(
    {"items": [{"smiles": "CCO"}, {"smiles": "Cn1cnc2c1c(=O)n(C)c(=O)n2C"}]}  # ethanol, caffeine
)

Architecture & training

Deep technical reference for the served DeepChem/ChemBERTa-100M-MLM checkpoint. For the API and a high-level summary, see README.md; for biological context, see BIOLOGY.md.

Architecture

ChemBERTa is a RoBERTa encoder (RobertaForMaskedLM) applied to molecules serialized as SMILES strings. It is architecturally a standard BERT-family masked language model; the chemistry lives entirely in the tokenizer vocabulary and the training corpus, not in any bespoke layers.

Component Specification
Base class RobertaForMaskedLM (model_type: roberta)
Encoder layers 12
Hidden size 768
Attention heads 12
Intermediate (FFN) size 3,072
Activation GELU
Position embeddings Absolute, learned; max_position_embeddings = 512, padding_idx = 1
Vocabulary 7,924 byte-level BPE tokens learned over SMILES
Tied embeddings Yes (tie_word_embeddings = true)
Total parameters 92,136,436 (~92.1M); encoder alone ~91.5M
Precision float32 (~369 MB on disk, safetensors)

Tokenizer. A byte-level BPE tokenizer (RobertaTokenizer / RobertaTokenizerFast) with special tokens <s> (id 0), <pad> (id 1), </s> (id 2), <unk> (id 3), and <mask> (id 4). SMILES are tokenized verbatim — the tokenizer segments the raw string into learned chemical subwords (whole rings, functional groups, and atoms can each map to single tokens). Because BPE merges characters, token count is typically well below character count (e.g. "CCO" → a single CCO token). This is why the schema carries two separate limits: a 512-character cap on the request field and a 510-token truncation limit for the model (max_position_embeddings 512 minus RoBERTa's position-id offset of 2).

Objective. Masked language modeling: during pretraining ~15% of SMILES tokens are masked and the model predicts them from bidirectional context. This is the same MLM head reused at inference for the log_prob action.

Performance & Benchmarks

Benchmarks are drawn only from the source papers and independent evaluations; no numbers are estimated here.

  • Scaling with data (Chithrananda et al., 2020). The original ChemBERTa study varied the pretraining set from 100K to 10M molecules and showed downstream MoleculeNet performance improves with pretraining set size — motivating the larger checkpoints such as this 100M one.
  • MoleculeNet competitiveness (Ahmad et al., 2022). ChemBERTa-2 reports competitiveness with contemporary state-of-the-art SMILES and graph models across the MoleculeNet suite. It compares two pretraining objectives — masked language modeling (MLM) and multi-task regression (MTR) — and finds the MTR variant generally outperforms MLM on supervised property-prediction tasks. The checkpoint served here is the MLM variant, chosen because it yields a general-purpose embedding and a well-defined pseudo-log-likelihood rather than being tuned to a fixed regression task set.
  • Applied demonstration (MolPROP, Rollins et al., 2024, J. Cheminformatics, DOI 10.1186/s13321-024-00846-9). Fuses ChemBERTa-2 embeddings with a graph neural network and evaluates on seven MoleculeNet datasets (FreeSolv, ESOL, Lipophilicity, QM7, BACE, BBBP, ClinTox) under scaffold splits.
  • Embedding-baseline caveat (Praski et al., 2025, arXiv:2508.06199). A benchmark of pretrained molecular embedding models across 25 datasets reports that most neural embeddings give negligible improvement over the classical ECFP fingerprint — i.e., a frozen SMILES embedding is not automatically better than a cheap fingerprint. Validate the lift on your own task.

Consult the paper tables directly for exact per-task metrics (they vary by split and metric and are best read in context).

Strengths & Limitations

Strengths. - Small (~92M params) and CPU-friendly; cheap for high-throughput embedding/scoring. - Standard RoBERTa — no trust_remote_code, robust across transformers versions, easy to audit. - Byte-level BPE tokenizes arbitrary SMILES without a hand-built atom vocabulary, capturing multi-character chemical motifs as single tokens. - Broad self-supervised pretraining (100M ZINC20 molecules) over drug-like / purchasable space. - MIT-licensed — unrestricted commercial and academic use.

Limitations. - Encoder-only masked LM: no molecular generation/design. - The MLM objective underperforms MTR on supervised property prediction (per ChemBERTa-2). - 1D SMILES only — no explicit 3D conformer or graph; sensitive to SMILES canonicalization. - Pseudo-log-likelihood is summed over BPE tokens (not per-atom), so it is length-dependent and not a normalized per-molecule quantity. - 512-position context; very large molecules are truncated. ZINC20 distribution limits out-of-distribution chemotypes. - Lightweight (non-rdkit) input validation does not guarantee chemical validity.

Implementation Details

  • Deployment. CPU (gpu=None), ModelMixinSnap with enable_memory_snapshot=True; the model is loaded to CPU inside a single @modal.enter(snap=True) and captured in the memory snapshot. Torch seeds are set for reproducibility; the forward pass runs under eval() (no dropout), so outputs are deterministic.
  • Image. debian_slim + CPU torch wheel (torch==2.6.0 from the PyTorch CPU index) + transformers==4.48.1, tokenizers==0.21.0, safetensors==0.5.3, huggingface_hub==0.26.0. No CUDA base image (CPU-only model).
  • Weights. r2_then_hf from DeepChem/ChemBERTa-100M-MLM pinned to f5c45f44d3061f0346888f5c09db17ec1146d29d. R2 cache is the fast primary; HuggingFace is the guaranteed fallback (and self-populates R2 for maintainer deploys).
  • encode. Batch-tokenizes SMILES (padding=True, truncation=True, max_length=510), runs the RoBERTa encoder (model.base_model), and mean-pools the final hidden state over non-padded positions (attention-mask-weighted) → one 768-dim vector per molecule.
  • log_prob. For each molecule, every non-special token position is replaced with <mask> in a batched copy of the sequence; the MLM logits at each masked position are log-softmaxed and the probability of the original token is summed → a per-molecule pseudo-log-likelihood.
  • Batching. Up to 16 molecules per request (ChemBERTaParams.batch_size).

Versions & Changelog

Date Change
2026-07 Initial implementation. Serves DeepChem/ChemBERTa-100M-MLM @ f5c45f44d3061f0346888f5c09db17ec1146d29d. Actions: encode, log_prob. CPU deployment. First small-molecule model in the catalog.

Biology

How ChemBERTa fits into cheminformatics and drug discovery. For the API see README.md; for architecture and training see MODEL.md.

Molecule Coverage

ChemBERTa operates on small molecules encoded as SMILES (Simplified Molecular-Input Line-Entry System) strings — the 1D text serialization of a molecular graph. It is pretrained on a 100M-molecule subset of ZINC20, a database of commercially available (purchasable), largely drug-like compounds. In catalog terms its input molecule type is ligand.

Well covered: organic drug-like small molecules — the kind found in medicinal-chemistry and virtual-screening libraries (typical molecular weights, standard organic elements, common functional groups, ring systems, stereochemistry, and charges expressible in SMILES).

Poorly covered / out of distribution: very large molecules (long peptides, oligonucleotides, polymers) that exceed the ~510-token context; organometallics and exotic inorganic chemistry under-represented in ZINC20; and any molecule whose SMILES exceeds the 512-character input cap. ChemBERTa reads only the 1D SMILES — it has no explicit 3D conformer or stereochemically resolved geometry.

Biological Problems Addressed

Small molecules are the dominant modality in drug discovery, chemical biology, and agrochemical research. Turning a molecule into a fixed-length numerical vector ("molecular featurization") is the gateway to machine learning on chemistry. ChemBERTa addresses:

  • Molecular representation. A learned, dense embedding of a molecule that downstream models can consume — an alternative to hand-crafted descriptors and circular fingerprints (ECFP).
  • Property / activity prediction (QSAR/QSPR). Physicochemical properties (solubility, lipophilicity), ADMET endpoints (toxicity, blood–brain-barrier penetration), and bioactivity are predicted by training a lightweight head on ChemBERTa embeddings.
  • Library triage and novelty scoring. The pseudo-log-likelihood provides a model-based "typicality" signal for ranking or filtering molecules — for example, flagging chemically unusual SMILES or screening the output of a generative model for plausibility.
  • Chemical similarity and clustering. Embeddings support nearest-neighbor search, deduplication, and diversity analysis across compound collections.

Applied Use Cases

  • QSAR feature extraction. Embed a screening library with encode and train a classifier or regressor for a bioactivity or ADMET endpoint — the canonical ChemBERTa workflow. MolPROP (Rollins et al., 2024) demonstrates exactly this, fusing ChemBERTa-2 embeddings with a graph neural network and evaluating on seven MoleculeNet datasets (solubility, lipophilicity, toxicity, BBB penetration, and more).
  • Virtual-screening pre-filter. Use embeddings (similarity to known actives) and/or log_prob (plausibility) to shortlist candidate ligands before committing expensive structure-based methods.
  • Generative-model quality control. Score de-novo-generated SMILES with log_prob to down-weight implausible or degenerate outputs.
  • Baseline for representation benchmarks. As a widely used pretrained SMILES encoder, ChemBERTa is a natural baseline against ECFP fingerprints and graph neural networks — noting that such embeddings do not always beat fingerprints (Praski et al., 2025).

ChemBERTa is the first small-molecule model in the catalog; the closest relatives are the other masked-LM sequence encoders for different molecule types, and the structure models that consume small-molecule ligands:

  • DNABERT-2 (dnabert2) — the analogous BPE-tokenized masked-LM encoder for DNA (encode + log_prob); use it when the input is nucleotides rather than a small molecule.
  • ESM-2 (esm2) / ESM C (esmc) — masked-LM protein encoders; combine ChemBERTa (ligand) with ESM (protein) embeddings for protein–ligand interaction models.
  • Chai-1 (chai1) / BoltzGen (boltzgen) — structure models that accept small-molecule SMILES ligands; use ChemBERTa to screen/rank ligands, then these to predict 3D protein–ligand structure.

Biological Background

SMILES linearizes a molecular graph into a string by walking its atoms and bonds, encoding elements, bonds (=, #), rings (matched digits), branches (parentheses), charges, and stereochemistry. The same molecule can be written as many valid SMILES; canonicalization picks one deterministically. A chemical language model treats these strings like sentences: by learning to fill in masked tokens across 100M molecules, ChemBERTa internalizes statistical regularities of chemical "grammar" — which atoms and groups co-occur, how rings close, which substructures are common — and encodes them into a representation that transfers to downstream property-prediction tasks. This is the direct chemical analogue of masked-language-model protein encoders (ESM) and DNA encoders (DNABERT-2): the same self-supervised recipe, applied to the language of molecules.


Sources & license

License: MIT (text) — Upstream ships no LICENSE file; the license is declared only via the HuggingFace model-card metadata tag license: mit. MIT is permissive (OSI-approved) — no commercial or academic-use restrictions.

Papers

  • ChemBERTa-2: Towards Chemical Foundation Models — ELLIS Machine Learning for Molecule Discovery Workshop, 2022 · arXiv
  • ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction — ML4Molecules Workshop, NeurIPS 2020 · arXiv

Source repositories

Cite

@article{ahmad2022chemberta2,
  title={ChemBERTa-2: Towards Chemical Foundation Models},
  author={Ahmad, Walid and Simon, Elana and Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
  journal={ELLIS Machine Learning for Molecule Discovery Workshop},
  year={2022},
  eprint={2209.01712},
  archivePrefix={arXiv}
}

@article{chithrananda2020chemberta,
  title={ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction},
  author={Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
  journal={ML4Molecules Workshop, NeurIPS},
  year={2020},
  eprint={2010.09885},
  archivePrefix={arXiv}
}