DNABERT-2¶

A 117M-parameter BERT-style masked language model for DNA sequences using BPE tokenization, providing sequence embeddings and pseudo-likelihood scoring for multi-species genomic analysis.

License: Apache-2.0 · Molecules: dna · Tasks: embedding, property_prediction

Actions: encode, log_prob · Variants: 1

At a glance¶

Use it when

You need lightweight, cost-effective DNA embeddings for regulatory element classification (promoters, enhancers, splice sites) where the analysis region fits within 2,048 nucleotides (~2 kbp)
You need a permissively licensed (Apache-2.0) DNA foundation model for commercial applications without license restrictions
You want fine-grained BPE tokenization that captures motif-level patterns, rather than fixed 6-mer tokens (NT) or single-nucleotide tokens (Evo)
You are running high-throughput variant effect screening and need a fast, lightweight model that can score thousands of variants on a T4 GPU
You need DNA embeddings as input features for downstream classifiers and want a model validated on the GUE benchmark (28 datasets, 7 task categories)
You are performing multi-species genomic analyses and need embeddings with demonstrated cross-species generalization
You are comparing DNA foundation model approaches and need a BPE-tokenized baseline alongside k-mer (NT) and byte-level (Evo) models

Strengths

BPE tokenization provides finer-grained resolution than fixed k-mer approaches (NT), capturing variable-length motifs that can represent regulatory grammar at multiple scales simultaneously
Lightweight at 117M parameters on a T4 GPU, making it one of the cheapest DNA foundation models to deploy — suitable for high-throughput screening and resource-constrained environments
Strong multi-species generalization demonstrated on the GUE benchmark (28 datasets across 7 task categories), outperforming larger models on many regulatory element classification tasks
Apache-2.0 license permits unrestricted commercial and academic use, unlike NT's CC-BY-NC-SA-4.0 non-commercial license
Zero-shot variant effect prediction via pseudo-log-likelihood scoring requires no task-specific fine-tuning and benefits from BPE's ability to detect disruptions at multiple sequence scales
Dual-action API (encode + log_prob) covers both embedding extraction and sequence fitness scoring in a single model deployment
BPE vocabulary learned from genomic data may capture biologically meaningful subword units (codons, common regulatory motifs) as individual tokens, providing interpretable tokenization
Well-validated in independent benchmarks (Nature Communications 2025, Frontiers in Genetics 2025) alongside NT, HyenaDNA, Caduceus, and GROVER

Limitations

Input limit of 2,048 nucleotides (~2 kbp) is shorter than Nucleotide Transformers (~12 kbp) and much shorter than Evo/Evo2 (8-131+ kbp), limiting analysis of extended genomic regions
No generative capability — cannot design or sample novel DNA sequences; limited to scoring and embedding existing sequences
Masked language model architecture (BERT-style) provides pseudo-log-likelihood only, not true autoregressive log-probabilities for sequence fitness scoring
Single model size (117M parameters) offers no size-accuracy tradeoff — unlike NT (250M/500M) or Omni-DNA (multiple sizes up to 1B)
Training data details are less extensively documented than NT's 850+ genomes, making it harder to assess coverage of specific organism groups
Does not support RNA sequences — input validation rejects non-ACGT characters
BPE tokenization may fragment sequences differently for closely related variants, making direct position-level comparison across variant alleles less straightforward than byte-level models

Reach for something else when

You need to analyze genomic regions longer than 2,048 nucleotides (~2 kbp), such as full gene loci or long-range regulatory interactions (use Nucleotide Transformers for ~12 kbp or evo/evo2 for 8-131+ kbp contexts)
You need to generate or design novel DNA sequences for synthetic biology (use evo or evo2, which have autoregressive generation capability)
You need the highest-capacity DNA embedding model available and compute cost is not a concern (use omni_dna for larger representation capacity)
You are working with RNA sequences or need RNA-specific features (use an RNA-specific model instead)
You need codon optimization or synthesis constraint checking for gene design (use dna_chisel, which is purpose-built for these tasks)
You need protein-level analysis from coding DNA (translate and use esm2 for protein embeddings directly)
You need long-range variant effect prediction across entire gene loci or enhancer-promoter distances exceeding 10 kbp (use evo2 instead)

Alternatives

Model	Better when	Worse when
omni_dna	Larger model capacity (1B parameters), autoregressive architecture for true log-probability scoring, and designed for cross-modal multi-task DNA understanding	Very recent model (2025) with limited independent benchmarking; DNABERT-2 has extensive validation on the GUE benchmark and in multiple independent studies
evo	Much longer context (8 kbp), autoregressive generation capability, and 7B-parameter capacity for richer sequence modeling	Prokaryotic training bias, much higher compute cost (L4 GPU vs T4), and no embedding endpoint; DNABERT-2 is more practical for embedding and scoring on standard hardware
evo2	Multi-domain training (prokaryotes + eukaryotes + viruses), generation capability, embedding extraction, and up to 1M-token context with the 7B variant	Substantially higher compute requirements; DNABERT-2 provides a lightweight, fast alternative for gene-scale embedding and scoring tasks

API & schema¶

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

`encode`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/dnabert2/encode \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — DNABERT2EncodeRequest

Field	Type	Required	Constraints	Description
`items`	list[DNABERT2EncodeRequestItem]	yes	items 1–10	Batch of inputs to process in a single request. Up to 10 sequences per request.

Nested types

DNABERT2EncodeRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–2048	A DNA sequence (A/C/G/T).

Raw JSON Schema

{
  "$defs": {
    "DNABERT2EncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A DNA sequence (A/C/G/T).",
          "maxLength": 2048,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "DNABERT2EncodeRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 10 sequences per request.",
      "items": {
        "$ref": "#/$defs/DNABERT2EncodeRequestItem"
      },
      "maxItems": 10,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "DNABERT2EncodeRequest",
  "type": "object"
}

Response — DNABERT2EncodeResponse

Field	Type	Required	Constraints	Description
`results`	list[DNABERT2EncodeResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

DNABERT2EncodeResponseResult

Field	Type	Required	Constraints	Description
`embedding`	list[number]	yes		Mean-pooled embedding vector for the sequence.

Raw JSON Schema

{
  "$defs": {
    "DNABERT2EncodeResponseResult": {
      "properties": {
        "embedding": {
          "description": "Mean-pooled embedding vector for the sequence.",
          "items": {
            "type": "number"
          },
          "title": "Embedding",
          "type": "array"
        }
      },
      "required": [
        "embedding"
      ],
      "title": "DNABERT2EncodeResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/DNABERT2EncodeResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "DNABERT2EncodeResponse",
  "type": "object"
}

`log_prob`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/dnabert2/log_prob \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — DNABERT2LogProbRequest

Field	Type	Required	Constraints	Description
`items`	list[DNABERT2LogProbRequestItem]	yes	items 1–10	Batch of inputs to process in a single request. Up to 10 sequences per request.

Nested types

DNABERT2LogProbRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–2048	A DNA sequence (A/C/G/T).

Raw JSON Schema

{
  "$defs": {
    "DNABERT2LogProbRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A DNA sequence (A/C/G/T).",
          "maxLength": 2048,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "DNABERT2LogProbRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 10 sequences per request.",
      "items": {
        "$ref": "#/$defs/DNABERT2LogProbRequestItem"
      },
      "maxItems": 10,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "DNABERT2LogProbRequest",
  "type": "object"
}

Response — DNABERT2LogProbResponse

Field	Type	Required	Constraints	Description
`results`	list[DNABERT2LogProbResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

DNABERT2LogProbResponseResult

Field	Type	Required	Constraints	Description
`log_prob`	number	yes		Pseudo-log-likelihood of the sequence under the model.

Raw JSON Schema

{
  "$defs": {
    "DNABERT2LogProbResponseResult": {
      "properties": {
        "log_prob": {
          "description": "Pseudo-log-likelihood of the sequence under the model.",
          "title": "Log Prob",
          "type": "number"
        }
      },
      "required": [
        "log_prob"
      ],
      "title": "DNABERT2LogProbResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/DNABERT2LogProbResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "DNABERT2LogProbResponse",
  "type": "object"
}

Usage¶

One-line summary: A 117M-parameter BERT-style masked language model for DNA sequences using BPE tokenization, providing sequence embeddings and pseudo-likelihood scoring for multi-species genomic analysis.

Overview¶

DNABERT-2 is a foundation model for genomic DNA developed by Zhou et al. at Northwestern University. It applies the BERT masked language modeling paradigm to DNA sequences, producing dense embeddings and pseudo-log-likelihood scores useful for variant effect prediction, regulatory element classification, and general genomic representation learning.

The key innovation of DNABERT-2 over its predecessor (DNABERT) and other DNA language models is the use of Byte Pair Encoding (BPE) tokenization instead of fixed-length k-mer tokenization. BPE learns a compact vocabulary of variable-length DNA subwords from the training data, enabling the model to represent patterns at multiple scales -- from short motifs to longer conserved blocks -- in a single unified vocabulary. This design allows DNABERT-2 to achieve competitive or superior performance to much larger models (up to 21x its size) on the Genome Understanding Evaluation (GUE) benchmark while requiring only 117M parameters.

Architecture¶

Property	Value
Architecture	Transformer encoder (BERT-style)
Parameters	117M
Hidden dimensions	768
Attention heads	12
Layers	12
Tokenization	Byte Pair Encoding (BPE), ~4,096 tokens
Training data	Multi-species genomic DNA
Training objective	Masked language modeling (MLM)
Max sequence length	2,048 nucleotides

See MODEL.md for detailed architecture specifications and tokenization comparison.

Capabilities & Limitations¶

CAN be used for: - Extracting mean-pooled 768-dimensional embeddings from DNA sequences for downstream tasks - Computing pseudo-likelihood log-probabilities for DNA sequences (zero-shot variant effect assessment) - Regulatory element classification (promoters, enhancers, splice sites) via embedding-based downstream classifiers - Cross-species genomic analysis leveraging BPE's multi-species generalization

CANNOT be used for: - Sequences containing ambiguous bases (N, R, Y, W, S, etc.) -- only A, C, G, T accepted - Sequences longer than 2,048 nucleotides - RNA sequences (use RNA-specific models) - Protein sequences (use ESM-2 or similar protein language models) - DNA sequence generation (use Evo for generative tasks) - Tasks requiring very long genomic context (>8 kbp) -- consider Nucleotide Transformers or Evo

Other considerations: - The request schema enforces a 2,048-nucleotide character limit (~2 kbp); BPE tokens are variable-length so the token count is always fewer than the nucleotide count - The log_prob action requires N forward passes (one per non-special token) and is significantly slower than encode - Uses GPU memory snapshots for reduced cold start times

Usage Examples¶

from models.dnabert2.schema import (
    DNABERT2EncodeRequest,
    DNABERT2EncodeRequestItem,
)

# Encode DNA sequences to get embeddings
request = DNABERT2EncodeRequest(
    items=[
        DNABERT2EncodeRequestItem(sequence="ACGTACGTACGTACGT"),
        DNABERT2EncodeRequestItem(sequence="ATGATGATGATGATG"),
    ]
)

from models.dnabert2.schema import (
    DNABERT2LogProbRequest,
    DNABERT2LogProbRequestItem,
)

# Score DNA sequences via pseudo-likelihood
# Compare reference vs. variant to assess mutation impact
reference_request = DNABERT2LogProbRequest(
    items=[
        DNABERT2LogProbRequestItem(sequence="ACGTACGTACGTACGT"),
    ]
)

variant_request = DNABERT2LogProbRequest(
    items=[
        DNABERT2LogProbRequestItem(sequence="ACGTTCGTACGTACGT"),
    ]
)
# delta = variant_log_prob - reference_log_prob
# Negative delta suggests the variant disrupts a learned pattern

Architecture & training¶

Architecture¶

Model Type & Innovation¶

DNABERT-2 is a BERT-style masked language model for DNA sequences, developed by Zhou et al. at Northwestern University. It is the successor to the original DNABERT model and introduces a fundamental shift in how DNA sequences are tokenized for language modeling.

The key innovation of DNABERT-2 is replacing k-mer tokenization with Byte Pair Encoding (BPE). The original DNABERT (and models like Nucleotide Transformers) use fixed-length k-mer tokenization, where DNA is split into overlapping or non-overlapping subsequences of a fixed length (e.g., 6-mers). This approach has drawbacks: the vocabulary grows exponentially with k (4^k possible k-mers), information leaks between overlapping tokens, and the model cannot represent sub-k-mer patterns.

DNABERT-2's BPE tokenizer learns a data-driven vocabulary of variable-length DNA subwords. This provides:

Compact vocabulary: A BPE vocabulary of ~4,096 tokens replaces the exponentially large k-mer space.
Multi-resolution encoding: The tokenizer can represent both short motifs and longer conserved patterns as single tokens, adapting granularity to the data.
Multi-species generalization: BPE tokens generalize across species because they are learned from the data distribution rather than imposed as a fixed-length sliding window.

The underlying architecture uses the Transformer encoder from the BertForMaskedLM framework (via the triton_flash_bert configuration), yielding a compact 117M-parameter model that achieves performance comparable to or exceeding much larger k-mer-based DNA models.

Parameters & Layers¶

Component	Details
Architecture	Transformer encoder (BERT-style)
Parameters	117M
Hidden dimensions	768
Attention heads	12
Layers	12
Vocabulary	~4,096 tokens (BPE-learned)
Positional encoding	Learned absolute positional embeddings
Activation	GELU
Max input length	2,048 nucleotides (request schema enforced)

DNABERT-2 is a single-variant model with no size options.

Training Data¶

Property	Details
Dataset	Multi-species genomes
Scope	Genomes from diverse species across all domains of life
Data type	Raw genomic DNA sequences
Preprocessing	BPE tokenization learned from training corpus
Scale	Large-scale genomic corpus (exact token count not specified in paper)

Known biases: like other multi-species genomic models, training data is skewed toward well-sequenced model organisms. Performance on underrepresented taxa may be lower.

Loss Function & Objective¶

Masked language modeling (MLM): a fraction of BPE tokens are replaced with a [MASK] token, and the model is trained to predict the original token from context via cross-entropy loss.

L = - sum_i log P(x_masked_i | x_visible)

This self-supervised objective forces the model to learn contextual relationships between DNA subsequences, capturing conservation patterns, regulatory grammar, and coding-region structure.

Tokenization / Input Processing¶

Tokenizer: Byte Pair Encoding (BPE), learned from multi-species genomic DNA. This is the defining innovation of DNABERT-2 compared to predecessors.
Vocabulary: ~4,096 tokens of variable-length DNA subwords.
Special tokens: [CLS], [SEP], [PAD], [MASK], [UNK]
Input alphabet: A, C, G, T only (validated by validate_dna_unambiguous; ambiguous IUPAC codes are rejected).
Max length: 2,048 nucleotides (request schema enforced). The BPE tokenizer processes variable-length subwords, so the token count is always fewer than the character count; the practical ceiling is the 2,048-character input limit (~2 kbp).
Truncation: Sequences exceeding max token length are truncated.

Comparison of DNA tokenization strategies:

Model	Tokenization	Resolution	Vocabulary	Context (approx.)
DNABERT-2	BPE	Variable (sub-k-mer to multi-k-mer)	~4,096	~2 kbp (API limit: 2,048 nt)
Nucleotide Transformers	6-mer	6 nt fixed	~4,105	~12 kbp
Evo	Byte-level	Single nucleotide	~8	Up to 131 kbp
DNABERT (v1)	k-mer (k=3..6)	k nt fixed	4^k	~2 kbp

Performance & Benchmarks¶

Published Benchmarks¶

From Zhou et al. (arXiv 2306.15006), DNABERT-2 was evaluated on the Genome Understanding Evaluation (GUE) benchmark, which spans 28 datasets across 7 task categories.

GUE Benchmark (Overall)¶

Model	Parameters	GUE Aggregate	Notes
DNABERT-2	117M	Best overall	Outperforms models up to 21x larger
NT-v2-500M	500M	Second best	4.3x larger
HyenaDNA	1.6M-6.6M	Competitive on some tasks	Much smaller
DNABERT v1 (6-mer)	~117M	Baseline	Original k-mer approach

Key findings from the paper: - DNABERT-2 achieves top performance on most GUE tasks despite having only 117M parameters -- significantly fewer than competing models. - The BPE tokenization strategy contributes to strong multi-species generalization, with consistent performance across human, mouse, and yeast genomic tasks. - DNABERT-2 outperforms models up to 21x its size (NT 2.5B) on several benchmark tasks.

BioLM Verification Results¶

Action	Test Input	Tolerance	Status
`encode`	"ACGTACGT"	rel_tol=1e-4 (golden fixture)	PASS
`log_prob`	"ACGT", "ACGTACGT"	rel_tol=1e-4 (golden fixture)	PASS

Comparison to Alternatives¶

Model	Parameters	Tokenization	Context	When to prefer
DNABERT-2	117M	BPE	~2 kbp (API limit: 2,048 nt)	Lightweight; fine-grained tokenization; multi-species generalization
NT-v2-250M	250M	6-mer	~12 kbp	Longer context; strong multi-species genomic benchmarks
NT-v2-500M	500M	6-mer	~12 kbp	Best NT accuracy; longer context
Evo	7B	Byte-level	131 kbp	Very long genomic contexts; generative DNA tasks

Strengths & Limitations¶

Pros¶

Compact model: At 117M parameters, DNABERT-2 is substantially smaller than competing DNA foundation models while achieving competitive or superior performance.
BPE tokenization: Variable-length tokens capture multi-scale DNA patterns, from short motifs to longer conserved blocks, without the rigid k-mer vocabulary.
Multi-species generalization: BPE tokens learned from diverse genomes transfer well across species, unlike fixed k-mers that may over-fit species-specific codon usage.
Low resource requirements: Runs on a T4 GPU with 4 GB memory, making it one of the most cost-effective DNA language models available.

Cons¶

Shorter effective context than NT or Evo: the 2,048-nucleotide API limit is sufficient for promoters and regulatory elements but too short for full gene bodies or large regulatory domains.
Single variant only: No size options -- users cannot trade off between accuracy and speed across model sizes.
BPE token boundaries are not biologically meaningful: Unlike k-mers (which have a fixed biological interpretation), BPE token boundaries are data-driven and may not align with codons, motifs, or other biologically relevant boundaries.

Known Failure Modes¶

Repetitive low-complexity DNA (e.g., microsatellites, telomeric repeats): BPE may collapse repetitive regions into few tokens, yielding embeddings with limited discriminative power.
Very short sequences (< ~10 nt): insufficient context for meaningful embeddings or log-probability scores.
Non-standard DNA (RNA, modified bases, ambiguous IUPAC codes): input validation rejects non-A/C/G/T characters.
Sequences at the nucleotide limit: the 2,048-nucleotide character cap is enforced by the request schema before tokenization.

Implementation Details¶

Inference Pipeline¶

encode¶

Request
  |-- 1. Validate DNA sequences (A/C/G/T only, length <= 2048 nt)
  |-- 2. Batch tokenize with BPE tokenizer (padding + truncation)
  |-- 3. Forward pass on GPU (base_model output)
  |      |-- 12 transformer encoder layers
  |      +-- Extract last hidden states: [B, L, 768]
  |-- 4. Mean pool over non-padded tokens (attention mask weighted)
  +-- 5. Return per-sequence embedding vectors (768-dimensional)

log_prob¶

Request (processed per-sequence)
  |-- 1. Validate DNA sequence
  |-- 2. Tokenize single sequence with special tokens
  |-- 3. Identify valid (non-special) token positions
  |-- 4. Create masked batch: one copy per valid position, each with one token masked
  |-- 5. Batch forward pass on GPU (MLM head logits)
  |-- 6. Gather log P(original_token) at each masked position
  +-- 7. Sum log-probs -> pseudo-likelihood score

Memory & Compute Profile¶

Property	Value
GPU	T4 (16 GB)
GPU Memory (model)	~1-2 GB
CPU	2 cores
System Memory	4 GB
Batch size	Up to 10 items per request

Attention complexity is O(n^2) in token length. The 2,048 token limit keeps memory usage well within T4 capacity even at full batch size.

The log_prob action is significantly more expensive than encode because it requires N forward passes (one per non-special token in the sequence).

Determinism & Reproducibility¶

Setting	Value
Torch manual seed	42
CUDA manual seed	42 (all devices)
Model mode	eval() with torch.no_grad()

Seeds are set during model loading. Inference uses torch.no_grad() and the model is in eval() mode. Results should be deterministic for the same input on the same hardware. Minor floating-point variations may occur across different GPU architectures.

Caching Behavior¶

Memory snapshots: GPU memory snapshots are enabled (enable_memory_snapshot=True, enable_gpu_snapshot=True) for faster cold starts.
Response caching: Handled outside the model container by the serving layer; cache key is determined by action name, input payload, and model variant.

Versions & Changelog¶

Version	Date	Changes
v1 (weights_version)	2025	Initial BioLM deployment with encode and log_prob actions

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

DNABERT-2 is designed for genomic DNA sequences. The model operates on raw nucleotide sequences composed of the four canonical bases (A, C, G, T) and does not accept ambiguous IUPAC codes or RNA (uracil).

Trained on multi-species genomic data, DNABERT-2 has broad coverage of:

Coding regions: exons, open reading frames, codon usage patterns across species
Non-coding regulatory DNA: promoters, enhancers, silencers, UTRs
Structural genomic elements: splice sites, polyadenylation signals, CpG islands, TATA boxes
Epigenetic signal regions: sequences associated with histone modifications (though the model does not directly predict modifications)

The API accepts sequences up to 2,048 nucleotides (~2 kbp) per request. This covers most gene-proximal regulatory features and short gene bodies. The underlying model's BPE tokenizer compresses variable-length subwords, so the model's theoretical capacity per 2,048 BPE tokens could span more nucleotides — but the request schema enforces the 2,048-nucleotide character cap.

Cross-Applicability¶

Molecule Type	Applicability	Evidence	Caveats
Genomic DNA (multi-species)	High	Trained on diverse species; GUE benchmark spans human, mouse, yeast	Performance varies with evolutionary distance from training organisms
Human genomic DNA	High	Strong performance on human-specific GUE tasks
Synthetic DNA	Moderate	Can score synthetic constructs via pseudo-likelihood	Not explicitly trained on synthetic libraries; unusual motifs may be out-of-distribution
RNA sequences	Not supported	Input validation rejects non-ACGT characters	Use RNA-specific models instead
Protein sequences	Not applicable	DNA-only model	Use ESM-2 or similar protein language models

Biological Problems Addressed¶

Genomic Variant Effect Prediction¶

Why it matters: Understanding how single-nucleotide variants (SNVs) and short insertions/deletions affect gene function is central to human genetics and clinical genomics. Genome-wide association studies (GWAS) identify statistical associations between variants and traits, but determining which variants are causal and how they affect molecular function remains a major bottleneck. Experimental characterization through massively parallel reporter assays (MPRAs) or saturation mutagenesis is costly and limited in throughput.

How DNABERT-2 addresses it: The log_prob action computes a pseudo-log-likelihood score for any DNA sequence. By comparing scores between reference and variant alleles, users can estimate the functional impact of mutations:

delta_score = log_prob(variant_sequence) - log_prob(reference_sequence)

A large negative delta suggests the variant disrupts a pattern learned by the model (e.g., a conserved regulatory motif or splice signal), indicating potential functional impact. This zero-shot approach requires no labeled training data for the specific variant class.

DNABERT-2's BPE tokenization is particularly well-suited for this task because variable-length tokens can capture disruptions at multiple scales -- from single-nucleotide changes that break a short motif to larger rearrangements that disrupt extended conserved regions.

Gene Regulatory Element Classification¶

Why it matters: Identifying functional elements in the genome -- promoters, enhancers, splice sites, polyadenylation signals -- is essential for understanding gene regulation. While databases like ENCODE provide experimental annotations for some cell types and organisms, the vast majority of the genome across most species remains functionally unannotated.

How DNABERT-2 addresses it: The encode action produces dense 768-dimensional vector embeddings that capture the functional character of a DNA region. These embeddings can be used as features for downstream classifiers:

Promoter detection: Distinguishing promoter regions from non-promoter DNA
Enhancer identification: Classifying enhancer regions and predicting cell-type specificity
Splice site detection: Identifying donor and acceptor splice sites
Transcription factor binding site (TFBS) prediction: Scoring whether a sequence contains functional binding motifs

The GUE benchmark demonstrates that DNABERT-2 embeddings achieve strong performance across all of these tasks, often surpassing larger models, suggesting that the BPE tokenization captures regulatory grammar efficiently.

Sequence-Level Representation Learning¶

Why it matters: Many genomic analyses require a fixed-dimensional numerical representation of variable-length DNA sequences -- for clustering, similarity search, visualization, or as input features to multi-modal models.

How DNABERT-2 addresses it: The encode action returns mean-pooled embeddings from the final transformer layer. These 768-dimensional vectors place DNA sequences in a learned representation space where functionally similar regions are geometrically closer. Applications include:

Clustering regulatory elements by function across species
Building sequence similarity indices for large genomic datasets
Providing input features for multi-modal models combining genomic and proteomic data

Applied Use Cases¶

DNABERT-2 has been applied in the following published studies:

Benchmarking DNA foundation models (Nature Communications, 2025; DOI: 10.1038/s41467-025-65823-8): Benchmarks DNABERT-2 against NT v2, HyenaDNA, Caduceus, and GROVER across classification, variant effect prediction, and gene expression tasks.
Colorectal enhancer classification (arXiv 2509.25274, 2025): First application of DNABERT-2 with BPE tokenization to enhancer classification in colorectal cancer, achieving ROC-AUC 0.743 on 2.34 million sequences.
Gene-LLMs survey (Frontiers in Genetics, 2025; DOI: 10.3389/fgene.2025.1634882): A comprehensive survey of transformer-based genomic language models covering DNABERT-2 and other genomic LLMs, analyzing tokenization strategies and downstream regulatory genomics applications.
DeepVRegulome (arXiv 2511.09026, 2025): Combines 700 DNABERT fine-tuned models trained on ENCODE gene regulatory regions with variant scoring and motif analysis. Applied to TCGA glioblastoma WGS dataset, identifying 572 splice-disrupting and 9,837 TFBS-altering mutations.
TFBS-Finder (arXiv 2502.01311, 2025): Uses pre-trained DNABERT embeddings combined with CNN and attention modules for transcription factor binding site prediction, trained and tested on 165 ENCODE ChIP-seq datasets.

Predecessor Models¶

DNABERT (original, 2021): The first DNABERT model used k-mer tokenization (k=3 to k=6), where DNA was split into overlapping subsequences of fixed length. This required training separate models for each k value and created an exponentially growing vocabulary. DNABERT-2 replaces this with BPE tokenization, unifying all k-mer scales into a single model with a compact learned vocabulary. The original DNABERT is no longer recommended when DNABERT-2 is available.

Complementary Models¶

ESM-2 (protein embeddings): For analyses spanning both DNA and protein, DNABERT-2 embeddings can characterize regulatory DNA while ESM-2 characterizes the encoded protein. This is useful for studying how non-coding variants affect protein function through regulatory mechanisms.
Nucleotide Transformers: For tasks requiring longer genomic context (>2 kbp), NT's 12 kbp context window complements DNABERT-2's shorter but finer-grained representations.

Alternative Models¶

Alternative	Advantage over DNABERT-2	Disadvantage vs. DNABERT-2
NT-v2-250M	Longer context (~12 kbp); larger capacity (250M params)	Coarser 6-mer tokenization; larger compute footprint
NT-v2-500M	Longest NT context; highest NT accuracy	4x the parameters; 6-mer resolution
Evo	Much longer context (131 kbp); generative capability	Much larger (7B params); prokaryotic bias; no embedding endpoint
HyenaDNA	Very long-range context (up to 1M bp); sub-quadratic attention	Less validated on standard genomic benchmarks; smaller model capacity

When to choose DNABERT-2: - Lightweight deployments where compute cost matters (117M params, T4 GPU) - Fine-grained tokenization where BPE resolution matters more than long context - Multi-species analyses leveraging DNABERT-2's strong cross-species generalization on the GUE benchmark - Regulatory element tasks within the 2,048-nucleotide API limit (promoters, enhancers, splice sites)

Biological Background¶

DNA (deoxyribonucleic acid) is the molecule that encodes genetic information in nearly all living organisms. It consists of two complementary strands of nucleotides -- adenine (A), cytosine (C), guanine (G), and thymine (T) -- wound into a double helix. The sequence of these bases encodes genes (transcribed into RNA and often translated into proteins) and regulatory instructions (controlling when, where, and how much each gene is expressed).

Genomics is the study of entire genomes -- the complete DNA content of an organism. A central challenge in genomics is moving from sequence to function: given a stretch of DNA, determining what it does. Only approximately 1.5% of the human genome encodes proteins; the remaining approximately 98.5% includes regulatory elements, structural DNA, transposable elements, and regions of unknown function.

Foundation models for genomics apply self-supervised learning (typically masked language modeling) to large corpora of genomic DNA. By predicting masked nucleotide tokens from context, these models learn statistical patterns that reflect:

Conservation: bases under purifying selection are harder to predict when masked, because the model has learned they are constrained.
Regulatory grammar: motifs recognized by transcription factors, splice machinery, and other regulatory proteins emerge as learned patterns.
Codon usage: in coding regions, the model captures codon frequency biases and reading frame structure.

Tokenization is a critical design choice for DNA language models. Different strategies trade off between resolution, vocabulary size, and context length:

k-mer tokenization (DNABERT v1, Nucleotide Transformers): DNA is split into fixed-length subsequences. Simple but rigid -- vocabulary grows as 4^k and resolution is fixed at k nucleotides.
Byte Pair Encoding (BPE) (DNABERT-2): A data-driven vocabulary of variable-length subwords is learned from the training corpus. Balances resolution and context, adapting token granularity to the data.
Byte-level tokenization (Evo): Each nucleotide is a single token. Maximum resolution but requires specialized architectures (e.g., StripedHyena) to handle the resulting very long token sequences.

Key terminology:

BPE (Byte Pair Encoding): A tokenization algorithm that iteratively merges the most frequent adjacent character pairs to build a vocabulary of variable-length subwords. Originally developed for text compression, widely adopted in NLP.
Pseudo-log-likelihood: An approximation of sequence probability computed by masking each position individually and summing log-probabilities. Higher values indicate the sequence is more "natural" according to the model.
Promoter: A DNA region upstream of a gene that initiates transcription.
Enhancer: A distal regulatory element that increases gene expression, often in a cell-type-specific manner.
Splice site: The boundary between an exon (retained in mRNA) and an intron (removed during RNA processing).
GUE (Genome Understanding Evaluation): A standardized benchmark introduced alongside DNABERT-2, comprising 28 datasets across 7 genomic task categories for evaluating DNA language models.

Sources & license¶

License: Apache-2.0 (text)

Papers

DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome — arXiv preprint, 2023 · arXiv

Source repositories

github: https://github.com/MAGICS-LAB/DNABERT_2
huggingface: https://huggingface.co/zhihan1996/DNABERT-2-117M

Cite

@article{zhou2023dnabert2,
  title={DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome},
  author={Zhou, Zhihan and Ji, Yanrong and Li, Weijian and Dutta, Pratik and Davuluri, Ramana and Liu, Han},
  journal={arXiv preprint arXiv:2306.15006},
  year={2023}
}

DNABERT-2¶

At a glance¶

API & schema¶

encode¶

log_prob¶

Usage¶

Overview¶

Architecture¶

Capabilities & Limitations¶

Usage Examples¶

Architecture & training¶

Architecture¶

Model Type & Innovation¶

Parameters & Layers¶

Training Data¶

Loss Function & Objective¶

Tokenization / Input Processing¶

Performance & Benchmarks¶

Published Benchmarks¶

GUE Benchmark (Overall)¶

BioLM Verification Results¶

Comparison to Alternatives¶

Strengths & Limitations¶

Pros¶

Cons¶

Known Failure Modes¶

Implementation Details¶

Inference Pipeline¶

encode¶

log_prob¶

Memory & Compute Profile¶

Determinism & Reproducibility¶

Caching Behavior¶

Versions & Changelog¶

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

Cross-Applicability¶

Biological Problems Addressed¶

Genomic Variant Effect Prediction¶

Gene Regulatory Element Classification¶

Sequence-Level Representation Learning¶

Applied Use Cases¶

Related Models¶

Predecessor Models¶

Complementary Models¶

Alternative Models¶

Biological Background¶

Sources & license¶

`encode`¶

`log_prob`¶