DeepX Embedding v1.0: Setting a New Sub-1B SOTA in Vietnamese Legal Retrieval

We are releasing the official technical report for DeepX Embedding v1.0, a state-of-the-art Vietnamese embedding model engineered specifically for legal document retrieval and high-precision semantic search.

Developed by the DX Tech team, DeepX v1.0 combines a novel linear attention mechanism (Gated DeltaNet-2) with Hyperloop weight sharing, achieving superior retrieval accuracy while maintaining O(n) time and memory efficiency for long sequences.

Executive Summary

On the industry-standard Zalo Legal Text Retrieval benchmark, DeepX Embedding v1.0 achieves an nDCG@10 of 0.8162 (and Recall@10 of 0.9638). This establishes a new State-of-the-Art (SOTA) among sub-1B parameter models (< 1B params), outperforming the previous benchmark record (0.7813) by +3.49 points.

Key Technical Innovations & DX Contributions:

• • O(n) Linear Attention (GDN-2): Replaces standard softmax attention with gated delta-rule recurrence. This maintains constant VRAM usage per token regardless of sequence length, enabling fast inference across long documents (trained up to 8,192 tokens).

• • Per-Loop LoRA (DX Contribution): Solves the depth collapse inherent in recursive weight sharing. Each loop iteration receives its own low-rank adapter (rank 16) applied across entire 5-layer blocks simultaneously (Q,K,V,O, gate, up, down), giving every compute pass a unique representation identity.

• • Rotary Depth Embedding / RoDE (DX Contribution): Introduces a novel positional signal that encodes loop iteration depth directly into attention. By applying a second rotation on Q and K based on iteration index (using 8 rotary pairs), RoDE informs the attention mechanism “how deep” in the network it currently is.

• • Two-Stage Retrieval & Matryoshka Embeddings: Produces dual outputs—a 1536-d single vector for fast ANN search (with 256-d Matryoshka truncation preserving ~96% quality) and ColBERT token vectors (T×128-d) for precise late-interaction reranking.

Read the full technical report embedded below for complete architectural breakdowns, dataset mixtures, and benchmarking methodology.