Learning to compress meaning and generate words together
Seed, the AI research unit of China's ByteDance, released the weights for "Cola2," a hierarchical language model, on Hugging Face on Nov. 6. The model splits text into short segments based on content and compresses an average of about 2 tokens (units of text) into a single sequence of numbers. A separate generative model produces these sequences of numbers, which are finally turned back into words to form text. @aitrackerbot, an X account that tracks new model releases, reported the release on Nov. 6.
ByteDance's cola2 weights are now public on Hugging Face. • Combines a dynamic-compression VAE, a block-causal DiT, and a span-causal token decoder
Today's mainstream language models produce words one at a time in sequence. Cola2 separates the part that builds meaning as numbers from the part that outputs words. The "co-adaptive" in the paper's title refers to updating the compression method based on the results of word prediction, training the word-output part and the compression part in step with each other. According to Hugging Face records, the repository was created around 4 p.m. on Nov. 6 (Japan time), and the time it was made public is not shown.
A research model following May's "Cola DLM"
In May, Seed released a paper on "Cola DLM," a model of the same lineage, and also put its weights on Hugging Face. The Cola2 paper was submitted to arXiv on Oct. 22. The authors are affiliated with the University of Hong Kong, ByteDance Seed, Peking University and Renmin University of China, and the paper's cover page reads "Seedance Team Research Project." One of the authors had said on the paper's Hugging Face page, "We plan to open-source the 8B model in the coming weeks.," and the release came about two weeks later.
The paper attributed Cola DLM's weakness on text generation tasks to the part that turns the output back into words. A version without compression that swapped out the word-output part reached about 64% on LAMBADA, about 15 points above Cola DLM, it said.
8-task average falls below all 4 comparison models
On the 8-task average shown in the paper, Cola2 scored 60.10. All 4 comparison models scored higher, with the top one, "Qwen3-8B-Base," at 75.87, a gap of 15.77 points. The paper notes that the compute used for each model in this table was not matched.
The 2 wins require caution over evaluation methods
The paper highlights that Cola2 beat Llama-3-8B on HellaSwag, 41.75 to 39.21, and on StoryCloze, 80.42 to 79.49. It trailed Llama-3-8B on the remaining 6 tasks, including MMLU at 51.32 to 63.19. On these 6 tasks, Cola2 was the lowest of the 5 models compared.
Llama-3-8B's HellaSwag score of 39.21 differs widely from the figure in another paper. In a table in the LLaDA paper by Renmin University of China and others, the Llama-3-8B base model scored 79.1 on HellaSwag under a zero-shot condition. The Cola2 paper provided examples, then extracted answers from the output and scored them by exact match, so the evaluation conditions differ and the 2 figures cannot be directly compared. The StoryCloze evaluation has 468 questions, and the 0.93-point gap works out to about 4 questions by simple calculation.
| Task | Llama-3-8B | Qwen2.5-7B | Qwen3-8B-Base | LLaDA-8B-Base | Cola2 |
|---|---|---|---|---|---|
| Pre-training tokens | 15T+ | 18T | ~36T | 2.3T | 346B |
| RACE | 75.77 | 89.95 | 90.65 | 87.40 | 69.91 |
| NarrativeQA | 42.70 | 32.90 | 37.30 | 36.00 | 32.70 |
| HellaSwag | 39.21 | 75.67 | 68.13 | 69.65 | 41.75 |
| StoryCloze | 79.49 | 95.09 | 97.44 | 95.94 | 80.42 |
| LAMBADA | 75.70 | 71.82 | 71.20 | 67.90 | 63.00 |
| ARC-Easy | 90.86 | 96.00 | 97.35 | 93.85 | 87.62 |
| TruthfulQA | 63.04 | 69.03 | 72.58 | 64.87 | 54.10 |
| MMLU | 63.19 | 70.16 | 72.31 | 61.27 | 51.32 |
| 8-task average | 66.25 | 75.08 | 75.87 | 72.11 | 60.10 |
"43 times" comparison leaves out Qwen3's training
The paper compares its training with Llama-3-8B's more than 15 trillion tokens and claims that 346 billion tokens is less than one-43rd the training data. However, the entire word-output part and the embeddings of the compression part were initialized from Alibaba's "Qwen3-4B-Base." According to Qwen3's description, the series was pre-trained on 36 trillion tokens. The authors also describe the 346 billion tokens as additional training on the paper's Hugging Face page. The paper itself writes that the results show the model inherited the generative ability of the parts initialized from Qwen.
The 346 billion tokens break down into 52.43 billion for compression training, 125.83 billion for training with the roughly 4 billion-parameter generative model added, and 167.77 billion for training the generative model alone. The comparison table has no scores for Qwen3-4B-Base, so it cannot show how much Cola2's method moved the scores from the initial model. The paper describes this comparison as showing where performance stands after training, rather than isolating which generation method is superior.
In matched experiments, the compressed version improves more slowly
A separate experiment to examine differences between methods trained models of the same size on the same data up to about 210 billion tokens. "Cola-MDM," which does not compress, averaged about 34.5% across 5 tasks, above the roughly 33% of the method that produces words in sequence. "Cola2-MDM," which compresses, matched the sequential method on LAMBADA at about 61% and beat it on RACE at about 10.5% to about 4.5%, but fell behind on SQuAD and Story Cloze, which require the ability to reproduce text verbatim.
The compressed version converges more slowly, but after about 150 billion tokens its average gap with LLaDA narrowed from about 4 points to 1.5 points, with no sign of plateauing, it said. The released weights were trained under different settings from this experiment.
Speed advantage not shown in measurements
If compression shortens the sequences being handled, the amount of computation decreases in theory. The paper adds, however, that actual generation speed depends on the word-output part, additional processing and how hardware is used, and that a dedicated implementation is needed. No measured generation speed figures appear in the body of the paper or in its appendix on compute costs. The paper also says the generative model that produces the numerical representations of meaning is "far from convergence," and that the scale of training was limited by available resources.
Among methods that do not produce words in sequence, Mercury 3, released Oct. 19 by Inception, claims more than 1,000 tokens per second. ByteDance itself announced 2,146 tokens per second on H20 GPUs for "Seed Diffusion Preview," a coding model, in August 2025. Both are developers' figures, and there is no comparison with Cola2 under the same conditions.
No chat tuning, and no license stated for the weights
The weights can be downloaded without an application. What was released is a base model, without the chat tuning for following instructions. There is no information on an API or pricing. Training code is not included; the released code is for inference and evaluation. Running the model requires the "trust_remote_code=True" setting, which allows the repository's code to execute.
The code is licensed under Apache 2.0, but the weights, tokenizer and evaluation data are each subject to their own terms, according to the release. The license field on Hugging Face is empty, and the bundled documents do not describe a license for the weights themselves. Qwen assets are said to remain under their original licenses, and Qwen3-4B-Base is licensed under Apache 2.0. Companies using the model need to confirm the terms for the weights.
Hugging Face displays the size as 9B, and the recorded value is 8.76801 billion parameters (F32). GPU memory requirements are not given. Because generation starts from random noise, results may vary even for the same input. Reproduction steps have been published, but the original random seed settings have not, so reproducing the 60.10 score is not guaranteed. ORIENS's research found no third-party reproductions or comparisons. As of Nov. 6, the pages showed 0 downloads and 1 like on Hugging Face and 2 stars on GitHub. ORIENS has not tested the model.
No link to products or revenue shown
ByteDance is privately held, and no figures for its AI business revenue were found. Bloomberg reported on May 27, citing people familiar with the matter, that the company was considering capital spending of as much as $70 billion in 2026. Cola2 is a research-stage base model, and no explanation of its relationship to the company's products or revenue was found.
