Expert Contributions
AI-TechScienceGovernmentBusinessSportsEntertainment
AI-Tech

Synthetic voices learn to read aloud without ballooning hardware demands

A tiered approach to audio tokens allows expressive long-form speech synthesis to run on a single consumer graphics card, according to new benchmark findings.

By Primary News Team
18 statements added by Jonathan Cremer · 28 Oct see what was added
Synthetic voices learn to read aloud without ballooning hardware demands
Source: Lee from Brooklyn, NY, USA (CC BY-SA 2.0)
Published28 Oct 2026, 13:26 Last updated28 Oct 2026, 13:26 Sources
Source mode Marks each sentence drawn from a source or a contributor

When a skilled narrator reads a passage of prose aloud, the voice changes constantly in pitch, cadence, and emotional intensity. Spoken words do not arrive in isolation, because the reader instinctively senses the flow of a paragraph, preparing for incoming punctuation, slowing down before a key transition, and pitching upward to mark an unresolved thought. Human speech blends meaning and acoustic texture so tightly that listeners immediately identify when an automated voice sounds flat, stiff, or disconnected from the sense of the sentence. Recreating that natural flow in software requires automated systems to parse written words while calculating how every syllable should resonate.

For speech synthesis, sustaining this melodic rhythm across extended passages presents an enduring computational challenge. Generating digital audio that mirrors the expressive rise and fall of human storytelling typically requires deep transformer networks processing dense audio frames. Commercial services capable of convincing narration usually operate across massive cloud data centers with clusters of enterprise graphics chips. When software engineers attempt to shrink those systems down to everyday local hardware, the synthetic speech often degrades into a mechanical drone, misplaces accents across compound words, or loses rhythmic coherence after a few sentences.

To understand why this happens, consider how human vocalization actually works. Spoken communication begins with linguistic intent, where the brain selects vocabulary and establishes an overarching rhythmic structure, technically termed prosody, which governs pitch variation, stress, and silent intervals. Once that structural guide is in place, the vocal tract, tongue, and lips produce the acoustic textures, harmonics, and breathiness that give a voice its physical presence. If a speech generator models sentence-level timing and fine acoustic harmonics simultaneously in one dense mathematical pass, computing costs quickly multiply. By dividing the global rhythm of a sentence from its acoustic polish, an automated system can direct processing power where human ears notice it most.

In a preprint paper posted on the arXiv repository in September 2026, researchers Fritz Cremer and Jonathan Cremer introduced TontaubeV1, a speech generation model designed to run expressive voice synthesis on a single consumer graphics processing unit.12 The authors structured the system to generate realistic long-form speech while preserving low processing latency.13 By adopting a hierarchical audio representation that splits meaning from sound texture, the model matches the prosodic expressiveness of major commercial cloud services while keeping its entire parameter footprint small enough to operate locally.14

How can an automated model separate rhythm from acoustic detail?

Speech models achieve efficiency by separating the broad cadence of a sentence from its detailed audio surface. To represent sound as digital code, TontaubeV1 relies on DualCodec, an audio tokenizer that compresses twenty-four-kilohertz sound into eight discrete numerical streams operating at twelve and a half frames per second.41 The first stream in this sequence acts as a semantic layer, which the developers extracted from self-supervised speech representations that capture linguistic structure, phrasing, and timing.14 The remaining seven streams capture acoustic residuals, which are the subtle textural details that remain after the basic linguistic rhythm is established.14

Instead of retaining all eight layers, Fritz Cremer and Jonathan Cremer kept the semantic stream and only the first three acoustic refinement streams.14 Decoded on its own, the semantic stream already produces understandable speech with recognizable phrasing and intonation, even though it lacks polished tonal fidelity.14 Discarding the four finest acoustic layers reduced the data rate from 1,225 bits per second for the full stack down to 625 bits per second for the retained streams.41 The authors reported that their informal listening tests indicated diminishing audio quality returns from calculating the finest acoustic layers.14

Jonathan said in response to questions from Primary that pruning these layers involved a deliberate technical balance. "We prioritized expressive prosody and low-latency local streaming over maximum acoustic fidelity," Jonathan told Primary. Jonathan explained that trimming the upper streams "creates a slight high-frequency 'dullness,' but we found diminishing quality returns from computing them." Jonathan noted that the developers would only restore those higher streams if the omission caused severe quantization noise, metallic artifacts, or muffled consonants that hurt intelligibility, adding that the team could resolve minor dullness in future updates by improving their downstream latent processing.

Under this tiered design, the first model stage determines the entire timeline of the spoken output. A larger transformer based on the Qwen3-1.7B architecture, containing roughly 1.83 billion stored parameters across 28 blocks, reads the written text and generates the initial semantic stream.6 Because this first predictor dictates the total frame count, it locks in the overall utterance duration, cadence, and pitch contour.1 Three progressively smaller transformers based on the Qwen3-0.6B architecture, containing 16 blocks, eight blocks, and four blocks respectively, then sequentially generate the three acoustic layers to enrich the sound.61

Jonathan told Primary that the initial semantic predictor can set expressive cadence before sound details exist because of how the underlying codec was built. Rather than merely compressing raw sound waveforms, the primary stream directly quantizes semantic features from w2v-BERT-2.0, an automated speech recognition model trained on thousands of hours of spoken audio. Jonathan said that because this layer contains rich pronunciation and linguistic information, "before any acoustic texture is generated, this first token stream acts as a highly detailed linguistic blueprint that dictates the rhythm, pitch, and intonation."

Synthetic voices learn to read aloud without ballooning hardware demands
Source: Original article

Once the primary semantic model establishes a speech trajectory, the subsequent acoustic refinement stages cannot alter its timeline or change the words. By default, these secondary acoustic transformers run at a temperature setting of zero, producing fully deterministic audio detail conditioned on the fixed semantic foundation.16 Jonathan told Primary that introducing sampling randomness into these acoustic refinement layers produced severe tonal instability during testing. Jonathan said that while the semantic stream serves as a blueprint, the acoustic stages resemble physical vocal cords, and adding temperature noise caused the output voice to tremble with an unpredictable flutter.

In total, the four predictive stages encompass roughly 2.87 billion stored parameters, representing 2.19 billion parameters when unused input embedding rows are excluded.6 This coarse-to-fine division concentrates the bulk of the computational capacity on the linguistic foundation that dictates natural expression. The acoustic stages operate independently during inference, maintaining no state across chunk boundaries and running without recursive back-and-forth calculations.

Why does character-by-character spelling help a narrator sound authentic?

Spelling words out character by character allows a speech generator to learn reliable phonetic rules across an entire language. Standard large language models routinely group common clusters of letters into subword tokens, which speeds up text processing but obscures how individual consonants and vowels interact. To prevent pronunciation glitches, TontaubeV1 tokenizes ordinary spoken text strictly one letter at a time, bypassing subword chunking so that each character occupies exactly one discrete token in the model input.

By evaluating text at the character level, the model learns pronunciation mechanics over a compact alphabet of a few hundred character identifiers rather than navigating tens of thousands of fragmented subwords.41 This setup allows the system to map letter combinations to vocal sounds systematically, ensuring that rare words and unusual names follow coherent acoustic rules. The approach also establishes a predictable spatial timeline, because text positions and context windows remain tied to character lengths without being distorted by arbitrary token boundaries.14

Because the network reads literal characters, it cannot automatically determine how to voice numbers, symbols, or abbreviations. A written number such as 1984 could be spoken as a historical year or as a whole mathematical quantity, a contextual distinction the raw character sequence cannot reliably convey.1 To address this ambiguity, the developers require input text to arrive in pre-verbalized spoken form, releasing a companion text verbalizer for English that expands digits, dates, and currency symbols into plain words before speech generation begins.6

The system also accommodates prompt audio to guide voice cloning. Callers can supply up to 60 seconds of reference audio, which DualCodec encodes into matching prompt streams that condition the four generative stages without needing a written transcript.16 To handle continuous generation across extended book chapters, the architecture uses paired boundary markers that align text and audio on a shared logical timeline.14 This bounded context structure allows earlier sections of synthesized audio to slide out of the memory window, preventing the computational demands from accumulating as an audiobook progresses.16

How fast can speech stream on everyday computer hardware?

Speech models achieve low interactive delay by emitting playable sound before an entire paragraph finishes computing. Streaming audio from discrete codec predictions normally presents a mechanical obstacle, because DualCodec utilizes a non-causal audio decoder that requires future audio frames to reconstruct sound near a chunk boundary.14 If a system must wait for downstream frames to finalize earlier speech, real-time playback stalls.

To overcome this limitation, Fritz Cremer and Jonathan Cremer mapped overlapping DualCodec sound reconstructions into the continuous acoustic latent space of VibeVoice, a different speech system.16 By processing these overlapping segments through the causal decoder of VibeVoice, the software emits stable, uninterrupted audio chunks while subsequent text remains in generation.14 On a single NVIDIA GeForce RTX 5090 graphics card, the streaming pipeline produces its first audible sound within approximately 200 milliseconds of receiving warmed input.16

Jonathan said in response to questions from Primary that routing audio through a second codec introduced noticeable operational penalties. Jonathan explained that the system cannot immediately turn early tokens into playable sound, requiring a brief buffer of initial frames before synthesis begins. Jonathan also pointed out that maintaining continuous causal cache stability requires generating overlapping segments, passing them through the secondary decoder, and discarding redundant context margins, creating measurable computational duplication.

Synthetic voices learn to read aloud without ballooning hardware demands
Source: Original article

In separate non-streaming tests measuring pure throughput, the software achieved an end-to-end real-time factor of 0.08 for a single spoken input on an RTX 5090 card, meaning it synthesized speech over twelve times faster than natural reading speed.13 When processing eight concurrent requests in batched execution, the aggregate real-time factor dropped to 0.02, delivering throughput roughly 50 times faster than real time.13 The underlying transformers were trained using supervised fine-tuning across roughly 200,000 hours of paired audio and transcripts drawn primarily from public-domain audiobooks and open speech datasets across seven languages.63

To evaluate expressive reading quality, the authors assembled an automated benchmark comprising 400 English prose passages ranging from 250 to 500 characters.61 Using Gemini 3.1 Pro Preview as an automated judge evaluating audio pairs in balanced presentation orders, TontaubeV1 achieved a 50.1 percent prosody preference against ElevenLabs Flash v2.5, indicating parity.61 Against other commercial competitors on prosodic quality, the model scored an 82.1 percent preference over Fish Audio S2 Pro, an 86.2 percent preference over the April 2026 Gradium API, and an 82.3 percent preference over Cartesia Sonic 3, alongside competitive word-level accuracy.61

What the benchmark results cannot confirm

The reported benchmark scores reflect automated language-model evaluations rather than controlled listening tests conducted with human subjects.61 Automated judges can exhibit subtle biases when assessing audio fidelity, and the evaluation rubric specifically focused on reading prosody and text correctness rather than overall acoustic clarity or background noise. Furthermore, the 400 test passages measured English prose alone, offering no empirical evidence regarding the system's performance in German or its other secondary languages.16

Jonathan told Primary that while the developers found the automated judge persuasive during training, independent human validation remains untested. "We didn't conduct a formal comparison with an independent human listening panel, so whether human raters perfectly mirror Gemini's prosody scores remains an open question," Jonathan said. Jonathan noted that blind voting on crowdsourced evaluation platforms will provide clearer verification of how the automated scoring correlates with human perception.

The testing environment also contained an inherent structural asymmetry that may have favored TontaubeV1. In the comparative benchmark trials, TontaubeV1 and Fish Audio both used a frozen cloning reference clip, whereas ElevenLabs, Gradium, and Cartesia relied on fixed provider voices.6 Although the evaluation prompt instructed the automated judge to ignore vocal timbre and speaker identity, natural prosody remains difficult to untangle from reference audio, which could introduce bias into the relative rankings.6

Jonathan acknowledged this disparity, explaining that the evaluation protocol attempted to mitigate bias by normalizing volume to -20 dBFS and instructing the judge to overlook timbre and audio blemishes. Jonathan conceded, however, that because cloned models absorb the lively cadence of human reference speakers while competitors use fixed studio voices, "our system's ability to borrow an expressive cadence may have naturally favored it during the evaluation."

Finally, the authors trained their model predominantly on narrated audiobooks, meaning its expressive capabilities do not automatically transfer to spontaneous dialogue.61 Conversational speech involves rapid turn-taking, colloquial interruptions, and varied speaking tempos that differ fundamentally from formal reading. Jonathan told Primary that while the current release targets single-narrator delivery rather than multi-speaker collisions, the architecture provides a functional foundation for conversational pipelines where basic turn-taking operates out of the box.

The authors distributed the model weights on Hugging Face under the Tontaube Community Model License 1.0, which allows research and qualifying commercial use under specified revenue thresholds, alongside code for local serving.16 The release provides checkpoint weights alongside local serving scripts, enabling independent research into bounded-context speech synthesis.

This piece was prepared from the arXiv preprint and public records together with answers from Jonathan to seven questions from the Primary news team, completed September 2026.

The record

54 sentences trace to 5 sources and 1 contributor.

  1. 1 FC Fritz Cremer, Jonathan Cremer Paper · arxiv.org · 9 Sep 2026 TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context Preprint · may not have been peer reviewed See the source
  2. 2 A arxiv.org Paper · arxiv.org Computer Science Preprint · may not have been peer reviewed See the source
  3. 3 L linkedin.com Third-party reporting · linkedin.com · 6 Aug 2026 SDK documentation: Rebuilt | Treble Technologies See the source
  4. 4 A arxiv.org Paper · arxiv.org TontaubeV1: Streaming Text-to-Speechwith Hierarchical Codec Modelingand Bounded Context Preprint · may not have been peer reviewed See the source
  5. 5 JC Jonathan Cremer Contributor · interview 10 Sep 2026 · in the article since 28 Oct 2026 Contribution — Jonathan Cremer 18 statements added to this article
  6. 6 H huggingface.co Third-party reporting · huggingface.co TontaubeAI/TontaubeV1 · Hugging Face See the source

Article history

  1. Published 28 Oct 2026, 13:26
    Assembled by the Primary desk from 5 sources · 1 contributor · 54 cited sentences
  2. Jonathan Cremer 18 statements added 49 days before publication
    1. Go to the sentence: Jonathan said in response to questions from Primary that pruning these layers involved a deliberate technical balance.
    2. Go to the sentence: "We prioritized expressive prosody and low-latency local streaming over maximum acoustic fidelity," Jonathan told Primary.
    3. Go to the sentence: Jonathan explained that trimming the upper streams "creates a slight high-frequency 'dullness,' but we found diminishing quality returns from computing them."
    4. Go to the sentence: Jonathan noted that the developers would only restore those higher streams if the omission caused severe quantization noise, metallic artifacts, or muffled consonants that hurt intelligibility, adding that the team could resolve minor dullness in future updates by improving their downstream latent processing.
    5. Go to the sentence: Jonathan told Primary that the initial semantic predictor can set expressive cadence before sound details exist because of how the underlying codec was built.
    6. Go to the sentence: Jonathan said that because this layer contains rich pronunciation and linguistic information, "before any acoustic texture is generated, this first token stream acts as a highly detailed linguistic blueprint that dictates the rhythm, pitch, and intonation."
    7. Go to the sentence: Jonathan told Primary that introducing sampling randomness into these acoustic refinement layers produced severe tonal instability during testing.
    8. Go to the sentence: Jonathan said in response to questions from Primary that routing audio through a second codec introduced noticeable operational penalties.
    9. Go to the sentence: Jonathan explained that the system cannot immediately turn early tokens into playable sound, requiring a brief buffer of initial frames before synthesis begins.
    10. Go to the sentence: Jonathan also pointed out that maintaining continuous causal cache stability requires generating overlapping segments, passing them through the secondary decoder, and discarding redundant context margins, creating measurable computational duplication.
    11. Go to the sentence: Jonathan told Primary that while the developers found the automated judge persuasive during training, independent human validation remains untested.
    12. Go to the sentence: "We didn't conduct a formal comparison with an independent human listening panel, so whether human raters perfectly mirror Gemini's prosody scores remains an open question," Jonathan said.
    13. Go to the sentence: Jonathan noted that blind voting on crowdsourced evaluation platforms will provide clearer verification of how the automated scoring correlates with human perception.
    14. Go to the sentence: In the comparative benchmark trials, TontaubeV1 and Fish Audio both used a frozen cloning reference clip, whereas ElevenLabs, Gradium, and Cartesia relied on fixed provider voices.
    15. Go to the sentence: Although the evaluation prompt instructed the automated judge to ignore vocal timbre and speaker identity, natural prosody remains difficult to untangle from reference audio, which could introduce bias into the relative rankings.
    16. Go to the sentence: Jonathan acknowledged this disparity, explaining that the evaluation protocol attempted to mitigate bias by normalizing volume to -20 dBFS and instructing the judge to overlook timbre and audio blemishes.
    17. Go to the sentence: Jonathan conceded, however, that because cloned models absorb the lively cadence of human reference speakers while competitors use fixed studio voices, "our system's ability to borrow an expressive cadence may have naturally favored it during the evaluation."
    18. Go to the sentence: Jonathan told Primary that while the current release targets single-narrator delivery rather than multi-speaker collisions, the architecture provides a functional foundation for conversational pipelines where basic turn-taking operates out of the box.