Poly-InstructTTS: Learning In-the-Wild Expressive Speech Synthesis from Open-Ended Instructions

🎉 Accepted to Interspeech 2026

Abstract

While recent text-to-speech (TTS) models achieve high naturalness, controlling fine-grained expression via natural-language instructions remains challenging. We introduce Poly-InstructTTS, which learns expressive speech from open-ended instructions using in-the-wild audiovisual data. We build a scalable multi-modal pipeline to construct a 1,000-hour instruction-annotated corpus covering 1,000+ fine-grained emotions and styles. The framework uses a prompt-free GPT with attribute-based thinking tokens, followed by a flow-matching module that injects timbre from a reference audio. We also present a speaker fine-tuning procedure to transfer instruction control to specific speakers while preserving persona. We further extend InstructTTSEval with broader tasks. Experiments show that Poly-InstructTTS delivers strong performance in instruction adherence and expressiveness. Audio demos and the expanded testset are available on our project page.

Method

Model Architecture Diagram
The proposed Poly-InstructTTS system, including the data pipeline and the model architecture.

Audio Sample of Proposed System

In this section, we present some samples generated by our proposed system that highlight fine‑grained control over subtle and extreme emotions and accents from open-ended instructions, as well as support for paralinguistic labels.

Text Inputs Generated Audio
Instruction Text Mrs. Smith pleads quietly, her voice pitched high with a steady, gentle concern.
Content Text "We have to be more careful. Your health is important, and I want us to have more time together."
Instruction Text The girl's voice rises to a higher pitch, sharp with a clear tone of disgusted disbelief.
Content Text "Seriously? How can you stand the stench? It's like living in a barn."
Instruction Text The realtor's voice is bright and animated, lifted to a higher pitch by an intense, overflowing enthusiasm for the property.
Content Text "This place is a dream! It has vaulted ceilings, a brand new kitchen, and the landscaping is to die for!"
Instruction Text Jack yells his evidence with a furious, high-pitched intensity, his volume booming with conviction.
Content Text "I saw it with my own eyes! The man is a thief! He stole my pen!"
Instruction Text Richard's high anxiety is palpable as he speaks, his words coming out slowly and hesitantly, fumbling to correct himself before awkwardly transitioning to his announcement.
Content Text "So, um, the, the reason I've called you all here is, well, it's about the, uh, funding. We, uh... we have an announcement."
Instruction Text The woman's voice is filled with an intense, playful happiness, with laughter bubbling up through her words as she flirts.
Content Text "Oh, I like a man who's not afraid of a little competition. This is going to be fun."
Instruction Text Continuing his furious tirade, the man shouts a series of desperate, self-justifying questions, his voice strained with the high intensity of his rage.
Content Text "Do you have any idea what they would have done to me? What they threatened to do to my little girl? I had to do it!"
Instruction Text The girl's apology is a loud, high-pitched rush of words, broken by intense sobs of profound sadness and guilt.
Content Text "I messed everything up. I'm so sorry, I never meant for any of this to happen."
Instruction Text A woman speaks with a calm, neutral tone, her Indian-accented English conveying a polite but firm business inquiry.
Content Text "Hello, I'm calling about the invoice from last month. It seems it is still outstanding."
Instruction Text With a clear note of feigned outrage, the gentleman's voice rises in pitch and volume, his German-accented words sharp and defensive.
Content Text "This is an outrage! I am a respected businessman! I will not be treated like a common criminal!"
Instruction Text A playful, teasing lilt dances in her voice as she playfully scolds someone, her tone more amused than annoyed.
Content Text "You are so much trouble. <giggles> What am I going to do with you?"
Instruction Text He is about to give an important speech, his voice slightly shaky as he clears his throat to begin.
Content Text "Good evening, everyone. <clear throat> On behalf of the entire team, I'd like to thank you for being here tonight."
Instruction Text An intimate, close‑mic ASMR delivery with a slow, breathy whisper. The voice sits very near the listener’s ear, vowels are gently elongated, and soft, audible breaths weave between phrases, creating a confidential, almost secret‑sharing tone. The mood can tilt playful and teasing, warm and reassuring.
Content Text "<whisper> It's me again, sure. </whisper> I'm here to bring you content. Yeah anyways, um.. Hi, welcome."

Baseline System Comparison

In this section, we present side‑by‑side audio samples comparing Poly‑InstructTTS with four representative, high‑performing systems. Since some systems only support system‑provided voices and do not accept external timbre prompts, this demo only focuses on instruction adherence and expressiveness instead of timbre similarity.

Gemini Flash TTS GPT-4o mini TTS Qwen3 TTS-Voice Design OV-InstructTTS Poly-InstructTTS (Ours)
Instruction Incorporate a rapid and fluent speech style highlighted by an extroverted, confident tone, maintaining high clarity throughout while brightening your vocal resonance during the energetic segments.
Content "And I'm happy to say we've got an extra judge on the panel tonight. Quite simply, without her, there would be no disco. Ladies and gentlemen, please welcome Donna Summer."
Instruction Inch your delivery towards a frantic pace where phrases occasionally repeat, creating a strained yet insistent flow where clarity becomes a casualty of the rushing torrent of fear and desperation.
Content "Listen to me. He killed her, and I'm not crazy. Jodi and I had a seance in the bathroom. Jodi and I, nothing happened, but when I went back there, she was there. She was."
Instruction Start at a relatively rapid speed and continuously accelerate your pace, transforming your delivery into an almost seamless flow, brimming with an aggressive and strained pitch that skyrockets during insults.
Content "I'm going to return to a rape-free workplace, alright? Because if you so much as look at my sexy little ass, Julia, I will have yours locked the fuck up, you crazy bitch whore!"
Instruction Conclude your speech with a cheerful and melodic farewell, blending a rush of playful energy with a discernible Hispanic influence, especially in the phrase 'See you mañana', to convey a warm, friendly demeanor.
Content "Make sure you lock homes. You know how to lock up, right? Make sure you put the rags in the washing machine. Don't clean the washing machine. Just put the rags in the washing machine. Understand? See you manana."
Instruction Dynamically articulate with a bright and smooth vocal texture combined with a brisk pace, enhancing your British accent for a confident and engaging presentation.
Content "During his reign, William the Conqueror abolished the death penalty in England. But what did he replace it with? Was it A, having your feet cut off, B, being tickled with a feather, or C, having your eyes gouged out? The answer is C, having your eyes gouged out. And if you think that's gory, then you should hear what happened at William's funeral."
Instruction Commence with a strained, high-pitched voice infused with a sense of panic, then allow the pitch to slightly decrease while keeping the strained emotion intact. Maintain clear articulation despite your flustered character under pressure.
Content "No, don't! Don't, computer! I'm just talking to the computer, because sometimes I feel like it helps. I need you to reboot on the count of three. One, two, three."
Instruction Craft your speech as a practiced performance, showcasing extroverted energy and exaggerated enthusiasm. Let the British accent shine through with a high volume approach, while deftly navigating the hurried pace with fluency.
Content "up and says, whoa, there's too many of you and your job sounds silly. Why not just have one person in charge of everything? Someone like, oh, I don't know, me. Yes, Julius Caesar becomes dictator. He keeps the Senate. Basically, he's in charge. A bit like a headmaster, but with a lot less shouting. Then Caesar gets murdered. Yeah. And a fellow called Augustus takes over and he decides the Senate is still too powerful. So he makes himself emperor and says the Senate can only give him advice. And being emperor is a great job. There's bang."

Ablation Study: Instruction-Conditioned SFT

Text Inputs Base Poly-InstructTTS
+ FM timbre injection		 Proposed SFT paradigm
Speaker Persona:
Instruction Delivering her words with cutting finality, the wife's voice seethes with intense fury and profound hurt, commanding Leon to leave with an unyielding resolve.
Content "You don't get to say that anymore. Get dressed and leave. NOW."
Instruction A visceral disgust fills her voice, which is strained and guttural, as if she's trying not to gag while describing something vile.
Content "The smell... Ugh, I can't even describe it. It was like rotting meat and chemicals. I think I'm going to be sick."
Speaker Persona:
Instruction His voice is laced with a cold, condescending arrogance, speaking down to someone he deems utterly pathetic. The tone is bored and dismissive.
Content "Is that supposed to impress me? Truly? Please, don't waste my time with your pathetic little power plays. It's tiresome."
Instruction As a leader in a crisis, his voice is a sharp, authoritative bark, devoid of any warmth. It's a tool for absolute command and control.
Content "Medic! I need a medic over here, now! Keep pressure on the wound! Do not let him bleed out!"

Supplementary Materials