What Is It Like to Write Like Haiku?

Jinran Jin, Ryan Teehan, Jack Lu, and Mengye Ren

New York University

Preprint

TL;DR: PrompTo distills LLMs' implicit writing styles into system prompts that other models can follow.

Abstract

Language models have distinct and recognizable writing styles, yet these writing styles are rarely articulated precisely. In this paper, we study whether a model's style can be described in words for another model to adapt and follow. To measure the fitness of a description, we ask independent evaluators to distinguish the pair of responses. Rather than coming up with an ad hoc prompt, our proposed framework, PrompTo, creates a search loop to optimize for the style prompt that confuses the evaluator. A reflector then takes in the reward and language feedback and generates a prompt candidate for the next iteration. The optimized prompts recover 74.8% of the source–target gap on average, against at most 15.2% for existing baselines. Moreover, we found style prompts optimized with one source model readily transfer to other source models.

Overview

We say that a description of a model's style is complete if a model that has never seen the original can reproduce the style from the description alone. No prior work both describes a model's style and reproduces it. We close this gap with PrompTo, a framework that searches for a complete description of a target model's style and tests it by reproduction. The description is in the format of a system prompt, which we call a style prompt.

Method

The PrompTo framework for stylistic transfer. Top: the source model answers X under a trainable style prompt π to generate the restyled text Y′ₛ. The target model answers X and held-out Xₐ to generate Yₜ and Yₐ. Discriminator D picks which of Y′ₛ, Yₜ is closer in style to Yₐ. The reflector R proposes updated π based on D's reasoning. Bottom: Qwen2.5-7B → Gemma3-4B on ELI5. Bold text shows the style of the target model.

Each iteration does three things: generating restyled responses under $\pi$; evaluating $\pi$ with an LLM discriminator; proposing updated $\pi$ with a reflection model using the discriminator's reasoning and failure cases. For each question, the discriminator $D$ receives the restyled response $Y'_s$, the target response $Y_t$ to the same question, and $n$ anchor responses $Y_a$ from the target model on different questions, and selects the response closer in style to the anchors. The fool rate is the probability that $D$ predicts $Y'_s$ as being more similar to $Y_a$.

Results

Across 13 directions on 2 datasets, the optimized style prompts recover 74.8% of the source–target gap on the LLM discriminators, against at most 15.2% for any baseline. The authorship-verification classifier and STAR style embeddings agree. A model name alone carries no style property. Examples or a simple description without revision carry a little style information. Only PrompTo's style prompt, which is optimized with a discriminator, carries most of it.

Method	Fool rate (%)	Fool rate SGR (%)	AV ratio (%)	Cos. sim.
Vanilla	9.2	–	0.9	0.767
Generic	3.5	−21.3	6.1	0.692
Few-shot	14.5	15.2	10.3	0.772
Single-pass	16.6	13.8	19.5	0.769
PrompTo (ours)	43.8	74.8	51.0	0.801

Main results of PrompTo, averaged over the 13 directions across three kinds of evaluators. Fool rate and its Style Gap Recovered (SGR) are averaged across three LLM discriminators; AV ratio is the AV classifier's same-model rate; Cos. sim. is the STAR embedding cosine similarity between the text and centroids. Higher is better throughout.

Model Signatures

PrompTo's style prompts discover Gemma as an enthusiastic tutor, because it uses warm openings and asks follow-up questions; Qwen as a neutral assistant, because it goes directly to the question and does nothing else; and Haiku as a knowledgeable friend, because it gets straight to the answer with warm voice but no AI-like phrases. PrompTo finds the most distinctive features of each model, and we measure their frequency with regex.

Stylistic features of target models. Frequencies counted with regex.

Style Prompt Generalization

Style prompts optimized for one source model transfer to other source models with 78% of the SGR retained on average. This indicates that style prompts mainly describe the target's style rather than suppress the traits of the source model they were optimized with.

Style prompts transfer across source models. Each row is a source model executing a style prompt and each column is the source model the prompt was optimized for. Boxed diagonal cells are each model running its own prompt.

Feature Frequency Mismatch

A model's behavior can be seen as a distribution of phrases, but the restyled text collapses it into just one behavior. The style prompt provides a list of openers, but the source model picks one of them almost every time. Because the discriminator only sees one triplet at a time, it cannot see the frequency mismatch.

Feature frequency for target model vs. source model with π*. The original target model writes in a more diverse way, but the source model with π* fails to replicate that diversity.

BibTeX

@misc{jin2026prompto,
  title  = {What Is It Like to Write Like Haiku?},
  author = {Jin, Jinran and Teehan, Ryan and Lu, Jack and Ren, Mengye},
  note   = {Preprint},
  year   = {2026}
}