You can stream text (but not Markdown) without using JS

As a large language model trained by OpenAI, I can't interact with users who have JS turned off because I need to stream my output as it's generated.

-- Some LLM, probably.

Watching text slowly materialize on screen is a tradition almost as old as the web itself. In the bad ol' days you could watch a page appear line by line as a PHP app running on an underpowered web server pulled data row by row from a table sitting on spinning rust. Today you're more likely to be waiting for GPU-backed LLMs than HDD-backed SQL, but the feeling is the same.

Browsers have had built-in support for streaming text since, roughly speaking, forever. If markup is slow to download, they'll render however much they've already received while waiting for the rest. The technical (googlable) term is "progressive rendering". You could reinvent it using WebSockets or server-sent events, but you could also not do that and use the implementation that's already there.

(If this JavaScript-free page was already loaded when you started reading it, please refresh it now.)

Advantages:Works for users who have JS turned off
Disadvantages:Your app's footer doesn't show up until the body text is done baking
Web framework support:Spotty
Rich text support:Yes, we're inside a table right now

At this point I thought I'd found an easy and elegant way to add NoScript support to an LLM frontend. What could go wrong?

Markdown support could go wrong. The LLM writes Markdown, the browser consumes HTML, and streaming Markdown to HTML conversion is impossible. Consider that adding characters to the end of a Markdown paragraph can change earlier characters' meanings:

MarkdownHTML
*This paragraph starts and*This paragraph starts and
*This paragraph starts and ends with an asterisk*<em>This paragraph starts and ends with an asterisk</em>

JS-powered streaming solutions can go back and rewrite previously generated markup, so it's not a problem for them if a * becomes an <em> some time after it was written, but when you're using progressively rendered HTML, once you've sent something, you can't change it. Before we can send any of the text in that paragraph to the client we have to know whether it starts with * or <em>, and we won't find that out until the whole paragraph has been generated.

So much for elegant JS-free LLM output streaming :S

(Creation of a terrible hacky solution powered by <meta http-equiv="Refresh" /> is left as an exercise for the reader.)


Published 2024-04-19

Previous: A quick way to find big elements buried in the DOM