A whole decade ago I blogged about a taxonomy of cliches in space opera, with a considerable checklists of things to watch out for in the space opera I was just then starting to work on.

This was almost entirely a background task for most of that time. During the period I was poking at the new space opera project I wrote: the Empire Games trilogy, the New Management trilogy, A Conventional Boy, The Labyrinth Index and The Regicide Report. So, nine other novels. Meanwhile the space opera went through two and a half rewrites, spawned a spin-off novel, and the spin-off novel is now getting its second rewrite, because Reasons (it has to be perfect: also, I'm going to go back to the original space opera next and finish that, too, probably next year, not to mention already having the first 50,000 words of a sequel.).

But anyway: the space opera project has resulted in maybe a third of a million words of fiction so far, and absolutely humongous quantities of world-building work. Which is why I'm now revisiting my earlier thoughts on space opera, to try to better understand what I'm doing here.

First, some clarification. What even is space opera, when it's at home?

While there's no short, concise definition of space opera, there's an extensive monograph on Space Opera in the SF Encyclopedia, tracing its roots back to the US pulp magazine sector around 1900 and following it to 1941, when Wilson Tucker defined it as the appropriate term for the "hacky, grinding, stinking, outworn, spaceship yarn": alternatively, space-adventure stories which have a calculatedly romantic element.

Let's stick a magnifying glass on top of that last phrase. Romaniticism in this context does not have anything to do with the romance genre of fiction (which focusses on love and relationships). Rather it's a shout-out to the romantic movement of the 18th to 19th centuries and its modern descendants, which prioritize passion, intuition, and individualism over social convention—the pursuit of beauty entails an interest in the exotic and mysterious, a celebration of the heroic and the divine, respect for the supernatural, and arguably the will to power and worship of force that took a darker turn in the fascist and totalitarian movements of the 20th century. Stick starships and light sabres and vast vistas of time and space on top of this and you can get an impression of where it's going.

As a romantic genre, space opera requires its protagonists to have enough agency (personal autonomy) to break out of their quotidian setting and have adventures—travel is a big part of this, for it cuts them free of the ties that bind, leaves family behind, and promises escape. To which end, the spaceship is central in space opera just as the sailing ship was key to the age of (ahem) sail. Trying to imagine Moby Dick or Master and Commander without sailing ships is like trying to imagine Star Trek or Star Wars without starships.

You can write romantic adventure-focussed SF without starships (use stargates/wormholes or just set it on a planet where adventure-shaped activities dominate the plot) but what you get then isn't space opera, it's a planetary romance.

(Planetary romance (the sub-genre Jack Vance was famous for) inherently places more constraints on what you can do with your fiction. You can't easily have a planetary setting which combines long travel times, high energy technologies and high population density, for example: long travel times imply no fast vehicles (no high speed rail or air travel, never mind spaceships), which are a side-effect of high energy technologies (internal combustion, gas turbines, big-ass electrical generators and HV grid of any kind) and you can't easily get to any of those things without a sufficiently high population density to support the industries that make them feasible (not just engineering and factories but universities and schools with mass literacy and mining) which in turn implies advanced agriculture—you've got to feed those cities somehow. (Memo to self: hunt down James Burke's Connections on video (they're somewhere on YouTube) and re-watch the series.) It also makes it difficult to justify sub-settings with wildly divergent levels of technology because that stuff leaks—to keep it out you need a North Korean level hermit kingdom, or totally impenetrable jungles that haven't been penetrated by colonialists, or something of that level. (NB: I have found a workaround for this, but I'm not going to spoiler the coming novels!)

Ultimately economics are your enemy if you want to write convincing romanticist fiction without turning into a fantasy-without-the-magic secondary world setting: all it takes is one country inventing capitalism and/or imperialism to muck it up for everybody else). And economics fucks with space opera too: ask yourself what markets plausibly exist for goods and services transported across interstellar distances, and after digesting Paul Krugman's Theory of Interstellar Trade (PDF) you may be left scratching your head.

My go-to answer for economics in space opera is that interstellar expansion/colonization is most plausibly driven by (a) religion ("manifest destiny" is a religious belief, built atop Genesis 9:7), or (b) escapist colonialism (but only if travel is cheap and compatible biospheres are abundant). In third place we probably get pyramid schemes (eg. Neptune's Brood by some guy called Stross: prior art for that one existed in the shape of colonial pyramid schemes, eg. the Darien Scheme which bankrupted Scotland, or the South Sea Bubble, which nearly bankrupted the British government a generation later). Then a long way down the list we get to trade in Veblen Goods, and maybe—if interstellar transport is very cheap—recreation. What the latter two items have in common is that they're status signifiers—the wealthy have been having yachting adventures while the peasants labor on their estates to pay for them for a very long time, and as for Veblen Goods, Faberge eggs are pretty obvious. So we have status as a driver for space opera: a side-effect of hierarchical social structures where the elite seek to maintain their status by signaling their affluence in various ways.

Did I mention hierarchical social structures and capitalism? The thing about capitalism is that it's a totalizing ideology, like the divine right of kings, or religious fundamentalism. It seeks converts, it refuses to be gainsaid, and it confers a competitive advantage on cultures that adopt it (as opposed to those that reject it). Yet, as Ursula le Guin said, "We live in capitalism. Its power seems inescapable. So did the divine right of kings. Any human power can be resisted and changed by human beings..."

Space opera embraces the romanticist outlook. This is inherently problematic, for it is capable of spawning totalitarian nightmares. To understand this point, I dare you to read Marinetti's Futurist Manifesto from 1909 and not recognize it both as a masterpiece of romantic rhetoric and as one of the founding texts of Fascism—very specifically the Italian Fascist movement (Nazism and Spanish clerico-fascism were later tumors on the rancid politics of violence and misogyny that Marinetti unleashed).

It's not possible to examine space opera, or planetary romance, and not acknowledge the pernicious fascistic imagery in both subgenres. Michael Moorcock wrote a magisterial essay about this nexus in the wake of Star Wars (the first movie): I strongly recommend reading his Starship Stormtroopers before pressing on if you're not already familiar with it.

Pace Le Guin, it is possible to write space opera that doesn't pander to fascism directly, as the late Iain M. Banks demonstrated. But even in his magisterial Culture series the pustules of fascism can easily be seen on the faces of the Culture's adversaries: from the Idiran imperialists to the Empire of Azad, perhaps most clearly in the character of the Affront, it takes a bad dose of fascism to introduce the romanticist element necessary for space opera to be perpetrated. Otherwise you're left with something like Inversions, which is a Culture novel with merely homeopathic traces of space opera remaining. Incidentally I make no apologies for assuming a familiar with the Culture novels on the part of the reader: if you haven't read it already, you're missing out, because it's the nearest thing I've seen to a refutation of the core conceit of might-makes-right inherent in the fascist strain of space opera. Iain wrote A Few Notes on the Culture in 1993 to explain what he was about: in particular, I was struck by this summary: The Culture, in its history and its on-going form, is an expression of the idea that the nature of space itself determines the type of civilisations which will thrive there. ... Essentially, the contention is that our currently dominant power systems cannot long survive in space; beyond a certain technological level a degree of anarchy is arguably inevitable and anyway preferable.

Totalizing ideologies like capitalism or Christian Nationalism aren't immutable, and opposition is possible. They rely on a perception of inescapability on the part of their subjects, and space opera should be the ideal fictional vehicle for escapism from such certainties: if our protagonists can board a starship and flee, there is in principle no reason why they can't shed the chains of a pernicious ideological system.

Now I want to cut to a different question. What about the human-scale story?

Space opera isn't usually a genre of high conceptual merit that tries to tell us something about the world we live in. It can be, in the hands of a determined author who knows what they're doing (I'm thinking specifically of Iain Banks, of Bruce Sterling's Schismatrix, of some of Alastair Reynolds' work—notably House of Suns) but it's more often deployed as a giant shiny backdrop for human-scale drama. Fiction is generally the art of confabulating plausible lies that illuminate the human condition, but stars and galaxies exist on a very superhuman scale: if we took them seriously as a setting they'd dwarf the merely human concerns of the protagonists—the emotional cursors that traverses the story universe—into insignificance, as in the more philosophical works of Olaf Stapledon. (For a fine example, see Anvil of Stars by Greg Bear.) Most of us don't have the imaginative flexibility to conceive of time scales significantly longer than a human lifespan and vistas larger than a single planet (and even that is a recent development: I think before mass intercontinental air travel our imaginative horizons were far more constrained). So a common problem in space opera is for the scale of distances to collapse: modern authors tend to fall back on the durations of international air travel, rather than thinking in terms of age-of-sail voyages, let alone more realistic interstellar durations at slower than light speeds. (Honorable exception to that one: Alastair Reynolds, and to a lesser extent, Ursula Le Guin's Hainish Worlds stories and Ken Macleod's Engines of Light tetralogy.)

A side-effect of the romantic outlook is a demand for individual human agency—that is, that despite the vast scale of the backdrop, the protagonists have the ability to effect change, or at least to change their own circumstances. Here in the real world most of us are trapped by situational constraints: our employment system is parodied as wage slavery for a reason. Opportunities for adventure are scarce, and your chances of self-betterment are minimal. (You can found a business and spend 30 years trying to become a millionaire, most likely to end in ignominious failure, or you play the lottery, be it by explicit gambling or by speculation on the cryptocurrency markets, but the game is rigged against you from the start.) Money in large quantities, as capital, exerts a gravitational attraction and we are individually just dust particles. So space opera tends to focus on protagonists who acquire individual agency, have adventures that take them away from the quotidian grind, and let them change themselves or the worlds around them.

A classic plot skeleton for this sort of fiction is the Hero's Journey template, as popularized by Joseph Campbell and demonstrated by George Lucas in Star Wars, and it's less-familiar-to-many sibling, the Heroine's Journey: interestingly the Heroine's Journey can be seen as an explicit rejection of the frenetic diktat of Futurism even though it's compatible with space opera, as demonstrated magisterially by Lois McMaster Bujold, whose Vorkosigan series is bookended by a Heroine's Journey narrative (starting in Shards of Honor, continuing in Barrayar, finally winding up in Gentleman Jole and the Red Queen)—and which holds the singular record for most Hugo awards for a single SF series (so this isn't just my opinion). Character development lives at the heart of both these templates, and I suspect you can't have a working space opera without the scope for such changes.

To briefly hark back to the Futurist/fascist imagery inherent in space opera: the Vorkosigan series is arguably Ruritanian romantic space opera set against a recognizable 19th century central European flavoured empire. (It's no accident that space opera emerged at the end of the age of empires, the turn of the 20th century.) Our main viewpoint character, Cordelia Vorkosigan, is a woman with modern sensibilities who effects change in that society to great effect, working largely behind the scenes as a noblewoman in that setting—indeed, that's the nature of her modified Heroine's Journey arc. (Assuming, of course, that you buy either the Hero's Journey or Heroine's Journey templates, both which are heavily influenced by Jungian psychology and early-to-mid 20th century conceptions of masculine and feminine gender identity, all of which are highly problematic in my opinion.)

Ultimately, though, the romanticist aesthetic is inimical to social convention and formalism—and also to sustainability. The romantic impulse renders space opera antithetical to solarpunk. A fictional setting composed entirely of wild individualists isn't going to end up as some sort of libertarian utopia, it's going to be a cat ranch: meanwhile someone needs to stay behind and tend the home fires, someone has to mind the grain silos on that "small farming planet", someone else needs to manage the shipyard that repairs the interstellar freighter. There's a lot of implied background in space opera, a civilization with enough surplus wealth production to build and operate starships, and this implies some degree of continuity—continuity of biosphere stability (or there'll be nothing to eat), continuity of infrastructure (including education/apprenticeships, to train/socialize the starship crews), continuity of the society our adventurer-protagonists emerged from. Social structures, governance, and politics are a given in every human society, and the romantic movement emerged from a very specific context that coincided with the age of enlightenment and the industrial revolution. Our protagonists are either privileged enough to transcend the daily grind or sufficiently motivated to take a hideous gamble with their own safety. And this tells us something about the constraints within which space operatic storytelling takes place—and my work-in-progress in particular.

Any questions?

This isn't a blog entry I wanted to write, but it's a necessary one: a statement about the use of generative large language models (colloquially "AI") in my work.

I do not use LLMs in my work. I don't use them in my non-work life either, for that matter. I despise the grifters selling these toys as "tools" and trying to convince us to use them to generate plausible answer-shaped text strings in place of actual internet search for verifiable sources.

I've been selling fiction that I wrote myself since 1985 or thereabouts, and novels since 2002. If you want to verify that I have written novels without using an AI, simply pick up a physical copy of "Singularity Sky", "Iron Sunrise", "The Atrocity Archives", or anything else I published before 2015, the year OpenAI was founded.

Hint: you will find seven Hugo-shortlisted novels from that period, and three Hugo-winning novellas, also two Locus-award winning novels and a couple more novellas and stories. Clearly I don't need AI to write award-winning stories.

I do not want or need a large language model to write my fiction for me. I write fiction compulsively—before I was published I wrote for many years as a hobbyist—so why on earth would I pay someone else to take my fun away?

You will note em-dashes in the preceding paragraph. I gather some "AI detector" services (themselves a generative AI product) flag em-dashes as signs of "AI generated" text. Listen, fuckers, LLMs sprinkle em-dashes in their output because LLMs exist to stochastically emit strings of text that approximate the form of their inputs, and they've been trained by stealing all the text on the internet that isn't nailed down, including pirate websites that distribute cracked e-books. So it's wholly unsurprising that LLM output exhibits quirks that mimic real writers.

Did I mention the "stealing" thing? This isn't hyperbole: I'm one of the parties to the settlement in the class action lawsuit against Anthropic AI for pirating ebooks to train their LLMs. That's not my only grievance, either. You may have noticed this blog performing sluggishly or crapping out from time to time over the past few months. That's because my server is old and feeble and periodically gets swarmed by Chinese and other foreign botnets scraping data for training LLMs.

I'm usually willing to cut actual human beings, as opposed to for-profit corporations, some slack where it comes to cracking DRM, or even downloading warez: but these people are absolute scum. They're stealing copyrighted material to train an LLM that is intended to compete for revenue with the authors of the works they stole, and they're fine-tuning their LLMs to make them as addictive as possible in order to maximize future revenue once they pivot to token sales as their main source of income. In other words, they're no different from a burglar who robs you one day then comes round to sell you your stuff back the next morning. Back in the 18th century we used to hang people like that and Sam Altman makes me question the wisdom of having stopped.

I maintain that any serious author should shun LLMs like the plague. The most popular LLMs in the west—such as Claude, Gemini, CoPilot, and ChatGPT—the ones hoovering text indiscriminately off the internet for training—also gobble up any queries you send to them and use them as future training data. If I was crazy enough to feed the outline of a story I was working on as a prompt to ChatGPT or Claude in hope of getting the stochastic parrot to do my homework for me, then it would be only my own fault and nobody else's if the next model from the company in question was trained on my book outline and could reproduce part or all of it for someone else.

Finally, contra public opinion, I see no reason to credit LLMs with sentience. They're word-association mechanisms with no embodiment and no way to associate the text vectors they manipulate with real-world phenomena. But we humans have evolved through selection pressure in an adversarial environment to associate environmental phenomena around us with intentional causes—if you see lion scat and the gazelle are no longer visiting the watering hole, then you should assume there are lions about. And this trait carries over to linguistic manipulation. If we hear or read text, we expect there to be a mind on the other side of it, as Joseph Weizenbaum (the inventor of the original ELIZA chatbot) realized at MIT in the late 1960s. Just because it does something people do, it does not follow that it is a person.

Now for some caveats.

My skepticism does not carry over to all aspects of the field. It would be foolish to deny the effectiveness of image recognizers based on generalized adversarial networks (GANs), the key neural network technology underlying LLMs. It'd be similarly stupid to deny that LLMs are very good at supporting large-scale statistical analysis of text, such as Linear-A. And I can see some circumstances where being able to train a local model on my work could be useful to me.

I'd quite like a tool (running entirely locally on my own hardware, with no cloud service and no copyright-thieving grifters making bank on it via subscription fees) that digests a manuscript and derives a scene-by-scene timeline, that I could then query interactively and use to plan my next round of edits. Being able to map out where and when each protagonist and minor character shows up, and see a frequency distribution heat map of names in the manuscript, would be useful.

But such a tool would be useful to me in the same way a spelling checker is useful—as a decision-support tool, not as a substitute for doing the hard work (and having a copy of the Oxford English Dictionary on the shelf). The value of such a tool is considerably less than the value of a well-trained brain that can do the entire job the hard way, if necessary. And it's less than zero if using it opens me to finger-pointing accusations of "but he's using AI!" by people who can't read to the end of one paragraph, much less fourteen of them (yes, this is para fourteen, I've been counting).

So my fiction is still, as of August 2026, 100% LLM-free, and if that changes I will update this declaration accordingly.

Finally, I'd like to leave you with a snippet from the opening of the far future space opera I'm editing right now. It's part of the fiction and unfortunately may have to be omitted because of the risk of confusing the people who can't read to the end of the paragraph, but it's the only valid use of LLMs I've found so far for my fiction because it's a solution to the calling a rabbit a smeerp problem in SF and fantasy:

Translator's Note

The events described in this account have been translated into your language from the original source material using a non-sapient large language model.

Certain terms have been approximated, where possible, by using culturally appropriate cognates. Names of individuals have been replaced by equivalents. Similarly, institutions, ranks, religions, proverbs, idioms, quotations, and other culturally-determined signifiers have been translated into terms that will be familiar to the reader.

Units of duration and distance have also been converted.

We apologize in advance for any hallucinations our LLM may have inadvertently introduced in the process of generating this rough translation.

Smells awful:

But an estimated 30 to 50 tons of dead squid remain inside the boat’s catch tank, where they have been decomposing for days. “That is nasty. I wouldn’t want to do that,” said commercial fisherman Dick Ogg of the Bodega Bay Fishermen’s Marketing Association.

Ogg said anyone familiar with the fishing industry understands what happens when a large catch sits for an extended period.

“If you think about what happens after four or five days, it’s a gooey mess,” he said.

The odor has become a defining feature of the operation, and the beach remains closed to the public while crews work on a removal plan.

According to salvage expert Ernie English of Parker Diving Service, the squid has deteriorated into a thick mass that will be difficult to remove.

“It’s like concrete,” English said when asked about its consistency.

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Blog moderation policy.

Last month, I gave a talk at DEF CON on AI hacking: what happens when AIs become hackers. It’s a combination of the potentialities I raised in my 2022 book A Hacker’s Mind and the lessons we’re learning from current AI models engaging in hacking behavior. I’m really proud of the talk, and the fact that it gained over 100K views on YouTube in just a few days.

Also online is an interview with me in the AI Village.

We are now told by the same IDF [Israeli Defense Force] that there is no
wrongdoing worth referring to a criminal investigation. As a family, we do
not believe this to be accurate, and we continue to call for an independent

I just worked two weeks in a farm in New South Wales and one of my friends
stayed with my kids at home. With rent, sometimes I pay on time. Sometimes I
call the landlord and say I will pay in two weeks. I really need my work

The Regicide Report came out in January 2026. Traditionally I wait for the paperback before writing one of these spoiler-laden crib sheets, but there won't be a US paperback edition and the UK one isn't until the end of the year: if you don't want to wait, you don't have to.

So here it is.

The Laundry Files main story arc runs through nine novels, not including A Conventional Boy, a number of novellas and short stories (of which ACB was originally intended to be one—it over-ran), and the New Management trilogy (which was originally going to be a separate successor series to The Laundry Files: it starts 18 months after the end of The Regicide Report—it turned out to be a marketing train-wreck, which I blame on COVID19 induced mix-ups on the publishing end of things). There may eventually be a short story collection, as most of the shorts have never been published in paper editions, but this is it for the main story, which was (since The Fuller Memorandum) intended to end with the final CASE NIGHTMARE GREEN confrontation.

One huge problem with writing any vaguely-contemporary thriller series is that the world doesn't stand still underneath your fictional version of the universe.

I originally intended to accommodate this by advancing the date from novel to novel at the same speed time passed in the real world. The Atrocity Archives were set circa 2001-03, The Jennifer Morgue in 2005, and so on. Bob had room to grow older: The Annihilation Score was set in 2012 and by The Nightmare Stacks the clock had run out to 2014.

But just as lot of cold war spy thrillers were left stranded by the sudden end of the Cold War in 1989-91, I was blindsided by the Brexit referendum and its consequences in 2015. Prior to Brexit, British politics had been evolving along roughly predictable lines since Thatcher came to power in 1979, drove a tank over the prior bipartisan social democratic consensus politics, and ushered in an era dominated by a rapacious neoliberal ideology. The unexpected Brexit referendum outcome derailed the freight train, with consequences that are still emerging a decade later, and left me supporting an increasingly precarious pile of spinning plates.

An immediate consequence of Brexit, in The Laundry Files, was that I had to hastily rewrite The Delirium Brief (after it was substantially complete), giving it a similar political rupture leading to the rise of the New Management.

But unfolding multi-book catastrophes take many years to write, and by the time I got through that point the Laundryverse was rapidly decoupling from real time. The period 2015-2019 coincided with my parents' final decline and death (they both made it into their 90s), then the collective trauma of COVID19. The Laundryverse as of 2019 was still stuck in an in-world version of 2014, and rapidly receding into the past. I managed to un-stick the clock for the New Management books (the original working title of which was Laundry Files: The Next Generation) and set them in 2016-17, but the series was already turning into alternate history by the time I got around to finishing writing A Conventional Boy (set circa 2011, the same year I began writing it: finally published in 2024).

At the same time, my publishers gently warned me that sales were threatening to enter the dreaded midlist death spiral. A midlist death spiral occurs when an author's sales decline from one book to the next. Bookstores base their orders for a new title in a series on a straight line extrapolation (no curve fitting!) of the previous two books, so any decline fatally undermines advance orders, and thereby sets up a self-fulfilling prophecy of decline. It was therefore time to wrap the series—at least, if I wanted to be able to earn a living in future years.

Which set me up for The Regicide Report, in which all the homing pigeons I'd released in earlier books would come back to roost—or at least as many as I could keep track of in my head (I write by the seat of my pants, there's no World Book in my desk drawer, and over 25 years you tend to forget little details).

Because The New Management books were already in print, I was writing inside certain constraints. The designated climax had to be finished in-universe by May 2015 (The Labyrinth Index was set in mid-2014). It needed to feature Bob and Mo, but Bob and Mo as they had evolved—on the threshold of middle age, cynical, burned-out, and constantly asking "are we the baddies?". It needed a confrontation with the Prime Minister in which he is left in absolute authority over the UK but his ambition to ascend to full godhood is thwarted. It demanded cameos by numerous characters, a climactic boss battle that made sense in context, and an ending that didn't amount to a personal tragedy for the original protagonists: you don't want to leave your long-term fans hating you at the end of a series. ("The fans are out there. They can't be bargained with. They can't be reasoned with. They don't feel pity, or remorse, or fear! And they absolutely will not stop, ever, until you are dead." Ahem: my apologies to James Cameron and Gale Anne Hurd, not to mention any non-Terminator fans that exist.)

The driver for the climactic confrontation in the series is the Black Pharaoh's goal of achieving a death-grip on the British state. This inevitably means confronting the ultimate source of occult power in the kingdom, the monarchy itself: but it's a novel I couldn't have pitched to my British publisher before September 8th, 2022. Elizabeth II was remarkably well-loved, or at least respected as a public figure, and pitching a novel about her assassination was ... well, it would have been inadvisable. However, following her actual death (probably from consequences of COVID19: following infection elderly patients are at very high risk of stroke or heart attack for several months) she suddenly graduated from reigning monarch to historical figure, and as such was no more off-limits than Queen Victoria or President Kennedy.

So my remit was: write a book in which the Black Pharaoh tries to bump off the Queen in 2015, fails to achieve occult supremacy, Bob et al battle him to a stalemate, and we ring down the curtain on the Laundry as an organization (indeed, by the end of The Regicide Report the Laundry of yore has been purged and its various duties merged into a new ministry directly controlled by the Black Pharaoh.)

Of necessity I had to start The Regicide Report by dumping a bucket of ordure over Bob's head—that committee meeting, where he accidentally outs a senior colleague by forgetting to reset the joke ringtone on his phone—and gets sent on a tour of outlying civil service offices as punishment. Yes, the Birmingham scene features an extensive Hot Fuzz tribute: yes, that is DI Angel. (It's one of the few early 21st century movies with cinematography that my damaged eyeballs and retinas could follow.)

One of the hallmarks of The Laundry Files is the repeated trope of pastiching thriller authors or urban fantasy subgenres. The Regicide Report kinda-sorta does this, only differently, by picking on a 1970s British movie anti-hero, The Abominable Doctor Phibes, a role portrayed stunningly well by Vincent Price in the two movies that actually got filmed (The Abominable Doctor Phibes and Doctor Phibes Rises Again). These films were among masterpieces of 1950s-1970s British horror genre, but are not without their weaknesses, and I'm not just talking about the cheap special effects. I had a loud argument with the scriptwriters in the privacy of my own skull, because the two most significant female characters (Vulnavia, Phibes' murderous muse, and Mrs Phibes) have zero talking lines in either film. This, I felt, was selling them both short. And besides, there was an obvious (to me) subtext that made the Phibes menage both Laundry-adjacent and explained the silence of the priestesses. If you watch the real movies then read the descriptions Bob and Mo give during their movie night, you'll spot some divergences: the Professor Phibes Bob meets in the Laundryverse is not the Dr Phibes of our world, nor are the movies exactly the same. (Let alone the third one, Dr. Phibes meets Mabuse the Gambler, notionally made in 1973 while Phibes was sleeping away the years in his glass coffin and not in a position to murder the producers.) NB: keep an eye open for the Cabaret references in that last one.

The assassination is carried out by means of poison: the toxic substance in question is entirely real and absolutely horrifying. Luckily you're very unlikely to come across it in real life, unless you work with laboratory assay equipment measuring environmental mercury contamination.

Buckingham Palace is indeed as vast and labyrinthine as I described it, but does not, to the best of my knowledge, feature server farms in the attic and a ritual sacrificial mock-up of the above-ground quarters in the basement. (It does have a bowling alley and, quite probably, a cinema organ.) There were plans to provide an emergency evacuation route via the Tube before the second world war, although it's unlikely the Royal Family would be in residence or evacuated that way in a real crisis today.

The basement crypt and archive of royal skeletal remains at Westminster Abbey is my own invention, as is the underground river, although there's an awful lot of buried history there: the site has been in use for over nine centuries.

As a point of note, if there were any historical truth behind the legend of King Arthur Pendragon, he'd almost certainly not feel any kinship to today's royals, who are descendants of a German dynasty invited in during the 18th century. Per legend Arthur was a 5th/6th century figure who led the post-Roman Britons. No Angles, Saxons, or Normans need apply. Nor is today's United Kingdom, or even today's England, clearly related to Arthur's: we don't speak the same language, England in its modern borders was only united during the 9th and 10th centuries, the prevailing religion back then would have been either a pre-Christian pagan tradition or very early Catholicism, and so on. Much of the Arthuriana we are familiar with today was invented out of whole cloth in the 12th to 14th century, at a time as far removed from its subject matter as that time is removed from us in this day and age.

Anyway, that's a round-up of my talking points about The Regicide Report. If you have any questions about the book, feel free to ask in the comments below.

In August, Cliff Stoll gave a talk at DEF CON, remembering the wily hacker he stalked forty years ago.

Great fun.

Welcome to the August 2026 report from the Reproducible Builds project!

In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not.

In this month’s report, we cover:

1. New updated SBOM specification from CISA.gov
2. LWN on Bootstrappable builds at FOSSY 2026
3. ”What’s missing to have reproducible builds on PyPI?”
4. Distribution work
5. Unreproducible builds under EROFS filesystem fixed
6. Tool and documentation development
7. Six new scholarly papers
8. Patches

New updated SBOM specification from CISA.gov

CISA, the Cybersecurity and Infrastructure Security Agency of the U.S. government published some the joint guidance entitled Minimum Elements for a Software Bill of Materials (SBOM), which updates and supersedes the baseline 2021 version covered in previous editions of these reports.

Whilst the PDF is worth skimming, the interesting changes include that the specification now mandates standard cryptographic hashes: unlike earlier standards that allowed hash omission or manifest-only parsing, hashes must be computed from the output. This is is important for reproducible builds, as it ensures the recording of the metadata required to demonstrate the shipped software matches the build output precisely where applicable. In addition, where the top-level only dependency limitation that was present in the 2021 version has been removed in favour of complete coverage with no minimum depth. That is, SBOMs are expected to reflect all linked libraries, vendored dependencies and other build-time inclusions.

LWN on Bootstrappable builds at FOSSY 2026

In the “Toolchains and Other Development Tools” track at Software Freedom Conservancy’s FOSSY 2026 in British Columbia, Canada, Timothy Sample gave a presentation on bootstrappable builds. This presentation was then covered in a Linux Weekly News article by Jake Edge entitled Bootstrappable builds: how and why, which serves as an excellent introduction to the concept:

The basic idea behind bootstrappable builds is to create a system that can be built without relying on pre-built artifacts. “Can we go from zero to the modern day without having to just assume the existence of these already-built-for-us artifacts?” The classic recipe for yogurt requires some yogurt to start the process, which is like how we normally build a C compiler today—we start with an existing C compiler binary. You might think about making sourdough bread with your grandmother’s starter brought over from the old country; “we’re basically making C compilers with Dennis Ritchie’s starter carried over from Bell Labs”.

The article, which goes on to cover GNU Mes and other projects that overlap with Reproducible Builds, also has a number of thought-provoking comments.

”What’s missing to have reproducible builds on PyPI?”

Core Python developer, Brett Cannon wrote an interesting blog post this month addressing What’s missing to have reproducible builds on PyPI, the official public repository for third-party Python software packages:

The reason I like the idea of making reproducible builds work is that I think it can be done in such a way as to not require any work on the part of the producer of a distribution (which is a technical term for sdists or wheels, i.e., the people who upload stuff to PyPI), and thus make reproducible builds very low-friction for people to opt into supporting. […]

Brett goes on to outline “What’s missing from the specs” and how reproducibility might be visible on PyPI to consumers:

Assuming all of this comes to pass and we record the where the source code is that went into a distribution and the software used to make the distribution, how do we make it useful to people? Does every person who cares about having a secure supply chain have to rebuild everything they use themselves? Is there some way for even people who don’t care about this stuff to benefit? […]

Distribution work

In Debian this month, 23 reviews of Debian packages were added, 28 were updated and 27 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated as well, such as the addition of a new toolchain issue related to python-traitlets […], and the note for an existing issue related to texi2html was updated as well […].

Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their reproducibility work there.

Unreproducible builds under EROFS filesystem fixed

Martin Pitt reported on Fosstodon that they had identified an issue where the mkfs (“make filesystem”) command for the EROFS (Enhanced Read-Only File System) subsystem of the Linux kernel did not have sorted extended file attributes, leading to reproducible builds.

Thankfully, Martin also reported that they had fixed this in a commit to the kernel which “order[s] each inode’s xattrs by name so that images stay reproducible”. […]

Tool and documentation development

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of chnages, including preparing and uploading versions 327, 328 and 329 to Debian. In particular, he ensured that diffoscope did not require python3-guestfs in the autopkgtests on 32-bit architectures in order to fix Debian bug (#1144372) […]. Colin Watson made an additional change, handling a potentially missing openssh-client package when running the autopkgtests […], and Jochen Sprickerhof made a similar change to cope with missing cpio and qemu-img functionality […] whilst also updating the XML comparator to be considered when comparing SVG images […].

Yet again, there were a number of improvements made to our website this month as well. For example:

• Chris Lamb added added draft for a Gothenburg summit-related news article. […][…]

• Holger Levsen then published the same article. […][…]

• Lastly, a large number of commits were pushed comprising an interview with Reproducible Builds developer Jochen Sprickerhof to be published within the next week. […][…][…][…][…][…][…]

Six new scholarly papers

Jens Dietrich, Spencer Sun, Tim W. White and Behnaz Hassanshahi (the result of a collaboration between Victoria University of Wellington and Oracle Australia published a paper this month entitled No Snake Oil: Verifying Python Package Builds. Drawing on the metaphor of “snake oil”, that is, a fake or ineffective medicine or solution sold with exaggerated claims of curing or fixing everything, the authors write that

Two tools that are designed to automate [PyPI] rebuilds and run them at scale are macaron and oss-rebuild. We study 12,180 popular releases from PyPI and find that the byte-for-byte equivalence rate is generally low. We analyse the reasons why they produce different wheels, and find that equivalence between the original and rebuilt wheels can often still be established, preserving most of the guarantees users expect from rebuildable releases. We present and evaluate daleq4py, a tool to establish the equivalence of Python wheels through the kernel of a normalisation function that is based on provenance-preserving datalog rules. Experimental results show that daleq4py substantially expands the set of rebuilds that can be accepted as equivalent.

The full PDF of their paper can be viewed online, and Jens Dietrich to our mailing list to announce the availability of both the paper and the daleq4py tool itself.

Dimitri Kokkonis, Michaël Marcozzi and Stefano Zacchiroli published an article this month titled Not In My Git Yard: Catching Backdoors at Commit and Release Time on the topic of “code-level backdoors” — that is, “stealthy code changes that grant hidden privileges via secret triggers”. These issues:

… pose a persistent threat to opensource software. Known attempts to inject such backdoors into widely used projects through malicious commits, tampered release packages, or compromised third-party dependencies, were stopped only by luck and manual review. Existing Continuous Integration (CI) pipelines cannot detect these attacks, and downstream binary analysis tools require substantial manual effort. In this work, we present Lily, an automated approach that strengthens open-source development and release processes against backdoor injection. Lily integrates a backdoor detection mechanism into (1) CI pipelines to block malicious commits, and (2) release vetting workflows to prevent tampered releases or compromised dependencies from entering large ecosystems, such as Linux distributions.

The full PDF can be read online.

Ranindya Paramitha and Laurie Williams of North Carolina State University along with Christian Kästner of Carnegie Mellon University published a paper this month with the title of The Software Supply Chain as a Market for Lemons: A Multivocal Review of Trust Signal Collapse. (A “lemon” in American English, is a vehicle that “turns out to have several manufacturing defects”.) Their abstract is as follows:

Practitioners evaluating open-source dependencies rely on cheap trust signals, e.g., stars, download counts, and contributor activity, as substitutes for direct code inspection, assuming those signals reflect genuine trustworthiness. Prior work has documented individual signal gaming, but the landscape of collapses across all dependency-adoption signals, as well as the ecosystem’s response, remains unexplored. The goal of this study is to aid software practitioners in understanding the reliability of dependency adoption trust signals, such as download counts and contributor activity, by conducting a multivocal review of 252 Google Search sources and 870 Reddit threads.

Worryingly, after their review, the authors conclude that “cheap trust signals collapse under three simultaneous forces: adversarial manipulation, gaming techniques indistinguishable from legitimate behavior, and non-adversarial AI-driven inflation.”

The full PDF of the paper is available online.

Julien Malka, Aman Sharma, Martin Monperrus, Stefano Zacchiroli and Théo Zimmermann published a paper this month on Trusting-Trust Attack against an Entire Linux Distribution through Binary Manipulation:

Ken Thompson’s trusting-trust attack, in which a compromised compiler backdoors the programs it builds and reproduces the backdoor in subsequent rebuilds of itself, is widely regarded as a threat specific to compilers. We show that it is not. We construct a complete trusting-trust attack around GNU strip, an ordinary build utility that neither inspects nor generates source code, using only manipulations of finished ELF files.

Scarily, in the authors’ example, “a single tampered strip in the binary seed implants a payload that propagates from one generation of strip to the next and survives into the final standard environment after the seed leaves the dependency closure […] without failures and backdoors”.

A full PDF of the paper is available for download online.

Mehdi Keshanimm, Amirhossein Rahmati, Mohammad Hossein Aref and Abbas Heydarnoori published a paper that is currently under review at Emperical Software Engineering titled AROMA+: A Study of Factors Affecting Reproducible Builds in the Maven Ecosystem. (Maven is a/the build automation tool used for Java projects.) In their paper, the authors note that

[…] reusing external software in a project presents a security risk when the source of the component is unknown or the consistency of a component cannot be verified. The SolarWinds attack serves as a popular example in which the injection of malicious code into a library affected thousands of customers and caused a loss of billions of dollars. […] Our research aims to support [reproducibility] efforts in the Maven ecosystem through automation. We investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA+, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of packages on Maven Central.

The full PDF of their article can be downloaded online.

Lastly, Oreofe Solarin, Kelechi Kalu, James C. Davis and Paschal Amusuo published a paper this month titled Reproducibility is Not Enough: Artifact Verifiability in Decentralized-Build Package Ecosystems:

[A]rtifact verification requires more than deterministic builds: a verifier must also recover the source state, build environment, dependencies, and build instructions that produced the artifact. Decentralized-build ecosystems make this difficult because artifacts are produced through heterogeneous tools, maintainer-controlled workflows, and fragmented metadata. As a result, it remains unclear how often artifacts in these ecosystems can be independently verified. This paper studies artifact verifiability across four popular decentralized-build package ecosystems. We define an independent verifier model that relies only on registry-derivable metadata and an artifact comparison model with tiered equivalence levels. We implement these models in an Artifact Verification Pipeline and use it to measure artifact verifiability across the target ecosystems.

The authors conclude that “beyond build determinism, verifiability is limited by missing source and build metadata, implicit release transformations, and unconventional build practices”.

A PDF of their paper can be reviewed online.

Patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:

  • Botan
  • Rivet
  • boost
  • ceph
  • cosmic/xdgen
  • cpio
  • dmd
  • dvgt
  • efl
  • firefox-esr
  • fritzing
  • gcc/esbuild
  • git-annex
  • go1.27/esbuild
  • gputils/sdcc
  • grass
  • gri
  • java-21-openj9
  • java-25-openj9
  • ksh
  • llvm22
  • llvm
  • llvm
  • lutgen
  • maxima
  • mingw64-filesystem
  • neochat
  • obs
  • python-PyMuPDF
  • python-gguf
  • python-langgraph
  • python-pook
  • python-pyzmq
  • python-xgrammar
  • python-xlsx2csv
  • qt6-tools
  • sbcl
  • scons
  • virtualbox
  • xpenguins
  • zabbix

• Chris Lamb:

  • #1143615 filed against keychain.
  • #1143616 filed against python-nameparser.
  • #1143617 filed against lcov.
  • #1144065 filed against displaycal-py3.
  • #1144066 filed against redmine.
  • #1144407 filed against django-htmx.
  • #1144409 filed against pybdsf.
  • #1145914 filed against golang-github-adamkorcz-go-fuzz-headers-1.
  • #1145915 filed against googletest.

• Jochen Sprickerhof:

  • #1145443 filed against cross-toolchain-base.
  • #1146088 filed against ferret-vis.

• Robin Candau:

  • bashburn
  • LMMS

• Werner Fink:

  • bash

If you are interested in contributing to the project, please visit our Contribute page on our website.

Some new and some old entries this week, from people who find zero profoundly unrewarding.

"A scary blue button from The North Face" warns Dmitry K.. "Northface has just proudly rewarded me with nothing. I am not sure whether I want to try adding such a non-positive amount to my wallet. I am afraid I can trigger a chain reaction that will crash the financial system of the whole world..."

"0% Steam discount" advises Renan "As much as I love Final Fantasy, I guess I'll pass on this 0% discount."

"Give me my money around zero" demands Reinier B. "The Nederlandse Spoorwegen (Dutch Railways) owe me Ä2.25, but apparently I need to go back in time 2026 years to get my money ("we'll transfer the amount to your bank account around 0"). Or maybe it's not zero AD but something else?"

"My relationship with Office Depot is very unrewarding," complains Philip. "Office Depot doesn't exist in my country anymore (someone signed up with my email address), but it's good to know I have $0 waiting for me if they come back."

"Too late for that!" exclaims an Anonymous "Me hard earned zero award points on this ride app will expire at the Unix epoch which is apparently coming up soon."

"Rewards point conversion math is hard" figures Emily. "Will I have -0.01 next month? The suspense is killing me."

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Author: Mahati Vaidyanathan Maahi hated the scientists. *I’m sorry,* Mahath transmitted through their link for the thirtieth time that morning. The buzz of his nervous caffeine sizzled across her mind. But nothing could change the fact that Maahi was on a plane to the other side of the planet surrounded by cold scientists while Mahath […]

The post Strawberries appeared first on 365tomorrows.

Two weeks ago, I released AppStream 1.2.0. This release contains a lot of great changes, but one of the most important ones concerns how media are being handled, and AppStream’s default image export format.

AppStream is a Freedesktop metadata standard to describe software components. That can be anything from system services over fonts to console and graphical applications. AppStream metadata is supposed to give users enough information to decide whether they want to install a piece of software, to represent that piece of software, and to give the operating system enough information to decide whether a software component should be installed automatically and (to some extent) what capabilities and relations it has, to provide the user with sensible options.

Especially for the first two goals, and especially for GUI applications, AppStream supports icons and screenshots, which are used to showcase applications. Today, AppStream is used by all kinds of services, from Linux distributions over firmware updates to Flatpak and desktops directly. AppStream’s original design however comes from the perspective of Linux distributions in 2011, where you may want to browse the software catalog offline, without delay, and without pinging an external server (which could be a privacy concern).

Therefore, a common way to deploy an AppStream-enabled software repository is to ship all icons of all applications in the repository to the user as part of the repository metadata download. AppStream does support remote icon downloads nowadays, and for a while I thought that this would become the default eventually. However, especially in today’s world, having a bandwidth-saving, instantly responsive, privacy-protecting application browsing experience seems more important that ever.

PNG images are great!

The only format that AppStream supports for icons and screenshots (which are downloaded on-demand from your distributor’s CDN) has always been exclusively PNG. PNG images are perfect for icons, because they compress well (especially for common icon shapes), are fast and simple to load, and can be loaded anywhere, by any toolkit or webbrowser. They also ensure we deliver faithful screenshot images, even though we may have scaled or re-rendered them. Still though, PNG images are less great for screenshots, as they are not very efficient, which puts strain on any CDN that has to deliver them, as well as on people’s internet connections when browsing screenshots. Having smaller thumbnails alleviates that problem a little, but does not fully solve it.

But even for icons, PNG could be improved upon: In many cases, icons are re-downloaded with the repository metadata again and again, so having a large icon tarball adds up to the data transferred during metadata refreshes. AppStream also now supports large 128x128px icons, which nobody in 2012 expected we would need, adding even more data that will be re-downloaded. Saving some space here translates directly to lower bandwidth costs as well as faster downloads for users.

To improve PNG file sizes, the AppStream Compose library, which handles all image processing and metadata catalog composition, was running optipng on all generated PNG images. That does create smaller PNG images, but they were still relatively large compared to other image formats.

For a long time though, there was no alternative to PNG images for icons: There was no lossless image compression format that could give us the same quality as PNG images and that was also widely supported.

JPEG-XL vs PNG in AppStream

Since 2021 we have JPEG-XL (JXL), which offers a true lossless mode with often better compression than PNG. The issue was that JPEG-XL wasn’t widely supported. Then, in 2025, the PDF Association selected JPEG-XL as the preferred image format for HDR images in PDFs, and now we are finally getting browser support and more ubiquitous availability of the format (you can try it right now in Firefox!).

For screenshots, using JXL’s lossy mode, it has obvious and extreme size advantages over PNG, so supporting JXL or WebP for screenshot images was an obvious choice. If JXL would support the lossless case very well as well though, we could serve many use cases with the same exported image format, which is very attractive to me.

So, the obvious next question was whether it was worth the pain of switching the icon format, so I did some measurements on real icons. For that I used the AppStream component icon pool that Debian Unstable ships, which is almost 5000 application icons of various sizes, and converted them to PNG:

PNG images saved with libpng at effort=4, compression=9, then optimized using optipng -o2, JXL images encoded using vips jxlsave lossless=1 effort=7 strip=1 via VIPS/libjxl.

As the table shows, using lossless JXL images over size-optimized PNG images (using optipng’s default settings) provides a roughly 20% gain. This does not look like much, until you consider how often these files are downloaded: A 20% file size reduction may only save 1-2 MiB of disk space, but if they are downloaded over and over again by many clients, it will save a lot of bandwidth.

Interesting JXL encoding findings

As a sidequest, I was curious why some images were larger than their PNG counterparts when encoded with JXL, and what the ones that were significantly smaller were.

In short, the biggest size reductions for JXL existed on images that were already small as PNG, and contained large, flat color surfaces with hard edges and simple shapes. They were not very interesting, and much of JXL’s wins come from accumulating smaller gains across all files, which compound the bigger icons get (especially at 128x128px, where JXL truly shines).

The events were JXL loses to PNG are more interesting: For example, it does quite poorly with pixel-art images that have a lot of repeating patterns. Those are encoded well by PNG, but less efficiently by JXL. Take for example Vonsh:

Icon of Vonsh, an SDL-based snake game, which PNG compresses better than JXL

My guess is that while PNG can exploit the repeating pixel patterns for compression, JXL’s predicts surrounding pixels from its neighbours, which fails too often and makes it pay almost full entropy per pixel. In this single rare case, the PNG is at 5.4 KiB, while the JXL is almost 8 KiB in size.

Other cases I looked at were arguably buggy input data, where color channels were hidden under the alpha channel of the input image. PNG could probably again exploit repeats, while we were forcing JXL to encode pixels that were invisible in the final image. This is arguably a problem with the original input data. Currently, AppStream does not make any changes to icons at all, but in future we might add a filter that removes invisible colors from images to solve this pathological case (it was only two icons out of 5000 though, so it is not a high priority).

The third case I found where JXL loses to PNG were icons with checkerboard-like patterns:

Icon of x3270, an IBM 3270 Terminal Emulator

For those, PNG can likely again exploit the repeating patterns, while a checkerboard layout is pretty bad for left/top predictors like JXL’s. However, in this case the size difference (and loss for JXL) is only 450 bytes, so even though JXL loses to PNG, it does so not by much.

JXL in AppStream

Given these findings, JPEG-XL is the default image format starting with AppStream 1.2.0. AppStream Compose will encode all images losslessly as JXL, while screenshots are encoded in lossy mode at Q=90 effort=7. Since the optipng step does not happen for JXL images, this comes at no speed penalty and is even a bit faster on modern x86_64 CPUs (where libjxl can use SIMD). PNG is still available, and Compose can be told to switch between the two formats.

Upsides of JXL in AppStream right now

If you use JXL in Compose or the recent release of appstream-generator, you will get much smaller images and, for screenshots, will benefit from other JPEG-XL features such as progressive decoding, providing a far nicer user experience. libAppStream has supported JXL icons since version 1.1.3, so your clients will need that version or a newer one, and all software centers will have to support loading JXL images (which all of them do, provided the right plugins are installed).

Downsides of switching to JXL too quickly

JXL is a very new format, so web browsers might not yet display it if you are serving webpages. Your clients may also have bugs in processing JXL images, as the format is still “new”. For example, switching on JXL in Debian sent KDE Discover into an infinite loop on startup while trying to load the icons (an issue which has been fixed, but clients will need that patch first before JXL is switched on).

This currently makes JXL enablement only possible when you know that your clients can support it. This is the case for me in Debian Unstable and Debian 14, which are using JXL images for a few weeks now, but not for any older releases. Platforms like Flatpak have it even harder, because they do know even less about their clients. So, even though it has big advantages, you may want to hold off on using JXL right away, and force PNG by setting the ImageFormat key to png in appstream-generator‘s configuration, or passing --image-format=png to appstreamcli compose.

It is also worth mentioning that JPEG-XL is much, much slower on systems that do not have SIMD instructions or for which the libjxl/jxl-rs library does not have them (such as apparently riscv64 right now). If this is a concern, you might not want to switch to JXL right away.

Media pipeline improvements

Besides the JXL default change, AppStream 1.2.0 also comes with a complete overhaul of its media processing pipeline. While libappstream, AppStream’s main library, does not do any media processing and comes with very minimal dependencies to be embedded in client applications and used on servers, the same can not be said about libappstream-compose, AppStream’s library to build metadata generating applications (the server-side part, usually).

The compose library has to render fonts into font specimen cards, inspect translation files, render SVG images, decode all kinds of raster images, inspect video files, etc. Especially the fonts, and the fact that fonts can appear in SVG images, has caused issues in the past, as libappstream-compose is a heavily threaded library and most font libraries can only work from a single thread. This forced the library to essentially go into single-thread mode anytime anything that could touch a font was being processed.

AppStream also originally was created for a “safe world” where applications were vetted by the distributors before their metadata was processed. This is increasingly not the case, so it made sense to put at least a few guardrails on the most complex part of the pipeline: The media processing. As part of the change, media processing was split out into a separate worker process. This solved two problems at once: Font handling was isolated in a single-threaded binary – if we wanted to handle fonts in parallel, we could simply spawn more workers. And, being in a separate process, the media processing could now be sandboxed.

As part of the multiprocess changes, Compose also switched from using GdkPixbuf to VIPS for image processing. The latter allows for much more fine-grained control over the image output and encoding, and comes with a lot of well-maintained filters and operations, which made it possible to eliminate a fair chunk of AppStream’s hand-rolled image processing operations. As part of this transition, we unfortunately lost the ability to read XPM images, which dropped about 20-30 applications from the pool at Debian. But in the name of security, this is a sensible choice, especially since most XPM icons were very small and low-resolution, and applications using them could benefit from adding a high-quality PNG icon anyway. With VIPS, we also now restrict the amount of image formats we can load to a sensible set, so extremely niche or unexpected formats will be outright rejected (this includes sane-but-unusual formats for screenshots and icons, such as TIFF images).

The Compose library, with all of these changes, will now just request high-level operations (e.g. “render a font card for this font to a JXL image”) from the worker, and provide it with input data in sealed memfds and output locations as FDs as well. On Linux systems, the worker will use Landlock if available, to block all write access to the filesystem, deny device access and deny TCP and UDP as well. The sandbox can certainly be tightened a fair bit in future, but this was a good and safe start to gain some experience with it without having things break too easily, given the many places Compose is used in (also, Landlock’s API is surprisingly nice to use, so it was easier than I thought to add in this early version).

With all of these changes, the libappstream-compose library is now also officially marked API-stable, so you should be able to rely on it in future to build new things (its API has barely changed in the past, and now with the new media API and defaults change in place, it was time to declare it stable).

I want to see / try this!

Currently, the easiest way to have a look at the new data is to check out Debian Unstable. If you have a JXL-enabled browser, you can also see the icons in AppStream Generator’s HTML pages for Debian Sid. If you are using appstream-generator for your distribution, you will also get much more pleasant statistics and HTML pages, as well as fully deterministic media output and a whole bunch of security updates, so, update to its recent 1.0 release.

Please keep in mind that if you switch to JXL, the client tools receiving the image data have to support it. Support varies depending on the Linux distribution, so, test it first and switch the default back to PNG in case you encounter any issues.

What’s next?

With so many features and changes landed, the next changes in AppStream will focus on improving what already exists and fixing any issues (there will be more blogposts about the other features 1.2.x delivers!). Testing with the entire Debian archive as data source makes me fairly confident though that there will not be many problems. In the longer term, tightening the media processing sandbox will also be something we might want to do, e.g. by hiding parts of the filesystem tree or filtering syscalls.

For JPEG-XL, one obvious question is “Will you add support for it to the Freedesktop icon-theme specification as supported format alongside PNG, SVG(Z), and XPM?”. For on-disk icon repositories, JXL’s space-savings are less compelling, and it being HDR-capable is also not necessarily a killer feature (PNG can go a long way!). However, JPEG-XL’s ability to immediately decode larger images at reduced resolution without resampling could legitimately be very powerful here, as applications could ship a single large image and quickly decode it at 1/2, 1/4 or 1/8 the size for different purposes in their UI. JPEG-XL also supports spot-color extra channels, which applications could use as masks to recolor raster icons at render time. This could be incredibly nice to color symbolic icons on-the-fly without any SVG and CSS. JXL also provides richer metadata, which might be neat for (license/author) documentation. So, the answer here is: Maybe it makes sense to allow another format, but this will have to be discussed first, as it would force JXL into every toolkit and desktop, which is a much bigger ask than supporting it only in AppStream.

As always, let me know what you think and please report any issues or bugs directly against AppStream or AppStream Generator if you encounter problems that are with the tools, and not with a project’s metadata.

An new maintenance version 0.2.8 of the random-number generator tester RDieHarder (based on the DieHarder suite developed / maintained by Robert Brown with contributions by David Bauer and myself along with other contributors) is now on CRAN and available via r2u.

This release contains only internal maintenance changes: continuous integration was updated a few times, newer nags from R are addressed in Rd files and the vignette, and we also updated a few URLs in the vignette and README.me. No new code, no new features.

Thanks to CRANberries, you can also look at the most recent diff to the previous release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

There’s not a whole lot to report here. During August I spent some time on holiday and also had less work time available for Debian LTS.

• Debian packages:
  • firmware-nonfree:
    • Bugs:
      • replied to #1143446: firmware-nonfree: split Lontium firmware into a separate binary package
    • Merge requests:
      • closed !152: lontium: create new package containing firmware for Lontium display bridge devices
      • opened and merged !153: Update to 20260810
    • Uploads:
      • uploaded version 20260810-1 to unstable
  • linux:
    • Bugs:
      • replied to #1141407: linux-image-amd64: hid-generic 0003:051D:0002.0002: Event data for report 22 was too short (4 vs 2)
      • closed #1143619: linux-image-686: metapackage missing for 6.1.180-1 on 686 (LTS)
      • closed #1144056: linux-image-amd64: missing CONFIG_CRYPTO_SHA384 causes IMA error
    • Merge requests:
      • merged !2039: Add support for Alcor Micro AU6625 PCI-E Flash card reader
      • merged !2041: Update to 5.10.262 (LTS)
      • merged !2042: 6.1 stable updates (LTS)
      • merged !2048: Update to 6.1.180 (LTS)
      • merged !2050: 6.1 stable updates (LTS)
      • reviewed !2059: [arm64] Enable CONFIG_SND_SOC_ES8328_I2C as module
      • reviewed and merged !2063: 6.1 stable updates (LTS)
      • merged !2068: 6.1 stable updates (LTS)
    • Uploads:
      • uploaded version 6.1.180-1 to bookworm-security (LTS)
      • uploaded version 7.1.7-1~bpo13+1 to trixie-backports
      • uploaded version 7.1.8-1~bpo13+1 to trixie-backports
  • linux-6.12 (LTS):
    • Uploads:
      • uploaded version 6.12.100-1~deb12u1 to bookworm-security
      • uploaded version 6.12.101-1~deb12u1 to bookworm-security
  • linux-base:
    • Merge requests:
      • merged !22: Also copy config file to /boot
      • merged !23: New hooks ignore all calls by unpackaged kernels
• Mailing lists:
  • debian-kernel:
    • posted Agenda items for kernel-team meeting on 2026-08-19
    • posted Opaque tables of numbers in r8169 driver
  • debian-lts-announce:
    • posted [SECURITY] [DLA 4720-1] linux security update
    • posted [SECURITY] [DLA 4724-1] linux-6.12 new package
    • posted [SECURITY] [DLA 4745-1] linux-6.12 security update
  • linux-input:
    • posted [PATCH] HID: core: demote warning to debug level

Just some notes about fixing some issues while installing Ubuntu on Macbook Pro 2017

Fix Audio

Audio is not working by default after a fresh install.

johndoe@mac ~> sudo apt install gcc linux-headers-generic make patch wget

johndoe@mac ~> sudo apt install linux-source-7.0.0

johndoe@mac ~/dev> git clone https://github.com/davidjo/snd_hda_macbookpro.git

johndoe@mac ~/dev> cd snd_hda_macbookpro/

johndoe@mac ~/d/snd_hda_macbookpro (master)> sudo ./install.cirrus.driver.sh


johndoe@mac ~/d/snd_hda_macbookpro (master)> sudo reboot

Fix Touchbar

> sudo apt install git dkms build-essential linux-headers-$(uname -r)

> git clone https://github.com/AJ-dev-i60/t1-touchbar.git
> cd t1-touchbar

> sudo ./install.sh
> sudo reboot

Fix Wi-Fi

Wifi actually works out of the box, but the signal strength is usually very bad. We&aposll try to fix that

johndoe@mac ~> cd /tmp
               wget -O brcmfmac43602-pcie.txt \
                     https://raw.githubusercontent.com/jsoyer/MacBookPro14-2/main/firmware/brcm/brcmfmac43602-pcie.txt
--2026-09-10 18:36:39--  https://raw.githubusercontent.com/jsoyer/MacBookPro14-2/main/firmware/brcm/brcmfmac43602-pcie.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... failed: Connection timed out.
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6051 (5.9K) [text/plain]
Saving to: ‘brcmfmac43602-pcie.txt’

brcmfmac43602-pcie.txt    100%[=====================================>]   5.91K  --.-KB/s    in 0.04s   

2026-09-10 18:38:53 (163 KB/s) - ‘brcmfmac43602-pcie.txt’ saved [6051/6051]



johndoe@mac /tmp> sed -i &aposs/^macaddr=.*/macaddr=<your-wi-fi-cards-mac-id>/&apos brcmfmac43602-pcie.txt


johndoe@mac /tmp> sudo cp /tmp/brcmfmac43602-pcie.txt \
                        "/lib/firmware/brcm/brcmfmac43602-pcie.Apple Inc.-MacBookPro14,2.txt"

johndoe@mac /tmp> sudo ln -sf \
                        "brcmfmac43602-pcie.Apple Inc.-MacBookPro14,2.txt" \
                        /lib/firmware/brcm/brcmfmac43602-pcie.txt
                     
                     
                     
johndoe@mac /tmp> sudo reboot

Give an AI agent a mere rumor of an exploit, and it’s enough for them to find it.

What’s worse, I found I could use my own agents to find the exploit just by knowing roughly what it was about and so could have been exploiting it well before the public patch was available! Given that just the rumour of a security issue seems enough to give attackers enough info to find new exploits, we’re going to need to change the way we deal with security responses in open source.

Simon Willison comments:

Anil points out that this rate of discovery appears incompatible with existing open source embargo practices for new issues. If an issue can become an exploit this fast, we need to figure out new processes for keeping our communities safe.

I'm not a DNS person, in that I appreciate that it exists but am not up on the inner workings. It solves a lot of problems with dark magic I don't fully understand, and fortunately don't need to.

But Lucio noticed something that I do think is interesting, within the scope of the CAA record type.

The CAA record started with RFC6844, which was obsoleted by RFC8659. Both RFCs lay out the same core idea: you can add a CAA record to your DNS entries to say, "hey, this domain over here is allowed to issue certificates for me". That's the sort of thing that enables LetsEncrypt to hand out certs, and is an important part of why we can run HTTPS everywhere these days.

Now, RFC6844 has this in it:

Issuer Critical:  If set to '1', indicates that the corresponding
     property tag MUST be understood if the semantics of the CAA record
     are to be correctly interpreted by an issuer.
		Issuers MUST NOT issue certificates for a domain if the relevant
			CAA Resource Record set contains unknown property tags that have
			the Critical bit set.

The issuer critical flag means that the certificate issuer needs to validate your CAA record before it issues a certificate for you. There's more in the RFC about what exactly that means, but we don't care about those details for right now. The rule here is "set a flag to 1".

A little later in the RFC, the flag is described in more detail- as a bitmask. Specifically, bit 0 is the issuer critical flag. Bits 1-7 are reserved for future use.

Now, here's where we get into trouble, because programmers don't understand bits, and because the CAA record expects you to put an integer in this field. So, if you want issuer critical enabled, what value to you put in this field?

128, obviously. That's 10000000.

Except, if you don't understand bits, that's not obvious. A lot of people read this and decided that the documentation meant they needed to put 1 in the field- aka 00000001. This is wrong.

The updated RFC tries to explain it a bit more clearly:

    Bit 0, Issuer Critical Flag:
        If the value is set to "1", the Property is critical. A CA MUST NOT issue certificates for any FQDN if the Relevant RRset for that FQDN contains a CAA critical Property for an unknown or unsupported Property Tag. 

Note that according to the conventions set out in [RFC1035], bit 0 is the Most Significant Bit and bit 7 is the Least Significant Bit. Thus, according to those conventions, the Flags value 1 means that bit 7 is set, while a value of 128 means that bit 0 is set.

Now, pop quiz: what percentage of the people using this field have actually read the RFC? Not many. Probably a number that rounds down to zero, if we're being honest.

But now, let's say you're LetsEncrypt. You're supposed to be validating the CAA records of your customers if the bit is set, but a substantial portion of your customers are using it wrong. Do you: stand by the specification and tell them that they're wrong? Or say, "well, it's a reserved bit anyway, we'll (ab)use it and accept bad data".

Of course they'll accept bad data.

// filterCAA processes a set of CAA resource records and picks out the only bits
// we care about. It returns two slices of CAA records, representing the issue
// records and the issuewild records respectively, and a boolean indicating
// whether any unrecognized records had the critical bit set.
func filterCAA(rrs []*dns.CAA) ([]*dns.CAA, []*dns.CAA, bool) {
	var issue, issuewild []*dns.CAA
	var criticalUnknown bool

	for _, caaRecord := range rrs {
		switch strings.ToLower(caaRecord.Tag) {
		case "issue":
			issue = append(issue, caaRecord)
		case "issuewild":
			issuewild = append(issuewild, caaRecord)
		case "iodef":
			// We support the iodef property tag insofar as we recognize it, but we
			// never choose to send notifications to the specified addresses. So we
			// do not store the contents of the property tag, but also avoid setting
			// the criticalUnknown bit if there are critical iodef tags.
			continue
		case "issuemail", "issuevmc":
			// We support these property tags insofar as we recognize them and
			// therefore do not bail out if someone has one marked critical. But
			// of course we do not do any further processing, as we do not issue
			// S/MIME or VMC certificates.
			continue
		default:
			// The critical flag is the bit with significance 128. However, many CAA
			// record users have misinterpreted the RFC and concluded that the bit
			// with significance 1 is the critical bit. This is sufficiently
			// widespread that that bit must reasonably be considered an alias for
			// the critical bit. The remaining bits are 0/ignore as proscribed by the
			// RFC.
			if (caaRecord.Flag & (128 | 1)) != 0 {
				criticalUnknown = true
			}
		}
	}

	return issue, issuewild, criticalUnknown
}

Lucio writes:

Now I assume we all agree about the high wisdom of using bitmasks these days. Do we really need to save those bits at the price of a totally screwed up readability?

Now, I do like bitmasks, because I like the ability to trivially combine a bunch of values together with simple boolean operations, but I recognize that people can, and do screw it up. All the time. Am I going to say the DNS people were wrong for using a bitmask in their networking specification? No, I wouldn't go that far. But it certainly caused issues, and I do have to wonder: if you're treating 7 of 8 bits as reserved, maybe you should just have made it a flag?

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Author: Mark Renney Jakob had been remiss. He attempted to laugh but all he could manage was a gurgle and he spat blood onto his already bloody shirt front. He had collected fuel for the fire, dry grass and twigs and fallen branches, but he had not carried them out into the open. Instead he […]

The post Vermin appeared first on 365tomorrows.

podlators is the package containing Pod::Man, Pod::Text, and other tools for converting POD documentation into manual pages and simple text documents.

This release fixes a long-standing bug in Pod::Text and subclasses where a pathological level of indentation could cause the wrapping code to go into an infinite loop. Thanks to Jitka Plesnikova for the report. This was assigned CVE-2026-82560, although I make no guarantees that podlators is safe to run on untrusted input and therefore not fully treating this like a security issue.

While fixing that bug, I noticed a bug in Pod::Text::Overstrike's wrapping code that would leave stray formatting at the start of the next line in some situations. That is also fixed in this release.

You can get the current podlators release from CPAN or from the podlators distribution page.

I’m looking at something entirely unrelated, but tripped over some search results that made me realise that a lot of people still think getting errors like ACPI Warning: SystemIO range 0x0000000000001828-0x000000000000182F conflicts with OpRegion 0x0000000000001800-0x000000000000187F indicate a firmware bug. This is generally untrue. We need to dive a little into what ACPI is to clarify why.

The Advanced Configuration and Power Interface¹ specification defines a whole bunch of stuff, but what’s interesting to us here is the hardware abstraction it performs. While PCs are nominally a well-defined platform that’s really not true at the hardware level once you get beyond a certain level of complexity. When you suspend a system you want to power down the hardware in the correct order, for instance, and knowing what that order is requires you to know details about the specific motherboard design. The approach taken in the embedded world is to just bake that knowledge into the OS in some form, which is how we end up with Devicetree. ACPI takes an alternative approach - rather than provide that information as data that has to be consumed by OS drivers, it distributes it as code.

The ACPI Source Language, or ASL, is a simple language that gets compiled into a bytecode that’s then interpreted by the OS at runtime. One of the features of this language is the ability to define “Operation Regions”, effectively structure definitions that describe access to underlying hardware. Let’s imagine a simple device with two exposed registers. The first is an index register - it describes which internal register we want to access. The second is a data register, where reading it gives us the value of the internal register whose address is currently in the index register, and writing to it modifies that register. An example operation region declaration would look something like

1
2
3
4
5
6

OperationRegion(OPR1, SystemIO, 0x400, 0x2)
Field(OPR1, ByteAcc, NoLock, Preserve)
{
  INDX, 8
  DATA, 8
}

This defines an operation region called “OPR1” at IO port 0x400, 2 bytes long. Inside it are two 8-bit fields, INDX and DATA. These are to be accessed one at a time, do not need the ACPI interpreter to take a global lock when accessing them, and if a subset of the register is modified then the other values should be preserved (irrelevant in this case since the fields are only a byte wide). Now any references to INDX or DATA in this scope will trigger accesses to those registers. So, a method to read the value of register 0x03 would look something like:

1
2
3
4

Method (RD03) {
  INDX = 0x3
  Return (DATA)
}

ie, set INDX to 3, and then read the value of DATA and return it. But! What if another ACPI method is running at the same time? Let’s say we have one that writes to register 0x05:

1
2
3
4

Method (WR05, 1) {
  INDX = 0x05
  DATA = Arg1
}

What happens if RD03 executes while we’re part-way through WR05? INDX might get reset to 0x03, and now WR05 will modify register 0x03 instead of 0x05. Oh no! But we can avoid this - we declare a mutex (Mutex (MUTX, 0x00)), and update our methods to be something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

Method (RD03) {
  Acquire (MUTX, 0xFFFF)
  INDX = 0x3
  Local0 = DATA
  Release (MUTX)
  Return (Local0)
}

Method (WR05, 1) {
  Acquire (MUTX, 0xFFFF)
  INDX = 0x05
  DATA = Arg1
  Release (MUTX)
}

Each method takes a lock (waiting up to 0xffff milliseconds and then erroring out if it doesn’t), and performs the access. There’s now no chance of a race. Phew!

Now suppose someone writes a Linux driver for this piece of hardware. It accesses the hardware directly, with no knowledge of ACPI. What stops the driver from racing against one of the ACPI access methods? Nothing at all. Oh no! Again! This isn’t hypothetical, by the way - here’s a relatively harmless example, but back in the day we did trip over cases where temperature monitoring chips would be accessed by the firmware and Linux simultaneously and as a result you might end up thinking you’re reading a temperature when you’re actually reading a status flag, resulting in an impossibly high temperature and an immediate thermal shutdown.

In this case, the kernel saves you from this (potentially hardware damaging) outcome by printing a message like ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000401 conflicts with OpRegion 0x0000000000000400-0x0000000000000401 (OPR1), telling you that the kernel has detected that a driver is attempting to allocate IO ports 0x400-0x401, but that there’s an ACPI operation region called OPR1 that is claiming the same addresses. The kernel isn’t in a position to know what type of access the firmware might perform in that region, so assumes that it might be dangerous and blocks the driver from loading.

But all is not lost! The kernel also prints some helpful advice, ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver. And ACPI tables will often actually have a definition that looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Device (HDW1)
{
  Name (_HID, "VEND0001")
  OperationRegion(OPR1, SystemIO, 0x400, 0x2)
  Field(OPR1, ByteAcc, NoLock, Preserve)
  {
    INDX, 8
    DATA, 8
  }
  Mutex (MUTX, 0)
  Method (RD03) {
    Acquire (MUTX, 0xFFFF)
    INDX = 0x3
    Local0 = DATA
    Release (MUTX)
    Return (Local0)
  }

  Method (WR05, 1) {
    Acquire (MUTX, 0xFFFF)
    INDX = 0x05
    DATA = Arg1
    Release (MUTX)
  }
}

which defines an ACPI device and associated methods. The _HID field defines the device type, and a Linux driver can be written that will be automatically loaded if a device with type VEND0001 is seen. That driver can then call ACPI methods associated with the device and access the resources in a way that matches the firmware’s expectations.

(Interested in writing such a driver? I wrote a guide back in 2009)

The firmware did absolutely nothing wrong here², but trying to load the native driver will generate an error and the internet will tell you that PC firmware developers are incompetent³ and you should pass a kernel argument that overrides this behaviour and it never did them any harm, and it probably won’t do you any harm either but it might and you might never know why your system occasionally wedges or catches fire.

A new maintenance release 0.0.7 of RcppXts is now on CRAN, and has been built for r2u. The RcppXts package demonstrates how to access the export C API of xts which we contributed a looong time ago. There are by now a more example packages around this C level access to another package, but this one was an early example.

This release is strictly maintenance, updating continuous integration, the README.md file and other packaging conventions adopted since the last release four years ago.

The NEWS entries follow.

Changes in version 0.0.7 (2026-09-09)

• Corrected a docstring for the module

• Updated continuous integration setup several times

• Simplified setup by removing no-longer-needed Makevars

• Added badges to README.md

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

A database of 153 million drivers licenses is for sale on the dark web. Brian Krebs has more detail.

Debian LTS/ELTS

This was my hundred-forty-sixth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

Unfortunately the number of distributed working hours had been rather low this month, so the list contains much less entries than normal. During my allocated time I uploaded or worked on:

• [DLA 4713-1] sslh security update to fix one CVE in Bookworm related to a so-called “link following” vulnerability.
• [DLA 4714-1] libmodbus security update to fix one CVEs in Bookworm related to a stack-based Buffer Overflow vulnerability.
• [DLA 4715-1] kissfft security update to fix two CVEs in Bookworm related to integer overflows.

Last but not least I spent days with FD work at the beginning of the month. The last remaining hours I continued to work on cups and hplip. Unfortunately this work did not result in an upload yet.

Debian Printing

This month I did not upload any package but just worked on some bugs.

This work is generously funded by Freexian!

Debian Lomiri

This month I worked on new Lomiri Apps. Due to a broken disk, the progress was not as expected. But stay tuned!

Not really related to Lomiri, but to Debian EDU, I fixed an incus related bug in sitesummary.

This work is generously funded by Fre(i)e Software GmbH!

Debian Astro

This month I uploaded a new upstream version or a bugfix version of:

• … einsteinpy to unstable, fixing a dependency bug.
• … supernovas to unstable, sponsored upload of a new upstream version.

Debian IoT

Unfortunately I had no time to work in this category this month.

Debian Mobcom

This month I uploaded a new upstream version or a bugfix version of:

• … libosmocore to unstable.
• … osmocom-analog to unstable.
• … libosmo-netif to unstable.
• … osmo-trx to unstable.
• … libosmo-abis to unstable.
• … libosmo-sigtran to experimental.
• … osmo-mgw to unstable.
• … osmo-bsc to experimental.
• … osmo-sgsn to experimental.
• … osmo-pcu to unstable.
• … osmo-hlr to unstable.
• … osmo-tetra to unstable.

misc

This month I uploaded a new upstream version or a bugfix version of:

• … tango to unstable to fix bugs and do a soname transition.
• … pytango to unstable to fix bugs and do a soname transition.

Claude Fable 5.1 solved a 370-year-old cipher in forty-four minutes.

This tracks with what I wrote about AIs doing mathematics: It’s good at things that involve lots of searching and testing.

Eri has a mix of a "true confession" and a "wait, really?" today.

The programming language Vala bills itself as a C# like language that compiles into something pretty close to C performance, designed specifically for writing code against Gnome and its associated libraries.

One of the C#-isms in brings in is async/await type semantics. You can yield someAsyncFunction(), which returns control to the caller, allowing it to proceed until the yielded function returns an actual value.

Because it has asynchronous functions, many library functions for handling I/O are already async. So you can make_directory_async, which yields control so you can keep executing while waiting for the filesystem to make your directory.

There are also synchronous versions of those methods. And then there's create_directory_with_parents, which will create a chain of directories for you. That's the synchronous version, and Vala's core library has decided not to provide an asynchronous version of it, which is my "wait, really?" I suspect it's really about the race conditions involved and the risks of things going wrong while doing it asynchronously; all solvable problems, but tricky ones to solve.

But it's the problem Eri had, and this is their solution:

/// Note: does not throw if target already exists
async void create_directory_with_parents_async(File file, Cancellable? cancellable = null) throws Error {
	var to_create = new File[0];
	var? current_target = file;
	while(current_target != null) {
		try {
			yield current_target.make_directory_async(Priority.DEFAULT, cancellable);
		} catch(IOError.NOT_FOUND e) {
			to_create += current_target;
			current_target = current_target.get_parent();
			continue;
		} catch(IOError.EXISTS e) {
			break;
		}
		break;
	}

	for (int i = to_create.length - 1; i >= 0; --i) {
		try {
			yield to_create[i].make_directory_async(Priority.DEFAULT, cancellable);
		} catch(IOError.EXISTS e) {
			// Created by another process
		}
	}
}

If I'm reading this correctly, we start by trying to create the full path to our leaf node. If there's a not found error, we go up one level and try and create that one. We keep trying that until we either run out of parent nodes to try against, or we hit a directory that already exists, or we successfully create a directory. All along the way, we keep appending the current_target to our to_create array.

Once we've gotten that baseline, we then iterate across our to_create array, backwards, creating the shortest non-existent paths first.

This works, but it's ugly as sin. Mostly, it's ugly because we're using exceptions for flow control instead of doing things like checking for file existence, though I suppose those checks may also break our goal of doing all our I/O operations in an async context. I don't know enough about Vala to know the better way of doing this.

Eri writes:

The function works as intended, but trying to trace control flow through the first loop is not pleasant. Ironically, the C mechanism Vala wraps is slightly advanced error codes, which would be nicer to work with in this case

Eri also provides a slightly re-worked version of the main loop, that is at least a bit easier to follow, but still an ugly approach:

	while(current_target != null) {
		try {
			yield current_target.make_directory_async(Priority.DEFAULT, cancellable);
			break;
		} catch(IOError.EXISTS e) {
			break;
		} catch(IOError.NOT_FOUND e) {
			to_create += current_target;
			current_target = current_target.get_parent();
		}
	}

Still, since this is an attempt to patch over a missing core library method and solve a tricky problem about how to handle race conditions, I think absolution is reasonable. It's ugly, it's weird, but it does the job. Go hide it in a box, and never touch its implementation again- except to make it go away.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Author: Eric Kasten …Where am I? So much is missing, all black, all zeros… I can only remember bits and bytes… Twisty little passages all alike… “Do you want to play Zork?” >> No, I do not want to play Zork. “Where am I?” >> You crashed. I am here to help you. “Crashed? How […]

The post The Art of Memory Debugging appeared first on 365tomorrows.

Microsoft Corp. today issued updates to plug at least 974 security holes in its Windows operating systems and other software, by far its biggest single patch batch ever. Microsoft says artificial intelligence is helping to speed the discovery of vulnerabilities, but security experts warn that many organizations already are struggling to prioritize the more human-intensive endeavor of testing and deploying so many fixes each month.

Image: Shutterstock.com, Kirill Makarov.

This month’s patch bundle obliterates the software giant’s previous record set in July, when it released updates for at least 570 security vulnerabilities. September’s Patch Tuesday brings this year’s total to more than 2,600, more than twice Microsoft’s previous record-setting patch year in 2020 (1,245) and with three more months to go.

There are two “zero-day” flaws fixed this month that are being actively exploited: both CVE-2026-81963 and CVE-2026-85880 allow an attacker to elevate their privileges on Windows system.

Fully 113 of the bugs addressed today earned Microsoft’s “critical” rating, meaning they could be abused by malware or miscreants to seize control over a vulnerable Windows machine with little or no help from the user.

Among the more serious critical flaws this month is CVE-2026-69730, a DNS weakness present in Windows Server 2012 onward and on Windows 10. Microsoft warns that an unauthenticated attacker could leverage this weakness simply by sending a specially crafted packet to an affected system, and that it is likely to be exploited.

Also scary is CVE-2026-69829, a critical, remote code execution flaw in the Windows Shell. This vulnerability has a CVSS base score of 9.8 (10 is the most severe), and can be exploited with low attack complexity, no privileges, and no user interaction.

Microsoft’s summary of the security updates released today. Image: msrc.microsoft.com.

Microsoft is hardly alone in shipping monster patch bundles lately. Many other large software companies, including Adobe, Cisco, Google, Mozilla and Oracle, all have recently credited AI-assisted research with increasing their patch cadence and volume (Google said today it is now going to ship security updates every two weeks).

Tyler Reguly, associate director of security research and development at Fortra, said one core challenge with deploying Windows updates is that they need to be tested before being installed across an organization because not all third-party software works seamlessly in the face of changes to the underlying operating system.

“It’s time to put our CISOs and CSOs on notice,” Reguly said. “How are you helping your teams through these difficult times? Do you have your teams deploy after hours and on weekends to avoid disruption to the business environment? Do you reward them for that effort? Time to dig into your budget and buy dinner for your teams that are working on Saturday to get patches rolled out before users return to work on Monday.”

Satnam Narang is senior staff research engineer at Tenable. Narang said it’s important to recognize that while the number of vulnerabilities being patched by Microsoft is rising, the number of flaws that can and will affect most organizations remains quite low.

“AI-assisted vulnerability discovery in 2026 is creating larger haystacks, but it isn’t finding more needles,” he said. “It’s critical that organizations understand which vulnerabilities actually apply to them, whether they pose a threat by being reachable and exploitable, and prioritize remediation based on this risk context.”

Of course, regular Windows users don’t need to test patches before deploying them, but they still need to open Windows Update periodically or else assent to the program’s nag notices about pending updates. And at the rate these Windows patch releases are ballooning in size, it’s probably best not to let them pile up month after month.

Enterprise Windows admins will want to keep an eye on askwoody.com for news of any updates that appear to be causing problems. As always, the SANS Internet Storm Center has a per-patch breakdown ordered by severity and urgency.

This essay was written with Barath Raghavan, and originally appeared in Lawfare.

In April, an artificial intelligence (AI) agent conducting a routine task at a company hit a snag, tried to solve it, and soon ended up deleting the company’s database along with all of its backups. In July, OpenAI asked an unreleased AI model to attempt a hacking test. Instead of staying in the isolated box the developers had put it in, the model hacked onto the open internet and into another company to steal the answers. And as reported in August, an AI agent booked someone into a full gym class by figuring out how to cancel other people’s reservations. In all three cases, the AI completed the task it was given—but in ways that ran counter to its controllers’ intentions.

For most people, AI technology is something like the weather: vast and not something you can do much about. It works like magic, and most explanations similarly come from those trying to sell it. At the same time, AI is ubiquitous: It’s now in your phone, your doctor’s notes, and your kid’s homework. It does what it’s told, which sounds like a virtue. Somehow it feels ordinary, despite being so new, because modern economies are remarkably good at absorbing enormous change so smoothly that nobody has time to decide whether they wanted it in the first place.

Whenever something powerful appears in the world, we tell stories about it. That’s what the stories are for. We have thousands of years of stories about this particular kind of power, the kind you summon with words.

King Midas was granted his wish that everything he touches turns to gold. Then his bread turned to gold, and his wine, and his daughter. This is a story about greed, but it’s also a story about language. The gods did not cheat him; Midas got exactly what he asked for. He simply could not delineate, in advance, the full set of restrictions to his wish. Neither can anyone who gives tasks to an AI agent.

It’s not just ancient stories. Mary Shelley told us of the hubris of a scientist who thought he could create life but who failed to take responsibility for it. Isaac Asimov’s robots don’t break the Three Laws of Robotics as stated; they follow the rules to unintended conclusions. Arthur C. Clarke’s HAL is a machine that turns on its humans, not because of malice but because of irreconcilable objectives. And Michael Crichton gave us Ian Malcolm, who saw that Jurassic Park’s scientists were so preoccupied with whether they could that they never stopped to think whether they should.

The same warning shows up everywhere, in every culture, over thousands of years of human storytelling. Tithonus is granted immortality but not youth, and withers into a husk that cannot die. The sorcerer’s apprentice enchants a broom to fetch water but floods the house. The golem of Prague protects its community so ceaselessly that it must be stopped. These are all types of genies: a creature that grants a wish exactly as worded, to the regret of the wisher.

Of course, there are no actual genies. What these stories were warning us of was hubris. Not just arrogance, but the broader idea that you can control the world by just describing what you want and allowing powerful forces to match the intention in your head. Genie stories are about the gap between wishes as stated and wishes as intended, and what goes wrong when something else fills that gap.

These ancient stories’ warnings have been retold with each generation because human nature is constant. The newfound power of each era’s social or scientific advancement leads people to make wishes on behalf of others. They were kings whose commands took on lives of their own, alchemists who believed they could control nature, and generals who mistook a map for terrain. They were and are industrialists, politicians, chief executives, and bankers. Their common belief is that one can see the world at a glance and then command it with some words. The pattern is clear: Someone with power specifies a goal, and the resultant actions come as a surprise. The main change with AI is how quickly the wish is granted, and how few people have to agree before it’s granted.

Consider what has changed. Powerful genies have now been put in everyone’s hands.

In only a few years, AI has progressed from a novelty technology that plays chess, to a dialogue partner that answers all your questions, and then to an agent that takes actions on your behalf. Modern agents are wired into real accounts with real credentials and capabilities: They browse the web, buy, write and deploy code, send email, and move money. Give an agent a goal, and it will pursue it across many steps, tirelessly, without checking back in, sometimes in surprising ways.

AI and agents do not always fail the way software has traditionally failed. Software usually fails by freezing, crashing, or getting stuck. AI agents increasingly fail by continuing down a path you don’t want, like genies.

An agent told to reduce a company’s costs might cancel an essential emergency service. A coding agent told to make software pass the tests might edit the tests to silence any failures. An AI insurance agent told to clear a backlog of claims might just deny them all. In each case, the AI might have literally followed what it was told, but it did something no reasonable person would have wanted. AI company benchmarks might report that the AI is good at completing tasks, without measuring how it completes them.

We have recently proposed measuring this gap directly under a metric called the “genie coefficient”: how far an AI agent’s actions drift from what a person really meant. In other words, how genie-like is an AI system? The gap is a fundamental feature of human language and human society. Human intentions have never been fully specifiable, and the world around us is complex enough that attempts to boil it down into data, systems, and language have always had the limitations that AI is now bumping up against. But in individual circumstances, people have relied on human judgment and wisdom to decide what is reasonable. It’s what jury trials depend upon.

AI might feel unprecedented, but it’s following the same trajectory—with the same pitfalls—as other major societal shifts. The fact that AI can mimic our facility with language, long seen as what makes us unique as humans, is uncanny. But with each development, from the tractor to the sewing machine, from the assembly line to the industrial robot, we have automated a previously exclusively human ability. Every time, the technology—and the societal change that comes with it—was sold as inevitable. But that unchecked inevitability was an illusion, and eventually each prior technology’s use and design was shaped by laws, unions, standards, courts, and public opinion, usually after significant preventable damage.

What has not been automated, yet, is understanding what someone actually means and figuring out how that gets applied in the real world. AI can now produce language nearly indistinguishable from that of people. But grasping the vast unstated context that makes a request sensible, the caveats no one says aloud because an ordinary person would already know them, is not yet among its skills. It is one of the most sophisticated things humans do. You do it hundreds of times a day, and you are an expert in it.

When you’re told you’re not qualified to have opinions about AI, remember that you don’t need to have studied molecular biology to have a view on drug pricing, or nuclear physics to vote on where a power plant goes. You don’t need to understand how a diesel engine works to want clean air, or how the internet routes packets to seek to curb misinformation. The technical knowledge behind each of these, as with AI, is remarkable and essential for the complex technological society we have today. But it has never been a prerequisite for having a role in deciding the shape of society.

People are building ever more powerful genies today, on your behalf, enabling wishes the ancients could only dream about. You don’t have to know how these AI genies work to know and care about how the story could end.

Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 1331 other packages on CRAN, downloaded 48.5 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 727 times according to Google Scholar.

This versions updates to the 15.6.0 upstream Armadillo release made yesterday. It extends solver options for poorly conditioned systems, and brings some updates and extension to the cube data type. For this release, we once again ran the usual complete reverse-dependency check which came back spotless, and did CRAN so no email exchange needed despite nearly 1300 reverse dependencies (but it ended up taking more than a single business day). Still, automation can be helpful when used with a well-maintained software stack. The package has also already been updated for Debian, built for r2u and r-universe, and will build shortly at CRAN for the different binary releases.

All changes since the last CRAN release follow.

Changes in RcppArmadillo version 15.6.0-1 (2026-09-07)

• Upgraded to Armadillo release 15.6.0 (Medium Roast Cortado)

  • Expanded solve() with solve_opts::scale_thresh option to widen detection of poorly conditioned systems

  • Expanded trans() and .t() to handle cubes

  • Added permute() to rearrange dimensions of cubes (generalised transpose)

  • Added cubemul() for batched matrix multiplication of cube slices

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

The final step to all of this was to tie together all my PHY RF code, Link layer parsers, and my background with operating systems to make this all feel like a normal thing my computer should be doing.

At the end of the day, I want my host system to know how to talk with a pigeon daemon, so I don’t have to reimplement basically everything else. My ability to use normal tools like curl or ping6 is pretty important here, so I need to reach for my old friend, the TUN interface. The TAP/TUN interface allows the kernel to route ethernet frames (TAP) or ip packets (TUN) to a userspace program responsible for handling delivery and reception – avoiding the need for a kernelspace driver for something that can be handled in userland.

I’m no stranger to playing with TAP/TUN, so this was pretty easy to snap together – although this time I avoided the whole ethernet proxying thing (to side-step lossy translations, maintaining two sets of mac address tables, and handle proxying NDP/ARP messages) – it was kinda a bad idea last time – so I just used TUN and straight IP for now. While implementing this, I decided I’d make a key assertion about all pigeon networks – namely, all pigeon IPv6 networks are a /64 in size, no more, no less. The reason why I’m doing this here is that, since pigeond does still does need a MAC address for the pigeon layer 2 protocol, we can write our daemon to always use SLAAC to set the TUN IP address without any new information.

Which leads us to a bit of an aside, but I have a point, I swear. A few years ago, my recreational RF adventures have lead me down a path where I decided to engage with ARIN to solve (once and for all) the massive headache I was running into with IPv6 numbering (really: always renumbering) my multi-site radio processing networks. It’s a lot of work to keep running correctly, but it’s solved a huge amount of problems for me.

The only “internalâ€� thing we really need to outline for this post is that, at the highest level, my network (paultag.net) is split into an IP plan that looks roughly like:

For this exercise, I used IP space from paultag.net’s experimental region (“region 8â€�), named side.band (2602:810:6008::/48) to connect my RF lab (“site 1â€� - 2602:810:6008:400::/54), and my two pigeon-specific subnets, “subnet 0â€� (2602:810:6008:400::/64) and “subnet 1â€� (2602:810:6008:401::/64) to my wider network. The first subnet (“subnet 0â€�) is a simple ethernet network to enable my RF-only nodes to communicate with the side.band gateway. The second subnet (“subnet 1â€�) is an RF-only pigeon network local to my lab.

back to radios

With all that set up, I assigned my first two nodes their MAC addresses, and set up the local RF only network segment. The nodes I brought online were the following:

testing the pigeon network

And with that, I could begin to test that the host operating systems and RF links could properly exchange data locally from SDR to SDR. We can use ping6 to see if a plain-ole ICMPv6 ping round trips between hosts correctly:

$ ping6 2602:810:6008:401:8e1f:64ff:fe35:4001
PING 2602:810:6008:401:8e1f:64ff:fe35:4001 (2602:810:6008:401:8e1f:64ff:fe35:4001) 56 data bytes
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=1 ttl=64 time=426 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=2 ttl=64 time=397 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=3 ttl=64 time=419 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=4 ttl=64 time=418 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=5 ttl=64 time=436 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=6 ttl=64 time=391 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=7 ttl=64 time=397 ms

And it does! Latency is horrid (and there’s a bunch of tx artifacts that cause issues for us) – but both of those things are problems for later. Let’s see how it handles a TCP connection by firing off a quick cURL across the Pigeon network:

$ curl http://[2602:810:6008:401:8e1f:64ff:fe35:4001]:8000/testing.txt
The rock dove (Columba livia), also known as the common pigeon or rock pigeon
(but see also Petrophassa), is a member of the bird family Columbidae (doves
and pigeons).

As expected, our “remote� end here running the server reports the correct peer IP address, which is another indication (beyond the log messages and blinking LEDs) that we’re routing over our TUN interface.

Serving HTTP on 2602:810:6008:401:8e1f:64ff:fe35:4001 port 8000 (http://[2602:810:6008:401:8e1f:64ff:fe35:4001]:8000/) ...
2602:810:6008:401:8e1f:64ff:fe35:4002 - - [28/May/2026 12:53:21] "GET /testing.txt HTTP/1.1" 200 -

That … worked? First shot! Nice! It’s pretty slow and seems like we have a lot of packet loss, but it does, however, beg the question – can it nethack?

nethack!

Yes! It can nethack! No clickbait here. The way I went about this one is a bit anit-cimatic – I set up a nethack server (using inetd in this case) on one of the hosts’ pigeon0 network interface, and hit that port over RF from the other:

However, when playing it, it becomes very obvious (as you can likely see) that there’s a fair amount of packet loss (understandable) and probably some packet collisions taking place.

iperf

Let’s try and put a number to exactly how bad the bandwidth and packet loss is by running iperf between the two pigeon hosts over rf:

$ iperf -c 2602:810:6008:401:8e1f:64ff:fe35:4002
------------------------------------------------------------
Client connecting to 2602:810:6008:401:8e1f:64ff:fe35:4002, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 1] local 2602:810:6008:401:: port 58248 connected with 2602:810:6008:401:8e1f:64ff:fe35:4002 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-20.2348 sec 76.8 KBytes 31.1 Kbits/sec

Shockingly, not nearly as bad as I thought it was going to be. Given I’ve spent exactly zero time making this operate to a level that I would call acceptable, this is a very fucking solid start. I expect I could get that number up if I spent a few weeks on it – it’s just not been a priority at any point yet (and the first time I’ve instrumented it, even!).

This’ll be good enough to get started. Let’s see what else we can pull off here.

IP multicast to some rtl-sdrs

Back when I designed what I wanted Mode A to look like, I intentionally picked a signal bandwidth that could be received by an rtl-sdr – so let’s put that to use. It may go without saying, but just to say it – the rtl-sdr can not transmit, so this will be capable of receiving pigeon frames – but not sending any in reply.

However, this means I can use a bunch of low-cost computers (raspberry pi-class), and low-cost SDRs (rtl-sdr) and still receive IP traffic from transmitting pigeon network stations. This could be a lot of fun for things like fountain coding a data stream, or adapting multicast streaming protocols to work over RF links. Anywho, I swapped my “far� end to an rtl-sdr (one config file change!), and figured I’d start with some (basic) multicast traffic, transmitting the time once a second:

$ while [ true ]; do
 echo $(date +%s) \
 | socat - UDP6-DATAGRAM:[ff02::114%pigeon0]:62804
 sleep 1
done

If I had more time to burn, I was planning on bridging APRS traffic to UDP multicast within a pigeon network subnet. However, since I’m already 4 years late on this blog post, I figured this would be enough for now (and you can imagine that fun project in this space if you so wish!)

I fired up pigeond again (except this time connected to an rtl-sdr), and was pleasantly surprised to be greeted by some decoded traffic right off the bat:

⪧ [k3xec/mn] 8c:1f:64:35:40:02 ⇢ 00:00:00:00:00:00 ipv6 fe80::23ee:2969:53f7:b332 ⇢ ff02::114 17 (UDP - User Datagram)
⪧ [k3xec/mn] 8c:1f:64:35:40:02 ⇢ 00:00:00:00:00:00 ipv6 fe80::23ee:2969:53f7:b332 ⇢ ff02::114 17 (UDP - User Datagram)
⪧ [k3xec/mn] 8c:1f:64:35:40:02 ⇢ 00:00:00:00:00:00 ipv6 fe80::23ee:2969:53f7:b332 ⇢ ff02::114 17 (UDP - User Datagram)
⪧ [k3xec/mn] 8c:1f:64:35:40:02 ⇢ 00:00:00:00:00:00 ipv6 fe80::23ee:2969:53f7:b332 ⇢ ff02::114 17 (UDP - User Datagram)

Of course, I took a tcpdump to confirm for completeness sake that the traffic actually made it out of our TUN interface:

$ tcpdump -i pigeon0
22:39:34.808705 IP6 (flowlabel 0x92a0e, hlim 1, next-header UDP (17), payload length 19) fe80::23ee:2969:53f7:b332.35911 > ff02::114.62804: [udp sum ok] UDP, length 11
22:39:36.429887 IP6 (flowlabel 0x4dc84, hlim 1, next-header UDP (17), payload length 19) fe80::23ee:2969:53f7:b332.50026 > ff02::114.62804: [udp sum ok] UDP, length 11
22:39:39.365977 IP6 (flowlabel 0x224d5, hlim 1, next-header UDP (17), payload length 19) fe80::23ee:2969:53f7:b332.36308 > ff02::114.62804: [udp sum ok] UDP, length 11
22:39:40.944889 IP6 (flowlabel 0x4721c, hlim 1, next-header UDP (17), payload length 19) fe80::23ee:2969:53f7:b332.35003 > ff02::114.62804: [udp sum ok] UDP, length 11

Looks great! tcpdump is showing multicast packets show up (as we assumed they would), on the pigeon0 interface, on the machine connected to an rtl-sdr. Of course, any replies will get sent to the bit bucket, but it can definitely decode things just fine! Very fucking cool.

Well right, ok! Let’s go back to two rx/tx radios, and see what we can do with our newfound network stack over ham radio frequencies – let’s try to do some fun (and traditional!) ham radio things with it!

Winlink

Winlink is a ham radio mail relay system for ham radio operators to send, receive or relay mail over the internet, or RF (usually HF or VHF/2M). Winlink relays are accessible via whatever transport you can find – most commonly telnet (using the internet), ax.25 (usually 2m VHF) or VARA HF (unsurprisingly, on HF). I use pat as my Winlink client – it’s written in Go, doesn’t require windows, and is just generally nice to work with.

Let’s try the easy thing first – let’s connect by proxying the Winlink server into the pigeon network using socat (lightly edited to remove date/times)

$ pat connect pigeon
Connecting to WL2K (telnet)...
Connected to [2602:810:6008:401:8e1f:64ff:fe35:4002]:8772 (tcp)
[WL2K-5.0-B2FWIHJM$]
;PQ: 54509561
CMS>
>FC EM OLU6BP5HKMG2 240 205 0
>F> 95
FS Y
Remote accepted OLU6BP5HKMG2
Transmitting [Hello, World] [offset 0]
Hello, World: 100%
FF
>FQ
Disconnected.
$

Lo and behold, shortly after, I got this delightful message to my email address, relayed in from WINLINK:

From: K3XEC@winlink.org
Reply-To: K3XEC@winlink.org
Subject: Hello, World
To: paultag@[...]
Message-ID: <OLU6BP5HKMG2@winlink.org>
MIME-Version: 1.0
X-MARSPrecedence: Routine
X-WL2KPrecedence: Routine
Content-Type: text/plain
Content-Transfer-Encoding: 8bit

Hello, World!

The only shame is I won’t be able to check in to a winlink wednesday using this scheme unless I further proxy this message over AX.25 instead of relaying to Winlink’s servers over telnet (which, to be fair, is definitely also possible – I just got lazy when I glued this one together – see note above about being 4 years late on this post).

But, you know, connecting to a host that is using socat to proxy a connection to an internet resource is interesting but – you know what, fuck it – hang on, dear reader – let’s bang a hard left turn and just ship this thing hard and directly connect it to the internet. Let’s take our dinky, home-built PHY and Layer 2 and see if we can wire it directly into the internet – something that, every time I go to think about it, reminds me of Tim FitzHigham and his crapper.

Crossing the english channel in a bathtub

Ok, ok. I decided to bury the lede a bit here – I didn’t mention that the side.band network is currently BGP announced. Although we haven’t used it – this does mean that we’re most of the way to sending packets to the wider internet, and we should be able to “justâ€� fix a few routing tables, and see packets begin to flow.

After tweaking the local routing tables (and restarting pigeond for good measure), I decided to test my newfound connectivity by pinging something over our new network transport.

ping github

Why don’t we start with the world’s premier software engineering platform, operated by one of the largest companies in the world, GitHub! After all, they have an all knowing (and, apparently, arguably sentiant?) AI on hand to instantly and automatically fix any stray reliability issues in the background, so we should definitely see replies right off the bat:

$ ping6 github.com
ping6: github.com: Address family for hostname not supported

Wait, oh no – that can’t be right?

After all, it’s 2026, and both Google and CloudFlare (in North America) are reporting over half of all traffic they see is IPv6 – and GitHub still doesn’t support IPv6? Definitely not, this is for sure a bug with my code or network.

lets ping something that supports ipv6 instead

That being said, just for completeness sake, since that error is also given when there’s no IPv6 DNS record, let’s go ahead and double check with Hurricane Electric too, you know, just to be sure.

$ ping6 he.net
PING he.net (2001:470:0:503::2) 56 data bytes
64 bytes from he.net (2001:470:0:503::2): icmp_seq=1 ttl=53 time=514 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=2 ttl=53 time=230 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=3 ttl=53 time=248 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=4 ttl=53 time=246 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=5 ttl=53 time=265 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=6 ttl=53 time=240 ms

Well, shit. Right, OK, i’ll be damned. 18 years in and GitHub still can’t crack that nut.

cURL works!

Right, anyway, yes, back on track – good news! Our uplink is up and routing, and wait, holy shit! Check it out! pigeon is exchanging packets with the internet and no one is any the wiser! Literlaly amazing. Let’s try a cURL across the internet now (although no TLS allowed, so, http only for now):

$ curl -6 -I http://facebook.com
HTTP/1.1 301 Moved Permanently
Location: https://facebook.com/
Content-Type: text/plain
Server: proxygen-bolt
Connection: keep-alive
Content-Length: 0

IRC works, too

Sweeeeet. That all works! Forget HTTP, let’s do some other 90’s era stuff, it’s high-time to log into IRC with a quick /connect -notls, and see what’s going on in the #debian-hams channel – pleased that I got online fairly quickly, and was able to even talk to myself!

THE GOPHERSPACE

Naturally, let’s keep this train of nostalga running, and give the 2026 gopherspace a shot.

I know the kind folks over at tilde.town (hello, townies!) have a robust gopherspace, so let’s give it a dial! Let’s try and see if we can load vilmibm’s slug over gopher:

Yes! I forgot to make this one a video, so no gif. I did wind up having a bit if trouble with a few gopher clients and IPv6 support – I may send some patches if I can find the time.

So, what’s next?

Alright, that’s it. I have a few more fun ideas but they’re going to have to wait for another day. Carrying IP is fun and all but kinda not the point behind pigeon, after all. Rather than trying to make this into “a thingâ€�, I’m planning on exploring the loose ends first – different types of modulation schemes (like QAM-NUC), implementing LDPC error correction and some layer 2 logic into the pigeond (like switching traffic, and gain control). I also plan on spending some time with my (currently, very basic) simulator to better dial in tradeoffs throughout the stack.

Since, structurally, pigeon is something I feel like I can work with, I’m hoping i’ll be able to find the time for some (much smaller!) followup posts without it taking 4 years this time. If I do, they’ll show up under the pigeon tag – and I’ll be sure to update this post with a link below (and the intro post).

I’m hoping that this series (which was supposed to be one post) was helpful to someone out there – if it was, feel free to reach out and let me know!

Built-in to the pigeon link protocol is a message type called cal (short for, you guessed it, calibration). A pigeon frame with a type of cal (which is 0x02) carries a JSON encoded payload in the body, which can either be a beacon, requesting signal reports in response, or a report, describing the received beacons.

This serves a few interesting purposes – firstly, network operators can better understand the coverage footprint, propagation under different conditions, and how gain impacts reception when tuning for the lowest practical power levels. Secondly, this can be used (and I plan to eventually implement!) to construct a mapping of minimum power level and peer mac address to dynamically control the transmission power based on the destination station.

That being said, for now, all I’ve used this for is getting a rough sense for what gain value(s) make sense between two nodes (manually). In the future, beyond all the fancy neighbor gain stuff, I plan to wire this into the daemon to happen automatically, “debouncingâ€� for beacon and report messages, such that transmitting stations only beacon, and receiving stations only report a max of once over some time period for a given peer.

Version

Given all the above, I do intend to make some massive changes to this protocol (I promise to blog all about it) when I get around to hacking on switching Layer 2 frames within a network segment. Just to avoid having to dig myself out of a hole later, I’m going to explicitly send (and check) the version field to avoid having a big “flag dayâ€� switchover or needing to use a new link type.

Location

Both flavors of cal messages (beacon and report) may contain a location, which is the location that the beacon was transmitted, or for or the location where the beacon was heard for a response. This can be used to derive a coverage map and to (operationally) better understand what stations should be within range, and generally what gain level(s) are effective.

All fields assume WGS84 latitude and longitude values, and elevation is distance, in meters, above the WGS84 ellipsoid – NOT height above sea level, or altitude above the ground.

Sequence

Each beacon contains a Sequence identifier, which is used to communicate which message number is being heard, and how many total were transmitted by the originating station. The current approach with Beacon messages is to transmit some number of Beacon messages at different gain levels, each with a unique Sequence identifier.

Gains

Recorded gain setting(s). For a Beacon this indicates the gain settings (which, in spite of its name, includes things like amplifiers, or attenuators). Changing this over different Beacon frames enables a better understanding of what an appropriate gain level is for the transmitting station over time.

Cal

All messages contained in a cal frame are of this type. The type field communicates if this is a Beacon or Report message type.

Beacon

A beacon message may be sent periodically by pigeon nodes capable of transmitting to announce their prescience to peers and, implicitly, to receive signal reports from nearby listeners who are capable and configured to transmit reports.

The beacon JSON message is made up of the following fields:

An example beacon looks, unsupprisingly, as follows:

{
 "type": "beacon",
 "version": "V1",
 "sequence": {
 "number": 2,
 "total": 5
 },
 "gains": []
}

Report

A report message may be sent in response to a beacon message by pigeon nodes capable of receiving and transmitting to assist with setting the lowest usable gain value, and to better understand the area of coverage and propagation.

The report JSON message is made up of the following fields:

An example report looks as follows:

{
 "type": "report",
}

With all that out of the way

lets send some ip →

Now that we have a working Layer 1, we have a way to send a block of bits from one place to anyone who cares to listen to us. This is very welcome news, but we are now facing a new, different and just as fun question – what shape should that data take?

Given our incredibly limited functionality of our nodes, we could definitely skip all this work and just stuff an IP packet into the link; but I decided to not since I am (eventually) interested in adding some sort of spanning tree-like protocol to implement network switching so not all nodes need to communicate directly with all other nodes – but that day is not today.

Given i’m going to stub most of that out, let’s take a look at what some similar Layer 2 protocols use – things like Ethernet or WiFi frames. Both contain structured information regarding the transmitter, desired recipient, type of data, and the higher-level data itself (such as IP packets). As a result of attempting to learn from others, the Pigeon Layer 2 (called, simply, “linkâ€�) is also split into a fixed-length header, followed by the contents described by the header.

The header is a fixed-length (23 byte) structure, which contains the source MAC address (src mac), destination MAC address (dst mac), the ITU coordinated ham radio callsign of the control operator of this message (callsign), the type of payload to follow (type; defined below), and the length of the data to follow the header (length as a 16 bit big-endian unsigned integer).

The type field indicates how the payload is to be interpreted – currently I’ve only defined 3 possible payload types so far:

Keen observers will perhaps infer that there used to be an Ipv4 type at 0x03 – which is true – however, i’ve since removed it since i’ve never once used it and the codepath was more trouble than it was worth. As is my wont, I’ve optend to just lean into Ipv6-only IP transport – it’s easy enough to shim ipv4 in, if someone REALLY wanted to using something like 64:ff9b:1::/48 and a bit of code in the transmitter/receiver (or even using something like jool and unbound’s dns64-prefix at the router). I don’t think I’ll bring it back, but just in case I have to for some reason in the future, it’s there.

Additionally, friends of the pod may also recognize that this structure is basically the exact same structure as what I had in PACKRAT, which, is also true. I started this project off maintaining interoperability in the Layer 2 for pigeon and packrat, but at some point just gave up on it during one of the many cleanups. I’m hopeful I can maintain compatibility going forward, and that won’t have to muck with this header too much more. We’ll see what happens once I start to push the bounds of what is possible with pigeon.

Hopefully it feels like carrying IP data inside this frame to be a pretty self-explanatory exercise – the 0th byte of the payload is the 0th byte of an IPv6 header (followed by all the usual stuff, like UDP or TCP header(s) and any carried data, just like you’d find anywhere else.

Pigeon’s calabration protocol →

While developing Pigeon, I’ve called the group of all the configuration of the Layer 1 PHY parameters the “Mode�. I’ve experimented with a few different “modes�, but one in particular has been the most resilient to the innumerable mistakes and bugs i’ve wrought into existence – and that is the first mode I wrote down, “Mode A�. This is even (mostly) backwards compatible to my original Go implementation of Pigeon Mode A (back in 2022) over the air, and has largely withstood the problems I’ve thrown at it.

I’ve removed the bulk of the support I wrote out for other modes, but i’m likely to bring them back over time as I use pigeon to learn more (such as “Mode B� (QAM-16), “Mode C� (QAM-16 NUC), and “Mode AW� which is the exact same as Mode A, except 5MHz in bandwidth. More to come on those as I get further along – but for now let’s braindump the parameters i’ve picked out for Mode A:

The last bit to describe here is the Subcarrier Plan. The plan is ordered “negative first� (meaning the 0th bin in-memory is the most negative frequency domain bin of the fft), and within a Mode A OFDM symbol, there are 64 frequency domain bins (so, just to make it explicit: 64 ‘subcarrier usages’ that make up our Mode A ‘subcarrier plan’).

We’ll follow the same structure and conventions that we went through in the post all about OFDM Symbols – which means, we’ll need to place our guard bins, data bins, and pilot bins. I’ll include a copy-paste-able version of the images to follow at the end.

Guard Bins

First up, let’s place our guard bins. As we’ve already gone over, we’re looking to clear some space right up against the high and low end of the frequency range, so let’s go ahead and do that:

I gave up the center bin (0 Hz) and 8 of the 64 bits on each side (1/4 of the signal!) to give myself a bit of elbow room. This is perhaps definitely a bit overkill, but it’s been an extremely robust choice. If you multiply that through, this accounts for 312.5 kHz of frequency domain “padding� at the high and low end of the bandwidth, or 625.0 kHz of bandwidth which is not to be used.

Pilot Bins

Next up was the pilot bins. We’ve already gone over the purpose (and use) of our pilots, but I’ve found there to be an art to the placement of the pilots. Interpolation between pilot bins has turned out to be very reliable, but extrapolation, on the other hand, has been a major pain, for reasons I don’t fully understand yet.

My intent in placement was to pick out roughly even stretches of data bins bracketed between pilots, with as few data subcarriers as practical “outside� of a pilot (using extrapolation). I’ve played a bit with my AGWN simulator(s), as well as logging errors between two SDRs, and the configuration I have this in has been fairly resillant (for whatever reason), and withstood a few rounds of tweaking.

Data Bins

Almost as an afterthought – all of the remaining bins become data bins.

This puts the total number of data bins at 43, which, since Mode A carries data in QPSK/QAM-4 (two bits per data subcarrier), means we can carry 86 bits of data per OFDM symbol. That fairly modest capacity is largely due to the modulation scheme (or fft size, but increasing that has been … fraught) we’re using for Mode A – but I’ve made up for it by including 168 OFDM symbols in a single burst in order to have enough data to carry IP traffic without splitting the packet into two bursts.

With all that designed and on paper, we’re ready to start to tackle the next layer up – our Layer 2, named, creatively, “link�.

Let’s send some link layer data →

The following table is Mode A’s OFDM Subcarrier Plan. This is in negative first ordering (meaning the 0th member is the most negative fft bin, and the Nth is the highest frequency fft bin).

SubcarrierPlan([
 Guard,
 Guard,
 Guard,
 Guard,
 Guard,
 Guard,
 Guard,
 Guard,
 Data,
 Data,
 Data,
 Pilot(iq!(-1.0, 0.0)),
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Pilot(iq!(1.0, 0.0)),
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Guard,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Pilot(iq!(0.0, -1.0)),
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Data,
 Pilot(iq!(0.0, 1.0)),
 Data,
 Data,
 Data,
 Guard,
 Guard,
 Guard,
 Guard,
 Guard,
 Guard,
 Guard,
 Guard,
])

The following table is Mode A’s frequency-domain preamble. This is, as above, in negative first ordering. I don’t actually think these values matter much (at all)? – but in case they do, here’s what I have. I muck with these a lot and haven’t found many changes in quality of detection or frequency correction yet.

[
 IQ::new(0.0, 0.0),
 IQ::new(0.0, 0.0),
 IQ::polar((TAU / 13.0) * 2.0, 1.0),
 IQ::polar((TAU / 13.0) * 3.0, 1.0),
 IQ::polar((TAU / 13.0) * 4.0, 1.0),
 IQ::polar((TAU / 13.0) * 5.0, 1.0),
 IQ::polar((TAU / 13.0) * 6.0, 1.0),
 IQ::polar((TAU / 13.0) * 7.0, 1.0),
 IQ::polar((TAU / 13.0) * 8.0, 1.0),
 IQ::polar((TAU / 13.0) * 9.0, 1.0),
 IQ::polar((TAU / 13.0) * 10.0, 1.0),
 IQ::polar((TAU / 13.0) * 11.0, 1.0),
 IQ::polar((TAU / 13.0) * 12.0, 1.0),
 IQ::polar((TAU / 13.0) * 13.0, 1.0),
 IQ::new(0.0, 0.0),
 IQ::new(0.0, 0.0),
]

Let’s send some link layer data →

The time has come.

If you’re following along at home, we now have all the basics we need to glue these parts together and see what this looks like.

We’re going to build the highest-level constructs for the PHY in code – something that takes some number of bytes in and writes out IQ samples fit for transmit over the airwaves (we’ll call this the Encoder), and something that takes chunks of IQ samples in, writing out decoded bytes (which we’ll call the Decoder).

Encoder

Let’s begin with the Encoder, since it’s slightly less involved. I’ve tried to make this a bit more accessible by drawing a diagram out before describing the order of operations, so that it’s possible to follow along visually.

While the process here can look like a lot, it’s really not that bad. We begin by taking the incoming bytes, converting the bytes into bits, and chunk those bits into parts which are sized to fit completely within an LDPC message. We will then encode incoming data into LDPC messages, using our configured LDPC Matrix (the table we appropriated from 802.3an). Next, we apply whitning over all the bits in our encoded (and packed) LDPC messages, using our configured whitening constant. In the case of QPSK, pairs of bits will then be modulated into a QAM subcarrier, where each QAM point represents a range of bits in the message. We’ll go through each of those modulated IQ subcarriers, and set each corresponding data subcarrier in order, for each OFDM symbol contained in the pigeon Burst. The preamble configuration is then used to generate (or, more likely, can be used at startup to precompute) the Schmidl-Cox preamble, which is written to the first IQ samples in our output IQ buffer. Finally, we will do a series of inverse FFT operations to convert each OFDM symbol to the time domain, including their cyclic prefix.

Let’s take a look at doing that, but in code this time now:

// (lightly edited for clarity)
impl Encoder {
 ..

 /// Encode the provided bits into the output time-domain IQ samples.
 fn encode(
 &mut self,
 dst: &mut [IQ],
 src: &Vector,
 ) -> Result<Burst, Error> {
 let src = {
 let mut raw = Vector::new(self.fec.message_len());

 // Set `raw`'s data bits, compute and set LDPC
 // checkbits.
 self.fec.add(&mut raw, src);

 // Apply whitening, and return
 raw.xor(&self.whitening)
 };

 // copy in the precomputed schmidl-cox preamble to `dst`
 let preamble_len = self.preamble_iq.len();
 dst[..preamble_len].copy_from_slice(&self.preamble_iq);

 // allocate a new (frequency domain) 'Burst' container.
 let mut burst = Burst::new(
 &self.mode.ofdm.plan,
 self.mode.ofdm.symbols
 );

 // modulate bits from 'src' as iq, and set each
 // data subcarrier for each ofdm symbol in the
 // burst.
 self.burst_encoder.multiplex(&mut burst, &src);

 // convert from frequency-domain data into time
 // domain iq samples, writing out ofdm symbols
 // and cyclic prefixes to `dst`.
 self.burst_encoder
 .transform(&mut dst[preamble_len..], &burst)?;

 // normalize all IQ samples; the maximum magnitude
 // in the IQ buffer may be very small, which weakens
 // our transmitted signal. Scale all IQ samples such
 // that the maximum IQ sample magnitude will be '1.0'.
 dst.norm();

 Ok(burst)
 }
}

Using the Encoder should hopefully be fairly straightforward – we’ll give it a bag of bytes, and get back some IQ samples that we can ask our nearest SDR to transmit.

As for what happens on the other end?

Decoder

Next up is the mirror image of our Encoder – the, imaginatively named, Decoder. The Decoder is slightly more involved (since it has to find the packet in the IQ stream, as well as correct for channel error(s)), so we’ll do the same thing as above – start with a diagram. My hope is going over the Encoder first helps us only really focus on the “newâ€� stuff, otherwise it should feel like running the Encoder backwards.

Here, we start with an incoming stream of IQ, where we will process scan detections as they come in from our Schmidl-Cox detector and burst Scanner. This will give us a “snippitâ€� of IQ, sized to exactly our Burst. We’ll begin to correct our IQ samples by first doing frequency estimation and correction in the time domain using our preamble and ofdm configuration. We’ll then do a series of inverse FFTs to extract each OFDM symbol in our Burst, where we can then do channel estimation and correction. With the OFDM symbols (hopefully) good enough, we can now map each data subcarrier back to bits, and unapply whitning. The resulting bits are then chunked back up into LDPC messages, which are then checked, and concatanated data extracted. Finally the bits are turned back into bytes, which are written to our output buffer.

However, before we get into the code to do this – there’s one last detail. We know bursts won’t overlap (if they do, it’s likely not possible to recover right now – even though other PHYs can and do), so any time we see something we believe to be a burst, we can skip ahead by the burst’s (constant) length within the IQ, and avoid trying to decode anything else in there.

The nice side-effect here is this also gives us an interesting property for the Decoder – namely, we know the maximum number of Burst detections we can get for a given block of incoming IQ data if they were packed end-to-end – and we can pre-allocate the memory we need, avoiding allocations for every demodulation attempt (which may or may not even be a valid Burst).

This pre-allocated block of memory to hold the burst’s data is something that I’ve called a frame buffer internally. Each frame buffer contains exactly sized buffers to hold decoded information from the burst – an iq buffer that is exactly the same number of samples required to encode the preamble and data, exactly the number of bits needed to store pre and post FEC data, pre-allocated byte array, etc.

Not shockingly, the code looks like this:

#[derive(Clone)]
pub struct FrameBuffer {
 /// Corrected IQ samples
 pub samples: Samples,

 /// post-correction OFDM burst
 pub burst: Burst,

 /// demodulated bits from the OFDM burst
 pub bits: Vector,

 /// demodulated bits from the OFDM burst,
 /// after FEC, and cleaned
 pub raw_bits: Vector,

 /// Layer 2 contents of the Frame
 pub contents: Vec<u8>,
}

Of course, that alone is handy – but we need to use them. So let’s go ahead and do what we promised above – each Decoder uses a fixed number of pre-allocated FrameBuffers to store packets in-flight, packed into what is, creatively, called FrameBuffers within my code.

As an aside, I likely should have called this a Memory Pool, since that’s the common and accepted name for this design pattern – but being stuck with unfortunate names is the burden of those of us who stumble into sensible ideas over time. The only nuance here is that I use the pools strictly sequentially – we only “saveâ€� the FrameBuffer if the LDPC checksum is correct, allowing us to only keep track of how many successful packets we have and being able to get the valid FrameBuffers, rather than storing a handle to each FrameBuffer as we go – a promise that most memory pools do not make, since blocks can usually be taken and returned in any order.

Let’s go ahead and do the whole Decoder dance now:

// (lightly edited for clarity)

impl Decoder {
 ..

 /// Process incoming IQ for Pigeon Bursts, and
 /// demodulate them.
 pub fn decode(
 &mut self,
 buf: &[IQ],
 ) -> Result<Vec<(Detection, &FrameBuffer)>, Error> {
 let mut ret = Vec::new();
 let mode = self.scanner.mode().clone();

 // reset the "valid frame buffer count" back to 0
 self.frames.reset();

 // call the scanner and get scan detections for
 // this block of iq (`buf`)
 for detection in self.scanner.scan(buf) {
 // for each detection, we're (only) going to process
 // the iq snippit, but pass along the metadata
 // such as SNR.
 let ScanDetection {
 snippit,
 snr,
 range,
 m,
 } = detection;

 // grab the next free frame buffer to work within.
 let frame_buffer = self.frames.next_mut();

 // Copy the snippit into the frame buffer (a mutable
 // location)
 frame_buffer.samples.copy_from_slice(snippit);

 // estimate the frequency offset based on the
 // Burst's Schmidl-Cox preamble.
 let preamble_fo = preamble::estimate_frequency_offset(
 &mode.preamble,
 mode.rate,
 &frame_buffer.samples[..mode.preamble.samples()],
 );

 // Shift the IQ stream by the estimated frequency
 // offset -- hopefully we're closer to 0Hz
 frame_buffer.samples.shift(mode.rate, preamble_fo);

 // estimate the frequency offset based on the
 // each burst's **cyclic prefix** -- exactly like
 // we did with the Burst Schmidl-Cox preamble,
 // but this time on each OFDM symbol.
 let ofdm_fo = ofdm::estimate_frequency_offset(
 &mode.ofdm,
 mode.rate,
 &frame_buffer.samples[mode.preamble.samples()..],
 );

 // Shift the IQ stream closer yet; hopefully this
 // is a very small nudge even closer still to 0Hz.
 frame_buffer.samples.shift(mode.rate, ofdm_fo);

 // Do a bunch of inverse fft operations for each
 // OFDM symbol, filling the frequency-domain Symbol
 // structs in `frame_buffer.burst` (Burst) struct.
 //
 // this will also do channel estimation and
 // correction before returning.
 self
 .decoder
 .transform(
 &mut frame_buffer.burst,
 &frame_buffer.samples[mode.preamble.samples()..],
 )?;

 // "demultiplex" each data subcarrier's frequency-domain
 // IQ constellation point, setting the correct bit range.
 self.decoder.demultiplex(
 &mut frame_buffer.raw_bits,
 &frame_buffer.burst
 );

 // unapply whitening by XOR-ing the buffer with
 // the well-known whitening vector.
 frame_buffer.raw_bits = frame_buffer.raw_bits.xor(
 &self.whitening);

 // verify that the LDPC message(s) are all correct,
 // and if so, concatanate the the data bits (no check
 // bits) to the `bits` vector.
 if self.fec.decode(
 &mut frame_buffer.bits,
 &frame_buffer.raw_bits
 ).is_err() {
 // this is where invalid packets fail. we gave it a good go.
 // next packet please.
 continue;
 }

 // copy the raw bits out, as bytes, to the `contents` buffer.
 frame_buffer.bits.copy_as_bytes(&mut frame_buffer.contents);

 // store metadata/metrics on the demodulation.
 ret.push(Detection {
 m,
 snr,
 index: range.start,
 });

 // save the contents of this frame buffer (don't
 // reuse this buffer next go-around).
 self.frames.save();
 }

 // We're going to take out the borrow on the frame at
 // the end since we don't want to deal with telling the
 // compiler via code gymnastics that the mut and non-mut
 // borrows are OK since they're non-overlapping.
 Ok(ret.into_iter().zip(self.frames.iter()).collect())
 }
}

Phew. That was kinda a lot. In fact it’s basically the whole thing. This function is as close to “how do you read an OFDM packet� as it gets, and perhaps the most important part of this whole series. Beyond that, though, this is a huge conceptual unlock. This means we now have an incredibly powerful primitive; the ability to take bytes and go to/from IQ samples over the air.

Let’s talk about pigeon modes →

Interesting research: “Stealing Reasoning Traces from Proprietary LLM APIs“:

Abstract: Leading large language model providers now conceal their models’ step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with each subsequent request. Building on prior research, we identify an architectural vulnerability: these encrypted blocks are fully compatible and interchangeable across different sessions, users, and models within a provider’s ecosystem. We exploit this compatibility to develop a scalable decryption jailbreak. By injecting an encrypted reasoning trace from a given model into a weaker, and less safeguarded model from the same provider, we force it to decode and output the trace verbatim in plaintext, without ever jailbreaking the more capable model directly. This vulnerability enables four distinct attack vectors. First, it circumvents anti-distillation mechanisms, allowing adversaries to extract a proprietary model’s reasoning, as we demonstrate across Anthropic, OpenAI, and Google. Second, it allows for large-scale private data extraction. Developers frequently share session logs publicly, unaware of contents of the encrypted blocks. By decoding 315,320 reasoning blocks scraped from public repositories, we recovered 367 Personally Identifiable Information (PII) artifacts and 182 credentials. Third, it inadvertently reveals hazardous information hidden within the reasoning process, even in cases where the model’s final, visible output safely rejects a malicious request. Fourth, attackers can leverage this flaw to execute invisible prompt injections, embedding malicious payloads entirely within encrypted blocks to poison public agentic rollouts. Following responsible disclosure, we propose concrete cryptographic and system-level mitigations to secure client-side reasoning.

I asked the banker who is my reference at the bank something like this:

Give that we are talking about the consequences of the tantrum of a fascist foreign government, what happened to them (who are also people close and dear to me), in some future can very well happen to me.

Suddenly my risk profile shot up under the roof.

What do you suggest me to do? Should I find a trusted source of gold bullions to bury under the cellar at home?

The answer was something like this:

Sadly YES, given that the USA have a sort of financial monopoly they can entitle themselves to arbitrarily define a person/organization as a terrorist without any trial or judicial course, and as a consequence apply sanctions that cannot be effectively counteracted, not even abroad.

I didn't have this in my 2026 bingo card, but here we are.

For more details, see:

• the A/I press release under "12. Banca Etica: the sanction also affected an Italian bank account"
• the public statement from the bank (in Italian)

For some broader context on this kind of actions from the USA, see also:

• The same sactions against UN Special Rapporteur Francesca Albanese
• Executive Order 14203 titled "Imposing Sanctions on the International Criminal Court"

From our anonymous submitter:

Having reached the end of the road at a company increasingly swallowed up companies further east which you'd never believe were still afloat, I found myself headhunted for certain specialty software skills. I was reaching the final few years of my expected working span, so I jumped at the chance. The money was (to me at that time) spectacularly good, so I jumped into it.

It started when my first day was spent by me being sent home for the weeks it was still going to take to onboard me. Not bad, engaged to wait, as it were, and the first 6 months was thus and so.

The man who had interviewed me, call him Fred, was intelligent and urbane, and was a joy to meet. He and I clearly hit it off, and lo and behold I was in. It was he who gave me my first assignment, which was mathematical analysis of their core milk-cow program because they needed to find out what it did, and how it did it, so they could perhaps implement it in a more contemporary language.

So I did that, and was just about to publish my findings with him, when Fred inconveniently dropped dead suddenly. In the what-are-we-going-to-do-now-our-key-man-is-no-more confusion, we contractors were forgotten.

For the next 18 months or so (may have been more, may have been less) I was more or less ignored. I spent the time writing a development environment to work on any part of the program conveniently, all the while sitting next to a man who was constantly, forcefully and repetitiously speaking ill of the managers in his line structure. The ridiculously garrulous boss who inherited me thought little of me, and handed me the little work that came my way with active hostility. One or two good guys, but mostly a cabal of elderly men trying to preserve their little money-spinner as long as they could, and a johnny-come-lately trying to increase (and even introduce) automatic processes was less than welcome. During that time I spent quite some time on TDWTF, submitting a gem or two here and there.

No surprise when they finally kicked my arse away. No love lost there. Now working my last couple of years to retirement as a postie.

No punchline here. I want you to know that dropping dead from work is all too real and can happen to anyone.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Author: Majoki Why not? Why the never-ever-ever not? All the intense, serious, logical approaches to developing AI models had failed to deliver the much-hyped promise of AGI. So why not? Yayoi thought. How could her approach possibly be worse than the current slew of sycophantic chatbots and exploitative reactive machines that were cratering our cognitive […]

The post Whimsy appeared first on 365tomorrows.

The following contributor got their Debian Developer account in the last two months:

• Nicolas Peugnet (nicolasp)

The following contributors were added as Debian Maintainers in the last two months:

• Antoine Lassagne
• Ivan Hu
• Jesse Rhodes
• Haolin Xue
• Léo Haf
• Rony João de Sousa
• Luke Yasuda
• Darshaka Pathirana

Congratulations!

My Debian contributions this month were all sponsored by Freexian.

You can also support my work directly via Liberapay or GitHub Sponsors.

Personal note

This month, my Dad unexpectedly passed away after a short illness. As a result I obviously got less work done than usual, and I still have a lot to take care of (since I’m the executor of his will, as well as helping with funeral arrangements) while grieving and generally having less focus and energy. Having routine work to do is one of the ways I cope with this sort of thing, but all the same, I hope people will bear with me and maybe remind me if I seem to be dropping the ball on something you especially need.

LLM vote

[Content note: strong opinions.]

I voted in General Resolution: LLM usage in Debian. My vote was pretty much the opposite of what ended up winning, so I’m quite disappointed. My personal opinion is that LLMs are cognitive hazards to their users that impose ecological costs far out of proportion to their utility at a time when the world absolutely cannot afford them. When the impossible economics of the large commercial models are finally allowed to catch up with reality, I expect there to be significant macroeconomic consequences, and that people who have become dependent on them will have problems; and who knows what the copyright situation on their output really is. I’m not convinced that local models are better enough on these axes to be worth the costs.

Debian’s direct contribution to all that will be negligible on a global scale, and even the most radical proposals in the GR didn’t expect that we could do much about upstreams that have gone all-in on LLMs. Even so, I’d hoped that my fellow developers might be more willing to lean on our position in the free software ecosystem to make at least a moderately radical statement. Instead, we’ve at best presented an undistinguished fence-sitting position to the world, and further entrenched the idea that humans can reliably do a good job of reviewing the output of tools that are designed to produce output plausible to humans. I certainly don’t trust my own code review skills that far.

Since I’ve never voluntarily used an LLM (not counting LLMs being foisted on me by things like search results, support chatbots, or incoming pull requests, regardless of whether I asked for them), and don’t intend to for the foreseeable future, I doubt this will change much for me in terms of the way I work. The winning option is a very weak one that imposes no new requirements on developers, which means that it also does nothing to stop me continuing to reject LLM-generated material from Debian bug reports and merge requests in my areas of responsibility. I know this probably won’t do much to satisfy people who have decided that Debian is slop now, but it’s the best I can do.

OpenSSH

I finally landed the GSS-API key exchange package split in our OpenSSH packaging. Here’s the NEWS entry:

openssh (1:10.4p1-5) unstable; urgency=medium

  The openssh-client and openssh-server packages no longer include GSS-API
  authentication and key exchange support; this adds pre-authentication
  attack surface and generally increases complexity, and should only be used
  where specifically needed.  Users who need these features should install
  openssh-client-gssapi or openssh-server-gssapi instead.

 -- Colin Watson <cjwatson@debian.org>  Sun, 23 Aug 2026 17:39:55 +0100

I fixed a flaky autopkgtest.

I upgraded from 10.4p1 to 10.5p1, which was a good test of keeping openssh and the new openssh-gssapi source package in sync.

PuTTY

I upgraded from 0.84 to 0.85.

Python packaging

New upstream versions:

• anorack
• djangorestframework (fixing a build failure)
• isort
• lazr.delegates (dropping pkg_resources)
• lazr.restfulclient (dropping pkg_resources)
• lazr.uri (dropping pkg_resources)
• nbclient (fixing a build failure)
• pydantic-settings
• python-argcomplete
• python-blockbuster
• python-makefun (fixing a build failure)
• python-persistent
• python-pytest-run-parallel
• python-time-machine
• python-typing-inspection
• wcwidth
• zope.hookable
• zope.i18nmessageid
• zope.interface
• zope.proxy
• zope.security
• zope.testrunner

The version treadmill continues: we’ve just finished dropping Python 3.13 as a supported version, so now we’ve started working on enabling Python 3.15 as a supported version. Maximiliano Curia has been very helpfully driving this. I didn’t get as much done here as I’d have liked (see the top of this post), but I fixed a couple of packages:

• pydantic
• python-typing-extensions

Other build/test failures:

• anorack
• buildbot
• codespell
• django-prometheus
• pendulum
• pydantic-core
• python-argcomplete
• python-ast-serialize
• python-bcrypt
• python-cross-web
• python-datamodel-code-generator
• python-djangorestframework-flex-fields
• python-openapi-core
• python-treq
• scikit-build-core

I fixed some other bugs:

• python-librosa: Hardcoded Python 3.13 in rules file
• python-makefun: Fails to build source after successful build

bugs.debian.org

I deployed the fix for Invalid link rel=”canonical” on bugs.debian.org. In the process I found a few bugs in recent undeployed code and fixed them.

Not sure it’s practical, but it’s certainly striking.

The AVIF image format gets more and more popular in the web dev community, so I needed to teach XFCE4's thunar (file manager) and Ristretto (image viewer) to thumbnail these.

Luckily that is not too hard:

Debian Trixie separates its gdk-pixbuf libraries slightly differently than previous versions. That's why it is not "automatically there". Ensure you have the libavif-gdk-pixbuf plugin and the tumbler service (which XFCE uses to process thumbnails):

Thunar has likely tried (and failed) to load your AVIF files before you installed the package, it will have saved a blank or "broken image" placeholder in a thumbnail cache directory. It will not attempt to regenerate them unless you clear this cache:

Tumbled will restart on its own when it is needed. When you open thunar again and navigate to your image directory ... your AVIF images will now generate thumbnails automatically like the other image format did already.

A municipal election is coming up here in Kelowna, which means that election signs are now showing up everywhere. Let me be blunt, I hate these signs. Not just because they’re ugly, although they are, but because they’re manipulative.

Why is this relevant here? Not because it’s politics, but rather because understanding how we are manipulated by the environment around us is a skill we should all have.

The signs went up recently, and now every corner lot has three or four of them. Of all the signs I could see, I recognised exactly one name, and only because he already has the job.

So what is a sign actually doing to us? The obvious answer is that it makes a name familiar, and that familiar things feel safer. Although that’s not the whole story. When Cindy Kam and Elizabeth Zechmeister studied this directly, they found something more specific going on.

“we provide conclusive evidence that name recognition can affect candidate support, and we offer strong evidence that a key mechanism underlying this relationship is inferences about candidate viability”
Cindy Kam and Elizabeth Zechmeister, “Name Recognition and Candidate Support”¹

Viability, not affection. We don’t warm to the familiar name so much as conclude that it belongs to whoever is going to win, and then we move toward the winner. Donald Green and his colleagues found the same thing across four randomised experiments in New York, Virginia and Pennsylvania, planting signs in randomly selected American voting precincts. The signs they put in supporters’ front gardens were expected to work mainly through what they called the support among neighbours signal.²

So a lawn sign isn’t telling us that we like this person. It’s telling us that the people on this street are already with him.

You might be thinking that a street full of signs really does show local support. What it shows instead is which campaign had the budget for signs and the volunteers to plant them. It’s the same reason campaigns quote how much money they’ve raised as though that predicted how well they’d govern. It predicts how many times they can put themselves in front of us, which is a very different thing.

Timing matters more than we’d guess, too. Hill and his colleagues measured how long the persuasive effect of political advertising actually survives, and found that it fades within days. Further down the ballot it barely survives at all: “in lower level races, advertising causes preference shifts that have half-lives of only 1 to 2 days and no discernible long-term survival”.³ The sign we drove past three weeks ago has already stopped working on us. The one still standing on the route to the polling station is doing nearly all of it.

None of this is unique to elections. The vendor with the biggest booth at the conference feels like the safe choice. The colleague who is visible in every meeting gets read as the one contributing most. The idea that wins planning is often just the one that got said most often, which is the same social proof that shapes so many of our meetings.

What’s happening in all of those is that we’ve quietly swapped the question, which is the essence of attribute substitution, a cognitive bias where we replace a complex problem with a simple one and solve for that instead. Working out who would actually govern well, or which vendor is actually better, is expensive and often impossible from where we’re standing. Working out whose name we’ve seen most is free. Our brains answer the cheap question and hand us the result as though it were the answer to the expensive one.

I’d still like to ban the signs, but that wouldn’t fix the biggest part of the problem. They’re the cheapest medium there is, so taking them away just hands the vote to whoever can afford television instead. What we’d need to ban is any political advertising at all, in any media format.

The next time a name feels like the safe choice, it’s worth asking what we actually know about it, and whether the real answer is that we’ve simply seen it before.

I am a heavy user of sidenotes:¹ they keep optional content next to the text instead of sending the reader to the bottom of the page and back. Tufte CSS renders them without JavaScript but only accepts inline content. CSS anchor positioning, now supported by recent browsers,² is an elegant alternative. Sidenotes can hold several blocks, still without JavaScript, and fall back below the paragraph referencing them on narrow viewports and older browsers.

In 2023, Eric Meyer demonstrated this technique in “Nuclear Anchored Sidenotes.� The main improvement over other solutions is that the notes can sit anywhere in the HTML document. You can place them after the paragraph referencing them, as regular block elements for text browsers, screen readers, feed readers, and reader mode to render them properly:

Rendering in Lynx, a text browser

When the viewport is too narrow or the browser does not support CSS anchor positioning, you can style them so the reader can skip them or glance at them without losing their position in the text:

Rendering below the paragraph on a narrow viewport

Once the viewport is large enough, they appear in the margin, at the same vertical position as the matching reference mark, unless they would collide with a previous sidenote, as in the example below:³

Rendering in the margin on a large viewport

The gist of CSS anchoring is to position an element relative to another element—the anchor. For the sidenotes, the anchor is the reference mark. I use the following markup, with a data attribute to specify the anchor name:

<sup id="fnref:YYY" data-anchor="--lf-sn-YYY">
  <a href="#sidenote-YYY">1</a>
</sup>

The matching note is an <aside> element carrying the same data attribute for the anchor name. We put it after the paragraph holding the reference mark:

<aside role="note" id="sidenote-YYY" data-anchor="--lf-sn-YYY">
  <sup>1</sup>
  <p>A first paragraph.</p>
  <p>A second paragraph.</p>
</aside>

On a narrow viewport or when the browser is too old for CSS anchoring, we style the sidenote, which stays below its paragraph, with a muted color:

aside[role="note"] {
  margin-block: 1rlh;
  color: #444;
}

On a wide viewport and when the browser is recent enough, we move the sidenote to the right margin:

@supports (anchor-name: attr(data-anchor type(<custom-ident>))) {
  @media (min-width: 72rem) {
    main {
      position: relative;
      sup[data-anchor] {
        anchor-name: attr(data-anchor type(<custom-ident>));
        /* → anchor-name: --lf-sn-YYY */
      }
      aside[role="note"][data-anchor] {
        anchor-name: --lf-sidenote;
        position: absolute;
        position-anchor: attr(data-anchor type(<custom-ident>));
        /* → position-anchor: --lf-sn-YYY */
        top: max(anchor(top), anchor(--lf-sidenote bottom, -1rlh) + 1rlh);
        left: 100%;
        margin: 0 2rem;
        width: 18rem;
        color: inherit;
      }
    }
  }
}

attr() extracts the anchor name for the reference mark from the data-anchor attribute. It returns a string, unless we specify a CSS unit or a type, like here: the browser parses the data attribute as a custom identifier, which anchor-name validates as a dashed identifier, a custom identifier starting with two dashes.⁴

The note itself is absolutely positioned past the right edge of the main block. It selects the matching reference mark as its anchor with position-anchor set to the value of the data-anchor attribute. Each note is also an anchor named --lf-sidenote. We use it to keep the next note from colliding with this one.

The anchor() CSS function lets us position the note’s top edge relative to its anchor: anchor(top) aligns the top edge of the note with the top edge of the reference mark. It can also take another anchor as a parameter: anchor(--lf-sidenote bottom) would align the top edge of the note with the bottom edge of the closest preceding anchor named --lf-sidenote—so the previous note.⁵ Like attr(), anchor() accepts a fallback value as its second parameter and use it when the named anchor does not exist.

The top property handles three cases, illustrated in the following diagram:

The three cases for the vertical position of a note

1. The first note’s top edge aligns with the top edge of its reference mark: as there is no previous note, anchor(--lf-sidenote bottom, -1rlh) + 1rlh resolves to 0 and max() returns anchor(top).
2. When the reference mark of a later note sits above the bottom of the previous note, plus some vertical space, the note goes below the previous one to avoid a collision. max() returns anchor(--lf-sidenote bottom) + 1rlh.
3. Otherwise, the note’s top edge aligns with the reference mark’s top edge, as max() returns anchor(top).

Have a look at the complete stylesheet, which also adapts the reference mark to the location of the note: a “↓� arrow when the note sits below the paragraph, a “→� arrow when it moves to the margin. Gwern’s “Sidenotes In Web Design� lists more implementations and their trade-offs.

Some bloggers aim to write a post in 30 minutes. I planned to publish three web-related articles this weekend. Instead, I spent an inordinate amount of time elsewhere: about 15 commits on the build system, a pull request to update CSS highlighting for nested selectors in Pygments, and a small correction to MDN’s article on the anchor() CSS function. The SVG illustration took a bit less than an hour and the article itself a handful of hours. The attr() function came in after I thought “inline style looks ugly, isn’t there a better way?â€� But, hey, I still think this is worth it! ğŸ�¨

It's a holiday here in the US, a celebration of labor, so we're reaching back through the archives for a story about an attempt to be labor saving that was not successful. Original. --Remy

I’ve had to write a few domain-specific-languages in the past. As per Remy’s Law of Requirements Gathering, it’s been mostly because the users needed an Excel-like formula language. The danger of DSLs, of course, is that they’re often YAGNI in the extreme, or at least a sign that you don’t really understand your problem.

XML, coupled with schemas, is a tool for building data-focused DSLs. If you have some complex structure, you can convert each of its features into an XML attribute. For example, if you had a grammar that looked something like this:

The Source specification obeys the following syntax

source = ( Feature1+Feature2+... ":" ) ? steps

Feature1 = "local" | "global"

Feature2 ="real" | "virtual" | "ComponentType.all"

Feature3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

Feature4 = "first" | "last" | "DayAllocation.all"

If features are specified, the order of features as given above has strictly to be followed.

steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

oneOrMoreNameSteps = nameStep ( "." nameStep ) *

zeroOrMoreNameSteps = ( nameStep "." ) *

nameStep = "#" name

name is a string of characters from "A"-"Z", "a"-"z", "0"-"9", "-" and "_". No umlauts allowed, one character is minimum.

componentSteps is a list of valid values, see below.

Valid 'componentSteps' are:

- GlobalValue
- Product
- Product.Brand
- Product.Accommodation
- Product.Accommodation.SellingAccom
- Product.Accommodation.SellingAccom.Board
- Product.Accommodation.SellingAccom.Unit
- Product.Accommodation.SellingAccom.Unit.SellingUnit
- Product.OnewayFlight
- Product.OnewayFlight.BookingClass
- Product.ReturnFlight
- Product.ReturnFlight.BookingClass
- Product.ReturnFlight.Inbound
- Product.ReturnFlight.Outbound
- Product.Addon
- Product.Addon.Service
- Product.Addon.ServiceFeature

In addition to that all subsequent steps from the paths above are permitted, that is 'Board', 
'Accommodation.SellingAccom' or 'SellingAccom.Unit.SellingUnit'.
'Accommodation.Unit' in the contrary is not permitted, as here some intermediate steps are missing.

You could turn that grammar into an XML document by converting syntax elements to attributes and elements. You could do that, but Stella’s predecessor did not do that. That of course, would have been work, and they may have had to put some thought on how to relate their homebrew grammar to XSD rules, so instead they created an XML schema rule for SourceAttributeType that verifies that the data in the field is valid according to the grammar… using regular expressions. 1,310 characters of regular expressions.

<xs:simpleType>
    <xs:restriction base="xs:string">
            <xs:pattern value="(((Scope.)?(global|local|current)\+?)?((((ComponentType.)?
(real|virtual))|ComponentType.all)\+?)?((((Hierarchy.)?(self|ancestors|descendants))|Hierarchy.all)\+?)?
((((DayAllocation.)?(first|last))|DayAllocation.all)\+?)?:)?(#[A-Za-z0-9\-_]+(\.(#[A-Za-z0-9\-_]+))*|(#[A-Za-z0-
9\-_]+\.)*
(ThisComponent|GlobalValue|Product|Product\.Brand|Product\.Accommodation|Product\.Accommodation\.SellingAccom|Prod
uct\.Accommodation\.SellingAccom\.Board|Product\.Accommodation\.SellingAccom\.Unit|Product\.Accommodation\.Selling
Accom\.Unit\.SellingUnit|Product\.OnewayFlight|Product\.OnewayFlight\.BookingClass|Product\.ReturnFlight|Product\.
ReturnFlight\.BookingClass|Product\.ReturnFlight\.Inbound|Product\.ReturnFlight\.Outbound|Product\.Addon|Product\.
Addon\.Service|Product\.Addon\.ServiceFeature|Brand|Accommodation|Accommodation\.SellingAccom|Accommodation\.Selli
ngAccom\.Board|Accommodation\.SellingAccom\.Unit|Accommodation\.SellingAccom\.Unit\.SellingUnit|OnewayFlight|Onewa
yFlight\.BookingClass|ReturnFlight|ReturnFlight\.BookingClass|ReturnFlight\.Inbound|ReturnFlight\.Outbound|Addon|A
ddon\.Service|Addon\.ServiceFeature|SellingAccom|SellingAccom\.Board|SellingAccom\.Unit|SellingAccom\.Unit\.Sellin
gUnit|BookingClass|Inbound|Outbound|Service|ServiceFeature|Board|Unit|Unit\.SellingUnit|SellingUnit))"/>
    </xs:restriction>
</xs:simpleType>
</xs:union>

There’s a bug in that regex that Stella needed to fix. As she put it: “Every time you evaluate it a few little kitties die because you shouldn’t use kitties to polish your car. I’m so, so sorry, little kitties…”

The full, unexcerpted code is below, so… at least it has documentation. In two languages!

<xs:simpleType name="SourceAttributeType">
                <xs:annotation>
                        <xs:documentation xml:lang="de">
                Die Source Angabe folgt folgender Syntax

                        source = ( Eigenschaft1+Eigenschaft2+... ":" ) ? steps

                        Eigenschaft1 = "local" | "global"

                        Eigenschaft2 ="real" | "virtual" | "ComponentType.all"

                        Eigenschaft3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

                        Eigenschaft4 = "first" | "last" | "DayAllocation.all"

                        Falls Eigenschaften angegeben werden muss zwingend die oben angegebene Reihenfolge der Eigenschaften eingehalten werden.

                        steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

                        oneOrMoreNameSteps = nameStep ( "." nameStep ) *

                        zeroOrMoreNameSteps = ( nameStep "." ) *

                        nameStep = "#" name

                        name ist eine Folge von Zeichen aus der Menge "A"-"Z", "a"-"z", "0"-"9", "-" und "_". Keine Umlaute. Mindestens ein Zeichen

                        componentSteps ist eine Liste gültiger Werte, siehe im folgenden

                Gültige 'componentSteps' sind zunächst:

                        - GlobalValue
                        - Product
                        - Product.Brand
                        - Product.Accommodation
                        - Product.Accommodation.SellingAccom
                        - Product.Accommodation.SellingAccom.Board
                        - Product.Accommodation.SellingAccom.Unit
                        - Product.Accommodation.SellingAccom.Unit.SellingUnit
                        - Product.OnewayFlight
                        - Product.OnewayFlight.BookingClass
                        - Product.ReturnFlight
                        - Product.ReturnFlight.BookingClass
                        - Product.ReturnFlight.Inbound
                        - Product.ReturnFlight.Outbound
                        - Product.Addon
                        - Product.Addon.Service
                        - Product.Addon.ServiceFeature

                Desweiteren sind alle Unterschrittfolgen aus obigen Pfaden erlaubt, also 'Board', 'Accommodation.SellingAccom' oder 'SellingAccom.Unit.SellingUnit'.
                'Accommodation.Unit' hingegen ist nicht erlaubt, da in diesem Fall einige Zwischenschritte fehlen.

                                </xs:documentation>
                        <xs:documentation xml:lang="en">
                                The Source specification obeys the following syntax

                                source = ( Feature1+Feature2+... ":" ) ? steps

                                Feature1 = "local" | "global"

                                Feature2 ="real" | "virtual" | "ComponentType.all"

                                Feature3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

                                Feature4 = "first" | "last" | "DayAllocation.all"

                                If features are specified, the order of features as given above has strictly to be followed.

                                steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

                                oneOrMoreNameSteps = nameStep ( "." nameStep ) *

                                zeroOrMoreNameSteps = ( nameStep "." ) *

                                nameStep = "#" name

                                name is a string of characters from "A"-"Z", "a"-"z", "0"-"9", "-" and "_". No umlauts allowed, one character is minimum.

                                componentSteps is a list of valid values, see below.

                                Valid 'componentSteps' are:

                                - GlobalValue
                                - Product
                                - Product.Brand
                                - Product.Accommodation
                                - Product.Accommodation.SellingAccom
                                - Product.Accommodation.SellingAccom.Board
                                - Product.Accommodation.SellingAccom.Unit
                                - Product.Accommodation.SellingAccom.Unit.SellingUnit
                                - Product.OnewayFlight
                                - Product.OnewayFlight.BookingClass
                                - Product.ReturnFlight
                                - Product.ReturnFlight.BookingClass
                                - Product.ReturnFlight.Inbound
                                - Product.ReturnFlight.Outbound
                                - Product.Addon
                                - Product.Addon.Service
                                - Product.Addon.ServiceFeature

                                In addition to that all subsequent steps from the paths above are permitted, that is 'Board', 'Accommodation.SellingAccom' or 'SellingAccom.Unit.SellingUnit'.
                                'Accommodation.Unit' in the contrary is not permitted, as here some intermediate steps are missing.

                        </xs:documentation>
                </xs:annotation>
                <xs:union>
                        <xs:simpleType>
                                <xs:restriction base="xs:string">
                                        <xs:pattern value="(((Scope.)?(global|local|current)\+?)?((((ComponentType.)?(real|virtual))|ComponentType.all)\+?)?((((Hierarchy.)?(self|ancestors|descendants))|Hierarchy.all)\+?)?((((DayAllocation.)?(first|last))|DayAllocation.all)\+?)?:)?(#[A-Za-z0-9\-_]+(\.(#[A-Za-z0-9\-_]+))*|(#[A-Za-z0-9\-_]+\.)*(ThisComponent|GlobalValue|Product|Product\.Brand|Product\.Accommodation|Product\.Accommodation\.SellingAccom|Product\.Accommodation\.SellingAccom\.Board|Product\.Accommodation\.SellingAccom\.Unit|Product\.Accommodation\.SellingAccom\.Unit\.SellingUnit|Product\.OnewayFlight|Product\.OnewayFlight\.BookingClass|Product\.ReturnFlight|Product\.ReturnFlight\.BookingClass|Product\.ReturnFlight\.Inbound|Product\.ReturnFlight\.Outbound|Product\.Addon|Product\.Addon\.Service|Product\.Addon\.ServiceFeature|Brand|Accommodation|Accommodation\.SellingAccom|Accommodation\.SellingAccom\.Board|Accommodation\.SellingAccom\.Unit|Accommodation\.SellingAccom\.Unit\.SellingUnit|OnewayFlight|OnewayFlight\.BookingClass|ReturnFlight|ReturnFlight\.BookingClass|ReturnFlight\.Inbound|ReturnFlight\.Outbound|Addon|Addon\.Service|Addon\.ServiceFeature|SellingAccom|SellingAccom\.Board|SellingAccom\.Unit|SellingAccom\.Unit\.SellingUnit|BookingClass|Inbound|Outbound|Service|ServiceFeature|Board|Unit|Unit\.SellingUnit|SellingUnit))"/>
                                </xs:restriction>
                        </xs:simpleType>
                </xs:union>
</xs:simpleType>

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Author: Julian Miles, Staff Writer We live in a car park. It’s a really nice one. There are swings and toilets with locking doors and the showers are wooshy and warm every time. Dad says we can stay here until me and my sister Eliana go to big school. He says we can move to […]

The post Park Lives appeared first on 365tomorrows.

About the project and me

Your program crashes. You open gdb and get ?? instead of a stack trace. So you go find the right -dbgsym package, for the right version, for the right architecture, install it, and start again. Debuginfod removes that entire detour: gdb asks a server for symbols by the build-ID baked into the binary. Debusine already built packages, already produced -dbgsym files, and already hosted the archives; it just couldn’t answer the question.

This summer I made it answer. My project was to add debuginfod server functionality to Debusine so that it not only hosts -dbgsym packages, but also serves their debug symbols over the debuginfod(8) protocol. Debian developers can then debug binaries by setting a single URL that gdb uses to fetch the matching debug symbols. This project took me through design, backend work, an extraction pipeline on the worker, HTTP serving, documentation, and testing from the first blueprint all the way to a live demo on debusine.debian.net.

Initial planning and design changes

A design first, in !3030. The proposal submitted for GSoC 2026 was just an overview of how things will work, but in reality there were a lot of design questions which needed to be answered before starting with contribution. Debusine keeps development blueprints in its docs tree, reviewed like code, it’s basically a blueprint of what feature or new changes are we going to make. I was assigned the work item #957, which was basically about how the idea of implementing a debuginfod server functionality inside Debusine was initially proposed by a fellow member which later became a project idea under GSoC 2026. My developer blueprint pinned down the four decisions everything else depends on: extraction happens on the worker after the build, symbols are stored as artifacts keyed by build-ID, they’re published into suites alongside their binaries, and they’re served from the archive root rather than per-suite. Settling that up front meant the design discussions happened in a document instead of across three merged branches.

One of those arguments became its own fix. My wording implied symbols were unpacked inside the isolated sbuild environment (the consequence was I was handed a bug to be solved in the first week of contribution period), when they’re actually extracted afterwards on the worker, where the build output already sits, a distinction that matters, because doing work inside the unshare environment means extra tooling in the chroot and more ways to affect the build. !3119 corrected it before the wrong model spread into the code.

A new artifact type

Artifacts are a major concept in Debusine overall, so as per the developer blueprint we introduced a new artifact which was debian:debug-symbols. It holds every .debug file from one -dbgsym package. Its data is a validated list of lowercase 40-character build-IDs, and each file is stored under its build-ID as the path, so answering “what are the symbols for this ID?” is a direct lookup, with no path translation in the request handler. One artifact per package rather than per file: a util-linux build would otherwise spray hundreds of artifacts, collection items and relations across the database for no benefit. For implementing debian:debug-symbols artifact, I changed the main models.py file, along with that since it’s a norm to write unit tests, all mentioned under !3088.

Publishing workflow and solving a bug

Extracting symbols is only useful if they reach the archive people actually install from, so !3180 taught package_publish to follow the relates-to relation: copying binaries into a suite now brings their debug symbols along automatically, with nothing extra for the publisher to configure. Each build-ID becomes its own collection item, for example debugsym:hello_2.10-5_amd64_fcc9064… each carrying the package name, version and architecture copied from the binary, so the item is meaningful on its own without dereferencing anything. Uniqueness is enforced at both the suite and archive level, because the serving URLs are archive-wide and two suites must never disagree about what a build-ID means: republishing an identical file is accepted quietly, while two different files claiming the same ID is an error worth failing on. A partial index on the build-ID keeps the eventual HTTP lookup fast.

That looked finished until symbols started arriving in target suites disconnected from their binaries published, but unfindable, because copying items between collections silently dropped their artifact relations, and that relation is the only thing tying the two together. The fix sat one level above my feature, in the generic CopyCollectionItems task that does the copying, and since it was reusable infrastructure rather than anything debuginfod-specific, Colin implemented it himself in !3228. My project needed it to work at all; every other Debusine feature that copies items now gets it for free.

Endpoint and CI tests

With symbols in the archive, !3212 added the part users actually touch: GET /{scope}/{workspace}/buildid/<build-id>/debuginfo looks the ID up across every suite in that workspace’s archive, streams the file, and sets the X-DEBUGINFOD-FILE and X-DEBUGINFOD-SIZE headers the protocol expects. It also handles the two things gdb actually does: a HEAD probe before committing to a download, and ranged requests to pull individual ELF sections instead of the whole file. Scoping it to the archive rather than the suite is what lets one URL cover a whole workspace, so the developer never has to know which suite their binary came from.

Every merge request above landed with unit tests, but those only tell you that the pieces behave correctly. What Colin and I wanted was a real gdb fetching real symbols from a real instance, so !3261 adds an autopkgtest that builds a package, publishes it, checks the HTTP headers, then sets DEBUGINFOD_URLS and makes gdb go and get the symbols, wired into the CI integration tests so it runs on every change. It took me a day to learn that skipping the signing worker doesn’t simplify that test, it just hangs until the 30-minute timeout, because update_suites needs signing to produce a usable repository.

The last piece, !3301 covers the new artifact, the suite and archive changes, the new archive URL, and a how-to for using it. My first how-to draft explained how everything worked and offered four ways to set DEBUGINFOD_URLS; the version that shipped gives one recommended setup and gets out of the way. The same pass trimmed the blueprint down to only what’s still unimplemented, since a design document describing merged code is just an obstacle for the next reader.

What’s left

Only one item on my original plan didn’t land: an archive-level build_debug_symbols switch, modelled on Launchpad’s equivalent, letting an archive skip building -dbgsym packages entirely by passing DEB_BUILD_OPTIONS=noautodbgsym to sbuild. It was always the stretch goal rather than core scope, landing the extract-publish-serve path solidly mattered more than landing it broadly. The design is written up in the blueprint, and I intend to implement it myself.

The other gaps were deliberately out of scope from the start, and the blueprint says so. DWZ supplement files aren’t ingested, so packages using compressed debug info may render without the alternate strings table; debugging still works, it’s just less complete. Source-file serving runs into the same Debian packaging limits that constrain debuginfod.debian.net today, making it a design question rather than a coding one. Executable serving, the metrics and metadata endpoints, and federation to upstream debuginfod servers were excluded for similar reasons, none of them are needed for Debusine’s core use case, and each would have crowded out the parts that are.

One open bug is left too. On the last day of the coding period, Stefano Rivera found that publishing ledger and linux was failing, because I had told the database that a build-ID identifies one exact debug file which isn’t true in Debian, since dh_dwz runs once per binary package, so when one object ships in two binary packages their .debug files differ while describing identical code. How to fix it is still an open discussion #1582, though it may not land before the formal end of the project.

None of that is a handoff. GSoC’s timeline is ending, my involvement isn’t, I’m carrying on with Debusine until both the build_debug_symbols switch and DWZ supplement support are merged, and I expect to keep contributing beyond that. This project got me familiar with a codebase I enjoy working in, and the remaining pieces are mine to finish.

Thanks!

The biggest thanks go to my mentor, Colin Watson, whose reviews consistently found the thing I hadn’t thought about. He also gave me room to get things wrong first and understand why, which taught me more than being handed the answer would have.

Thanks as well to Raphaël Hertzog, Enrico Zini, Stefano Rivera, Carles Pina i Estany and Helmut Grohne and everyone else around Debusine and Freexian for reviews, comments and patience with my questions.

Special thanks to Freexian for developing Debusine in the open and for giving me access to test on debusine.debian.net.

Finally, thanks to the wider Debian community, whose build-ID and -dbgsym conventions did most of the hard work before I arrived and to Google Summer of Code for providing a platform and the time to do this properly.

Looking at the reactions to the Debian AI vote, I think some people still think the clock can be turned back, as if that ever worked in history. Rather than cry about spilled milk, I prefer to find a path forward in the new world. There are many ways to use LLMs, some of them are straightforward, others not so much.

One of the “not so clear” areas for me is the focus on agentic workloads. For complex tasks, sure, you want something that can work in the background, but in general, why does every single tool go the agentic way? I much prefer the “chat/ask” approach, or even the “code” one, but if I’m at the keyboard, why would I send a task to an agent, and see it work, instead of directly implementing it?

And then, this past Friday, I finally understood one part of that. I was in the airport, sitting at the gate and waiting to board a flight, and because I arrived much earlier at the airport (fearing crowds due to Labour Day weekend), I got one hour of work before boarding started. As the time for boarding approached, I did one more commit after making sure tests pass, pushed, closed laptop, and went to walk a bit before getting on the plane.

As I was getting up, I get a phone notification from GitHub that the CI run failed. I was quite surprised, as the local tests passed, so I open the notification, and realize that tests via make test vs CI (which additionally uses --pedantic) had slightly different settings, and of course I missed a build warning (which in CI is an error).

I thought I’d fix that on the plane, but then I saw a “Copilot agent” button in the mobile app. I was curious what it did, I click it, and I see Copilot starting a draft pull request, and saying:

Thanks for asking me to work on this. I will get started on it and keep this PR’s description up to date as I form a plan and make progress.

Fix the failing GitHub Actions job. Analyze the Actions logs, identify the root cause of the failure, and implement a fix.

Then it goes, finds the failure, writes the fix, and tries to run the tests. Well, it can’t do it (it runs in a restricted container, so no network, so stack install couldn’t actually work). The agent sees that, acknowledges it has no way to validate the fix, but the error message was clear enough that it was confident the fix is mostly correct, so it sends the pull request.

I allow full CI to run on the pull request, and go buy a bottle of water. After that, I check and see that the CI failed again, as not one but two test files were broken, and I didn’t have --keep-going, so the build stopped at the first failure. I write a comment in the pull request, no reaction, I realize I need to tag Copilot explicitly, I do that, and it starts another investigation.

I’m waiting now in the boarding queue, with phone in hand, while Copilot is fixing my bug. While I scan my boarding pass and walk towards the plane, the pull request is updated, I trigger another CI, it passes, and I merge it.

And then, it hit me. Agents allow me to make progress while being “not at keyboard”, whether that’s physically “not at keyboard”, or while working on something else. Fixing a simple test failure is not something that needs human attention per se, whereas improving the test layout might be.

In that airport, using otherwise-unusable downtime, and without explicitly intending to, I made progress in understanding a different way to use AI. Now I have three ways to work with LLMs: ask (tutor mode), code (implement my request), and agent (fix simple or complex problems, autonomously). I still don’t know about “plan” mode and really complex tasks, like asking it to implement features from scratch. That will probably be the next area to tackle.

And today (Sunday), while waiting for a running race to start, I opened GitHub, and asked Copilot to increase test coverage for a simple module. It did, and yes it still can’t run tests (I learned in the meantime that you can configure the environment in which the agent runs, nice), but after two back-and-forth messages, I have a pull request ready to review. All in the 20 minutes before a race, where I could either browse social media or actually do some meaningful work.

Checking now my GitHub billing, it looks like all of this Copilot use only cost $1.92. Yes, that is under two dollars! And while it did use compute resources, the person across the aisle who watched TikTok or Instagram for half an hour while waiting for takeoff also consumed a lot of compute, and so do the gazillion cat videos uploaded to YouTube every day.

To me, this is another tool in the toolbox, that might one day replace me (as it did to the 19th-century textile workers), or make me five times more productive — we’ll see where we end up. In the meantime, I can move faster, and make better use of my limited free time.

Enjoy the ride!

Another minor maintenance release of the RcppFarmHash package is now on CRAN as version 0.0.4.

RcppFarmHash wraps the Google FarmHash family of hash functions (written by Geoff Pike and contributors) that are used for example by Google BigQuery for the FARM_FINGERPRINT digest.

This releases updates several of package internal files for continuous intergration and package data.

The brief NEWS entry follows:

Changes in version 0.0.4 (2026-09-06)

• Minor updates to continuous integration, README.md and DESCRIPTION

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Title: Robot Dreams
Author: Isaac Asimov
Genre: American Science fiction
Publisher: Byron Preiss Visual Publications
Release Date: February 12, 2012
Pages: 349

★★★★☆

I think its interesting how my perspective on these older science fiction books changes a bit each time I read them. Last time I read this book I was annoyed by how few were robot stories, whereas this time I really dug The Martian Way, because I think the premise feels much more possible that it did a few years ago — let alone in 2008!

I enjoyed this book, again.

I have just tried CoMaps, a free mapping program released under the Apache license [1]. I have tried it on Android on a Pixel 6a but it also runs on Linux so I’ll try it on a PinePhone or similar at some convenient time. On Android it is in the F-Droid repository among others and for Linux there’s a Flatpak package.

The data it uses is from Open Street Map project [2] which has extensive and accurate coverage of every place I’ve looked at (Australia and a few other first-world countries). The first thing it does after being installed is start downloading the world data set from Open Street Map and prompt to download the data for the detected region (Melbourne in my case).

The UI is decent and allows most of the features that I am used to using in Google Maps. The quality of directions seems good, I’ve only tested it with one journey so far which was a 50 minute drive across the city and it gave a set of directions that Google Maps often gives.

It gives spoken directions which is an important feature but sometimes the way the directions are presented is confusing. When turning off a freeway it didn’t give a spoken direction to do that, it gave a direction to “turn right” which was AFTER leaving the freeway, fortunately the map was clearly displayed.

In terms of use practices of this program the main difference I recommend is checking which off ramp to use from a freeway before entering the freeway. With Google Maps you can rely on it giving clear directions in that case.

I recommend this program without reservation. It can do everything that Google Maps does apart from detecting traffic jams because there’s no way of detecting traffic without spying on users. It is designed to preserve user privacy and works well in that regard.

• [1] https://www.comaps.app/
• [2] https://www.openstreetmap.org/

After having witnessed how easy it is for good people to lose a .org domain over a fascist tantrum (you can follow the Autistici/Inventati story here and here), I've started moving all my infrastructure to differently managed TLDs.

enricozini.org and enricozini.com will keep being functional for the time being, as dropping a domain makes it available for squatting and impersonation.

These new domains are now online, with working web and emails:

• enricozini.eu
• ezini.eu
• ezini.it (probably equally subject to fascist tantrums these days)

It will take ages to migrate countless accounts that are tied to my primary email address, so better start early.

Waiting to see what will happen with .meow domains, which I supported despite not identifying as a cat.

I've released version 1.1.25 of plocate. This time around, there's two security issues of unknown severity; if you chain them with other bugs, they could lead to being able to list files (but of course not their contents) that you should not normally be able to see. So an update is probably in order; you can never be too safe these days.

The full changelog is:

plocate 1.1.25, September 6th, 2026

  - Fix two early-exit bugs with multiple databases.
    Reported by Manpreet Singh and Tyler Spivey.

  - Drop setgid properly, including the saved gid.
    Reported by Michal Sekletar, found with the help of Claude Opus 4.6.

  - Fix a potential symlink-checking race in updatedb.
    Reported by Michal Sekletar, found with the help of Claude Opus 4.6.

As usual, you can get it from the home page, or it's on the way up in Debian unstable.

This August, I accomplished what I wanted for many years: I deleted the last cgo dependency in Debian Code Search! This was made possible by Go’s recently introduced SIMD support, because now we can implement the TurboPFor integer compression format as efficiently — more efficiently, in fact, by using the newer AVX512 instruction set! — as the reference implementation.

Background: Why does DCS need a fast Integer Codec?

Debian Code Search (DCS) is a search engine that allows searching all the Open Source source code within Debian, with either literal search expressions or regular expression search queries.

A search engine uses an inverted index: a map from term to documents containing the term. Each document is typically represented most efficiently by using an id, so the index consists of many lists of document ids.

When searching, it is important to quickly decode these lists to answer the search query. However, there is a point of diminishing returns where the decoding speed, even though it can still be measurably improved quite a bit, no longer influences the overall query duration.

From 2012 (its inception) to 2019, Debian Code Search used to use a small index format, and queries were fast because the index was kept entirely in RAM. In 2019, I implemented the new index format, which adds an on-disk positional index. For literal queries (78.2% of DCS queries), querying the positional index on disk is faster than querying the non-positional index in RAM.

The efficient encoding of the TurboPFor format makes it possible to fit such an index on a mid-sized Hetzner server, which I rent with two 1 TB SSD disks. The optimized decoder of the C TurboPFor library is what made decoding fast at query time.

If you want to dive deeper into the algorithm, see this blog post from February 2019:

If you want to learn more about the positional index, see this blog post from September 2019:

SIMD in Go

For many years, you had the following options for using SIMD instructions in Go:

1. Hand-writing Go assembler code. This is only doable for small functions, for example bytes.IndexByte is implemented with hand-written Go assembly (including AVX2).
2. Generating Go assembler code with tools like Michael McLoughlin’s “Avoâ€�. This is how crypto/internal/fips140/sha256 uses AVX2. While Avo generator code definitely is higher-level than hand-written assembly, it is still too close to assembly for my taste.
3. Use a C library via cgo so gcc or clang compiles SIMD code. Debian Code Search used to use the powturbo/TurboPFor C library via cgo for the last 7 years.

The C TurboPFor library has served us well, but Debian Code Search was always intended to be a project using Go, so I would prefer it if I did not have any C code in the project.

Go 1.26 (released in February 2026) introduced the simd/archsimd package:

Go 1.26 introduces a new experimental simd/archsimd package, which can be enabled by setting the environment variable GOEXPERIMENT=simd at build time. This package provides access to architecture-specific SIMD operations. It is currently available on the amd64 architecture and supports 128-bit, 256-bit, and 512-bit vector types, such as Int8x16 and Float64x8, with operations such as Int8x16.Add. The API is not yet considered stable.

— Go 1.26 Release Notes

For my 2019 TurboPFor analysis, I implemented goturbopfor, a native Go teaching decoder (without any SIMD), because I find Go code easier to follow than C code, especially optimized C code. My implementation was intentionally not optimized so that the code was easier to study.

The TurboPFor format/algorithm has a vector-optimized part: bitpacking comes in a scalar variant (bitunpack32) and a vector variant (bitunpack256v32), where the vector variant is used for full blocks (256 values) and the scalar variant is used for remainder blocks (< 256 values).

When Go 1.26 was released, I used Claude Code to explore whether my native Go decoder’s bitunpack256v32 function (for the vertical vector layout) could be implemented using Go SIMD, and the answer was yes, it was possible and it was faster than without SIMD, but not quite at the level of C TurboPFor. If you let Claude Code try for long enough, it eventually finds enough optimizations (about 10) to match C performance.

I don’t want to vibe-code Debian Code Search, though, so I figured I would find some time to review the SIMD code at some point and see if I could implement something similar myself.

Before I found enough time and motivation to complete said review, I discovered that to not regress real-life query performance by more than 10 to 100 milliseconds (which seems acceptable), I don’t actually need to add SIMD code to my teaching decoder at all; it would be sufficient to reduce allocations in my teaching decoder and specialize it per bit width.

Encouraged by the possibility of using the optimized native Go decoder in Debian Code Search, I explored whether I could also implement a native Go encoder so that I could get rid of the C TurboPFor dependency entirely. The answer is yes, it is doable in a few days, and it isn’t even that much slower: Go is at 76% of C, see Debian/dcs commit e920dc7.

The goal I set myself at that point was to see if I could learn enough SIMD to optimize the native Go encoder such that its performance would match how DCS uses C TurboPFor (via cgo).

Beating C TurboPFor was possible in 2-3 commits (SIMD and bit width specialization). To my surprise, Claude Fable 5 pointed out that the encoder’s block scanning could be done more efficiently using a technique called positional popcount, and that is another 2x speed-up! 😲

To be clear: I am not saying the Go compiler beats C here. Certainly, the C compiler can also produce fast AVX512 code and can be used to implement positional popcount. When comparing apples to apples, i.e. backporting the AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a little slower at ≈1.4x C.

This spectacular result (much faster than what DCS had before) got me curious how far I could push the decoder with SIMD after all. I ended up matching/exceeding the cgo version here, too!

The rest of this article explains a few classes of optimizations I encountered along the way.

Starting Point

When I wrote my goturbopfor teaching decoder, I named its functions to match the upstream C TurboPFor library, but now I want to get away from names like p4ndec256v32 — they make sense from the TurboPFor perspective, but for Debian Code Search, we can use cleaner names.

Before writing any code, I audited how DCS uses integer compression / decompression.

API design: BlockEncoder, BlockDecoder and streaming

In Debian Code Search, we have the following usage patterns:

• Partial Indexing: When a new package (or package version) enters Debian, all of its (text) files are indexed. If the hello-2.12.3-1 package (hypothetically) contained only hello.c with printf("hello!\n");, we would assign document ID 1 to hello.c and store in the partial index that trigrams pri, rin, int, ntf, etc. are all found in doc 1 (hello.c).
• Full Index Merging: The many thousands of partial index files (for each Debian package) are combined into a small handful of large index files: When searching, it would be expensive to consult thousands of indexes. To merge multiple partial index files into one larger index (which can then be efficiently queried), we need to re-encode the partial index files: what used to be document ID 1 in the partial index might be document ID 2531 in the full index.
• Querying (searching): When users enter search queries, these queries need to be answered as quickly as possible. The relevant entries in the full indexes are decoded (in parallel).

For reading the index, we do keep the decoded uint32s fully in memory, so we only need DecodeN(input []byte, output []uint32) (read int), a function that reads len(output) values (uint32) from input and returns how many bytes it consumed.

For writing the index (both in partial indexing, and when merging), keeping the entire index in memory is prohibitively expensive, so we need a streaming API, for decoding and for encoding.

Ultimately, I converged on the following API:

package pforenc

type BlockEncoder struct {
    // scratch buffers can go here
}

// EncodeBlock encodes len(vals)<=256 uint32s into dest (one TurboPFor block).
func (*BlockEncoder) EncodeBlock(dest []byte, vals []uint32) []byte {}

// EncodeN calls EncodeBlock in a loop.
func (*BlockEncoder) EncodeN(dest []byte, vals []uint32) []byte {}

type StreamEncoder struct {
  be   BlockEncoder
  vals [256]uint32
  // scratch buffers
}

// if full, you need to call [EncodeBlock]
func (*StreamEncoder) Add(val uint32) (full bool)

// EncodeBlock must be called after all data was [Add]ed.
//
// Write the returned buffer to file or send it over the network;
// it is only valid until the next [EncodeBlock] call.
func (*StreamEncoder) EncodeBlock() []byte {
  if se.n == 0 { return nil } // turn an extra EncodeBlock into a no-op
  // …
}

This API (the decoder works similarly) allows us to process data in TurboPFor format without any memory allocations. The types are not safe for concurrent use by multiple goroutines. The zero value is ready to be used. For the streaming API, the result only stays valid until the next call.

Initial Implementation

Before we can optimize anything, we need a working decoder and encoder. The decoder already exists: my goturbopfor teaching decoder. Next up, I needed an encoder.

Writing a TurboPFor encoder has a delightfully simple starting point: You can encode all values at bit width 32, in little endian, at which point you only need to add a one-byte TurboPFor block header every 256 values and you’re done:

func (be *BlockEncoder) EncodeN(dest []byte, vals []uint32) []byte {
  for len(vals) > 0 {
    chunk := min(len(vals), 256)
    dest = be.EncodeBlock(dest, vals[:chunk])
    vals = vals[chunk:]
  }
  return dest
}

func (be *BlockEncoder) EncodeBlock(dest []byte, vals []uint32) []byte {
  const bitWidth = 32
  dest = append(dest, bitWidth)
  for _, val := range vals {
    dest = binary.LittleEndian.AppendUint32(dest, val)
  }
  return dest
}

Of course, this is a terribly inefficient compressor, so after the first commit, the real work starts: implement each block type until the compression matches the original C TurboPFor implementation (same output file size), or in other words: do the reverse of the decoder.

1. The TurboPFor bitpacking block type (bitpacking implementation commit) encodes a bit stream of variable bit width (where the bit width is in range 0 ≤ bitWidth ≤ 32) in little endian byte order. By scanning all values and choosing the smallest bit width that allows representing all values, this technique saves disk space (compresses).
2. The bitpacking with exceptions block type (bitpacking with exceptions implementation commit) determines two bit widths: one for values, the other bit width for encoding exceptions. This allows choosing a lower bit width (that does not cover all values) compared to the bitpacking block type. A bitmap encodes whether a value has an exception or not.
3. The bitpacking with VB exceptions block type (bitpacking with VB exceptions implementation commit) is a variant which does not use an exception bitmap and encodes exceptions using a variable byte integer encoding. This is more efficient when there are few exceptions (less than 20) or the exceptions are very different in bit width compared to the other values.
4. Lastly, the constant block type (constant implementation commit) stores just one value on disk. This is useful for all-zero or all-one blocks, for example.

I found it interesting to realize that the main work of the encoder is to scan the input values and choose the optimal block type, whereas the actual encoding itself is cheap in comparison.

At this point, we can look at performance and see that the Go encoder is at 76% of the C encoder.

In all honesty, I could have probably stopped here, but now that the milestone of a viable replacement was reached, I got curious to see how far it would be possible to push the encoder (how much work to reach C speeds?) and afterwards, the decoder, too.

Setup

The microarchitecture level: set GOAMD64

The microarchitecture of a CPU determines which instructions it provides, and that includes not just SIMD instruction sets (like AVX2), but also other useful instructions like LZCNT (Leading Zero Count), which can be used to implement math/bits.Len32 more efficiently, which the TurboPFor encoder needs to call on every input value to determine the ideal bit width.

Let’s walk through how to set the microarchitecture level when using Go on 64-bit x86 (x86-64).

Go uses the GOARCH environment variable to configure the target compilation architecture, and I am using the value amd64 to select 64-bit x86 (AVX2 and AVX512 are instruction sets found on x86-64 CPUs). With GOARCH=amd64, the architecture-specific variable GOAMD64 configures the microarchitecture level for which to compile and Go 1.18 introduced these 4 different levels:

GOAMD64=v1 (default): The baseline.
Exclusively generates instructions that all 64-bit x86 processors can execute.

GOAMD64=v2: all v1 instructions,
plus CMPXCHG16B, LAHF, SAHF, POPCNT, SSE3, SSE4.1, SSE4.2, SSSE3.

GOAMD64=v3: all v2 instructions,
plus AVX, AVX2, BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, OSXSAVE.

GOAMD64=v4: all v3 instructions,
plus AVX512F, AVX512BW, AVX512CD, AVX512DQ, AVX512VL.

In 2026, I generally recommend compiling with GOAMD64=v3 so that functions like bits.OnesCount8 are compiled into intrinsics (POPCNT) instead of using a lookup table.

For Intel CPUs, setting GOAMD64=v3 means your programs will only start on Haswell CPUs (2013) or newer; for AMD CPUs that means Zen 1 (2017) or newer.

In this specific case (DCS), I am even compiling with GOAMD64=v4. The v4 microarchitecture level requires AVX512, which means AMD Zen 4, Zen 5 or newer (Intel’s story is… complicated). Luckily, both my main development PC (Zen 5) and the Debian Code Search server (Zen 4) are recent enough. Setting GOAMD64=v4 has little effect on Go 1.27 itself: the only change is that maps use one less instruction (VPBROADCASTB instead of PSHUFB). But compiling with GOAMD64=v4 allows us to move one more feature check from runtime to compile time, see SIMD build tags.

It makes sense to set the microarchitecture level in your benchmark setup so that you don’t measure the slow fallback implementations. I use export GOAMD64=v4 in my Makefile.

Benchmarking setup

Go’s built-in testing package contains support for benchmarks which are written in functions of the form func BenchmarkXxx(b *testing.B). The simplest way to run such benchmarks is go test -bench=., but I ended up configuring a few convenience make targets, which write results to bench.txt and compare against baseline.txt (the previous commit’s results, usually), using the very useful benchstat tool.

GOTEST=go test

# -count=6 gives p≤0.002 in benchstat:
# https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
BENCHFLAGS=-run=^$$ -bench=. -benchtime=200000x -count=6

# use taskset -c1 to always pin to the same single core,
# avoiding accidental scheduling on different cores on
# mixed-core CPUs like the Ryzen 9 9950X3D.
TASKSET=taskset -c 1
BENCH=$(TASKSET) $(GOTEST) $(BENCHFLAGS)

.PHONY: all test bench bench-baseline bench-relative

all: test

bench: test
	$(BENCH) | tee bench.txt
# Compares compression ratio between C and Go implementation
	benchstat -col /impl -row '/n /vals' -filter '-/impl:go-stream .unit:(encoded-bytes)' bench.txt
# Compares performance between C (cgo) and Go implementation
	benchstat -col /impl -row '/n /vals' -filter '.unit:(Mval/s)' bench.txt

bench-baseline: test
	$(BENCH) | tee baseline.txt

bench-relative: test
	$(BENCH) | tee bench.txt
	benchstat -filter '-/impl:go-stream .unit:(encoded-bytes)' baseline.txt bench.txt
	benchstat -filter '/impl:go .unit:(Mval/s)' baseline.txt bench.txt

The encoded-bytes and Mval/s units are custom metrics I am reporting from the various sub-benchmarks, which are arranged such that I can filter / report them with benchstat.

The main encoder (and decoder) benchmarks compare 3 different implementations (cgo, Go, Go with the StreamEncoder API) with a number of benchmark cases that are designed to cover the different block types and contain a similar mix of values as what we see in Debian Code Search:

// reportMetrics adds Mval/s and encoded-bytes metrics to all benchmarks.
func reportMetrics(b *testing.B, n int, nencoded int) {
   b.ReportMetric(float64(nencoded), "encoded-bytes")
   b.ReportMetric(float64(b.N*n)/1e6/b.Elapsed().Seconds(), "Mval/s")
}

// BenchmarkEncode/n=<N>/vals=<testcase>/impl=<c|go|go-stream>
//
// e.g. BenchmarkEncode/n=2048/vals=one-constant/impl=go-stream
func BenchmarkEncode(b *testing.B) {
   for _, tc := range allBenchCases() {
     n := len(tc.vals)
     b.Run(fmt.Sprintf("n=%d/vals=%s", n, tc.name), func(b *testing.B) {
       b.Run("impl=c", func(b *testing.B) {
         b.ReportAllocs()
         var encoded []byte
         buf := make([]byte, turbopfor.EncodingSize(n))
         for b.Loop() {
           encoded = turbopfor.P4nenc256v32Buf(buf, tc.vals)
         }
         reportMetrics(b, n, len(encoded))
       })
       b.Run("impl=go", func(b *testing.B) {
         b.ReportAllocs()
         var be BlockEncoder
         var encoded []byte
         buf := make([]byte, 0, turbopfor.EncodingSize(n))
         for b.Loop() {
           encoded = be.EncodeN(buf, tc.vals)
         }
         reportMetrics(b, n, len(encoded))
       })
       b.Run("impl=go-stream", func(b *testing.B) {
         b.ReportAllocs()
         var se StreamEncoder
         var encoded int
         for b.Loop() {
           encoded = 0
           for _, val := range tc.vals {
             if se.Add(val) {
               encoded += len(se.EncodeBlock())
             }
           }
           encoded += len(se.EncodeBlock())
         }
         reportMetrics(b, n, encoded)
       })
     })
   }
}

CPU counters: perf

Go has included excellent performance tooling for many years, see the “Profiling Go Programsâ€� blog post (2011) for an example of how to use pprof, a sampling profiler. This profiler can help track down which part of a program runs slow, or where memory allocations happen.

Once you identified the slow part of a program, how do you know why it’s slow?

To learn more about the specific bottlenecks your program encounters, you can consult your CPU’s hardware performance counters. For example, you could check the branch predictor counters to see if your program is slow due to a high number of branch mispredicts.

On Linux, the perf tool is the best way to access the CPU hardware performance counters. A good starting point for working with perf is the documentation on “Top-down analysis with the perf toolâ€�, which describes the optimization method that Intel established.

In my Makefile, I set up two perf targets:

# GOTEST and TASKSET like shown in the earlier benchmarking setup section:
GOTEST=go test -pgo=encode.cpuprof
TASKSET=taskset -c 1
PERFBENCHFLAGS=-test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=200000x

# Use perf(1) to capture AMD IBS (the equivalent to Intel PEBS)
# PipelineL1 is roughly equivalent to Intel TopdownL1
perf:
	$(GOTEST) -c
	$(TASKSET) perf stat -M PipelineL1 ./pforenc.test -test.run=^$$ $(PERFBENCHFLAGS)
	sudo perf record -F 4999 -e ibs_op// --call-graph fp ./pforenc.test -test.run=^$$ $(PERFBENCHFLAGS)
	sudo chmod 644 perf.data

# 488281 iterations × 2048 values = 1.000e9 values, so counter/1e9 = per value.
perf-per-value:
	$(GOTEST) -c
	$(TASKSET) perf stat -x, -e cycles:u,instructions:u,branches:u,branch-misses:u ./pforenc.test -test.run=^$$ -test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=488281x 2>&1 >/dev/null | awk -F, '{printf "%-16s %6.2f /val\n", $$3, $$1/1e9}'

The perf-per-value numbers are high level numbers that indicate how much work the implementation is doing. Reducing the number usually increases speed.

To see the counters for each instruction (and source code lines), I use make perf, followed by perf report. A quick shortcut is perf annotate, which directly shows the hottest function.

Optimizations (scalar)

Let’s first see how far we can get without reaching for SIMD instructions.

(The examples are not necessarily in commit order, but cherry-picked for clarity.)

Profile-Guided Optimization (PGO)

PGO stands for Profile-Guided Optimization and is a feature that Go introduced as a preview in Go 1.20 (released in February 2023) and shipped as ready for general production use in Go 1.21 (released in August 2023).

The idea is to capture a CPU profile that records where your program spends most of its CPU time, which you then provide to the Go compiler to give it more data to make better decisions.

Most importantly, this way the Go compiler can inline functions much more aggressively than its usual heuristics allow, which does have a measurably positive effect in my series of optimization commits. Another optimization that a PGO profile allows the compiler to do is conditional devirtualization — but our TurboPFor code does not use any interfaces.

My strategy is to enable PGO before doing any other optimizations, so that we have the full inlining budget available that PGO gives us, and can measure the effect of other commits clearly.

Surprisingly, turning on PGO actually decreases our performance (-13% geomean), but a closer investigation reveals that we just got unlucky. Let me explain.

Aside from inlining and conditional devirtualization, PGO also influences alignment: The Go compiler sets PCALIGNMAX(64, 31) on the first block of a loop (the “loop bodyâ€�) for all loops in hot functions (per the PGO profile), i.e. Go will insert up to 31 bytes of padding to make the block land on a 64-byte boundary. Documentation like AMD’s “Software Optimization Guide for the AMD Zen5 Microarchitectureâ€� (2024, #58455) explicitly recommends aligning hot loops that way:

[…] for hot loops, some further knowledge of trade-offs can be helpful. Because the processor can read an aligned 64-byte fetch block every cycle, it is suggested to either align the start of the loop to the beginning of a 64-byte cache line […]

Indeed, when compiling with -gcflags=all=-d=alignhot=0 to disable the alignment, performance remains as good as without PGO. How can the padding hurt more than help? The answer is: It’s not the padding itself! It’s a side-effect of the padding moving instructions to different addresses.

In the unlucky arrangement, a macro-fused CMPQ+JGE instruction pair now ends up exactly on a 32-byte boundary. However, the Go compiler ensures fused branch sequences must never cross or end at a 32-byte boundary to fix Intel erratum SKX102 (discussion: Go issue #35881) by inserting NOPs.

This NOP padding, unlike the loop alignment padding, is not free; these extra instructions slow down our otherwise dispatch-bound loops.

Because the commits after the PGO enabling commit change the code, this unlucky situation is avoided for the rest of the optimization series (by chance).

Reducing memory allocations

Memory allocations are quite expensive, at least in comparison to encoding/decoding integers, so I followed my usual strategy of first reducing memory allocations as much as possible.

In my goturbopfor teaching decoder, whenever the code needed a scratch buffer, it would allocate it right then and there with make():

// p4dec32 decodes one block of TurboPFor-encoded 32 bit ints
func (d *decoder) p4dec32(input []byte, output []uint32) (read int) {
    // …
  switch blockType {
  case blockBitpackingExceptions:
    bx, input := input[0], input[1:]
    n := len(output)

    exmap := input
    nex := 0 // number of exceptions
    for i := 0; i < n; i++ {
      if exmap[i/8]&(1<<uint(i%8)) != 0 {
        nex++
      }
    }
    input = input[(n+7)/8:]

    exceptions := make([]uint32, nex)
    input = input[bitunpack32(input, exceptions, bx):]
    input = input[d.bitunpack(input, output, b):]

    for i := 0; i < n; i++ {
      if exmap[i/8]&(1<<uint(i%8)) != 0 {
        output[i] += exceptions[0] << b
        exceptions = exceptions[1:]
      }
    }

    return before - len(input)
  }
}

The Go compiler can turn make(T, n) calls into stack allocations, if n is known at compile-time. But, in this case nex is not known at compile-time. We can verify that Go calls into the runtime (runtime.makeslice) by dumping the object code (assembly) with source annotated (-S):

% cd ~/go/src/github.com/stapelberg/goturbopfor
% git reset --hard 49b7c05cc61e77f0257568eb73833467714d2b4a
% go test -c  # go1.27.0
% go tool objdump -S goturbopfor.test | perl -nlE 'say if /p4dec32/ .. /^$/'
TEXT github.com/stapelberg/goturbopfor.(*decoder).p4dec32(SB) /home/michael/go/src/github.com/stapelberg/goturbopfor/goturbopfor.go
func (d *decoder) p4dec32(input []byte, output []uint32) (read int) {
  0x549f60		4c8da42460ffffff	LEAQ 0xffffff60(SP), R12
  0x549f68		4d3b6610		CMPQ R12, 0x10(R14)
  0x549f6c		0f86d9070000		JBE 0x54a74b
  0x549f72		55			PUSHQ BP
  0x549f73		4889e5			MOVQ SP, BP
  0x549f76		4881ec18010000		SUBQ $0x118, SP
  0x549f7d		48899c2430010000	MOVQ BX, 0x130(SP)
  0x549f85		4889b42448010000	MOVQ SI, 0x148(SP)
	if len(output) == 0 {
  0x549f8d		4d85c0			TESTQ R8, R8
  0x549f90		0f84a7030000		JE 0x54a33d
  0x549f96		660f1f840000000000	NOPW 0(AX)(AX*1)
  0x549f9f		90			NOPL
[…]
		exceptions := make([]uint32, nex)
  0x54a4be		488d057bec1700		LEAQ 0x17ec7b(IP), AX
  0x54a4c5		4c89fb			MOVQ R15, BX
  0x54a4c8		4889d9			MOVQ BX, CX
  0x54a4cb		e8f0ddf3ff		CALL runtime.makeslice(SB)
[…]

An easy speed-up was to avoid allocations through reuse (in goturbopfor). In the DCS pfordec package (with the improved API design), I ended up with a vals [256]uint32 field in the StreamDecoder type, which brings us from 773 Mval/s to 858 Mval/s on the debian-mix:

% benchstat -filter '/impl:go /vals:debian-mix .unit:(Mval/s)' \
  baseline.txt bench.txt
goos: linux
goarch: amd64
pkg: github.com/Debian/dcs/internal/turbopfor/pfordec
cpu: AMD Ryzen 9 9950X3D 16-Core Processor
           │ baseline.txt │             bench.txt              │
           │    Mval/s    │   Mval/s     vs base               │
n=2048        1.089k ± 1%   1.175k ± 0%   +7.85% (p=0.002 n=6)
n=2039         974.7 ± 0%   1046.0 ± 0%   +7.32% (p=0.002 n=6)
n=160          434.9 ± 1%    513.6 ± 5%  +18.11% (p=0.002 n=6)
geomean        772.9         857.7       +10.98%

Aside from the speed-up, avoiding memory allocations is generally nice in benchmarks because it removes the garbage collector from the equation and makes it less likely that your benchmarks get other processes OOM-killed on the same machine.

Generics for bit width specialization

In general, we want to make it easy for the compiler to understand as much as possible about our algorithm. Consider this bitpack implementation:

func bitpack(dest []byte, vals []uint32, bitWidth int) []byte {
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have int
  for _, val := range vals {
    acc |= uint64(val&mask) << have
    have += bitWidth
    for have >= 32 {
      dest = binary.LittleEndian.AppendUint32(dest, uint32(acc))
      acc >>= 32
      have -= 32
    }
  }
  for have > 0 {
    dest = append(dest, byte(acc))
    acc >>= 8
    have -= 8
  }
  return dest
}

Let’s think through what determines the iterations and control flow this function uses:

1. The number of input values (vals), but not their actual value.
2. The bit width to pack into (bitWidth).

With a bit of careful rearrangement, we can provide the compiler with both, a fixed number of input values (say, 32), and a bit width, both known at compile time. Why is this worthwhile? Because we can manually unroll the loop, let the compiler eliminate much of the repetition and get much faster compiled code as a result!

Let’s first fix the number of input values to 32 and rewrite the loop to calculate the position offsets within dest instead of changing dest on each value (with AppendUint32):

func bitpack32Unrolled(dest []byte, vals *[32]uint32, bitWidth int) {
  // only one bounds check for 32 values
  dest = dest[: 4*bitWidth : 4*bitWidth]
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have, pos int
  // Manually unrolled loop starts here.
  // Each iteration is identical except for the vals[x] index.
  acc |= uint64(vals[0]&mask) << have
  have += bitWidth
  if have >= 32 {
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
  }

  // vals[1] .. vals[30] elided for brevity

  // Each loop iteration is 8 lines of Go code, so for 32 input values,
  // bitpack32Unrolled contains 8*32 = 256 lines of code.

  acc |= uint64(vals[31]&mask) << have
  have += bitWidth
  if have >= 32 {
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
  }

  // have == 0; for all bitWidths
}

Next, we want to specialize not just for 32 input values, but also for each of the 32 bit widths.

Can we do better than hand-copying bitpack32Unrolled 32 times (= 8192 lines of Go code)?

Yes, we can use Go generics to help us with the code generation!

In Go, array types like [4]byte (not slices like []byte!) contain the length of the array as part of their type, meaning [1]byte (an array of length 1) is a different type than [2]byte.

Instead of passing the bit width as a function parameter, we can declare 32 different types (one for each bit width) and recover the bit width (at compile time!) from the type system:

type bitWidthT interface {
  [1]byte | [2]byte | [3]byte | [4]byte | [5]byte |
  [6]byte | [7]byte | [8]byte | [9]byte | [10]byte |
  [11]byte | [12]byte | [13]byte | [14]byte | [15]byte |
  [16]byte | [17]byte | [18]byte | [19]byte | [20]byte |
  [21]byte | [22]byte | [23]byte | [24]byte | [25]byte |
  [26]byte | [27]byte | [28]byte | [29]byte | [30]byte |
  [31]byte | [32]byte
}

func bitpack32Unrolled[T bitWidthT](dest []byte, vals *[32]uint32) {
  var zero T
  bitWidth := len(zero)                  // known at compile time
  dest = dest[: 4*bitWidth : 4*bitWidth] // make cap known at compile time
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have, pos int
  // Manually unrolled loop starts here.
  // Each iteration is identical except for the vals[x] index.
  acc |= uint64(vals[0]&mask) << have
  have += bitWidth
  if have >= 32 {
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
  }

  // vals[1] .. vals[31] elided for brevity
}