Planet Javasummit

September 23, 2017

etbeConverting Mbox to Maildir

MBox is the original and ancient format for storing mail on Unix systems, it consists of a single file per user under /var/spool/mail that has messages concatenated. Obviously performance is very poor when deleting messages from a large mail store as the entire file has to be rewritten. Maildir was invented for Qmail by Dan Bernstein and has a single message per file giving fast deletes among other performance benefits. An ongoing issue over the last 20 years has been converting Mbox systems to Maildir. The various ways of getting IMAP to work with Mbox only made this more complex.

The Dovecot Wiki has a good page about converting Mbox to Maildir [1]. If you want to keep the same message UIDs and the same path separation characters then it will be a complex task. But if you just want to copy a small number of Mbox accounts to an existing server then it’s a bit simpler.

Dovecot has a mb2md.pl script to convert folders [2].

cd /var/spool/mail
mkdir -p /mailstore/example.com
for U in * ; do
  ~/mb2md.pl -s $(pwd)/$U -d /mailstore/example.com/$U
done

To convert the inboxes shell code like the above is needed. If the users don’t have IMAP folders (EG they are just POP users or use local Unix MUAs) then that’s all you need to do.

cd /home
for DIR in */mail ; do
  U=$(echo $DIR| cut -f1 -d/)
  cd /home/$DIR
  for FOLDER in * ; do
    ~/mb2md.pl -s $(pwd)/$FOLDER -d /mailstore/example.com/$U/.$FOLDER
  done
  cp .subscriptions /mailstore/example.com/$U/ subscriptions
done

Some shell code like the above will convert the IMAP folders to Maildir format. The end result is that the users will have to download all the mail again as their MUA will think that every message had been deleted and replaced. But as all servers with significant amounts of mail or important mail were probably converted to Maildir a decade ago this shouldn’t be a problem.

September 22, 2017

Worse Than FailureError'd: Choose Wisely

"I'm not sure how I can give feedback on this course, unless, figuring out this matrix is actually a final exam," wrote Mads.

 

Brian W. writes, "Sorry that you're not happy with our spam, but before you go...just one more."

 

"I was looking forward to getting this Gerber Dime, but I guess I'll have to wait till they port it to OS X," wrote Peter G.

 

"Deleting 7 MB frees up 6.66 GB? I smell a possible unholy alliance," Mike W. writes.

 

Bill W. wrote, "I wonder if they're wanting to know to what degree I'm 'not at all likely' to recommend Best Buy to friends and family?"

 

"So, is this a new way for the folks at WebEx to make sure that you don't get bad answers?" writes Andy B.

 

[Advertisement] Otter, ProGet, BuildMaster – robust, powerful, scalable, and reliable additions to your existing DevOps toolchain.

XKCDThread

September 21, 2017

Worse Than FailureTales from the Interview: The In-House Developer

James was getting anxious to land a job that would put his newly-minted Computer Science degree to use. Six months had come to pass since he graduated and being a barista barely paid the bills. Living in a small town didn't afford him many local opportunities, so when he saw a developer job posting for an upstart telecom company, he decided to give it a shot.

Lincoln Log Cabin 2

We do everything in-house! the posting for CallCom emphasized, piquing James' interest. He hoped that meant there would be a small in-house development team that built their systems from the ground up. Surely he could learn the ropes from them before becoming a key contributor. He filled out the online application and happily clicked Submit.

Not 15 minutes later, his phone rang with a number he didn't recognize. Usually he just ignored those calls but he decided to answer. "Hi, is James available?" a nasally female voice asked, almost sounding disinterested. "This is Janine with CallCom, you applied for the developer position."

Caught off guard by the suddenness of their response, James wasn't quite ready for a phone screening. "Oh, yeah, of course I did! Just now. I am very interested."

"Great. Louis, the owner, would like to meet with you," Janine informed him.

"Ok, sure. I'm pretty open, I usually work in the evenings so I can make most days work," he replied, checking his calendar.

"Can you be here in an hour?" she asked. James managed to hide the fact he was freaking out about how to make it in time while assuring her he could be.

He arrived at the address Janine provided after a dangerous mid-drive shave. He felt unprepared but eager to rock the interview. The front door of their suite gave way to a lobby that seemed more like a walk-in closet. Janine was sitting behind a small desk reading a trashy tabloid and barely looked up to greet him. "Louis will see you now," she motioned toward a door behind the desk and went back to reading barely plausible celebrity rumors.

James stepped through the door into what could have been a walk-in closet for the first walk-in closet. A portly, sweaty man presumed to be Louis jumped up to greet him. "John! Glad you could make it on short notice. Have a seat!"

"Actually, it's James..." he corrected Louis, while also forgiving the mixup. "Nice to meet you. I was eager to get here to learn about this opportunity."

"Well James, you were right to apply! We are a fast growing company here at CallCom and I need eager young talent like you to really drive it home!" Louis was clearly excited about his company, growing sweatier by the minute.

"That sounds good to me! I may not have any real-world experience yet, but I assure you that I am eager to learn from your more senior members," James replied, trying to sell his potential.

Louis let out a hefty chuckle at James' mention of senior members. "Oh you mean stubborn old developers who are set in their ways? You won't be finding those around here! I believe in fresh young minds like yours, unmolded and ready to take the world by storm."

"I see..." James said, growing uneasy. "I suppose then I could at least learn how your code is structured from your junior developers? The ones who do your in-house development?"

Louis wiped his glistening brow with his suit coat before making the big revelation. "There are no other developers, James. It would just be you, building our fantastic new computer system from scratch! I have all the confidence in the world that you are the man for the job!"

James sat for a moment and pondered what he had just heard. "I'm sorry but I don't feel comfortable with that arrangement, Louis. I thought that by saying you do everything in-house, that implied there was already a development team."

"What? Oh, heavens no! In-house development means we let you work from home. Surely you can tell we don't have much office space here. So that's what it means. In. House. Got it?

James quickly thanked Louis for his time and left the interconnected series of closets. In a way, James was glad for the experience. It motivated him to move out of his one horse town to a bigger city where he eventually found employment with a real in-house dev team.

[Advertisement] Otter, ProGet, BuildMaster – robust, powerful, scalable, and reliable additions to your existing DevOps toolchain.

Tim Bray You Might Be Evil

Or at least, your employer might be. Over the years we in the tech sector have gotten used to being well-regarded. After all, we make people’s lives better, on balance. That’s changing. At the moment it’s rumblings from thought leaders, not pervasive popular anger. The other thing that’s new is that they’re thought leaders who are progressives and liberals; just like most of us in the tech professions. It notably involves the M-word and those of us on the inside need to be thinking about it.

The general public, by and large, love reading the news of their friends and the world on Facebook, buying stuff cheap on Amazon, using Google maps and mail for free, and using recent Windows releases at work.

But these days, it seems like every other day I read a chilling anti-tech rant, usually written by someone smart, articulate, and (like me) leftist. Here are a few recent offerings:

  • A Serf on Google’s Farm: About how the advertising end of the business fails to combine customer support and scalability, and what it feels like to be a minor customer: “It’s a bit like being assimilated by the Borg. You get cool new powers. But having been assimilated, if your implants were ever removed, you’d certainly die.” “Google is so big and so powerful that even when it’s trying to do something good, it can be dangerous and frightening.”

  • You can’t quit Facebook, a Twitter rant by Matt Stoller: “Your data and identity is trapped inside a machine that spends huge $$$ to addict and manipulate you, your friends, and your culture.” “We cannot as individual consumers resist the tens of billions spent to manipulate us. But we as citizens can do so through politics.”

  • Margrethe Vestager’s growing American fan club, on the savvy Eurocrat who’s been tormenting Google, Apple, and Facebook: “There is growing concern … about bigness and size, and power because power corrupts absolutely.”

  • There’s Blood In The Water In Silicon Valley: “This sort of political change happens slowly until it happens fast. Uber provided a new model for a transformative tech giant to crash through with a dark, negative brand.”

  • Facebook’s Heading Toward a Bruising Run-In With the Russia Probe, interesting not so much for the Russian angle but for the visceral contempt for Facebook: “Facebook’s ‘internal policies’ amount to a kind of Stepford Wives version of civic liberalism and speech and privacy rights, the outward form of the things preserved while the innards have been gutted and replaced by something entirely different, an aggressive and totalizing business model which in many ways turns these norms and values on their heads.”

Disclosures

I’m not going to claim my curation is unbiased. I left out Microsoft because, weirdly, nobody seems to hate Microsoft that much any more. I certainly don’t. I left out Twitter because it’s not actually a company, it’s a dysfunctional non-profit that accidentally provides a valuable service. I left out Amazon (although it appears in a few of those pieces) because I’d have no chance of coming anywhere near balance.

The M-Word

It’s “Monopoly” of course. If you follow the links above and read, the authors come at the tech giants from every which direction, but always ending up banging out the monopoly melody. Sometimes they say “corporate concentration” or another euphemism, because being anti-monopoly sounds kind of old-fashioned; and anyhow, shouldn’t you be talking about Comcast or United?

Not any more. A lot of smart people think it’s good economics, good policy, and good politics to aim the anti-trust gun at the tech sector. I’m not saying they’re wrong. I’m also not predicting that they’ll get any traction, particularly in the America where the short-term focus has to be on combating Nazis and pussy-grabbers.

But this is a trend that nobody in technology leadership should ignore.

September 20, 2017

Worse Than FailureCodeSOD: A Dumbain Specific Language

I’ve had to write a few domain-specific-languages in the past. As per Remy’s Law of Requirements Gathering, it’s been mostly because the users needed an Excel-like formula language. The danger of DSLs, of course, is that they’re often YAGNI in the extreme, or at least a sign that you don’t really understand your problem.

XML, coupled with schemas, is a tool for building data-focused DSLs. If you have some complex structure, you can convert each of its features into an XML attribute. For example, if you had a grammar that looked something like this:

The Source specification obeys the following syntax

source = ( Feature1+Feature2+... ":" ) ? steps

Feature1 = "local" | "global"

Feature2 ="real" | "virtual" | "ComponentType.all"

Feature3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

Feature4 = "first" | "last" | "DayAllocation.all"

If features are specified, the order of features as given above has strictly to be followed.

steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

oneOrMoreNameSteps = nameStep ( "." nameStep ) *

zeroOrMoreNameSteps = ( nameStep "." ) *

nameStep = "#" name

name is a string of characters from "A"-"Z", "a"-"z", "0"-"9", "-" and "_". No umlauts allowed, one character is minimum.

componentSteps is a list of valid values, see below.

Valid 'componentSteps' are:

- GlobalValue
- Product
- Product.Brand
- Product.Accommodation
- Product.Accommodation.SellingAccom
- Product.Accommodation.SellingAccom.Board
- Product.Accommodation.SellingAccom.Unit
- Product.Accommodation.SellingAccom.Unit.SellingUnit
- Product.OnewayFlight
- Product.OnewayFlight.BookingClass
- Product.ReturnFlight
- Product.ReturnFlight.BookingClass
- Product.ReturnFlight.Inbound
- Product.ReturnFlight.Outbound
- Product.Addon
- Product.Addon.Service
- Product.Addon.ServiceFeature

In addition to that all subsequent steps from the paths above are permitted, that is 'Board', 
'Accommodation.SellingAccom' or 'SellingAccom.Unit.SellingUnit'.
'Accommodation.Unit' in the contrary is not permitted, as here some intermediate steps are missing.

You could turn that grammar into an XML document by converting syntax elements to attributes and elements. You could do that, but Stella’s predecessor did not do that. That of course, would have been work, and they may have had to put some thought on how to relate their homebrew grammar to XSD rules, so instead they created an XML schema rule for SourceAttributeType that verifies that the data in the field is valid according to the grammar… using regular expressions. 1,310 characters of regular expressions.

<xs:simpleType>
    <xs:restriction base="xs:string">
            <xs:pattern value="(((Scope.)?(global|local|current)\+?)?((((ComponentType.)?
(real|virtual))|ComponentType.all)\+?)?((((Hierarchy.)?(self|ancestors|descendants))|Hierarchy.all)\+?)?
((((DayAllocation.)?(first|last))|DayAllocation.all)\+?)?:)?(#[A-Za-z0-9\-_]+(\.(#[A-Za-z0-9\-_]+))*|(#[A-Za-z0-
9\-_]+\.)*
(ThisComponent|GlobalValue|Product|Product\.Brand|Product\.Accommodation|Product\.Accommodation\.SellingAccom|Prod
uct\.Accommodation\.SellingAccom\.Board|Product\.Accommodation\.SellingAccom\.Unit|Product\.Accommodation\.Selling
Accom\.Unit\.SellingUnit|Product\.OnewayFlight|Product\.OnewayFlight\.BookingClass|Product\.ReturnFlight|Product\.
ReturnFlight\.BookingClass|Product\.ReturnFlight\.Inbound|Product\.ReturnFlight\.Outbound|Product\.Addon|Product\.
Addon\.Service|Product\.Addon\.ServiceFeature|Brand|Accommodation|Accommodation\.SellingAccom|Accommodation\.Selli
ngAccom\.Board|Accommodation\.SellingAccom\.Unit|Accommodation\.SellingAccom\.Unit\.SellingUnit|OnewayFlight|Onewa
yFlight\.BookingClass|ReturnFlight|ReturnFlight\.BookingClass|ReturnFlight\.Inbound|ReturnFlight\.Outbound|Addon|A
ddon\.Service|Addon\.ServiceFeature|SellingAccom|SellingAccom\.Board|SellingAccom\.Unit|SellingAccom\.Unit\.Sellin
gUnit|BookingClass|Inbound|Outbound|Service|ServiceFeature|Board|Unit|Unit\.SellingUnit|SellingUnit))"/>
    </xs:restriction>
</xs:simpleType>
</xs:union>

There’s a bug in that regex that Stella needed to fix. As she put it: “Every time you evaluate it a few little kitties die because you shouldn’t use kitties to polish your car. I’m so, so sorry, little kitties…”

The full, unexcerpted code is below, so… at least it has documentation. In two languages!

<xs:simpleType name="SourceAttributeType">
                <xs:annotation>
                        <xs:documentation xml:lang="de">
                Die Source Angabe folgt folgender Syntax

                        source = ( Eigenschaft1+Eigenschaft2+... ":" ) ? steps

                        Eigenschaft1 = "local" | "global"

                        Eigenschaft2 ="real" | "virtual" | "ComponentType.all"

                        Eigenschaft3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

                        Eigenschaft4 = "first" | "last" | "DayAllocation.all"

                        Falls Eigenschaften angegeben werden muss zwingend die oben angegebene Reihenfolge der Eigenschaften eingehalten werden.

                        steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

                        oneOrMoreNameSteps = nameStep ( "." nameStep ) *

                        zeroOrMoreNameSteps = ( nameStep "." ) *

                        nameStep = "#" name

                        name ist eine Folge von Zeichen aus der Menge "A"-"Z", "a"-"z", "0"-"9", "-" und "_". Keine Umlaute. Mindestens ein Zeichen

                        componentSteps ist eine Liste gültiger Werte, siehe im folgenden

                Gültige 'componentSteps' sind zunächst:

                        - GlobalValue
                        - Product
                        - Product.Brand
                        - Product.Accommodation
                        - Product.Accommodation.SellingAccom
                        - Product.Accommodation.SellingAccom.Board
                        - Product.Accommodation.SellingAccom.Unit
                        - Product.Accommodation.SellingAccom.Unit.SellingUnit
                        - Product.OnewayFlight
                        - Product.OnewayFlight.BookingClass
                        - Product.ReturnFlight
                        - Product.ReturnFlight.BookingClass
                        - Product.ReturnFlight.Inbound
                        - Product.ReturnFlight.Outbound
                        - Product.Addon
                        - Product.Addon.Service
                        - Product.Addon.ServiceFeature

                Desweiteren sind alle Unterschrittfolgen aus obigen Pfaden erlaubt, also 'Board', 'Accommodation.SellingAccom' oder 'SellingAccom.Unit.SellingUnit'.
                'Accommodation.Unit' hingegen ist nicht erlaubt, da in diesem Fall einige Zwischenschritte fehlen.

                                </xs:documentation>
                        <xs:documentation xml:lang="en">
                                The Source specification obeys the following syntax

                                source = ( Feature1+Feature2+... ":" ) ? steps

                                Feature1 = "local" | "global"

                                Feature2 ="real" | "virtual" | "ComponentType.all"

                                Feature3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

                                Feature4 = "first" | "last" | "DayAllocation.all"

                                If features are specified, the order of features as given above has strictly to be followed.

                                steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

                                oneOrMoreNameSteps = nameStep ( "." nameStep ) *

                                zeroOrMoreNameSteps = ( nameStep "." ) *

                                nameStep = "#" name

                                name is a string of characters from "A"-"Z", "a"-"z", "0"-"9", "-" and "_". No umlauts allowed, one character is minimum.

                                componentSteps is a list of valid values, see below.

                                Valid 'componentSteps' are:

                                - GlobalValue
                                - Product
                                - Product.Brand
                                - Product.Accommodation
                                - Product.Accommodation.SellingAccom
                                - Product.Accommodation.SellingAccom.Board
                                - Product.Accommodation.SellingAccom.Unit
                                - Product.Accommodation.SellingAccom.Unit.SellingUnit
                                - Product.OnewayFlight
                                - Product.OnewayFlight.BookingClass
                                - Product.ReturnFlight
                                - Product.ReturnFlight.BookingClass
                                - Product.ReturnFlight.Inbound
                                - Product.ReturnFlight.Outbound
                                - Product.Addon
                                - Product.Addon.Service
                                - Product.Addon.ServiceFeature

                                In addition to that all subsequent steps from the paths above are permitted, that is 'Board', 'Accommodation.SellingAccom' or 'SellingAccom.Unit.SellingUnit'.
                                'Accommodation.Unit' in the contrary is not permitted, as here some intermediate steps are missing.

                        </xs:documentation>
                </xs:annotation>
                <xs:union>
                        <xs:simpleType>
                                <xs:restriction base="xs:string">
                                        <xs:pattern value="(((Scope.)?(global|local|current)\+?)?((((ComponentType.)?(real|virtual))|ComponentType.all)\+?)?((((Hierarchy.)?(self|ancestors|descendants))|Hierarchy.all)\+?)?((((DayAllocation.)?(first|last))|DayAllocation.all)\+?)?:)?(#[A-Za-z0-9\-_]+(\.(#[A-Za-z0-9\-_]+))*|(#[A-Za-z0-9\-_]+\.)*(ThisComponent|GlobalValue|Product|Product\.Brand|Product\.Accommodation|Product\.Accommodation\.SellingAccom|Product\.Accommodation\.SellingAccom\.Board|Product\.Accommodation\.SellingAccom\.Unit|Product\.Accommodation\.SellingAccom\.Unit\.SellingUnit|Product\.OnewayFlight|Product\.OnewayFlight\.BookingClass|Product\.ReturnFlight|Product\.ReturnFlight\.BookingClass|Product\.ReturnFlight\.Inbound|Product\.ReturnFlight\.Outbound|Product\.Addon|Product\.Addon\.Service|Product\.Addon\.ServiceFeature|Brand|Accommodation|Accommodation\.SellingAccom|Accommodation\.SellingAccom\.Board|Accommodation\.SellingAccom\.Unit|Accommodation\.SellingAccom\.Unit\.SellingUnit|OnewayFlight|OnewayFlight\.BookingClass|ReturnFlight|ReturnFlight\.BookingClass|ReturnFlight\.Inbound|ReturnFlight\.Outbound|Addon|Addon\.Service|Addon\.ServiceFeature|SellingAccom|SellingAccom\.Board|SellingAccom\.Unit|SellingAccom\.Unit\.SellingUnit|BookingClass|Inbound|Outbound|Service|ServiceFeature|Board|Unit|Unit\.SellingUnit|SellingUnit))"/>
                                </xs:restriction>
                        </xs:simpleType>
                </xs:union>
</xs:simpleType>
[Advertisement] Release! is a light card game about software and the people who make it. Play with 2-5 people, or up to 10 with two copies - only $9.95 shipped!

XKCDUSB Cables

September 19, 2017

Worse Than FailurePoor Shoe

OldShoe201707

"So there's this developer who is the end-all, be-all try-hard of the year. We call him Shoe. He's the kind of over-engineering idiot that should never be allowed near code. And, to boot, he's super controlling."

Sometimes, you'll be talking to a friend, or reading a submission, and they'll launch into a story of some crappy thing that happened to them. You expect to sympathize. You expect to agree, to tell them how much the other guy sucks. But as the tale unfolds, something starts to feel amiss.

They start telling you about the guy's stand-up desk, how it makes him such a loser, such a nerd. And you laugh nervously, recalling the article you read just the other day about the health benefits of stand-up desks. But sure, they're pretty nerdy. Why not?

"But then, get this. So we gave Shoe the task to minify a bunch of JavaScript files, right?"

You start to feel relieved. Surely this is more fertile ground. There's a ton of bad ways to minify and concatenate files on the server-side, to save bandwidth on the way out. Is this a premature optimization story? A story of an idiot writing code that just doesn't work? An over-engineered monstrosity?

"So he fires up gulp.js and gets to work."

Probably over-engineered. Gulp.js lets you write arbitrary JavaScript to do your processing. It has the advantage of being the same language as the code being minified, so you don't have to switch contexts when reading it, but the disadvantage of being JavaScript and thus impossible to read.

"He asks how to concat JavaScript, and the room tells him the right answer: find javascripts/ -name '*.js' -exec cat {} \; > main.js"

Wait, what? You blink. Surely that's not how Gulp.js is meant to work. Just piping out to shell commands? But you've never used it. Maybe that's the right answer; you don't know. So you nod along, making a sympathetic noise.

"Of course, this moron can't just take the advice. Shoe has to understand how it works. So he starts googling on the Internet, and when he doesn't find a better answer, he starts writing a shell script he can commit to the repo for his 'jay es minifications.'"

That nagging feeling is growing stronger. But maybe the punchline is good. There's gotta be a payoff here, right?

"This guy, right? Get this: he discovers that most people install gulp via npm.js. So he starts shrieking, 'This is a dependency of mah script!' and adds node.js and npm installation to the shell script!"

Stronger and stronger the feeling grows, refusing to be shut out. You swallow nervously, looking for an excuse to flee the conversation.

"We told him, just put it in the damn readme and move on! Don't install anything on anyone else's machines! But he doesn't like this solution, either, so he finally just echoes out in the shell script, requires npm. Can you believe it? What a n00b!"

That's it? That's the punchline? That's why your friend has worked himself into a lather, foaming and frothing at the mouth? Try as you might to justify it, the facts are inescapable: your friend is TRWTF.

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!

September 18, 2017

XKCDObsolete Technology

Tim Bray Photos of the Sky

I mean No Man’s Sky the game, which I’ve been playing again lately. It’s been accused of being mostly a platform for generating cheesy sci-fi book covers, but that’s not true, and also I love cheesy sci-fi book covers, so this is mostly to show you some. With a few words on the game.

Starship flying in No Man’s Sky

Oh, just flying my starship over the landscape of
a planet I recently discovered and named. As one does.

Updates

No Man’s Sky launched just over a year ago, preceded by a massive hype wave and followed by howls of disappointment when it fell short of expectations. There’ve been three major updates, each with massive additions. and it’s becoming increasingly like a “normal” game, with missions and NPC’s and economies and so on.

Its biggest charm probably remains that it’s an easy game, occasional flashes of excitement, but mostly just cruising along from star to star, enjoying the views.

Nice view in No Man’s Sky Desert-scape, No Man’s Sky Nice view in No Man’s Sky

The game is starting to add inklings of multi-player, and multiple players are self-organizing, most notably to create The Galactic Hub, which I hope to reach some day.

Aliens!

In NMS they’re, more than anything, fun. The planets have their own procedurally-generated menageries, but there are three main species you can actually have relationships with, and learn the languages of. There’s lots of back-story that you soak up as you move through the game. Here’s a Gek.

a Gek in No Man’ Sky

Aren’t they cute?

The beasts in NMS are chatty and come in all sizes, shapes, and colors. They’re more playful than scary. And like I said, it’s an easy game, any creature that attacks you is pretty well toast. But, here’s Jabba the Flowerpott.

No Man’s Sky with fat alien

My experience

In the picture just above, that’s my current starship, parked on the left. Nice ride, eh? For the cognoscenti: A decent little Samamoga 27-slot B-class fighter, modest stats but a reliable pirate-killer. Which sort of sums up my playing style. I’m busy, with a job and family; thus not a serious gamer. One of the really nice things about NMS is you can do it for an hour here and an hour there and keep progressing and never really get stuck.

Ship parked on peak, No Man’s Sky

Good parking spot!

I restarted after the 1.3 update (like a lot of others) and I’m going to keep playing till I get a freighter.

Gender

The three primary alien races you deal with do not participate in the human notion of gender. Thus, when individuals are mentioned in text (and there’s a lot of text in NMS) it’s always in a scrupulous third-person: They, them, their. It’s amazing how quickly you get used to it.

On the other hand, the beasts that populate the planets are gendered, and while male and female do appear, one also finds Asymptotic, Asymmetric, Orthogonal, Prime, Radical, Indeterminate,and Vectorised.

So the authors of NMS are having some real postmodern twenty-first century fun; which on balance is a fair description of the whole game. I’m still on board.

September 15, 2017

XKCDWhat to Bring

September 10, 2017

Tim Bray Canadian Tax Wrangling

Our media, pro and social, echo with blasts of self-righteous anger over proposed legislation which would eliminate a few popular tax dodges. Weirdly, I see no-one arguing the other side; that the tax proposals are reasonable. I think I’m qualified to make that argument, so I will. [If you’re not Canadian, you can probably stop reading here.]

The proposed tax changes

They’re summarized pretty well here. Basically, if you have a business and it’s incorporated — say you’re a doctor, lawyer, contractor, accountant, that kind of thing — you can use your corporation for tax tricks, the effect being that you pay less tax on the same income.

The tricks have been well-known for years; every competent financial advisor and wealth manager in the country tells every client about them.

Disclosures

(And I wish a few more people publishing op-eds on this subject would offer theirs.)

  1. I have a good income and pay a lot of tax.

  2. I’ve had capital gains over the years, from a successful startup and a couple of lucky investments.

  3. My wife and I have a corporation, which is useful in supporting the consulting businesses both of us have run from time to time. But we’ve hardly ever been able to use any tricks; one time by accident and acqui-hire the corporation ended up with pre-IPO Twitter shares, and we did save some tax bucks when we sold them.

  4. I was a co-founder of a company that currently has just over 14,000 employees.

  5. I didn’t vote for the government that’s making the proposals.

My feelings on tax generally? My bills are shocking, but on the other hand I’m a heavy direct user of government services: roads, bridges, public transit, bikeways, libraries, athletic facilities, public broadcasting, health care, emergency first responders. Plus, having grown up in a third-world country, I have a hearty appreciation for the rule of law and the social safety net. So yeah, I’d like a lower tax bill and I might vote for a party that had specific proposals on paying for one with cutbacks on the parts of government I don’t like.

But I generally do think that people who make about as much money as me should pay about as much tax as me.

The complaints

It amounts to a bunch of well-off people who are going to start paying the same tax rate I do explaining why they shouldn’t have to, because they work hard, create jobs, and are just all-around nice people.

Here’s an anesthesiologist who’s so upset that, he says, he and the other doctors are going to go mini-John Galt and start working less.

Here’s a guy who pays $225K in tax and says he’s “already contributing FAR more than his fair share”.

The complainers complain that they don’t get a pension. Neither do I; in fact nobody does any more, except civil servants.

They also bitch about not getting paid vacation or “any other benefits”. Oddly enough, when we’re on vacation in sunny destinations, we tend to encounter lots of doctors and lawyers and contractors and so on, so somehow they manage to get away. And now that we’re older and know a few retirees, I gotta say that the small-biz-owner contingent seems well-represented among those in their “golden years” where by “golden” I mean “rolling in dough”.

Also, do the math: Anyone who’s paying $225K in tax is a very well-off individual, with lots of room to save for retirement, and enough cash-flow to visit Maui or Cabo every winter.

But they do have a point; It wouldn’t seem completely insane to me if there were tax deductions for those whose employment situation is low on benefits. But that should be done explicitly, rather than nudge-nudge-wink-winking at small-biz corporate fiddles.

That aside, I’m sorry but I just don’t see any reason why someone who makes my kind of money shouldn’t have to pay my kind of taxes, just because they’re a small business. And in the testimonies I read, they come across as hypocrisy-drenched greedheads.

If you want lower taxes, I think there are two good courses of action: Move to a jurisdiction that has them, or do the political work to elect a party that will cut them (and correspondingly, services). Spare me, please, the “supply-side” fantasy in which tax cuts generate increased revenue. It’s been tried.

So, tax all income equally?

Maybe. In Canada, the tax on capital gains is half that on salary. The idea is to encourage people to start businesses and invest in other businesses. I’ve created businesses and invested in them too, so it’d be easy for me to say “Look, it works” and maybe I’d be right. But maybe not; I probably would have gone ahead and done those things anyhow. It’s amazing how many tax policies officially aimed at one good end or another seem mostly to result in rich people paying less.

Also I’ve started to hear progressive economists arguing against this kind of thing. And a hard-line policy of “income is income” has the advantage that people don’t pay tax consultants to fool around and try to make one kind of income look like another. So it’s not a slam-dunk.

What’s going to happen?

The politics is interesting. The people lined up against the tax proposals have loud, well-funded voices, and donate lots to political parties, so their concerns are going to get careful attention from legislators. And it’s easy to convince the public that any government tax tactic is a grubby revenue grab.

But at the end of the day the noise is coming from a bunch of fat cats trying to pay less tax than other fat cats, based on what feel like really flimsy arguments. They exude entitlement. I think the politics on this one could go either way.

September 09, 2017

etbeObserving Reliability

Last year I wrote about how great my latest Thinkpad is [1] in response to a discussion about whether a Thinkpad is still the “Rolls Royce” of laptops.

It was a few months after writing that post that I realised that I omitted an important point. After I had that laptop for about a year the DVD drive broke and made annoying clicking sounds all the time in addition to not working. I removed the DVD drive and the result was that the laptop was lighter and used less power without missing any feature that I desired. As I had installed Debian on that laptop by copying the hard drive from my previous laptop I had never used the DVD drive for any purpose. After a while I got used to my laptop being like that and the gaping hole in the side of the laptop where the DVD drive used to be didn’t even register to me. I would prefer it if Lenovo sold Thinkpads in the T series without DVD drives, but it seems that only the laptops with tiny screens are designed to lack DVD drives.

For my use of laptops this doesn’t change the conclusion of my previous post. Now the T420 has been in service for almost 4 years which makes the cost of ownership about $75 per year. $1.50 per week as a tax deductible business expense is very cheap for such a nice laptop. About a year ago I installed a SSD in that laptop, it cost me about $250 from memory and made it significantly faster while also reducing heat problems. The depreciation on the SSD about doubles the cost of ownership of the laptop, but it’s still cheaper than a mobile phone and thus not in the category of things that are expected to last for a long time – while also giving longer service than phones usually do.

One thing that’s interesting to consider is the fact that I forgot about the broken DVD drive when writing about this. I guess every review has an unspoken caveat of “this works well for me but might suck badly for your use case”. But I wonder how many other things that are noteworthy I’m forgetting to put in reviews because they just don’t impact my use. I don’t think that I am unusual in this regard, so reading multiple reviews is the sensible thing to do.

September 04, 2017

Tim Bray Murder at Adolf’s Cottage

I recently read Prussian Blue by Philip Kerr. It’s good — a Fifties-Iron-Curtain spy thriller gracefully mashed up with a pre-war murder mystery set in Hitler’s Bavarian country getaway, Berghof. It’s a repeat appearance for Kerr’s Bernie Gunther, an appealingly hard-boiled veteran socialist cop who finds himself working for National Socialist management.

Prussian Blue by Philip Kerr

And yep, there are Nazis in this story; the real not tiki-torch flavor. A couple of mega-Nazis, Bormann and Heydrich, and some relatively minor odd fish like Karl Brandt and Gerdy Troost.

Nazis are convenient for a novelist, because they are reliably evil and twisted, so he can economize on characterization and leave room for plot and atmospherics. Having said that, he cuts Troost a little slack; you’d have to be a better historian than me to know whether that’s a travesty or not.

Did I mention atmospherics? You’ve come to the right place. While substantial parts of the novel aren’t at der Führer’s country digs, the ones that are bask in deep you-are-there weirdness, and (I thought) a pretty deep take on what it’d be like for the yokels in a pretty backwater when the Fascist architects come to put up palaces. Some, predictably, react a lot better than others. But Kerr doesn’t pretend the presence of any serious resistance.

Hey, and here’s a fascinating little sidelight: The lead contractor for the Berghof construction was a company called Polensky & Zöllner; some part of which apparently still exists, albeit in Abu Dhabi. Their motto: “All knowledge comes from experience”. And yeah, they’ll still build you a villa.

Oops, I got distracted. This is a fine piece of writing, intense and atmospheric and instructive and just really good fun. Recommended.

August 01, 2017

etbeQEMU for ARM Processes

I’m currently doing some embedded work on ARM systems. Having a virtual ARM environment is of course helpful. For the i586 class embedded systems that I run it’s very easy to setup a virtual environment, I just have a chroot run from systemd-nspawn with the --personality=x86 option. I run it on my laptop for my own development and on a server my client owns so that they can deal with the “hit by a bus” scenario. I also occasionally run KVM virtual machines to test the boot image of i586 embedded systems (they use GRUB etc and are just like any other 32bit Intel system).

ARM systems have a different boot setup, there is a uBoot loader that is fairly tightly coupled with the kernel. ARM systems also tend to have more unusual hardware choices. While the i586 embedded systems I support turned out to work well with standard Debian kernels (even though the reference OS for the hardware has a custom kernel) the ARM systems need a special kernel. I spent a reasonable amount of time playing with QEMU and was unable to make it boot from a uBoot ARM image. The Google searches I performed didn’t turn up anything that helped me. If anyone has good references for getting QEMU to work for an ARM system image on an AMD64 platform then please let me know in the comments. While I am currently surviving without that facility it would be a handy thing to have if it was relatively easy to do (my client isn’t going to pay me to spend a week working on this and I’m not inclined to devote that much of my hobby time to it).

QEMU for Process Emulation

I’ve given up on emulating an entire system and now I’m using a chroot environment with systemd-nspawn.

The package qemu-user-static has staticly linked programs for emulating various CPUs on a per-process basis. You can run this as “/usr/bin/qemu-arm-static ./staticly-linked-arm-program“. The Debian package qemu-user-static uses the binfmt_misc support in the kernel to automatically run /usr/bin/qemu-arm-static when an ARM binary is executed. So if you have copied the image of an ARM system to /chroot/arm you can run the following commands like the following to enter the chroot:

cp /usr/bin/qemu-arm-static /chroot/arm/usr/bin/qemu-arm-static
chroot /chroot/arm bin/bash

Then you can create a full virtual environment with “/usr/bin/systemd-nspawn -D /chroot/arm” if you have systemd-container installed.

Selecting the CPU Type

There is a huge range of ARM CPUs with different capabilities. How this compares to the range of x86 and AMD64 CPUs depends on how you are counting (the i5 system I’m using now has 76 CPU capability flags). The default CPU type for qemu-arm-static is armv7l and I need to emulate a system with a armv5tejl. Setting the environment variable QEMU_CPU=pxa250 gives me armv5tel emulation.

The ARM Architecture Wikipedia page [2] says that in armv5tejl the T stands for Thumb instructions (which I don’t think Debian uses), the E stands for DSP enhancements (which probably isn’t relevant for me as I’m only doing integer maths), the J stands for supporting special Java instructions (which I definitely don’t need) and I’m still trying to work out what L means (comments appreciated).

So it seems clear that the armv5tel emulation provided by QEMU_CPU=pxa250 will do everything I need for building and testing ARM embedded software. The issue is how to enable it. For a user shell I can just put export QEMU_CPU=pxa250 in .login or something, but I want to emulate an entire system (cron jobs, ssh logins, etc).

I’ve filed Debian bug #870329 requesting a configuration file for this [1]. If I put such a configuration file in the chroot everything would work as desired.

To get things working in the meantime I wrote the below wrapper for /usr/bin/qemu-arm-static that calls /usr/bin/qemu-arm-static.orig (the renamed version of the original program). It’s ugly (I would use a config file if I needed to support more than one type of CPU) but it works.

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
  if(setenv("QEMU_CPU", "pxa250", 1))
  {
    printf("Can't set $QEMU_CPU\n");
    return 1;
  }
  execv("/usr/bin/qemu-arm-static.orig", argv);
  printf("Can't execute \"%s\" because of qemu failure\n", argv[0]);
  return 1;
}

July 31, 2017

etbeRunning a Tor Relay

I previously wrote about running my SE Linux Play Machine over Tor [1] which involved configuring ssh to use Tor.

Since then I have installed a Tor hidden service for ssh on many systems I run for clients. The reason is that it is fairly common for them to allow a server to get a new IP address by DHCP or accidentally set their firewall to deny inbound connections. Without some sort of VPN this results in difficult phone calls talking non-technical people through the process of setting up a tunnel or discovering an IP address. While I can run my own VPN for them I don’t want their infrastructure tied to mine and they don’t want to pay for a 3rd party VPN service. Tor provides a free VPN service and works really well for this purpose.

As I believe in giving back to the community I decided to run my own Tor relay. I have no plans to ever run a Tor Exit Node because that involves more legal problems than I am willing or able to deal with. A good overview of how Tor works is the EFF page about it [2]. The main point of a “Middle Relay” (or just “Relay”) is that it only sends and receives encrypted data from other systems. As the Relay software (and the sysadmin if they choose to examine traffic) only sees encrypted data without any knowledge of the source or final destination the legal risk is negligible.

Running a Tor relay is quite easy to do. The Tor project has a document on running relays [3], which basically involves changing 4 lines in the torrc file and restarting Tor.

If you are running on Debian you should install the package tor-geoipdb to allow Tor to determine where connections come from (and to not whinge in the log files).

ORPort [IPV6ADDR]:9001

If you want to use IPv6 then you need a line like the above with IPV6ADDR replaced by the address you want to use. Currently Tor only supports IPv6 for connections between Tor servers and only for the data transfer not the directory services.

Data Transfer

I currently have 2 systems running as Tor relays, both of them are well connected in a European DC and they are each transferring about 10GB of data per day which isn’t a lot by server standards. I don’t know if there is a sufficient number of relays around the world that the share of the load is small or if there is some geographic dispersion algorithm which determined that there are too many relays in operation in that region.

November 18, 2014

Kelvin Lawrence - personal25 Years of the World Wide Web

I have been so busy that I am a few days late putting this post together but hopefully better late than never!

A few days ago, hard though it is to believe, the Worldwide Web, that so many of us take for granted these days, celebrated it's 25th anniversary. Created in 1989 by Sir Tim Berners-Lee , for many of us, "Web" has become as essential in our daily lives as electricity or natural gas. Built from its earliest days upon the notion of open standards the Web has become the information backbone of our current society. My first exposure, that I can remember, to the concept of the Web was in the early 1990s when I was part of the OS/2 team at IBM and we put one of the earliest browsers, Web Explorer, into the operating system and shipped it. Back then, an HTML web page was little more than text, images, animated GIFs and most importantly of all hyperlinks. I was also involved with the team that did some of the early ports of Netscape Navigator to OS/2 and I still recall being blown away by some of what I saw that team doing upon some of my many visits to Netscape in California what seems like a lifetime ago now!

 From those modest but still highly effective beginnings, the Web and most importantly perhaps, the Web browser, has evolved into the complete business and entertainment platform that it is today.


The Web, and open standards, have been part of my personal and work life ever since. I am honored to have been a small part of the evolution of the web myself. I have worked on a number of different projects with great people from all over the World under the auspices of the W3C for longer than I care to remember! I have done a lot of fun things in my career, but one of the highlights was definitely working with so many talented people on the original Scalable Vector Graphics (SVG) specification which is now supported by most of the major browsers and of course you can find my library of SVGsamples here on my site.

It is also so fitting that the latest evolution of Web technology, the finished HTML 5 specification was announced to coincide with the 25th anniversary of the Web.

I could write so much more about what the Web has meant to me but most of all I think my fondest memory will always be all of the great friends I have met and the large number of very talented people that I have had the good fortune to work with through our joint passion to make the Web a better and even more open, place.

Happy (slightly belated) Birthday Worldwide Web and here's to the next 25!

November 13, 2014

Kelvin Lawrence - personalAsian Tiger Mosquitoes

The weather has been unusually cold for the time of year the last day or so. I was actually hoping that if we get a hard freeze it will kill off for now the Asian Tiger mosquitoes that we have been overrun with this year. However I have my doubts as apparently, unlike other mosquitoes, their eggs, which they lay in vegetation and standing water, can survive a harsh winter. They apparently got into the USA in a shipment of waterlogged tires (tyres for my UK friends) some time ago and they are now spreading more broadly. They are covered in black and white stripes and look quite different than the regular "brown" colored mosquitoes we are used to seeing here. They are also a lot more aggressive. They bite all day long (not just at dusk) and even bite animals but definitely prefer humans. It has got so bad that we have had to pay to have our yard sprayed regularly almost all year so that we even have a chance to sit outside and enjoy our yard. These nasty little guys also transmit the chikungunya virus for which I currently believe there is no vaccine. It's not usually fatal but does have some nasty symptoms if you are unlucky enough to catch it. Here's a link to a WebMD write up on these little nasties.

November 12, 2014

Kelvin Lawrence - personalPink Floyd's Endless River - The End of an Era

I just purchased the new Pink Floyd CD from Amazon which includes a free digital download as well. I have been listening to it while I work today. Given the way the album was put together (using material the late Richard Wright recorded almost 20 years ago during the making of The Division Bell) much of the music is immediately familiar. I definitely also hear flashbacks to Wish You Were Here, Dark Side of the Moon and many other albums as well. It's mostly instrumental and there is a lot of it - four sides if you buy the vinyl version!! A lot of the music has an almost eerie tone to it - definitely a good one for the headphones with the lights off. It's a really good listen but left me feeling sad in a way, in a good way I guess, as much of their music has been the backdrop to the last 40 years or so of my life and this is definitely the end of a musical era as supposedly this is the last album the band plan to release. It has a bit of everything for Pink Floyd fans, especially those who like some of the "more recent" albums. Don't expect a bunch of rocking songs that you will be humming along to all day but as a complete work, listened to end to end, I found it very moving. Very much not your modern day pop tune and thank goodness for that!

October 26, 2014

Kelvin Lawrence - personalSeven years post cancer surgery

Today marks another big milestone for me. It has now been seven years since my cancer surgery. As always, I am grateful for all of my family, friends and doctors and every minute that I get to spend with them.

June 03, 2009

Software Summit June 3, 2009: The Finale of Colorado Software Summit

To Our Friends and Supporters,

In these challenging economic times, business has slowed, many companies have had to resort to layoffs and/or closures, and everyone has been tightening their belts. Unfortunately, Colorado Software Summit has not been immune to this downturn. As have so many companies and individuals, we too have experienced a severe decline in our business, and as a result we are not able to continue producing this annual conference.

This year would have been our 18th conference, and we had planned to continue through our 20th in 2011, but instead we must end it now.

Producing this conference has been a wonderful experience for us, truly a labor of love, and we have been extremely privileged to have been able to do well by doing good.  We are very proud of the many people whose careers flourished through what they learned here, of the extensive community we built via the conference, and of the several businesses that were begun through friendships made here. We treasure the friends we made, and we consider them to be part of our extended family. Just as in any family, we celebrated with them through joyous life events and grieved with them through tragic ones.

This is a sad time for us, of course, but not overwhelmingly so. It's sort of the feeling you have when your son leaves for college, or your daughter gets married. You knew it was coming someday, but it is here much sooner than you imagined, and the sadness is sweetened with the joy you had in all that has come before.

We have been privileged to have created a thriving community of friends who met for the first time at the conference, and we want that community to continue. We hope that all of you will stay in touch with us and with each other, and that the Colorado Software Summit community will continue as a source of wisdom and friendship to all of you. If you have ever attended one of our conference, we hope you will consider joining the Colorado Software Summit LinkedIn group as one means to keep in touch.

With our very best wishes for your future, and with unbounded gratitude for your support,

- Wayne and Peggy Kovsky -

All presentations from Colorado Software Summit 2008 have been posted.

May 18, 2009

Software Summit May 17, 2009: Additions to Preliminary Agenda for Colorado Software Summit 2009

We have posted additions to the preliminary agenda for Colorado Software Summit 2009, in two formats:

We will continue to post additions to this agenda during the coming weeks. Please check back here from time to time for additions and/or changes to the agenda, or subscribe to our RSS feed to receive notifications of updates automatically.

Presentations from the 2008 Conference

We have posted presentations for these speakers from Colorado Software Summit 2008:

Presentations from Colorado Software Summit 2008 will be posted periodically throughout the year.

May 03, 2009

Software Summit May 3, 2009: Additions to Preliminary Agenda for Colorado Software Summit 2009

We have posted additions to the preliminary agenda for Colorado Software Summit 2009, in two formats:

We will continue to post additions to this agenda during the coming weeks. Please check back here from time to time for additions and/or changes to the agenda, or subscribe to our RSS feed to receive notifications of updates automatically.

Presentations from the 2008 Conference

We have posted presentations for these speakers from Colorado Software Summit 2008:

Presentations from Colorado Software Summit 2008 will be posted periodically throughout the year.

April 26, 2009

Software Summit April 25, 2009: Preliminary Agenda for Colorado Software Summit 2009

We have posted the preliminary agenda for Colorado Software Summit 2009, in two formats:

We will continue to post additions to this agenda during the coming weeks. Please check back here from time to time for additions and/or changes to the agenda, or subscribe to our RSS feed to receive notifications of updates automatically.

Presentations from the 2008 Conference

We have posted presentations for these speakers from Colorado Software Summit 2008:

Presentations from Colorado Software Summit 2008 will be posted periodically throughout the year.