Planet Bozo

July 20, 2018

Worse Than FailureError'd: Upon Reaching a Certain Age...

"Evidently, once you hit 55, LinkedIn thinks you'll age until your buffer overflows," writes Jonathan L.

 

"I started out looking for shower gel, but now, thanks to Google, I'm considering if a GBIC in Cadet Blue is worth the extra money," writes Robin M.

 

Matthew B. wrote, "So, an article about AI shows that the AI behind generating the summary rasied an exception. Maybe the AIs aren't speaking to each other?"

 

"Wait...did I just fail a Turing Test?" writes Daniel.

 

Rob J. wrote, "I got a 2 on a vision test but apparently only people from Krypton or blind people test on it, because there were very large negative and positive scores."

 

Pieter V. writes, "Thankfully this combo error didn't occur on the plane I took."

 

[Advertisement] Forget logs. Next time you're struggling to replicate error, crash and performance issues in your apps - Think Raygun! Installs in minutes. Learn more.

XKCDSports Champions

July 19, 2018

Worse Than FailureClassic WTF: Flawless Compilation

Just today I was joking with my co-workers: I had written software for which we had no viable test hardware, but the code compiled, therefore I was done. The difference is I was joking… --Remy (Originally)

Back in the heady days of Internet speculation, the giant retailer JumboStores contracted with Fred’s software company, TinyWeb, to develop the region’s first web-based supermarket. Customers would be able to assemble carts online and receive their groceries the next day.

The virtual supermarket had to communicate with JumboStores’s inventory system in real-time. The former was bleeding-edge web technology, the latter a cobweb-laden mainframe with no external point of access.

“How will we get around this?” Fred asked early in the specification process.

“We can stage an intermediate server.” Nick, a programmer from JumboStores IT, assured him around a mouthful of doughnut. “You guys send your requests there, we’ll write software to forward them to the mainframe and back.”
Engine overhauled
Fred was optimistic. Both companies were *nix shops; the JumboStores IT department were his geek kindred. Equally optimistic, JumboStores management scheduled a live media demo several months out, well after the estimated project completion date.

Deadlines slipped, as they are wont to do. The week before the big demo, the online supermarket still wasn’t ready. TinyWeb had implemented the website and database back-end, but JumboStores’ relay software lagged behind. At the urging of multiple strata of nervous managers, Fred took an emergency trip to JumboStores to investigate.

“We don’t know, man, we just don’t know.” The confident Nick of months prior shook now, leading Fred to his cubicle. “We coded the application. We debugged until it compiled without errors. When we run it- core dump!” He threw up his hands, then dropped into his swivel chair. “We’ve been pestering IBM support, but they haven’t been very helpful.”

“Well, why would they be?” Fred frowned, pausing at the cube threshold. “I mean, who knows what might be wrong with the code?”

“Nothing’s wrong with it. It compiles!”

“So? It could still have errors.”

Nick swiveled around to face him. “Dude. It compiles.

Fred faltered in the wake of Nick’s earnest insistence. “That… doesn’t mean the code is perfect.” He all but fell into the spare chair presented to him. “How do I explain this?” Am I actually trying to explain this? To a programmer? “Let’s say you’re building an engine.”

“This isn’t an engine,” Nick said. “It just passes-“

“No, a car engine! OK? You have all the parts spread out on the desk here.” He waved his arm out over a layer of branded cube toys and post-it notes. “You’ve never built an engine from scratch before, but you have a blueprint with pictures and directions, so you grab your wrench and your welder and whatever, and go to town. At the end, all the parts get used up, and the result looks vaguely engine-like. Still, would you expect to drop it under the hood and have it start up flawlessly the first time you turn over the ignition?”

Nick stared. “I… don’t see what this has to do with anything.”

Fred refrained from smacking his forehead. “Uh, OK. Forget the engine. It’s like sheet music. Just because all the dots are on the staff doesn’t mean it’s the song you want.“

“Dude! The compiler would bug out if there were any problems.” Nick graciously omitted the Duh.

Fred took one last chance. “No- it’s like, if you were building a house. Just because all the parts fit together doesn’t mean it will stand up.”

Nick’s face brightened. “It’s like the home inspector! I see what you mean."

“If that works for you…” Fred said, carefully.

After long consideration, Fred took the intermediate server back home to TinyWeb for some down-to-the-wire recoding, resulting in a flawless demo for the press. JumboStores was delighted.

With their collaboration at an end, Fred wondered how JumboStores IT would ever manage on their own.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

July 18, 2018

Worse Than FailureClassic WTF: The Mega Bureaucracy

Part of the reason we need a summer break is because we simply don't have the organizational skills of this particular company. I wonder if they sell consulting. Original -- Remy

Photo credit: 'digicla' at Flickr At my daytime corporate-type job, if I need to even sneeze in the general direction of a production environment, I need both a managerial and customer approvals with documentation solemnly stating that I thoroughly tested my changes and swear on a stack of MSDN licenses and O'Reilly books that I am NOT going to break anything as a result of my changes. Sure, the whole thing is a pain (and admittedly, a necessary evil), but what Bruce W. has to go through beats the pants off of anything I've ever had to go through.

For the most part, Bruce loves his job. He gets to work with a lot of intelligent and motivated people. He has been developing a new system to support a new product that has the possibility of earning his division several million dollars per year and saving the corporate parent several hundred thousand dollars per year. The net effect on the corporate parent's bottom line will be quite nice. He developed a Web front end while a fellow developer put together the data feeds. The initial development work was estimated to take about six weeks; pretty good since we only had eight weeks to work with.

However, Bruce works in a very large corporation (70,000 plus employees through out the US and several countries) and IT for the corporation has been highly centralized to the world headquarters. Smaller IT work, like the development and support for only a single division, isn't centralized but must pass through the central Mega Bureaucracy for approval and placement on the centralized servers.

...and Bruce needs their "help" to officially set up his environments.

You see, while Bruce and his group can test all day long on their local computers and servers, any kind of "live" environments must be created, blessed, and centralized by the Mega Bureaucracy. They're bigger, badder, and have more connections than anybody in your division's rank-and-file. Remember: in the Mega Bureaucracy, processes and procedures are to be followed, respected, and if necessary worshipped. Oh, and forget even thinking of installing Web services on one of the existing centralized servers. That would bring down the wrath of the entire blessed Bureaucracy for changing the purpose of an existing machine without first going through Mega Change Server Process.

Here's a brief overview of what Bruce had to go through to get four (one each for development, testing, staging, and production) Windows-based Web servers:

Week 1 - At the same time Bruce's group started the project he went to procure the servers. He was told that all he needed to do was put in a Service Request with the Windows Server Team and they would get what we needed. However, that request is cancelled because when the Windows Server Team saw that the servers were for a new application they said, "Whoa, you have violated rule #38,991 of the Mega Bureaucracy! New applications must go through the Process for Application Implementation and Navigation."

Bruce starts into the fill the first two PAIN forms (one being 20 pages long with 150 questions), sends them off to the server team, and immediately receives a response that, no, do not directly send PAIN forms to the group they go to. Instead, open a project with the Mega Bureaucracy's project tracking system, attach the forms and THEN assign the project to the group.

A few days later, he receives word that the project has been accepted, slotted, and a project manager assigned. Bruce figures, "Cool, now we are moving! I'll have my servers in no time!" He and his boss have a conference call with the PM and express to him the time critical nature of these servers. The PM agrees to push them forward saying that the request isn't complex and shouldn't take much effort.


Week 2 - Bruce receives the initial project estimate and immediately replies with his approval.


Week 4 - Bruce calls the PM to find out what's going on. He says that due to staffing cuts only a handful of requests are being processed at a time. Despite being reminded that this project is literally worth millions, he says that other projects are ahead of us and that this is simply how things are. Bruce boss escalates the issue to the head of IT for the entire division who just happens to be a member of the Project Approving Council and supposedly has the power to move the project forward.


Week 6 - Only three weeks until the promised delivery date, Bruce learns that the project still has not moved. His boss fires off a series of emails saying that the app is about to go live on a system that will earn the company millions of dollars that is running on a desktop machine sitting in a cubicle.


Week 7 - The system is now fully coded. Bruce is walking around, shaking his head, saying to himself "We have done user testing and end-to-end testing on a desktop machine-based server!"


Week 8 - The new system goes live and is serving dozens of customers daily. The difference between Production and Test environments is a Post-it Note. Power strips and network hub are carefully labeled "DO NOT TOUCH! HIGH VOLTAGE!" to prevent cleaning staff misfeance.


Week 10 - Bruce and the Windows Server Team finally have the project kick off meeting for the servers. About 15 of the 30 minute call was spent with Bruce repeatedly saying, "All I need is a Windows Server with IIS and .NET. I do not need a database server, no access to the mainframe, no massive SAN space, no Internet access, no interplanetary probe, just servers." "BUT", they say, "You stated on page 16, question 113 that your application uses a database. Where will that database come from?" Bruce explains again, "We are using existing databases assigned to our group. The database is outside of the scope of the project of setting up four Web servers."

Week 12 - Bruce and the Windows Server Team get together for their status meeting. The server team says they haven't budged since last meeting. Why? Everyone says, "Well, we're just waiting for the other shoe to drop and this becoming a big, complex, hairy project requiring massive time." Bruce once again states that all they he needs is four Web servers. Nothing more. The server design engineer says, "Wow, that is pretty simple. Shouldn't take too long at all."


Week 14 - Bruce has another status meeting with the PM and the server engineer. The engineer has put together the required diagram of the requested infrastucture and states that he only had to change a handful of things from the initial template. He says that everything should be ok and once they have the infrastructure readiness, the server builds can start. Bruce thinks, "Finally! All the other people initially assigned to the project must have realized that building four web servers isn't that big if a deal! ...haven't they?"


Week 18 - The head of IT for our division finds out that we are still waiting. Heads start rolling...even poor Bruce's. "WHY DIDN'T YOU CALL ME SIX %($$!*& WEEKS AGO???" the IT head blasts.


Week 19 - The servers are built (it only took 2 days to build them!) and are signed off for production support.

Week 20 - Bruce distributes the application URL pointing to the brand new servers.

Through all of this Bruce learned a couple things. First, don't even think of going around the Mega Bureaucracy, even if somebody says you can. The Mega Bureaucracy remembers and brands you a heretic. Second, if you think you will need help from the Mega Bureaucracy, start early, fill out all of the forms, stand in the right lines, sacrifice to the appropriate gods, and don't even hint that you would think of going around them. Finally, he who yells loudest gets move the front of the queue soonest - as holy and almighty as The Mega Bureaucracy is, they're happiest to get rid of their crabbiest customers first.

The silver lining in all of this? Apparently, the Guardians of the Mega Bureaucracy seem to now be willing to consider that there is a different tier of requests that don't require so many stopping points, designed to make sure that users really, REALLY know what they want to request. Bruce remains positive saying that, maybe in a few years, after meetings to plan meetings, forms to request forms, they will have a process that only has an initial questionnaire of 10 pages and 75 questions.

[Advertisement] Otter - Provision your servers automatically without ever needing to log-in to a command prompt. Get started today!

XKCDSoftware Development

July 17, 2018

Worse Than FailureClassic WTF: The Source Control Shingle

Our summer break continues. I once worked on a team which made "shingles"- software modules that were layered on top of a packaged product. There were a lot of WTFs in those shingles, but nothing that can compare to this once. Original--Remy

The year was 1999 and the dot-com boom was going full-throttle. Companies everywhere were focused on building revolutionary applications using nothing but top-shelf hardware and state-of-the-art software tools. Developers everywhere were trying to figure out if they should play more foosball, more air hockey, or sit back down on their Aeron and write more code. Everywhere, that is, except Boise, Idaho. Or at least, Dave's small corner of it.

At Dave's company, developers worked at a solid pace, using reliable tools, for a stable industry. They were sub-sub-contractors on a giant project commissioned by the U.S. Navy to condense naval vessel documentation. Generally speaking, the complete documentation required for a modern warship-from the GPS calibration instructions to the giant 130-millimeter cannon repair guide-is measured in tons. By condensing the documentation into the electronic equivalent, they could not only save tremendous physical space, but they could make it much easier to navigate.

A Simple Plan

Dave's company's small piece of the pie involved writing a very specific application for a particular group of users. Their application needed to track who moved which box of classified documentation from where to where, and why. Given the very simple requirements, the entire application was assigned to Mark.

Mark believed in keeping things simple: he rarely left the command line, his text editor was notepad and his source repository was a few backup folders on a network drive. He didn't need or want more than that. It was a simple task that called for his simple methodologies.

As their app neared completion, a whole new set of requirements came in. Now, they had to add in security and logging. When Dave joined Mark's one-man team to help out with this, the current system of source control -- nothing -- became inconvenient for collaborating.

Dave suggested they set up a source-control repository, but Mark wanted to keep things simple. He devised a solution called the "source-control shingle."

Roofing and Revisions

The source-control shingle was literally that: an actual shingle from someone's house that somehow ended up in their office. It acted like a "talking stick," in that only he who possessed the shingle was allowed to edit the common libraries.

As time went on, the project's scope grew immensely. More and more developers came on board, and the source-control shingle was pushed to its limits. Despite not being in possession of the shingle, some developers broke protocol and edited the library files on the share drive. Finally, Mark agreed to use a simple source repository. He wanted to use the only source-control system that guaranteed file locks: Visual Source Safe.

Unfortunately, Source Safe was so painful to license and manage that Mark had no choice but to explore other options, some of which involved a piece of painted wood. After much arguing and cajoling, Mark agreed to try out open source CVS. Things went well for the first few days, but quickly took a turn for the worse.

"What happened to my code?" Mark asked. "I just did a CVS UPDATE and everything I wrote this morning is gone!"

"It's working fine for me," one of the developers replied.

"Same here," another joined in. "I just checked in my changes a few minutes ago, and they're still here."

"Wait," a third one questioned, "did you do an UPDATE before the COMMIT?"

"Did I what?" the second developer replied. "Oh. Crap."

Exasperated, Mark jumped. "That's it! We're going back to the shingle!"

Fortunately, some of the other developers managed to convince Mark to stick with CVS, at least for a little while longer. One of the developers even managed to enforce better source control practices using some server-side scripts. And despite Mark's constant reservations, they ended up staying with CVS throughout the project. But the whole while, Mark kept the shingle handy, just in case.

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

July 16, 2018

XKCDNegative Results

July 13, 2018

XKCDAn Apple for a Dollar

July 05, 2018

Dave HallMigrating AWS System Manager Parameter Store Secrets to a new Namespace

When starting with a new tool it is common to jump in start doing things. Over time you learn how to do things better. Amazon's AWS System Manager (SSM) Parameter Store was like that for me. I started off polluting the global namespace with all my secrets. Over time I learned to use paths to create namespaces. This helps a lot when it comes to managing access.

Recently I've been using Parameter Store a lot. During this time I have been reminded that naming things is hard. This lead to me needing to change some paths in SSM Parameter Store. Unfortunately AWS doesn't allow you to rename param store keys, you have to create new ones.

There was no way I was going to manually copy and paste all those secrets. Python (3.6) to the rescue! I wrote a script to copy the values to the new namespace. While I was at it I migrated them to use a new KMS key for encryption.

Grab the code from my gist, make it executable, pip install boto3 if you need to, then run it like so:

copy-ssm-ps-path.py source-tree-name target-tree-name new-kms-uuid

The script assumes all parameters are encrypted. The same key is used for all parameters. boto3 expects AWS credentials need to be in ~/.aws or environment variables.

Once everything is verified, you can use a modified version of the script that calls ssm.delete_parameter() or do it via the console.

I hope this saves someone some time.

June 18, 2018

etbeCooperative Learning

This post is about my latest idea for learning about computers. I posted it to my local LUG mailing list and received no responses. But I still think it’s a great idea and that I just need to find the right way to launch it.

I think it would be good to try cooperative learning about Computer Science online. The idea is that everyone would join an IRC channel at a suitable time with virtual machine software configured and try out new FOSS software at the same time and exchange ideas about it via IRC. It would be fairly informal and people could come and go as they wish, the session would probably go for about 4 hours but if people want to go on longer then no-one would stop them.

I’ve got some under-utilised KVM servers that I could use to provide test VMs for network software, my original idea was to use those for members of my local LUG. But that doesn’t scale well. If a larger group people are to be involved they would have to run their own virtual machines, use physical hardware, or use trial accounts from VM companies.

The general idea would be for two broad categories of sessions, ones where an expert provides a training session (assigning tasks to students and providing suggestions when they get stuck) and ones where the coordinator has no particular expertise and everyone just learns together (like “let’s all download a random BSD Unix and see how it compares to Linux”).

As this would be IRC based there would be no impediment for people from other regions being involved apart from the fact that it might start at 1AM their time (IE 6PM in the east coast of Australia is 1AM on the west coast of the US). For most people the best times for such education would be evenings on week nights which greatly limits the geographic spread.

While the aims of this would mostly be things that relate to Linux, I would be happy to coordinate a session on ReactOS as well. I’m thinking of running training sessions on etbemon, DNS, Postfix, BTRFS, ZFS, and SE Linux.

I’m thinking of coordinating learning sessions about DragonflyBSD (particularly HAMMER2), ReactOS, Haiku, and Ceph. If people are interested in DragonflyBSD then we should do that one first as in a week or so I’ll probably have learned what I want to learn and moved on (but not become enough of an expert to run a training session).

One of the benefits of this idea is to help in motivation. If you are on your own playing with something new like a different Unix OS in a VM you will be tempted to take a break and watch YouTube or something when you get stuck. If there are a dozen other people also working on it then you will have help in solving problems and an incentive to keep at it while help is available.

So the issues to be discussed are:

  1. What communication method to use? IRC? What server?
  2. What time/date for the first session?
  3. What topic for the first session? DragonflyBSD?
  4. How do we announce recurring meetings? A mailing list?
  5. What else should we setup to facilitate training? A wiki for notes?

Finally while I list things I’m interested in learning and teaching this isn’t just about me. If this becomes successful then I expect that there will be some topics that don’t interest me and some sessions at times when I am have other things to do (like work). I’m sure people can have fun without me. If anyone has already established something like this then I’d be happy to join that instead of starting my own, my aim is not to run another hobbyist/professional group but to learn things and teach things.

There is a Wikipedia page about Cooperative Learning. While that’s interesting I don’t think it has much relevance on what I’m trying to do. The Wikipedia article has some good information on the benefits of cooperative education and situations where it doesn’t work well. My idea is to have a self-selecting people who choose it because of their own personal goals in terms of fun and learning. So it doesn’t have to work for everyone, just for enough people to have a good group.

June 06, 2018

etbeBTRFS and SE Linux

I’ve had problems with systems running SE Linux on BTRFS losing the XATTRs used for storing the SE Linux file labels after a power outage.

Here is the link to the patch that fixes this [1]. Thanks to Hans van Kranenburg and Holger Hoffstätte for the information about this patch which was already included in kernel 4.16.11. That was uploaded to Debian on the 27th of May and got into testing about the time that my message about this issue got to the SE Linux list (which was a couple of days before I sent it to the BTRFS developers).

The kernel from Debian/Stable still has the issue. So using a testing kernel might be a good option to deal with this problem at the moment.

Below is the information on reproducing this problem. It may be useful for people who want to reproduce similar problems. Also all sysadmins should know about “reboot -nffd”, if something really goes wrong with your kernel you may need to do that immediately to prevent corrupted data being written to your disks.

The command “reboot -nffd” (kernel reboot without flushing kernel buffers or writing status) when run on a BTRFS system with SE Linux will often result in /var/log/audit/audit.log being unlabeled. It also results in some systemd-journald files like /var/log/journal/c195779d29154ed8bcb4e8444c4a1728/system.journal being unlabeled but that is rarer. I think that the same
problem afflicts both systemd-journald and auditd but it’s a race condition that on my systems (both production and test) is more likely to affect auditd.

root@stretch:/# xattr -l /var/log/audit/audit.log 
security.selinux: 
0000   73 79 73 74 65 6D 5F 75 3A 6F 62 6A 65 63 74 5F    system_u:object_ 
0010   72 3A 61 75 64 69 74 64 5F 6C 6F 67 5F 74 3A 73    r:auditd_log_t:s 
0020   30 00                                              0.

SE Linux uses the xattr “security.selinux”, you can see what it’s doing with xattr(1) but generally using “ls -Z” is easiest.

If this issue just affected “reboot -nffd” then a solution might be to just not run that command. However this affects systems after a power outage.

I have reproduced this bug with kernel 4.9.0-6-amd64 (the latest security update for Debian/Stretch which is the latest supported release of Debian). I have also reproduced it in an identical manner with kernel 4.16.0-1-amd64 (the latest from Debian/Unstable). For testing I reproduced this with a 4G filesystem in a VM, but in production it has happened on BTRFS RAID-1 arrays, both SSD and HDD.

#!/bin/bash 
set -e 
COUNT=$(ps aux|grep [s]bin/auditd|wc -l) 
date 
if [ "$COUNT" = "1" ]; then 
 echo "all good" 
else 
 echo "failed" 
 exit 1 
fi

Firstly the above is the script /usr/local/sbin/testit, I test for auditd running because it aborts if the context on it’s log file is wrong. When SE Linux is in enforcing mode an incorrect/missing label on the audit.log file causes auditd to abort.

root@stretch:~# ls -liZ /var/log/audit/audit.log 
37952 -rw-------. 1 root root system_u:object_r:auditd_log_t:s0 4385230 Jun  1 
12:23 /var/log/audit/audit.log

Above is before I do the tests.

while ssh stretch /usr/local/sbin/testit ; do 
 ssh stretch "reboot -nffd" > /dev/null 2>&1 & 
 sleep 20 
done

Above is the shell code I run to do the tests. Note that the VM in question runs on SSD storage which is why it can consistently boot in less than 20 seconds.

Fri  1 Jun 12:26:13 UTC 2018 
all good 
Fri  1 Jun 12:26:33 UTC 2018 
failed

Above is the output from the shell code in question. After the first reboot it fails. The probability of failure on my test system is greater than 50%.

root@stretch:~# ls -liZ /var/log/audit/audit.log  
37952 -rw-------. 1 root root system_u:object_r:unlabeled_t:s0 4396803 Jun  1 12:26 /var/log/audit/audit.log

Now the result. Note that the Inode has not changed. I could understand a newly created file missing an xattr, but this is an existing file which shouldn’t have had it’s xattr changed. But somehow it gets corrupted.

The first possibility I considered was that SE Linux code might be at fault. I asked on the SE Linux mailing list (I haven’t been involved in SE Linux kernel code for about 15 years) and was informed that this isn’t likely at
all. There have been no problems like this reported with other filesystems.

March 16, 2018

etbeRacism in the Office

Today I was at an office party and the conversation turned to race, specifically the incidence of unarmed Afro-American men and boys who are shot by police. Apparently the idea that white people (even in other countries) might treat non-white people badly offends some people, so we had a man try to explain that Afro-Americans commit more crime and therefore are more likely to get shot. This part of the discussion isn’t even noteworthy, it’s the sort of thing that happens all the time.

I and another man pointed out that crime is correlated with poverty and racism causes non-white people to be disproportionately poor. We also pointed out that US police seem capable of arresting proven violent white criminals without shooting them (he cited arrests of Mafia members I cited mass murderers like the one who shot up the cinema). This part of the discussion isn’t particularly noteworthy either. Usually when someone tries explaining some racist ideas and gets firm disagreement they back down. But not this time.

The next step was the issue of whether black people are inherently violent. He cited all of Africa as evidence. There’s a meme that you shouldn’t accuse someone of being racist, it’s apparently very offensive. I find racism very offensive and speak the truth about it. So all the following discussion was peppered with him complaining about how offended he was and me not caring (stop saying racist things if you don’t want me to call you racist).

Next was an appeal to “statistics” and “facts”. He said that he was only citing statistics and facts, clearly not understanding that saying “Africans are violent” is not a statistic. I told him to get his phone and Google for some statistics as he hadn’t cited any. I thought that might make him just go away, it was clear that we were long past the possibility of agreeing on these issues. I don’t go to parties seeking out such arguments, in fact I’d rather avoid such people altogether if possible.

So he found an article about recent immigrants from Somalia in Melbourne (not about the US or Africa, the previous topics of discussion). We are having ongoing discussions in Australia about violent crime, mainly due to conservatives who want to break international agreements regarding the treatment of refugees. For the record I support stronger jail sentences for violent crime, but this is an idea that is not well accepted by conservatives presumably because the vast majority of violent criminals are white (due to the vast majority of the Australian population being white).

His next claim was that Africans are genetically violent due to DNA changes from violence in the past. He specifically said that if someone was a witness to violence it would change their DNA to make them and their children more violent. He also specifically said that this was due to thousands of years of violence in Africa (he mentioned two thousand and three thousand years on different occasions). I pointed out that European history has plenty of violence that is well documented and also that DNA just doesn’t work the way he thinks it does.

Of course he tried to shout me down about the issue of DNA, telling me that he studied Psychology at a university in London and knows how DNA works, demanding to know my qualifications, and asserting that any scientist would support him. I don’t have a medical degree, but I have spent quite a lot of time attending lectures on medical research including from researchers who deliberately change DNA to study how this changes the biological processes of the organism in question.

I offered him the opportunity to star in a Youtube video about this, I’d record everything he wants to say about DNA. But he regarded that offer as an attempt to “shame” him because of his “controversial” views. It was a strange and sudden change from “any scientist will support me” to “it’s controversial”. Unfortunately he didn’t give up on his attempts to convince me that he wasn’t racist and that black people are lesser.

The next odd thing was when he asked me “what do you call them” (black people), “do you call them Afro-Americans when they are here”. I explained that if an American of African ancestry visits Australia then you would call them Afro-American, otherwise not. It’s strange that someone goes from being so certain of so many things to not knowing the basics. In retrospect I should have asked whether he was aware that there are black people who aren’t African.

Then I sought opinions from other people at the party regarding DNA modifications. While I didn’t expect to immediately convince him of the error of his ways it should at least demonstrate that I’m not the one who’s in a minority regarding this issue. As expected there was no support for the ideas of DNA modifying. During that discussion I mentioned radiation as a cause of DNA changes. He then came up with the idea that radiation from someone’s mouth when they shout at you could change your DNA. This was the subject of some jokes, one man said something like “my parents shouted at me a lot but didn’t make me a mutant”.

The other people had some sensible things to say, pointing out that psychological trauma changes the way people raise children and can have multi-generational effects. But the idea of events 3000 years ago having such effects was ridiculed.

By this time people were starting to leave. A heated discussion of racism tends to kill the party atmosphere. There might be some people who think I should have just avoided the discussion to keep the party going (really I didn’t want it and tried to end it). But I’m not going to allow a racist to think that I agree with them, and if having a party requires any form of agreement to racism then it’s not a party I care about.

As I was getting ready to leave the man said that he thought he didn’t explain things well because he was tipsy. I disagree, I think he explained some things very well. When someone goes to such extraordinary lengths to criticise all black people after a discussion of white cops killing unarmed black people I think it shows their character. But I did offer some friendly advice, “don’t drink with people you work with or for or any other people you want to impress”, I suggested that maybe quitting alcohol altogether is the right thing to do if this is what it causes. But he still thought it was wrong of me to call him racist, and I still don’t care. Alcohol doesn’t make anyone suddenly think that black people are inherently dangerous (even when unarmed) and therefore deserving of being shot by police (disregarding the fact that police can take members of the Mafia alive). But it does make people less inhibited about sharing such views even when it’s clear that they don’t have an accepting audience.

Some Final Notes

I was not looking for an argument or trying to entrap him in any way. I refrained from asking him about other races who have experienced violence in the past, maybe he would have made similar claims about other non-white races and maybe he wouldn’t, I didn’t try to broaden the scope of the dispute.

I am not going to do anything that might be taken as agreement or support of racism unless faced with the threat of violence. He did not threaten me so I wasn’t going to back down from the debate.

I gave him multiple opportunities to leave the debate. When I insisted that he find statistics to support his cause I hoped and expected that he would depart. Instead he came back with a page about the latest racist dog-whistle in Australian politics which had no correlation with anything we had previously discussed.

I think the fact that this debate happened says something about Australian and British culture. This man apparently hadn’t had people push back on such ideas before.

March 05, 2018

etbeWordPress Multisite on Debian

WordPress (a common CMS for blogs) is designed to be copied to a directory that Apache can serve and run by a user with no particular privileges while managing installation of it’s own updates and plugins. Debian is designed around the idea of the package management system controlling everything on behalf of a sysadmin.

When I first started using WordPress there was a version called “WordPress MU” (Multi User) which supported multiple blogs. It was a separate archive to the main WordPress and didn’t support all the plugins and themes. As a main selling point of WordPress is the ability to select from the significant library of plugins and themes this was a serious problem.

Debian WordPress

The people who maintain the Debian package of WordPress have always supported multiple blogs on one system and made it very easy to run in that manner. There’s a /etc/wordpress directory for configuration files for each blog with names such as config-etbe.coker.com.au.php. This allows having multiple separate blogs running from the same tree of PHP source which means only one thing to update when there’s a new version of WordPress (often fixing security issues).

One thing that appears to be lacking with the Debian system is separate directories for “media”. WordPress supports uploading images (which are scaled to several different sizes) as well as sound and apparently video. By default under Debian they are stored in /var/lib/wordpress/wp-content/uploads/YYYY/MM/filename. If you have several blogs on one system they all get to share the same directory tree, that may be OK for one person running multiple blogs but is obviously bad when several bloggers have independent blogs on the same server.

Multisite

If you enable the “multisite” support in WordPress then you have WordPress support for multiple blogs. The administrator of the multisite configuration has the ability to specify media paths etc for all the child blogs.

The first problem with this is that one person has to be the multisite administrator. As I’m the sysadmin of the WordPress servers in question that’s an obvious task for me. But the problem is that the multisite administrator doesn’t just do sysadmin tasks such as specifying storage directories. They also do fairly routine tasks like enabling plugins. Preventing bloggers from installing new plugins is reasonable and is the default Debian configuration. Preventing them from selecting which of the installed plugins are activated is unreasonable in most situations.

The next issue is that some core parts of WordPress functionality on the sub-blogs refer to the administrator blog, recovering a forgotten password is one example. I don’t want users of other blogs on the system to be referred to my blog when they forget their password.

A final problem with multisite is that it makes things more difficult if you want to move a blog to another system. Instead of just sending a dump of the MySQL database and a copy of the Apache configuration for the site you have to configure it for which blog will be it’s master. If going between multisite and non-multisite you have to change some of the data about accounts, this will be annoying on both adding new sites to a server and moving sites from the server to a non-multisite server somewhere else.

I now believe that WordPress multisite has little value for people who use Debian. The Debian way is the better way.

So I had to back out the multisite changes. Fortunately I had a cron job to make snapshots of the BTRFS subvolume that has the database so it was easy to revert to an older version of the MySQL configuration.

Upload Location

update etbe_options set option_value='/var/lib/wordpress/wp-content/uploads/etbe.coker.com.au' where option_name='upload_path';

It turns out that if you don’t have a multisite blog then there’s no way of changing the upload directory without using SQL. The above SQL code is an example of how to do this. Note that it seems that there is special case handling of a value of ‘wp-content/uploads‘ and any other path needs to be fully qualified.

For my own blog however I choose to avoid the WordPress media management and use the following shell script to create suitable HTML code for an image that links to a high resolution version. I use GIMP to create the smaller version of the image which gives me a lot of control over how to crop and compress the image to ensure that enough detail is visible while still being small enough for fast download.

#!/bin/bash
set -e

if [ "$BASE" = "" ]; then
  BASE="http://www.coker.com.au/blogpics/2018"
fi

while [ "$1" != "" ]; do
  BIG=$1
  SMALL=$(echo $1 | sed -s s/-big//)
  RES=$(identify $SMALL|cut -f3 -d\ )
  WIDTH=$(($(echo $RES|cut -f1 -dx)/2))px
  HEIGHT=$(($(echo $RES|cut -f2 -dx)/2))px
  echo "<a href=\"$BASE/$BIG\"><img src=\"$BASE/$SMALL\" width=\"$WIDTH\" height=\"$HEIGHT\" alt=\"\" /></a>"
  shift
done

September 24, 2017

Dave HallDrupal Puppies

Over the years Drupal distributions, or distros as they're more affectionately known, have evolved a lot. We started off passing around database dumps. Eventually we moved onto using installations profiles and features to share par-baked sites.

There are some signs that distros aren't working for people using them. Agencies often hack a distro to meet client requirements. This happens because it is often difficult to cleanly extend a distro. A content type might need extra fields or the logic in an alter hook may not be desired. This makes it difficult to maintain sites built on distros. Other times maintainers abandon their distributions. This leaves site owners with an unexpected maintenance burden.

We should recognise how people are using distros and try to cater to them better. My observations suggest there are 2 types of Drupal distributions; starter kits and targeted products.

Targeted products are easier to deal with. Increasingly monetising targeted distro products is done through a SaaS offering. The revenue can funds the ongoing development of the product. This can help ensure the project remains sustainable. There are signs that this is a viable way of building Drupal 8 based products. We should be encouraging companies to embrace a strategy built around open SaaS. Open Social is a great example of this approach. Releasing the distros demonstrates a commitment to the business model. Often the secret sauce isn't in the code, it is the team and services built around the product.

Many Drupal 7 based distros struggled to articulate their use case. It was difficult to know if they were a product, a demo or a community project that you extend. Open Atrium and Commerce Kickstart are examples of distros with an identity crisis. We need to reconceptualise most distros as "starter kits" or as I like to call them "puppies".

Why puppies? Once you take a puppy home it becomes your responsibility. Starter kits should be the same. You should never assume that a starter kit will offer an upgrade path from one release to the next. When you install a starter kit you are responsible for updating the modules yourself. You need to keep track of security releases. If your puppy leaves a mess on the carpet, no one else will clean it up.

Sites build on top of a starter kit should diverge from the original version. This shouldn't only be an expectation, it should be encouraged. Installing a starter kit is the starting point of building a unique fork.

Project pages should clearly state that users are buying a puppy. Prospective puppy owners should know if they're about to take home a little lap dog or one that will grow to the size of a pony that needs daily exercise. Puppy breeders (developers) should not feel compelled to do anything once releasing the puppy. That said, most users would like some documentation.

I know of several agencies and large organisations that are making use of starter kits. Let's support people who are adopting this approach. As a community we should acknowledge that distros aren't working. We should start working out how best to manage the transition to puppies.

September 16, 2017

Dave HallTrying Drupal

While preparing for my DrupalCamp Belgium keynote presentation I looked at how easy it is to get started with various CMS platforms. For my talk I used Contentful, a hosted content as a service CMS platform and contrasted that to the "Try Drupal" experience. Below is the walk through of both.

Let's start with Contentful. I start off by visiting their website.

Contentful homepage

In the top right corner is a blue button encouraging me to "try for free". I hit the link and I'm presented with a sign up form. I can even use Google or GitHub for authentication if I want.

Contentful signup form

While my example site is being installed I am presented with an overview of what I can do once it is finished. It takes around 30 seconds for the site to be installed.

Contentful installer wait

My site is installed and I'm given some guidance about what to do next. There is even an onboarding tour in the bottom right corner that is waving at me.

Contentful dashboard

Overall this took around a minute and required very little thought. I never once found myself thinking come on hurry up.

Now let's see what it is like to try Drupal. I land on d.o. I see a big prominent "Try Drupal" button, so I click that.

Drupal homepage

I am presented with 3 options. I am not sure why I'm being presented options to "Build on Drupal 8 for Free" or to "Get Started Risk-Free", I just want to try Drupal, so I go with Pantheon.

Try Drupal providers

Like with Contentful I'm asked to create an account. Again I have the option of using Google for the sign up or completing a form. This form has more fields than contentful.

Pantheon signup page

I've created my account and I am expecting to be dropped into a demo Drupal site. Instead I am presented with a dashboard. The most prominent call to action is importing a site. I decide to create a new site.

Pantheon dashboard

I have to now think of a name for my site. This is already feeling like a lot of work just to try Drupal. If I was a busy manager I would have probably given up by this point.

Pantheon create site form

When I submit the form I must surely be going to see a Drupal site. No, sorry. I am given the choice of installing WordPress, yes WordPress, Drupal 8 or Drupal 7. Despite being very confused I go with Drupal 8.

Pantheon choose application page

Now my site is deploying. While this happens there is a bunch of items that update above the progress bar. They're all a bit nerdy, but at least I know something is happening. Why is my only option to visit my dashboard again? I want to try Drupal.

Pantheon site installer page

I land on the dashboard. Now I'm really confused. This all looks pretty geeky. I want to try Drupal not deal with code, connection modes and the like. If I stick around I might eventually click "Visit Development site", which doesn't really feel like trying Drupal.

Pantheon site dashboard

Now I'm asked to select a language. OK so Drupal supports multiple languages, that nice. Let's select English so I can finally get to try Drupal.

Drupal installer, language selection

Next I need to chose an installation profile. What is an installation profile? Which one is best for me?

Drupal installer, choose installation profile

Now I need to create an account. About 10 minutes I already created an account. Why do I need to create another one? I also named my site earlier in the process.

Drupal installer, configuration form part 1
Drupal installer, configuration form part 2

Finally I am dropped into a Drupal 8 site. There is nothing to guide me on what to do next.

Drupal site homepage

I am left with a sense that setting up Contentful is super easy and Drupal is a lot of work. For most people wanting to try Drupal they would have abandoned someway through the process. I would love to see the conversion stats for the try Drupal service. It must miniscule.

It is worth noting that Pantheon has the best user experience of the 3 companies. The process with 1&1 just dumps me at a hosting sign up page. How does that let me try Drupal?

Acquia drops onto a page where you select your role, then you're presented with some marketing stuff and a form to request a demo. That is unless you're running an ad blocker, then when you select your role you get an Ajax error.

The Try Drupal program generates revenue for the Drupal Association. This money helps fund development of the project. I'm well aware that the DA needs money. At the same time I wonder if it is worth it. For many people this is the first experience they have using Drupal.

The previous attempt to have simplytest.me added to the try Drupal page ultimately failed due to the financial implications. While this is disappointing I don't think simplytest.me is necessarily the answer either.

There needs to be some minimum standards for the Try Drupal page. One of the key item is the number of clicks to get from d.o to a working demo site. Without this the "Try Drupal" page will drive people away from the project, which isn't the intention.

If you're at DrupalCon Vienna and want to discuss this and other ways to improve the marketing of Drupal, please attend the marketing sprints.

AttachmentSize
try-contentful-1.png342.82 KB
try-contentful-2.png214.5 KB
try-contentful-3.png583.02 KB
try-contentful-5.png826.13 KB
try-drupal-1.png1.19 MB
try-drupal-2.png455.11 KB
try-drupal-3.png330.45 KB
try-drupal-4.png239.5 KB
try-drupal-5.png203.46 KB
try-drupal-6.png332.93 KB
try-drupal-7.png196.75 KB
try-drupal-8.png333.46 KB
try-drupal-9.png1.74 MB
try-drupal-10.png1.77 MB
try-drupal-11.png1.12 MB
try-drupal-12.png1.1 MB
try-drupal-13.png216.49 KB

April 27, 2017

Dave HallContinuing the Conversation at DrupalCon and Into the Future

My blog post from last week was very well received and sparked a conversation in the Drupal community about the future of Drupal. That conversation has continued this week at DrupalCon Baltimore.

Yesterday during the opening keynote, Dries touched on some of the issues raised in my blog post. Later in the day we held an unofficial BoF. The turn out was smaller than I expected, but we had a great discussion.

Drupal moving from a hobbyist and business tool to being an enterprise CMS for creating "ambitious digital experiences" was raised in the Driesnote and in other conversations including the BoF. We need to acknowledge that this has happened and consider it an achievement. Some people have been left behind as Drupal has grown up. There is probably more we can do to help these people. Do we need more resources to help them skill up? Should we direct them towards WordPress, backdrop, squarespace, wix etc? Is it is possible to build smaller sites that eventually grow into larger sites?

In my original blog post I talked about "peak Drupal" and used metrics that supported this assertion. One metric missing from that post is dollars spent on Drupal. It is clear that the picture is very different when measuring success using budgets. There is a general sense that a lot of money is being spent on high end Drupal sites. This has resulted in less sites doing more with Drupal 8.

As often happens when trying to solve problems with Drupal during the BoF descended into talking technical solutions. Technical solutions and implementation detail have a place. I think it is important for the community to move beyond this and start talking about Drupal as a product.

In my mind Drupal core should be a content management framework and content hub service for building compelling digital experiences. For the record, I am not arguing Drupal should become API only. Larger users will take this and build their digital stack on top of this platform. This same platform should support an ecosystem of Drupal "distros". These product focused projects target specific use cases. Great examples of such distros include Lightning, Thunder, Open Social, aGov and Drupal Commerce. For smaller agencies and sites a distro can provide a great starting point for building new Drupal 8 sites.

The biggest challenge I see is continuing this conversation as a community. The majority of the community toolkit is focused on facilitating technical discussions and implementations. These tools will be valuable as we move from talking to doing, but right now we need tools and processes for engaging in silver discussions so we can build platinum level products.