Planet Russell

,

Planet DebianRuss Allbery: Review: The Power of Habit

Review: The Power of Habit, by Charles Duhigg

Publisher: Random House
Copyright: 2012, 2014
Printing: 2014
ISBN: 0-679-60385-9
Format: Kindle
Pages: 366

One problem with reading pop psychology is that one runs into a lot of books like this one: summaries of valid psychological research that still leave one with the impression that the author was more interested in being dramatic and memorable than accurate. But without reproducing the author's research, it's hard to tell whether that fear is well-grounded or unfair, so one comes away feeling vaguely dissatisfied and grumpy.

Or at least I do. I might be weird.

As readers of my book reviews may have noticed, and which will become more apparent shortly, I'm going through another round of reading "self-help" books. This time, I'm focusing on work habits, concentration, and how to more reliably reach a flow state. The Power of Habit isn't on that topic but it's adjacent to it, so I picked it up when a co-worker recommended it.

Duhigg's project here is to explain habits, both good ones and bad ones, at a scientific level. He starts with a memorable and useful model of the habit loop: a cue triggers a routine, which results in a reward. The reward reinforcement strengthens the loop, and the brain starts internalizing the routine, allowing it to spend less cognitive energy and essentially codifying the routine like a computer program. With fully-formed habits (one's daily bathing routine, for example), the routine is run by a small, tuned part of your brain and requires very little effort, which is why we can have profound shower thoughts about something else entirely. That example immediately shows why habits are valuable and why our brain is so good at creating them: they reduce the mental energy required for routine actions so that we can spend that energy elsewhere.

The problem, of course, is that this mechanism doesn't first consult our conscious intent. It works just as well for things that we do repeatedly but may not want to automatically do, like smoking a pack of cigarettes a day. It's also exploitable; you are not the only person involved in creating your habits. Essentially every consumer product company is trying to get you to form habits around their products, often quite successfully. Duhigg covers marketing-generated habits as well as social and societal habits, the science behind how habits can be changed, and the evidence that often a large collection of apparently unrelated habits are based in a "keystone habit" that, if changed, makes changing all of the other habits far easier.

Perhaps the most useful part of this book is Duhigg's discussion of how to break the habit loop through substitution. When trying to break habits, our natural tendency is to consciously resist the link between cue and routine. This is possible, but it's very hard. It requires making an unconscious process conscious, and we have a limited amount of conscious decision-making energy available to us in a day. More effective than fighting the cues is to build a replacement habit with the same cue, but this requires careful attention to the reward stage so that the substituted habit will complete the loop and have a chance of developing enough strength to displace the original habit.

So far, so good. All of this seems consistent with other psychological research I've read (particularly the reasons why trying to break habits by willpower alone is rarely successful). But there are three things that troubled me about this book and left me reluctant to recommend it or rely on it.

The first is that a useful proxy for checking the research of a book is to look at what the author says about a topic that one already knows something about. Here, I'm being a bit unfair by picking on a footnote, but Duhigg has one anecdote about a woman with a gambling problem that has following definitive-sounding note attached:

It may seem irrational for anyone to believe they can beat the house in a casino. However, as regular gamblers know, it is possible to consistently win, particularly at games such as blackjack. Don Johnson of Bensalem, Pennsylvania, for instance, won a reported $15.1 million at blackjack over a six-month span starting in 2010. The house always wins in the aggregate because so many gamblers bet in a manner that doesn't maximize their odds, and most people do not have enough money to see themselves through losses. A gambler can consistently win over time, though, if he or she has memorized the complicated formulas and odds that guide how each hand should be played. Most players, however, don't have the discipline or mathematical skills to beat the house.

This is just barely this side of being outright false, and is dangerously deceptive to the point of being casino propaganda. And the argument from anecdote is both intellectually bogus (a lot of people gamble, which means that not only is it possible that someone will go on that sort of winning streak through pure chance, it is almost guaranteed) and disturbingly similar to how most points are argued in this book.

If one assumes an effectively infinite deck (in other words, assume each card dealt is an independent event), there is no complicated rule you can memorize to beat the house at blackjack. The best that you can do is to reduce the house edge to 1-2% depending on the exact local rules. Wikipedia has a comprehensive discussion if you want the details. Therefore, what Duhigg has to be talking about is counting cards (modifying your play based on what cards have already been dealt and therefore what cards are remaining in the deck).

However, and Duhigg should know this if he's going to make definitive statements about blackjack, US casinos except in Atlantic City (every other example in this book is from the US) can and do simply eject players who count cards. (There's a legal decision affecting Atlantic City that makes the story more complicated there.) They also use other techniques (large numbers of decks, frequent reshuffling) to make counting cards far less effective. Even if you are very good at counting cards, this is not a way to win "consistently over time" because you will be told to stop playing. Counting cards is therefore not a matter of memorizing complicated formulas and odds. It's a cat-and-mouse game against human adversaries to disguise your technique enough to not be ejected while still maintaining an edge over the house. This is rather far from Duhigg's description.

Duhigg makes another, if less egregious, error by uncritically accepting the popular interpretation of the Stanford marshmallow experiment. I'll spare you my usual rant about this because The Atlantic has now written it for me. Surprise surprise, new research shows that the original experiment was deeply flawed in its choice of subjects and that the effect drastically decreases once one controls for social and economic background.

So that's one problem: when writing on topics about which I already have some background, he makes some significant errors. The second problem is related: Duhigg's own sources in this book seem unconvinced by the conclusions he's drawing from their research.

Here, I have to give credit to Duhigg for publishing his own criticism, although you won't find it if you read only the main text of the book. Duhigg has extensive end notes (distinct from the much smaller number of footnotes that elaborate on some point) in which he provides excerpts from fact-checking replies he got from the researchers and interview subjects in this book. I read them all after finishing the rest of the book, and I thought a clear pattern emerged. After reading early drafts of portions of the book, many of Duhigg's sources replied with various forms of "well, but." They would say that the research is accurately portrayed, but Duhigg's conclusion isn't justified by the research. Or that Duhigg described part of the research but left out other parts that complicated the picture. Or that Duhigg has simplified dangerously. Or that Duhigg latched on to an ancillary part of their research or their story and ignored the elements that they thought were more central. Note after note reads as a plea to add more nuance, more complication, less certainty, and fewer sweeping conclusions.

Science is messy. Psychological research is particularly messy because humans are very good at doing what they're "supposed" to do, or changing behavior based on subtle cues from the researcher. And most psychological research of the type Duhigg is summarizing is based on very small sample sizes (20-60 people is common) drawn from very unrepresentative populations (often college students who are conveniently near the researchers and cheap to bribe to do weird things while being recorded). When those experiments are redone with larger sample sizes or more representative populations, often they can't be replicated. This is called the replication crisis.

Duhigg is not a scientist. He's a reporter. His job is to take complicated and messy stories and simplify them into entertaining, memorable, and understandable narratives for a mass audience. This is great for making difficult psychological research more approachable, but it also inherently involves amplifying tentative research into rules of human behavior and compelling statements about how humans work. Sometimes this is justified by the current state of the research. Sometimes it isn't. Are Duhigg's core points in this book justified? I don't know and, based on the notes, neither does Duhigg, but none of that uncertainty is on the pages of the main text.

The third problem is less foundational, but seriously hurt my enjoyment of The Power of Habit as a reader: Duhigg's examples are horrific. The first chapter opens with the story of a man whose brain was seriously injured by a viral infection and could no longer form new memories. Later chapters feature a surgeon operating on the wrong side of a stroke victim's brain, a woman who destroyed her life and family through gambling, and a man who murdered his wife in his sleep believing she was an intruder. I grant that these examples are memorable, and some are part of a long psychological tradition of learning about the brain from very extreme examples, but these were not the images that I wanted in my head while reading a book about the science of habits. I'm not sure this topic should require the reader brace themselves against nightmares.

The habit loop, habit substitution, and keystone habits are useful concepts. Capitalist manipulation of your habits is something everyone should be aware of. There are parts of this book that seem worth knowing. But there's also a lot of uncritical glorification of particular companies and scientific sloppiness and dubious assertions in areas I know something about. I didn't feel like I could trust this book, or Duhigg. The pop psychology I like the best is either written by practicing scientists who (hopefully) have a feel for which conclusions are justified by research and which aren't, or admits more questioning and doubt, usually by personalizing the research and talking about what worked for the author. This is neither, and I therefore can't bring myself to recommend it.

Rating: 6 out of 10

,

CryptogramFriday Squid Blogging: Dead Squid on Prince Edward Island

A beach on Prince Edward Island is littered with dead squid. No one knows why.

As usual, you can also use this squid post to talk about the security stories in the news that I haven't covered.

Read my blog posting guidelines here.

TEDWhat does TED look for in its Fellows?

Every year, TED opens applications for its new group of TED Fellows. We get thousands of applications from all corners of the world, representing every field under the sun — marine mammal conservation, biomechatronics, Khmer dance, space archeology. How do we select just 20 people to become TED Fellows?

It’s not an easy process. (Technically, our acceptance rate is lower than Harvard’s.) But we love reading your applications and hearing about your latest medical breakthroughs, ambitious art projects and incredible explorations in outer space and under the sea. We also love seeing the diversity of the people doing this groundbreaking work.

What exactly makes for a good application? Here are five traits that we look for in a TED Fellow.  

A track record of achievement. In order to be selected, you have to have done something in the world. What does that “something” look like? It depends. Maybe you’ve started a company or invented a new product. Maybe you’ve made a groundbreaking film or discovered a new galaxy. Whatever you’re doing, you should be deep in your craft, building something big.

Individuals on the cusp of a big break. Beyond a track record, we are looking for people who are ready to make a giant leap forward, and could benefit from support. Fellows are often in the early part of their careers, but we also know that big breaks can happen at any age. Fellows’ projects should have real potential for impact, and they should realistically be scalable in the next three to five years. What that scale looks like depends on the project, but we select Fellows whose ambitions are big and often global.

Originality and authenticity. An original “idea worth spreading” is the key to a successful Fellows applicant. Maybe you’re working to make a current system more efficient or equitable. Or maybe you’re working across fields, challenging the underlying assumptions of our current systems and creating brand-new ones. In fact, we’ve chosen Fellows whose work is just getting off the ground — but whose vision of the future is so imaginative and convincing that we know TED’s network can help them realize that future.   

Kind, collaborative character. The TED Fellows program now encompasses more than 450 Fellows in more than 90 countries. We’re looking for people who want to engage deeply in this amazing network — build companies together, start nonprofits, share research. Often, TED Fellows are engaging deeply with the communities around them, perhaps in the places where they were born or raised. In our experience, some of the best and most overlooked ideas for our contemporary global challenges come from those whose lives depend on the solutions.  

The truth is, we don’t always know what we’re looking for. Often, Fellows totally surprise and challenge us with brand-new ways of thinking about the world. There really is no secret formula to becoming a TED Fellow, but we know it when we see it. If you’re unsure about applying, do it anyway.

Does this sound like you or someone you know? Our application is now open. Dream bigger and apply by August 26, 2018.

Rondam RamblingsI've never seen anything like this before

Speaking of ominous developments, have you noticed how much  more often you hear the phrase, "I've never seen anything like this before" in reference to the weather nowadays?  Well, now it's my turn.  For those of you who don't know, I'm a private pilot.  I got my license in 1996 so I've been flying for over 22 years.  This is a map of south-western Oregon as it showed up on my flight planning

Rondam RamblingsThree ominous developments

Ominous development #1: Bank of America (and, apparently, only BofA) has started asking its customers whether or not they hold a dual citizenship. Ominous development #2: The Trump administration is moving aggressively to strip U.S. citizenship from anyone who lied on their application, even if the lie was immaterial or inadvertent.  The last time anything like that happened was during the

Planet DebianRaphaël Hertzog: Freexian’s report about Debian Long Term Support, June 2018

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In June, about 202 work hours have been dispatched among 13 paid contributors. Their reports are available:

  • Abhijith PA did 8 hours (out of 10 hours allocated, thus keeping 2 extra hours for July).
  • Antoine Beaupré did 24 hours (out of 12 hours allocated + 12 extra hours).
  • Ben Hutchings did 12 hours (out of 15 hours allocated, thus keeping 3 extra hours for July).
  • Brian May did 10 hours.
  • Chris Lamb did 18 hours.
  • Emilio Pozuelo Monfort did 17 hours (out of 23.75 hours allocated, thus keeping 6.75 extra hours for July).
  • Holger Levsen did nothing (out of 8 hours allocated, thus keeping 8 extra hours for July).
  • Hugo Lefeuvre did 4.25 hours (out of 23.75 hours allocated, but gave back 10 hours, thus keeping 9.5 hours for July).
  • Markus Koschany did 23.75 hours.
  • Ola Lundqvist did 6 hours (out of 8 hours allocated + 17.5 remaining hours, but gave back 15.5 unused hours, thus keeping 4 extra hours for July).
  • Roberto C. Sanchez did 29.5 hours (out of 18 hours allocated + 11.5 extra hours).
  • Santiago Ruano Rincón did 5.5 hours (out of 8 hours allocated + 7 extra hours, thus keeping 9.5 extra hours for July).
  • Thorsten Alteholz did 23.75 hours.

Evolution of the situation

The number of sponsored hours increased to 210 hours per month. We lost a silver sponsor but gained a new platinum sponsor with the Civil Infrastructure Platform project (hosted by the Linux Foundation, see their announce).

We are very happy to see the CIP project engage directly with the Debian project and try to work together to build the software stack for tomorrow’s world’s infrastructure.

The security tracker currently lists 57 packages with a known CVE and the dla-needed.txt file 52.

Thanks to our sponsors

New sponsors are in bold.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

CryptogramNew Report on Chinese Intelligence Cyber-Operations

The company ProtectWise just published a long report linking a bunch of Chinese cyber-operations over the past few years.

The always interesting gruqq has some interesting commentary on the group and its tactics.

Lots of detailed information in the report, but I admit that I have never heard of ProtectWise or its research team 401TRG. Independent corroboration of this information would be helpful.

Worse Than FailureError'd: Upon Reaching a Certain Age...

"Evidently, once you hit 55, LinkedIn thinks you'll age until your buffer overflows," writes Jonathan L.

 

"I started out looking for shower gel, but now, thanks to Google, I'm considering if a GBIC in Cadet Blue is worth the extra money," writes Robin M.

 

Matthew B. wrote, "So, an article about AI shows that the AI behind generating the summary rasied an exception. Maybe the AIs aren't speaking to each other?"

 

"Wait...did I just fail a Turing Test?" writes Daniel.

 

Rob J. wrote, "I got a 2 on a vision test but apparently only people from Krypton or blind people test on it, because there were very large negative and positive scores."

 

Pieter V. writes, "Thankfully this combo error didn't occur on the plane I took."

 

[Advertisement] Forget logs. Next time you're struggling to replicate error, crash and performance issues in your apps - Think Raygun! Installs in minutes. Learn more.

Planet DebianWouter Verhelst: PKCS#11 v2.20

By way of experiment, I've just enabled the PKCS#11 v2.20 implementation in the eID packages for Linux, but for now only in the packages in the "continuous" repository. In the past, enabling this has caused issues; there have been a few cases where Firefox would deadlock when PKCS#11 v2.20 was enabled, rather than the (very old and outdated) v2.11 version that we support by default. We believe we have identified and fixed all outstanding issues that caused such deadlocks, but it's difficult to be sure. So, if you have a Belgian electronic ID card and are willing to help me out and experiment a bit, here's something I'd like you to do:

  • Install the eID software (link above) as per normal.
  • Enable the "continuous" repository and upgrade to the packages in that repository:

    • For Debian, Ubuntu, or Linux Mint: edit /etc/apt/sources.list.d/eid.list, and follow the instructions there to enable the "continuous" repository. Don't forget the dpkg-reconfigure eid-archive step. Then, run apt update; apt -t continuous upgrade.
    • For Fedora and CentOS: run yum --enablerepo=beid-continuous install eid-mw
    • For OpenSUSE: run zypper mr -e beid-continuous; zypper up

The installed version of the eid-mw-libs or libbeidpkcs11-0 package should be v4.4.3-42-gf78d786e or higher.

One of the new features in version 2.20 of the PKCS#11 API is that it supports hotplugging of card readers; in version 2.11 of that API, this is not the case, since it predates USB (like I said, it is outdated). So, try experimenting with hotplugging your card reader a bit; it should generally work. Try leaving it installed and using your system (and webbrowser) for a while with that version of the middleware; you shouldn't have any issues doing so, but if you do I'd like to know about it.

Bug reports are welcome as issues on our github repository.

Thanks!

Don MartiBrowser privacy improvements and anti-fraud

(I work for Mozilla. None of this is secret. None of this is official Mozilla policy. Not speaking for Mozilla here.)

The good news is that interesting competition among web browsers is back, not just because of ongoing performance improvements in Firefox, but also because of Apple Safari's good work on protecting users from some kinds of cross-site tracking by default. Now the challenge for other browsers is to learn from the Safari work and build on it, to even more accurately implement the user's preferences on sharing their personal information. According to research by Tini Sevak at YouGov, 36% of users are more likely to engage with adverts that are tailored to them, while 55% are creeped out by personalized ads. The browser has to get its data sharing settings right for the individual user, while minimizing the manual settings and decision fatigue that the user has to go through.

A short-term problem for sites, though, is that the current price for highly tracked ad impressions facilitated by cross-site tracking is way below the price of impressions delivered to users who choose to protect themselves. Tim Peterson, on Digiday, covers the natural experiment of GDPR consenters and non-consenters:

If an exchange or SSP declines to sign the agreement, it is limited to only selling non-personalized ads through DBM. Those generic ads generate less revenue for publishers than personalized ads that are targeted to specific audiences based on data collected about them. Some publishers that are heavily reliant on DBM have seen their revenues decline by 70-80 percent since GDPR took effect because they were limited to non-personalized ads, said another ad tech exec.

(‘It’s impossible’: Google has asked ad tech firms to guarantee broad GDPR consent, assume liability - Digiday)

In the medium to long term, better browser privacy settings will give an advantage to high-reputation sites for two reasons:

  • ads on high-value content have signaling value

  • users are more likely to share information with a site they trust

But in the short term, what can browsers do to help address the market dislocation from the user data crunch?

One possibility is to take advantage of an important side effect of browser privacy improvements: better anti-fraud data.

Today, unprotected browsers and fraudbots are hard to tell apart. Both maintain a single "cookie jar" across trusted and untrusted sites. For fraudbots, cross-site trackability is not a bug as it is for a human user's browser—it's a feature. A fraudbot can only produce valuable ad impressions on a fraud site if it is somehow trackable from a legit site.

As browser users start to upgrade to nightly releases that include more protection, though, a trustworthy site's real users will start to look more and more different from fraudbots. Low-reputation and fraud sites claiming to offer the same audience will have a harder and harder time trying to sell impressions to agencies that can see it's not the same people. This does require better integration with anti-fraud tools, so it's something sites and anti-fraud vendors can do in parallel with the brower release process.

Can the anti-fraud advantages of browser privacy improvements completely swamp out the market effects of reducing cross-site trackability? Depends on how much adfraud there is. We don't know.

,

Krebs on SecurityHuman Resources Firm ComplyRight Breached

Cloud-based human resources company ComplyRight said this week that a security breach of its Web site may have jeopardized sensitive consumer information -- including names, addresses, phone numbers, email addresses and Social Security numbers -- from tax forms submitted by the company's thousands of clients on behalf of employees. Cloud-based human resources company ComplyRight said this week that a security breach of its Web site may have jeopardized sensitive consumer information -- including names, addresses, phone numbers, email addresses and Social Security numbers -- from tax forms submitted by the company's clients on behalf of employees. Pompano Beach, Fla-based ComplyRight began mailing breach notification letters to affected consumers late last week, but the form letters are extremely vague about the scope and cause of the breach. Indeed, many readers who received these letters wrote to KrebsOnSecurity asking for more information, as the company hadn't yet published any details about the breach on its Web site. Also, most of those folks said they'd never heard of ComplyRight and could not remember ever doing business with a company by that name.

Planet DebianLars Wirzenius: Building Debian packages in CI (ick)

I've recently made the first release of ick, my CI engine, which was built by ick itself. It went OK, but the process needs improvement. This blog post is some pondering on how the process of building Debian packages should happen in the best possible taste.

I'd appreciate feedback, preferably by email (liw@liw.fi).

Context

I develop a number of (fairly small) programs, as a hobby. Some of them I also maintain as packages in Debian. All of them I publish as Debian packages in my own APT repository. I want to make the process for making a release of any of my programs as easy and automated as possible, and that includes building Debian packages and uploading them to my personal APT repository, and to Debian itself.

My personal APT repository contains builds of my programs against several Debian releases, because I want people to have the latest version of my stuff regardless of what version of Debian they run. (This is somewhat similar to what OEMs that provide packages of their own software as Debian packages need to do. I think. I'm not an OEM and I'm extrapolating wildly here.)

I currently don't provide packages for anything but Debian. That's mostly because Debian is the only Linux distribution I know well, or use, or know how to make packages for. I could do Ubuntu builds fairly easily, but supporting Fedora, RHEL, Suse, Arch, Gentoo, etc, is not something I have the energy for at this time. I would appreciate help in doing that, however.

I currently don't provide Debian packages for anything other than the AMD64 (x86-64, "Intel 64-bit") architecture. I've previously provided packages for i386 (x86-32), and may in the future want to provide packages for other architectures (RISC-V, various Arm variants, and possibly more). I want to keep this in mind for this discussion.

Overview

For the context of this blog post, let's assume I have a project Foo. Its source code is stored in foo.git. When I make a release, I tag it using a signed git tag. From this tag, I want to build several things:

  • A release tarball. I will publish and archive this. I don't trust git, and related tools (tar, compression programs, etc) to be able to reproducibly produce the same bit-by-bit compressed tarball in perpetuity. There's too many things that can go wrong. For security reasons it's important to be able to have the exact same tarball in the future as today. The simplest way to achive this is to not try to reproduce, but to archive.

  • A Debian source package.

  • A Debian binary package built for each target version of Debian, and each target hardware architecture (CPU, ABI, possibly toolchain version). The binary package should be built from the source package, because otherwise we don't know the source package can be built.

The release tarball should be put in a (public) archive. A digital signature using my personal PGP key should also be provided.

The Debian source and binary packages should be uploaded to one or more APT repositories: my personal one, and selected packages also the Debian one. For uploading to Debian, the packages will need to be signed with my personal PGP key.

(I am not going to give my CI access to my PGP key. Anything that needs to be signed with my own PGP key needs to be a manual step.)

Package versioning

In Debian, packages are uploaded to the "unstable" section of the package archive, and then automatically copied into the "testing" section, and from there to the "stable" section, unless there are problems in a specific version of a package. Thus all binary packages are built against unstable, using versions of build dependencies in unstable. The process of copying via testing to stable can take years, and is a core part of how Debian achieves quality in its releases. (This is simplified and skips consideration like security updates and other updates directly to stable, which bypass unstable. These details are not relevant to this discussion, I think.)

In my personal APT repository, no such copying takes place. A package built for unstable does not get copied into section with packages built for a released version of Debian, when Debian makes a release.

Thus, for my personal APT repository, there may be several builds of the any one version of Foo available.

  • foo 1.2, built for unstable
  • foo 1.2, built for Debian 9
  • foo 1.2, built for Debian 8

In the future, that list may be expanded by having builds for several architectures:

  • foo 1.2, built for unstable, on amd64
  • foo 1.2, built for Debian 9, on amd64
  • foo 1.2, built for Debian 8, on amd64

  • foo 1.2, built for unstable, on riscv

  • foo 1.2, built for Debian 9, on riscv
  • foo 1.2, built for Debian 8, on riscv

When I or my users upgrade our Debian hosts, say from Debian 8 to Debian 9, any packges from my personal APT archive should be updated accordingly. When I upgrade a host running Debian 8, with foo 1.2 built for Debian 8, gets upgraded to Debian 9, foo should be upgraded to the version of 1.2 built for Debian 9.

Because the Debian package manager works on combinations of package name and package version, that means that the version built for Debian 8 should have a different, and lesser, version than the one built for Debian 9, even if the source code is identical except for the version number. The easiest way to achieve this is probably to build a different source package for each target Debian release. That source package has no other differences than the debian/changelog entry with a new version number, so it doesn't necessarily need to be stored persistently.

(This is effectively what Debians "binary NMU" uploads do: use the same source package version, but do a build varying only the version number. Debian does this, among other reasons, to force a re-build of a package using a new version of a build depenency, for which it is unnecessary to do a whole new sourceful upload. For my CI build purposes, it may be useful to have a new source package, for cases where there are other changes than the source package. This will need further thought and research.)

Thus, I need to produce the following source and binary packages:

  • foo_1.2-1.dsc — source package for unstable
  • foo_1.2-1.orig.tar.xz — upstream tarball
  • foo_1.2-1.debian.tar.xz — Debian packaging and changes
  • foo_1.2-1_amd64.deb — binary package for unstable, amd64
  • foo_1.2-1_riscv.deb — binary package for unstable, riscv

  • foo_1.2-1~debian8.dsc — source package for Debian 8

  • foo_1.2-1~debian8.debian.tar.xz — Debian packaging and changes
  • foo_1.2-1~debian8_amd64.deb — binary package for Debian 8, amd64
  • foo_1.2-1~debian8_riscv.deb — binary package for Debian 8, riscv

  • foo_1.2-1~debian9.dsc — source package for Debian 9

  • foo_1.2-1~debian9.debian.tar.xz — Debian packaging and changes
  • foo_1.2-1~debian9_amd64.deb — binary package for Debian 9, amd64
  • foo_1.2-1~debian9_riscv.deb — binary package for Debian 9, riscv

The orig.tar.xz file is a bit-by-bit copy of the upstream release tarball. The debian.tar.xz files have the Debian packaging files, plus any Debian specific changes. (For simplicity, I'm assuming a specific Debian source package format. The actual list of files may vary, but the .dsc file is crucial, and references the other files in the source package. Again, these details don't really matter for this discussion.)

To upload to Debian, I would upload the foo_1.2-1.dsc source package from the list above, after downloading the files and signing them with my PGP key. To upload to my personal APT repository, I would upload all of them.

Where should Debian packaging be stored in version control?

There seems to be no strong consensus in Debian about where the packaging files (the debian/ subdirectory and its contents) should be stored in version control. Several approaches are common. The examples below use git as the version control system, as it's clearly the most common one now.

  • The "upstream does the packaging" approach: upstream's foo.git also contains the Debian packaging. Packages are built using that. This seems to be especially common for programs, where upstream and the Debian package maintainer are the same entity. That's also the OEM model.

  • The "clone upstream and add packaging" approach: the Debian package maintainer clonse the upstream repository, and adds the packaging files in a separate branch. When upstream makes a release, the master branch in the packaging repository is updated to match the upstream's master branch, and the packaging branch is rebased on top of that.

  • The "keep it separate" approach: the Debian packager puts the packaging files in their own repository, and the source tree is constructed from botht the upstream repository and the packaging repository.

For my own use, I prefer the "upstream does packaging" approach, as it's the least amount of friction for me. For ick, I want to support any approach.

There are various tools for maintaining package source in git (e.g., dgit and git-buildpackage), but those seem to not be relevant to this blog post, so I'm not discussing them in any detail.

The build process

Everything starts from a signed git tag in the foo.git plus additional tags in any packaging repository. The tags are made by the upstream developers and Debian package maintainers. CI will notice the new tag, and build a release from that.

  • Create the upstream tarball (foo-1.2.tar.gz).

  • Manully download and sign the upstream tarball with PGP.

  • Manully publish the upstream tarball and its signature in a suitable place.

  • Create the Debian source package for unstable (foo_1.2-1.dsc), using a copy of the upstream tarball, renamed.

  • Using the Debian source package, build a Debian binary package for unstable for each target architecture (foo_1.2-1_amd64.deb etc).

  • For each target Debian release other than unstable, create a new source package by unpacking the source package for unstable, and adding a debian/changelog entry with ~debianN appended to the version number. If there is a need, make any additional Debian release specific changes to the source package.

  • Build each of those source packages for each target architecture, in a build environment with the target Debian release. (foo_1.2-1~debianN_amd64.deb etc).

  • Upload all the Debian source and binary packages to an APT repository that allows upload by CI. Have that APT repository sign the resulting Packages file with its own PGP key.

  • Manully download the Debian packages and sign the unstable build to Debian, and upload it to Debian. (Source package only, except in cases where the binary package also needs to be uploaded, such as for new packages.)

TEDElectric and empowered: Monica Araya on Costa Rica’s clean energy future


28001654635_f128f0b6b6_o.jpg

Monica Araya made a big prediction on the TED stage in 2016: Costa Rica, her home country, will be the first nation in the world to pursue 100% renewable energy. Fast forward to 2018, and they’re on their way. Costa Rica already generates over 99% of their electricity through renewable energy, and went 300 days on clean energy in 2017. And in May, in a visionary next step, new president Carlos Alvarado announced at his inauguration that Costa Rica would phase out the use of fossil fuels in transportation, calling it a “generational imperative.” We talked to Monica, the director of Costa Rica Limpia (Clean Costa Rica), about what lies ahead.

This interview has been edited and condensed.

Can you tell us about the clean energy movement in Costa Rica? What are the core objectives and how is Costa Rica positioned to lead the way?

I went on the TED stage [to share] a vision of a small country thinking big. We should completely get rid of fossil fuels. Why not? The country already runs on renewable energy, which is not the case for the world — it’s not the case for Europe, the US, India or China. We’ve already broken free from fossil fuels for power and electricity generation. We’ve done the work with civil society from the ground up, but we needed it to become a vision for the country. Costa Rica is a young nation that’s going to turn 200 in 2021. 200 years ago, we broke free from Spain and we became a free nation — and that matches perfectly with this timing. We’re now ready to say, “We are going to free ourselves from fossil fuels.”

“This is the new Costa Rica, and in that new Costa Rica, we know that the future is renewable and electric.”

We have all the conditions — we have clean electricity, we have a young president who wants to do right, and we have technology on our side. Renewable energy has become a part of the country’s identity. People feel proud: they believe it’s a Costa Rican thing to go green. If you look at the citizen consultations we’ve done with Costa Rica Limpia, people disagree on many things but they agree on this. The president knows that he can set a precedent at a time when the world is trying to figure out how to transition to electric mobility. We have to show that it’s doable and beneficial, that it works technologically; I think that’s the value of a small country doing it first.

What are the challenges that Costa Rica will face in transitioning to 100% clean energy? I’m particularly interested in transportation, and moving from gasoline to electric energy — what are the challenges of that?

In practice, there are five things we have to do. We managed to pass the first electric zero-emissions law in Latin America. That came out of a coalition led by congresswoman Marcela Guerrero Campos. We created that coalition and it led to a law — Argentina and Columbia are going to try to do the same — and now, the law needs to be implemented. It calls for electrification of at least 10 percent of all the transportation owned by the state, and gives financial incentives for five years for electrification. This law is the first step — and it was hard — but we won it. It was a big day. I had some tears in my eyes when we passed it.

Second, on June 5th, on World Environment Day, we launched an initiative to electrify buses. That’s going to take some time because that’s a sector that is resistant to change — in Costa Rica, the buses belong to companies and they run for concessions every seven years. We have to make sure when they apply for concessions for the next seven-year cycle, the mandate for the buses are embedded in this requirement. In the meantime, we’re going to start testing three bus lines. Public transportation is very important in Latin America and in Costa Rica. Latin America has the highest number of people in the world using public transit. So the electrification of buses is a very important step.

27388329844_c9a5fb4095_o

Monica Araya: “By 2022, electric cars and conventional cars are expected to cost the same, and cities are already trying electric buses…if we want to get rid of oil-based transportation, we can, because we have options now that we didn’t have before.” Photo: Bret Hartman / TED

The next element that is very important is the First Lady’s Office. The first lady is amazing — she’s an architect, and she’s totally into decarbonization. Her office is focusing on urban issues, and public transit is a big part of that. Her priority is to lead the process towards the urban electric train. The train is very important to this administration — it’s a symbol of modernization. For Costa Ricans, the train is something that was wanted for a long time and was blocked by bus companies. The First Lady has taken this on; by the end of the four years, we should have started the first electric train.

I think there’s a new generation around the world — it doesn’t matter if it’s Costa Rica, or Columbia, or the Philippines  — that aspire to have bikes and safe bike paths. It’s about democratizing the street and making sure the streets don’t belong to private cars. The President of our Congress, Carolina Hidalgo Herrera, goes to work on a bike — she rode her bike to the inauguration in high heels. That’s another route to decarbonization; the bike path is a symbol of good planning, and that is where we have failed in the past. In emerging economies, it’s common to just let cars rule. The electric bus was used to transport all of the ministers to the transportation and it was important for the people to see a zero-emission bus arriving to the inauguration. There’s a lot of backcasting — looking to the ideal future and working backwards from there to see what we need to do. It’s about having a direction of travel.

The President and Minister put a draft law in Congress that makes it impossible for Costa Rica to do any drilling and any exploitation of fossil fuels. We already have a moratorium on oil exploration and exploitation from around 15 years ago that has been sustained by five different governments from three different parties; it cannot be removed. This new government wants to make sure it is the law. It’s a way of saying that they’re serious about fossil fuels not being the future for us. In the early 2000s, there was lobbying by a company in Texas who wanted to do oil exploration in Costa Rica, and there was a lot of pressure on us. The Minister of Energy and Environment at the time said, “No way, this is not going to happen,” — and I know this because I asked him — he said, “Look, I don’t know what will happen, but I can assure you that as long as I’m the minister, they will have to go over my dead body.” That was very reassuring for me to hear as a young advocate.

“There’s a long tradition of environmental protection in Costa Rica.”

Here’s what’s interesting: the Minister of Energy and Environment at that time, Carlos Manuel Rodriguez, is the minister again. It’s reassuring to have a confident and experienced minister because it means we’re going to think big. We organized a free citizen encounter with him a few weeks after he was appointed — we brought him to a museum and sat him in front of citizens. The two of us were on the stage — two chairs, nothing fancy — and I asked him questions and he answered. We also used Facebook Live so people could listen from home. And he says he wants to do these kinds of citizen encounters every six months.

That’s great — connecting the citizens to what can be a more abstract concern is important. Environmental changes can be very macro so bringing it to the citizens in an accessible place of understanding and engagement is necessary.

It’s very important to have symbols. It doesn’t matter if you’re trying to get rid of plastics or protect the ocean — you have to know what your symbols are. We came up with a logo of a contour of Costa Rica’s map that connects through a plug, meaning that there’s clean electricity that connects us as Costa Ricans, as a country.

20170909_120543

Photo: Costa Rica Limpia

We created the Costa Rican Association for Electric Mobility as a separate entity that represents users of electric mobility — electric buses, motorbikes, cars, etcetera. It’s helped as we talk to young people, mothers, grandmothers — people who don’t spend a lot of time thinking about the climate. It’s easy to feel small and scared, and feel like it all depends on what China or Trump does. That’s a dangerous framing of the problem because it’s so easy to do nothing and have a “why bother?” mentality. And when advocates and governments have that kind of framing, you lose the citizens, the people. So we had to think about the symbols of success. What is the symbol of success if we decarbonize? I’m obsessed with exhaust pipes and the fumes that come out of cars — they’re a symbol of the last century that we really need to get rid of.

“The day we are a country without exhaust pipes — the buses won’t have them, the cars won’t have them — then we have succeeded in our mission to decarbonize the country. Hopefully, the world will get there someday; Costa Rica will need to get there as soon as possible to show that it’s possible.“

The plug has become an important symbol for us. We show a very modern-looking plug and say — look, you have electricity at home to toast your bread, charge your phone, make your coffee. Everything you do is electric. Why on earth would you want an old technology that burns, that’s liquid, that’s not even Costa Rican? It costs a lot to bring it in, it causes climate change, and when you put it in your car, you have to burn it, then it comes out of an exhaust pipe and pollutes the air. People are really intrigued by the idea that everything they use is already electric other than their cars.

This technology will allow us to meet the Paris Agreement targets, and that’s important — we don’t walk around the Paris Agreement targets like other countries do. We won’t have a global impact on emissions or average temperature, because we’re too small. It’s easy to be cynical: people will say, “What’s the point? Whatever you’re reducing in Costa Rica won’t make a difference.” But we’re the ones who benefit the most. You have to win this on the basis of the benefits for the people and avoid the argument that you do it for the 2-degree temperature change — that framing won’t work for a family in Costa Rica.

It’s important to communicate that the situation is tough but it’s also important to pivot to resilience and to ideas of what is possible for us to protect ourselves. The TED Talk let us use a storytelling format — you can share it on Facebook, watch it on a phone. The TED Talk expanded the imagination of the people who listened to it. Even bigger countries like India have told me, “Maybe India can’t move forward the same way that Costa Rica can, but that doesn’t mean that a city in India the size of Costa Rica cannot think big and move faster to clean energy.” That was a very empowering idea. There’s something about smaller locations that’s great because we can move forward and just wait for the rest of the country to get there. In my country, if you want to get people excited, you have to say that this will make us a country that could inspire others.

We matter because of our ideas, not our size. Being small doesn’t mean thinking small.”

Can you tell us about your work with Costa Rica Limpia? How do you involve and center citizens in your approach? ​

Costa Rica Limpia (Clean Costa Rica) is centered on engaging citizens and consumers in the transition to a fossil free society. We educate, inspire and empower citizens by translating technical issues such as decarbonization, Paris targets and NDCs into layman’s language. We are very focused on zero emissions mobility because being carbon free in Costa Rica means using electricity instead of oil for transportation. We design education materials like infographics and videos that respond to common questions and myths. We also conduct citizen consultations on climate change and renewables, based on a Danish Board of Technology methodology. We pioneered the concept of Electric Mobility Citizen Festivals (we organized two in 2017 and 2018) because it is critical to get people to experience these new technologies.

20170422_163051.jpg

Congresswoman Marcela Guerrero and Monica Araya attend an Electric Mobility Citizen Festival with their mothers. Photo: Monica Araya

In your talk, you mention that Costa Rica disbanded its army in 1948 and has been able to redirect those funds to programs that develop social progression and growth. In a world that, in a lot of respects, seems unwilling and unable to change, how has Costa Rica been able to cultivate a culture of forward-thinking innovation?

This would not be possible if we didn’t have a social contract that takes care of people’s needs by giving them free health care and free education. We do this work because it makes life better for people who are taking public transit. There’s something about the social guarantee in the ’40s before the abolition of the army that was important. It allowed people to have a safety net, and when you do that, you build a more resilient society. Social progress was able to develop in Costa Rica partially because we have the infrastructure for it. When you go to other places in Latin America, there is a very small group of people who have nearly everything, and you have a very large population that is very poor; we have been very fortunate in Costa Rica to be able to negotiate with those stakeholders.

If there’s something I’ve learned about Costa Rica, it’s that we’ve succeeded because we have a strong middle ground in politics. The new president, Carlos Alvarado, as a political scientist, is trying to practice this lesson from Costa Rica’s history. This is an environmental story, yes, but it’s also about balance. You have to do the environmental work but you can do it better when you have invested in the people’s social progress and have turned it into a good business opportunity. Costa Rica has a larger group of people making money off ecotourism now than in the ‘80s. This bet on natural capital has paid off — when you look at the materials and marketing of Costa Rica in the world, it emphasizes that we have a social safety net. It’s a balancing act between social, environment and economic concerns that we need to get right. It’s worked in the past, and if we want to make sure it works now with fossil-free Costa Rica, we will have to be able to bring on board the private sector but also be very socially oriented. We have to make sure that the people who have the least benefit the most.

What are some other ways Costa Rica is working to protect the environment?

There’s a big movement in Costa Rica to do more about the oceans and our plastic consumption as well. There is a protected area that was launched last year in the south of Costa Rica — it continues with our tradition to resist the exploitation of our natural capital for fossil fuels. The conservation agenda today is not just the land — it’s the oceans too. The relationship between the oceans and the fossil fuel agenda is extremely close because the drilling often happens offshore. If we keep protecting areas around the world, it’ll hopefully create an awareness that the gasoline you put in your car comes from somewhere. The same thing with plastics — there’s a cultural shift and awareness about our unsustainable plastic use. When you link it to oil, it’s really interesting: it comes from oil, from petroleum and natural gas. We continue to work in different bubbles — I’m in the fossil fuel and energy transportation bubbles, but other people are in the ocean bubbles and plastic bubbles. What links us is that we all advocate that we fundamentally have to change our relationship to fossil fuels.

Are you going to play a role in the energy transition? What are your next steps?

I’m going to help with the decarbonization pathways — that takes time, and it takes not just technical work but also consultation with key stakeholders. There’s methodologies with this but the Minister doesn’t want to end up with something too theoretical but rather, is grounded in our political reality. I’ll be helping with that. We need to find as many partners as possible — in Costa Rica, obviously — but also outside. My role is to tell the story as best as I can so that we can attract anyone around the world with brilliant ideas. We want to be the testing ground for a fossil-free society. In Costa Rica Limpia, I see the electrification of buses as a very strategic action plan. This is something that is going to transform life in a very tangible way. The buses are beautiful, quiet, and they don’t pollute. Imagine a single mom with two kids who will be commuting on that bus — her life will be transformed for the better.

 

TEDTiq Milan talks to Netflix, the determined search for aliens, and other TED news

30308530590_032c31041e_o (1).jpg

As usual, the TED community is bursting with new projects and discoveries. Here are a few highlights.

The power of representation. Writer and trans activist Tiq Milan, at left in the photo above, was interviewed for a new initiative by Netflix and GLAAD called “First Time I Saw Me.” Alongside Elliot Fletcher, Jamie Clayton, Jazz Jennings and other trans actors and media makers, Tiq spoke on the realities of being marginalized in media, and what representation means to him. “With representation, we’re going to see the hearts and minds of people change,” he said. “And then, we see policies change.” Bonus: As a beautiful part of the project (and a surprise to Tiq!), Netflix commissioned visual artist Rae Senarighi to live-paint a larger-than-life color portrait of Tiq as he spoke. (Watch Tiq’s TED Talk here.)

The most fearless comedian alive. In a new profile in Glamour, Palestinian-American comedian Maysoon Zayid offers her thoughts on the internet, our political landscape and the limitless possibilities of humor. Maysoon uses her comedy to shine a light on Islamophobia and disability, and is a vocal advocate against bigotry of all kinds. She co-founded the New York Arab-American Comedy Festival (now in its 14th season!), “to combat the negative images about Arabs and Muslims in media.” (Watch Maysoon’s TED Talk here.)

Illuminating truth by mining map data. In a feature by the BBC, political scientist and megacity expert Robert Muggah revealed fascinating insights from several super-maps he helped develop at Instituto Igarapé. By sifting through key data points found while researching geographical patterns, these maps can offer fascinating information about climate change, refugee and migration patterns, even light pollution. Instituto Igarapé has just released a new website, Earth Time, for global citizens to dive into to understand comprehensive information through highly visual, accessible formats. (Watch Robert’s TED Talk here.)

Where are the aliens? American senior astronomer Seth Shostak was recently interviewed for Vox’s Explained series on Netflix on the famous Fermi paradox and the possibilities of discovering intelligent extraterrestrial life. At TED in 2012, Seth shared a bold prediction: we’ll find aliens within the next two dozen years. Others are not quite so sure, retooling a probability equation called the Drake equation to shut down our hopes of finding and communicating with otherworldly beings. As a researcher at the SETI (Search for Extraterrestrial Intelligence) Institute, Seth is determined to prove them wrong. (Watch Seth’s TED Talk.)

Cancer: a descendent of the ancient dog. New research from Elizabeth Murchison and others has found that the closest relative to ancient American dogs isn’t a dog at all — it’s a canine cancer. According to an article by TED speaker Ed Yong in the Atlantic, canine-transmissible venereal tumors (CTVT) likely began in the genitals of a dog thousands of years ago. By remaining alive in dogs dozens of generations (and continents) beyond its humble origins, CTVT stands as the closest living descendent to indigenous American dogs. (Watch Elizabeth’s TED Talk here.)

Dance us to the end of love. One important update isn’t about a speaker at all, rather about someone we got to know through a TED Talk: the dancer and choreographer Gillian Lynne, who died last week after an astonishing life. Here is what Sir Ken Robinson said about her, and it’s worth reading at length:

Gillian Lynne. Have you heard of her? Some have. She’s a choreographer, and everybody knows her work. She did “Cats” and “Phantom of the Opera.” She’s wonderful. Gillian and I had lunch one day and I said, “How did you get to be a dancer?” It was interesting. When she was at school, she was really hopeless. And the school, in the ’30s, wrote to her parents and said, “We think Gillian has a learning disorder.” She couldn’t concentrate; she was fidgeting. I think now they’d say she had ADHD. Wouldn’t you? But this was the 1930s, and ADHD hadn’t been invented at this point. It wasn’t an available condition. People weren’t aware they could have that.

Anyway, she went to see this specialist. So, this oak-paneled room, and she was there with her mother, and she was led and sat on this chair at the end, and she sat on her hands for 20 minutes while this man talked to her mother about the problems Gillian was having at school. Because she was disturbing people; her homework was always late; and so on, little kid of eight. In the end, the doctor went and sat next to Gillian, and said, “I’ve listened to all these things your mother’s told me, I need to speak to her privately. Wait here. We’ll be back; we won’t be very long,” and they went and left her.

But as they went out of the room, he turned on the radio that was sitting on his desk. And when they got out, he said to her mother, “Just stand and watch her.” And the minute they left the room, she was on her feet, moving to the music. And they watched for a few minutes and he turned to her mother and said, “Mrs. Lynne, Gillian isn’t sick; she’s a dancer. Take her to a dance school.”

I said, “What happened?” She said, “She did. I can’t tell you how wonderful it was. We walked in this room and it was full of people like me. People who couldn’t sit still. People who had to move to think.” Who had to move to think. They did ballet, they did tap, jazz; they did modern; they did contemporary. She was eventually auditioned for the Royal Ballet School; she became a soloist; she had a wonderful career at the Royal Ballet. She eventually graduated from the Royal Ballet School, founded the Gillian Lynne Dance Company, met Andrew Lloyd Webber. She’s been responsible for some of the most successful musical theater productions in history, she’s given pleasure to millions, and she’s a multi-millionaire. Somebody else might have put her on medication and told her to calm down.

Planet DebianLucas Kanashiro: My DebCamp/DebConf 18 plans


Tomorrow I am going to another DebCamp and DebConf; this time at Hsinchu, Taiwan. Thanks to Debian project, I received a sponsor to attend the event, in this sense I plan to do the following contributions:

  • Bootstrap the DebConf 19 website. I volunteered myself to lead the DebConf 19 website things, and to do that I intend to get in touch with more experienced people from the DebConf team.

  • Participate part-time at the Perl team sprint. Despite I have not been so active in the team as I used to be, I’ll try to use the opportunity to help with packages update and some bug fixing.

  • Keep working with Arthur Del Esposte in our GSoC project, which aims to improving distro-tracker to better support Debian teams workflow. Also, prepare him to make an excellent presentation in the GSoC session. Hope see you there!

  • If I have enough time I want to work on some of my packages too, specially Redmine.

If anyone is interested in what I’ll do these days just reach me out! Could be in person, via IRC (my nickname: kanashiro) or just mail me (kanashiro@debian.org).

I hope meet you soon in Hsinchu!

CryptogramSuing South Carolina Because Its Election Machines Are Insecure

A group called Protect Democracy is suing South Carolina because its insecure voting machines are effectively denying people the right to vote.

Note: I am an advisor to Protect Democracy on its work related to election cybersecurity, and submitted a declaration in litigation it filed, challenging President Trump's now-defunct "election integrity" commission.

Worse Than FailureClassic WTF: Flawless Compilation

Just today I was joking with my co-workers: I had written software for which we had no viable test hardware, but the code compiled, therefore I was done. The difference is I was joking… --Remy (Originally)

Back in the heady days of Internet speculation, the giant retailer JumboStores contracted with Fred’s software company, TinyWeb, to develop the region’s first web-based supermarket. Customers would be able to assemble carts online and receive their groceries the next day.

The virtual supermarket had to communicate with JumboStores’s inventory system in real-time. The former was bleeding-edge web technology, the latter a cobweb-laden mainframe with no external point of access.

“How will we get around this?” Fred asked early in the specification process.

“We can stage an intermediate server.” Nick, a programmer from JumboStores IT, assured him around a mouthful of doughnut. “You guys send your requests there, we’ll write software to forward them to the mainframe and back.”
Engine overhauled
Fred was optimistic. Both companies were *nix shops; the JumboStores IT department were his geek kindred. Equally optimistic, JumboStores management scheduled a live media demo several months out, well after the estimated project completion date.

Deadlines slipped, as they are wont to do. The week before the big demo, the online supermarket still wasn’t ready. TinyWeb had implemented the website and database back-end, but JumboStores’ relay software lagged behind. At the urging of multiple strata of nervous managers, Fred took an emergency trip to JumboStores to investigate.

“We don’t know, man, we just don’t know.” The confident Nick of months prior shook now, leading Fred to his cubicle. “We coded the application. We debugged until it compiled without errors. When we run it- core dump!” He threw up his hands, then dropped into his swivel chair. “We’ve been pestering IBM support, but they haven’t been very helpful.”

“Well, why would they be?” Fred frowned, pausing at the cube threshold. “I mean, who knows what might be wrong with the code?”

“Nothing’s wrong with it. It compiles!”

“So? It could still have errors.”

Nick swiveled around to face him. “Dude. It compiles.

Fred faltered in the wake of Nick’s earnest insistence. “That… doesn’t mean the code is perfect.” He all but fell into the spare chair presented to him. “How do I explain this?” Am I actually trying to explain this? To a programmer? “Let’s say you’re building an engine.”

“This isn’t an engine,” Nick said. “It just passes-“

“No, a car engine! OK? You have all the parts spread out on the desk here.” He waved his arm out over a layer of branded cube toys and post-it notes. “You’ve never built an engine from scratch before, but you have a blueprint with pictures and directions, so you grab your wrench and your welder and whatever, and go to town. At the end, all the parts get used up, and the result looks vaguely engine-like. Still, would you expect to drop it under the hood and have it start up flawlessly the first time you turn over the ignition?”

Nick stared. “I… don’t see what this has to do with anything.”

Fred refrained from smacking his forehead. “Uh, OK. Forget the engine. It’s like sheet music. Just because all the dots are on the staff doesn’t mean it’s the song you want.“

“Dude! The compiler would bug out if there were any problems.” Nick graciously omitted the Duh.

Fred took one last chance. “No- it’s like, if you were building a house. Just because all the parts fit together doesn’t mean it will stand up.”

Nick’s face brightened. “It’s like the home inspector! I see what you mean."

“If that works for you…” Fred said, carefully.

After long consideration, Fred took the intermediate server back home to TinyWeb for some down-to-the-wire recoding, resulting in a flawless demo for the press. JumboStores was delighted.

With their collaboration at an end, Fred wondered how JumboStores IT would ever manage on their own.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Planet DebianArturo Borrero González: Things you can do with Debian: multimedia editing

Debian

The Debian operating system serves many purposes and you can do amazing things with it. Apart of powering the servers behind big internet sites like Wikipedia and others, you can use Debian in your PC or laptop. I’ve been doing that for many years.

One of the great things you can do is some multimedia editing. It turns out I love nature, outdoor sports and adventures, and I usually take videos and photos with my friends while doing such activities. And when I arrive home I love editing them for my other blog, or putting them together in a video.

kdenlive

The setup I’ve been using is composed of several different programs:

  • gimp - image processing
  • audacity - quick audio recording / editing
  • ardour - audio recording / editing / mixing / mastering
  • kdenlive - video editing / mixing
  • openshot - video editing / mixing
  • handbrake - video transcoding

My usage of these tools ranges from very simple to more complex. In the case of gimp, for example, I mostly do quick editting, crop, resize, fix colours, etc. I use audacity for quick audio recording and editing, like cutting a song in half or quickly record my mic. Ardour is such a powerfull DAW, which is more complex to use. I can use it because my background in the audio business (did you know I worked as recording/mixing/mastering engineer in a recording studio 10 years ago?). The last amazing feature I discovered in Ardour was the hability to do side-chain compression, great!

For video editing, I started using openshot some years ago, but I recently switched to kdenlive, which from my point of view is more robust and more fine-tunned. You should try both and decide which one fits your needs.

And another awesome tool in my setup is handbrake, which allows to easily convert and transcode video between many formats, so you can reproduce your videos in different platforms.

It amazes me how these FLOSS tools can be so usefull, powerful and easy to install/use. From here, I would like to send a big thanks you a lot! to all those upstream communities.

Ardour

In Debian, getting them is a matter of installing the packages from the repositories. All this setup is waiting for you in the Debian archive. This wouldn’t be possible without the hard work of the Debian Multimedia team and other collaborators, who maintain these packages ready to install and use. Well, in fact, thanks to every single Debian contributor :-)

Planet DebianArthur Del Esposte: Plans for DebCamp and DebConf 18

DebConf18

I recently became an active contributor to the Debian project, which has been consolidated throughout my GSoC project. In addition to the great learning with my mentors, Lucas Kanashiro and Raphäel Hertzog, the feedback from other community members has been very valuable to the progress we are making in the Distro Tracker. Tomorrow, thanks to Debian project sponsorship, I will take off for Hsinchu, Taiwan to attend DebCamp and DebConf18. It is my first DebConf and I’m looking forward to meeting new people from the Debian community, learn a lot and make useful contributions during the time I am there.

During DebCamp, I plan to make the following contributions:

  • Keep working with Lucas Kanashiro in our GSoC project on Distro Tracker. In particular, I intend to finish my two open Merge Requests to improve Team’s page performance and to highlight packages with RC bugs. Also, I plan to advance in adding new packages tables to Team’s page based on PET’s categories.
  • Help DebConf19 team to bootstrap the website and help with other things that are needed.

In DebConf, I’ll make a presentation in the GSoC Session to present the advances in my project.

I hope to talk to more experienced people and collect feedback to improve my work. If anyone is interested in what I will be working on, feel free to talk to me personally, via IRC (nick: arthurmde), or email (arthurmde@gmail.com).

I am certainly looking forward to meeting Taiwan too. I’m sure I’ll be positively surprised by the culture, food, places, and people on the other side of the world.

See you soon in Taiwan!

Taipei - Taiwan

Let’s get moving on! ;)

Planet DebianDirk Eddelbuettel: nanotime 0.2.2

A new maintenance release of the nanotime package for working with nanosecond timestamps just arrived on CRAN.

nanotime uses the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it now uses a more rigorous S4-based approach thanks to a rewrite by Leonardo Silvestri.

This release re-disables tests for xts use. At some point we had hoped a new xts version would know what nanotime is. That xts version is out now, and it doesn’t. Our bad for making that assumption.

Changes in version 0.2.2 (2018-07-18)

  • Unit tests depending on future xts behaviour remain disabled (Dirk in #41).

We also have a diff to the previous version thanks to CRANberries. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

,

Planet DebianLars Wirzenius: Ick version 0.53 released: CI engine

I have just made a new release of ick, my CI system. The new version number is 0.53, and a summary of the changes is below. The source code is pushed to my git server (git.liw.fi), and Debian packages to my APT repository (code.liw.fi/debian). See https://ick.liw.fi/download/ for instructions.

See the website for more information: https://ick.liw.fi/

A notable change from previous releases should be invisible to users: the release is built by ick2 itself, instead of my old mostly-manual CI script. This means I can abandon the old script and live in a brave, new world with tea, frozen-bubble, and deep meaningful relationships with good people.

Version 0.53, released 2018-07-18

  • Notification mails now include controller URL, so it's easy to see which ick instance they come from. They also include the exit code (assuming the notification itself doesn't fail), and a clear SUCCESS or FAILURE in the subject.

  • Icktool shows a more humane error message if getting a token fails, instead of a Python stack trace.

  • Icktool will now give a more humane error message if user triggers the build of a project that doesn't exist, instead of a Python stack trace.

  • Icktool now looks for credentials using both the controller URL, and the authentication URL.

  • Icktool can now download artifacts from the artifact store, with the new get-artifact subcomand.

  • The archive: workspace action now takes an optional globs field, which is a list of Unix filename globs, for what to include in the artifact. Also, optionally the field name_from can be used to specify the name of a project parameter, which contains the name of the artifact. The default is the artifact_name parameter.

  • A Code of Conduct has been added to the ick project. https://ick.liw.fi/conduct/ has the canonical copy.

Rondam RamblingsRepublican voters are completely insane

If you were hoping that Helsinki might be Donald Trump's Joseph-Welch moment, think again.  Donald Trump will not suffer any negative consequences from his disastrous and treasonous remarks.  This is why: A new tracking poll from Reuters/ Ipsos on Tuesday showed that rank-and-file Republicans not only continue to support President Trump but refuse to believe he’s doing anything wrong. The most

TEDYou can now get customized TED Talk recommendations in your inbox

As the number of TED Talks on TED.com grows, we’ve created a new way to discover talks you’ll love: Tell us your favorite topics and areas of interest, and we’ll send you a customized email brimming with talks worth your personal attention.

Here’s how it works: Visit ted.com/recommends and tell us the topics that fascinate you most, as well as your personal goals in watching TED Talks. In other words: What do you want to get out of your time online? After you’ve answered these two quick questions, you’ll be asked to sign up for TED, or log in with your existing TED account. In less than a minute, you’ll get a personalized recommendation.

At the TED Recommends sign-in page, you can decide what kind of talks you’d most like to watch. Ask yourself: What do you hope to learn from watching a talk?

We’ll take that input and combine it with your watch history to serve up jaw-dropping, a-ha-moment-inducing, worldview-altering talks—picked just for you. The more you watch, the better the recommendations will get.

And you won’t just be taking our word for it. The recommended talks are selected by members of our community who share your passions and have strong opinions about what you need to see right now. You’ll hear from these community members in your personal email and learn why they served up what they did.

CryptogramDefeating the iPhone Restricted Mode

Recently, Apple introduced restricted mode to protect iPhones from attacks by companies like Cellebrite and Greyshift, which allow attackers to recover information from a phone without the password or fingerprint. Elcomsoft just announced that it can easily bypass it.

There is an important lesson in this: security is hard. Apple Computer has one of the best security teams on the planet. This feature was not tossed out in a day; it was designed and implemented with a lot of thought and care. If this team could make a mistake like this, imagine how bad a security feature is when implemented by a team without this kind of expertise.

This is the reason actual cryptographers and security engineers are very skeptical when a random company announces that their product is "secure." We know that they don't have the requisite security expertise to design and implement security properly. We know they didn't take the time and care. We know that their engineers think they understand security, and designed to a level that they couldn't break.

Getting security right is hard for the best teams on the world. It's impossible for average teams.

Worse Than FailureClassic WTF: The Mega Bureaucracy

Part of the reason we need a summer break is because we simply don't have the organizational skills of this particular company. I wonder if they sell consulting. Original -- Remy

Photo credit: 'digicla' at Flickr At my daytime corporate-type job, if I need to even sneeze in the general direction of a production environment, I need both a managerial and customer approvals with documentation solemnly stating that I thoroughly tested my changes and swear on a stack of MSDN licenses and O'Reilly books that I am NOT going to break anything as a result of my changes. Sure, the whole thing is a pain (and admittedly, a necessary evil), but what Bruce W. has to go through beats the pants off of anything I've ever had to go through.

For the most part, Bruce loves his job. He gets to work with a lot of intelligent and motivated people. He has been developing a new system to support a new product that has the possibility of earning his division several million dollars per year and saving the corporate parent several hundred thousand dollars per year. The net effect on the corporate parent's bottom line will be quite nice. He developed a Web front end while a fellow developer put together the data feeds. The initial development work was estimated to take about six weeks; pretty good since we only had eight weeks to work with.

However, Bruce works in a very large corporation (70,000 plus employees through out the US and several countries) and IT for the corporation has been highly centralized to the world headquarters. Smaller IT work, like the development and support for only a single division, isn't centralized but must pass through the central Mega Bureaucracy for approval and placement on the centralized servers.

...and Bruce needs their "help" to officially set up his environments.

You see, while Bruce and his group can test all day long on their local computers and servers, any kind of "live" environments must be created, blessed, and centralized by the Mega Bureaucracy. They're bigger, badder, and have more connections than anybody in your division's rank-and-file. Remember: in the Mega Bureaucracy, processes and procedures are to be followed, respected, and if necessary worshipped. Oh, and forget even thinking of installing Web services on one of the existing centralized servers. That would bring down the wrath of the entire blessed Bureaucracy for changing the purpose of an existing machine without first going through Mega Change Server Process.

Here's a brief overview of what Bruce had to go through to get four (one each for development, testing, staging, and production) Windows-based Web servers:

Week 1 - At the same time Bruce's group started the project he went to procure the servers. He was told that all he needed to do was put in a Service Request with the Windows Server Team and they would get what we needed. However, that request is cancelled because when the Windows Server Team saw that the servers were for a new application they said, "Whoa, you have violated rule #38,991 of the Mega Bureaucracy! New applications must go through the Process for Application Implementation and Navigation."

Bruce starts into the fill the first two PAIN forms (one being 20 pages long with 150 questions), sends them off to the server team, and immediately receives a response that, no, do not directly send PAIN forms to the group they go to. Instead, open a project with the Mega Bureaucracy's project tracking system, attach the forms and THEN assign the project to the group.

A few days later, he receives word that the project has been accepted, slotted, and a project manager assigned. Bruce figures, "Cool, now we are moving! I'll have my servers in no time!" He and his boss have a conference call with the PM and express to him the time critical nature of these servers. The PM agrees to push them forward saying that the request isn't complex and shouldn't take much effort.


Week 2 - Bruce receives the initial project estimate and immediately replies with his approval.


Week 4 - Bruce calls the PM to find out what's going on. He says that due to staffing cuts only a handful of requests are being processed at a time. Despite being reminded that this project is literally worth millions, he says that other projects are ahead of us and that this is simply how things are. Bruce boss escalates the issue to the head of IT for the entire division who just happens to be a member of the Project Approving Council and supposedly has the power to move the project forward.


Week 6 - Only three weeks until the promised delivery date, Bruce learns that the project still has not moved. His boss fires off a series of emails saying that the app is about to go live on a system that will earn the company millions of dollars that is running on a desktop machine sitting in a cubicle.


Week 7 - The system is now fully coded. Bruce is walking around, shaking his head, saying to himself "We have done user testing and end-to-end testing on a desktop machine-based server!"


Week 8 - The new system goes live and is serving dozens of customers daily. The difference between Production and Test environments is a Post-it Note. Power strips and network hub are carefully labeled "DO NOT TOUCH! HIGH VOLTAGE!" to prevent cleaning staff misfeance.


Week 10 - Bruce and the Windows Server Team finally have the project kick off meeting for the servers. About 15 of the 30 minute call was spent with Bruce repeatedly saying, "All I need is a Windows Server with IIS and .NET. I do not need a database server, no access to the mainframe, no massive SAN space, no Internet access, no interplanetary probe, just servers." "BUT", they say, "You stated on page 16, question 113 that your application uses a database. Where will that database come from?" Bruce explains again, "We are using existing databases assigned to our group. The database is outside of the scope of the project of setting up four Web servers."

Week 12 - Bruce and the Windows Server Team get together for their status meeting. The server team says they haven't budged since last meeting. Why? Everyone says, "Well, we're just waiting for the other shoe to drop and this becoming a big, complex, hairy project requiring massive time." Bruce once again states that all they he needs is four Web servers. Nothing more. The server design engineer says, "Wow, that is pretty simple. Shouldn't take too long at all."


Week 14 - Bruce has another status meeting with the PM and the server engineer. The engineer has put together the required diagram of the requested infrastucture and states that he only had to change a handful of things from the initial template. He says that everything should be ok and once they have the infrastructure readiness, the server builds can start. Bruce thinks, "Finally! All the other people initially assigned to the project must have realized that building four web servers isn't that big if a deal! ...haven't they?"


Week 18 - The head of IT for our division finds out that we are still waiting. Heads start rolling...even poor Bruce's. "WHY DIDN'T YOU CALL ME SIX %($$!*& WEEKS AGO???" the IT head blasts.


Week 19 - The servers are built (it only took 2 days to build them!) and are signed off for production support.

Week 20 - Bruce distributes the application URL pointing to the brand new servers.

Through all of this Bruce learned a couple things. First, don't even think of going around the Mega Bureaucracy, even if somebody says you can. The Mega Bureaucracy remembers and brands you a heretic. Second, if you think you will need help from the Mega Bureaucracy, start early, fill out all of the forms, stand in the right lines, sacrifice to the appropriate gods, and don't even hint that you would think of going around them. Finally, he who yells loudest gets move the front of the queue soonest - as holy and almighty as The Mega Bureaucracy is, they're happiest to get rid of their crabbiest customers first.

The silver lining in all of this? Apparently, the Guardians of the Mega Bureaucracy seem to now be willing to consider that there is a different tier of requests that don't require so many stopping points, designed to make sure that users really, REALLY know what they want to request. Bruce remains positive saying that, maybe in a few years, after meetings to plan meetings, forms to request forms, they will have a process that only has an initial questionnaire of 10 pages and 75 questions.

[Advertisement] Otter - Provision your servers automatically without ever needing to log-in to a command prompt. Get started today!

Rondam RamblingsI don't even know what to say here

Oh my.  Sharp-eyed CNBC reporter Christina Wilkie noticed that in a photo of the written prepared remarks of Trump's lame rollback of his catastrophic Helsinki news conference, there was a line that was crossed out.  But it wasn't crossed out all the way, you could still see what it said in the image. Here's a close-up (flipped 180 degrees so the text is right-side-up): It says, "Anyone

,

Rondam RamblingsBut what about the witch hunt?

Donald Trump is frantically backpedalling on his treasonous remarks at yesterday's Helsinki news conference. “The sentence should have been, ‘I don’t see any reason why it wouldn’t be Russia.’ Sort of a double negative,” Trump told reporters. “So you can put that in, and I think that probably clarifies things pretty good by itself.” In other words, Donald Trump wants you to believe that what he

Planet DebianErich Schubert: Facebook is overly optimistic with respect to Cambridge Analytica data scope

Facebook is too optimistic when it comes to Cambridge Analytica extends.

Sorry for this post on a fairly old topic. I just did not get around to write this up.

Several media outlets (e.g., Bloomberg) ran the story that Facebook privacy policy director Stephen Satterfield claimed that “European’s data” may not have been accessed by Cambridge Analytica in an EU hearing.

This claim is nonsense. It is almost a lie - except that he used the weasel word “may”.

For fairly trivial reasons, you can be sure that the data of at least some European’s data has been accessed. Largely because it’s pretty much impossible to perfectly separate U.S. and EU users. People move. People use Proxies. People use wrong locations. People forget to update their location. Location does not imply residency nor citizenship. People may have multiple nationalities. On Facebook, people may make up all of this, too.

Even if Dr. Aleksandr Kogan did try his best to provide only U.S. users to Cambridge Analytica, there ought to be some mistakes. Even if he only provided the data of users he could map to U.S. voter records, there likely is someone in there that has both U.S. and EU citizenship. Or that became a EU citizen since.

Because they shared the data of 87 million people. According to some numbers I found, there are around 70,000 people with U.S. and German citizenship. That is “just” a tiny 0.02% of U.S. citizens. Since Facebook users are younger than average, and in particular kids will often have both citizenships if their parents have different nationalities, we can expect the rate to be higher than that. If you now draw 87 million random samples, the chance of not having at least one of these U.S.-EU-citizens in your sample is effectively 0. This does not even take other EU nationalities into account yet.

Already a random sample of 100,000 U.S. citizens will with very high probability contain at least one E.U. citizen (in fact, at least one German citizen, because I didn’t include any other numbers but the 70,000 above). In 87 million, you likely have even several accounts created for a cat.

Says math.

To anyone trained in statistics, this should be obvious version of the birthday paradoxon.

So yes, I bet that at least one EU citizen was affected.

Just because the data is too big (and too unreliable) to be able to rule this out.

Apparently, neither the U.S. nor Germany (or the EU) even have reliable numbers on how many people have multiple nationalities. So do not trust Facebook (or Kogan’s) data to be better here…

Planet DebianLouis-Philippe Véronneau: Taiwan Travel Blog - Day 6 & 7


This is the fifth entry of my Taiwan Travel blog series! You can find my previous entries here:


I wasn't sure if people were enjoying my travel blog or if I was spamming planet.d.o with pictures of random mountain paths, but several people told me they liked it. Thanks for the feedback!

I've been busy in the last few days so for convenience's sake, I'll merge together what I did on the 14th and the 15th.

From mountain to sea

The view from the 193 road from Taroko to Hualien

I left late in the morning on the 14th from the Taroko national park were I was staying to move to Hualien. Taroko was beautiful, but there is only so much to do there and I think I did most of it.

The bike ride was easier than I thought it would be. Taroko is in the mountains so I was travelling on a downward slope pretty much the whole way. There wasn't a dedicated bike path, but the road I took (n° 193) had a speed limit of 40km/h. The view was beautiful, as this road follows the shoreline all the way to Hualien.

I guess I must have been quite a sight for the locals: a foreigner riding a bicycle a few sizes to small for him on a small country road with a large bag and hiking boots strapped behind on the rack.

Fun times! I also caught a bad sunburn, as it seems the sun is stronger here than at home :(

Hualien

After more than a week of travel, Hualien was the first large city in Taiwan I visited. Although its the largest city on the east coast, Hualien only has 100'000 inhabitants. That's a manageable size for me.

The Hualien beach

I arrived in Hualien in the beginning of the afternoon and after having checked-in at my hotel, I decided to go out for the night to have a meal and enjoy the city. I'm really happy I bought a bicycle, as it makes moving around so much easier than walking!

Following Andrew's advice, I first stopped at Danji's Bianshi shop (戴记扁食) to enjoy one of the best Bianshi I ever had. From what I understood from the pictures on the wall, this shop has been there for a least a few decades and only sells one dish: Bianshi. For those of you not familiar with Bianshi, it's a dumpling soup similar to Wonton soup, but where celery plays the leading role in flavouring the clear chicken broth. Hmmmmmm.

At the night market I got to eat a ton of food once again, from donuts filled with red bean paste to octopus takoyaki. It's also the harvest season here so I enjoyed many fresh juices like watermelon (the first time I had some juiced) and later on white jade bitter melon (白玉苦瓜), deliciously sweet and bitter.

And the Lord said ‘This bike shop shall be closed on Sunday’

On the 15th, I went out to try to get my bike fixed. I wanted to buy a longer saddle post and the bottom bracket on my bike was loose and I wanted to have it fixed.

The litchis I ate, with part of my new bike in the background

Since the bike I bought is still under warranty, I had the great idea to cycle across town to the Giant store to see what could be done. Of course, I didn't check if the store was open and ended up realising it was Sunday when I saw the sign on the door.

Some say bad things happen for a reason so I comprised, bought fruits from the fruit stall across the street and decided to go watch the waves on the beach. The mango I ate was delicious and it had been a really long time since I ate fresh litchis.

I came back to my hotel in the middle of the afternoon, once again thinking I would have a quiet night listening to podcasts while working on some DebConf stuff when aLiao (Andrew's friend who owns a recording studio in Hualien) reached out to me to ask if I had plans for the night.

I ended up at his studio listening to him and a bunch of his friends jamming. We later went to a seafood restaurant on the edge of the night market called The Tall Knight (高大侠) to have some fresh seafood grilled on charcoal.

We ate a bunch of grilled shrimps as appetizers, very large scallops cooked in their shells and oysters the size of a toy football. I'm used to the small and delicious oysters we eat raw in Canada, but this one was served cooked with homemade hot sauce. They also had Hardcore beer (a local Taiwan craft beer) on tap and it went very well with the meal.

Cheers!

Cory DoctorowSee you at Comic-Con!

I’m one of the “special guests” at this year’s San Diego Comic-Con! If you’re attending, I hope you’ll come by and see some of my programming items, especially my spotlight interview with Cecil Castellucci (Friday, July 20, 1330h-1430h, Room 24ABC), where I’ll be making an exciting announcement.

Here’s my full schedule, including some extracurriculars not listed in the regular program:


* Thursday, 1PM: Panel – Finding Comfort in the Apocalypse, Room 32AB, with Emily Suvada, Elizabeth Hand, Douglas Holgate, Andrew Smith, Scott Westerfeld and Adron Buske

* Thursday, 2:30PM: Signing (Autograph Area Table AA09)


* Thursday, 6PM: Tor Books/Den of Geek happy hour (Horton Grand Hotel Courtyard)

* Friday, 1:30PM: Building a Skeptical Techno-Utopia with Optimistic Disaster Stories (spotlight), Room 24ABC, with Cecil Castellucci

* Friday, 2:30PM: Signing (Autograph Area Table AA21)

* Friday, 5PM: Signing (Tor Books booth #2701)

* Saturday, 12PM: Fan meet & greet/signing at Nerdist House, Sparks Gallery

* Sunday, 1PM: Panel – How I Got Here: Paths to Comics-Making, Room 28DE, with Aminder Dhaliwa, Nalo Hopkinson and Calvin Reid.

* Sunday, 2:30PM: Signing (Autograph Area #18)

Planet DebianReproducible builds folks: Reproducible Builds: Weekly report #168

Here’s what happened in the Reproducible Builds effort between Sunday July 8 and Saturday July 14 2018:

Packages reviewed and fixed, and bugs filed

diffoscope development

diffoscope is our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages. This week, diffoscope version 99 was uploaded to Debian unstable by Mattia Rizzolo. It includes contributions already covered in previous weeks as well as new ones from:

reprotest development

reprotest is our “end-user” tool to build arbitrary software and check it for reproducibility. This week, version 0.7.8 was uploaded to Debian unstable by Mattia Rizzolo. It includes contributions already covered in previous weeks as additional contributions from Mattia, including:

Misc.

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

CryptogramInstalling a Credit Card Skimmer on a POS Terminal

Watch how someone installs a credit card skimmer in just a couple of seconds. I don't know if the skimmer just records the data and is collected later, or if it transmits the data back to some base station.

Worse Than FailureClassic WTF: The Source Control Shingle

Our summer break continues. I once worked on a team which made "shingles"- software modules that were layered on top of a packaged product. There were a lot of WTFs in those shingles, but nothing that can compare to this once. Original--Remy

The year was 1999 and the dot-com boom was going full-throttle. Companies everywhere were focused on building revolutionary applications using nothing but top-shelf hardware and state-of-the-art software tools. Developers everywhere were trying to figure out if they should play more foosball, more air hockey, or sit back down on their Aeron and write more code. Everywhere, that is, except Boise, Idaho. Or at least, Dave's small corner of it.

At Dave's company, developers worked at a solid pace, using reliable tools, for a stable industry. They were sub-sub-contractors on a giant project commissioned by the U.S. Navy to condense naval vessel documentation. Generally speaking, the complete documentation required for a modern warship-from the GPS calibration instructions to the giant 130-millimeter cannon repair guide-is measured in tons. By condensing the documentation into the electronic equivalent, they could not only save tremendous physical space, but they could make it much easier to navigate.

A Simple Plan

Dave's company's small piece of the pie involved writing a very specific application for a particular group of users. Their application needed to track who moved which box of classified documentation from where to where, and why. Given the very simple requirements, the entire application was assigned to Mark.

Mark believed in keeping things simple: he rarely left the command line, his text editor was notepad and his source repository was a few backup folders on a network drive. He didn't need or want more than that. It was a simple task that called for his simple methodologies.

As their app neared completion, a whole new set of requirements came in. Now, they had to add in security and logging. When Dave joined Mark's one-man team to help out with this, the current system of source control -- nothing -- became inconvenient for collaborating.

Dave suggested they set up a source-control repository, but Mark wanted to keep things simple. He devised a solution called the "source-control shingle."

Roofing and Revisions

The source-control shingle was literally that: an actual shingle from someone's house that somehow ended up in their office. It acted like a "talking stick," in that only he who possessed the shingle was allowed to edit the common libraries.

As time went on, the project's scope grew immensely. More and more developers came on board, and the source-control shingle was pushed to its limits. Despite not being in possession of the shingle, some developers broke protocol and edited the library files on the share drive. Finally, Mark agreed to use a simple source repository. He wanted to use the only source-control system that guaranteed file locks: Visual Source Safe.

Unfortunately, Source Safe was so painful to license and manage that Mark had no choice but to explore other options, some of which involved a piece of painted wood. After much arguing and cajoling, Mark agreed to try out open source CVS. Things went well for the first few days, but quickly took a turn for the worse.

"What happened to my code?" Mark asked. "I just did a CVS UPDATE and everything I wrote this morning is gone!"

"It's working fine for me," one of the developers replied.

"Same here," another joined in. "I just checked in my changes a few minutes ago, and they're still here."

"Wait," a third one questioned, "did you do an UPDATE before the COMMIT?"

"Did I what?" the second developer replied. "Oh. Crap."

Exasperated, Mark jumped. "That's it! We're going back to the shingle!"

Fortunately, some of the other developers managed to convince Mark to stick with CVS, at least for a little while longer. One of the developers even managed to enforce better source control practices using some server-side scripts. And despite Mark's constant reservations, they ended up staying with CVS throughout the project. But the whole while, Mark kept the shingle handy, just in case.

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

Planet DebianDirk Eddelbuettel: pinp 0.0.6: Two new options

A small feature release of our pinp package for snazzier one or two column vignettes get onto CRAN a little earlier.

It offers two new options. Saghir Bashir addressed a longer-standing help needed! issue and contributed code to select papersize options via the YAML header. And I added support for the collapse option of knitr, also via YAML header selection.

A screenshot of the package vignette can be seen below. Additional screenshots of are at the pinp page.

The NEWS entry for this release follows.

Changes in pinp version 0.0.6 (2018-07-16)

  • Added YAML header option 'papersize' (Saghir Bashir in #54 and #58 fixing #24).

  • Added YAML header option 'collapse' with default 'false' (#59).

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Rondam RamblingsRemember this day

If you remember nothing else about this extraordinary time in American history, remember this day, and remember this quote: “Dan Coats came to me and said they think it's Russia.  I have, uh, president Putin, uh, who just said it's not Russia.  I will say this, I don't see any reason why it would be." Let that sink in.  "I don't see any reason why it would be." So... president Trump does not

Planet DebianJacob Adams: PGP Clean Room Beta

My Google Summer of Code 2018 Project is finally far enough along that I feel that it should have a proper public beta.

PGP Clean Room

This summer I’m working on the PGP Clean Room Live CD project. The goal of this project is to make it easy to create and maintain an offline GPG key. It creates and backs up your GPG key to USB drives which can be stored in a safe place, and exports subkeys for you to use, either via an export USB or a PGP smartcard. It also allows you to sign other people’s keys, revoke your own keys, and change your keys expiration dates. The live system is built on live-build with an added python application (using GPGME to manage keys) and all networking functionality removed.

Testers Wanted

I would love to get some feedback on this build and how it could be improved. You can report bugs via salsa.d.o

Downloads

Prebuilt ISOs are available here (sig) or on Google Drive (sig)

The source code is available on salsa.d.o

Screenshots

main menu Main Menu

load menu Load Menu

advanced menu Advanced Menu

signing subkey Signing subkey

subkey algorithm Picking subkey algorithm

pick disk Picking a backup disk (in QEMU)

Planet DebianGunnar Wolf: A very nice side-project that has come to fruition: Fresh from the 1960s, my father's travel memories

So... Everybody I've interacted with along the last couple of weeks knows I'm basically just too busy. If I'm not tied up with stuff regarding my privacy/anonymity project at the university, I am trying to get the DebConf scheduling, or trying to catch up with my perpetual enemy, mail backlog. Of course, there's also my dayjob — Yes, it's vacation time, but I'm a sysadmin, and it's not like I want to give software updates much of a vacation! Of course, my family goes to Argentina for a couple of weeks while I go to DebConf, so there's quite a bit of work in that sphere as well, and... And... And... Meh, many other things better left unaccounted for ☺
But there's one big extra I was working on, somewhat secretly, over the last two months. I didn't want to openly spill the beans on it until it was delivered in hand to its recipient.
Which happened this last weekend. So, here it is!

During the late 1960s, my father studied his PhD in Israel and had a posdoctoral stay in Sweden. During that time, he traveled through the world during his vacations as much as he could — This book collects his travels through Ethiopia (including what today is Eritrea), Kenya, Uganda, Rwanda, Burundi, Turkey, Afghanistan, Pakistan, Czechoslovakia, Sweden, Norway, Iceland and India. As he took those trips, he wrote chronicles about them, and sent them to Mexico's then-most-important newspaper (Excélsior), which published each of them in four to six parts (except for the Czechoslovakia one, which is a single page, devoted to understanding Prague two years after the Soviet repression and occupation).

I did this work starting from the yellow-to-brown and quite brittle copies of the newspaper he kept stored in a set of folders. I had the help of a digitalization professional that often works for the University, but still did a couple of cleanup and QA reads (and still, found typos... In the first printed page, in the first title! :-/ ). The text? Amazing. I thoroughly enjoyed it. He wrote the chronicles being between 23 and 27 years old, but the text flows quick and easy, delightful, as if coming from a professional writer. If you can read Spanish, I am sure you will enjoy the read:

Chronicles of a backpacker in a more naïve world

Why am I publishing this now, amid the work craze I've run into? Because my father is turning 75 year old next weekend. We rushed the mini-party for him (including the book-as-a-present) as we wanted my kids to deliver the present, and they are now in a plane to South America.

The book run I did was quite limited — Just 30 items, to give away to family and close friends. I can, of course, print more on demand. But I want to take this work to a publisher — There are many reasons I believe these youth chronicles are of general interest.

AttachmentSize
libro_bwolf.jpg64.11 KB
portada.jpg368.39 KB
cronicas_de_un_mochilero.pdf25.3 MB

,

Cory DoctorowPodcast: Zuck’s Empire of Oily Rags


Here’s my reading (MP3) of Zuck’s Empire of Oily Rags, a Locus Magazine column about the corruption implicit in surveillance capitalism, which creates giant risks to users by collecting sensitive information about them in order to eke out tiny gains in the efficacy of targeted advertising. The commercial surveillance industry may not be very good at selling us fridges, but they’re very good at locating racists and thugs and getting them to support violent political movements.

MP3

Cory DoctorowPortuguese translation of Zuck’s Empire of Oily Rags

Brent Longborough did me the enormous favor of translating my latest Locus column, Zuck’s Empire of Oily Rags, into Portuguese, and sent it to me to publish.


Cory Doctorow: Zuck e seu Império de Trapos Oleosos

2 de julho de 2018 Cory Doctorow

Durante vinte anos, os defensores da privacidade vem soando o alarme sobre a vigilância comercial on-line, a maneira de as empresas coletarem dossiês detalhados sobre nós para ajudar os profissionais de marketing a nos almejar com seus anúncios. Esse alarde não surtiu efeito: em geral, as pessoas eram céticas em relação à eficácia da publicidade direcionada; os anúncios que recebíamos raramente eram muito persuasivos e, quando funcionavam, era porque os anunciantes haviam descoberto o que queríamos e se ofereceram para nos vender: as pessoas que já haviam comprado um sofá viam anúncios de sofás e, se comprassem um sofá, os anúncios persistiam por algum tempo porque os sistemas almejadores de anúncios não eram inteligentes o suficiente para saber que seus serviços não eram mais necessários, mas, de fato, onde estava o problema? O pior cenário era que os anunciantes desperdiçariam seu dinheiro com anúncios que não tinham efeito, e o melhor cenário era que as compras se tornariam mais convenientes, conforme os algoritmos preditivos facilitassem cada vez mais a busca das coisas que estávamos prestes a procurar .
Os defensores da privacidade tentaram explicar que a persuasão era apenas a ponta do iceberg. Bancos de dados comerciais eram alvos suculentos para espiões e ladrões de identidade, ainda sem falar das oportunidades para chantagem das pessoas cujos dados revelavam práticas sexuais socialmente arriscadas, crenças religiosas ou visões políticas.

Agora estamos vivendo a reação técnica, e, finalmente, as pessoas estão voltando aos defensores da privacidade, para dizer que tínhamos razão o tempo todo; dada vigilância suficiente, as empresas podem nos vender qualquer coisa: Brexit, Trump, limpeza étnica no Mianmar, e lances eleitorais bem-sucedidos para bastardos absolutos como Erdoğan na Turquia, e Orban, na Hungria.

É ótimo que a mensagem, que a privacidade é importante, esteja finalmente alcançando um público mais amplo, e é emocionante pensar que estamos nos aproximando de um ponto de inflexão pela indiferença à privacidade e à vigilância.

Mas, apesar de o reconhecimento do problema da Big Tech seja muito bem-vindo, estou preocupado que o diagnóstico esteja errado.

O problema é este: estamos confundindo o ‘persuadir’ automatizado com o ‘almejar’ automatizado. Mentiras risíveis sobre o Brexit, os estupradores mexicanos e a infiltração rasteira da lei Sharia não convenceram pessoas de outras maneiras ajuizadas que ‘para cima’ era ‘para baixo’, nem sequer que o céu era verde.

Em vez disso, os sofisticados sistemas almejadores disponíveis através do Facebook, Google, Twitter e outras plataformas de publicidade da Big Tech facilitaram a localização de pessoas racistas, xenófobas, medrosas e iradas que queriam acreditar que eram estrangeiros, financiados, por sinal, por George Soros, que estavam destruindo seu país.

Lembre-se que as eleições são no geral equilibradas na ponta da faca, mesmo para aqueles políticos que ocupam seus assentos, durante décadas, com margens pequenas: 60% dos votos é uma vitória excelente. Lembre-se, também, que o vencedor na maioria das corridas é “nenhum das opções acima”, com um grande número de eleitores se abstendo da eleição. Se mesmo um número pequeno desses não-eleitores puderem ser motivados a comparecer às urnas, os lugares seguros poderão se tornar contestáveis. Numa corrida apertada, possuir uma maneira barata de alcançar todos os sócios latentes da KKK em determinado distrito, e silenciosamente informá-los na surdinha que o Donald J. Trump é o homem deles, é fator de mudança.

Cambridge Analytica são como mentalistas de palco: estão fazendo algo trabalhoso enquanto fingem que é algo sobrenatural. Um mentalista de palco vai treinar por anos para aprender a memorizar rapidamente um baralho e, em seguida, afirmar que podem identificar sua carta graças aos seus poderes psíquicos. Você nunca vê a prática de memorização nada glamourosa e inexpressiva. A Cambridge Analytica usa o Facebook para encontrar idiotas racistas e dizer-lhes para votar em Trump e, em seguida, eles afirmam que descobriram uma maneira mística de fazer pessoas de outro modo ajuizadas votarem em maníacos.

Isso não quer dizer que a persuasão é impossível. As campanhas automatizadas de desinformação podem inundar o canal com relatos contraditórios e aparentemente plausíveis sobre o atual estado das coisas, dificultando que um observador casual compreenda os eventos. A repetição a longo prazo duma narrativa consistente, mesmo cabalmente desequilibrada, pode criar dúvidas e encontrar adeptos – pense na negação da mudança climática, ou nas conspirações sobre George Soros, ou no movimento anti- vacinas.

Esses são processos longos e lentos, entretanto, que ao longo dos anos fazem pequenas mudanças na opinião pública, e funcionam melhor quando há outras condições que os alimentam – por exemplo, movimentos fascistas, xenófobos e nativistas que são as servas da austeridade e da privação. Quando por longos tempos você não tem o suficiente, você está maduro para mensagens culpando seus vizinhos por tê-lo privado de seu quinhão.

Mas não precisamos de vigilância comercial para criar multidões furiosas: Goebbels e Mao fizeram muito bem com técnicas analógicas, sem recursos digitais.

O Facebook não é um raio de controle mental. É uma ferramenta para encontrar pessoas que possuem características incomuns e difíceis de localizar, seja “pessoa pensando em comprar uma nova geladeira”, “pessoa com a mesma doença rara que você” ou “pessoa que pode participar de um pogrom genocida”. E depois vendendo-os um belo lado-a-lado ou algumas tochas tiki, enquanto os mostra a prova social da boa utilidade de seu curso de ação, na forma de outras pessoas (ou bots) que estão fazendo a mesma coisa, de forma que eles se sentem parte da multidão.

Mesmo que os raios de controle mental continuem uma ficção científica, o Facebook e outras plataformas de vigilância comercial ainda são preocupantes, e não apenas porque permitem que pessoas com pontos de vista extremos se encontrem. O ato de reunir enormes dossiês sobre todos no mundo é assustador por si só: na Camboja, o governo autocrático usa o Facebook para identificar dissidentes e sujeitá-los à prisão e tortura; o serviço de Alfândega e Proteção de Fronteiras dos EUA usa as mídias sociais para encontrar visitantes culpados por asociação, impedindo-os de entrar no país com base em seus amigos, afiliações e interesses. Também há os ladrões de identidade, chantagistas e vigaristas que usam dados de agências de crédito, dados vazados de usuários, e mídias sociais para arruinar a vida das pessoas. Finalmente, há os hackers que reforçam seus ataques de “engenharia social” pela coleta de informações pessoais vazadas a fim de realizar falsas e convincentes representações que ludibriam seus alvos de forma a revelar informações que os permitam invadir redes vulneráveis.

Está na moda tratar as disfunções das mídias sociais como desfechar da ingenüidade dos primeiros tecnólogos, que não conseguiram prever esses resultados. A verdade é que a capacidade de construir serviços semelhantes ao Facebook é relativamente comum. O que era raro era a imprudência moral necessária para levar isso a cabo.

A questão é esta: sempre foi óbvio que, ao espionar os usuários da Internet, você poderia melhorar a eficácia da publicidade. Isso não é tanto porque a espionagem fornece discernimento fantástico sobre novas maneiras de convencer as pessoas a comprar produtos, mas sim uma homenagem ao quão ineficiente é o marketing. Quando a taxa de sucesso esperada de um anúncio está bem abaixo de um por cento, dobrar ou triplicar sua eficácia ainda deixa você com uma taxa de conversão de menos de um por cento.

Mas também ficou óbvio desde o início que acumular enormes dossiês sobre todos os que usavam a Internet, enquanto geraria ganhos mínimos para os anunciantes, poderia criar problemas reais para toda a sociedade, e que por comparação tornariam pífios tais ganhos.

É como se Mark Zuckerberg acordasse uma manhã e percebesse que os trapos oleosos que acumulava em sua oficina poderiam ser refinados para um óleo cru de baixo teor e baixo grau. Ninguém pagaria muito por esse óleo, mas havia muitos trapos oleosos, e desde que ninguém lhe pedisse para pagar pelos inevitáveis incêndios horríveis que resultariam do enchimento das garagens do mundo com trapos oleosos, ele poderia obter um lucro considerável.

Dez anos mais tarde, tudo está em chamas e estamos tentando dizer a Zuck e seus amigos que eles vão precisar pagar pelos danos e instalar os tipos de equipamentos de supressão de incêndio nos quais qualquer pessoa que esteja armazenando panos oleosos deveria ter investido desde o começo. Mas a indústria de vigilância comercial reluta absolutamente em contemplar qualquer coisa desse tipo.

Isso porque os dossiês de bilhões de pessoas possuem o poder de causar danos quase inimagináveis e, no entanto, cada dossiê traz apenas alguns dólares por ano. Para que a vigilância comercial seja rentável, é preciso socializar todos os riscos associados à vigilância em massa e privatizar todos os ganhos.

Há uma palavra antiquada para isso: corrupção. Em sistemas corruptos, alguns maus atores custam bilhões a todos os demais, a fim de gerar milhões – as economias que uma fábrica pode obter ao despejar a poluição no abastecimento de água são muito menores do que os custos que todos suportamos ao sermos envenenados por efluentes. Mas os custos são amplamente difusos, enquanto os ganhos são fortemente concentrados, de modo que os beneficiários da corrupção sempre podem gastar mais do que suas vítimas para permanecerem limpos.

O Facebook não tem um problema de controle mental, tem um problema de corrupção. A Cambridge Analytica não convenceu pessoas decentes a se tornarem racistas; convenceram os racistas a se tornarem eleitores.


Tradução por: Brent Longborough (@orelhoes)

Esta tradução está licenciada com uma Licença Creative Commons – Atribuição 4.0 Internacional.

Krebs on Security‘LuminosityLink RAT’ Author Pleads Guilty

A 21-year-old Kentucky man has pleaded guilty to authoring and distributing a popular hacking tool called "LuminosityLink," a malware strain that security experts say was used by thousands of customers to gain unauthorized access to tens of thousands of computers across 78 countries worldwide.

CryptogramReasonably Clever Extortion E-mail Based on Password Theft

Imagine you've gotten your hands on a file of e-mail addresses and passwords. You want to monetize it, but the site it's for isn't very valuable. How do you use it? You convince the owners of the password to send you money.

I recently saw a spam e-mail that ties the password to a porn site. The e-mail title contains the password, which is sure to get the recipient's attention.

I do know, yhhaabor, is your password. You may not know me and you're most likely thinking why you're getting this email, right?

actually, I actually setup a malware on the adult video clips (pornographic material) web site and you know what, you visited this web site to have fun (you know what I mean). While you were watching videos, your web browser began operating as a RDP (Remote Desktop) having a key logger which provided me accessibility to your display and web camera. after that, my software obtained your entire contacts from your Messenger, social networks, and email.

What exactly did I do?

I created a double-screen video. First part shows the video you were viewing (you've got a fine taste ; )), and 2nd part displays the recording of your webcam.

What should you do?

Well, I believe, $2900 is a reasonable price for our little secret. You will make the payment through Bitcoin (if you don't know this, search "how to buy bitcoin" in Google).

This is clever. The valid password establishes legitimacy. There's a decent chance the recipient has visited porn sites, and maybe set up an account for which they can't remember the password. The RDP attack is plausible, as is turning on the camera and downloading the contacts file.

Of course, it all fails because there isn't enough detail. If the attacker actually did all of this, they would include the name of the porn site and attached the video file.

But it's a clever attack, and one I have not seen before. If the attacker asked for an order of magnitude less money, I think they would make more.

EDITED TO ADD: Brian Krebs has written about this, too.

Worse Than FailureClassic WTF: The Virtudyne Saga

As we usually do around this time of year, it's summer break season for TDWTF. This week, we're going to rerun some old classics, starting with this legend from 2006, compiled into a single article. --Remy

The Virtudyne saga (published 2006-Oct-10 through 2006-Oct-13) is my all time favorite. It tells the story of the rise and fall of Virtudyne, one of the largest privately-financed ($200M) disasters in our industry. Like most articles published here, all names have been changed to protect the guilty, and I've worked very closely with Rob Graves (the submitter) to ensure that this presentation is as close to how it happened as possible.


Part I - The Founding

By most people's standard, The Founder was very wealthy. A successful entrepreneur since age seventeen, he built several multi-million dollar companies and amassed a fortune larger than that of most A-list Hollywood celebrities. He prided himself on having one of the largest private collections of Egyptian artifacts in the world and prominently displayed many of them in his Great Room. And it truly was a great room: having been to The Founder's mansion several times, Rob recalls that his two-story, four-bedroom home could easily fit inside the Great Room.

The Founder was at home one day, doing whatever retired rich people did in 1999, and became extremely aggravated with how slow his brand-new, top-of-the-line computer was running. While cursing Microsoft Office, he had an "ah-ha" moment: he could build a better Microsoft Office.

Recalling his days as a Digital PDP-11 programmer, he knew that he could write financial software that would support fifty users, perform great, and run in 256-bytes of memory. Given the monumental advances in the twenty-years since he coded, he was elated just to think what would be possible with a bunch of top-notch programmers such as himself. He wondered just how many people it would take to build a Microsoft Office killer.

One thing led to another and Virtudyne was born. Its goal was modest: become the next Microsoft Office killer. The Founder hired his long-time colleague as the Chief Information Officer and together, they would create The Plan. It was simple: develop an internet/intranet based Office/Collaboration system that would deliver "90% of functionality that 90% of [Microsoft Office] users use."

An avid programmer himself, the CIO knew exactly how they could accomplish this. He convinced The Founder that, with a handful of programmers helping him, he could develop a client/server Microsoft Office Killer using Visual Basic 6. And with the latest hardware available, their application could easily scale to support twenty million users using one, maybe two servers. And best of all, it would all take only six months to create.

It was the perfect opportunity to jump on the .com bandwagon. They just could feel the IPO beckoning them. The Founder invested a few million of his own dollars and the CIO started hiring.

One of the first people the CIO approached was Rob Graves. He wanted Rob to become the database administrator, telling him only that Virtudyne was a pre-IPO startup bankrolled by The Founder. As tempting as it was, Rob had a second kid on the way and declined the offer. The CIO would have to find another DBA for the project.

What better place to find a DBA than the same place he turned for the rest of the initial hiring: the local Visual Basic special interest group. In fact, not only did he find a DBA at the SIG, he found one that proclaimed to be one of the greatest DBA in the world. And not because he possessed extensive database administration skills, but because he was willing to admit to The Truth: with the GUI-tools and automagic processes that modern databases offer, all those extensive database administration skills are meaningless.

Sadly, the DBA was one of the more talented members of the initial Virtudyne team.

Part II - The Gathering

The Founder had little trouble convincing his millionaire friends to invest in Virtudyne. It wasn't so much the idea of a Microsoft Office Killer, but that fact that it was 1999 and just about anyone with an internet company could go public and become an overnight billionaire. Within one month of The Founder's grandiose idea, he had secured an impressive eleven million in funding.

While The Founder solicited investors, the Chief Information Officer solicited employees. The CIO knew it would take "only a handful of strong programmers" to develop the Microsoft Office Killer and hired ten of the best programmers he could find. He promised a high salary, good stock options, and the chance to beat the market leader at their own game. Though his team's competence was minimal, their confidence was as strong as ever. They were all eager to build the Microsoft Office Killer.

It was the opportunity of a lifetime handed to the CIO on a silver platter: millions in capital and a dedicated team of developers. It was up to him to get busy with a clear vision, detailed requirements, a throughout market analysis, an extensive design, and solid architecture. Instead, he discovered something much more important: Magic: The Gathering.

The CIO dedicated his "lunch break" to his Magic card collection. This, of course, meant that he'd spend much of his day thinking up new deck concepts, building them, and testing them out. He even got some of his developers hooked: they'd all get together during their "lunch break" and play, trade, and chat about the latest happenings in the world of Magic: The Gathering.

Don't get me wrong, Magic wasn't the Chief Information Officer's only focus. With his new job title, he was eligible to receive executive-level trade publications for free. In fact, one of his first acts as CIO was to purchase a top-of-the-line solid ink printer. In addition to producing sharp full-color graphs for presentation packets, it printed up some wicked high-quality "proxy cards" for everyone's Magic decks.

Days turned into weeks, weeks turned into months, and next thing they knew, six months had passed and not a single line of code had been written. What made this especially bad was the fact that the investors were flying in to town to check on everyone's progress. They were all eager to see just how their Microsoft Office Killer was coming along.

Thank goodness that the Chief Information Officer chose Visual Basic 6 as their platform. Real magic ensued when the following were combined: a handful of developers, a caffeine-filled all-nighter, and VB6's wonderful ability to drag & drop controls onto Windows form and "hard code" what shows in the labels, text boxes, drop downs, etc.

The investors were not impressed. They were astonished. In fact, the demonstration convinced them that, not only the project was on track, but that Virtudyne was poised to take on Microsoft and its ubiquitous office suite. Word spread fast and even more investors signed up. Tens of millions of dollars started pouring into Virtudyne.

The new investment might have been the CIO's motivation to finally get cracking on the project. Or it could have been the fact that the .com bubble was starting to burst and that meant they'd have to make a real attempt at making a product. He immediately started hiring again. And I mean hiring. A massive recruiting campaign was initiated and developers from all over the country were brought in. Within a year, the Virtudyne CIO commanded an army of I.T. professionals whose skill levels ranged between complete ineptitude and moderate competence.

The Chief Information Officer also purchased the best server he could find advertised in his executive trade publications: the Unisys ES7000. It was a thirty-two processor beast with sixty-four gigabytes of RAM and an attached EMC CLARiiON storage server. This $1.3M machine would be the single production server for their anticipated 20,000,000 users.

With all the new talent and the fancy new hardware, development of the Microsoft Office Killer finally began. The biggest hurdle that faced the developers was the new requirements. You see, one of the major selling points to investors was that Virtudyne's office suite already had every feature they asked for: it ran on Windows, Linux, and even Palm OS. All the developers had to do was make it actually do that.

Rob Graves joined Virtudyne around its second-year anniversary. He had been contacting part-time, off-and-on since day one, and they finally made him an offer he could not refuse: lead role in a company of 100+ developers, top-of-the-line development hardware, a dedicated QA team, and most of all, a $50,000 raise with five weeks paid vacation. No one could top that in the post .com-bubble.

In the year that followed, Rob found himself in the middle of quite a few political battles between the "do it right" and the "do it now" developers. Nothing too spectacular, especially in the context of this entire Virtudyne saga, but Rob did note who won the argument over whether or not to use the special coding techniques recommended by Unisys and Microsoft to utilize the server's full potential. I'll let you guess which side that was.

Despite all this, Virtudyne lacked one thing: customers. Allow me to clarify that because saying that they lacked "customers" might imply they had "a" customer. They didn't. The sales department of eight was unable to find a single organization willing to license their product.

This was especially problematic because their initial $94M war chest had dwindled to less than $10M. Investors were starting to wonder about their "six-months-to-develop Microsoft Office Killer" and stopped pouring money into Virtudyne. Something needed to be done.

Part III - The Savior Cometh

Virtudyne's first three years are best summed up with a single word: disastrous. Nearly $90M had been spent developing a product that was barley functional and completely unsalable. Most would call that "miserable failure" and encourage all involved to salvage what they could, abandon ship, scuttle the remains, and never look back. But one person saw it as the golden opportunity; he was known as The Savior

The Savior was a self-made billionaire who struck it rich doing the type of business that makes unregulated industries regulated. He heard about Virtudyne's struggles and wanted to help out. He contacted the powers that be and offered some very reasonable terms. In exchange for investing $100M, he would take over operations and sit as chairman on the board of directors. It seemed to be a a win-win for everyone.

Even the Virtudyne employees were excited. They welcomed their new overlord with open arms and truly believed that The Savior would turn the company around with his "new management team of highly-qualified executives with a proven track record." Such statements tend to be very convincing when accompanied with a hundred million dollar investment.

Unfortunately, employee confidence wore off almost immediately. It wasn't so much that the fact that the superstar executives consisted primarily of The Savior's immediate family, but more the fact that they managed to set the bar of incompetence even higher. I suspect that, given yesterday's article, this might seem impossible, so I'll share my favorite three people that The Savior brought in.

First and foremost, there was the new chief of operations, heralded as a "brilliant innovator" and "technological wizard." He was also The Savior's eldest son. Junior's grasp on technology is best illustrated with this simple anecdote: one day, Junior was walking past Rob Graves' office and saw a graph actively moving around on the screen. He got incredibly exited and wanted to know how he could get the cool looking monitoring software Rob was using to watch their World Wide Server. Rob just didn't have the heart to tell him it was the "Bars and Waves" visulization from Windows Media Player.

One of Junior's first acts as operations chief was to partner up with a major hardware vendor peddling another completely unsalable product. It was a massively-parallel server that featured a proprietary operating system with an integrated database. The sales rep told them that "reliability, not speed, is our primary concern" and they meant it. The $350,000 development server had the same processing power as a 600MHz Pentium II that even a charity organization would refuse as a donation.

Perhaps Junior's logic was that anyone stupid enough to buy the hardware would be stupid enough to buy their Microsoft Office Killer. Unfortunately, Virtudyne seemed to hold a monopoly on the world's supply of stupidity. The expensive hardware and vast amount of effort to port the software resulted in only a single sale, and it was a sale for the hardware vendo to Virtudyner.

The next person on the list was known as The VP of Nothing. I don't that's a very fair title because he actually did two things. First and foremost, despite having no direct reports or job responsibilities, he collected a six-figure paycheck. And secondly, he was allowed to bypass the proxy filter to surf the web; it was no secret why. A curious network administrator looked in the logs and discovered that The VP of Nothing spent a lot of time looking at pictures of large Amazonian women wrestling with little men. Seriously.

My personal favorite that The Savior brought in was The Janitor. Now it may seem odd that the chairman of the board would insist upon changing cleaning companies, but it's even stranger what the new "cleaning company" consisted of: The Savior's youngest son. The Janitor was a trust-fund baby and wealthier than most of us will ever be. It became pretty apparent why he couldn't keep a job anywhere else: after a few months of his cleaning service, ants and cockroaches were everywhere, the VB team had a gnat infestation, and the restrooms became so dirty that most managers allowed their employees to go home if they needed to use the facilities.

Amazingly, this new team was able to find a paying customer. It was The City. Virtudyne's office suite was to be installed at all libraries and made available for download by city residents. All it took was some lobbying at city council, a few calls to the local media, and a sizable march on city hall with "unemployed" protestors demanding the city provide free office software. Although the majority of the protesters were Virtudyne employees, the city finally agreed and signed an $8,500,000 three-year contract, collectable upon timely delivery of the software specified in the contract.

The main problem (well, aside from the fact that Virtudyne's Office Killer was a joke compared to any office suite) was that the contract called for a product that would replace Microsoft Access. In fact, it was a major selling point: the sales VP gave a demo of a product that didn't exist.

It fell on Rob Graves and a handful of other developers to create an application in two weeks that would "allow users with no database and minimal computer knowledge to: build applications; add users and groups to access the application; set security at the form-, record-, and field- level;" and so on. Rob is embarrassed to report that they actually managed to deliver a completely useless application that met every word in the ambiguous requirements to technically fulfill the contract.

None of tha mattered, though. Employee morale was at an all time high and things were finally starting to look good. It only took three and half years and nearly $150M dollars, but they finally made eight and half million dollars. Unfortunately, the sale also brought something else to the Virtudyne: paranoia.

Junior held a company-wide meeting to discuss a very serious issue: Microsoft was onto them. They were shaking in their boots and saw Virtudyne as a major threat. They would stop at nothing to get their grubby hands on the product and might even try to steal the source code. Of course, at that point, about half the people at Virtudyne realized that all one would have to do to "get their grubby hands" on their Microsoft Office Killer was to go into any of The City's public libraries and ask for an installation disk. Obviously, Junior wasn't in that half.

Within days, Junior ordered cameras to cover every square inch of the Virtudyne facility. Fingerprint scanners were installed at every door, both inside and out, and full-time security guards were placed at key locations throughout the building. All exterior windows were covered with a translucent film to prevent Microsoft from peeping in and, just to be safe, computer monitors could no longer face outside windows. Key employees were issued with special pagers that allowed them to discretely press a button to alert the private investigator if they found themselves being followed by Microsoft's white vans.

The draconian security measures didn't help the recent boost in employee morale. In fact, over the next year, employee morale sunk to an all-time low, leading to a mass exodus from the company. Key employees were dropping like flies and all Junior would do to maintain headcount was to hire more employees.

Eventually, Junior struck a good balance between tight security and employee indifference and managed to stabilize the loss. Unfortunately, that didn't help business much. Almost two years had passed since the single sale to The City and Virtudyne couldn't even give away their product. It's hard to say whether it was the terrible product itself or the fact that the Virtydune's contract with The City sparked a state-wide scandal with accusations of impropriety going all around.

What Virtudyne needed was a new market. A market that hadn't heard of Virtudyne before. And preferably, one that wouldn't do any research before spending millions to license their product.

Part IV - The Digital Donkey

After three years of full-time employment at Virtudyne, Rob Graves finally decided to call it quits. Most of Rob's friends and family thought he was insane to leave a cushy job where he was making 30% more than he could anywhere else in town. But then again, most of his current and former coworkers thought he was insane for staying so long.

Virtudyne had blown through nearly $200,000,000 of investor capital over five years to develop their Microsoft Office Killer. All they had to show for it was a barely functional product with a small subset of Microsoft Office's features and a single $8.5M sale. And technically, they only collected $5.8M on the sale because their customer withdrew the contract.

The sales and marketing department were desperate for ideas. They literally couldn't give their software away; anyone with even the most basic knowledge of Google could find out how well Virtudyne's first customer worked out. No one wanted to be their second.

But just then, it dawned on the sales team. They needed to find a market where the Internet had not yet reached. In such a market, their office suite would develop interest and that interest would lead right in to sales. One of the executives knew exactly how to find and penetrate such a market. They would use The Digital Donkey.

The CEO was a bit skeptical at first, but eventually realized how great of an idea it was. It was the perfect plan to sell their product to a market that no one else has ever tapped before. He signed off on the project.

Virtudyne engineers were tasked with figuring out a way to attach a satellite dish, laptops, and solar cells to a donkey. This Digital Donkey would then take Virtudyne's software along with a satellite internet connection to disenfranchised villagers living in the rural parts of India. The villagers would then be able to surf the 'net and use Virtudyne's software suite to create documents and communicate with "God knows who."

Everything would go through Virtudyne's server and they would eventually start billing the local governments for usage. I think it goes without saying that, like virtually every idea coming from Virtudyne's management team, the Digital Donkey was a miserable failure.

Virtudyne's offices are still open to this day, but no one's sure for how long. They've slightly improved their product over the years and tried to sell it under many different names to many different people. But still, nothing seems to work. Rob keeps in touch with a programmer at Virtudyne and confirmed that, as of two or three months ago to this day, there has yet to be a second sale.

Addendum

Several readers had a hard time believing that Virtudyne actually tried to create a Digital Donkey. It's important to note how ubiquitous donkeys/mules/camels/etc are in everyday tasks and transportation in the third world. So much so that the idea of a "Digital Donkey" did not even originate at Virtudyne. Consider this successful experiment from the International Federation of Library Associations and Institutions:

"The mobile units are Donkey Drawn Electro-Communication Library Carts. Besides functioning as a mobile library with a collection of books and other printed works, it works as a centre for electric and electronic communication: radio, telephone, fax, e-mail, Internet."
-- http://www.ifla.org/V/press/pr0225-02.htm (with pictures, thanks to Antitorgo for finding)

Obviously the Digital Donkey illustration is hyperbolized for fun. What should strike you as unbelievable is not the digital donkey, but the fact that Virtudyne actually considered trying to profiting from such a market. But then again, looking at their track record, that part is not so hard to believe ...


Happy New Year, All! I'll be back on Tuesday, January 2nd with some fresh, new content =-)

[Advertisement] Forget logs. Next time you're struggling to replicate error, crash and performance issues in your apps - Think Raygun! Installs in minutes. Learn more.

Planet DebianRuss Allbery: Review: Effective Python

Review: Effective Python, by Brett Slatkin

Publisher: Addison-Wesley
Copyright: 2015
ISBN: 0-13-403428-7
Format: Trade paperback
Pages: 216

I'm still looking for a programming language book that's as good as Joshua Bloch's Effective Java, which goes beyond its surface mission to provide valuable and deep advice about how to think about software construction and interface design. Effective Python is, sadly, not that book. It settles for being a more pedestrian guide to useful or tricky corners of Python, with a bit of style guide attached (although not as much as I wanted).

Usually I read books like this as part of learning a language, but in this case I'd done some early experimenting with Python and have been using it extensively for my job for about the past four years. I was therefore already familiar with the basics and with some coding style rules, which made this book less useful. This is more of an intermediate than a beginner's book, but if you're familiar with list and hash comprehensions, closures, standard method decorators, context managers, and the global interpreter lock (about my level of experience when I started reading), at least half of this book will be obvious and familiar material.

The most useful part of the book for me was a deep look at Python's object system, including some fully-worked examples of mix-ins, metaclasses, and descriptors. This material was new to me and a bit different than the approach to similar problems in other programming languages I know. I think this is one of the most complex and hard-to-understand parts of Python and will probably use this as a reference the next time I have to deal with complex class machinery. (That said, this is also the part of Python that I think is the hardest to read and understand, so most programs are better off avoiding it.) The description of generators and coroutines was also excellent, and although the basic concepts will be familiar to most people who have done parallelism in other languages, Slatkin's treatment of parallelism and its (severe) limitations in Python was valuable.

But there were also a lot of things that I was surprised weren't covered. Some of these are due to the author deciding to limit the scope to the standard library, so testing only covers unittest and not the (IMO far more useful) pytest third-party module. Some are gaps in the language that the author can't fix (Python's documentation situation for user-written modules is sad). But there was essentially nothing here about distutils or how to publish modules properly, almost nothing about good namespace design and when to put code into __init__.py (a topic crying out for some opinionated recommendations), and an odd lack of mention of any static analysis or linting tools. Most books of this type I've read are noticeably more comprehensive and have a larger focus on how to share your code with others.

Slatkin doesn't even offer much of a style guide, which is usually standard in a book of this sort. He does steer the reader away from a few features (such as else with for loops) and preaches the merits of decomposition and small functions, among other useful tidbits. But it falls well short of Damian Conway's excellent guide for Perl, Perl Best Practices.

Anyone who already knows Python will be wondering how Slatkin handles the conflict between Python 2 and Python 3. The answer is that it mostly doesn't matter, since Slatkin spends little time on the parts of the language that differ. In the few places it matters, Effective Python discusses Python 3 first and then mentions the differences or gaps in Python 2. But there's no general discussion about differences between Python 2 and 3, nor is there any guide to updating your own programs or attempting to be compatible with both versions. That's one of the more common real-world problems in Python at the moment, and was even more so when this book was originally written, so it's an odd omission.

Addison-Wesley did a good job on the printing, including a nice, subtle use of color that made the physical book enjoyable to read. But the downside is that this book has a surprisingly expensive retail ($40 USD) for a fairly thin trade paperback. At the time of this writing, Amazon has it on sale at 64% off, which takes the cost down to about the right territory for what you get.

I'm not sorry I read this, and I learned a few things from it despite having used Python fairly steadily for the last few years. But it's nowhere near good enough to recommend to every Python programmer, and with a bit of willingness to search out on-line articles and study high-quality code bases, you can skip this book entirely and never miss it. I found it oddly unopinionated and unsatisfying in the places where I wish Python had more structure or stronger conventions. This is particularly strange given that it was written by a Google staff engineer and Google has a quite comprehensive and far more opinionated coding style guide for Python.

If you want to dig into some of Python's class and object features or see a detailed example of how to effectively use coroutines, Effective Python is a useful guide. Otherwise, you'll probably learn some things from this book, but it's not going to significantly change how you approach the language.

Rating: 6 out of 10

,

Planet DebianDirk Eddelbuettel: RcppClassic 0.9.11

A new maintenance release, now at version 0.9.11, of the RcppClassic package arrived earlier today on CRAN. This package provides a maintained version of the otherwise deprecated initial Rcpp API which no new projects should use as the normal Rcpp API is so much better.

Per another request from CRAN, we updated the source code in four places to no longer use dynamic exceptions specification. This is something C++11 deprecated, and g++-7 and above now complain about each use. No other changes were made.

CRANberries also reports the changes relative to the previous release.

Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Planet DebianMike Hommey: Announcing git-cinnabar 0.5.0 beta 4

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What’s new since 0.5.0 beta 3?

  • Fixed incompatibility with Mercurial 3.4.
  • Performance and memory consumption improvements.
  • Work around networking issues while downloading clone bundles from Mozilla CDN with range requests to continue past failure.
  • Miscellaneous metadata format changes.
  • The prebuilt helper for Linux now works across more distributions (as long as libcurl.so.4 is present, it should work)
  • Updated git to 2.18.0 for the helper.
  • Properly support the pack.packsizelimit setting.
  • Experimental support for initial clone from a git repository containing git-cinnabar metadata.
  • Changed the default make rule to only build the helper.
  • Now can successfully clone the pypy and GNU octave mercurial repositories.
  • More user-friendly errors.

Cory DoctorowThe economics of Walkaway




Edgeryders’ Alberto Cottica has published a detailed analysis of the economics of Walkaway, at the micro-, mezzo-, and macroscale. It’s a good, crisp analysis that really captures what I was going for.


Writers are notoriously bad at knowing what they’re doing and why, and good criticism is just as interesting for writers to read as it is for readers.

The economics in Walkaway are my attempt to nail down a bunch of half-formed ideas that have been knocking around in my own thoughts for decades. Cottica’s analysis actually improves on some of what I was able to do, and was a great read.


These statements are important in Walkaway, because they dispose of methodological individualism. The reasoning works like this:

1. Most people like building things together. As long as the two elements of building and sociality are present, you do not need to obsess too much about incentives. In practice, you can blackbox individual behavior: observe what they do, then build a model in which they do it. No need to derive this behavior as the equilibrium strategy of a problem. This is a position close to behavioral economics.

2. What matters, instead, are technologies for cooperation. Groups of humans that are better at cooperating will prosper at the expense of other groups that are not as good. Groups of humans get better at cooperating by adopting systems of rules that make cooperation easier. Therefore, humans are subject to evolutionary pressure both at the individual level and at the group level, and the in the group level the pressure is cultural. This is the interpretation proposed by cultural evolution biologists like E.O. Wilson and Joseph Henrich.

3. It follows that an effective economic theory should not focus on individual behavior as an equilibrium of a set of individual incentives, but on system-level behavior as an equilibrium of interaction protocols.

Planet DebianElena Gjevukaj: Google Summer of Code with a Debian Project

GETTING ACCEPTED TO GOOGLE SUMMER OF CODE

Yes! My project proposal was selected.

First of all I want to mention that I began my open source adventure with Debian.

I started to participate in the open source events like Hackathons, BSP and Conferences and doing small contribution to different projects and this is how everything started.

WHAT IS GOOGLE SUMMER OF CODE?

Google Summer of Code is a global program focused on bringing more student developers into open source software development. Undergrads, students in graduate programs, PhD candidates can take part in GSoC. Due to the prestigiousness of the program and the weight of Google’s name to go with it, acceptance into the program is no easy task. GSoC is based on the model of different organizations providing projects for students all around the world. Each organization assign mentors for every project and students are paired up with mentors once they get selected.


MY PROJECT: Wizard/GUI helping students/interns apply and get started

Throughout the application process and first few weeks of programs like Google Summer of Code and Outreachy, applicants typically need to work through many things for the first time, such as creating their own domain name and blog, mail account with proper filters, creating SSH and PGP keys, linking these keys with a Github account, joining mailing lists, IRC and XMPP channels, searching for free software groups in their local area and installing useful development tools on their computer. Daniel Pocock’s blog “Want to be selected for GSoC?” lists some of these steps with more details. This project involves creating a Python script with a GUI that helps applicants and interns complete as many of these steps as possible, in less than an hour. Imagine that a student has only just installed Debian, they install this script from a package using Synaptic and an hour later they have all their accounts, communications tools, development tools and a blog (using Jekyll/Git) up and running.


COMMUNITY BONDING

In practice, the community bonding period is all about, well, community bonding. Rather than jumping straight into coding, you’ve got some time to learn about your organization’s processes - release and otherwise - developer interactions, codes of conduct, etc.


WEEK 1

In the very frist week of GSoC we all were preparing for OSCAL’18, one of the biggest technology conferences held in my Ballkan. I was fortune enough to meet my mentor Daniel Pocock in his visit for this conference. He helped me a lot to get started with the project.

It was the week that mentors devided the tasks between us interns. After getting my tasks, I started with the research for the Python API’s, templates and other tools that I needed to use for my work on the project.

I read about python-git API, about python templates and together with my mentor we decided which template was the best to use.

Event: I meet other GSoC interns here in Kosovo, Diellza Shabani and Enekelena Haxhiu, where we discussed about our projects.


WEEK 2

The second week of GSoC I spent mostly learning new things and updating packages that I needed for the project, also I started writing the script to create a blog and learning Python and Jekyll Github pages.

Why Jekyll you ask?

  • Jekyll sites are basically static sites with an extra templating language called Liquid so it’s a small step to learn if you already know HTML, CSS and JavaScript. We can actually start with a static site and then introduce Jekyll functionality as it’s needed.

For templating we decided to use the Mako template.

<%inherit file="base.html"/>
<%
    rows = [[v for v in range(0,10)] for row in range(0,10)]
%>
<table>
    % for row in rows:
        ${makerow(row)}
    % endfor
</table>

<%def name="makerow(row)">
    <tr>
    % for name in row:
        <td>${name}</td>\
    % endfor
    </tr>
</%def>

Towards the beginning of the month, Daniel recommended that we could use Salsa for the development phase.


WEEK 3 & 4

In the begining og the third week of the project I was busy with two important things.

As asked from my mentors, I was doing a research on what things should be included in Wizard, in order to make our project more helpful. Meeting first hand with new people starting in the open source world I knew excatlly what should we include.

Things as:

  • Email account

  • Blog created

  • Github account

  • Debian wiki personal home page

  • IRC nick

  • IRC client configuration file generated with IRC nick in it

  • SSH key

  • PGP key

  • Fingerprint slips printed, PDF generated

  • Thunderbird and Lightning configured

  • IMAP

  • Labels

  • Enigmail

  • Lightning polling events and tasks for the chosen communities

  • CardBook plugin for contacts

  • Package install

  • Domain name

The other thing was my part on the creating the Jekyll blog.

I want to say that as a developer this project allowed me to explore and learn a lot of new things.


WEEK 5 & 6

I finished the first phase of the program succesfully. These weeks were the most challenging week so far.

I was getting started with Django Framework and Bootstrap, dynamic blog and compiling function.

You can take a look at my code in these repositories:

In this process I learned a lot about the Liquid templating language, Pipenv, Kivy and the Django Framework.


WEEK 7 & 8

I’m getting used to the dictionary in the Python and I wrote the function to generate the blogs from the templates and for the moment I’m in the testing phase.

def main():
    dictionary = createDictionary()

blog_params['first_name'] =   'Test'
blog_params['last_name'] = 'Student'
blog_params['email_address'] = 'test@school.edu'

create_blog(blog_params)

Planet DebianSteve Kemp: Automating builds via CI

I've been overhauling the way that I build Debian packages and websites for myself. In the past I used to use pbuilder but it is pretty heavyweight and in my previous role I was much happier using Gitlab runners for building and deploying applications.

Gitlab is something I've no interest in self-hosting, and Jenkins is an abomination, so I figured I'd write down what kind of things I did and explore options again. My use-cases are generally very simple:

  • I trigger some before-steps
    • Which mostly mean pulling a remote git repository to the local system.
  • I then start a build.
    • This means running debuild, hugo, or similar.
    • This happens in an isolated Docker container.
  • Once the build is complete I upload the results somehere.
    • Either to a Debian package-pool, or to a remote host via rsync.

Running all these steps inside a container is well-understood, but I cheat. It is significantly easier to run the before/after steps on your host - because that way you don't need to setup SSH keys in your containers, and that is required when you clone (private) remote repositories, or wish to upload to hosts via rsync over SSH.

Anyway I wrote more thoughts here:

Then I made it work. So that was nice.

To complete the process I also implemented a simple HTTP-server which will list all available jobs, and allow you to launch one via a click. The output of the build is streamed to your client in real-time. I didn't bother persisting output and success/failure to a database, but that would be trivial enough.

It'll tide me over until I try ick again, anyway. :)

,

CryptogramCalifornia Passes New Privacy Law

The California legislature unanimously passed the strongest data privacy law in the nation. This is great news, but I have a lot of reservations. The Internet tech companies pressed to get this law passed out of self-defense. A ballot initiative was already going to be voted on in November, one with even stronger data privacy protections. The author of that initiative agreed to pull it if the legislature passed something similar, and that's why it did. This law doesn't take effect until 2020, and that gives the legislature a lot of time to amend the law before it actually protects anyone's privacy. And a conventional law is much easier to amend than a ballot initiative. Just as the California legislature gutted its net neutrality law in committee at the behest of the telcos, I expect it to do the same with this law at the behest of the Internet giants.

So: tentative hooray, I guess.

Planet DebianSteinar H. Gunderson: Solskogen 2018

I've been so busy I've forgotten to blog: Solskogen 2018 is underway! And so is the stream. http://live.solskogen.no has it all, and Saturday evening (CEST) is the big night; the home page has schedule (scroll down), and everything you missed is going up on the The YouTube channel as soon as time and bandwidth permits. :-)

,

Planet DebianSean Whitton: Debian Policy call for participation -- July 2018

Thanks to efforts from several contributors I was able to push a substantive release of Policy a little over a week ago. It contains a lot of changes that I was very pleased to release:

Changes:
 debian-policy (4.1.5.0) unstable; urgency=medium
 .
   * Policy: Update section 4.1, "Standards conformance"
     Wording: Ian Jackson
     Seconded: Sean Whitton
     Seconded: Holger Levsen
     Closes: #901160
   * Policy: Require d-devel consultation for each epoch bump
     Wording: Ian Jackson
     Seconded: Sean Whitton
     Seconded: Paul Gevers
     Closes: #891216
   * Policy: Document Rules-Requires-Root
     Wording: Niels Thykier
     Wording: Guillem Jover
     Wording: Sean Whitton
     Seconded: Paul Gevers
     Closes: #880920
   * Policy: Update version of POSIX standard for shell scripts
     Wording: Sean Whitton
     Seconded: Simon McVittie
     Seconded: Julien Cristau
     Seconded: Gunnar Wolf
     Closes: #864615
   * Policy: Update version of FHS from 2.3 to 3.0
     Wording: Simon McVittie
     Seconded: Sean Whitton
     Seconded: Julien Cristau
     Closes: #787816
   * Add reference link to section 4.1 to section 5.6.11.

We’re now looking forward to the rolling Debian Policy spring at DebCamp18. You do not need to be physically present at DebCamp18 to participate. My goal is firstly to enable others to work on bugs by providing advice about the process and how to move things along, and secondarily to get some patches written for uncontroversial bugs.

See the linked wiki page for more information on the sprint.

CryptogramGas Pump Hack

This is weird:

Police in Detroit are looking for two suspects who allegedly managed to hack a gas pump and steal over 600 gallons of gasoline, valued at about $1,800. The theft took place in the middle of the day and went on for about 90 minutes, with the gas station attendant unable to thwart the hackers.

The theft, reported by Fox 2 Detroit, took place at around 1pm local time on June 23 at a Marathon gas station located about 15 minutes from downtown Detroit. At least 10 cars are believed to have benefitted from the free-flowing gas pump, which still has police befuddled.

Here's what is known about the supposed hack: Per Fox 2 Detroit, the thieves used some sort of remote device that allowed them to hijack the pump and take control away from the gas station employee. Police confirmed to the local publication that the device prevented the clerk from using the gas station's system to shut off the individual pump.

Slashdot post.

Hard to know what's true, but it seems like a good example of a hack against a cyber-physical system.

Planet DebianAndrej Shadura: Upcoming git-crecord release

More than 1½ years since the first release of git-crecord, I’m preparing a big update. Not aware how exactly many people are using it, I neglected the maintenance for some time, but last month I’ve decided I need to take action and fix some issues I’ve known since the first release.

First of all, I’ve fixed a few minor issues with setup.py-based installer some users reported.

Second, I’ve ported a batch of updates from a another crecord derivative merged into Mercurial. That also brought some updates to the bits of Mercurial code git-crecord is using.

Third, long waited Python 3 support is here. I’m afraid at the moment I cannot guarantee support of patches in encodings other than the locale’s one, but if that turns out to be a needed feature, I can think about implementing it.

Fourth, missing staging and unstaging functionality is being implemented, subject to the availability of free time during the holiday :)

The project is currently hosted at GitHub: https://github.com/andrewshadura/git-crecord.

P.S. In case you’d like to support me hacking on git-crecord, or any other of my free software activities, you can tip my Patreon account.

Worse Than FailureError'd: All the Way from Sweden

"And to think, this price doesn't include assembly," wrote Adam G.

 

Martin B. writes, "Don't worry...I'm sure you'll find your group eventually."

 

"As always I appreciate Google's relevant search results for dealing with specific issues in the office," Michael T. wrote.

 

Bruce W. writes, "How exactly were podcasts distributed in 1938? 45 RPM Record-of-the-Month club? Also, anyone have have some of those 2038 podcasts?"

 

"I realize the page says it's 'free', but I don't really think you should be selling that," writes Rob W.

 

Peter L. writes, "So, does the 's' stand for 'segfault'?"

 

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

Planet DebianSteve Kemp: Odroid-go initial impressions

Recently I came across a hacker news post about the Odroid-go, which is a tiny hand-held console, somewhat resembling a gameboy.

In terms of hardware this is powered by an ESP32 chip, which is something I'm familiar with due to my work on Arduino, ESP8266, and other simple hardware projects.

Anyway the Odroid device costs about $40, can be programmed via the Arduino-studio, and comes by default with a series of emulators for "retro" hardware:

  • Game Boy (GB)
  • Game Boy Color (GBC)
  • Game Gear (GG)
  • Nintendo Entertainment System (NES)
  • Sega Master System (SMS)

I figured it was cheap, and because of its programmable nature it might be fun to experiment with. Though in all honesty my intentions were mostly to play NES games upon it. The fact that it is programmable means I can pretend I'm doing more interesting things than I probably would be!

Most of the documentation is available on this wiki:

The arduino setup describes how to install the libraries required to support the hardware, but then makes no mention of the board-support required to connect to an ESP32 device.

So to get a working setup you need to do two things:

  • Install the libraries & sample-code.
  • Install the board-support.

The first is documented, but here it is again:

 git clone https://github.com/hardkernel/ODROID-GO.git \
     ~/Arduino/libraries/ODROID-GO

The second is not documented. In short you need to download the esp32 hardware support via this incantation:

    mkdir -p ~/Arduino/hardware/espressif && \
     cd ~/Arduino/hardware/espressif && \
     git clone https://github.com/espressif/arduino-esp32.git esp32 && \
     cd esp32 && \
     git submodule update --init --recursive && \
     cd tools && \
     python2 get.py

(This assumes that you're using ~/Arduino for your code, but since almost everybody does ..)

Anyway the upshot of this should be that you can:

  • Choose "Tools | Boards | ODROID ESP32" to select the hardware to build for.
  • Click "File | Examples |ODROID-Go | ...." to load a sample project.
    • This can now be compiled and uploaded, but read on for the flashing-caveat.

Another gap in the instructions is that uploading projects fails. Even when you choose the correct port (Tools | Ports | ttyUSB0). To correct this you need to put the device into flash-mode:

  • Turn it off.
  • Hold down "Volume"
  • Turn it on.
  • Click Upload in your Arduino IDE.

The end result is you'll have your own code running on the device, as per this video:

Enough said. Once you do this when you turn the device on you'll see the text scrolling around. So we've overwritten the flash with our program. Oops. No more emulation for us!

The process of getting the emulators running, or updated, is in two-parts:

  • First of all the system firmware must be updated.
  • Secondly you need to update the actual emulators.

Confusingly both of these updates are called "firmware" in various (mixed) documentation and references. The main bootloader which updates the system at boot-time is downloaded from here:

To be honest I expect you only need to do this part once, unless you're uploading your own code to it. The firmware pretty much just boots, and if it finds a magic file on the SD-card it'll copy it into flash. Then it'll either execute this new code, or execute the old/pre-existing code if no update was found.

Anyway get the tarball, extract it, and edit the two files if your serial device is not /dev/ttyUSB0:

  • eraseflash.sh
  • flashall.sh

One you've done that run them both, in order:

$ ./eraseflash.sh
$ ./flashall.sh

NOTE: Here again I had to turn the device off, hold down the volume button, turn it on, and only then execute ./eraseflash.sh. This puts the device into flashing-mode.

NOTE: Here again I had to turn the device off, hold down the volume button, turn it on, and only then execute ./flashall.sh. This puts the device into flashing-mode.

Anyway if that worked you'll see your blue LED flashing on the front of the device. It'll flash quickly to say "Hey there is no file on the SD-card". So we'll carry out the second step - Download and place firmware.bin into the root of your SD-card. This comes from here:

(firmware.bin contains the actual emulators.)

Insert the card. The contents of firmware.bin will be sucked into flash, and you're ready to go. Turn off your toy, remove the SD-card, load it up with games, and reinsert it.

Power on your toy and enjoy your games. Again a simple demo:

Games are laid out in a simple structure:

  ├── firmware.bin
  ├── odroid
  │   ├── data
  │   └── firmware
  └── roms
      ├── gb
      │   └── Tetris (World).gb
      ├── gbc
      ├── gg
      ├── nes
      │   ├── Lifeforce (U).nes
      │   ├── SMB-3.nes
      │   └── Super Bomberman 2 (E) [!].nes
      └── sms

You can find a template here:

Planet Debianbisco: Fifth GSoC Report

No shiny screenshots this time ;-)

In the last two weeks i’ve finished evaluating the SSO solutions. I’ve added the evaluation of ‘ipsilon’, a python based single sign on solution. I’ve also updated the existing evaluations with a bit more information about their possibility to work with multiple backends. And i’ve added gitlab configuration snippets for some of the solutions. Formorer asked me to create a tabular overview of the outcome of the evaluations, so i did that in the README of the corresponding salsa repository. I’ve also pushed the code for the test client application i used to test the SSO solutions.

This week i used to look into the existing Debian SSO solution that works with certificates. The idea is not to change it, but to integrate it with the chosen OAuth2/SAML solution. To test this, i’ve pulled the code and set up my own instance of it. Fortunatly it is a Django application, so i now have some experience with that. Its not working yet, but i’m getting there.

I’ve also reevaluated a design desicion i made with nacho and came to the same conclusion: that storing the temporary accounts in ldap too is the way to go. There are also still some small feature requests i want to implement.

Planet DebianLouis-Philippe Véronneau: Taiwan Travel Blog - Day 5

The view from my hostel this morning

This is the fourth entry of my Taiwan Travel blog series! You can find posts on my first few days in Taiwan by following these links:

Old Zhuilu Road (锥麓古道)

Between all the trails I did in the Taroko National Park, I think this one was the best.

Old Zhuilu Road has quite a story behind it. It was built in the 1910s under the Japanese occupation of the Taiwan island. Most of the trail already existed, but the Japanese forced the natives to make it at least a meter large everywhere (it used to be only 10cm wide) using dynamite.

It's quite sad that such a beautiful trail is soaked the in blood of the Taroko natives.

The path at the top of Old Zhuilu Road

The Japanese wanted to widen the trail to be able to bring cannons and weapons up the mountain to subdue the native villages resisting the occupation. They eventually won and killed a bunch of them.

Even though this trail is less taxing than the Dali-Datong trail I did two days ago, it easily scores a hard 3/5 as half of the path is made of very narrow paths on the side of a 700m cliff. Better not trip and fall!

Sadly out of the original 10km, only a third of the trail is currently open. A large typhoon destroyed part of the path a few years ago. Once you arrive at the cliffside outpost you have to turn back and start climbing down.

Night market in Xincheng (新成乡)

Tonight was the night market in Xincheng township, the rural township where my hostel is located. I was pretty surprised by how large it was (more than 50 stalls) considering that only 20'000 people reside in the whole township. The ambiance was great and of course I ended up eating too much.

Looking down into the abyss

Fried and grilled chicken hearts, fried bits of fat, fried fish balls, grilled squid, hollow egg-shaped cakes, corndogs, sausages, soft-serve ice cream, sweet black tea, I guess I should have paced myself.

Although the food stalls were numerous, there were also a large number of clothes vendors, random toys stalls for the kids and a bunch of electronic pachinko machines where old people were placing bets on one another. The kids were pretty funny: since I seemed to be the only westerner in the whole market, they kept coming in groups of 3 or 4 to laugh at me for not speaking mandarin, and then left running and squealing when I started talking to them.

Blurp! More chicken hearts please!

,

CryptogramFriday Squid Blogging: Antifungal Squid-Egg Coating

The Hawaiian bobtail squid coats its eggs with antifungal bacteria.

As usual, you can also use this squid post to talk about the security stories in the news that I haven't covered.

Read my blog posting guidelines here.

Sociological ImagesThe Rise of Adblock Shaming

If you walked through a city without looking up at any billboards, could advertisers yell at you? Could the owner of an iconic building shame you for “stealing” a beautiful view while weaseling them out of their livelihood? It sounds absurd, and you might remember a viral quote from Banksy (riffing on original writing from Sean Tejaratchi), tearing the idea apart.

But what about digital advertising? The internet looks very different if you are using software to block advertisements. Use it for a long time you’ll forget how much junk a user has to slog through to read or watch anything.

Of course, blocking ads cuts into the main source of support for online publications. Lately, many have taken up a new approach to discourage their users from blocking ads: good old fashioned shame and guilt.

We can have an important conversation about the ethics of paying for content online, but what strikes me the most about these pop-ups are some core sociological questions about the shaming tactic: why here, and why now?

For a long time, social scientists have seen a “digital divide” in how unequal access to the internet reinforces social inequality. Research also shows that the digital divide isn’t just about access; people learn to use the internet in different ways from these early access experiences. From the design side, sociologists Jenny Davis and James Chouinard have also written about affordance theory: the way technology requests, demands, allows, encourages, discourages, and refuses different kinds of behavior from users.

Yes, you can see the important weather alert, but first…

For some, the internet is about abundance and agency. Take as much time as you need to figure out your problems, and, if things don’t work out, bend the world to your will! Grab open source software or write a script to automate the boring stuff! Open your app of choice to hail a ride if the bus is delayed or the taxis are busy! For others, these choices aren’t as readily apparent. If you had to trek to the library and sign up for time-limited computer access, the internet can seem a lot less helpful and a lot less free, at least at first glance.

These ideas help us understand the biggest problem for ad-block shaming: “soft” barriers, delays, and emotional appeals are trying to change the behavior of people who already have the upper hand from learning to seek out and use blocking software to make the internet work better for them. David Banks’ writing on this over at Cyborgology in 2015 shows the power struggle at work:

The ad blocker should not be seen as a selfish technology. It is a socialist cudgel—something that forces otherwise lazy capitalists to find new and inventive ways to make their creations sustainable. Ad blockers are one of the few tools users have to fight against the need to monetize fast and big because it troubles the predictability of readily traceable attention.

Now, emotional appeals like guilt and shame are the next step after stronger power plays like rigid paywalls largely failed for publishing companies. The challenge is that guilt and shame require a larger sense of community obligation for people to feel their effects, and I am not sure a pop-up is ever going to be anything other than an obstacle to get around.

It’s not that online advertising is inherently good or bad, and the problem of paying artists and writers in the digital age is a serious concern. But in addition to these considerations, looking directly at the way web design tries to shape our online interactions can better prepare us to see how the rules of the social world can be challenged and changed.

Evan Stewart is a Ph.D. candidate in sociology at the University of Minnesota. You can follow him on Twitter.

(View original at https://thesocietypages.org/socimages)

Planet DebianPetter Reinholdtsen: Simple streaming the Linux desktop to Kodi using GStreamer and RTP

Last night, I wrote a recipe to stream a Linux desktop using VLC to a instance of Kodi. During the day I received valuable feedback, and thanks to the suggestions I have been able to rewrite the recipe into a much simpler approach requiring no setup at all. It is a single script that take care of it all.

This new script uses GStreamer instead of VLC to capture the desktop and stream it to Kodi. This fixed the video quality issue I saw initially. It further removes the need to add a m3u file on the Kodi machine, as it instead connects to the JSON-RPC API in Kodi and simply ask Kodi to play from the stream created using GStreamer. Streaming the desktop to Kodi now become trivial. Copy the script below, run it with the DNS name or IP address of the kodi server to stream to as the only argument, and watch your screen show up on the Kodi screen. Note, it depend on multicast on the local network, so if you need to stream outside the local network, the script must be modified. Also note, I have no idea if audio work, as I only care about the picture part.

#!/bin/sh
#
# Stream the Linux desktop view to Kodi.  See
# http://people.skolelinux.org/pere/blog/Streaming_the_Linux_desktop_to_Kodi_using_VLC_and_RTSP.html
# for backgorund information.

# Make sure the stream is stopped in Kodi and the gstreamer process is
# killed if something go wrong (for example if curl is unable to find the
# kodi server).  Do the same when interrupting this script.
kodicmd() {
    host="$1"
    cmd="$2"
    params="$3"
    curl --silent --header 'Content-Type: application/json' \
	 --data-binary "{ \"id\": 1, \"jsonrpc\": \"2.0\", \"method\": \"$cmd\", \"params\": $params }" \
	 "http://$host/jsonrpc"
}
cleanup() {
    if [ -n "$kodihost" ] ; then
	# Stop the playing when we end
	playerid=$(kodicmd "$kodihost" Player.GetActivePlayers "{}" |
			    jq .result[].playerid)
	kodicmd "$kodihost" Player.Stop "{ \"playerid\" : $playerid }" > /dev/null
    fi
    if [ "$gstpid" ] && kill -0 "$gstpid" >/dev/null 2>&1; then
	kill "$gstpid"
    fi
}
trap cleanup EXIT INT

if [ -n "$1" ]; then
    kodihost=$1
    shift
else
    kodihost=kodi.local
fi

mcast=239.255.0.1
mcastport=1234
mcastttl=1

pasrc=$(pactl list | grep -A2 'Source #' | grep 'Name: .*\.monitor$' | \
  cut -d" " -f2|head -1)
gst-launch-1.0 ximagesrc use-damage=0 ! video/x-raw,framerate=30/1 ! \
  videoconvert ! queue2 ! \
  x264enc bitrate=8000 speed-preset=superfast tune=zerolatency qp-min=30 \
  key-int-max=15 bframes=2 ! video/x-h264,profile=high ! queue2 ! \
  mpegtsmux alignment=7 name=mux ! rndbuffersize max=1316 min=1316 ! \
  udpsink host=$mcast port=$mcastport ttl-mc=$mcastttl auto-multicast=1 sync=0 \
  pulsesrc device=$pasrc ! audioconvert ! queue2 ! avenc_aac ! queue2 ! mux. \
  > /dev/null 2>&1 &
gstpid=$!

# Give stream a second to get going
sleep 1

# Ask kodi to start streaming using its JSON-RPC API
kodicmd "$kodihost" Player.Open \
	"{\"item\": { \"file\": \"udp://@$mcast:$mcastport\" } }" > /dev/null

# wait for gst to end
wait "$gstpid"

I hope you find the approach useful. I know I do.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Krebs on SecuritySextortion Scam Uses Recipient’s Hacked Passwords

Here's a clever new twist on an old email scam that could serve to make the con far more believable. The message purports to have been sent from a hacker who's compromised your computer and used your webcam to record a video of you while you were watching porn. The missive threatens to release the video to all your contacts unless you pay a Bitcoin ransom. The new twist? The email now references a real password previously tied to the recipient's email address.

Planet DebianJonathan Dowland: The Cure's 40th Anniversary

Last Saturday I joined roughly 65,000 other people to see the Cure play a 40th Anniversary celebration gig in Hyde Park, London. It was a short gig (by Cure) standards of about 2½ hours due to the venue's strict curfew, and as predicted, the set was (for the most part) a straightforward run through the greatest hits. However, the atmosphere was fantastic. It may have been helped along by the great weather we were enjoying (over 30°C), England winning a World Cup match a few hours earlier, and the infectious joy of London Pride that took place a short trip up the road. A great time was had by all.

Last year, a friend of mine who had never listened to the Cure had asked me to recommend (only) 5 songs which would give a reasonable overview. (5 from over 200 studio recorded songs). As with Coil, this is quite a challenging task, and here's what I came up with. In most cases, the videos are from the Hyde Park show (but it's worth seeking out the studio versions too)

1. "Pictures of You"

Walking a delicate line between their dark and light songs, "Pictures of You" is one of those rare songs where the extended remix is possibly better than the original (which is not short either)

2. "If Only Tonight We Could Sleep"

I love this song. I'm a complete sucker for the Phrygian scale. I was extremely happy to finally catch it live for the first time at Hyde Park, which was my fourth Cure gig (and hopefully not my last)

The nu-metal band "Deftones" have occasionally covered this song live, and they do a fantastic job of it. They played it this year for their Meltdown appearance, and a version appears on their "B-Side and Rarities". My favourite take was from a 2004 appearance on MTV's "MTV Icon" programme honouring the Cure:

3. "Killing An Arab"

The provocatively-titled first single by the group takes its name from the pivotal scene in the Albert Camus novel "The Stranger" and is not actually endorsing the murder of people. Despite this it's an unfortunate title, and in recent years they have often performed it as "Killing Another". The song loses nothing in renaming, in my opinion.

The original recording is a sparse, tight, angular post-punk piece, but it's in the live setting that this song really shines, and it's a live version I recommend you try.

4. "Just Like Heaven"

It might be obvious that my tastes align more to the Cure's dark side than the light, but the light side can't be ignored. Most of their greatest hits and best known work are light, accessible pop classics. Choosing just one was amongst the hardest decisions to make. For the selection I offered my friend, I opted for "Friday I'm In Love", which is unabashed joy, but it didn't meet a warm reception, so I now substitute it for "Just Like Heaven".

Bonus video: someone proposed in the middle of this song!

5. "The Drowning Man"

From their "Very Dark" period, another literature-influenced track, this time Mervyn Peake's "Gormenghast": "The Drowning Man"

If you let the video run on, you'll get a bonus 6th track, similarly rarely performed live: Faith. I haven't seen either live yet. Maybe one day!

CryptogramWPA3

Everyone is writing about the new WPA3 Wi-Fi security standard, and how it improves security over the current WPA2 standard.

This summary is as good as any other:

The first big new feature in WPA3 is protection against offline, password-guessing attacks. This is where an attacker captures data from your Wi-Fi stream, brings it back to a private computer, and guesses passwords over and over again until they find a match. With WPA3, attackers are only supposed to be able to make a single guess against that offline data before it becomes useless; they'll instead have to interact with the live Wi-Fi device every time they want to make a guess. (And that's harder since they need to be physically present, and devices can be set up to protect against repeat guesses.)

WPA3's other major addition, as highlighted by the Alliance, is forward secrecy. This is a privacy feature that prevents older data from being compromised by a later attack. So if an attacker captures an encrypted Wi-Fi transmission, then cracks the password, they still won't be able to read the older data -- they'd only be able to see new information currently flowing over the network.

Note that we're just getting the new standard this week. Actual devices that implement the standard are still months away.

Worse Than FailureCodeSOD: A Symbol of Bad Code

As developers, when we send data over the network, we can usually safely ignore the physical implementation of that network. At some level, though, the bits you’re sending become physical effects in your transmission medium, whether it’s radio waves or electrical signals.

You can’t just send raw bits over the wire. Those bits have to be converted into a symbol suitable for the transmission medium. Symbols could be the dots-and-dashes of morse code, tones transmitted over a phone line, or changing duty cycles on a pulse-width-modulated signal. The number of symbols per second is the baud rate of the channel. What this means for digital transmission is that even if your channel has a potential bit rate of one gigabit per second, the actual baud rate may be different- either much larger or much smaller. For example, modems might send 4-bits per symbol, meaning a 2,400 baud modem actually can transmit 9,600 bits per second. GPS, on the other hand, can transmit 50 bits/s, but over one million symbols per second thanks to spread spectrum broadcast.

Marnie works for a telcoms company which greatly cares about these sorts of details. There’s a variety of client-side web apps which accrued over time which can help with things like symbol rate calculations.

For their environment, this calculation is straightforward: whatever the bits-per-second of the channel is, divide by 1.25 to find the symbol rate. Of course, this simple calculation can’t be performed without regexes. Marnie found this code in the code base:

private bandwidthList = [6.4, 3.2, 1.6];
private symbolRateList = [5.12, 2.56, 1.28]; 

public getSymbolRate(_bandwidth: number): string {
	let BW1DecimalPoint = _bandwidth.toString().match(/^-?\d+(?:\.\d{0,1})?/)[0];
	let symbolRate = this.symbolRateList[0];
	let symbolIndex = this.bandwidthList.indexOf(parseInt(BW1DecimalPoint));

	if (symbolIndex > 0) {
		symbolRate = this.symbolRateList[symbolIndex];
	}

	return formatValue(symbolRate);
}

Now, this is TypeScript, so there is no guarantee that, at runtime, _bandwidth will actually be a number. But there’s no need to convert it to a string, match against a regex, slice up the results, only to parse it back into a an int later. Which, it’s important to note, because they use parseInt inside of the indexOf call, this will never find the target entry inside of bandwidthList- 6 is not 6.4.

So, as written, this method only ever returns 5.12. It’s been in use, in production, for a few years now. Not only is in overly complex, it also just doesn’t work.

Marnie fixed the code with a much simpler version:

public getSymbolRate(_bandwidth: number): string {
	let symbolRate = _bandwidth / 1.25;
	return formatValue(symbolRate);
}
[Advertisement] Forget logs. Next time you're struggling to replicate error, crash and performance issues in your apps - Think Raygun! Installs in minutes. Learn more.

Don MartiWeb ad bargain?

Tim Peterson, on Digiday:

If an exchange or SSP declines to sign the agreement, it is limited to only selling non-personalized ads through DBM. Those generic ads generate less revenue for publishers than personalized ads that are targeted to specific audiences based on data collected about them. Some publishers that are heavily reliant on DBM have seen their revenues decline by 70-80 percent since GDPR took effect because they were limited to non-personalized ads, said another ad tech exec. That revenue drop has put pressure on exchanges and SSPs to sign Google’s consent agreement lest their publishers move their inventory to other platforms that can run DBM’s personalized ads on their sites, the second exec said.

(‘It’s impossible’: Google has asked ad tech firms to guarantee broad GDPR consent, assume liability - Digiday)

A lot of those "specific audiences" are, of course, adfraud bots. Fraud hackers are better at adtech than adtech firms are. So ads shown to bots, on shitty sites, are going for more than ads seen by humans on legit sites.

Meanwhile, tracking-resistant, personalization-averse readers are overrepresented in some customer categories. Web developers are a good example. (40% protected based on recent data from one popular site.)

Of course, today's web ad system is based on tracking the best possible prospect to the cheapest possible site, so it won't be easy to take advantage of this nice piece of market inefficiency. First step is figuring out how well protected the people you want to reach are.

More: blog.aloodo.org - Beware of averages: why you need a local tracking protection metric

Rondam RamblingsRoe is a distraction. The real problem is much, much worse.

The United States of America has always had a somewhat tenuous relationship with its own ideals.  The disconnect between "We hold these truths to be self-evident, that all men are created equal" on the one hand, and chattel slavery on the other, cannot result in anything other than some pretty severe cognitive dissonance.  But despite being deeply rooted in contradictions, the history of this

Planet DebianLouis-Philippe Véronneau: Taiwan Travel Blog - Day 4

This is the third entry of my Taiwan Travel blog series! You can find posts on my first few days in Taiwan by following these links:

The Baiyang Waterfall

I had to take care of a few things this morning so I left the hostel a little bit later than I would have liked. I've already done quite a few trails and I'm slowly starting to exhaust the places I wanted to visit in the Taroko National Park, or at least the ones I can reach via public bus.

I've got another day left here and I plan to use the mountain permit I got to visit Old Zhuilu Road (锥麓古道) and then I'll cycle back to Hualien.

Baiyang Waterfall Trail (白杨步道)

I took the bus to go to Tianxiang (天祥), a mountain station about 20km inland from the Taroko National Park entrance. I considered riding my bike there but the mountain road has a lot of blind turns and buses frequently pass one another.

I wanted to visit the Baiyang Waterfall and the Huoran temple, but only found the trail for the waterfall. I blame the heavy construction work in the area.

Formosan Rock Macaque, Yang Hsiao Chi, CC-BY-SA-4.0

Baiyang Waterfall was a very nice and leisurely trail. Paved all the way, it follows the Tacijili River (塔次基里) for a while. The trail starts by a tunnel 250m long carved into the mountain. There are no lights in there so it's pitch dark for a while!

I was very lucky to see a colony of bats sleeping with their pups at the exit of the entrance tunnel. So cute!

I also encountered a family of Formosan Rock Macaques in the forest next to the trail. Apparently, they are pretty common in Taiwan, but it was the first time I saw them. Andrew Lee tells me that if I want to meet more monkeys, I can go to the nearby hot spring and bathe with them. If it wasn't so darn hot I just might go.

Planet DebianPetter Reinholdtsen: Streaming the Linux desktop to Kodi using VLC and RTSP

PS: See the followup post for a even better approach.

A while back, I was asked by a friend how to stream the desktop to my projector connected to Kodi. I sadly had to admit that I had no idea, as it was a task I never had tried. Since then, I have been looking for a way to do so, preferable without much extra software to install on either side. Today I found a way that seem to kind of work. Not great, but it is a start.

I had a look at several approaches, for example using uPnP DLNA as described in 2011, but it required a uPnP server, fuse and local storage enough to store the stream locally. This is not going to work well for me, lacking enough free space, and it would impossible for my friend to get working.

Next, it occurred to me that perhaps I could use VLC to create a video stream that Kodi could play. Preferably using broadcast/multicast, to avoid having to change any setup on the Kodi side when starting such stream. Unfortunately, the only recipe I could find using multicast used the rtp protocol, and this protocol seem to not be supported by Kodi.

On the other hand, the rtsp protocol is working! Unfortunately I have to specify the IP address of the streaming machine in both the sending command and the file on the Kodi server. But it is showing my desktop, and thus allow us to have a shared look on the big screen at the programs I work on.

I did not spend much time investigating codeces. I combined the rtp and rtsp recipes from the VLC Streaming HowTo/Command Line Examples, and was able to get this working on the desktop/streaming end.

vlc screen:// --sout \
  '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{dst=projector.local,port=1234,sdp=rtsp://192.168.11.4:8080/test.sdp}'

I ssh-ed into my Kodi box and created a file like this with the same IP address:

echo rtsp://192.168.11.4:8080/test.sdp \
  > /storage/videos/screenstream.m3u

Note the 192.168.11.4 IP address is my desktops IP address. As far as I can tell the IP must be hardcoded for this to work. In other words, if someone elses machine is going to do the steaming, you have to update screenstream.m3u on the Kodi machine and adjust the vlc recipe. To get started, locate the file in Kodi and select the m3u file while the VLC stream is running. The desktop then show up in my big screen. :)

When using the same technique to stream a video file with audio, the audio quality is really bad. No idea if the problem is package loss or bad parameters for the transcode. I do not know VLC nor Kodi enough to tell.

Update 2018-07-12: Johannes Schauer send me a few succestions and reminded me about an important step. The "screen:" input source is only available once the vlc-plugin-access-extra package is installed on Debian. Without it, you will see this error message: "VLC is unable to open the MRL 'screen://'. Check the log for details." He further found that it is possible to drop some parts of the VLC command line to reduce the amount of hardcoded information. It is also useful to consider using cvlc to avoid having the VLC window in the desktop view. In sum, this give us this command line on the source end

cvlc screen:// --sout \
  '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{sdp=rtsp://:8080/}'

and this on the Kodi end

echo rtsp://192.168.11.4:8080/ \
  > /storage/videos/screenstream.m3u

Still bad image quality, though. But I did discover that streaming a DVD using dvdsimple:///dev/dvd as the source had excellent video and audio quality, so I guess the issue is in the input or transcoding parts, not the rtsp part. I've tried to change the vb and ab parameters to use more bandwidth, but it did not make a difference.

I further received a suggestion from Einar Haraldseid to try using gstreamer instead of VLC, and this proved to work great! He also provided me with the trick to get Kodi to use a multicast stream as its source. By using this monstrous oneliner, I can stream my desktop with good video quality in reasonable framerate to the 239.255.0.1 multicast address on port 1234:

gst-launch-1.0 ximagesrc use-damage=0 ! video/x-raw,framerate=30/1 ! \
  videoconvert ! queue2 ! \
  x264enc bitrate=8000 speed-preset=superfast tune=zerolatency qp-min=30 \
  key-int-max=15 bframes=2 ! video/x-h264,profile=high ! queue2 ! \
  mpegtsmux alignment=7 name=mux ! rndbuffersize max=1316 min=1316 ! \
  udpsink host=239.255.0.1 port=1234 ttl-mc=1 auto-multicast=1 sync=0 \
  pulsesrc device=$(pactl list | grep -A2 'Source #' | \
    grep 'Name: .*\.monitor$' |  cut -d" " -f2|head -1) ! \
  audioconvert ! queue2 ! avenc_aac ! queue2 ! mux.

and this on the Kodi end

echo udp://@239.255.0.1:1234 \
  > /storage/videos/screenstream.m3u

Note the trick to pick a valid pulseaudio source. It might not pick the one you need. This approach will of course lead to trouble if more than one source uses the same multicast port and address. Note the ttl-mc=1 setting, which limit the multicast packages to the local network. If the value is increased, your screen will be broadcasted further, one network "hop" for each increase (read up on multicast to learn more. :)!

Having cracked how to get Kodi to receive multicast streams, I could use this VLC command to stream to the same multicast address. The image quality is way better than the rtsp approach, but gstreamer seem to be doing a better job.

cvlc screen:// --sout '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{mux=ts,dst=239.255.0.1,port=1234,sdp=sap}'

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

,

Krebs on SecurityNotorious ‘Hijack Factory’ Shunned from Web

Score one for the good guys: Bitcanal, a Portuguese Web hosting firm long accused of helping spammers hijack large swaths of dormant Internet address space over the years, was summarily kicked off the Internet this week after a half-dozen of the company's bandwidth providers chose to sever ties with the company.

CryptogramDepartment of Commerce Report on the Botnet Threat

Last month, the US Department of Commerce released a report on the threat of botnets and what to do about it. I note that it explicitly said that the IoT makes the threat worse, and that the solutions are largely economic.

The Departments determined that the opportunities and challenges in working toward dramatically reducing threats from automated, distributed attacks can be summarized in six principal themes.

  1. Automated, distributed attacks are a global problem. The majority of the compromised devices in recent noteworthy botnets have been geographically located outside the United States. To increase the resilience of the Internet and communications ecosystem against these threats, many of which originate outside the United States, we must continue to work closely with international partners.

  2. Effective tools exist, but are not widely used. While there remains room for improvement, the tools, processes, and practices required to significantly enhance the resilience of the Internet and communications ecosystem are widely available, and are routinely applied in selected market sectors. However, they are not part of common practices for product development and deployment in many other sectors for a variety of reasons, including (but not limited to) lack of awareness, cost avoidance, insufficient technical expertise, and lack of market incentives

  3. Products should be secured during all stages of the lifecycle. Devices that are vulnerable at time of deployment, lack facilities to patch vulnerabilities after discovery, or remain in service after vendor support ends make assembling automated, distributed threats far too easy.

  4. Awareness and education are needed. Home users and some enterprise customers are often unaware of the role their devices could play in a botnet attack and may not fully understand the merits of available technical controls. Product developers, manufacturers, and infrastructure operators often lack the knowledge and skills necessary to deploy tools, processes, and practices that would make the ecosystem more resilient.

  5. Market incentives should be more effectively aligned. Market incentives do not currently appear to align with the goal of "dramatically reducing threats perpetrated by automated and distributed attacks." Product developers, manufacturers, and vendors are motivated to minimize cost and time to market, rather than to build in security or offer efficient security updates. Market incentives must be realigned to promote a better balance between security and convenience when developing products.

  6. Automated, distributed attacks are an ecosystem-wide challenge. No single stakeholder community can address the problem in isolation.

[...]

The Departments identified five complementary and mutually supportive goals that, if realized, would dramatically reduce the threat of automated, distributed attacks and improve the resilience and redundancy of the ecosystem. A list of suggested actions for key stakeholders reinforces each goal. The goals are:

  • Goal 1: Identify a clear pathway toward an adaptable, sustainable, and secure technology marketplace.
  • Goal 2: Promote innovation in the infrastructure for dynamic adaptation to evolving threats.
  • Goal 3: Promote innovation at the edge of the network to prevent, detect, and mitigate automated, distributed attacks.
  • Goal 4: Promote and support coalitions between the security, infrastructure, and operational technology communities domestically and around the world
  • Goal 5: Increase awareness and education across the ecosystem.

Worse Than FailureReproducible Heisenbug

Illustration of Heisenberg Uncertainty Principle

Matt had just wrapped up work on a demo program for an IDE his company had been selling for the past few years. It was something many customers had requested, believing the documentation wasn't illustrative enough. Matt's program would exhibit the IDE's capabilities and also provide sample code to help others get started on their own creations.

It was now time for the testers to do their thing with the demo app. Following the QA team's instructions, Matt changed the Debug parameter in the configuration file from 4 (full debugging) to 1 (no debugging). Build and deploy completed without a hitch. Matt sent off the WAR file, feeling good about his programming aptitude and life in general.

And then his desk phone rang. The caller ID revealed it was Ibrahim, one of the testers down in QA.

Already? Matt wondered. With a mix of confusion and annoyance, he picked up the phone, assuming it was something PEBKAC-related.

"I've got no descriptors for the checkboxes on the main page," Ibrahim told him. "And the page after that has been built all skew-whiff."

"Huh?" Matt frowned. "Everything works fine on my side."

What could be different about Ibrahim's setup? The first thing Matt thought of was that he'd disabled debugging before building the WAR file for QA.

That can't be it! But it was easy enough to test.

"Hang on one sec here." Matt muted his phone, then changed the Debug parameter on his local deployment from 4 to 1. Indeed, upon refreshing, the user interface went wonky, just as Ibrahim had described. Unfortunately, with debugging off, Matt couldn't check the logs for a clue as to what was going wrong.

Back on the phone, Matt explained how he was able to do reproduce the problem, then instructed Ibrahim on manually hacking the WAR file to change the Debug parameter. Ibrahim reported that with full debugging enabled, the program worked perfectly on his end.

"OK. Lemme see what I can do," Matt said, trying not to sound as hopeless as he felt.

With absolutely no hints to guide him, Matt spent hours stepping through his code to figure out what was going wrong. At long last, he isolated a misbehaving repeat-until loop. When the Debug parameter was set to 4, the program exited the loop and returned data as expected. But when Debug was set to anything less than 4, it made an extra increment of the loop counter, leading to the graphical mayhem experienced earlier.

Horror crept down Matt's spine. This problem would affect anyone using repeat-until loops in conjunction with the IDE. Such programs were bound to fail in unexpected ways. He immediately issued a bug report, suggesting this needed to be addressed urgently.

Later that day, he received an email from one of the IDE developers. I found where it was testing the wrong boolean. Should we raise this as a defect?

"Yes! Duh!" Matt grumbled out loud, then took to typing. And can we find out where this bug crept in? All projects released since that time are compromised!!

As it turned out, the bug had been introduced to the IDE 2 years earlier. It'd been found almost immediately and fixed. Unfortunately, it'd only been fixed in one specific branch within source control—a branch that had never been merged to the trunk.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Krebs on SecurityPatch Tuesday, July 2018 Edition

Microsoft and Adobe each issued security updates for their products today. Microsoft's July patch batch includes 14 updates to fix more than 50 security flaws in Windows and associated software. Separately, Adobe has pushed out an update for its Flash Player browser plugin, as well as a monster patch bundle for Adobe Reader/Acrobat.

Planet DebianBen Hutchings: Debian LTS work, June 2018

I was assigned 15 hours of work by Freexian's Debian LTS initiative and worked 12 hours, so I have carried 3 hours over to July. Since Debian 7 "wheezy" LTS ended at the end of May, I prepared for Debian 8 "jessie" to enter LTS status.

I prepared a stable update of Linux 3.16, sent it out for review, and then released it. I rebased jessie's linux package on this, but didn't yet upload it.

Since the "jessie-backports" suite is no longer accepting updates, and there are LTS users depending on the updated kernel (Linux 4.9) there, I prepared to add it to the jessie-security suite. The source package I have prepared is similar to what was in jessie-backports, but I have renamed it to "linux-4.9" and disabled building some binary packages to avoid conflicting with the standard linux source package. I also disabled building the "udeb" packages used in the installer, since I don't expect anyone to need them and building them would require updating the "kernel-wedge" package too. I didn't upload this either, since there wasn't a new linux version in "stretch" to backport yet.

,

CryptogramRecovering Keyboard Inputs through Thermal Imaging

Researchers at the University of California, Irvine, are able to recover user passwords by way of thermal imaging. The tech is pretty straightforward, but it's interesting to think about the types of scenarios in which it might be pulled off.

Abstract: As a warm-blooded mammalian species, we humans routinely leave thermal residues on various objects with which we come in contact. This includes common input devices, such as keyboards, that are used for entering (among other things) secret information, such as passwords and PINs. Although thermal residue dissipates over time, there is always a certain time window during which thermal energy readings can be harvested from input devices to recover recently entered, and potentially sensitive, information.

To-date, there has been no systematic investigation of thermal profiles of keyboards, and thus no efforts have been made to secure them. This serves as our main motivation for constructing a means for password harvesting from keyboard thermal emanations. Specifically, we introduce Thermanator, a new post factum insider attack based on heat transfer caused by a user typing a password on a typical external keyboard. We conduct and describe a user study that collected thermal residues from 30 users entering 10 unique passwords (both weak and strong) on 4 popular commodity keyboards. Results show that entire sets of key-presses can be recovered by non-expert users as late as 30 seconds after initial password entry, while partial sets can be recovered as late as 1 minute after entry. Furthermore, we find that Hunt-and-Peck typists are particularly vulnerable. We also discuss some Thermanator mitigation strategies.

The main take-away of this work is three-fold: (1) using external keyboards to enter (already much-maligned) passwords is even less secure than previously recognized, (2) post factum (planned or impromptu) thermal imaging attacks are realistic, and finally (3) perhaps it is time to either stop using keyboards for password entry, or abandon passwords altogether.

News article.

Worse Than FailureCodeSOD: Is the Table Empty?

Sean has a lucrative career as a consultant/contractor. As such, he spends a great deal of time in other people’s code bases, and finds things like a method with this signature:

public boolean isTableEmpty()

Already, you’re in trouble. Methods which operate directly on “tables” are a code-smell, yes, even in a data-driven application. You want to operate on business objects, and unless you’re a furniture store, tables are not business objects. You might think in those terms when building some of your lower-level components, but then you’d expect to see things like string tableName in the parameter list.

Now, maybe I’m just being opinionated. Maybe there’s a perfectly valid reason to build a method like this that I can’t imagine. Well, let’s check the implementation.

public boolean isTableEmpty()
{
    boolean res = false;
    Connection conn = cpInstance.getConnection();
    try (PreparedStatement ps = conn.prepareStatement("select * from some_table")) {
        try (ResultSet rs = ps.executeQuery()) {
            if (rs.first()) {
 	        res = true;
            }
        }
    catch (SQLException e) {
        e.printStackTrace();
    } finally {
        try {
            conn.close();
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }

    return res;
}

Even if you think this method should exist, it shouldn’t exist like this. No COUNT(*) or LIMIT in the query. Using exceptions as flow control. And the best part: returning the opposite of what the method name implies. false tells us the table is empty.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Don MartiBug futures: business models

Recent question about futures markets on software bugs: what's the business model?

As far as I can tell, there are several available models, just as there are multiple kinds of companies that can participate in any securities or commodities market.

Cushing, Oklahoma

Oracle operator: Read bug tracker state, write futures contract state, profit. This business would take an agreed-upon share of any contract in exchange for acting as a referee. The market won't work without the oracle operator, which is needed in order to assign the correct resolution to each contract, but it's possible that a single market could trade contracts resolved by multiple oracles.

Actively managed fund: Invest in many bug futures in order to incentivize a high-level outcome, such as support for a particular use case, platform, or performance target.

Bot fund: An actively managed fund that trades automatically, using open source metrics and other metadata.

Analytics provider: Report to clients on the quality of software projects, and the market-predicted likelihood that the projects will meet the client's maintenance and improvement requirements in the future.

Stake provider: A developer participant in a bug futures market must invest to acquire a position on the fixed side of a contract. The stake provider enables low-budget developers to profit from larger contracts, by lending or by investing alongside them.

Arbitrageur: Helps to re-focus development efforts by buying the fixed side of one contract and the unfixed side of another. For example, an arbitrageur might buy the fixed side of several user-facing contracts and the unfixed side of the contract on a deeper issue whose resolution will result in a fix for them.

Arbitrageurs could also connect bug futures to other kinds of markets, such as subscriptions, token systems, or bug bounties.

Previous items in the bug futures series:

Bugmark paper

A trading market to incentivize secure software: Malvika Rao, Georg Link, Don Marti, Andy Leak & Rich Bodo (PDF) (presented at WEIS 2018)

Corporate Prediction Markets: Evidence from Google, Ford, and Firm X (PDF) by Bo Cowgill and Eric Zitzewitz.

Despite theoretically adverse conditions, we find these markets are relatively efficient, and improve upon the forecasts of experts at all three firms by as much as a 25% reduction in mean squared error.

(This paper covers a related market type, not bug futures. However some of the material about interactions of market data and corporate management could also turn out to be relevant to bug futures markets.)

Creative Commons

Pipeline monument in Cushing, Oklahoma: photo by Roy Luck for Wikimedia Commons. This file is licensed under the Creative Commons Attribution 2.0 Generic license.

Planet DebianReproducible builds folks: Reproducible Builds: Weekly report #167

Here’s what happened in the Reproducible Builds effort between Sunday July 1 and Saturday July 7 2018:

Packages reviewed and fixed, and bugs filed

diffoscope development

Misc.

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Planet DebianMinkush Jain: KubeCon + CloudNativeCon, Copenhagen

I attended KubeCon + CloudNativeCon 2018, Europe that took place from 2nd to 4th of May. It was held in Copenhagen, Denmark. I know it’s quite late since I attended it, but still I wanted to share my motivating experiences at the conference, so here it is!

I got scholarship from the Linux Foundation which gave me a wonderful opportunity to attend this conference. This was my first developer conference aboard and I was super-excited to attend it. I got the chance to learn more about containers, straight from the best people out there.

I attended the opening keynote sessions on 2nd May in Bella Centre, Copenhagen. The opening keynote was given by Dan Kohn, Executive Director, CNCF in an enormous hall filled with more than 4100 people. It was like everybody from the container community was present there!

The conference was very well organised considering the large scale of the event. People from all around the world were present, sharing their experience with Kubernetes.

Apart from the keynotes, I mostly attended beginner and intermediate level talks, due to the fact that some of the sessions required high technical knowledge that I didn’t possess yet.

One of the speech that I enjoyed was given by Oliver Beattie, Engineering head, Monzo Bank where he talked about the Kubernetes outrage that they experienced a few months ago and how they handled its consequences.

Other talks that interested me were how Adidas is using cloud native technologies and closing remarks given by the inspiring Kelsey Hightower. It was wonderful to see the growth of cloud and container technologies and its communities.

A large number of sponsor booths were present, including RedHat, AWS, IBM and Google cloud, Azure. They shared their workflow and technologies they were using. I visited several booths, interacted with amazing people and got lots of stickers and goodies!

I was fortunate enough to win two raffles conducted by the sponsors! A big thank you to node.js for the Raspberry Pi kit and Mesosphere for the drone.

Raffle Winner!

Our sponsors also organised Diversity Lunch event for the scholarship recipients. The committee had a great discussion on inclusion and diversity along with excellent meals.

I had face-to-face interactions with some inspiring developers and employees of tech giants. Being among the youngest to attend the conference, I had a lot to learn from everyone around me and grow my network.

On the day before the last, an all attendee party and dinner was held in Tivoli Gardens, in the heart of the city. The evening was filled with amusement rides, beautiful gardens, and more. What more would you expect?

Tivoli Gardens Event in Tivoli Gardens, Copenhagen

I would like to express gratitude to CNCF, The Linux Foundation and Wendy West for this opportunity, and for helping the community involve more diversity. I look forward to attend more such events in the future!

Photographs by Cloud Native Foundation

Planet DebianLouis-Philippe Véronneau: Taiwan Travel Blog - Day 2 & 3

My Taiwan Travel blog continues! I was expecting the weather to go bad on July 10th, but the typhoon arrived late and the rain only started around 20:00. I'm pretty happy because that means I got to enjoy another beautiful day of hiking in Taroko National Park.

I couldn't find time on the 10th to sit down and blog about my trip, so this blog will also include what I did on the 11th.

Xiaozhuilu Trail (小锥麓步道)

Suspension bridge in Xiaozhuilu

The first path I did on the 10th was Xiaozhuilu to warm my muscles a little bit. It links the Shakadang Trail to the Taroko Visitor center and it's both easy and enjoyable. The path is mainly composed of stairs and man-made walkways, but it's the middle of the forest and goes by the LiWu river.

To me, the highlight of the trail was the short rope suspension bridge. How cute!

Dekalun Trail (得卡伦步道)

Once I finished the Xiaozhuilu trail, I decided I was ready for something a little more challenging. Since the park was slowly closing down because of the incoming Typhoon Maria, the only paths I could do were the ones where I didn't need to ride a bus.

I thus started climbing the Dekalun Trail, situated right behind the Taroko Visitor Center.

Although the path is very steep and goes through the wild forest/jungle, this path is also mainly man-made walkways and stairs. Here is a forest interpretation poster I really liked:

The leaves of a tree are its name cards. The name cards of the Macaranga tree are very special. They are large and round and the petiole is not on the leaf margin, it is inside the leaf blade. They are called perforated leaves and look like shields. [...] The Macaranga tree is like a spearhead. When the village here relocated and the fields were abandoned, it quickly moved in. The numerous leaves form a large umbrella that catches a large amount of sunlight and allows it to grow quickly. It can be predicted that in the future, the Macaranga will gradually be replaced by trees that are more shade tolerant. In the meantime however, its leaves, flowers and fruits are a source of food loved by the insects and birds.

A very fengshui tree yo.

Here is a bonus video of one of the giant spiders I was describing yesterday being eaten by ants. For size comparison, the half step you can see is about 10cm large...

Dali - Datong Trail (大礼-大同步道)

The Dekalun Trail ends quite abruptly and diverges into two other paths: one that goes back down and the other one that climbs to the Dali village and then continues to the Datong village.

The view from Dali Village

It was still early in the afternoon when I arrived at the crossroad so I decided that I was at least going to make it to Dali before turning back. Turns out that was a good idea, since the Dali path was a really beautiful mountainside path with a very challenging heigh difference. If the Dekalun Trail is a light 3/5, I'd say the Dali trail is a heavy 3/5. Although I'm in shape, I had to stop multiple times to sit down and try to cool myself. By itself the trail would be fine, but it's the 35+°C with a high level of humidity that made it challenging to me.

Once I arrived at Dali, I needed a permit to continue to Datong but the path was very easy, the weather beautiful and the view incredible, so I couldn't stop myself. I think I walked about half of the 6km trail from Dali to Datong before running out of water. Turns out 4L wasn't enough. The mixed guilt of not having a mountain permit and the concern I wouldn't have anything left to drink for a while made me turn back and start climbing down.

Still, no regrets! This trail was clearly the best one I did so far.

A Wild Andrew Appears!

So there I was in my bed after a day of hiking in the mountains, ready to go to sleep when Andrew Lee reached out to me.

He decided to come by my hostel to talk about the DebConf18 daytrip options. Turns out I'll be the one to lead the River Tracing daytrip on the MeiHua river (梅花溪). River tracing is a mix of bouldering and hiking, but in a river bed.

I'm a little apprehensive of taking the lead of the daytrip since I don't know if my mandarin will be good enough to fully understand the bus driver and the activity guide, but I'll try my best!

Anyway, once we finished talking about the daytrip, Andrew proposed we go to the Hualien night market. After telling him I wasn't able to rent a bike because of the incoming typhoon (nobody would rent me one), we swerved by Carrefour (a large super market chain) and ended up buying a bicycle! The clerk was super nice and even gave me a lock to go with it.

I'm now the proud owner of a brand new Giant bicycle for the rest of my trip in Taiwan. I'm retrospective, I think this was a pretty good idea. It'll end up cheaper than renting one for a large amount of time and will be pretty useful to get around during DebConf.1 It's a little small for me, but I will try to buy a longer seat post in Hualien.

Music and Smoked Flying Fish

After buying the bike, I guess we said fuck the night market and met up with one of Andrew's friend who is a musician and owns a small recording studio. We played music with him for a while and sang songs, and then went back to Andrew's place to eat some flying fish that Andrew had smoked. We drank a little and I decided to sleep there because it was getting pretty late.

Andrew was a wonderful guest and brought me back to my hostel the next day in the afternoon after showing me the Hualien beach and drinking some tea in a local teashop with me. I had a very good time.

What an eventful two days that was! Turns out the big typhoon that was supposed to hit on the 11th turned out to be a fluke and passed to the north of Taiwan: in Hualien we only had a little bit of rain. So much for the rainpocalyspe I was expecting!

Language Rant bis

Short but heartfelt language rant: Jesus Christ on a paddle-board, communication in a language you don't really master is exhausting. I recently understood one of the sentences I was trying to decipher was a pun and I laughed. Then cried a little.


  1. If you plan to stay in Taiwan after DebConf and need a bicycle, I would be happy to sell it for 1500 NTD$ (40€), half of what I paid. It's a little bit cheap, but it's brand new and comes with a 1 year warranty! Better than walking if you ask me. 

,

Planet DebianCharles Plessy: Still not going to Debconf....

I was looking forward to this year's Debconf in Taiwan, the first in Asia, and the perspective of attending it with no jet lag, but I happen to be moving to Okinawa and changing jobs on August 1st, right at the middle of it...

Moving is a mixed feeling of happiness and excitation for what I am about to find, and melancholy about what and whom I am about to leave. But flights to Tôkyô and Yokohama are very affordable.

Special thanks to the Tôkyô Debian study group, where I got my GPG key signed by Debian developers a long time ago :)

Planet DebianBálint Réczey: Run Ubuntu on Windows, even multiple releases in parallel!

Running Linux terminals on Windows needs just a few clicks since we can install Ubuntu, Debian and other distributions right from the Store as apps, without the old days’ hassle of dual-booting or starting virtual machines. It just works and it works even in enterprise environments where installation policies are tightly controlled.

If you check the Linux distribution apps based on the Windows Subsystem for Linux technology you may notice that there is not only one Ubuntu app, but there are already three, Ubuntu, Ubuntu 16.04 and Ubuntu 18.04. This is no accident. It matches the traditional Ubuntu release offering where the LTS releases are supported for long periods and there is always a recommended LTS release for production:

  • Ubuntu 16.04 (code name: Xenial) was the first release really rocking on WSL and it will be updated in the Store until 16.04’s EOL, April, 2021.
  • Ubuntu 18.04 (code name: Bionic) is the current LTS release (also rocking :-)) and the first one supporting even ARM64 systems on Windows. It will be updated in the Store until 18.04’s EOL, April, 2023.
  • Ubuntu (without the release version) always follows the recommended release, switching over to the next one when it gets the first point release. Right now it installs Ubuntu 16.04 and will switch to 18.04.1, on 26th July, 2018.

The apps in the Store are like installation kits. Each app creates a separate root file system in which Ubuntu terminals are opened but app updates don’t change the root file system afterwards. Installing a different app in parallel creates a different root file system allowing you to have both Ubuntu LTS releases installed and running in case you need it for keeping compatibility with other external systems. You can also upgrade your Ubuntu 16.04 to 18.04 by running ‘do-release-upgrade’ and have three different systems running in parallel, separating production and sandboxes for experiments.

What amazes me in the WSL technology is not only that Linux programs running directly on Windows perform surprisingly well (benchmarks), but the coverage of programs you can run unmodified without any issues and without the large memory overhead of virtual machines.

I hope you will enjoy the power or the Linux terminals on Windows at least as much we enjoyed building the apps at Canonical working closely with Microsoft to make it awesome!

Planet DebianMarkus Koschany: My Free Software Activities in June 2018

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Games

  • I advocated Phil Morrell to become Debian Maintainer with whom I have previously worked together on corsix-th. This month I sponsored his updates for scorched3d and the new play.it package, an installer for drm-free commercial games. Play.it is basically a collection of shell scripts that create a wrapper around games from gog.com or Steam and put them into a Debian package which is then seamlessly integrated into the user’s system.  Similar software are game-data-packager, playonlinux or lutris (not yet in Debian).
  • I packaged new upstream releases of blockattack, renpy, atomix and minetest, and also backported Minetest version 0.4.17.1 to Stretch later on.
  • I uploaded RC bug fixes from Peter de Wachter for torus-trooper, tumiki-fighters and val-and-rick and moved the packages to Git.
  • I tackled an RC bug (#897548) in yabause, a Saturn emulator.
  • I sponsored connectagram, cutemaze and tanglet updates for Innocent de Marchi.
  • Last but not least I refreshed the packaging of trophy and sauerbraten which had not seen any updates for the last couple of years.

Debian Java

  • I packaged a new upstream release of activemq and could later address #901366 thanks to a bug report by Chris Donoghue.
  • I also packaged upstream releases of bouncycastle, libpdfbox-java, libpdfbox2-java because of reported security vulnerabilities.
  • I investigated and fixed RC bugs in openjpa (#901045), osgi-foundation-ee (#893382) and ditaa (#897494, Java 10 related).
  • A snakeyaml update introduced a regression in apktool (#902666) which was only visible at runtime. Once known I could fix it.
  •   I worked on Netbeans again. It can be built from source now but there is still a runtime error (#891957) that prevents users from starting the application. The current plan is to package the latest release candidate of Netbeans 9 and move forward.

Debian LTS

This was my twenty-eight month as a paid contributor and I have been paid to work 23,75 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

  • From 18.06.2018 until 24.06.2018 I was in charge of our LTS frontdesk. I investigated and triaged CVE in jasperreports, 389-ds-base, asterisk, lava-server, libidn, php-horde-image, tomcat8, thunderbird, glusterfs, ansible, mercurial, php5, jquery, redis, redmine, libspring-java, php-horde-crypt, mupdf, binutils, jetty9 and libpdfbox-java.
  • DSA-4221-1. Issued a security update for libvncserver fixing 1 CVE.
  • DLA-1398-1. Issued a security update for php-horde-crypt fixing 2 CVE.
  • DLA-1399-1. Issued a security update for ruby-passenger fixing 2 CVE.
  • DLA-1411-1. Issued a security update for tiff fixing 5 CVE.
  • DLA-1410-1. Issued a security update for python-pysaml fixing 2 CVE.
  • DLA-1418-1. Issued a security update for bouncycastle fixing 7 CVE.

ELTS

Extended Long Term Support (ELTS) is a new project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 7 „Wheezy“. This was my first month and I have been paid to work 7 hours on ELTS.

  • ELA-1-1. Issued a security update for Git fixing 1 CVE.
  • ELA-8-1. Issued a security update for ruby-passenger fixing 1 CVE.
  • ELA-14-1. Backported the Linux 3.16 kernel from Jessie to Wheezy. This update also included backports of initramfs-tools and the linux-latest source package. The new kernel is available for amd64 and i386 architectures.

Misc

  • I prepared security updates for libvncserver (Stretch, DSA-4221-1) and Sid) and bouncycastle (Stretch, DSA-4233-1)

Thanks for reading and see you next time.

TEDGhana eliminates trachoma, The Bail Project opens a fifth site: Updates from The Audacious Project

At One Acre Fund’s soil lab in Kenya, soil samples from small farms are analyzed to help the farmers select the right seeds and fertilizer to maximize their yields. Photo: Courtesy of One Acre Fund

Their ideas are big — aimed at impacting millions of lives or creating sweeping global change. Three months after the first project leaders of The Audacious Project stood on the TED stage and shared their ambitious plans, things are already starting to happen. Below, enjoy the latest news.

The drive to end trachoma

When Sightsavers and its partners started working in Ghana in 2000, about 2.8 million people in the country were at risk of contracting trachoma, an ancient disease that eventually causes blindness. But on June 13, 2018, the World Health Organization announced: Ghana has eliminated trachoma. It’s a very big deal, the first country in sub-Saharan Africa to reach this milestone. Caroline Harper and her team expect more countries to follow — their goal is to end trachoma across twelve African countries. Last month’s news, she says, is proof it can be done when a country’s ministry of health teams up with the right coalition of partners.

The launch of The Bail Project

The Bail Project is gaining national momentum — since launching in January 2018, the project has bailed out more than 1,000 people in four US cities. And it recently opened a fifth site in Louisville, Kentucky, where on any given night there are about 2,100 people and fewer than 1,800 beds in the Department of Corrections jail. The department estimates that 77 percent of those being held are there because they can’t afford to pay bail. The Bail Project aims to help as many of them as possible return to their families to await trial. Next up for Robin Steinberg and her team: Detroit, where The Bail Project will work with the Detroit Justice Center to assist residents who can’t otherwise pay their bail bonds.

Two new missions explore the twilight zone

Most people know The Twilight Zone as a vintage television show. Now, more people are getting to know it as the vast, dark midwater region of the ocean. On World Ocean Day in early June, as TED posted a talk from Heidi Sosik of Woods Hole Oceanographic Institution (WHOI), both The New York Times and Washington Post ran op-eds on why exploring the twilight zone is so critical. Both called for increasing our knowledge of the region before commercial interests can exploit it. WHOI’s far-reaching twilight zone exploration will begin in August. One mission, leaving from Rhode Island, will test DEEP-SEE, a new instrument designed to gather acoustical data and imagery. And a second, leaving from Seattle (funded by both NASA and the National Science Foundation), will study how phytoplankton and other organisms move carbon through the ocean to the twilight zone, making it a critical part of the climate system.

The satellite to curb methane

Last month, Environmental Defense Fund (EDF) released a study showing that US oil and natural gas companies are leaking 60 percent more methane than EPA estimates predicted. About 2.3 percent of overall natural gas output is lost, meaning that companies are essentially leaking $2 billion of their product. But EDF stresses the potential for this to motivate action — in fact, Shell, ExxonMobil and BP have already committed to reduction efforts. At the World Gas Conference, held in Washington D.C. in late June, EDF continued to share this message, showing how the launch of MethaneSAT will help companies and governments take action. During a panel, Fred Krupp said the satellite should be in orbit in three years. And at a booth, EDF demoed a virtual reality experience that showed just how easily methane leaks can be spotted and fixed. With headsets on, Methane CH4llenge users could play hero by stopping multiple leaks.

At the World Gas Conference in June, an attendee plays the Methane CH4llenge, spotting and fixing methane leaks. Photo: Courtesy of Environmental Defense Fund

The Woodstock for Black women’s health

T. Morgan Dixon and Vanessa Garrison of GirlTrek are laying the groundwork for next summer’s big event, the Summer of Selma. They’re on a 12-month, 50-city wellness revival that they’re calling the Road to Selma, and they’re making stops all around the country, holding Civil Rights Movement-style teach-ins for Black women. So far, they’ve been to New York, Detroit and New Orleans and are gearing up for stops in Houston, Baltimore and Kansas City. The Summer of Selma festival will be held May 24–27, 2019, and registration is expected to open later in the year.

The community health work revolution

Living Goods and Last Mile Health are on the way to their 2018 goal of equipping nearly 14,000 community health workers with mobile technology that will allow them to more effectively diagnose and treat members of their community at their doorsteps. “No one should die because they live too far from a doctor. Not in the 21st century,” said Raj Panjabi at a TIME 100 x WeWork Speaker event in June, where he highlighted how training community health workers in 30 life-saving skills has the potential to save 30 million lives by 2030. In the fall, Last Mile Health’s Community Health Academy will begin enrolling students in its first course, designed to help local leaders build community health worker programs in their countries. Reps from both Living Goods and Last Mile Health spoke on this topic at the World Health Assembly in late May, just as community health workers were applauded for circulating the vaccines that squashed the Ebola flare-up in Democratic Republic of the Congo.

Support for small-scale farmers

One Acre Fund is thinking a lot about soil and how optimizing it can help smallholder farmers boost their income and feed their families. On their blog, they gave readers a peek inside their soil analytics lab in Kakamega, Kenya, just as 3,000 samples had arrived from small farms in Rwanda to be analyzed. The goal of the analysis is two-fold: to determine the best kinds of seeds and fertilizer mixes for each farmer, and to collect data for a study on how farming practices affect soil health. This kind of research is helping Andrew Youn and his team scale and improve their overall operations. By the end of the year, they plan to serve 760,000 small-scale farmers, tracking well ahead of their goal of working with one million by 2020. This expansion is key for preventing another global food crisis — and promoting gender equality in a region where a high percentage of farmers are women, yet systems are not designed to help them thrive.

CryptogramPROPagate Code Injection Seen in the Wild

Last year, researchers wrote about a new Windows code injection technique called PROPagate. Last week, it was first seen in malware:

This technique abuses the SetWindowsSubclass function -- a process used to install or update subclass windows running on the system -- and can be used to modify the properties of windows running in the same session. This can be used to inject code and drop files while also hiding the fact it has happened, making it a useful, stealthy attack.

It's likely that the attackers have observed publically available posts on PROPagate in order to recreate the technique for their own malicious ends.

Worse Than FailureWalking on the Sun

In 1992, I worked at a shop that was all SunOS. Most people had a Sparc-1. Production boxes were the mighty Sparc-2, and secretaries had the lowly Sun 360. Somewhat typical hardware for the day.

SPARCstation 1

Sun was giving birth to their brand spanking new Solaris, and was pushing everyone to convert from SunOS. As with any OS change in a large shop, it doesn't just happen; migration planning needs to occur. All of our in-house software needed to be ported to the new Operating System.

This planning boiled down to: assign it to snoofle; let him figure it out.

This was before Sun made OpCom available to help people do their migrations.

I took the latest official code, opened an editor, grepped through the include files and compiled, for each OS. Then I went into a nine month long compile-edit-build cycle, noting the specifics of each item that required different include files/syntax/whatever. Basically, Sun had removed the Berkeley libraries when they first put out Solaris, so everything signal or messaging related had to change.

Finally, I naively thought the pain was over; it compiled. I had coalesced countless functions that had nearly identical multiple versions, deleted numerous blocks of dead code, and reduced 1.4 million LOC to about 700K. Then began the debugging cycle. That took about 3 weeks.

Then I was told not to merge it because another subteam in our group was doing a 9-month sub-project and couldn't be interrupted. Naturally, they were working in the main branch, which forced me to keep pulling and porting their code into mine several times a week, for months. Ironically, they were constantly changing dead code as part of trying to fix their own code.

You can only do this for so long before getting fed up; I'd had it and let it be known to boss+1 (who was pushing for Solaris) that this had to end. He set a date three months out, at which time I would do the merge and commit; other tasks be damned! The subteam was repeatedly informed of this drop-dead date.

So I put up with it for 3 months, then did the final merge; over 3,500 diffs. I went through them all, praying the power wouldn't cut out. After fixing a few typos and running the cursory test, I held my breath and committed. Then I told everyone to pull and merge.

It turns out that I missed 3 little bugs, but they were suffiently visible that it prevented the application from doing anything useful. The manager of the sub-team ordered me to roll it back because they were busy. I handed her the written memo from B+1 ordering me to do it on this date and told her to suck it up and give me a chance to debug it.

An hour later, it was working and committed.

I instructed everyone to pull and build, and to follow the instructions in my handout for coding going forward. Anything that broke the Solaris build would be summarily rolled back per orders from B+1.

It took a few months and hundreds of rollbacks for them to start to follow my instructions, but when they finally did, the problems ceased.

Then the managers from the other teams took my instructions and all my global edit scripts (it wasn't a perfect parser, but it at least left syntax errors if it tried to change code that was really badly formatted, so you could trivially find them and fix them very quickly).

Using my scripts and cheat sheets, my peers on the other projects managed to do their ports in just a couple of hours, and mercilessly rode me about it for the next 3 years.

[Advertisement] Otter - Provision your servers automatically without ever needing to log-in to a command prompt. Get started today!

Planet DebianPetter Reinholdtsen: What is the most supported MIME type in Debian in 2018?

Five years ago, I measured what the most supported MIME type in Debian was, by analysing the desktop files in all packages in the archive. Since then, the DEP-11 AppStream system has been put into production, making the task a lot easier. This made me want to repeat the measurement, to see how much things changed. Here are the new numbers, for unstable only this time:

Debian Unstable:

  count MIME type
  ----- -----------------------
     56 image/jpeg
     55 image/png
     49 image/tiff
     48 image/gif
     39 image/bmp
     38 text/plain
     37 audio/mpeg
     34 application/ogg
     33 audio/x-flac
     32 audio/x-mp3
     30 audio/x-wav
     30 audio/x-vorbis+ogg
     29 image/x-portable-pixmap
     27 inode/directory
     27 image/x-portable-bitmap
     27 audio/x-mpeg
     26 application/x-ogg
     25 audio/x-mpegurl
     25 audio/ogg
     24 text/html

The list was created like this using a sid chroot: "cat /var/lib/apt/lists/*sid*_dep11_Components-amd64.yml.gz| zcat | awk '/^ - \S+\/\S+$/ {print $2 }' | sort | uniq -c | sort -nr | head -20"

It is interesting to see how image formats have passed text/plain as the most announced supported MIME type. These days, thanks to the AppStream system, if you run into a file format you do not know, and want to figure out which packages support the format, you can find the MIME type of the file using "file --mime <filename>", and then look up all packages announcing support for this format in their AppStream metadata (XML or .desktop file) using "appstreamcli what-provides mimetype <mime-type>. For example if you, like me, want to know which packages support inode/directory, you can get a list like this:

% appstreamcli what-provides mimetype inode/directory | grep Package: | sort
Package: anjuta
Package: audacious
Package: baobab
Package: cervisia
Package: chirp
Package: dolphin
Package: doublecmd-common
Package: easytag
Package: enlightenment
Package: ephoto
Package: filelight
Package: gwenview
Package: k4dirstat
Package: kaffeine
Package: kdesvn
Package: kid3
Package: kid3-qt
Package: nautilus
Package: nemo
Package: pcmanfm
Package: pcmanfm-qt
Package: qweborf
Package: ranger
Package: sirikali
Package: spacefm
Package: spacefm
Package: vifm
%

Using the same method, I can quickly discover that the Sketchup file format is not yet supported by any package in Debian:

% appstreamcli what-provides mimetype  application/vnd.sketchup.skp
Could not find component providing 'mimetype::application/vnd.sketchup.skp'.
%

Yesterday I used it to figure out which packages support the STL 3D format:

% appstreamcli what-provides mimetype  application/sla|grep Package
Package: cura
Package: meshlab
Package: printrun
%

PS: A new version of Cura was uploaded to Debian yesterday.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Planet DebianLouis-Philippe Véronneau: Taiwan Travel Blog - Day 1

I'm going to DebConf18 later this month, and since I had some free time and I speak a somewhat understandable mandarin, I decided to take a full month of vacation in Taiwan.

I'm not sure if I'll keep blogging about this trip, but so far it's been very interesting and I felt the urge to share the beauty I've seen with the world.

This was the first proper day I spent in Taiwan. I arrived on the 8th during the afternoon, but the time I had left was all spent traveling to Hualien County (花蓮縣) were I intent to spend the rest of my time before DebConf.

Language Rant

I'm mildly annoyed at Taiwan for using traditional Chinese characters instead of simplified ones like they do in Mainland China. So yeah, even though I've been studying mandarin for a while now, I can't read much if anything at all. For those of you not familiar with mandarin, here is an example of a very common character written with simplified (后) and traditional characters (後). You don't see the resemblance between the two? Me neither.

I must say technology is making my trip much easier though. I remember a time when I had to use my pocket dictionary to lookup words and characters and it used to take me up to 5 minutes to find a single character1. That's how you end up ordering cold duck blood soup from a menu without pictures after having given up on translating it.

Now, I can simply use my smartphone and draw the character I'm looking for in my dictionary app. It's fast, it's accurate and it's much more complete than a small pocket dictionary.

Takoro National Park (太鲁阁国家公园)

Since I've seen a bunch of large cities in China already and I both dislike pollution and large amounts of people squished up in too few square meters, I rapidly decided I wasn't going to visit Taipei and would try to move out and explore one of the many national parks in Taiwan.

After looking it up, Takoro National Park in the Hualien County seemed the best option for an extended stay. It's large enough that there is a substantial tourism economy built around visiting the multiple trails of the park, there are both beginner and advanced trails you can choose from and the scenery is incredible.

Also Andrew Lee lives nearby and had a bunch of very nice advice for me, making my trip to Takoro much easier.

Swallow Gorge (燕子口)

Picture of the LiWu river in Yanzikou

The first trail I visited in the morning was Swallow Gorge. Apparently it's frequently closed because of falling rocks. Since the weather was very nice and the trail was open, I decided to start by this one.

Fun fact, at first I thought the swallow in Swallow Gorge meant swallowing, but it is swallow as in the cute bird commonly associated with spring time. The gorge is named that way because the small holes in the cliffs are used by swallows to nest. I kinda understood that when I saw a bunch of them diving and playing in the wind in front of me.

The Gorge was very pretty, but it was full of tourists and the "trail" was actually a painted line next to the road where car drives. It was also pretty short. I guess that's ok for a lot of people, but I was looking for something a little more challenging and less noisy.

Shakadang Trail (砂卡礑步道)

The second trail I visited was the Shakadang trail. The trail dates back to 1940, when the Japanese tried to use the Shakadang river for hydroelectricity.

Shakadang's river water is bright blue and extremely clear

This trail was very different from Yanzikou, being in the wild and away from cars. It was a pretty easy trail (2/5) and although part of it was paved with concrete, the more you went the wilder it got. In fact, most of the tourist gave up after the first kilometer and I had the rest of the path to myself afterwards.

Some cute purple plant growing along the river

The path is home to a variety of wild animals, plants and insects. I didn't see any wild board, but gosh damn did I saw some freakingly huge spiders. As I learnt later, Taiwan is home of the largest spiders in the world. The ones I saw (Golden silk orb-weaver, Nephila pilipes) had bodies easily 3 to 5cm long and 2cm thick, with an overall span of 20cm with their legs.

I also heard some bugs (I guess it was bugs) making a huge racket that somewhat reminded me of an old car's loose alternator belt strap on a cold winter morning.


  1. Using a Chinese dictionary is a hard thing to do since there is no alphabet. Instead, the characters are classified by the number of strokes in their radicals and then by the number of strokes in the rest of the character. 

Planet DebianIan Wienand: uwsgi; oh my!

The world of Python based web applications, WSGI, its interaction with uwsgi and various deployment methods can quickly turn into a incredible array of confusingly named acronym soup. If you jump straight into the uwsgi documentation it is almost certain you will get lost before you start!

Below tries to lay out a primer for the foundations of application deployment within devstack; a tool for creating a self-contained OpenStack environment for testing and interactive development. However, it is hopefully of more general interest for those new to some of these concepts too.

WSGI

Let's start with WSGI. Fully described in PEP 333 -- Python Web Server Gateway Interface the core concept a standardised way for a Python program to be called in response to a web request. In essence, it bundles the parameters from the incoming request into known objects, and gives you can object to put data into that will get back to the requesting client. The "simplest application", taken from the PEP directly below, highlights this perfectly:

def simple_app(environ, start_response):
     """Simplest possible application object"""
     status = '200 OK'
     response_headers = [('Content-type', 'text/plain')]
     start_response(status, response_headers)
     return ['Hello world!\n']

You can start building frameworks on top of this, but yet maintain broad interoperability as you build your application. There is plenty more to it, but that's all you need to follow for now.

Using WSGI

Your WSGI based application needs to get a request from somewhere. We'll refer to the diagram below for discussions of how WSGI based applications can be deployed.

Overview of some WSGI deployment methods

In general, this is illustrating how an API end-point http://service.com/api/ might be connected together to an underlying WSGI implementation written in Python (web_app.py). Of course, there are going to be layers and frameworks and libraries and heavens knows what else in any real deployment. We're just concentrating on Apache integration -- the client request hits Apache first and then gets handled as described below.

CGI

Starting with 1 in the diagram above, we see CGI or "Common Gateway Interface". This is the oldest and most generic method of a web server calling an external application in response to an incoming request. The details of the request are put into environment variables and whatever process is configured to respond to that URL is fork() -ed. In essence, whatever comes back from stdout is sent back to the client and then the process is killed. The next request comes in and it starts all over again.

This can certainly be done with WSGI; above we illustrate that you'd have a framework layer that would translate the environment variables into the python environ object and connect up the processes output to gather the response.

The advantage of CGI is that it is the lowest common denominator of "call this when a request comes in". It works with anything you can exec, from shell scripts to compiled binaries. However, forking processes is expensive, and parsing the environment variables involves a lot of fiddly string processing. These become issues as you scale.

Modules

Illustrated by 2 above, it is possible to embed a Python interpreter directly into the web server and call the application from there. This is broadly how mod_python, mod_wsgi and mod_uwsgi all work.

The overheads of marshaling arguments into strings via environment variables, then unmarshaling them back to Python objects can be removed in this model. The web server handles the tricky parts of communicating with the remote client, and the module "just" needs to translate the internal structures of the request and response into the Python WSGI representation. The web server can manage the response handlers directly leading to further opportunities for performance optimisations (more persistent state, etc.).

The problem with this model is that your web server becomes part of your application. This may sound a bit silly -- of course if the web server doesn't take client requests nothing works. However, there are several situations where (as usual in computer science) a layer of abstraction can be of benefit. Being part of the web server means you have to write to its APIs and, in general, its view of the world. For example, mod_uwsgi documentation says

"This is the original module. It is solid, but incredibly ugly and does not follow a lot of apache coding convention style".

uwsgi

mod_python is deprecated with mod_wsgi as the replacement. These are obviously tied very closely to internal Apache concepts.

In production environments, you need things like load-balancing, high-availability and caching that all need to integrate into this model. Thus you will have to additionally ensure these various layers all integrate directly with your web server.

Since your application is the web server, any time you make small changes you essentially need to manage the whole web server; often with a complete restart. Devstack is a great example of this; where you have 5-6 different WSGI-based services running to simulate your OpenStack environment (compute service, network service, image service, block storage, etc) but you are only working on one component which you wish to iterate quickly on. Stopping everything to update one component can be tricky in both production and development.

uwsgi

Which brings us to uwsgi (I call this "micro-wsgi" but I don't know if it actually intended to be a μ). uwsgi is a real Swiss Army knife, and can be used in contexts that don't have to do with Python or WSGI -- which I believe is why you can get quite confused if you just start looking at it in isolation.

uwsgi lets us combine some of the advantages of being part of the web server with the advantages of abstraction. uwsgi is a complete pluggable network daemon framework, but we'll just discuss it in one context illustrated by 3.

In this model, the WSGI application runs separately to the webserver within the embedded python interpreter provided by the uwsgi daemon. uwsgi is, in parts, a web-server -- as illustrated it can talk HTTP directly if you want it to, which can be exposed directly or via a traditional proxy.

By using the proxy extension mod_proxy_uwsgi we can have the advantage of being "inside" Apache and forwarding the requests via a lightweight binary channel to the application back end. In this model, uwsgi provides a uwsgi:// service using its internal protcol on a private port. The proxy module marshals the request into small packets and forwards it to the given port. uswgi takes the incoming request, quickly unmarshals it and feeds it into the WSGI application running inside. Data is sent back via similarly fast channels as the response (note you can equally use file based Unix sockets for local only communication).

Now your application has a level of abstraction to your front end. At one extreme, you could swap out Apache for some other web server completely and feed in requests just the same. Or you can have Apache start to load-balance out requests to different backend handlers transparently.

The model works very well for multiple applications living in the same name-space. For example, in the Devstack context, it's easy with mod_proxy to have Apache doing URL matching and separate out each incoming request to its appropriate back end service; e.g.

  • http://service/identity gets routed to Keystone running at localhost:40000
  • http://service/compute gets sent to Nova at localhost:40001
  • http://service/image gets sent to glance at localhost:40002

and so on (you can see how this is exactly configured in lib/apache:write_uwsgi_config).

When a developer makes a change they simply need to restart one particular uwsgi instance with their change and the unified front-end remains untouched. In Devstack (as illustrated) the uwsgi processes are further wrapped into systemd services which facilitates easy life-cycle and log management. Of course you can imagine you start getting containers involved, then container orchestrators, then clouds-on-clouds ...

Conclusion

There's no right or wrong way to deploy complex web applications. But using an Apache front end, proxying requests via fast channels to isolated uwsgi processes running individual WSGI-based applications can provide both good performance and implementation flexibility.

,

Planet DebianJonathan McDowell: Fixing a broken ESP8266

One of the IoT platforms I’ve been playing with is the ESP8266, which is a pretty incredible little chip with dev boards available for under £4. Arduino and Micropython are both great development platforms for them, but the first board I bought (back in 2016) only had a 4Mbit flash chip. As a result I spent some time writing against the Espressif C SDK and trying to fit everything into less than 256KB so that the flash could hold 2 images and allow over the air updates. Annoyingly just as I was getting to the point of success with Richard Burton’s rBoot my device started misbehaving, even when I went back to the default boot loader:

 ets Jan  8 2013,rst cause:1, boot mode:(3,6)

load 0x40100000, len 816, room 16
tail 0
chksum 0x8d
load 0x3ffe8000, len 788, room 8
tail 12
chksum 0xcf
ho 0 tail 12 room 4
load 0x3ffe8314, len 288, room 12
tail 4
chksum 0xcf
csum 0xcf

2nd boot version : 1.2
  SPI Speed      : 40MHz
  SPI Mode       : DIO
  SPI Flash Size : 4Mbit
jump to run user1

Fatal exception (0):
epc1=0x402015a4, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000
Fatal exception (0):
epc1=0x402015a4, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000
Fatal exception (0):

(repeats indefinitely)

Various things suggested this was a bad flash. I tried a clean Micropython install, a restore of the original AT firmware backup I’d taken, and lots of different combinations of my own code/the blinkenlights demo and rBoot/Espressif’s bootloader. I made sure my 3.3v supply had enough oompf (I’d previously been cheating and using the built in FT232RL regulator, which doesn’t have quite enough when the device is fully operational, rather than in UART boot mode, such as doing an OTA flash). No joy. I gave up and moved on to one of the other ESP8266 modules I had, with a greater amount of flash. However I was curious about whether this was simply a case of the flash chip wearing out (various sites claim the cheap ones on some dev boards will die after a fairly small number of programming cycles). So I ordered some 16Mb devices - cheap enough to make it worth trying out, but also giving a useful bump in space.

They arrived this week and I set about removing the old chip and soldering on the new one (Andreas Spiess has a useful video of this, or there’s Pete Scargill’s write up). Powered it all up, ran esptool.py flash_id to see that it was correctly detected as a 16Mb/2MB device and set about flashing my app onto it. Only to get:

 ets Jan  8 2013,rst cause:2, boot mode:(3,3)

load 0x40100000, len 612, room 16
tail 4
chksum 0xfd
load 0x88380000, len 565951362, room 4
flash read err, ets_unpack_flash_code
ets_main.c

Ooops. I had better luck with a complete flash erase (esptool.py erase_flash) and then a full program of Micropython using esptool.py --baud 460800 write_flash --flash_size=detect -fm dio 0 esp8266-20180511-v1.9.4.bin, which at least convinced me I’d managed to solder the new chip on correctly. Further experimention revealed I needed to pass all of the flash parameters to esptool.py to get rBoot entirely happy, and include esp_init_data_default.bin (FWIW I updated everything to v2.2.1 as part of the process):

esptool.py write_flash --flash_size=16m -fm dio 0x0 rboot.bin 0x2000 rom0.bin \
    0x120000 rom1.bin 0x1fc000 esp_init_data_default_v08.bin

Which gives (at the default 76200 of the bootloader bit):

 ets Jan  8 2013,rst cause:1, boot mode:(3,7)

load 0x40100000, len 1328, room 16
tail 0
chksum 0x12
load 0x3ffe8000, len 604, room 8
tail 4
chksum 0x34
csum 0x34

rBoot v1.4.2 - richardaburton@gmail.com
Flash Size:   16 Mbit
Flash Mode:   DIO
Flash Speed:  40 MHz

Booting rom 0.
rf cal sector: 507
freq trace enable 0
rf[112]

Given the cost of the modules it wasn’t really worth my time and energy to actually fix the broken one rather than buying a new one, but it was rewarding to be sure of the root cause. Hopefully this post at least serves to help anyone seeing the same exception messages determine that there’s a good chance their flash has died, and that a replacement may sort the problem.

Planet DebianPetter Reinholdtsen: Debian APT upgrade without enough free space on the disk...

Quite regularly, I let my Debian Sid/Unstable chroot stay untouch for a while, and when I need to update it there is not enough free space on the disk for apt to do a normal 'apt upgrade'. I normally would resolve the issue by doing 'apt install <somepackages>' to upgrade only some of the packages in one batch, until the amount of packages to download fall below the amount of free space available. Today, I had about 500 packages to upgrade, and after a while I got tired of trying to install chunks of packages manually. I concluded that I did not have the spare hours required to complete the task, and decided to see if I could automate it. I came up with this small script which I call 'apt-in-chunks':

#!/bin/sh
#
# Upgrade packages when the disk is too full to upgrade every
# upgradable package in one lump.  Fetching packages to upgrade using
# apt, and then installing using dpkg, to avoid changing the package
# flag for manual/automatic.

set -e

ignore() {
    if [ "$1" ]; then
	grep -v "$1"
    else
	cat
    fi
}

for p in $(apt list --upgradable | ignore "$@" |cut -d/ -f1 | grep -v '^Listing...'); do
    echo "Upgrading $p"
    apt clean
    apt install --download-only -y $p
    for f in /var/cache/apt/archives/*.deb; do
	if [ -e "$f" ]; then
	    dpkg -i /var/cache/apt/archives/*.deb
	    break
	fi
    done
done

The script will extract the list of packages to upgrade, try to download the packages needed to upgrade one package, install the downloaded packages using dpkg. The idea is to upgrade packages without changing the APT mark for the package (ie the one recording of the package was manually requested or pulled in as a dependency). To use it, simply run it as root from the command line. If it fail, try 'apt install -f' to clean up the mess and run the script again. This might happen if the new packages conflict with one of the old packages. dpkg is unable to remove, while apt can do this.

It take one option, a package to ignore in the list of packages to upgrade. The option to ignore a package is there to be able to skip the packages that are simply too large to unpack. Today this was 'ghc', but I have run into other large packages causing similar problems earlier (like TeX).

Update 2018-07-08: Thanks to Paul Wise, I am aware of two alternative ways to handle this. The "unattended-upgrades --minimal-upgrade-steps" option will try to calculate upgrade sets for each package to upgrade, and then upgrade them in order, smallest set first. It might be a better option than my above mentioned script. Also, "aptutude upgrade" can upgrade single packages, thus avoiding the need for using "dpkg -i" in the script above.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Don Martitake the YouTube advertisers bowling

What if there is a better way forward on the whole Safe Harbor controversy and Article 13?

Companies don't advertise on sites like YouTube, sites teeming with copyright infringers and nationalist extremists, because those companies are run by copyright infringers or nationalist extremists. Marketing decision-makers are incentivized to play a corrupt online advertising game that rewards them for supporting infringement and extremism.

So the trick here is to help people move marketing money out of bad things (negative externalities) and toward good things (positive externalities). We know that YouTube is a brand-unsafe shitshow because Google won't advertise its own end-user-facing products and services there without a whole extra layer of brand safety protection.

Big Internet companies are set up to insulate decision-makers from the consequences of their own online asshattery, anyway. The way to affect those big Internet companies is through their advertisers. So how about a tweak to Article 13? Instead of putting the consequences of infringement on the "online content sharing service provider," put it on the brand advertised. This should help in several ways.

  • Give legit services some flexibility. If your web site's business model is anything other than "get cheap eyeballs with other people's creative work" or "get cheap eyeballs by recommending divisive bullshit" then you don't have to change a thing.

  • Incentivize sites to pay for new creative work, by making works covered by an author or artist contract a more attractive place for paid advertising than "content" uploaded by random users.

  • Make it easier for marketers who want to do the right thing, by pointing out the risks of supporting bad people.

  • Move some of the risks of online advertising away from the public and toward the people who can make a difference.

How about it?

Planet DebianMinkush Jain: Getting Started with Debian Packaging

One of my tasks in GSoC involved set up of Thunderbird extensions for the user. Some of the more popular add-ons like ‘Lightning’ (calendar organiser) already has a Debian package.

Another important add on is ‘Cardbook’ which is used to manage contacts for the user based on CardDAV and vCard standards. But it doesn’t have a package yet.

My mentor, Daniel motivated me to create a package for it and upload it to mentors.debian.net. It would ease the installation process as it could get installed through apt-get. This blog describes how I learned and created a Debian package for CardBook from scratch.

Since, I was new to packaging, I did extensive research on basics of building a package from the source code and checked if the license was DFSG compatible.

I learned from various Debian wiki guides like ‘Packaging Intro’, ‘Building a Package’ and blogs.

I also studied the amd64 files included in Lightning extension package.

The package I created could be found here.

Debian Package! Debian Package

Creating an empty package

I started by creating a debian directory by using dh_make command


# Empty project folder
$ mkdir -p Debian/cardbook

# create files
$ dh_make\
> --native \
> --single \
> --packagename cardbook_1.0.0 \
> --email minkush@example.com

Some important files like control, rules, changelog, copyright are initialized with it.

The list of all the files created:


$ find /debian
debian/
debian/rules
debian/preinst.ex
debian/cardbook-docs.docs
debian/manpage.1.ex
debian/install
debian/source
debian/source/format
debian/cardbook.debhelper.lo
debian/manpage.xml.ex
debian/README.Debian
debian/postrm.ex
debian/prerm.ex
debian/copyright
debian/changelog
debian/manpage.sgml.ex
debian/cardbook.default.ex
debian/README
debian/cardbook.doc-base.EX
debian/README.source
debian/compat
debian/control
debian/debhelper-build-stamp
debian/menu.ex
debian/postinst.ex
debian/cardbook.substvars
debian/files

I gained an understanding of Dpkg package management program in Debian and its use to install, remove and manage packages.

I build an empty package with dpkg commands. This created an empty package with four files namely .changes, .deb, .dsc, .tar.gz

.dsc file contains the changes made and signature

.deb is the main package file which can be installed

.tar.gz (tarball) contains the source package

The process also created the README and changelog files in /usr/share. They contain the essential notes about the package like description, author and version.

I installed the package and checked the installed package contents. My new package mentions the version, architecture and description!


$ dpkg -L cardbook
/usr
/usr/share
/usr/share/doc
/usr/share/doc/cardbook
/usr/share/doc/cardbook/README.Debian
/usr/share/doc/cardbook/changelog.gz
/usr/share/doc/cardbook/copyright


Including CardBook source files

After successfully creating an empty package, I added the actual CardBook add-on files inside the package. The CardBook’s codebase is hosted here on Gitlab. I included all the source files inside another directory and told the build package command which files to include in the package.

I did this by creating a file debian/install using vi editor and listed the directories that should be installed. In this process I spent some time learning to use Linux terminal based text editors like vi. It helped me become familiar with editing, creating new files and shortcuts in vi.

Once, this was done, I updated the package version in the changelog file to document the changes that I have made.


$ dpkg -l | grep cardbook
ii  cardbook       1.1.0          amd64        Thunderbird add-on for address book


changelog file Changelog file after updating Package

After rebuilding it, dependencies and detailed description can be added if necessary. The Debian control file can be edited to add the additional package requirements and dependencies.

Local Debian Repository

Without creating a local repository, CardBook could be installed with:


$ sudo dpkg -i cardbook_1.1.0.deb

To actually test the installation for the package, I decided to build a local Debian repository. Without it, the apt-get command would not locate the package, as it is not in uploaded in Debian packages on net.

For configuring a local Debian repository, I copied my packages (.deb) to Packages.gz file placed in a /tmp location.

Packages.gz Local Debian Repo

To make it work, I learned about the apt configuration and where it looks for files.

I researched for a way to add my file location in apt-config. Finally I could accomplish the task by adding *.list file with package’s path in APT and updating ‘apt-cache’ afterwards.

Hence, the latest CardBook version could be successfully installed by apt-get install cardbook

Installation CardBook Installation through apt-get

Fixing Packaging errors and bugs

My mentor, Daniel helped me a lot during this process and guided me how to proceed further with the package. He told me to use Lintian for fixing common packaging error and then using dput to finally upload the CardBook package.

Lintian is a Debian package checker which finds policy violations and bugs. It is one of the most widely used tool by Debian Maintainers to automate checks for Debian policies before uploading the package.

I have uploaded the second updated version of the package in a separate branch of the repository on Salsa here inside Debian directory.

I installed Lintian from backports and learned to use it on a package to fix errors. I researched on the abbreviations used in its errors and how to show detailed response from lintian commands


$ lintian -i -I --show-overrides cardbook_1.2.0.changes

Initially on running the command on the .changes file, I was surprised to see that a large number of errors, warnings and notes were displayed!

Running Lintian on changelog Brief errors after running Lintian on Package

Running Lintian Detailed Lintian errors (1)

Running Lintian Detailed Lintian errors (2) and many more…

I spend some days to fix some errors related to Debian package policy violations. I had to dig into every policy and Debian rules carefully to eradicate a simple error. For this I referred various sections on Debian Policy Manual and Debian Developer’s Reference.

I am still working on making it flawless and hope to upload it on mentors.debian.net soon!

It would be grateful if people from the Debian community who use Thunderbird could help fix these errors.

Planet DebianMinkush Jain: Getting Started with Debian Packaging

One of my tasks in GSoC involved set up of Thunderbird extensions for the user. Some of the more popular add-ons like ‘Lightning’ (calendar organiser) already has a Debian package.

Another important add on is ‘Cardbook’ which is used to manage contacts for the user based on CardDAV and vCard standards. But it doesn’t have a package yet.

My mentor, Daniel motivated me to create a package for it and upload it to mentors.debian.net. It would ease the installation process as it could get installed through apt-get. This blog describes how I learned and created a Debian package for CardBook from scratch.

Since, I was new to packaging, I did extensive research on basics of building a package from the source code and checked if the license was DFSG compatible.

I learned from various Debian wiki guides like ‘Packaging Intro’, ‘Building a Package’ and blogs.

I also studied the amd64 files included in Lightning extension package.

The package I created could be found here.

Debian Package! Debian Package

Creating an empty package

I started by creating a debian directory by using dh_make command


# Empty project folder
$ mkdir -p Debian/cardbook

# create files
$ dh_make\
> --native \
> --single \
> --packagename cardbook_1.0.0 \
> --email minkush@example.com

Some important files like control, rules, changelog, copyright are initialized with it.

The list of all the files created:


$ find /debian
debian/
debian/rules
debian/preinst.ex
debian/cardbook-docs.docs
debian/manpage.1.ex
debian/install
debian/source
debian/source/format
debian/cardbook.debhelper.lo
debian/manpage.xml.ex
debian/README.Debian
debian/postrm.ex
debian/prerm.ex
debian/copyright
debian/changelog
debian/manpage.sgml.ex
debian/cardbook.default.ex
debian/README
debian/cardbook.doc-base.EX
debian/README.source
debian/compat
debian/control
debian/debhelper-build-stamp
debian/menu.ex
debian/postinst.ex
debian/cardbook.substvars
debian/files

I gained an understanding of Dpkg package management program in Debian and its use to install, remove and manage packages.

I build an empty package with dpkg commands. This created an empty package with four files namely .changes, .deb, .dsc, .tar.gz

.dsc file contains the changes made and signature

.deb is the main package file which can be installed

.tar.gz (tarball) contains the source package

The process also created the README and changelog files in /usr/share. They contain the essential notes about the package like description, author and version.

I installed the package and checked the installed package contents. My new package mentions the version, architecture and description!


$ dpkg -L cardbook
/usr
/usr/share
/usr/share/doc
/usr/share/doc/cardbook
/usr/share/doc/cardbook/README.Debian
/usr/share/doc/cardbook/changelog.gz
/usr/share/doc/cardbook/copyright


Including CardBook source files

After successfully creating an empty package, I added the actual CardBook add-on files inside the package. The CardBook’s codebase is hosted here on Gitlab. I included all the source files inside another directory and told the build package command which files to include in the package.

I did this by creating a file debian/install using vi editor and listed the directories that should be installed. In this process I spent some time learning to use Linux terminal based text editors like vi. It helped me become familiar with editing, creating new files and shortcuts in vi.

Once, this was done, I updated the package version in the changelog file to document the changes that I have made.


$ dpkg -l | grep cardbook
ii  cardbook       1.1.0          amd64        Thunderbird add-on for address book


changelog file Changelog file after updating Package

After rebuilding it, dependencies and detailed description can be added if necessary. The Debian control file can be edited to add the additional package requirements and dependencies.

Local Debian Repository

Without creating a local repository, CardBook could be installed with:


$ sudo dpkg -i cardbook_1.1.0.deb

To actually test the installation for the package, I decided to build a local Debian repository. Without it, the apt-get command would not locate the package, as it is not in uploaded in Debian packages on net.

For configuring a local Debian repository, I copied my packages (.deb) to Packages.gz file placed in a /tmp location.

Packages.gz Local Debian Repo

To make it work, I learned about the apt configuration and where it looks for files.

I researched for a way to add my file location in apt-config. Finally I could accomplish the task by adding *.list file with package’s path in APT and updating ‘apt-cache’ afterwards.

Hence, the latest CardBook version could be successfully installed by apt-get install cardbook

Installation CardBook Installation through apt-get

Fixing Packaging errors and bugs

My mentor, Daniel helped me a lot during this process and guided me how to proceed further with the package. He told me to use Lintian for fixing common packaging error and then using dput to finally upload the CardBook package.

Lintian is a Debian package checker which finds policy violations and bugs. It is one of the most widely used tool by Debian Maintainers to automate checks for Debian policies before uploading the package.

I have uploaded the second updated version of the package in a separate branch of the repository on Salsa here inside Debian directory.

I installed Lintian from backports and learned to use it on a package to fix errors. I researched on the abbreviations used in its errors and how to show detailed response from lintian commands


$ lintian -i -I --show-overrides cardbook_1.2.0.changes

Initially on running the command on the .changes file, I was surprised to see that a large number of errors, warnings and notes were displayed!

Running Lintian on changelog Brief errors after running Lintian on Package

Running Lintian Detailed Lintian errors (1)

Running Lintian Detailed Lintian errors (2) and many more…

I spend some days to fix some errors related to Debian package policy violations. I had to dig into every policy and Debian rules carefully to eradicate a simple error. For this I referred various sections on Debian Policy Manual and Debian Developer’s Reference.

I am still working on making it flawless and hope to upload it on mentors.debian.net soon!

It would be grateful if people from the Debian community who use Thunderbird could help fix these errors.

,

Planet DebianDominique Dumont: New Software::LicenseMoreUtils Perl module

Hello

Debian project has rather strict requirements regarding package license. One of these requirements is to provide a copyright file mentioning the license of the files included in a Debian package.

Debian also recommends to provide this copyright information in a machine readable format that contain the whole text of the license(s) or a summary pointing to a pre-defined location on the file system (see this example).

cme and Config::Model::Dpkg::Copyright helps in this task using Software::License module. But this module lacks the following features to properly support the requirements of Debian packaging:

  • license summary
  • support for clause like “GPL version 2 or (at your option) any later version”

Long story short, I’ve written Software::LicenseMoreUtils to provide these missing features. This module is a wrapper around Software::License and has the same API.

Adding license summaries for Debian requires only to update this YAML file.

This modules was written for Debian while keeping other distros in minds. Debian derevatives like Ubuntu or Mind are supported. Adding license summaries for other Linux distribution is straightforward. Please submit a bug or a PR to add support for other distributions.

For more details. please see:

 

All the best

Planet DebianCraig Small: wordpress 4.9.7

No sooner than I had patched WordPress 4.9.5 to fix the arbitrary unlink bug than I realised there is a WordPress 4.9.7 out there. This release (just out for Debian, if my Internet behaves) fixes the unlink bug found by RIPS Technologies.  However, the WordPress developers used a different method to fix it.

There will be Debian backports for WordPress that use one of these methods. It will come down to do those older versions use hooks and how different the code is in post.php

You should update, and if you don’t like WordPress deleting or editing its own files, perhaps consider using AppArmor.

,

Planet DebianClint Adams: Solve for q

f 0   =  1.5875
f 0.5 =  1.5875
f 1   =  3.175
f 2   =  6.35
f 3   =  9.525
f 4   = 12.7
f 5   = 15.875
f 6   = 19.05
f 7   = 22.225
f 8   = 25.4
f 9   = 28.575
f 10  = 31.75
Posted on 2018-07-06
Tags: barks

CryptogramFriday Squid Blogging: Squid Unexpectedly Playing a Part in US/China Trade War

Chinese buyers are canceling orders to buy US squid in advance of an expected 25% tariff.

As usual, you can also use this squid post to talk about the security stories in the news that I haven't covered.

Read my blog posting guidelines here.

Krebs on SecurityExxonMobil Bungles Rewards Card Debut

Energy giant ExxonMobil recently sent snail mail letters to its Plenti rewards card members stating that the points program was being replaced with a new one called Exxon Mobil Rewards+. Unfortunately, the letter includes a confusing toll free number and directs customers to a parked page that tries to foist Web browser extensions on visitors.

CryptogramThe NSA's Domestic Surveillance Centers

The Intercept has a long story about the NSA's domestic interception points.

Includes some new Snowden documents.

Planet DebianJonathan Dowland: Newcastle University Historic Computing

some of our micro computers

some of our micro computers

Since first writing about my archiving activities in 2012 I've been meaning to write an update on what I've been up to, but I haven't got around to it. This, however, is noteable enough to be worth writing about!

In the last few months I became chair of the Historic Computing Committee at Newcastle University. We are responsible for a huge collection of historic computing artefacts from the University's past, going back to the 1950s, which has been almost single-handedly assembled and curated over the course of decades by the late Roger Broughton, who did much of the work in his retirement.

Segment of IBM/360 mainframe

Segment of IBM/360 mainframe

Sadly, Roger died in 2016.

Recently there has been an upsurge of interest and support for our project, partly as a result of other volunteers stepping in and partly due to the School of Computing moving to a purpose-built building and celebrating its 60th birthday.

We've managed to secure some funding from various sources to purchase proper, museum-grade storage and display cabinets. Although portions of the collection have been exhibited for one-off events, including School open days, this will be the first time that a substantial portion of the collection will be on (semi-)permanent public display.

Amstrad PPC640 portable PC

Amstrad PPC640 portable PC

Things have been moving very quickly recently. I am very happy to announce that the initial public displays will be unveiled as part of the Great Exhibition of the North! Most of the details are still TBC, but if you are interested you can keep an eye on this A History Of Computing events page.

For more about the Historic Computing Committee, cs-history Special Interest Group and related stuff, you can follow the CS History SIG blog, which we will hopefully be updating more often going forward. For the Historic Computing Collection specifically, please see the The Roger Broughton Museum of Computing Artefacts.

Planet DebianHolger Levsen: 20180706-rise-of-the-machines

Rise of the machines

Last week I was in a crowd of 256 people watching and cheering Compressorhead, some were stage-diving, many pogo dancing. Truely awesome.

Worse Than FailureError'd: Is Null News Good News?

"The Eugene (Oregon) Register-Guard knows when it's a slow news day, null happens," Bill T. writes.

 

"12 months for free or a year for not free...so hard to choose!" writes Paige S.

 

Rodrigo M. wrote, "GlobalProtect thinks the current version I have installed is not very good, so why not upgrade by downgrading?"

 

"After flying with Norwegian airways I got a mail asking to take the survey," Nathan K. wrote, "Now I apparently need to find out how to file a ticket with their sysadmins."

 

"So...my name is 'Marketing' now?" wrote Anon and totally not named Marketing.

 

Brad W. writes, "For having 'Caterpillar,' 'Revolver,' and 'Steel Toe' in the description the shoe seems a bit wimpy."

 

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

,

Rondam RamblingsTrump is a personality cult

If you want proof that Donald Trump has become a cult of personality look no further than this story in the LA Times: Workers in this town may become victims of Trump's trade war, but they're behind him 'no matter what' Jimmie Coffer, a machine programmer at the nation’s largest nail-making plant, voted for Donald Trump partly because he was confident he would bring manufacturing jobs back to

Planet DebianThorsten Alteholz: My Debian Activities in June 2018

FTP master

This month I accepted 166 packages and rejected only 7 uploads. The overall number of packages that got accepted this month was 216.

Debian LTS

This was my forty eighth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 23.75h. During that time I did LTS uploads of:

  • [DLA 1404-1] lava-server security update for one CVE
  • [DLA 1403-1] zendframework security update for one CVE
  • [DLA 1409-1] mosquitto security update for two CVE
  • [DLA 1408-1] simplesamlphp security update for two CVE

I also prepared a test package for slurm-llnl but got no feadback yet *hint* *hint*.

This month has been the end of Wheezy LTS and the beginning of Jessie LTS. After asking Ansgar, I did the reconfiguration of the upload queues on seger to remove the embargoed queue for Jessie and reduce the number of supported architectures.

Further I started to work on opencv.

Unfortunately the normal locking mechanism for work on packages by claiming the package in dla-needed.txt did not really work during the transition. As a result I worked on libidn and mercurial parallel to others. There seems to be room for improvement for the next transition.

Last but not least I did one week of frontdesk duties.

Debian ELTS

This month was the first ELTS month.

During my allocated time I made the first CVE triage in my week of frontdesk duties, extended the check-syntax part in the ELTS security tracker and uploaded:

  • ELA-3-1 for file
  • ELA-4-1 for openssl

Other stuff

During June I continued the libosmocore transition but could not finish it. I hope I can upload all missing packages in July.

Further I continued to sponsor some glewlwyd packages for Nicolas Mora.

The DOPOM package for this month was dvbstream.

I also upload a new upstream version of …

CryptogramBeating Facial Recognition Software with Face Makeup

At least right now, facial recognition algorithms don't work with Juggalo makeup.

Worse Than FailureCodeSOD: To Read or Parse

When JSON started to displace XML as the default data format for the web, my initial reaction was, "Oh, thank goodness." Time passed, and people reinvented schemas for JSON and RPC APIs in JSON and wrote tools which turn JSON schemas into UIs and built databases which store BSON, which is JSON with extra steps, and… it makes you wonder what it was all for.

Then people like Mark send in some code with a subject, "WHY??!??!". It's code which handles some XML, in C#.

Now, a useful fact- C# has a rich set of API- for handling XML, and like most XML APIs, they implement two approaches.

The simplest and most obvious is the DOM-style approach, where you load an entire XML document into memory and construct a DOM out of it. It's easy to manipulate, but for large XML documents can strain the available memory.

The other is the "reader" approach, where you treat the document as a stream, and read through the document, one element at a time. This is a bit trickier for developers, but scales better to large XML files.

So let's say that you're reading a multi-gigabyte XML file. You'd want to quit your job, obviously. But assuming you didn't, you'd want to use the "reader" approach, yes? There's just one problem: the reader approach requires you to go through the document element-by-element, and you can't skip around easily.

public void ReadXml(XmlReader reader) { string xml = reader.ReadOuterXml(); XElement element = XElement.Parse(xml); … }

Someone decided to give us the "best of both worlds". They load the multi-gigabyte file using a reader, but instead of going elementwise through the document, they use ReadOuterXml to pull the entire document in as a string. Once they have the multi-gigabyte string in memory, they then feed it into the XElement.Parse method, which turns the multi-gigabyte string into a multi-gigabyte DOM structure.

You'll be shocked to learn that this code was tested with small testing files, not multi-gigabyte files, worked fine in those conditions, and thus ended up in production.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Dave HallMigrating AWS System Manager Parameter Store Secrets to a new Namespace

When starting with a new tool it is common to jump in start doing things. Over time you learn how to do things better. Amazon's AWS System Manager (SSM) Parameter Store was like that for me. I started off polluting the global namespace with all my secrets. Over time I learned to use paths to create namespaces. This helps a lot when it comes to managing access.

Recently I've been using Parameter Store a lot. During this time I have been reminded that naming things is hard. This lead to me needing to change some paths in SSM Parameter Store. Unfortunately AWS doesn't allow you to rename param store keys, you have to create new ones.

There was no way I was going to manually copy and paste all those secrets. Python (3.6) to the rescue! I wrote a script to copy the values to the new namespace. While I was at it I migrated them to use a new KMS key for encryption.

Grab the code from my gist, make it executable, pip install boto3 if you need to, then run it like so:

copy-ssm-ps-path.py source-tree-name target-tree-name new-kms-uuid

The script assumes all parameters are encrypted. The same key is used for all parameters. boto3 expects AWS credentials need to be in ~/.aws or environment variables.

Once everything is verified, you can use a modified version of the script that calls ssm.delete_parameter() or do it via the console.

I hope this saves someone some time.

,

Worse Than FailureClassic WTF: Common Sense Not Found

It's the Forth of July in the US, where we all take a day off and launch fireworks to celebrate the power of stack based languages. While we participate in American traditions, like eating hot dogs without buns, enjoy this classic WTF about a real 455hole. --Remy

Mike was a server admin at your typical everyday Initech. One day, project manager Bill stopped by his cube with questions from Jay, the developer of an internal Java application.

“Hello there- thanks for your time!” Bill dropped into Mike’s spare chair. “We needed your expertise on this one.”

“No problem,” Mike said, swiveling to face Bill. “What can I help with?”

Bill’s pen hovered over the yellow notepad in his lap. He frowned down at some notes already scribbled there. “The WAS HTTP server- that’s basically an Apache server, right?”

HTTP Error 455 - User is a Jackass

“Basically,” Mike answered. “Some IBM customizations, but yeah.”

“So it has a… HT Access file, or whatever it’s called?” Bill asked.

He meant .htaccess, the config file. “Sure, yeah.”

“OK.” Bill glanced up with wide-eyed innocence. “So we could put something into that file that would allow a redirect, right?”

“Um… it’s possible.” Uneasiness crept over Mike, who realized he was about to discuss a custom solution to a problem he didn’t know about, on a server he was responsible for. “What’s going on?”

“Well, Jay wants a redirect in there to send people to another server,” Bill replied.

Mike frowned in confusion. “We just stood this server up. Now he wants another domain?”

“Huh? Oh, no, It’s not our domain. It’s someone else’s.”

“OK… I’m lost,” Mike admitted. “Let’s start at the beginning. What’s the problem Jay wants to fix?”

“Well, he has this broken link in his app, and he wants to redirect people to the correct site,” Bill explained.

Mike stared, dumbfounded for several moments. “Excuse me?”

“Yeah. He has this link that points off to some external federal website- IRS, I think- and the link is broken. He wants to automatically redirect users to the correct site so they don’t get a 404 error. We started looking into it, and found that Apache has this HT Access file thingy. It looks like that’s what we need.”

“You’re kidding, right?” Mike blurted ahead of discretion.

“No. Why?” Bill’s eyes widened. “Something wrong?”

Mike swiveled around to retrieve his coffee mug, and a measure of composure. “Why doesn’t he just fix the link within the app so it points to the right URL?”

“Well, that’s what I asked him. But he thinks it’d be more convenient to redirect people.”

“If the link is updated, they won’t need to be redirected.”

“I realize that.”

Mike took a long swig. “That’s not what the .htaccess file is for. It’s meant to redirect an incoming request to a different server of your own, not someone else’s.”

“Oh.” Bill scribbled this down on his notepad, then stared hard at the scribbles. Every moment of silence ratcheted Mike’s nervousness higher.

“So you’re saying we can’t do the HT Access thing?” Bill finally asked, looking up again.

“To fix a broken link?”

“Yeah!” Bill’s eyes lit up. Apparently, Mike’s clarifying question had given him new hope.

“No.” Mike crushed that hope as mercilessly as he could.

“OK, so the HT Access thing won’t work. Hmm, OK.” Bill frowned back down at his notes, falling silent again. Mike sensed, and dreaded, another inane line of questioning about to follow.

“Well, another thing Jay mentioned was a custom error page,” Bill’s next foray began. “Can we do that in Apache?”

Mike hesitated. “…Yes?”

“Great! I’ll tell him that. He can develop a custom 404 page with some Javascript in it or something to redirect people to the correct site.”

“Huh?”

“Not the prettiest solution, I know, but Jay said he can make it work.”

Mike spoke slowly. “He’s going to create a custom 404 error page… for that broken link of his?”

“Yeah.”

“And that 404 page is supposed to display… when his broken link sends users off to some IRS web server?”

“Yeah.”

“The IRS web server, when it gets a request for a page that doesn’t exist, is gonna display Jay’s custom 404 error page. Is that what you’re telling me?”

Bill’s confidence faltered. “Um… I think so.”

Mike dropped the bomb. “How’s he gonna get that custom page onto their server?”

“Well, it’d be on our server.”

“Right! So how would that custom 404 error get displayed?”

“When the user clicks the broken link.”

“I asked how. You just answered when.”

“Well, OK, I don’t know! I’m not the developer here.” Bill’s hands rose defensively. “Jay said he could make it work.”

“He’s wrong!” Mike snapped.

“He was pretty confident.”

Mike hesitated a moment before his shoulders dropped. Facts and common sense were not to prevail that day. “OK then. Lemme know when it works.”

Bill perked up. “Really? You’ll put it on the server?”

“Sure. Just have him fill out a service request and I’ll deploy it.”

“Excellent! Thank you!” Bill jumped up with pleasant surprise, and left the cube.


A few days later, Mike was completely unsurprised to find Jay frowning into his cube. “My 404 page isn’t displaying!”

Mike created a new email addressed to Jay, then copied and pasted a link to the IRS Help and Resources page. “Sorry- you’ll have to take it up with the taxman.”

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

,

Sociological ImagesThe Flag Fight

Every year I see the Fourth of July spark a social media fight. First, the flag swag comes out for the ritual parties and barbecues:

Then, somebody posts the U.S. flag code, especially this part:

(d) The flag should never be used as wearing apparel, bedding, or drapery.

It is interesting that flag apparel has become a quintessential dudebro look for the Fourth. Activist Abbie Hoffman was arrested for wearing a flag shirt in protest in 1968, and we still argue about whether flag burning in protest should be legal.

Are the dudebros disrespectful? Are the flag purists raining on the parade? Sociology shows us how this debate runs into deep assumptions about how we show respect for sacred things.

In 1966, the late sociologist Robert Bellah presented a now-classic essay, “Civil Religion in America.” The essay is about religion in public life, and how American politicians created a sense of shared national identity around general religious claims. Since then, sociologists and political theorists have argued about how inclusive civil religion really is (Does it include atheists or other minority groups who aren’t Christian? Lots of Americans don’t seem to think so.), but the theory is useful for highlighting how much of American political life takes on a religious tone.

While Bellah focused on religious references in speeches and texts, there is a more general point that stands out for the flag debate:

What we have, then, from the earliest years of the republic is a collection of beliefs, symbols, and rituals with respect to sacred things and institutionalized in a collectivity…

The American civil religion…borrowed selectively from the religious tradition in such a way that the average American saw no conflict between the two. In this way, the civil religion was able to build up without any bitter struggle with the church powerful symbols of national solidarity and to mobilize deep levels of personal motivation for the attainment of national goals.

It is pretty easy to see the flag as a sacred symbol—one that represents a long history of solidarity and commitment in the United States. The trick is that civil religion focuses on the content of political beliefs more than the conduct of honoring those beliefs. The rich variety of human religious experience shows us that just because people share a sacred symbol doesn’t mean they agree about how best to celebrate it. Sure, the styles of American Christianity might appreciate quiet reverence and contemplation, but other societies partied to show their piety (Bacchanalia, anyone?).

Photo Credits: Wikimedia Commons, Scott Sherrill-Mix and US Embassy Canada via Flickr CC.

Once you consider the range in how people express their deeply-held political and cultural beliefs, it gets easier to understand where they are coming from, even if you completely disagree with them. What starts as an argument about disrespect hides a deeper argument about different kinds of celebration (and, of course, whether it is appropriate to celebrate at all)Political tensions are high these days, but cases like this show how we can have more productive arguments by getting to the core of our cultural disagreements.

Evan Stewart is a Ph.D. candidate in sociology at the University of Minnesota. You can follow him on Twitter.

(View original at https://thesocietypages.org/socimages)

TEDTED en Español: TED’s first-ever Spanish-language speaker event in NYC

Host Gerry Garbulsky opens the TED en Español event in the TEDNYC theater, New York, NY. (Photo: Dian Lofton / TED)

Thursday marked the first-ever TED en Español speaker event hosted by TED in its New York City office. The all-Spanish daytime event featured eight speakers, a musical performance, five short films and fifteen one-minute talks given by members of the audience.

The New York event is just the latest addition to TED’s sweeping new Spanish-language TED en Español initiative, designed to spread ideas to the global Hispanic community. Led by TED’s Gerry Garbulsky, also head of the world’s largest TEDx event, TEDxRiodelaPlata in Argentina, TED en Español includes a Facebook community, Twitter feed, weekly “Boletín” newsletter, YouTube channel and — as of earlier this month — an original podcast created in partnership with Univision Communications.

Should we automate democracy? “Is it just me, or are there other people here that are a little bit disappointed with democracy?” asks César A. Hidalgo. Like other concerned citizens, the MIT physics professor wants to make sure we have elected governments that truly represent our values and wishes. His solution: What if scientists could create an AI that votes for you? Hidalgo envisions a system in which each voter could teach her own AI how to think like her, using quizzes, reading lists and other types of data. So once you’ve trained your AI and validated a few of the decisions it makes for you, you could leave it on autopilot, voting and advocating for you … or you could choose to approve every decision it suggests. It’s easy to poke holes in his idea, but Hidalgo believes it’s worth trying out on a small scale. His bottom line: “Democracy has a very bad user interface. If you can improve the user interface, you might be able to use it more.”

When the focus of failure shifts from what is lost to what is gained, we can all learn to “fail mindfully,” says Leticia Gasca. (Photo: Jasmina Tomic / TED)

How to fail mindfully. If your business failed in Ancient Greece, you’d have to stand in the town square with a basket over your head. Thankfully, we’ve come a long way — or have we? Failed-business owner Leticia Gasca doesn’t think so. Motivated by her own painful experience, she set out to create a way for others like her to convert the guilt and shame of a business venture gone bad into a catalyst for growth. Thus was born “Fuckup Nights” (FUN), a global movement and event series for sharing stories of professional failure, and The Failure Institute, a global research group that studies failure and its impact on people, businesses and communities. For Gasca, when the focus of failure shifts from what is lost to what is gained, we can all learn to “fail mindfully” and see endings as doorways to empathy, resilience and renewal.

From four countries to one stage. The pan-Latin-American musical ensemble LADAMA brought much more than just music to the TED en Español stage. Inviting the audience to dance with them, Venezuelan Maria Fernanda Gonzalez, Brazilian Lara Klaus, Colombian Daniela Serna and American Sara Lucas sing and dance to a medley of rhythms that range from South American to Caribbean-infused styles. Playing “Night Traveler” and “Porro Maracatu,” LADAMA transformed the stage into a place of music worth spreading.

Gastón Acurio shares stories of the power of food to change lives. (Photo: Jasmina Tomic / TED)

World change starts in your kitchen. In his pioneering work to bring Peruvian cuisine to the world, Gastón Acurio discovered the power that food has to change peoples’ lives. As ceviche started appearing in renowned restaurants worldwide, Gastón saw his home country of Peru begin to appreciate the diversity of its gastronomy and become proud of its own culture. But food hasn’t always been used to bring good to the world. With the industrial revolution and the rise of consumerism, “more people in the world are dying from obesity than hunger,” he notes, and many peoples’ lifestyles aren’t sustainable. 
By interacting with and caring about the food we eat, Gastón says, we can change our priorities as individuals and change the industries that serve us. He doesn’t yet have all the answers on how to make this a systematic movement that politicians can get behind, but world-renowned cooks are already taking these ideas into their kitchens. He tells the stories of a restaurant in Peru that supports native people by sourcing ingredients from them, a famous chef in NYC who’s fighting against the use of monocultures and an emblematic restaurant in France that has barred meat from the menu. “Cooks worldwide are convinced that we cannot wait for others to make changes and that we must jump into action,” he says. But professional cooks can’t do it all. If we want real change to happen, Gastón urges, we need home cooking to be at the center of everything.

The interconnectedness of music and life. Chilean musical director Paolo Bortolameolli wraps his views on music within his memory of crying the very first time he listened to live classical music. Sharing the emotions music evoked in him, Bortolameolli presents music as a metaphor for life — full of the expected and the unexpected. He thinks that we listen to the same songs again and again because, as humans, we like to experience life from a standpoint of expectation and stability, and he simultaneously suggests that every time we listen to a musical piece, we enliven the music, imbuing it with the potential to be not just recognized but rediscovered.

We reap what we sow — let’s sow something different. Up until the mid-’80s, the average incomes in major Latin American countries were on par with those in Korea. But now, less than a generation later, Koreans earn two to three times more than their Latin American counterparts. How can that be? The difference, says futurist Juan Enriquez, lies in a national prioritization of brainpower — and in identifying, educating and celebrating the best minds. What if in Latin America we started selecting for academic excellence the way we would for an Olympic soccer team? If Latin American countries are to thrive in the era of technology and beyond, they should look to establish their own top universities rather than letting their brightest minds thirst for nourishment, competition and achievement — and find it elsewhere, in foreign lands.

Rebeca Hwang shares her dream of a world where identities are used to bring people together, not alienate them. (Photo: Jasmina Tomic / TED)

Diversity is a superpower. Rebeca Hwang was born in Korea, raised in Argentina and educated in the United States. As someone who has spent a lifetime juggling various identities, Hwang can attest that having a blended background, while sometimes challenging, is actually a superpower. The venture capitalist shared how her fluency in many languages and cultures allows her to make connections with all kinds of people from around the globe. As the mother of two young children, Hwang hopes to pass this perspective on to her kids. She wants to teach them to embrace their unique backgrounds and to create a world where identities are used to bring people together, not alienate them.

Marine ecologist Enric Sala wants to protect the last wild places in the ocean. (Photo: Jasmina Tomic / TED)

How we’ll save our oceans If you jumped in the ocean at any random spot, says Enric Sala, you’d have a 98 percent chance of diving into a dead zone — a barren landscape empty of large fish and other forms of marine life. As a marine ecologist and National Geographic Explorer-in-Residence, Sala has dedicated his life to surveying the world’s oceans. He proposes a radical solution to help protect the oceans by focusing on our high seas, advocating for the creation of a reserve that would include two-thirds of the world’s ocean. By safeguarding our high seas, Sala believes we will restore the ecological, economic and social benefits of the ocean — and ensure that when our grandchildren jump into any random spot in the sea, they’ll encounter an abundance of glorious marine life instead of empty space.

And to wrap it up … In an improvised rap performance with plenty of well-timed dance moves, psychologist and dance therapist César Silveyra closes the session with 15 of what he calls “nano-talks.” In a spectacular showdown of his skills, Silveyra ties together ideas from previous speakers at the event, including Enric Sala’s warnings about overfished oceans, Gastón Acurio’s Peruvian cooking revolution and even a shoutout for speaker Rebeca Hwang’s grandmother … all the while “feeling like Beyoncé.”

TEDTED en Español: el primer evento de oradores TED de habla hispana

El presentador Gerry Garbulsky da inicio al evento TED en Español en el teatro TEDNYC, Nueva York, NY (Foto: Dian Lofton/TED)

El 26 de abril tuvo lugar el primer evento de oradores de TED en Español, presentado por TED en su oficina de Nueva York. El evento, completamente en español, contó con ocho oradores, una presentación musical, cinco cortometrajes y 13 charlas de un minuto dadas por miembros de la audiencia.

El evento en Nueva York es la última incorporación a la iniciativa “TED en Español” de TED, diseñada para difundir ideas en Español a la comunidad hispana mundial. El evento fue conducido por Gerry Garbulsky, director de TED en Español (también director del mayor evento de TEDx del mundo: TEDxRiodelaPlata en Argentina.) TED en Español, además, incluye su página en TED.com, una comunidad de Facebook, un feed de Twitter, un “Boletín” semanal, un canal de YouTube y, a principios de este mes, un podcast original creado en asociación con Univision.

¿Deberíamos automatizar la democracia? “¿Soy solo yo, o hay más personas que están un poco decepcionadas con la democracia?, pregunta César A. Hidalgo. Al igual que otros ciudadanos preocupados, el profesor e investigador de física del MIT quiere asegurarse de que hayamos elegido gobiernos que realmente representen nuestros valores y deseos. Su solución: ¿qué tal si los científicos pudieran crear una IA que votara por ti? Hidalgo visualiza un sistema en el que cada votante pueda enseñar a su propia IA, cómo pensar como ella, utilizando cuestionarios, listas de lectura y otros tipos de datos. Una vez que hayas entrenado a tu IA y validado algunas decisiones que toma por ti, puedes dejarla en piloto automático, votando y representándote… o puedes decidir aprobar cada cosa que sugiera. Es muy sencillo restarle credibilidad a su idea, pero Hidalgo cree que vale la pena probarlo a menor escala. Su conclusión: “la democracia tiene una pésima interfaz de usuario. Si se pudiera mejorar la interfaz, podríamos usarla más”.

Cuando el foco del fracaso cambia de lo que se pierde a lo que se gana, todos podemos aprender a “fallar conscientemente”, afirma Leticia Gasca (Foto: Jasmina Tomic/TED)

Cómo fallar conscientemente. Si tu negocio hubiera fallado en la Antigua Grecia, habrías tenido que pararte en la plaza del pueblo con una canasta sobre tu cabeza. Afortunadamente, hemos recorrido un largo camino… ¿o no? La dueña de un negocio fallido, Leticia Gasca, no lo cree. Motivada por su dolorosa experiencia, se dispuso a crear una forma para que otros como ella, transformaran la culpa y la vergüenza de un emprendimiento que salió mal, en un acelerador del crecimiento. En consecuencia, nació “Fuckup Nights” (FUN), una serie de eventos en diversos lugares del mundo para compartir historias de fracaso profesional; y “The Failure Institute” (el Instituto del Fracaso), un grupo de investigación, que estudia el fracaso y su impacto en las personas, empresas y comunidades. Para Gasca, cuando el foco del fracaso cambia de lo que se pierde a lo que se gana, todos podemos aprender a “fallar conscientemente” y ver los desenlaces como puertas a la empatía, la resiliencia y la renovación.

De cuatro países a un escenario. El grupo musical panlatinoamericano LADAMA trajo mucho más que música al escenario de TED en Español. La venezolana María Fernanda González, la brasilera Lara Klaus, la colombiana Daniela Serna y la estadounidense Sara Lucas cantan y bailan al son de una variedad de ritmos, que van desde estilos sudamericanos hasta fusiones caribeñas, invitando a la audiencia a bailar con ellas. Tocando “Night Traveler” y “Porro Maracatu”, LADAMA transformó el escenario en un espacio musical que vale la pena difundir.

Gastón Acurio comparte historias sobre el poder de la comida para cambiar vidas (Foto: Jasmina Tomic/TED)

El cambio mundial comienza en tu cocina. En su trabajo pionero por llevar la cocina peruana al mundo, Gastón Acurio descubrió el poder que tiene la comida para cambiar la vida de las personas. A medida que el ceviche apareció en restaurantes de renombre en todo el mundo, Gastón vio que su país natal, Perú, comenzaba a apreciar la diversidad de su gastronomía y se enorgullecía de su propia cultura. Pero la comida no siempre se ha usado para traer bien al mundo. Debido a la revolución industrial y al aumento del consumismo, “muere más cantidad de gente de obesidad que de hambre”, afirma, y el estilo de vida de muchas personas no es sostenible. Al interactuar y preocuparnos por los alimentos que comemos, dice Gastón, podemos cambiar nuestras prioridades como individuos y cambiar las industrias que nos sirven. Todavía no tiene las respuestas a cómo hacer de esto un movimiento sistemático que los políticos puedan respaldar, sin embargo, cocineros de renombre alrededor del mundo están llevando estas ideas a sus cocinas. Él cuenta historias sobre un restaurante en Perú que ayuda a los nativos obteniendo ingredientes de ellos, un chef famoso en Nueva York que lucha contra el uso de monocultivos y un restaurante emblemático en Francia que ha excluido la carne del menú. “Los cocineros alrededor del mundo estamos convencidos de que no podemos esperar a que otros hagan los cambios y que debemos ponernos en acción”, afirma. Pero los cocineros profesionales no pueden hacerlo todo. Si queremos realizar un cambio profundo, urge Gastón, necesitamos que la comida casera sea la clave.

La interconexión de la música y la vida. El director de orquesta chileno, Paolo Bortolameolli, envuelve su opinión sobre la música, alrededor de su recuerdo de haber llorado la primera vez que escuchó música clásica en vivo. Compartiendo las emociones que la música causó en él, Bortolameolli presenta la misma como una metáfora de la vida, llena de lo esperado y lo inesperado. Cree que escuchamos las mismas canciones una y otra vez porque, como humanos, nos gusta experimentar la vida desde un punto de vista de expectativa y estabilidad y, a la vez, sugiere que cada vez que escuchamos una canción, animamos la música, impregnándola con el potencial de no solo ser reconocida, sino también redescubierta.

Cosechamos lo que sembramos – sembremos algo distinto. Hasta mediados de los años 80, los ingresos en los principales países latinoamericanos estaban a la par de los de Corea. Pero ahora, menos de una generación después, los coreanos ganan entre dos y tres veces más que sus contrapartes latinoamericanos. ¿Cómo puede ser? La diferencia, afirma el futurista Juan Enríquez, radica en una priorización nacional de la capacidad intelectual y en identificar, educar y celebrar las mejores mentes. ¿Qué sucedería si en América Latina empezáramos a seleccionar la excelencia académica como lo hacemos hoy con la selección nacional de fútbol? Si los países latinoamericanos prosperan en la era de la tecnología y más, deberían buscar establecer sus propias universidades superiores en lugar de dejar que sus mentes más brillantes estén ansiosas de alimento, competencia y logros, y lo encuentren en otro lugar, en tierras extranjeras.

Rebeca Hwang comparte su sueño de un mundo donde las identidades se utilizan para unir a la gente, no para alienarlas (Foto: Jasmina Tomic/TED)

La diversidad es un superpoder. Rebeca Hwang nació en Corea, fue criada en Argentina y educada en los Estados Unidos. Como alguien que ha pasado su vida intercambiando varias identidades, Hwang afirma que tener un trasfondo variado, aunque a veces sea desafiante, es en realidad un superpoder. La inversora de riesgo compartió cómo su fluidez en muchos idiomas y culturas le permite establecer conexiones con todo tipo de personas de todo el mundo. Como madre de dos niños pequeños, Hwang espera transmitir esta perspectiva a sus hijos. Ella quiere enseñarles a abrazar sus orígenes y crear un mundo donde las identidades se utilicen para unir a las personas, no para alienarlas.

El ecologista marino Enric Sala desea proteger las últimas especies salvajes del océano (Foto: Jasmina Tomic/TED)

Cómo salvaremos nuestros océanos. Si saltas al océano en cualquier lugar, dice Enric Sala, tendrías un 98 por ciento de posibilidades de sumergirte en una zona muerta, un paisaje estéril, vacío de grandes peces y otras formas de vida marina. Como ecologista marino y explorador residente de National Geographic, Sala ha dedicado su vida a inspeccionar los océanos del mundo. Enfocándose en alta mar, propone una solución radical para ayudar a proteger los océanos, fomentando la creación de una reserva que incluiría dos tercios de los océanos del planeta. Al salvaguardar nuestra alta mar, Sala cree que restauraremos los beneficios ecológicos, económicos y sociales del océano y podremos asegurarnos de que cuando nuestros nietos salten a cualquier lugar en el mar, se encuentren con una gran cantidad de vida marina gloriosa en lugar de un espacio vacío.

Y para concluir… En una presentación improvisada de rap con muchos pasos de baile bien sincronizados, el psicólogo, rapero y bailarín César Silveyra cierra el evento. En una espectacular demostración de sus habilidades, Silveyra une las ideas de oradores anteriores del evento, incluyendo las advertencias de Enric Sala sobre la sobrepesca en los océanos, la revolución de la cocina peruana de Gastón Acurio e incluso un grito para la abuela de la oradora Rebeca Hwang… todo el tiempo “sintiéndose como Beyoncé”.

TEDIdeas from the intersections: A night of talks from TED and Brightline

Onstage to host the event, Corey Hajim, TED’s business curator, and Cloe Shasha, TED’s speaker development director, kick off TEDNYC Intersections, a night of talks presented by TED and the Brightline Initiative. (Photo: Ryan Lash / TED)

At the intersections where we meet and collaborate, we can pool our collective wisdom to seek solutions to the world’s greatest problems. But true change begs for more than incremental steps and passive reactions — we need to galvanize transformation to create our collective future.

To celebrate the effort of bold thinkers building a better world, TED has partnered with the Brightline Initiative, a noncommercial coalition of organizations dedicated to helping leaders turn ideas into reality. In a night of talks at TED HQ in New York City — hosted by TED’s speaker development director Cloe Shasha and co-curated by business curator Corey Hajim and technology curator Alex Moura — six speakers and two performers showed us how we can effect real change. After opening remarks from Brightline’s Ricardo Vargas, the session kicked off with Stanford professor Tina Seelig.

Creativity expert Tina Seelig shares three ways we can all make our own luck. (Photo: Ryan Lash / TED)

How to cultivate more luck in your life. “Are you ready to get lucky?” asks Tina Seelig, a professor at Stanford University who focuses on creativity, entrepreneurship and innovation. While luck may seem to be brought on by chance alone, it turns out that there are ways you can enhance it — no matter how lucky or unlucky you think you are. Seelig shares three simple ways you can help luck to bend a little more in your direction: Take small risks that bring you outside your comfort zone; find every opportunity to show appreciation when others help you; and find ways to look at bad or crazy ideas with a new perspective. “The winds of luck are always there,” Seelig says, and by using these three tactics, you can build a bigger and bigger sail to catch them.

A new mantra: let’s fail mindfully. We celebrate bold entrepreneurs whose ingenuity led them to success — but how do we treat those who have failed? Leticia Gasca, founder and director of the Failure Institute, thinks we need to change the way we talk about business failure. After the devastating closing of her own startup, Gasca wiped the experience from her résumé and her mind. But she later realized that by hiding her failure, she was missing out on a valuable opportunity to connect. In an effort to embrace failure as an experience to learn from, Gasca co-created the Failure Institute, which includes international Fuck-Up Nights — spaces for vulnerability and connection over shared experiences of failure. Now, she advocates for a more holistic culture around failure. The goal of failing mindfully, Gasca says, is to “be aware of the consequences of the failed business,” and “to be aware of the lessons learned and the responsibility to share those learnings with the world.” This shift in the way we address failure can help make us better entrepreneurs, better people, and yes — better failures.

A police officer for 25 years, Tracie Keesee imagines a future where communities and police co-produce public safety in local communities. Photo: Ryan Lash / TED

Preserving dignity, guaranteeing justice. We all want to be safe, and our safety is intertwined, says Tracie Keesee, cofounder of the Center for Policing Equity. Sharing lessons she’s learned from 25 years as a police officer, Keesee reflects on the challenges — and opportunities — we all have for creating safer communities together. Policies like “Stop, Question and Frisk” set police and neighborhoods as adversaries, creating alienation, specifically among African Americans; instead, Keesee shares a vision for how the police and the neighborhoods they serve can come together to co-produce public safety. One example: the New York City Police Department’s “Build the Block Program,” which helps community members interact with police officers to share their experiences. The co-production of justice also includes implicit bias training for officers — so they can better understand how this biases we all carry impact their decision-making. By ending the “us vs. them” narrative, Keesee says, we can move forward together.

We can all be influencers. ​Success was once defined by power, but today it’s tied to influence, or “the ability to have an effect on a person or outcome,” says behavioral scientist Jon Levy. It rests on two building blocks: who you’re connected to and how much they trust you. In 2010, Levy created “Influencers” dinners, gathering a dozen high-profile people (who are strangers to each other) at his apartment. But how to get them to trust him and the rest of the group? He asks his guests to cook the meal and clean up. “I had a hunch this was working,” Levy recalls, “when one day I walked into my home and 12-time NBA All-Star Isiah Thomas was washing my dishes, while singer Regina Spektor was making guac with the Science Guy himself, Bill Nye.” From the dinners have emerged friendships, professional relationships and support for social causes. He believes we can cultivate our own spheres of influence at a scale that works for us. “If I can encourage you to do anything, it’s to bring together people you admire,” says Levy. “There’s almost no greater joy in life.”

Yelle and GrandMarnier rock the TED stage with electro-pop and a pair of bright yellow jumpsuits. (Photo: Ryan Lash / TED)

The intersection of music and dance. All the way from France, Yelle and GrandMarnier grace the TEDNYC stage with two electro-pop hits, “Interpassion” and “Ba$$in.” Both songs groove with robotic beats, Yelle’s hypnotic voice, kaleidoscopic rhythms and hypersonic sounds that rouse the audience to stand up, let loose and dance in the aisles.

How to be a great ally. We’re taught to believe that working hard leads directly to getting what you deserve — but sadly, this isn’t the case for many people. Gender, race, ethnicity, religion, disability, sexual orientation, class and geography — all of these can affect our opportunities for success, says writer and advocate Melinda Epler, and it’s up to all of us to do better as allies. She shares three simple ways to start uplifting others in the workplace: do no harm (listen, apologize for mistakes and never stop learning); advocate for underrepresented people in small ways (intervene if you see them being interrupted); and change the trajectory of a life by mentoring or sponsoring someone through their career. “There is no magic wand that corrects diversity and inclusion,” Epler says. “Change happens one person at a time, one act at a time, one word at a time.”

AJ Jacobs explains the powerful benefits of gratitude — and takes us on his quest to thank everyone who made his morning cup of coffee. (Photo: Ryan Lash / TED)

Lessons from the Trail of Gratitude. Author AJ Jacobs embarked on a quest with a deceptively simple idea at its heart: to personally thank every person who helped make his morning cup of coffee. “This quest took me around the world,” Jacobs says. “I discovered that my coffee would not be possible without hundreds of people I take for granted.” His project was inspired by a desire to overcome the brain’s innate “negative bias” — the psychological tendency to focus on the bad over the good — which is most effectively combated with gratitude. Jacobs ended up thanking everyone from his barista and the inventor of his coffee cup lid to the Colombian farmers who grew the coffee beans and the steelworkers in Indiana who made their pickup truck — and more than a thousand others in between. Along the way, he learned a series of perspective-altering lessons about globalization, the importance of human connection and more, which are detailed in his new TED Book, Thanks a Thousand: A Gratitude Journey. “It allowed me to focus on the hundreds of things that go right every day, as opposed to the three or four that go wrong,” Jacobs says of his project. “And it reminded me of the astounding interconnectedness of our world.”

Worse Than FailureFlobble

The Inner Platform Effect, third only after booleans and dates, is one of the most complicated blunders that so-called developers (who think that they know what they're doing) do to Make Things Better.™ Combine that with multiple inheritance run-amok and a smartass junior developer who thinks documentation and method naming are good places to be cute, and you get todays' submission.

A cat attacking an impossible object illusion to get some tuna from their human

Chops,an experienced C++ developer somewhere in Europe, was working on their flagship product. It had been built slowly over 15 years by a core of 2-3 main developers, and an accompanying rotating cast of enthusiastic but inexperienced C++ developers. The principal developer had been one of those juniors himself at the start of development. When he finally left, an awful lot of knowledge walked out the door with him.

Enormous amounts of what should have been standard tools were homegrown. Homegrown reference counting was a particular bugbear, being thread dangerous as it was - memory leaks abounded. The whole thing ran across a network, and there were a half-dozen ways any one part could communicate with another. One such way was a "system event". A new message object was created and then just launched into the underlying messaging framework, in the hopes that it would magically get to whoever was interested, so long as that other party had registered an interest (not always the case).

A new system event was needed, and a trawl was made for anyone who knew anything about them. <Crickets> Nobody had any idea how they worked, or how to make a new one. The documentation was raked over, but it was found to mostly be people complaining that there was no documentation. The code suffered from inheritance fever. In a sensible system, there would be only one message type, and one would simply tag it appropriately with an identifier before inserting the data of interest.

In this system, there was an abstract base message type, and every specific message type had to inherit from it, implement some of the functions and override some others. Unfortunately, each time it seemed to be a different set of functions being implemented and a different set being overridden. Some were clearly cut and paste jobs, copying others, carrying their mistakes forward. Some were made out of several pieces of others; cut, paste and compiled until the warning messages were disabled compiler stopped complaining.

Sometimes, when developing abstract base types that were intended to be inherited from to create a concrete class for a new purpose, those early developers had created a simple, barebones concrete example implementation. A reference implementation, with "Example" in the name, that could be used as a starting point, with comments, making it clear what was necessary and what was optional. No such example class could be found for this.

Weeks of effort went into reverse-engineering the required messaging functionality, based on a few semi-related examples. Slowly, the shape of the mutant inside became apparent. Simple, do-nothing message objects were created and tested. Each time they failed, the logs were pored over, breakpoints were added, networks were watched, tracing the point of failure and learning something new.

Finally, the new message object was finished. It worked. There was still some voodoo coding in it; magic incantations that were not understood (the inheritance chain was more than five levels deep, with multiple diamonds, and one class being inherited from six times), but it worked, although nobody was certain why.

During the post development documentation phase, Mister Chops was hunting down every existing message object. Each would need reviewing and examination at some point, with the benefit of the very expensive reverse engineering. He came across one with an odd name; it wasn't used anywhere, so hadn't been touched since it was first committed. Nobody had ever had a reason to look at it. The prefix of the name was as expected, but the suffix - the part that told you at a glance what kind of message it was - was "Flobble". Chops opened it up.

It was a barebones example of a concrete implementation of the abstract base class, with useful explanatory comments on how to use/extend it, and how it worked. Back at the start, some developer, instead of naming the example class "Example" as was customary, or naming it anything at all that would have made it clear what it was, had named it "Flobble". It sat there for a decade, while people struggled to understand these objects over and over, and finally reverse engineered it at *significant* expense. Because some whimsical developer a decade previously had decided to be funny.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Don MartiNudgestock 2018 transcript

(This is a cleaned-up and lightly edited version of my talk from Nudgestock 2018.)

First I have to give everybody a disclaimer. This is 100% off message. I work for Mozilla. I am NOT speaking for Mozilla here.

If you follow Rory, you have probably heard a lot about signaling in advertising, so I'm going to go over this material pretty quickly. Why does Homo economicus read magazine advertising but hangs up on cold calls? To put it another way why is every car commercial the same? You could shoot the "car driving down the windy road" commercial with any car. All that the car commercial tells you is: if it was a waste of your time to test drive our car then it would have been a waste of our money to make this little movie about it.

There's a whole literature of economics and math about signaling involving deceptive senders and honest senders. With this paper, Gardete and Bart show that when the sender wants to really get a message across, counter-intuitively the best thing for the sender to do is deprive themselves of some information about the receiver. If you're in the audience and you know what the sender knows about you, then you can't tell are they honestly expressing their intentions in the market, or are they just telling you what you want to hear? Anyone who used to read Computer Shopper magazine for the ads didn't read for specific information about all the parts that you might put into your computer. You read it to find out which manufacturers are adopting which standards so you don't buy a motherboard that won't support the video card that you might want to upgrade to next year.

There are three sets of papers in the signaling literature. There are papers that have pure math where you devise kind of a mathematical game of buyers and sellers and see how that game works out. And there are papers where you take users in an experimental setting. Ambler and Hollier took 540 users, showed them different versions of expensive looking and cheap looking advertising that conveys the same information. Finally you've got the kind of research that looks at spending across different product categories, and in this study they found that types of product that have different advertising to sales ratios really depends on how much extra user experience it takes to evaluate that product.

The feedback loop here is that when brands have signaling power, then that means market power for the publishers that carry their advertising, which means advertising rates tend to go up, which means the publishers can afford to make obviously expensive content. And when you attach advertising to obviously expensive content, that means more signaling power. It's kind of a loop that builds more and more value for the advertiser.

Some people compare this to the signaling that a bank does when they build this monstrous stone building to keep your money. Really, the stuff that a bank does, having a stone building doesn't do any more for keeping money in it than having a metal building or a concrete building, but it just shows that they've got this big stone building with their name on it so if they turned out to be deceptive it would be more costly for them to do it. That's the pure signaling model. But the other area that we can see when we compare this kind of classic signal-carrying advertising to online advertising, the kind of ads that are targeted to you based on who you are, is what's up with the norms enforcers?

Rory has his blue checkmark on Twitter which means he doesn't see Twitter ads. I'm less Internet Famous, so I still get the advertising on Twitter. A lot of the ads that I get are deceptive issue ads. This is one. A company that's getting sued for lead paint related issues is trying to convince residents of California that government inspectors are coming to their houses to declare them a nuisance. This is bogus and it's the kind of thing that if it appeared in the newspaper that everyone got to see then journalists and public interest lawyers, and everyone else who enforces the norms on how we communicate, would call it out. But in a targeted ad medium this kind of deceptive advertising can target me directly.

So let me show a little simulation here. What we're looking at is deceptive sellers making a sale. When a deceptive seller makes a sale that's a red line. When an honest seller makes a sale, that's a green line. The little blue squares are norms enforcers, and the only thing that makes a norms enforcer different in this game from a regular customer is when a deceptive seller contacts a norms enforcer the deceptive seller pays a higher price than they would have made in profit from a sale. So with honest sellers and deceptive sellers evolving and competing in this primordial soup of customers, what ends up happening to the deceptive sellers that try to do a broad reach and hit a bunch of different customers is, well you saw them, they hit the norms enforcers, the blue squares lit up. Advertisers who are deceptive and try to reach a bunch of different people end up getting squeezed out in this version of the game. An honest advertiser like this little square down here can reach over the whole board because they don't pay the penalty for reaching the norms enforcer.

So what does this really mean for the real web? On the World Wide Web, have we inadvertently built a game that gives an unfair advantage to deceptive sellers? If somebody can take advantage of all the the user profiling information that's available out there, and say, "oh I believe that these people are rural, low-income, unlikely to be finance journalists, therefore I'm going to hit them with the predatory finance ads," does that cause users to pay less attention to the medium?

Online advertising effectiveness has declined since the launch of the first banner advertisement in 1994. That's certainly not news. This is a slide that appeared in Mary Meeker's famous Internet Trends presentation, and as you can see blue is percentage of ad spending, grey is percentage of people's time. So TV is 36% of the time 36% of the money. Desktop web 18%, 20%, about right.

What's going on with print? Print is 9% of the money for 4% of the time. Now you might say this is just inertia, that that this year people are finally just cutting back on spending money in print because of people spending less time on print and it'll eventually catch up. But I went back and plotted the same slide from the same presentation going back to 2011, and I've got time plotted across the bottom, money plotted on the y axis, and what do we see about print? Print is on a whole different trend line. Print is on a trend line of much more value to the advertiser per unit of time spent than these other ad medium. My hypothesis is that targeting breaks signaling and this means an opportunity.

Targeting means that when you see an ad coming in targeted to you it's more like a cold call. It doesn't carry credible information about the seller's intention in the market.

From the point of view of who has an incentive to to support signal-carrying ad media instead, the people who have an interest in that signal for attention bargain in that positive feedback loop are of course the publishers, high reputation brands that want to be able to send that signal, writers, photographers, and editors, people who get paid by that publisher, and people who benefit from the positive externalities of those signal carrying ads that support news and cultural works.

So if the signaling model is such a big thing then why are there so many targeted ads still out there?

Nudges.

Let's have a look at, just to pick an example, the Facebook advertising policy. As you know, the Facebook advertising platform will let you micro target individuals extremely specifically. You can pick out seven people in Florida, you can pick out everyone who's looking for an apartment who doesn't have a certain ethnic affinity, that kind of thing. But the one thing you're not allowed to do with Facebook targeting is put anything in your ad that might indicate how you're targeting it. The policy says:

ads must not contain content that asserts or implies personal attributes

You can't say, I know you're male or female, I know your sexual orientation, I know what you do for a living. The ad copy has to be generic even if the targeting can be extremely specific. You can't even say other. You can't say meet other singles because that implies that the advertiser knows that the reader is single. Facebook will let you target people with depression but you can't reveal that you know that about them. Aanother good example is Target. They do targeting of individuals who they believe to be pregnant, but they'll pad out those ads for baby stuff with ads for other types of products so as not to creep everybody out.

Back to our shared interest in signal for attention bargain. Pretty much everybody has an interest in that original positive feedback loop of getting the higher reputation for brands of getting reputation driven publishers that'll build high quality content for us. Writers and photographers have an interest in getting paid, and people who are shopping for goods are the ones who want the signal the most. All that stands on the opposite side is behavioral tricks to conceal targeting. Now I'm not going to say this as a privacy issue. I know that there are privacy issues here but that is really not my department. Besides, Facebook just announced a dating site so they're going to breed privacy preferences out of their user base anyway.

Can the web as an advertising medium be redesigned to make it work better for carrying signal? We know from the existence of print that this type of signal carrying ad medium can exist. Print is an existence proof of signal carrying advertising. We also know that building that kind of an ad medium can't be that hard because print was built when people were breathing fumes from molten lead all day.

The prize for building a signal-carrying ad medium is all the cultural works that you get when somebody like Kurt Vonnegut can quit his job as manager of a car dealership and write for Collier's magazine full-time. This book is still on sale with the resulting stories. And of course local news. Democracy depends on the the vital flow of information of public interest. Some people say that the problem with news and information on the web is that it's all been made free, and if people would just subscribe we could fix the system. But honestly if if free was the problem, then Walter Cronkite would have destroyed the media business in 1962. It's a market design problem and a signaling problem, not just a problem of who has to pay for what.

And the web browsers got a bunch of things wrong in the 1990s. There are certain patterns of information flow that the browser facilitated, like third-party tracking, where browsers enable some companies to follow your activity from site to site, and data leakage. Things that that just don't work according to the way that people expect. Most people don't want their activity on one site to follow them over to another site, and the original batch of web browsers got that terribly wrong. The good news is web browsers are getting it right, and web browsers are under tremendous pressure now to do so. As a product the web browser is pretty much complete and working and generic. The whole point of a web browser is it shows web sites the same as all the other web browsers do, so there's less and less reason for a user to want to switch web browsers. But everybody who is trying to get you to install a web browser needs for there to be a reason, so the opportunity for browsers is to align with those interests of users that the browser wasn't able to pick up on previously.

At Mozilla some user researchers recently did a study on users with no ad blocker installed and users within the first few weeks of installing an ad blocker. Anybody want to guess on the increased engagement? How much more time those ad blocker users spend with that same browser than the non ad blocker users? Anybody shout out a number. All right, 28%. From the point of view of the browser those kinds of numbers, moving user engagement in a way that helps that browser meet its goals, that's something that that the browser can't ignore. So that means we're going from the old web game where everyone tries win by collecting as much data on people can without their permission to a new game in which the browser, high reputation publishers, and high reputation brands are all aligned in trying to build enough trust to work on information that users choose to share.

I know when I say information that users choose to share you're going to think about all these GDPR dialogs and I know I've seen these too, and they're just tons of companies on these. To be honest, looking at some of these company names it looks like most of them were made up by guys from Florida who communicate primarily by finger guns. Users should not have to micromanage their consent for all this data collection activity any more than email users should have to go in and read their SMTP headers to filter spam. And really if you think about what brands are, it's offloading information about a product buying decision onto the reputation coprocessor in the user's brain. It's kind of like taking a computational task and instead of running it on the CPU in your data center where you have to to pay the power and cooling bills for it, you offload it and run it on on the GPU on the client. It'll run faster, it'll run better, and the audience is maintaining that reputation state.

The future is here, it's just not very evenly distributed, as William Gibson said. This picture is the cyberpunk of the 1990s. Today all of that stuff he's carrying, his video camera, his laptop, his scanner, all that stuff's on a phone and everybody has it.

Today, the privacy sensitive users, the ones who are already working based on sharing data with permission, they're out there. But they're in niches today. If you have a relationship with those people now, then now is an opportunity to connect with them, figure out how to build that signal carrying advertising game, and and create a reputation based advertising model for the web. Thank you very much.

CryptogramTraffic Analysis of the LTE Mobile Standard

Interesting research in using traffic analysis to learn things about encrypted traffic. It's hard to know how critical these vulnerabilities are. They're very hard to close without wasting a huge amount of bandwidth.

The active attacks are more interesting.

EDITED TO ADD (7/3): More information.

I have been thinking about this, and now believe the attacks are more serious than I previously wrote.

,

Cory DoctorowMark Zuckerberg and his empire of oily rags

Surveillance capitalism sucks: it improves the scattershot, low-performance success-rate of untargeted advertising (well below 1 percent) and doubles or triples it (to well below 1 percent!).


But surveillance captialism is still dangerous: all those dossiers on the personal lives of whole populations can be used for blackmail, identity theft and political manipulation. As I explain in my new Locus column, Cory Doctorow: Zuck’s Empire of Oily Rags, Facebook’s secret is that they’ve found a way to turn a profit on an incredibly low-yield resource — like figuring out how to make low-grade crude out of the oil left over from oily rags.

But because the margins on surveillance data are so poor, the business is only sustainable if it fails to take the kinds of prudent precautions that would make it safe to warehouse these unimaginably gigantic piles of oily rags.

It’s as though Mark Zuckerberg woke up one morning and realized that the oily rags he’d been accumulating in his garage could be refined for an extremely low-grade, low-value crude oil. No one would pay very much for this oil, but there were a lot of oily rags, and provided no one asked him to pay for the inevitable horrific fires that would result from filling the world’s garages with oily rags, he could turn a tidy profit.

A decade later, everything is on fire and we’re trying to tell Zuck and his friends that they’re going to need to pay for the damage and install the kinds of fire-suppression gear that anyone storing oily rags should have invested in from the beginning, and the commercial surveillance industry is absolutely unwilling to contemplate anything of the sort.

That’s because dossiers on billions of people hold the power to wreak almost unimaginable harm, and yet, each dossier brings in just a few dollars a year. For commercial surveillance to be cost effective, it has to socialize all the risks associated with mass surveillance and privatize all the gains.

There’s an old-fashioned word for this: corruption. In corrupt systems, a few bad actors cost everyone else billions in order to bring in millions – the savings a factory can realize from dumping pollution in the water supply are much smaller than the costs we all bear from being poisoned by effluent. But the costs are widely diffused while the gains are tightly concentrated, so the beneficiaries of corruption can always outspend their victims to stay clear.

Facebook doesn’t have a mind-control problem, it has a corruption problem. Cambridge Analytica didn’t convince decent people to become racists; they convinced racists to become voters.

Cory Doctorow: Zuck’s Empire of Oily Rags [Cory Doctorow/Locus]

TEDCuring cancer one nanoparticle at a time, and more news from TED speakers

As usual, the TED community is hard at work — here are some highlights:

A new drug-delivering nanoparticle. Paula Hammond, the head of the Department of Chemical Engineering at MIT, is part of a research team that has developed a new nanoparticle designed to treat a kind of brain tumor called glioblastoma multiforme. The nanoparticles deliver drugs to the brain that work in two ways — to destroy the DNA of tumor cells, and to impede the reparation of those cells. The researchers were able to shrink tumors and stop them from growing back in mice — and there’s hope this technology can be used for human applications in the future. (Watch Hammond’s TED Talk).

Reflections on grief, loss and love. Amy Krouse Rosenthal penned a poignant, humorous and heart-rending love letter to her husband — published in The New York Times ten days before her death — that resonated deeply with readers across the world. In the year since, Jason Rosenthal established a foundation in her name to fund ovarian cancer research and childhood literacy initiatives. Following the anniversary of Amy’s death, Rosenthal responded to her letter in a moving reflection on mourning and the gifts of generosity she left in her wake. “We did our best to live in the moment until we had no more moments left,” he wrote for The New York Times. “Amy continues to open doors for me, to affect my choices, to send me off into the world to make the most of it. Recently I gave a TED Talk on the end of life and my grieving process that I hope will help others.” (Watch Rosenthal’s TED Talk.)

Why we need to change our perceptions of teenagers. Neurologist Sarah-Jayne Blakemore urges us to reconsider the way we understand and treat teenagers, especially in school settings. (She wrote a book about the secret life of the teenage brain in March.) According to the latest research, teenagers shed 17% of their grey matter in the prefrontal cortex between childhood and adulthood, which, as Blakemore says, explains that traditional “bad” behaviors like sleeping in late and moodiness are a result of cognitive changes, not laziness or abrasiveness. (Watch Blakemore’s TED Talk.)

Half empty or half full? Research by Dan Gilbert indicates that our decisions may be more faulty than we think — and that we may be predisposed to seeing problems even when they aren’t there. In a recent paper Gilbert co-authored, researchers found that our judgment doesn’t follow fixed rules, but rather, our decisions are more relative. In one experiment, participants were asked to look at dots along a color spectrum from blue to purple, and note which dots were blue; at first, the dots were shown in equal measure, but when blue dots were shown less frequently, participants began marking dots they previously considered purple as blue (this video does a good job explaing). In another experiment, participants were more likely to mark ethical papers as unethical, and nonthreatening faces as threatening, when the previously-set negative stimulus was shown less frequently. This behavior — dubbed “prevalence-induced concept change” — has broad implications; the paper suggests it may explain why social problems never seem to go away, regardless of how much work we do to fix them. (Watch Gilbert’s TED Talk).

Terrifying insights from the world of parasites. Ed Yong likes to write about the creepy and uncanny of the natural world. In his latest piece for The Atlantic, Yong offered a deeper view into the bizarre habits and powers of parasitic worms. Based on research by Nicolle Demandt and Benedikt Saus from the University of Munster, Yong described how some tapeworms capitalize on the way fish shoals guide and react to each other’s behaviors and movements. Studying stickleback fish, Demandt and Saus realized parasite-informed decisions of infected sticklebacks can influence the behavior of uninfected fish, too. This means that if enough infected fish are led to dangerous situations by the controlling powers of the tapeworms, uninfected fish will be impacted by those decisions — without ever being infected themselves. (Read more of Yong’s work and watch his TED Talk.)

A new documentary on corruption within West African football. Ghanaian investigative journalist Anas Aremeyaw Anas joined forces with BBC Africa to produce an illuminating and hard-hitting documentary exposing fraud and corruption in West Africa’s football industry. In an investigation spanning two years, almost 100 officials were recorded accepting cash “gifts” from a slew of undercover reporters from Anas’ team posing as business people and investors. The documentary has already sent shock-waves throughout Ghana — including FIFA bans and resignations from football officials across the country. (Watch the full documentary and Anas’ TED Talk.)

 

Worse Than FailureCodeSOD: An Eventful Career Continues

You may remember Sandra from her rather inglorious start at Initrovent. She didn't intend to continue working for Karl for very long, but she also didn't run out the door screaming. Perhaps she should have, but if she had- we wouldn't have this code.

Initrovent was an event-planning company, and thus needed to manage events, shows, and spaces. They wrote their own exotic suite of software to manage that task.

This code predates their current source control system, and thus it lacks any way to blame the person responsible. Karl, however, was happy to point out that he used to do Sandra's job, and he knew a thing or two about programming. "My fingerprints are on pretty much every line of code," he was proud to say.

if($showType == 'unassigned' || $showType == 'unassigned' || $showType == 'new') { ... }

For a taster, here's one that just leaves me puzzling. Were it a long list, I could more easily see how the same value might appear multiple times. A thirty line conditional would be more of a WTF, but I can at least understand it. There are only three options, two of them are duplicates, and they're right next to each other.

What if you wanted to conditionally enable debugging messages. Try this approach on for size.

foreach($current_open as $key => $value) { if ($value['HostOrganization']['ticket_reference'] == '400220') { //debug($value); } }

What a lovely use of magic numbers. I also like the mix of PascalCase and snake_case keys. But see, if there's any unfilled reservation for a ticket reference number of 400220, we'll print out a debugging message… if the debug statement isn't commented out, anyway.

With that in mind, let's think about a real-world problem. For a certain set of events, you don't want to send emails to the client. The planner wants to send those emails manually. Who knows why? It doesn't matter. This would be a trivial task, yes? Simply chuck a flag on the database table- manual_emails and add a code branch. You could do that, yes, but remember how we controlled the printing of debugging messages before. You know how they actually did this:

$hackSkipEventIds = array('55084514-0864-46b6-95aa-6748525ee4db'); if (in_array($eventId, $hackSkipEventIds)) { // Before we implement #<redacted>, we prefer to skip all roommate // notifications in certain events, and just let the planner send // manual emails. return; }

Look how extensible this solution is- if you ever need to disable emails for more events, you can just extend this array. There's no need to add a UI or anything!

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Don MartiWorse is better, again?

Are there parallels between the rise of Worse Is Better in software and the success of the "uncreative counterrevolution" in advertising? (for more on that second one: John Hegarty: Creativity is receding from marketing and data is to blame) The winning strategy in software is to sacrifice consistency and correctness for simplicity. (probably because of network effects, principal-agent problems, and market failures.) And it seems like advertising has similar trade-offs between

  • Signal

  • Measurability (How well can we measure this project's effect on sales?)

  • Message (Is it persuasive and on brand?)

Just as it's rational for software decision-makers to choose simplicity, it can be rational for marketing decsion-makers to choose measurability over signal and message. (This is probably why there is a brand crisis going on—short-term CMOs are better off when they choose brand-unsafe tactics, sacrificing Message.)

As we're now figuring out how to use market-based tools to fix market failures in software, where can we use better market design to fix market failures in advertising? Maybe this is where it actually makes sense to use #blockchain: give people whose decisions can affect #brandEquity some kind of #skinInTheGame?

Against privacy defeatism: why browsers can still stop fingerprinting

How to get away with financial fraud

Google invests $22M in feature phone operating system KaiOS

Inside the investor revolt that’s trying to take down Mark Zuckerberg

Ryan Wallman: Marketers must loosen their grip on the creative process

Open source sustainability

K2’s Media Transparency Report Still Rocks The Ad Industry Two Years After Its Release

Mark Ritson: How ‘influencers’ made my arse a work of art

Ad fraud one of the most profitable criminal enterprises in the world, researcher says

Cover story: Adtech won’t fix ad fraud because it is too lucrative, say specialists

https://hackernoon.com/why-funding-open-source-is-hard-652b7055569d

Sir John Hegarty: Great advertising elevates brands to a part of culture

https://www.canvas8.com/blog/2018/ju/behavioural-science-insights-nudgestock-2018.html …

,

Valerie AuroraBryan Cantrill has been accused of verbal abuse by at least seven people

It sounds like Bryan Cantrill is thinking about organizing another computer conference. When he did that in 2016, I wrote a blog post about why I wouldn’t attend, because, based on my experience as Bryan’s former co-worker, I believed that Bryan Cantrill would probably say cruel and humiliating things to people who attended.

I understand that some people still supported Bryan and his conference after they read that post. After all, Bryan is so intelligent and funny and accomplished, and it’s a “he said, she said” situation, and if you can’t take the heat get out of the kitchen, etc. etc.

What’s changed since then? Well, at least six other people spoke up publicly about their own experiences with Bryan, many of which seem worse than mine. Then #metoo happened and we learned how many people a powerful person can abuse before any of their victims speak up, and why they stay quiet: worry about their careers being destroyed, being bankrupted by a lawsuit, or being called a liar and worse. If you’re still supporting Bryan, I invite you to read this story about Jeffrey Tambor verbally abusing Jessica Walter on the set of Arrested Development, and re-examine why you are supporting someone who has been verbally abusive to so many people.

Here are six short quotes from other people speaking about their experiences with Bryan Cantrill:

Having been a Joyent ‘customer’ and working to porting an application to run on SmartOS was like being a personal punching bag for Bryan.”

I worked at Joyent from 2010 through 2013. Valerie’s experience comports with my own. This warning is brave and wise.”

All that you say is true, and if anything, toned down from reality. Bryan is a truly horrible human being.”

I know for sure Bryan’s behavior prevented or at the very least delayed other developers from reaching their potential in the kernel group. Unfortunately the lack of moral and ethical leadership in Solaris allowed this to go on for far too long.”

Sun was such a toxic environment for so many people and it is very brave of you to share your experience. After six years in this oppressive environment, my confidence was all but destroyed.”

Having known Bryan from the days of being a junior engineer…he has always been a narcissistic f_ck that proudly leaves a wake of destruction rising up on the carcasses of his perceived foes (real and imagined). His brilliance comes at too high of a cost.”

This is what six people are willing to say publicly about how Bryan treated them. If you think that isn’t a lot, please take the time to read more about #metoo and consider how Bryan’s position of power would discourage people from coming forward with their stories of verbal abuse. If you do believe that Bryan has abused these people, consider what message you are sending to others by continuing to follow him on social media or otherwise validating his behavior.


If you have been abused by Bryan, I have a request: please do not contact me to tell me your story privately, unless you want help making your story public in some way. I’m exhausted and it doesn’t do any good to tell me—I’m already convinced he’s awful. Here’s what I can say: There are dozens of you, and you have remarkably similar stories.

I’ll be heavily moderating comments on this post and in particular won’t approve anything criticizing victims of abuse for speaking up. If your comment gets stuck in the spam filter, please email me at valerie.aurora@gmail.com and I’ll post it for you.

,

CryptogramFriday Squid Blogging: Fried Squid with Turmeric

Good-looking recipe.

As usual, you can also use this squid post to talk about the security stories in the news that I haven't covered.

Read my blog posting guidelines here.

CryptogramConservation of Threat

Here's some interesting research about how we perceive threats. Basically, as the environment becomes safer we basically manufacture new threats. From an essay about the research:

To study how concepts change when they become less common, we brought volunteers into our laboratory and gave them a simple task ­-- to look at a series of computer-generated faces and decide which ones seem "threatening." The faces had been carefully designed by researchers to range from very intimidating to very harmless.

As we showed people fewer and fewer threatening faces over time, we found that they expanded their definition of "threatening" to include a wider range of faces. In other words, when they ran out of threatening faces to find, they started calling faces threatening that they used to call harmless. Rather than being a consistent category, what people considered "threats" depended on how many threats they had seen lately.

This has a lot of implications in security systems where humans have to make judgments about threat and risk: TSA agents, police noticing "suspicious" activities, "see something say something" campaigns, and so on.

The academic paper.

Worse Than FailureError'd: Testing English in Production

Philip G. writes, "I found this gem when I was on the 'Windows USB/DVD Download Tool' page (yes, I know Rufus is better) and I decided to increment the number in the URL."

 

"Using a snowman emoji as a delimiter...yeah, I guess you could do that," writes George.

 

Seb wrote, "These signup incentives are just a little too variable for my tastes..."

 

"Wow. Vodafone UK really isn't selling the battery life of the Samsung Galaxy J3...or maybe they're just being honest?" Steve M. writes.

 

"Nice to see the Acer website here in South Africa being up front about their attempts at upselling," wrote Gabriel S.

 

"Thank you $wargaming_company_title$ for your friendly notice, I'll spend my $wot_gold_amount$ in $wot_gold_suggestion$.", Tassu writes.

 

[Advertisement] Ensure your software is built only once and then deployed consistently across environments, by packaging your applications and components. Learn how today!

Google AdsenseAdSense now understands Telugu

Today, we’re excited to announce the addition of Telugu, a language spoken by over 70 million in India and many other countries around the world, to the family of AdSense supported languages. With this launch, publishers can now monetize their Telugu content and advertisers can connect to a Telugu speaking audience with relevant ads.

To start monetizing your Telugu content website with Google AdSense:

Check the AdSense program policies and make sure your website is compliant.
Sign up for an AdSense account.
Add the AdSense code to start displaying relevant ads to your users.

Welcome to AdSense! Sign up now.


Posted by:
The AdSense Internationalization Team

Rondam RamblingsI have no words

So I'll let Lili Loofbourow speak for me.

,

Krebs on SecurityPlant Your Flag, Mark Your Territory

Many people, particularly older folks, proudly declare they avoid using the Web to manage various accounts tied to their personal and financial data -- from utilities and mobile phones to retirement benefits and online banking services. The reasoning behind this strategy is as simple as it is alluring: What's not put online can't be hacked. But increasingly, adherents to this mantra are finding out the hard way that if you don't plant your flag online, fraudsters and identity thieves may do it for you.

TEDAn ambitious plan to explore our oceans, and more news from TED speakers

 

The past few weeks have brimmed over with TED-related news. Below, some highlights.

Exploring the ocean like never before. A school of ocean-loving TED speakers have teamed up to launch OceanX, an international initiative dedicated to discovering more of our oceans in an effort to “inspire a human connection to the sea.” The coalition is supported by Bridgewater Capital’s Ray Dalio, along with luminaries like ocean explorer Sylvia Earle and filmmaker James Cameron, and partners such as BBC Studios, the American Museum of Natural History and the National Geographic Society. The coalition is now looking for ideas for scientific research missions in 2019, exploring the Norwegian Sea and the Indian Ocean. Dalio’s son Mark leads the media arm of the venture; from virtual reality demonstrations in classrooms to film and TV releases like the BBC show Blue Planet II and its follow-up film Oceans: Our Blue Planet, OceanX plans to build an engaged global community that seeks to “enjoy, understand and protect our oceans.” (Watch Dalio’s TED Talk, Earle’s TED Talk and Cameron’s TED Talk.)

The Ebola vaccine that’s saving lives. In response to the recent Ebola outbreak in the Democratic Republic of the Congo, GAVI — the Vaccine Alliance, led by Seth Berkeley — has deployed thousands of experimental vaccines in an outbreak control strategy. The vaccines were produced as part of a partnership between GAVI and Merck, a pharmaceutical company, committed to proactively developing and producing vaccines in case of a future Ebola epidemic. In his TED Talk, Berkeley spoke of the drastic dangers of global disease and the preventative measures necessary to ensure we are prepared for future outbreaks. (Watch his TED Talk and read our in-depth interview with Berkeley.)

A fascinating new study on the halo effect. Does knowing someone’s political leanings change how you gauge their skills? Cognitive neurologist Tali Sharot and lawyer Cass R. Sunstein shared insights from their latest research answering the question in The New York Times. Alongside a team from University College London and Harvard Law School, Sharot conducted an experiment testing whether knowing someone’s political leanings affected how we would engage and trust in other non-political aspects of their lives. The study found that people were more willing to trust someone who had the same political beliefs as them — even in completely unrelated fields, like dentistry or architecture. These findings have wide-reaching implications and can further our understanding of the social and political landscape. (Watch Sharot’s TED Talk on optimism bias).

A new essay anthology on rape culture. Roxane Gay’s newest book, Not That Bad: Dispatches from Rape Culture, was released in May to critical and commercial acclaim. The essay collection, edited and introduced by Gay, features first-person narratives on the realities and effects of harassment, assault and rape. With essays from 29 contributors, including actors Gabrielle Union and Amy Jo Burns, and writers Claire Schwartz and Lynn Melnick, Not That Bad offers feminist insights into the national and global dialogue on sexual violence. (Watch Gay’s TED Talk.)

One million pairs of 3D-printed sneakers. At TED2015, Carbon founder and CEO Joseph DeSimone displayed the latest 3D printing technology, explaining its seemingly endless applications for reshaping the future of manufacturing. Now, Carbon has partnered with Adidas for a bold new vision to 3D-print 100,000 pairs of sneakers by the end of 2018, with plans to ramp up production to millions. The company’s “Digital Light Synthesis” technique, which uses light and oxygen to fabricate materials from pools of resin, significantly streamlines manufacturing from traditional 3D-printing processes — a technology Adidas considers “revolutionary.” (Watch DeSimone’s TED Talk.)

CryptogramManipulative Social Media Practices

The Norwegian Consumer Council just published an excellent report on the deceptive practices tech companies use to trick people into giving up their privacy.

From the executive summary:

Facebook and Google have privacy intrusive defaults, where users who want the privacy friendly option have to go through a significantly longer process. They even obscure some of these settings so that the user cannot know that the more privacy intrusive option was preselected.

The popups from Facebook, Google and Windows 10 have design, symbols and wording that nudge users away from the privacy friendly choices. Choices are worded to compel users to make certain choices, while key information is omitted or downplayed. None of them lets the user freely postpone decisions. Also, Facebook and Google threaten users with loss of functionality or deletion of the user account if the user does not choose the privacy intrusive option.

[...]

The combination of privacy intrusive defaults and the use of dark patterns, nudge users of Facebook and Google, and to a lesser degree Windows 10, toward the least privacy friendly options to a degree that we consider unethical. We question whether this is in accordance with the principles of data protection by default and data protection by design, and if consent given under these circumstances can be said to be explicit, informed and freely given.

I am a big fan of the Norwegian Consumer Council. They've published some excellent research.

Worse Than FailureCodeSOD: Foggy about Security

Maverick StClare’s company recently adopted a new, SaaS solution for resource planning. Like most such solutions, it was pushed from above without regard to how people actually worked, and thus required the users to enter highly structured data into free-form, validation-free, text fields. That was dumb, so someone asked Maverick: “Hey, could you maybe write a program to enter the data for us?”

Well, you’ll be shocked to learn that there was no API, but the web pages themselves all looked pretty simple and the design implied they hadn’t changed since IE4, so Maverick decided to take a crack at writing a scraper. Step one: log in. Easy, right? Maverick fired up a trace on the HTTPS traffic and sniffed the requests. He was happy to see that his password wasn’t sent in plain text. He was less happy to see that it wasn’t sent using any of the standard HTTP authentication mechanisms, and it certainly wasn’t hashed using any algorithm he recognized. He dug into the code, and found this:

function Foggy(svInput)
{
  // Any changes must be duplicated in the server-side version of this function.
  var svOutput = "";
  var ivRnd;
  var i;
  var ivLength = svInput.length;

  if (ivLength == 0 || ivLength > 158)
  {
        svInput = svInput.replace(/"/g,"&qt;");
        return svInput;
  }

  for (i = 0; i < ivLength; i++)
  {
        ivRnd = Math.floor(Math.random() * 3);
        if (svInput.charCodeAt(i) == 32 || svInput.charCodeAt(i) == 34 || svInput.charCodeAt(i) == 62)
        {
          ivRnd = 1;
        }
        if (svInput.charCodeAt(i) == 33 || svInput.charCodeAt(i) == 58 || svInput.charCodeAt(i) == 59 || svInput.charCodeAt(i) + ivRnd > 255)
        {
          ivRnd = 0;
        }
        svOutput += String.fromCharCode(ivRnd+97);
        svOutput += String.fromCharCode(svInput.charCodeAt(i)+ivRnd);
  }

  for (i = 0; i < Math.floor(Math.random() * 8) + 8; i++)
  {
        ivRnd = Math.floor(Math.random() * 26);
        svOutput += String.fromCharCode(ivRnd+97);
  }

  svOutput += String.fromCharCode(svInput.length + 96);
  return svOutput;
}

I… have so many questions. Why do they only replace quotes if the string is empty or greater than 158 characters? Why are there random numbers involved in their “hashing” algorithm? I’m foggy about this whole thing, indeed. And ah, protip: security through obscurity works better when nobody can see how you obfuscated things. All I can say is: “aWcjaacvc0b!cVahcgc0b!cHaubdcmb/gmzyrcoqhp”.

[Advertisement] Ensure your software is built only once and then deployed consistently across environments, by packaging your applications and components. Learn how today!

Mark ShuttleworthFraud alert – scams using my name and picture

I have recently become aware of a fraudulent investment scam which falsely states that I have launched new software known as a QProfit System promoted by Jerry Douglas. I’ve seen some phishing sites like http://www.bbc-tech.news and http://pipeline-stats.club, and pop up ads on Facebook like this one:

I can’t comment on whether or not Jerry Douglas promotes a QProfit system and whether or not it’s fraud. But I can tell you categorically that there are many scams like this, and that this investment has absolutely nothing to do with me. I haven’t developed this software and I have no desire to defraud the South African government or anyone else. I’m doing what I can to get the fraudulent sites taken down. But please take heed and don’t fall for these scams.

,

CryptogramIEEE Statement on Strong Encryption vs. Backdoors

The IEEE came out in favor of strong encryption:

IEEE supports the use of unfettered strong encryption to protect confidentiality and integrity of data and communications. We oppose efforts by governments to restrict the use of strong encryption and/or to mandate exceptional access mechanisms such as "backdoors" or "key escrow schemes" in order to facilitate government access to encrypted data. Governments have legitimate law enforcement and national security interests. IEEE believes that mandating the intentional creation of backdoors or escrow schemes -- no matter how well intentioned -- does not serve those interests well and will lead to the creation of vulnerabilities that would result in unforeseen effects as well as some predictable negative consequences

The full statement is here.

Worse Than FailureRepresentative Line: Got Your Number

You have a string. It contains numbers. You want to turn those numbers into all “0”s, presumably to anonymize them. You’re also an utter incompetent. What do you do?

You already know what they do. Jane’s co-worker encountered this solution, and she tells us that the language was “Visual BASIC, Profanity”.

Private Function ReplaceNumbersWithZeros(ByVal strText As String) As String
     ReplaceNumbersWithZeros = Replace(Replace(Replace(Replace(Replace(Replace(Replace(Replace(Replace(strText, "1", "0"), "2", "0"), "3", "0"), "4", "0"), "5", "0"), "6", "0"), "7", "0"), "8", "0"), "9", "0")
End Function

Jane adds:

My co-worker found this function while researching some legacy code. Shortly after this discovery, it took us 15 minutes to talk him down off the ledge…and we’re on the ground floor.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

,

Krebs on SecurityHow to Avoid Card Skimmers at the Pump

Previous stories here on the proliferation of card-skimming devices hidden inside fuel pumps have offered a multitude of security tips for readers looking to minimize their chances of becoming the next victim, such as favoring filling stations that use security cameras and tamper-evident tape on their pumps. But according to police in San Antonio, Texas, there are far more reliable ways to avoid getting skimmed at a fuel station.

Sociological ImagesThe Half-Dozen Headline

Want to help fight fake news and manage political panics? We have to learn to talk about numbers.

While teaching basic statistics to sociology undergraduates, one of the biggest trends I noticed was students who thought they hated math experiencing a brain shutdown when it was time to interpret their results. I felt the same way when I started in this field, and so I am a big advocate for working hard to bridge the gap between numeracy and literacy. You don’t have to be a statistical wizard to make your reporting clear to readers.

Sociology is a great field to do this, because we are used to going out into the world and finding all kinds of cultural tropes (like pointlessly gendered products!). My new favorite trope is the Half-Dozen Headline. You can spot them in the wild, or through Google News with a search for “half dozen.” Every time I read one of these headlines, my brain echoes with “half of a dozen is six.”

Sometimes, six is a lot:

Sometimes, six is not:

(at least, not relative to past administrations)

Sometimes, well, we just don’t know:

Is this five deaths (nearly six)? Is a rate of about two deaths a year in a Walmart parking lot high? If people already struggle to interpret raw numbers, wrapping your findings in fuzzy language only makes the problem worse.

Spotting Half-Dozen Headlines is a great introductory exercise for classes in social statistics, public policy, journalism, or other fields that use applied data analysis. If you find a favorite Half-Dozen Headline, be sure to send it our way!

Evan Stewart is a Ph.D. candidate in sociology at the University of Minnesota. You can follow him on Twitter.

(View original at https://thesocietypages.org/socimages)

CryptogramBypassing Passcodes in iOS

Last week, a story was going around explaining how to brute-force an iOS password. Basically, the trick was to plug the phone into an external keyboard and trying every PIN at once:

We reported Friday on Hickey's findings, which claimed to be able to send all combinations of a user's possible passcode in one go, by enumerating each code from 0000 to 9999, and concatenating the results in one string with no spaces. He explained that because this doesn't give the software any breaks, the keyboard input routine takes priority over the device's data-erasing feature.

I didn't write about it, because it seemed too good to be true. A few days later, Apple pushed back on the findings -- and it seems that it doesn't work.

This isn't to say that no one can break into an iPhone. We know that companies like Cellebrite and Grayshift are renting/selling iPhone unlock tools to law enforcement -- which means governments and criminals can do the same thing -- and that Apple is releasing a new feature called "restricted mode" that may make those hacks obsolete.

Grayshift is claiming that its technology will still work.

Former Apple security engineer Braden Thomas, who now works for a company called Grayshift, warned customers who had bought his GrayKey iPhone unlocking tool that iOS 11.3 would make it a bit harder for cops to get evidence and data out of seized iPhones. A change in the beta didn't break GrayKey, but would require cops to use GrayKey on phones within a week of them being last unlocked.

"Starting with iOS 11.3, iOS saves the last time a device has been unlocked (either with biometrics or passcode) or was connected to an accessory or computer. If a full seven days (168 hours) elapse [sic] since the last time iOS saved one of these events, the Lightning port is entirely disabled," Thomas wrote in a blog post published in a customer-only portal, which Motherboard obtained. "You cannot use it to sync or to connect to accessories. It is basically just a charging port at this point. This is termed USB Restricted Mode and it affects all devices that support iOS 11.3."

Whether that's real or marketing, we don't know.

TEDApply now to be a TED2019 Fellow

The TED Fellows program is turning ten years old next year, and we are looking for our most ambitious class yet. We select people from every discipline and every country to be Fellows, and we give them support to scale their dreams and scale their impact.

Apply to be a TED Fellow by August 26.

Who are TED Fellows? Fellows are individuals with original work, a record of achievement in their field and exceptional potential. They are also courageous, collaborative people dedicated to improving life where they work.

How do we help you dream bigger? The Fellows program is robust, long-term and, we think, unlike any other Fellowship out there. From our open application process to our rigorous support systems, we have designed a program that maximizes innovation and collaboration.

Fellows get career coaching and speaker training as well as mentorship and public relations guidance. Fellows also give a talk at a TED Conference, a huge opportunity to share their work with a wide, new audience. And perhaps most important, Fellows join the community of 450+ other Fellows who inspire one another and collaborate on new projects.

What have Fellows done after joining the program? In our nearly 10-year history, the Fellows program has sparked remarkable cultural change and reached millions of people. With the support of TED, Fellows have conserved large swaths of our planet, protecting many species in the process. They’ve made headway in understanding complex diseases like Parkinson’s, cancer and malaria. They’ve created art that shines a light on injustice and made music that celebrates our history. They’ve made huge strides in robotics and 4-D printing and launched new startups. They’ve passed laws and have gone on to win Oscars, Grammys and MacArthur “genius” grants. And in the process, Fellows have improved conditions on our planet for countless communities and inspired others to pursue their own unconventional projects.

Our application is straightforward. It’s open to everyone (no one is appointed a Fellow; everyone has to apply), and we encourage you to apply even if you’re not sure you’re qualified. We have a way of picking winners before they know it.

The online application can take as little as 20 minutes. It asks for general biographical information, short essays on your work and three references. We don’t have an upper age limit, but you must be 18 or older to apply. If you’re selected, you will be part of our 10-year anniversary class, and you will need to reserve April 13 through April 20, 2019, for TED2019 and our own very special pre-conference.

So dream bigger. Apply to be a TED Fellow today.

For more information on the TED Fellows:

Visit: ted.com/fellows

Follow: @TEDFellow

Like: facebook.com/TEDFellow

Read: fellowsblog.ted.com

TED12 books from favorite TEDWomen speakers, for your summer reading list

We all have a story to tell. And in my work as curator of the TEDWomen conference, I’ve had the pleasure of providing a platform to some of the best stories and storytellers out there. Beyond their TED Talk, of course, many TEDWomen speakers are also accomplished authors — and if you liked them on the TED stage, odds are you will enjoy spending more time with them in the pages of their books.

All of the women and men listed here have given talks at TEDWomen, though some talks are related to their books and some aren’t. See what connects with you and enjoy your summer!

luvvie-ajayi-im-judging-you-cover.jpg

Luvvie Ajayi‘s 2017 TEDWomen talk has already amassed over 2.2 million views online! In it, she talks about how she wants to leave this world better than she found it and in order to do that, she says we all have to get more comfortable saying the sometimes uncomfortable things that need to be said. What’s great about Luvvie is that she delivers her commentary with a sly side eye that pokes fun at everyone, including herself.

In her book, I’m Judging You: The Do-Better Manual — written in the form of an Emily Post-type guidebook for modern manners — Luvvie doles out criticism and advice with equal amounts of wit, charm and humor that’s often laugh-out-loud funny. As Shonda Rhimes noted in her review, “This truth-riot of a book gives us everything from hilarious lectures on the bad behavior all around us to razor sharp essays on media and culture. With I’m Judging You, Luvvie brilliantly puts the world on notice that she is not here for your foolishness — or mine.”

madeleine-albright-fascism.jpg

At the first TEDWomen in 2010, Madeleine Albright talked to me about what it was like to be a woman and a diplomat. In her new book, entitled Fascism: A Warning, the former secretary of state writes about the history of fascism and the clash that took place between two ideologies of governing: fascism and democracy. She argues that “fascism not only endured the 20th century, but now presents a more virulent threat to peace and justice than at any time since the end of World War II.”

“At a moment when the question ‘Is this how it begins?’ haunts Western democracies,” the Economist notes in its review, “[Albright] writes with rare authority.”

gretchen-carlson-be-fierce-cover.jpg

Sometimes a talk perfectly captures the zeitgeist, and that was the case with Gretchen Carlson last November at TEDWomen. At the time, the #MeToo movement founded in 2007 by Tarana Burke was seeing a huge surge online, thanks to signal-boosting from Alyssa Milano and more women with stories to share.

Carlson took to the stage to talk about her personal experience with sexual harassment at Fox News, her historic lawsuit and the lessons she’d learned and related in her just-released book, Be Fierce. In her talk, she identifies three specific things we can all do to create safer places to work. “We will no longer be underestimated, intimidated or set back,” Carlson says. “We will stand up and speak up and have our voices heard. We will be the women we were meant to be.” In her book, she writes in detail about how we can stop harassment and take our power back.

john-cary-design-for-good-cover.jpg

John Cary is an architect who thinks deeply about diversity in design — and how the field’s lack of diversity leads to thoughtless, compassionless spaces in the modern world. As he said in his 2017 TEDWomen talk, “well-designed spaces are not just a matter of taste or a questions of aesthetics. They literally shape our ideas about who we are in the world and what we deserve.”

For years, as the executive director of Public Architecture, John has advocated for the term “public interest design” to become part of the architect’s lexicon, in much the same way as it is in fields like law and health care. In his new book, Design for Good, John presents 20 building projects from around the world that exemplify how good design can improve communities, the environment, and the lives of the people who live with it.

brittney-cooper-eloquent-rage-cover.jpg

In her thought-provoking 2016 TEDWomen talk, professor Brittney Cooper examined racism through the lens of time — showing how moments of joy, connection and well-being had been lost to people of color because of delays in social progress.

Last summer, I recommended Brittney’s book on the lives and thoughts of intellectual Black women in history who had been left out of textbooks. And this year, Brittney is back with another book, one that is more personal and also very timely in this election year in which women are figuring out what a truly intersectional feminist movement looks like.

As my friend Jane Fonda wrote in a recent blog post, in order to build truly multi-racial coalitions, white people need to do the work to truly understand race and racism. For white feminists in particular, the work starts by listening to the perspectives of women of color. Brittney’s book, Eloquent Rage: A Black Feminist Discovers Her Superpower, offers just that opportunity. Brittney’s sharp observations from high school (at a predominantly white school), college (at Howard University) and as a 30-something professional make the political personal. As she told the Washington Post, “When we figure out politics at a personal level, then perhaps it wouldn’t be so hard to figure it out at the more structural level.”

susan-david-emotional-agility-cover.jpeg

Susan David is a Harvard Medical School psychologist who studies how we process our emotions. In a deeply moving talk at TEDWomen 2017, Susan suggested that the way we deal with our emotions shapes everything that matters: our actions, careers, relationships, health and happiness. “I’m not anti-happiness. I like being happy. I’m a pretty happy person,” she says. “But when we push aside normal emotions to embrace false positivity, we lose our capacity to develop skills to deal with the world as it is, not as we wish it to be.”

In her book, Emotional Agility, Susan shares strategies for the radical acceptance of all of our emotions. How do we not let our self-doubts, failings, shame, fear, or anger hold us back?

“We own our emotions,” she says. “They don’t own us.”

all-the-women-in-my-family-sing-cover.jpg

Dr. Musimbi Kanyoro is president and CEO of Global Fund for Women, one of the world’s leading publicly supported foundations for gender equality. In her TEDWomen talk last year, she introduced us to the Maragoli concept of “isirika” — a pragmatic way of life that embraces the mutual responsibility to care for one another — something she sees women practicing all over the world.

In All the Women in My Family Sing, Musimbi is one of 69 women of color who have contributed prose and poetry to this “moving anthology” that “illuminates the struggles, traditions, and life views of women at the dawn of the 21st century. The authors grapple with identity, belonging, self-esteem, and sexuality, among other topics.” Contributors range in age from 16 to 77 and represent African-American, Native American, Asian-American, Muslim, Cameroonian, Kenyan, Liberian, Mexican-American, Korean, Chinese-American and LGBTQI experiences.

anjali-kumar-book-cover.jpg

In her 2017 TEDWomen talk, author Anjali Kumar shared some of what she learned in researching her new book, Stalking God: My Unorthodox Search for Something to Believe In. A few years ago, Anjali — a pragmatic lawyer for Google who, like more than 56 million of her fellow Americans, describes herself as not religious — set off on a mission to find God.

Spoiler alert: She failed. But along the way, she learned a lot about spirituality, humanity and what binds us all together as human beings.

In her humorous and thoughtful book, Anjali writes about her search for answers to life’s most fundamental questions and finding a path to spirituality in our fragmented world. The good news is that we have a lot more in common than we might think.

peggy-orenstein-dont-call-me-princess-cover.jpg

New York Times best-selling author Peggy Orenstein is out with a new collection of essays titled Don’t Call Me Princess: Girls, Women, Sex and Life. Peggy combines a unique blend of investigative reporting, personal revelation and unexpected humor in her many books, including Schoolgirls and the book that was the subject of her 2016 TEDWomen talk, Girls & Sex.

Don’t Call Me Princess “offers a crucial evaluation of where we stand today as women — in our work lives, sex lives, as mothers, as partners — illuminating both how far we’ve come and how far we still have to go.” Don’t miss it.

caroline-paul-you-are-mighty-cover.jpg

Caroline Paul began her remarkable career as the first female firefighter in San Francisco. She wrote about that in her first book, Fighting Fires. In the 20 years since, she’s written many more books, including her most recent, You Are Mighty: A Guide to Changing the World.

This well-timed book offers advice and inspiration to young activists. She writes about the experiences of young people — from famous kids like Malala Yousafzai and Claudette Colvin to everyday kids — who stood up for what they thought was right and made a difference in their communities. Paul offers loads of tactics for young people to use in their own activism — and proves you’re never too young to change the world.

cleo-wade-heart-talk-cover.png

I first encountered Cleo Wade‘s delightful, heartfelt words of wisdom like most people, on Instagram. Cleo has over 350,000 followers on her popular feed that features short poems, bits of wisdom and pics. Cleo has been called the poet of her generationeverybody’s BFF and the millennial Oprah. In her new poetry collection, Heart Talk: Poetic Wisdom for a Better Life, the poet, artist and activist shares some of the Instagram notes she wrote “while sitting in her apartment, poems about loving, being and healing” and “the type of good ol’-fashioned heartfelt advice I would share with you if we were sitting in my home at my kitchen table.”

girl-who-smiled-beads-clementine.jpg

In 1994, the Rwandan Civil War forced six-year-old Clemantine Wamariya and her fifteen-year-old sister from their home in Kigali, leaving their parents and everything they knew behind. In her 2017 TEDWomen talk, Clemantine shared some of her experiences over the next six years growing up while living in refugee camps and migrating through seven African countries.

In her new memoir, The Girl Who Smiled Beads: A Story of War and What Comes After, Clemantine recounts her harrowing story of hunger, imprisonment, and not knowing whether her parents were alive or dead. At the age of 12, she moved to Chicago and was raised in part by an American family. It’s an incredible, poignant story and one that is so important during this time when many are denying the humanity of people who are victims of war and civil unrest. For her part, Clemantine remains hopeful. “There are a lot of great people everywhere,” she told the Washington Post. “And there are also a lot of not-so-great people. It’s all over the world. But when we stepped out of the airplane, we had people waiting for us — smiling, saying, ‘Welcome to America.’ People were happy. Many countries were not happy to have us. Right now there are people at the airport still holding those banners.”

TEDWOMEN 2018

I also want to mention that registration for TEDWomen 2018 is open now! Space is limited and I don’t want you to miss out. This year, TEDWomen will be held Nov. 28–30 in Palm Springs, California. The theme is Showing Up.

The time for silent acceptance of the status quo is over. Women around the world are taking matters into their own hands, showing up for each other and themselves to shape the future we all want to see.We’ll explore the many aspects of this year’s theme through curated TED Talks, community dinners and activities.

Join us!

— Pat

Worse Than FailureCodeSOD: External SQL

"Externalize your strings" is generally good advice. Maybe you pull them up into constants, maybe you move them into a resource file, but putting a barrier between your code and the strings you output makes everything more flexible.

But what about strings that aren't output? Things like, oh… database queries? We want to be cautious about embedding SQL directly into our application code, but our SQL code often is our business logic, so it makes sense to inline it. Most data access layers end up trying to abstract the details of SQL behind method calls, whether it's just a simple repository or an advanced ORM approach.

Sean found a… unique approach to resolving this tension in some Java code he inherited. He saw lots of references to keys in a hash-map, keys like user or pw or insert_account_table or select_all_transaction_table. But where did these keys get defined?

Like all good strings, they were externalized into a file called sql.txt. A simple regex-based parser loaded the data and created the dictionary. Now, any module which wanted to query the database had a map of any query they could possibly want to run. Just chuck 'em into a PreparedStatement object and you're ready to go.

Here, in its entirety, is the sql.txt file.

user = root
pw = password
db_name = lrc_mydb

create_account_table = create table if not exists account_table(username varchar(45) not null, password text not null, last_name text, first_name text, mid_name text, suffix_name text, primary key (username))
create_course_table = create table if not exists course_table (course_abbr char(45) not null unique, course_name text, primary key(course_abbr))
create_student_table = create table if not exists student_table (username varchar(45) not null, registration_date date, year_lvl char(45), photolink longblob, freetime time, course_abbr char(45) not null, status char(45) not null, balance double not null, foreign key fk_username(username) references account_table(username) on update cascade on delete cascade, foreign key fk_course_abbr(course_abbr) references course_table(course_abbr) on update cascade on delete cascade, primary key(username))
create_admin_table = create table if not exists admin_table (username varchar(45) not null, delete_priv boolean, settle_priv boolean, db_access boolean, foreign key fk_username(username) references account_table(username) on update cascade on delete cascade, primary key(username))
create_reservation_table = create table if not exists reservation_table (username varchar(45) not null, foreign key fk_username(username) references account_table(username) on update cascade on delete cascade, primary key(username))
create_service_table = create table if not exists service_table (service_id int not null auto_increment, service_name text, amount double, page_requirement boolean, primary key (service_id))
create_pc_table = create table if not exists pc_table (pc_id char(45) not null, ip_address varchar(45), primary key (pc_id))
create_transaction_table = create table if not exists transaction_table (transaction_id int not null auto_increment, date_rendered date, amount_paid double unsigned not null,cost_payable double, username varchar(45) not null, service_id int not null, foreign key fk_username(username) references account_table(username) on update cascade on delete cascade, foreign key fk_service_id(service_id) references service_table(service_id) on update cascade on delete cascade, primary key (transaction_id))
create_pc_usage_table = create table if not exists pc_usage_table (transaction_id int not null, pc_id char(45) not null, login_time time, logout_time time, foreign key fk_pc_id(pc_id) references pc_table(pc_id) on update cascade on delete cascade, foreign key fk_transaction_id(transaction_id) references transaction_table(transaction_id) on update cascade on delete cascade, primary key(transaction_id))
create_pasa_hour_table = create table if not exists pasa_hour_table (transaction_id int not null auto_increment, date_rendered date, sender varchar(45) not null, amount_time time, current_free_sender time, deducted_free_sender time, receiver varchar(45) not null, current_free_receiver time, added_free_receiver time, primary key(transaction_id))
create_receipt_table = create table if not exists receipt_table (dates date, receipt_id varchar(45) not null, transaction_id int not null, username varchar(45) not null, amount_paid double, amount_change double, foreign key fk_transaction_id(transaction_id) references transaction_table(transaction_id) on update cascade on delete cascade, foreign key fk_username(username) references account_table(username) on update cascade on delete cascade)
create_cash_flow_table = create table if not exists cash_flow_table (dates date, cash_in double, cash_close double, cash_out double, primary key(dates))
create_free_pc_usage_table = create table if not exists free_pc_usage_table (transaction_id int not null, foreign key fk_transaction_id(transaction_id) references transaction_table(transaction_id) on update cascade on delete cascade, primary key(transaction_id))
create_diagnostic_table = create table if not exists diagnostic_table (sem_id int not null auto_increment , date_start date, date_end date, sem_num enum('first', 'second', 'mid year'), freetime time, time_penalty double, balance_penalty double, primary key(sem_id))
create_pasa_balance_table = create table if not exists pasa_balance_table (transaction_id int not null auto_increment, date_rendered date, sender varchar(45) not null, amount double, current_balance_sender double, deducted_balance_sender double, receiver varchar(45) not null, current_balance_receiver double, added_balance_receiver double, primary key(transaction_id))

insert_account_table = insert into account_table values (?, password(?), ?, ?, ?, ?)
insert_course_table = insert into course_table values (?, ?)
insert_student_table = insert into student_table values (?, now(), ?, ?, ?, ?, ?, ?)
insert_admin_table = insert into admin_table values (?, ?, ?, ?)
insert_reservation_table = insert into reservation_table values (?)
insert_service_table = insert into service_table (service_name, amount, page_requirement) values (?, ?, ?)
insert_pc_table = insert into pc_table values (?, ?)
insert_transaction_table = insert into transaction_table (date_rendered, amount_paid, cost_payable, username, service_id) values (now(), ?, ?, ?, ?)
insert_pc_usage_table = insert into pc_usage_table values (?, ?, ?, ?)
insert_pasa_hour_table = insert into pasa_hour_table (date_rendered, sender, amount_time, current_free_sender, deducted_free_sender, receiver, current_free_receiver, added_free_receiver) values (curdate(), ?, ?, ?, ?, ?, ?, ?)
insert_free_pc_usage_table = insert into free_pc_usage_table values (?)
insert_cash_flow_table = insert into cash_flow_table values (curdate(), ?, ?, ?)
insert_receipt_table = insert into receipt_table values (curdate(), ?, ?, ?, ?, ?)
insert_diagnostic_table = insert into diagnostic_table (date_start, date_end, sem_num, freetime, time_penalty, balance_penalty) values (?, ?, ?, ?, ?, ?)
insert_pasa_balance_table = insert into pasa_balance_table (date_rendered, sender, amount, current_balance_sender, deducted_balance_sender, receiver, current_balance_receiver, added_balance_receiver) values (curdate(), ?, ?, ?, ?, ?, ?, ?)

delete_reservation_table = delete from reservation_table where username = ?
delete_course_table = delete from course_table where course_abbr = ?
delete_user_assoc_to_course = delete account_table, student_table from student_table inner join account_table on account_table.username = student_table.username where student_table.course_abbr = ?
delete_service_table = delete from service_table where service_name = ?
delete_user_student = delete account_table, student_table from student_table inner join account_table on account_table.username = student_table.username where student_table.username = ?
delete_user_staff = delete account_table, admin_table from admin_table inner join account_table on account_table.username = admin_table.username where admin_table.username = ?

select_total_cost = select sum(cost_payable - amount_paid) from transaction_table where username = ? and cost_payable > amount_paid
select_time_penalty = select time_penalty from diagnostic_table where sem_id = ?
select_balance_penalty = select balance_penalty from diagnostic_table where sem_id = ?
select_balance = select balance from student_table where username = ?
select_accountabilities = select sum(cost_payable - amount_paid) from transaction_table where username = ? and cost_payable > amount_paid
select_count_service_table = select count(*) from service_table
select_count_course_table = select count(*) from course_table
select_course_count = select count(course_abbr) from student_table where course_abbr = ?
select_course_abbr = select course_abbr from course_table where course_name = ?
select_degree_name_abbr = select * from course_table
select_service_name = select * from service_table
select_service_name1 = select service_name from service_table where service_id = ?
select_services_amount = select * from service_table
select_username = select * from account_table where username = (?) and password = password(?)
select_user = select * from account_table where username = (?)
select_reserved_user = select * from reservation_table where username = (?)
select_existing_course = select * from course_table where course_abbr = (?)
select_existing_service = select * from service_table where service_name = (?)
select_existing_transaction_id = select transaction_id from transaction_table where transaction_id = ?
select_user_is_active = select status from student_table where username = ?
select_page_requirement = select page_requirement from service_table where service_name = ?
select_user_details = select account_table.username as 'Username', concat(account_table.last_name, ', ', account_table.first_name, ' ', account_table.suffix_name, ' ', account_table.mid_name) as 'Name',  student_table.course_abbr as 'Degree Program', student_table.year_lvl as 'Year Level', student_table.freetime as 'Free Time' from account_table inner join student_table on account_table.username = student_table.username where student_table.username = ?
select_amount_service = select amount from service_table where service_name = ?
select_id_service = select * from service_table where service_name = ?
select_freetime = select student_table.freetime from student_table inner join transaction_table on student_table.username = transaction_table.username where transaction_table.transaction_id = ?
select_timediff = select timediff(time(?), timediff(time(logout_time), time(login_time))) as 'timedifference' from pc_usage_table where transaction_id = ?
select_trans_user = select username from transaction_table where transaction_id = ?
select_pc_id1 = select pc_id from pc_table where ip_address = ?
select_timedifference = select timediff(time(?), timediff(curtime(), time(?))) as 'timedifference' from pc_usage_table where transaction_id = ?
select_logout_time = select logout_time from pc_usage_table where transaction_id = ?
select_login_time = select login_time from pc_usage_table where transaction_id = ?
select_now = select curtime()
select_time_consumed = select timediff(time(logout_time), time(login_time)) as 'timedifference' from pc_usage_table where time_to_sec(timediff(time(logout_time), time(login_time))) < time_to_sec(time(?)) and transaction_id = ?
select_freetime_user = select freetime from student_table where username = ?
select_cost_transaction = select cost_payable from transaction_table where transaction_id = ?
select_amount_transaction = select amount_paid from transaction_table where transaction_id = ?
select_pc_id_from_trans = select pc_id from pc_usage_table where transaction_id = ?
select_pc_id2 = select pc_table.pc_id from pc_table
select_transactions_with_accountabilities = select transaction_id from transaction_table where username = ? and amount_paid < cost_payable
select_picture = select photolink from student_table where username = ?
select_diagnostic_table2 = select * from diagnostic_table where sem_id = ?
select_diagnostic_table = select * from diagnostic_table order by diagnostic_table.date_end desc limit 1

select_filtered_username = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', student_table.year_lvl as 'Year Level', student_table.course_abbr as 'Degree Program', student_table.status as 'Status', student_table.balance as 'Balance', ifnull((select sum(transaction_table.cost_payable - transaction_table.amount_paid) from transaction_table where transaction_table.cost_payable > transaction_table.amount_paid and transaction_table.username = account_table.username),0) as 'Accountabilities' from account_table inner join student_table on account_table.username = student_table.username where account_table.username like (?) and student_table.username like (?) group by username
select_filtered_lastname = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', student_table.year_lvl as 'Year Level', student_table.course_abbr as 'Degree Program', student_table.status as 'Status', student_table.balance as 'Balance', ifnull((select sum(transaction_table.cost_payable - transaction_table.amount_paid) from transaction_table where transaction_table.cost_payable > transaction_table.amount_paid and transaction_table.username = account_table.username),0) as 'Accountabilities' from account_table join student_table on account_table.username = student_table.username where account_table.last_name like ? group by username
select_filtered_firstname = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', student_table.year_lvl as 'Year Level', student_table.course_abbr as 'Degree Program', student_table.status as 'Status', student_table.balance as 'Balance', ifnull((select sum(transaction_table.cost_payable - transaction_table.amount_paid) from transaction_table where transaction_table.cost_payable > transaction_table.amount_paid and transaction_table.username = account_table.username),0) as 'Accountabilities' from account_table join student_table on account_table.username = student_table.username where account_table.first_name like ? group by username
select_filtered_yearlvl = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', student_table.year_lvl as 'Year Level', student_table.course_abbr as 'Degree Program', student_table.status as 'Status', student_table.balance as 'Balance', ifnull((select sum(transaction_table.cost_payable - transaction_table.amount_paid) from transaction_table where transaction_table.cost_payable > transaction_table.amount_paid and transaction_table.username = account_table.username),0) as 'Accountabilities' from account_table join student_table on account_table.username = student_table.username where student_table.year_lvl like ? group by username
select_filtered_degprog = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', student_table.year_lvl as 'Year Level', student_table.course_abbr as 'Degree Program', student_table.status as 'Status', student_table.balance as 'Balance', ifnull((select sum(transaction_table.cost_payable - transaction_table.amount_paid) from transaction_table where transaction_table.cost_payable > transaction_table.amount_paid and transaction_table.username = account_table.username),0) as 'Accountabilities' from account_table join student_table on account_table.username = student_table.username where student_table.course_abbr like ? group by username

select_filtered_username2 = select transaction_table.date_rendered as 'Date', transaction_table.transaction_id as 'Transaction ID', transaction_table.username 'Username', service_table.service_name 'Service Name', substring(transaction_table.cost_payable, 1, 5) as 'Cost', substring(transaction_table.amount_paid, 1, 5) as 'Amount Rendered' from transaction_table inner join service_table on transaction_table.service_id = service_table.service_id where transaction_table.username like ? group by transaction_id
select_filtered_servicename = select transaction_table.date_rendered as 'Date', transaction_table.transaction_id as 'Transaction ID', transaction_table.username 'Username', service_table.service_name 'Service Name', substring(transaction_table.cost_payable, 1, 5) as 'Cost', substring(transaction_table.amount_paid, 1, 5) as 'Amount Rendered' from transaction_table inner join service_table on transaction_table.service_id = service_table.service_id where service_table.service_name like ? group by transaction_id
select_filtered_date = select transaction_table.date_rendered as 'Date', transaction_table.transaction_id as 'Transaction ID', transaction_table.username 'Username', service_table.service_name 'Service Name', substring(transaction_table.cost_payable, 1, 5) as 'Cost', substring(transaction_table.amount_paid, 1, 5) as 'Amount Rendered' from transaction_table inner join service_table on transaction_table.service_id = service_table.service_id where transaction_table.date_rendered like ? group by Transaction_id

select_all = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', student_table.year_lvl as 'Year Level', student_table.course_abbr as 'Degree Program', student_table.status as 'Status', student_table.freetime as 'Free Time', student_table.balance as 'Balance', ifnull((select sum(transaction_table.cost_payable - transaction_table.amount_paid) from transaction_table where transaction_table.cost_payable > transaction_table.amount_paid and transaction_table.username = account_table.username),0) as 'Accountabilities' from account_table join student_table on account_table.username = student_table.username
select_filtered_active = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', student_table.year_lvl as 'Year Level', student_table.course_abbr as 'Degree Program', student_table.status as 'Status', student_table.freetime as 'Free Time', student_table.balance as 'Balance', ifnull((select sum(transaction_table.cost_payable - transaction_table.amount_paid) from transaction_table where transaction_table.cost_payable > transaction_table.amount_paid and transaction_table.username = account_table.username),0) as 'Accountabilities' from account_table join student_table on account_table.username = student_table.username where student_table.status = 'active'
select_filtered_inactive = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', student_table.year_lvl as 'Year Level', student_table.course_abbr as 'Degree Program', student_table.status as 'Status', student_table.freetime as 'Free Time', student_table.balance as 'Balance', ifnull((select sum(transaction_table.cost_payable - transaction_table.amount_paid) from transaction_table where transaction_table.cost_payable > transaction_table.amount_paid and transaction_table.username = account_table.username),0) as 'Accountabilities' from account_table join student_table on account_table.username = student_table.username where student_table.status = 'inactive'

select_online_pc =
select_reserved_pc = select reservation_table.username as 'Username' from reservation_table
select_staff_table = select account_table.username as 'Username', account_table.last_name as 'Last Name', concat(account_table.first_name, ', ', account_table.suffix_name) as 'First Name', account_table.mid_name as 'Middle Name', admin_table.delete_priv as 'Delete Privilege', admin_table.settle_priv as 'Settle Privilege', admin_table.db_access as 'Database Access' from account_table inner join admin_table on account_table.username = admin_table.username
select_degree_table = select course_table.course_name as 'Degree Program', course_table.course_abbr as 'Abbreviation' from course_table
select_service_table = select service_name as 'Service Name', amount as 'Amount' from service_table
select_pasa_hour = select pasa_hour_table.date_rendered as 'Date', pasa_hour_table.amount_time as 'Amount Time', concat(pasa_hour_table.sender, '     ( ', pasa_hour_table.current_free_sender, '  -  ', pasa_hour_table.deducted_free_sender, ' )') as 'Sender (Current - Deducted)', concat(pasa_hour_table.receiver, '     ( ', pasa_hour_table.current_free_receiver, '  -  ', pasa_hour_table.added_free_receiver, ' )') as 'Receiver (Current - Added)' from pasa_hour_table
select_pasa_bal = select date_rendered as 'Date', amount as 'Amount Time', concat(sender, '     ( ', current_balance_sender, '  -  ', deducted_balance_sender, ' )') as 'Sender (Current - Deducted)', concat(receiver, '     ( ', current_balance_receiver, '  -  ', added_balance_receiver, ' )') as 'Receiver (Current - Added)' from pasa_balance_table

select_transaction_table = select transaction_table.date_rendered as 'Date', transaction_table.transaction_id as 'Transaction ID', transaction_table.username 'Username', service_table.service_name 'Service Name', substring(transaction_table.cost_payable, 1, 5) as 'Cost', substring(transaction_table.amount_paid, 1, 5) as 'Amount Rendered' from transaction_table inner join service_table on transaction_table.service_id = service_table.service_id where transaction_table.date_rendered = curdate()
select_all_transaction_table = select transaction_table.date_rendered as 'Date', transaction_table.transaction_id as 'Transaction ID', transaction_table.username 'Username', service_table.service_name 'Service Name', substring(transaction_table.cost_payable, 1, 5) as 'Cost', substring(transaction_table.amount_paid, 1, 5) as 'Amount Rendered' from transaction_table inner join service_table on transaction_table.service_id = service_table.service_id
select_paid_transaction_table = select transaction_table.date_rendered as 'Date', transaction_table.transaction_id as 'Transaction ID', transaction_table.username 'Username', service_table.service_name 'Service Name', substring(transaction_table.cost_payable, 1, 5) as 'Cost', substring(transaction_table.amount_paid, 1, 5) as 'Amount Rendered' from transaction_table inner join service_table on transaction_table.service_id = service_table.service_id where transaction_table.cost_payable <= transaction_table.amount_paid
select_unpaid_transaction_table = select transaction_table.date_rendered as 'Date', transaction_table.transaction_id as 'Transaction ID', transaction_table.username 'Username', service_table.service_name 'Service Name', substring(transaction_table.cost_payable, 1, 5) as 'Cost', substring(transaction_table.amount_paid, 1, 5) as 'Amount Rendered' from transaction_table inner join service_table on transaction_table.service_id = service_table.service_id where transaction_table.cost_payable > transaction_table.amount_paid

select_usage_daily = select distinct a.pc_id as 'PC Number', (select count(b.pc_id) from pc_usage_table b where b.pc_id = a.pc_id && b.transaction_id in (select transaction_id from transaction_table where date_rendered = ?)) as 'Total # of Transactions', (select count(distinct c.username) from transaction_table c where c.transaction_id in (select d.transaction_id from pc_usage_table d where d.pc_id = a.pc_id) && c.transaction_id in (select transaction_id from transaction_table where date_rendered = ?)) as 'Total # of Users' from pc_usage_table a join transaction_table e on a.transaction_id = e.transaction_id where date_rendered = ?
select_usage_monthly = select distinct a.pc_id as 'PC Number', (select count(b.pc_id) from pc_usage_table b where b.pc_id = a.pc_id && b.transaction_id in (select transaction_id from transaction_table where year(date_rendered) = ? and monthname(date_rendered) = ?)) as 'Total # of Transactions', (select count(distinct c.username) from transaction_table c where c.transaction_id in (select d.transaction_id from pc_usage_table d where d.pc_id = a.pc_id) && c.transaction_id in (select transaction_id from transaction_table where year(date_rendered) = ? and monthname(date_rendered) = ?)) as 'Total # of Users' from pc_usage_table a join transaction_table e on a.transaction_id = e.transaction_id where year(e.date_rendered) = ? and monthname(date_rendered) = ?
select_usage_annual = select distinct a.pc_id as 'PC Number', (select count(b.pc_id) from pc_usage_table b where b.pc_id = a.pc_id && b.transaction_id in (select transaction_id from transaction_table where year(date_rendered) = ?)) as 'Total # of Transactions', (select count(distinct c.username) from transaction_table c where c.transaction_id in (select d.transaction_id from pc_usage_table d where d.pc_id = a.pc_id) && c.transaction_id in (select transaction_id from transaction_table where year(date_rendered) = ?)) as 'Total # of Users' from pc_usage_table a join transaction_table e on a.transaction_id = e.transaction_id where year(e.date_rendered) = ?
select_usage_semestral = select distinct a.pc_id as 'PC Number', (select count(b.pc_id) from pc_usage_table b where b.pc_id = a.pc_id && b.transaction_id in (select transaction_id from transaction_table where date_rendered between (select date_start from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)) and (select date_end from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)))) as 'Total # of Transactions', (select count(distinct c.username) from transaction_table c where c.transaction_id in (select d.transaction_id from pc_usage_table d where d.pc_id = a.pc_id) && c.transaction_id in (select transaction_id from transaction_table where transaction_table.date_rendered between (select date_start from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)) and (select date_end from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)))) as 'Total # of Users' from pc_usage_table a join transaction_table e on a.transaction_id = e.transaction_id where e.date_rendered between (select date_start from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)) and (select date_end from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?))

select_student_daily = select account_table.username, concat(account_table.last_name, ', ', account_table.first_name, ', ', account_table.suffix_name, ', ', account_table.mid_name), student_table.course_abbr from account_table inner join student_table on student_table.username = account_table.username where account_table.username in (select transaction_table.username from transaction_table inner join pc_usage_table on transaction_table.transaction_id = pc_usage_table.transaction_id where year(transaction_table.date_rendered) = ? and transaction_table.date_rendered = ?)
select_student_monthly = select account_table.username as 'Student Number', concat(account_table.last_name, ', ', account_table.first_name, ', ', account_table.suffix_name, ', ', account_table.mid_name) as 'Name', student_table.course_abbr as 'Degree Program' from account_table inner join student_table on student_table.username = account_table.username where account_table.username in (select transaction_table.username from transaction_table inner join pc_usage_table on transaction_table.transaction_id = pc_usage_table.transaction_id where year(transaction_table.date_rendered) = ? and monthname(transaction_table.date_rendered) = ?)
select_student_annual = select account_table.username as 'Student Number', concat(account_table.last_name, ', ', account_table.first_name, ', ', account_table.suffix_name, ', ', account_table.mid_name) as 'Name', student_table.course_abbr as 'Degree Program' from account_table inner join student_table on student_table.username = account_table.username where account_table.username in (select transaction_table.username from transaction_table inner join pc_usage_table on transaction_table.transaction_id = pc_usage_table.transaction_id where year(transaction_table.date_rendered) = ?)
select_student_semestral = select account_table.username as 'Student Number', concat(account_table.last_name, ', ', account_table.first_name, ', ', account_table.suffix_name, ', ', account_table.mid_name) as 'Name', student_table.course_abbr as 'Degree Program' from account_table inner join student_table on student_table.username = account_table.username where account_table.username in (select transaction_table.username from transaction_table inner join pc_usage_table on transaction_table.transaction_id = pc_usage_table.transaction_id where transaction_table.date_rendered between (select date_start from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)) and (select date_end from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)))

select_transaction_daily = select service_table.service_name as 'Service Name', sum(transaction_table.cost_payable) as 'Cost Payable' from service_table join transaction_table on service_table.service_id = transaction_table.service_id where transaction_table.date_rendered = ? group by transaction_table.service_id
select_transaction_monthly = select service_table.service_name as 'Service Name', sum(transaction_table.cost_payable) as 'Cost Payable' from service_table join transaction_table on service_table.service_id = transaction_table.service_id where year(transaction_table.date_rendered) = ? and monthname(transaction_table.date_rendered) = ? group by transaction_table.service_id
select_transaction_annual = select service_table.service_name as 'Service Name', sum(transaction_table.cost_payable) as 'Cost Payable' from service_table join transaction_table on service_table.service_id = transaction_table.service_id where year(transaction_table.date_rendered) = ? group by transaction_table.service_id
select_transaction_semestral = select service_table.service_name as 'Service Name', sum(transaction_table.cost_payable) as 'Cost Payable' from service_table join transaction_table on service_table.service_id = transaction_table.service_id where transaction_table.date_rendered between (select date_start from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)) and (select date_end from diagnostic_table where sem_num = ? and (year(date_start) = ? or year(date_end) = ?)) group by transaction_table.service_id

select_latest_trans = select transaction_table.date_rendered as 'Date', service_table.service_name as 'Service Name', substring(transaction_table.amount_paid,1,5) as 'Cash Rendered', substring(transaction_table.cost_payable,1,5) as "Cost Payable" from transaction_table inner join service_table on service_table.service_id = transaction_table.service_id where transaction_table.username = ? order by transaction_table.transaction_id desc limit 5
select_trans_by_user = select service_table.service_name as 'Service Name', sum(transaction_table.amount_paid) as 'Amount Paid', sum(transaction_table.cost_payable) as 'Cost Payable' from service_table join transaction_table on service_table.service_id = transaction_table.service_id where transaction_table.username = ? and transaction_table.amount_paid < transaction_table.cost_payable group by transaction_table.service_id

update_activate_student = update student_table set status = 'active' where username = ?
update_deactivate_student = update student_table set status = 'inactive' where username = ?
update_profile_pic = update student_table set photolink = ? where username = ?
update_amount = update transaction_table set amount_paid = ? where transaction_id = ?
update_cash_close = update cash_flow_table set cash_close = cash_close + ? where dates = curdate()
update_balance = update student_table set balance = ? where username = ?
update_logout_expand = update pc_usage_table set logout_time = ? where transaction_id = ?
update_cost_transaction = update transaction_table set cost_payable = (select cost_payable + ? where transaction_id = ?) where transaction_id = ?
update_cost_transaction_plain = update transaction_table set cost_payable = ? where transaction_id = ?
update_amount_transaction = update transaction_table set amount_paid = (select amount_paid + ? where transaction_id = ?) where transaction_id = ?
update_pasa_hour_table = update pasa_hour_table set deducted_free_sender = ?, added_free_receiver = ? where transaction_id = ?
update_pasa_balance_table = update pasa_balance_table set deducted_balance_sender = ?, added_balance_receiver = ? where transaction_id = ?
update_receiver_time = update student_table set freetime = (select addtime(freetime,time(?)) where username = ?) where username = ?
update_sender_time = update student_table set freetime = (select timediff(freetime,time(?)) where username = ?) where username = ?
update_logout_pending = update pc_usage_table set logout_time = (select addtime(time(login_time), time(?))) where transaction_id = ?
update_logout_time = update pc_usage_table set logout_time = curtime() where transaction_id = ?
update_logout_time_with_reference = update pc_usage_table set logout_time = ? where transaction_id = ?
update_user_time = update student_table set freetime = ? where username = ?
update_reset_pw = update account_table set password = password(?) where username = ?
update_all_status = update student_table set status = 'inactive'
update_course_table = update course_table set course_abbr = ?, course_name = ? where course_abbr = ?
update_user_password = update account_table set password = password(?) where username = ? and password = password(?)
update_account_table = update account_table set username = ?, last_name = ?, first_name = ?, mid_name = ?, suffix_name = ? where username = ?
update_admin_table = update admin_table set username = ?, delete_priv = ?, settle_priv = ?, db_access = ? where username = ?
update_student_table = update student_table set username = ?, year_lvl = ?, course_abbr = ?, status = ? where username = ?
update_service_table = update service_table set service_name = ?, amount = ?, page_requirement = ? where service_name = ?
[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

,

Cory DoctorowPodcast: Let’s get better at demanding better from tech


Here’s my reading (MP3) of Let’s get better at demanding better from tech, a Locus Magazine column about the need to enlist moral, ethical technologists in the fight for a better technological future. It was written before the death of EFF co-founder John Perry Barlow, whose life’s work was devoted to this proposition, and before the Google uprising over Project Maven, in which technologists killed millions in military contracts by refusing to build AI systems for the Pentagon’s drones.

MP3

Worse Than FailureSponsor Post: Error Logging vs. Crash Reporting

A lot of developers confuse error and crash reporting tools with traditional logging. And it’s easy to make the relation without understanding the two in more detail.

Dedicated logging tools give you a running history of events that have happened in your application. Dedicated error and crash reporting tools focus on the issues users face that occur when your app is in production, and record the diagnostic details surrounding the problem that happened to the user, so you can fix it with greater speed and accuracy.

Most error logging activities within software teams remain just that. A log of errors that are never actioned and fixed.

Traditionally speaking, when a user reports an issue, you might find yourself hunting around in log files searching for what happened so you can debug it successfully.

Having an error reporting tool running silently in production means not only do users not need to report issues, as they are identified automatically, but each one is displayed in a dashboard, ranked by severity. Teams are able to get down to the root cause of an issue in seconds, not hours.

Full diagnostic details about the issue are presented to the developer immediately. Information such as OS, browser, machine, a detailed stack trace, a history of events leading up to the issue and even which individual users have encountered the specific issue are all made available.

In short, when trying to solve issues in your applications, you immediately see the needle, without bothering with the haystack.

Error monitoring tools are designed to give you answers quickly. Once you experience how they fit into the software development workflow and work alongside your logging, you won’t want to manage your application errors in any other way.

So next time you’re struggling to resolve problems in your apps - Think Raygun.

Your life as a developer will be made so much easier.

[Advertisement] Forget logs. Next time you're struggling to replicate error, crash and performance issues in your apps - Think Raygun! Installs in minutes. Learn more.

Worse Than FailureA Hard SQL Error

Prim Maze

Padma was the new guy on the team, and that sucked. When you're the new guy, but you're not new to the field, there's this maddening combination of factors that can make onboarding rough: a combination of not knowing the product well enough to be efficient, but knowing your craft well enough to expect efficiency. After all, if you're a new intern, you can throw back general-purpose tutorials and feel like you're learning new things at least. When you're a senior trying to make sense of your new company's dizzying array of under-documented products? The only way to get that knowledge is by dragging people who are already efficient away from what they're doing to ask.

By the start of week 2, however, Padma knew enough to get his hands dirty with some smaller bug-fixes. By the end of it, he'd begun browsing the company bug tracker looking for more work on his own. That's when he came across this bug report that seemed rather urgent:

Error: Can't connect to local MySQL server

It had been in the tracker for a month. That could mean a lot of things, all of them opaque when you're new enough not to know anyone. Was it impossible to reproduce? Was it one of those reports thrown in by someone who liked to tamper with their test environment and blame things breaking on the coders? Was their survey product just low priority enough that they hadn't gotten around to fixing it? Which client was this for?

It took Padma a few hours to dig into it enough to get to the root of the problem. The repository for their survey product was stored in their private github, one of dozens of repositories with opaque names. He found the codename of the product, "Santiago," by reading older tickets filed against the same product, before someone had renamed the tag to "Survey Deluxe." There was a branch for every client, an empty Master branch, and a Development branch as the default; he reached back out to the reporter for the name of the client so he could pull up their branch. Of course they had a "clientname" branch, a "clientname-new," and a "clientname3.0," but after comparing merge histories, he eventually discovered the production code: in a totally different branch, after they had merged two clients' environments together for a joint venture. Of course.

But finally, he had the problem reproduced in his local dev environment. After an hour of digging through folders, he found the responsible code:


<h2 id="survey">Surveys</h2>
        <div style="margin-left:10px;">
        <ul class="submenu">
                <li><a href="survey1.php">Survey #1</a></li>
                <li><a href="survey2.php">Survey #2</a><span style="color:red">Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)</span></li>
        </ul>
</div>

"But ... why?!" Padma growled at the screen.

"Oh, is that Santiago?" asked his neighbor, leaning over to see his screen. "Yeah, they requested a one-for-one conversion from their previous product. Warts and all. Seems they thought that was the name of the survey, and it was important that it be in red so they could find it easily enough."

Padma stared at the code in disbelief. After a long moment, he closed the editor and the browser, deleted the code from his hard drive, and closed the ticket "won't fix."

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

CryptogramSecure Speculative Execution

We're starting to see research into designing speculative execution systems that avoid Spectre- and Meltdown-like security problems. Here's one.

I don't know if this particular design secure. My guess is that we're going to see several iterations of design and attack before we settle on something that works. But it's good to see the research results emerge.

News article.

,

Valerie AuroraYesterday’s joke protest sign just became today’s reality

Tomorrow I’m going to a protest against the forcible separation of immigrant children from their families. When I started thinking about what sign to make, I remembered my sign for the first Women’s March protest, the day after Trump took office in January 2017. It said: “Trump hates kids and puppies… for real!!!

trump_hates_puppies
My  protest sign for the 2017 Women’s March

While I expected a lot of terrifying things to happen over the next few years, I never, never thought that Trump would deliberately tear thousands of children away from their families and put them in concentration camps. I knew he hated children; I didn’t know he hated children (specifically, brown children) so much that he’d hold them hostage to force Congress to pass his racist legislation. I did not expect him and his party to try to sell cages full of weeping little boys as future gang members. I did not expect 55% of Republican voters to support splitting up families and putting them in camps. I’m smiling at the cute dog in that photo; now the entire concept of that sign seems impossibly naive and inappropriate, much less my expression in that photo. I apologize for this sign and my joking attitude.

I remember being terrified during the months between Trump’s election and his inauguration. I couldn’t sleep; I put together a go-bag; I bought three weeks worth of food and water and stored them in the closet. I read a dozen books on fascism and failed democracies. I even built a spreadsheet tracking signs of fascism so I’d know when to leave the country.

I came up with the concept of that sign as a way to increase people’s disgust for Trump; what kind of pathetic low-life creep hates kids AND puppies? But I still didn’t get how bad things truly were; I thought Trump hated kids in the sense that he didn’t want any of them around him and wouldn’t lift a finger to help them. I didn’t understand that he—and many people in his administration—took actual pleasure in knowing they were building camps full of crying, desperate, terrified kids who may never be reunited with their parents. In January 2017, I thought I understood the evil of this administration and of a significant percentage of the people in this country; actually, I way underestimated it.

At that protest, several people asked me if Trump really hated puppies, but not one person asked me if Trump really hated kids. In retrospect, this seems ominous, not funny.

I’m going to think very carefully before creating any more “joke” protest signs. Today’s “joke” could easily be tomorrow’s reality.

,

CryptogramFriday Squid Blogging: Capturing the Giant Squid on Video

In this 2013 TED talk, oceanographer Edith Widder explains how her team captured the giant squid on video.

As usual, you can also use this squid post to talk about the security stories in the news that I haven't covered.

Read my blog posting guidelines here.

Krebs on SecuritySupreme Court: Police Need Warrant for Mobile Location Data

The U.S. Supreme Court today ruled that the government needs to obtain a court-ordered warrant to gather location data on mobile device users. The decision is a major development for privacy rights, but experts say it may have limited bearing on the selling of real-time customer location data by the wireless carriers to third-party companies.

CryptogramThe Effects of Iran's Telegram Ban

The Center for Human Rights in Iran has released a report outlining the effect's of that country's ban on Telegram, a secure messaging app used by about half of the country.

The ban will disrupt the most important, uncensored platform for information and communication in Iran, one that is used extensively by activists, independent and citizen journalists, dissidents and international media. It will also impact electoral politics in Iran, as centrist, reformist and other relatively moderate political groups that are allowed to participate in Iran's elections have been heavily and successfully using Telegram to promote their candidates and electoral lists during elections. State-controlled domestic apps and media will not provide these groups with such a platform, even as they continue to do so for conservative and hardline political forces in the country, significantly aiding the latter.

From a Wired article:

Researchers found that the ban has had broad effects, hindering and chilling individual speech, forcing political campaigns to turn to state-sponsored media tools, limiting journalists and activists, curtailing international interactions, and eroding businesses that grew their infrastructure and reach off of Telegram.

It's interesting that the analysis doesn't really center around the security properties of Telegram, but more around its ubiquity as a messaging platform in the country.

CryptogramDomain Name Stealing at Gunpoint

I missed this story when it came around last year: someone tried to steal a domain name at gunpoint. He was just sentenced to 20 years in jail.

Worse Than FailureError'd: Be Patient!...OK?

"I used to feel nervous when making payments online, but now I feel ...um...'Close' about it," writes Jeff K.

 

"Looks like me and Microsoft have different ideas of what 75% means," Gary S. wrote.

 

George writes, "Try this one at home! Head to tdbank.com, search for 'documents for opening account' and enjoy 8 solid pages of ...this."

 

"I'm confused if the developers knew the difference between Javascript and Java. This has to be a troll...right?" wrote JM.

 

Tom S. writes, "Saw this in the Friendo app, but what I didn't spot was an Ok button. "

 

"I look at this and wonder if someone could deny a vacation requests because of a conflict of 0.000014 days with another member of staff," writes Rob.

 

[Advertisement] Forget logs. Next time you're struggling to replicate error, crash and performance issues in your apps - Think Raygun! Installs in minutes. Learn more.

Sam VargheseRecycling Trump: Old news passed off as investigative reporting

Over the last three weeks, viewers of the Australian Broadcasting Corporation’s Four Corners program have been treated to what is the ultimate waste of time: a recapping of all that has gone on in the United States during the investigation into alleged Russian collusion with the Trump campaign in the 2016 presidential campaign.

There was nothing new in the nearly three hours of programming on what is the ABC’s prime investigative program. It only served as a vanity outlet for Sarah Ferguson, rated as one of the network’s better reporters, but after this, and her unnecessary Hillary Clinton interview, she appears to be someone who is interested in big-noting herself.

Exactly why Ferguson and a crew spent what must be between four to six weeks in the US, London and Moscow to put to air material that has been beaten to death by the US and other Western media is a mystery. Had Ferguson managed to unearth one nugget of information that has gone unnoticed so far, one would not be inclined to complain.

But this same ABC has been crying itself hoarse for the last few months over cuts to its budget and trumpeting its news credentials – and then it produces garbage like the three episode of the Russia-Trump series or whatever it was called.

As an aside, the investigation has been going on for more than a year now, with special counsel Robert Mueller, a former FBI director, having been appointed on May 17, 2017. The American media have had a field day and every time there is a fresh development, these are shrieks all around that this is the straw that breaks the camel’s back. But it all turns out to be an illusion in the end.

Every little detail of the process of electing Donald Trump has been covered and dissected over and over and over again. And yet Ferguson thought it a good idea to run three hours of this garbage.

Apart from the fact that this something akin to the behaviour of a dog that revisits its own vomit, Ferguson also paraded some very dodgy individuals to bolster her program.

One was James Clapper, the director of national intelligence during the Obama presidency. Clapper is a man who has committed perjury by lying to the US Congress under oath. Clapper also leaked information about the infamous anti-Trump dossier to CNN’s Jake Tapper and then was rewarded with a contract at CNN.

Clapper does not have the best of reputations when it comes to integrity. To call him a shady character would not be a stretch. Now Ferguson may have needed to speak to him once, because he was the DNI under Obama. But she did not need to have him appear every now and then, remarking on this and that. He added no weight to an already weak program.

Another person Ferguson gave plenty of air time to was Luke Harding, a reporter with the Guardian. Harding is known for a few things: plagiarising others’ reports while he was stationed in Moscow and writing a book about Edward Snowden without having met any of the principal players in the matter. Once again, a person of dubious character.

One would also have to ask: why does the camera focus on the reporter so much? Is she the story? Or is it a way to puff herself up and appear so important that she cannot be out of sight of the lens lest the story break down? It is a curse of modern journalism, this narcissism, and Ferguson suffers from it badly.

This is the second worthless program Ferguson has produced in recent times; the first was her puff interview with Hillary Clinton.

Maybe she is gearing up to take on some kind of job in the US. Wouldn’t surprise me if public money was being used to paint the meretricious as the magnificent.

CryptogramAlgeria Shut Down the Internet to Prevent Students from Cheating on Exams

Algeria shut the Internet down nationwide to prevent high-school students from cheating on their exams.

The solution in New South Wales, Australia was to ban smartphones.

EDITED TO ADD (6/22): Slashdot thread.