Planet Bozo

December 04, 2020

etbeKDE Icons Disappearing in Debian/Unstable

One of my workstations is running Debian/Unstable with KDE and SDDM on an AMD Radeon R7 260X video card. Recently it stopped displaying things correctly after a reboot, all the icons failed to display as well as many of the Qt controls. When I ran a KDE application from the command line I got the error “QSGTextureAtlas: texture atlas allocation failed, code=501“. Googling that error gave a blog post about a very similar issue in 2017 [1]. From that blog post I learned that I could stop the problem by setting MESA_EXTENSION_OVERRIDE=”-GL_EXT_bgra -GL_EXT_texture_format_BGRA8888″ in the environment. In a quick test I found that the environment variable setting worked, making the KDE apps display correctly and not report an error about a texture atlas.

I created a file ~/.config/plasma-workspace/env/bgra.sh with the following contents:

export MESA_EXTENSION_OVERRIDE="-GL_EXT_bgra -GL_EXT_texture_format_BGRA8888"

Then after the next login things worked as desired!

Now the issue is, where is the bug? GL, X, and the internals of KDE are things I don’t track much. I welcome suggestions from readers of my blog as to what the culprit might be and where to file a Debian bug – or a URL to a Debian bug report if someone has already filed one.

Update

When I run the game warzone2100 with this setting it crashes with the below output. So this Mesa extension override isn’t always a good thing, just solves one corner case of a bug.

$ warzone2100 
/usr/bin/gdb: warning: Couldn't determine a path for the index cache directory.
27      ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
No frame at level 0x7ffc3392ab50.
Saved dump file to '/home/etbe/.local/share/warzone2100-3.3.0//logs/warzone2100.gdmp-VuGo2s'
If you create a bugreport regarding this crash, please include this file.
Segmentation fault (core dumped)

Update 2

Carsten provided the REAL solution to this, run “apt remove libqt5quick5-gles” which will automatically install “libqt5quick5” which makes things work. Another workstation I run that tracks Testing had libqt5quick5 installed which was why it didn’t have the problem.

The system in question had most of KDE removed due to package dependency issues when tracking Unstable and when I reinstalled it I guess the wrong one was installed.

Worse Than FailureError'd: Mandatory Pants Day

"I wonder if the people behind the ad campaign ever said they could 'sell ice to a snowman'," Loren writes.

 

"Umm...Thanks for the suggestion?" John wrote.

 

Kolja writes, "So, wait, do I type 'ENTER' or just press the 'ENTER' key? And what if I hae a 'Return' key instead?"

 

"Given how annoying and defective software licensing usually is, an operation completing successfully should indeed be considered as an error," wrote Carl C.

 

Tim P. writes, "If you're thinking about buying your candy by the kilo, beware of that sugar tax!"

 

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

December 03, 2020

Worse Than FailureCodeSOD: WWJSD?

A few months ago, Lee was reviewing a pull request from Eddie. Eddie was a self-appointed "rockstar" developer. Eddie might be a "junior" developer in job title, but they're up on the latest, greatest, bestest practices, and if everyone would just get out of Eddie's way, the software would get so much better.

Which is why the pull request Lee was looking at touched nearly every file. Every change was some variation of this:

- if (obj == null) + if (obj is null)

"Did… did you do a 'Replace in Files' on the entire solution?" Lee asked.

"Well, I had to use the regex find-and-replace," Eddie said, "have you ever used that? It's a great refactoring tool, you can get it with control-H, and then you have to check a box-"

"I know how it works," Lee said. "Why?"

"Well, a regex, or a regular expression is a-"

"No, why did you make the change?"

"Oh! Well, that's the correct way to check for nulls. The equality operator could be overloaded, and that overload might have bugs."

"I mean, it's a correct way," Lee said, "but the equality operator is also fine. Any bugs in our overloads are likely already caught. There's no reason to make a change like this through the whole codebase."

The debate went on for longer than it should have. Eventually, Eddie defended his choice by saying, "Well, it's the way Jon Skeet does it," at which point Lee threw up his hands.

"Fine." Lee approved the request. After all, what was the worst that could happen?

What happened was that a month or two later, the application started throwing NullReferenceExceptions in production. The change which caused the behavior was on line:

Coordinate p = maybeGetCoordinate(); if (p != null) { p.Project(alternateCoordinateSystem); // NullReferenceException }

Look at that, someone using the more traditional equality operator! Maybe Eddie had a point, maybe an overloaded equality operator was the problem. Certainly, the bug went away if Lee used !(p is null) (or, in C# 9.0, p is not null).

Still, it would be good to fix the bug. Lee took a peek at the overloaded operators:

public static bool operator == (Coordinate pt1, Coordinate pt2) => !(pt1 is null) && !(pt2 is null) && pt1.Longitude == pt2.Longitude && pt1.Latitude == pt2.Latitude; public static bool operator !=(Coordinate pt1, Coordinate pt2) => !(pt1 == pt2);

The key bug is !(pt1 is null) && !(pt2 is null)- if one or more of the operands is null, then return false. This means null == null is always false, which means the p != null check returns false if p is null.

As it turned out, Eddie did have a point about favoring is null checks. Someone might add a buggy operator overload which could cause unexpected behavior. Who did add that particular buggy overload? Why, Eddie, of course.

Lee submitted a patch, and gently suggested Eddie might need to spend a little more time in the woodshed before declaring themselves a rockstar.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

December 02, 2020

Worse Than FailureA New Bean

Roasted coffee beans

It was Paramdeep's first corporate IT job. He was assigned a mentor, Rajiv, who would train him up on the application he would be supporting: a WebSphere-based Java application full of Enterprise Java Beans.

Paramdeep reserved time with Rajiv in Outlook, arranging to meet at Rajiv's cubicle for half an hour. Rajiv accepted. At the agreed-upon time, Paramdeep walked over with a notebook and pencil in hand, intent on copying down all the pertinent information he would hear. When he reached Rajiv's desk, however, the elder developer waved Paramdeep away from his spare chair before he had a chance to sit down.

"Sorry, more urgent stuff came up," Rajiv said, turning back to his monitor. "The best way to learn about the application is to dive right in. I'll give you one of the simpler tasks. All you need to do is write a bean that'll call a Sybase stored procedure to get the next sequence ID of a domain table, then create the next object based on that ID."

Paramdeep stood there wide-eyed and frozen, pencil hovering uselessly over paper. "What?"

"I've already built an EJB for the database connection layer and all the handling code," Rajiv continued, still intent on his screen. "There's also a stored procedure in the common schema for getting the ID. You just have to put them all together."

"Uh—?"

"I'll send an email with the relevant class names," Rajiv cut him off.

"OK. I'll, uh, let you know if I have any trouble?"

Rajiv was too caught up in his urgent business to respond.

Paramdeep staggered back to his desk: scared, confused, and a bit dejected. But hey, maybe Rajiv was right. Once he dove into the code, he'd have a better understanding of what to do. He'd done JDBC before. It wasn't hard.

Unfortunately, there was a stark difference between a computer science student's homework assignments and the inner workings of a creaky, enterprisey corporate behemoth. Try as he might, Paramdeep simply couldn't connect to the database schema Rajiv had told him about. He was connecting to a default schema instead, which lacked the stored procedure he needed. All of Paramdeep's attempts to solicit help from Rajiv—whether emailed, messaged, or sought in person—met with failure. His boss promised to get Rajiv talking, but that promise never materialized into Paramdeep and Rajiv spending quality time together.

A strict deadline was looming. Desperate times called for desperate measures. Paramdeep emailed the database developer, Amita, suggesting his solution: creating a new stored procedure to call and return the value from the actual stored procedure, which would then return the value to his new bean.

Minutes later, his phone rang. Caller ID showed it was Amita.

"You can't possibly want this!" she declared without preamble. "Just use the stored procedure in the schema."

"I can't connect to it," Paramdeep explained.

"What do you mean, you can't?"

"I just can't!"

"Who's the tech lead on this project?"

"Rajiv."

"Ohhh." A weary understanding permeated her tone, taking all the fight out of it. "OK, I get it. Fine, I'll make a synonym in the default schema."

And that was why the ID generating procedure also existed in the default schema. Paramdeep couldn't help but wonder how many of the procedures in the default schema had gotten there this way.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

XKCDPresidential Middle Names

December 01, 2020

Worse Than FailureCodeSOD: A Tight Fitter

Part of the pitch of Python is that it's a language which values simplicity and ease of use. Whether it lives up to that pitch is its own discussion, but the good news is that if you really want to create really complex code with loads of inheritance and nasty interrelationships, you can.

Today's anonymous submitter started out by verifying some of the math in some extremely math-y analytical code. Previously, they'd had problems where the documentation and the code were subtly different, so it was important that they understood exactly what the code was doing.

In one file, they found the core class they cared about:

class WeibullFitter(KnownModelParametricUnivariateFitter): # snip: some math

Now, the math looked more or less right. But there was just one problem: who knows how the superclass interacts with that? Subtle changes in behavior could appear there.

class KnownModelParametricUnivariateFitter(ParametricUnivariateFitter): _KNOWN_MODEL = True

The "base" class isn't terribly "base" at all, as you can see: all it does is set a property to True. So once again, up the inheritance tree we go, to see the base class:

class ParametricUnivariateFitter(UnivariateFitter): # ...snip...

We're still not at the actual base class, though this one has some implementation, at least? Is that a good thing? We haven't hit TRWTF just yet, but this strikes me as a good chance to talk about why "inheritance is considered harmful": inheritance is an automatic dependency. To understand the behavior of a child class, you have to also understand the behavior of its ancestor classes. Certainly, well implemented inheritance should keep those boundaries neat, but as we can see, this example isn't "well implemented".

More than that, Python is purposefully loosely typed, so one of the key benefits of inheritance, polymorphism isn't even a benefit here. And yes, one could use Python's type annotations to get some lint-time type checking, which would sort-of bring back polymorphism, it still doesn't justify this whole inheritance tree.

That all aside, one of the main reasons we use inheritance is so that we can cut out common conditional logic and let the type system worry about it. I call concreteInstance.someMethod() and the right thing happens, even if I have a dozen possible types, each with some differing behavior. I bring this up, because in the ParametricUnivariateFitter class, we have this:

def _fit_model(self, Ts, E, entry, weights, show_progress=True): if utils.CensoringType.is_left_censoring(self): # Oh no. negative_log_likelihood = self._negative_log_likelihood_left_censoring elif utils.CensoringType.is_interval_censoring(self): # Oh no no no. negative_log_likelihood = self._negative_log_likelihood_interval_censoring elif utils.CensoringType.is_right_censoring(self): # This is exactly what I think it is isn't it. negative_log_likelihood = self._negative_log_likelihood_right_censoring # ...snip...

Comments provided by the submitter. In addition to having a whole tree of child classes, each of these child classes may have a censoring type applied, and our behavior is different based on the censoring type. This is 100% a code smell, and it becomes more clear when we take a look at CensoringType.

class CensoringType(Enum): # enum.Enum from the standard library LEFT = "left" INTERVAL = "interval" RIGHT = "right" @classmethod def right_censoring(cls, function: Callable) -> Callable: @wraps(function) # functools.wraps from the standard library def f(model, *args, **kwargs): cls.set_censoring_type(model, cls.RIGHT) return function(model, *args, **kwargs) return f @classmethod def left_censoring(cls, function: Callable) -> Callable: @wraps(function) def f(model, *args, **kwargs): cls.set_censoring_type(model, cls.LEFT) return function(model, *args, **kwargs) return f @classmethod def interval_censoring(cls, function: Callable) -> Callable: @wraps(function) def f(model, *args, **kwargs): cls.set_censoring_type(model, cls.INTERVAL) return function(model, *args, **kwargs) return f @classmethod def is_right_censoring(cls, model) -> bool: return cls.get_censoring_type(model) == cls.RIGHT @classmethod def is_left_censoring(cls, model) -> bool: return cls.get_censoring_type(model) == cls.LEFT @classmethod def is_interval_censoring(cls, model) -> bool: return cls.get_censoring_type(model) == cls.INTERVAL @classmethod def get_censoring_type(cls, model) -> str: return model._censoring_type @classmethod def str_censoring_type(cls, model) -> str: return model._censoring_type.value @classmethod def set_censoring_type(cls, model, censoring_type) -> None: model._censoring_type = censoring_type

For those not up on Python, Enum is exactly what you think it is, and the @classmethod decorator is Python's way of making static methods. In the same way instance methods take self as their first parameter (the Python-ic "this"), static methods take cls as their first parameter- a reference to the class itself.

It's also important to note the methods like right_censoring, because those themselves are "decorator" definitions. See how they def a local function, itself decorated with @wraps? The right_censoring(cls, function) signature expects a callable (probably a constructor method), and replaced its implementation with the implementation of f- the inner function. Here, it tampers with the input parameters to the constructor before calling the constructor itself.

If you aren't doing a lot of Python, you might be completely confused at this point, so let me just show you how this gets used:

@CensoringType.right_censoring class SomeCurveFitterType(SomeHorribleTreeOfBaseClasses): def __init__(self, model, *args, **kwargs): # snip instance = SomeCurveFitterType(model, *args, **kwargs)

On that last line, it doesn't directly call the __init__ constructor, it first passes through that inner f function, which , most notably, does this: cls.set_censoring_type(model, cls.RIGHT) before invoking the constructor.

If you're confused by all of this, don't feel bad. Decorators are a Pythonic way to tamper with the implementation of classes and functions, allowing you to mix declarative programming with more traditional techniques. In the end, to understand what the WeibullFitter class does, you have to walk up a half dozen ancestor classes to reach the BaseFitter type, and you have to note what decorators are applied to it, and any of its ancestor classes, and know what their implementation is.

If you're the person who wrote this code, this mix of decorators and inheritance probably feels wonderfully extensible. It's probably quick and easy to slap a new curve fitting function into the framework, at least if you're the galaxy-brained individual who dreamed up this over-engineered framework. The rest of us just have to poke at it with sticks until the behavior we want falls out.

Our anonymous submitter adds:

I almost feel bad about this. The library is generally good, the math itself is good, this is a very useful library that has made my job a lot easier…
This sort of behavior actually has real performance implications. I had to re-implement a different piece because the sheer number of nested function calls (some of which were actually recursive, in a chain that spanned three different classes) completely destroyed the performance of a conceptually simple process, such that it could take upwards of ten minutes.
The rewritten function runs near-instantly and fits in 20 lines with comments.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

November 30, 2020

XKCDCyber Cafe

November 27, 2020

XKCDLife Before the Pandemic

November 25, 2020

XKCDLinguists

November 08, 2020

etbeLinks November 2020

KDE has a long term problem of excessive CPU time used by the screen locker [1]. Part of it is due to software GL emulation, and part of it is due to the screen locker doing things like flashing the cursor when nothing else is happening. One of my systems has an NVidia card and enabling GL would cause it to crash. So now I have kscreenlocker using 30% of a CPU core even when the screen is powered down.

Informative NYT article about the latest security features for iPhones [2]. Android needs new features like this!

Russ Allbery wrote an interesting review of the book Hand to Mouth by Linda Tirado [3], it’s about poverty in the US and related things. Linda first became Internet famous for her essay “Why I Make Terrible Decisions or Poverty Thoughts” which is very insightful and well written, this is the latest iteration of that essay [4].

This YouTube video by Ruby Payne gives great insights to class based attitudes towards time and money [5].

News Week has an interesting article about chicken sashimi, apparently you can safely eat raw chicken if it’s prepared well [6].

Vanity Fair has an informative article about how Qanon and Trumpism have infected the Catholic Church [7]. Some of Mel Gibson’s mental illness is affecting a significant portion of the Catholic Church in the US and some parts in the rest of the world.

Noema has an interesting article on toxic Internet culture, Japan’s 2chan, 4chan, 8chan/8kun, and the conspiracy theories they spawned [8].

Benjamin Corey is an ex-Fundie who wrote an amusing analysis of the Biblical statements about the anti-Christ [9].

NYMag has an interesting article The Final Gasp of Donald Trump’s Presidency [10].

Mother Jones has an informative article about the fact that Jim Watkins (the main person behind QAnon) has a history of hosting child porn on sites he runs [11], but we all knew QAnon was never about protecting kids.

Eand has an insightful article America’s Problem is That White People Want It to Be a Failed State [12].

October 19, 2020

etbeVideo Decoding

I’ve had a saga of getting 4K monitors to work well. My latest issue has been video playing, the dreaded mplayer error about the system being too slow. My previous post about 4K was about using DisplayPort to get more than 30Hz scan rate at 4K [1]. I now have a nice 60Hz scan rate which makes WW2 documentaries display nicely among other things.

But when running a 4K monitor on a 3.3GHz i5-2500 quad-core CPU I can’t get a FullHD video to display properly. Part of the process of decoding the video and scaling it to 4K resolution is too slow, so action scenes in movies lag. When running a 2560*1440 monitor on a 2.4GHz E5-2440 hex-core CPU with the mplayer option “-lavdopts threads=3” everything is great (but it fails if mplayer is run with no parameters). In doing tests with apparent performance it seemed that the E5-2440 CPU gains more from the threaded mplayer code than the i5-2500, maybe the E5-2440 is more designed for server use (it’s in a Dell PowerEdge T320 while the i5-2500 is in a random white-box system) or maybe it’s just because it’s newer. I haven’t tested whether the i5-2500 system could perform adequately at 2560*1440 resolution.

The E5-2440 system has an ATI HD 6570 video card which is old, slow, and only does PCIe 2.1 which gives 5GT/s or 8GB/s. The i5-2500 system has a newer ATI video card that is capable of PCIe 3.0, but “lspci -vv” as root says “LnkCap: Port #0, Speed 8GT/s, Width x16” and “LnkSta: Speed 5GT/s (downgraded), Width x16 (ok)”. So for reasons unknown to me the system with a faster PCIe 3.0 video card is being downgraded to PCIe 2.1 speed. A quick check of the web site for my local computer store shows that all ATI video cards costing less than $300 have PCI3 3.0 interfaces and the sole ATI card with PCIe 4.0 (which gives double the PCIe speed if the motherboard supports it) costs almost $500. I’m not inclined to spend $500 on a new video card and then a greater amount of money on a motherboard supporting PCIe 4.0 and CPU and RAM to go in it.

According to my calculations 3840*2160 resolution at 24bpp (probably 32bpp data transfers) at 30 frames/sec means 3840*2160*4*30/1024/1024=950MB/s. PCIe 2.1 can do 8GB/s so that probably isn’t a significant problem.

I’d been planning on buying a new video card for the E5-2440 system, but due to some combination of having a better CPU and lower screen resolution it is working well for video playing so I can save my money.

As an aside the T320 is a server class system that had been running for years in a corporate DC. When I replaced the high speed SAS disks with SSDs SATA disks it became quiet enough for a home workstation. It works very well at that task but the BIOS is quite determined to keep the motherboard video running due to the remote console support. So swapping monitors around was more pain than I felt like going through, I just got it working and left it. I ordered a special GPU power cable but found that the older video card that doesn’t need an extra power cable performs adequately before the cable arrived.

Here is a table comparing the systems.

2560*1440 works well 3840*2160 goes slow
System Dell PowerEdge T320 White Box PC from rubbish
CPU 2.4GHz E5-2440 3.3GHz i5-2500
Video Card ATI Radeon HD 6570 ATI Radeon R7 260X
PCIe Speed PCIe 2.1 – 8GB/s PCIe 3.0 downgraded to PCIe 2.1 – 8GB/s

Conclusion

The ATI Radeon HD 6570 video card is one that I had previously tested and found inadequate for 4K support, I can’t remember if it didn’t work at that resolution or didn’t support more than 30Hz scan rate. If the 2560*1440 monitor dies then it wouldn’t make sense to buy anything less than a 4K monitor to replace it which means that I’d need to get a new video card to match. But for the moment 2560*1440 is working well enough so I won’t upgrade it any time soon. I’ve already got the special power cable (specified as being for a Dell PowerEdge R610 for anyone else who wants to buy one) so it will be easy to install a powerful video card in a hurry.

October 12, 2020

etbeFirst Attempt at Gnocchi-Statsd

I’ve been investigating the options for tracking system statistics to diagnose performance problems. The idea is to track all sorts of data about the system (network use, disk IO, CPU, etc) and look for correlations at times of performance problems. DataDog is pretty good for this but expensive, it’s apparently based on or inspired by the Etsy Statsd. It’s claimed that the gnocchi-statsd is the best implementation of the protoco used by the Etsy Statsd, so I decided to install that.

I use Debian/Buster for this as that’s what I’m using for the hardware that runs KVM VMs. Here is what I did:

# it depends on a local MySQL database
apt -y install mariadb-server mariadb-client
# install the basic packages for gnocchi
apt -y install gnocchi-common python3-gnocchiclient gnocchi-statsd uuid

In the Debconf prompts I told it to “setup a database” and not to manage keystone_authtoken with debconf (because I’m not doing a full OpenStack installation).

This gave a non-working configuration as it didn’t configure the MySQL database for the [indexer] section and the sqlite database that was configured didn’t work for unknown reasons. I filed Debian bug #971996 about this [1]. To get this working you need to edit /etc/gnocchi/gnocchi.conf and change the url line in the [indexer] section to something like the following (where the password is taken from the [database] section).

url = mysql+pymysql://gnocchi-common:PASS@localhost:3306/gnocchidb

To get the statsd interface going you have to install the gnocchi-statsd package and edit /etc/gnocchi/gnocchi.conf to put a UUID in the resource_id field (the Debian package uuid is good for this). I filed Debian bug #972092 requesting that the UUID be set by default on install [2].

Here’s an official page about how to operate Gnocchi [3]. The main thing I got from this was that the following commands need to be run from the command-line (I ran them as root in a VM for test purposes but would do so with minimum privs for a real deployment).

gnocchi-api
gnocchi-metricd

To communicate with Gnocchi you need the gnocchi-api program running, which uses the uwsgi program to provide the web interface by default. It seems that this was written for a version of uwsgi different than the one in Buster. I filed Debian bug #972087 with a patch to make it work with uwsgi [4]. Note that I didn’t get to the stage of an end to end test, I just got it to basically run without error.

After getting “gnocchi-api” running (in a terminal not as a daemon as Debian doesn’t seem to have a service file for it), I ran the client program “gnocchi” and then gave it the “status” command which failed (presumably due to the metrics daemon not running), but at least indicated that the client and the API could communicate.

Then I ran the “gnocchi-metricd” and got the following error:

2020-10-12 14:59:30,491 [9037] ERROR    gnocchi.cli.metricd: Unexpected error during processing job
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gnocchi/cli/metricd.py", line 87, in run
    self._run_job()
  File "/usr/lib/python3/dist-packages/gnocchi/cli/metricd.py", line 248, in _run_job
    self.coord.update_capabilities(self.GROUP_ID, self.store.statistics)
  File "/usr/lib/python3/dist-packages/tooz/coordination.py", line 592, in update_capabilities
    raise tooz.NotImplemented
tooz.NotImplemented

At this stage I’ve had enough of gnocchi. I’ll give the Etsy Statsd a go next.

Update

Thomas has responded to this post [5]. At this stage I’m not really interested in giving Gnocchi another go. There’s still the issue of the indexer database which should be different from the main database somehow and sqlite (the config file default) doesn’t work.

I expect that if I was to persist with Gnocchi I would encounter more poorly described error messages from the code which either don’t have Google hits when I search for them or have Google hits to unanswered questions from 5+ years ago.

The Gnocchi systemd config files are in different packages to the programs, this confused me and I thought that there weren’t any systemd service files. I had expected that installing a package with a daemon binary would also get the systemd unit file to match.

The cluster features of Gnocchi are probably really good if you need that sort of thing. But if you have a small instance (EG a single VM server) then it’s not needed. Also one of the original design ideas of the Etsy Statsd was that UDP was used because data could just be dropped if there was a problem. I think for many situations the same concept could apply to the entire stats service.

If the other statsd programs don’t do what I need then I may give Gnocchi another go.