In the past couple weeks I contributed to a bunch of different open source projects in different ways and I thought I’d write about some of them.
Table of Contents
I switched from Dropbox to Syncthing a while ago and so far it’s been pretty great. I run it on my macOS server in the basement which mirrors everything on its large disks, and also on my various laptops where I selectively sync certain directories that I need.
Dropbox has a feature in its iOS app where it can upload photos taken to Dropbox, which would then sync to my Mac. I’ve mostly replicated this functionality with Syncthing by using the PhotoSync app which has SFTP support. It automatically uploads new photos to my macOS server via SFTP, which Syncthing picks up and can sync to my laptops.
Recently Syncthing gained kqueue support so it doesn’t have to walk the directory structure every n minutes looking for changes. When I opened the Syncthing web interface to add synced photos to my OpenBSD laptop, it prompted me with a “you can use kqueue now, do you want to?” dialog so I enabled it.
Unfortunately a few minutes later, my laptop started getting hot and I noticed that
the syncthing processes were using 100% CPU.
After doing a
I noticed it was caught in a tight loop trying to monitor a new directory, failing
because it was out of file descriptors, and then retrying with no delay, causing
the 100% CPU.
This class of problem was actually one that OpenBSD audited for not too long ago.
System daemons that spun when running out of file descriptors (such as from an
accept) were fixed not to do that as it could cause a remotely triggered DoS.
I opened an issue on the Syncthing issue tracker complaining about the problem, and a fix was committed 5 days later. These kinds of contributions are the easiest to do; just complain about a problem with the proper details, to the right people via the right channel, and they fix it for you.
Crystal gained OpenBSD support long ago, but there is no port for it in the OpenBSD ports tree. I found a work-in-progress and tried to get it to compile, but on my 6.3-current laptop, it would crash while trying to compile its compiler.
After the OpenBSD 6.3 release, the OpenBSD kernel was changed to require that
used for thread stacks be done with a new
This allows the kernel to do enforcement of the use of the allocation and kill
any program that doesn’t do things properly.
This required changes in OpenBSD’s ports of Go,
sbcl, and some others.
Crystal to pass the
MAP_STACK flag on OpenBSD, and submitted a pull request
to the author of the work-in-progress port.
Since Crystal should continue working on OpenBSD out of the box, I submitted a
to the upstream Crystal project for formal
MAP_STACK support, which
was accepted for their 0.25.1 release.
Now that the port would no longer require patches, I
the work-in-progress port for the new 0.25.1 release.
After running Crystal’s
std_spec test suite, I noticed some garbage printed
when running on OpenBSD because it uses an alternative method to determine the
number of CPUs.
I submitted another
to clean this up, which was quickly merged.
These kinds of contributions are time consuming and technical, but as an OpenBSD developer I find it rewarding to get proper OpenBSD support in upstream projects.
At some point during this work, someone submitted a
project to update its Sinatra dependency to fix a security issue in Sinatra.
After merging it, another user
breakage because the Sinatra update pulled in ActiveSupport which somehow broke
.try() functionality I was already using
(which doesn’t make sense because ActiveSupport includes a
Not wanting to fight things in the future or spend much time on it, I just
removed my use of
Maintaining public projects and accepting contributions from others is not one of my strong points (which is partly why I gave up maintaining Lobsters). Users want to take the project in a different direction and add things that I don’t want or care about, and I feel bad telling them “no” just because I don’t like it, not really for any technical reason.
After looking into the OpenBSD
libc code and seeing how other OSes handled it,
I realized it was actually an incompatibility in OpenBSD’s
posted about it
on our tech@ mailing list and another developer helped come up with a fix.
One of our system tools,
was actually relying on the incompatible behavior and would need to be changed
realpath(3) could be fixed in
Martijn and I worked together to develop a fix and I
a few days ago.
Our ports team is doing a full package build of our entire ports tree with the
libc patch to determine any fallout.
Since OpenBSD is behaving differently than most other OSes, hopefully there won’t
be anything else relying on our incompatible behavior.
OpenBSD’s structure shines here; an issue in our
libc was identified, we could
easily see which of our own software relied on the behavior and fix it, and we
could then build the thousands of 3rd party software packages that work on OpenBSD
to identify any fallout before it’s committed.
While testing the
realpath behavior on other OSes, I needed to access my macOS
server through VNC.
OpenBSD’s current VNC port is
which is a modified version of TightVNC.
I noticed that when connecting to my Mac even over my fast network, the window
would draw very slowly and mouse movements would take a few seconds to respond
on the other end.
I had chalked it up to some weird incompatibility between TightVNC’s viewer and macOS’s native VNC server, but since I was on a streak of fixing things that bothered me this week, I spent some time digging into why it was so slow.
vncviewer spit out a warning that it was unable to create a
but continued on its way.
I traced this error in the ssvnc code and thought it may have been some problem in
the MIT-SHM Xorg extension on OpenBSD.
Some hours later debugging Xorg’s
XShm* functions and OpenBSD’s kernel shm
functionality, it all turned out to be for nothing.
The ssvnc package was simply not being compiled with SHM support because its
only enabled it on Linux
(as Theo sarcastically said a decade ago, “all the world is an i386 running
Once I built ssvnc with
-DMITSHM, the window would draw nearly instantly and
my mouse movements reflected in near real-time.
I sent a patch to the ssvnc OpenBSD port maintainer to properly enable
-DMITSHM on OpenBSD, who quickly ok’d it and
I committed it.
Since we’d prefer to have as few patches as possible in our ports tree, I should really send this patch upstream but it seems like the kind of project where the patch will languish for years because nobody is maintaining it anymore.
While doing most of this work, I was using the console-based
audio player to play music (which is synced from my Mac via Syncthing, see how
this is all connected?).
One thing that has always bothered me about
cmus was that it didn’t support
Not many music players do, but iPods and iTunes have always supported it and
I’ve gotten used to it.
I don’t like listening to songs on normal shuffle mode and prefer to listen to an album all the way through from its first track to the last, then skip to another random album by a different artist and continue playing from its first track.
Continuing my streak of digging in and fixing things, I spent an hour or so
reading code and
implementing album shuffle
I’m not sure whether it should be implemented the way I did it, so I’m waiting
for the project maintainer to
before submitting a pull request for merging.
I like these kinds of contributions where I implement something for myself but it’s also something a bunch of other people were waiting for someone to come along and do. I did this when adding configurable margins to iTerm2 and it seemed well received (hi /r/unixporn).