Don't understand german? Read or subscribe to my english-only feed.

What to expect from Debian/trixie #newintrixie

July 20th, 2025

Trixie Banner, Copyright 2024 Elise Couper

Update on 2025-07-28: added note about Debian 13/trixie support for OpenVox (thanks, Ben Ford!)

Debian v13 with codename trixie is scheduled to be published as new stable release on 9th of August 2025.

I was the driving force at several of my customers to be well prepared for the upcoming stable release (my efforts for trixie started in August 2024). On the one hand, to make sure packages we care about are available and actually make it into the release. On the other hand, to ensure there are no severe issues that make it into the release and to get proper and working upgrades. So far everything is looking pretty well and working fine, the efforts seemed to have payed off. :)

As usual with major upgrades, there are some things to be aware of, and hereby I’m starting my public notes on trixie that might be worth for other folks. My focus is primarily on server systems and looking at things from a sysadmin perspective.

Further readings

As usual start at the official Debian release notes, make sure to especially go through What’s new in Debian 13 + issues to be aware of for trixie (strongly recommended read!).

Package versions

As a starting point, let’s look at some selected packages and their versions in bookworm vs. trixie as of 2025-07-20 (mainly having amd64 in mind):

Package bookworm/v12 trixie/v13
ansible 2.14.3 2.19.0
apache 2.4.62 2.4.64
apt 2.6.1 3.0.3
bash 5.2.15 5.2.37
ceph 16.2.11 18.2.7
docker 20.10.24 26.1.5
dovecot 2.3.19 2.4.1
dpkg 1.21.22 1.22.21
emacs 28.2 30.1
gcc 12.2.0 14.2.0
git 2.39.5 2.47.2
golang 1.19 1.24
libc 2.36 2.41
linux kernel 6.1 6.12
llvm 14.0 19.0
lxc 5.0.2 6.0.4
mariadb 10.11 11.8
nginx 1.22.1 1.26.3
nodejs 18.13 20.19
openjdk 17.0 21.0
openssh 9.2p1 10.0p1
openssl 3.0 3.5
perl 5.36.0 5.40.1
php 8.2+93 8.4+96
podman 4.3.1 5.4.2
postfix 3.7.11 3.10.3
postgres 15 17
puppet 7.23.0 8.10.0
python3 3.11.2 3.13.5
qemu/kvm 7.2 10.0
rsync 3.2.7 3.4.1
ruby 3.1 3.3
rust 1.63.0 1.85.0
samba 4.17.12 4.22.3
systemd 252.36 257.7-1
unattended-upgrades 2.9.1 2.12
util-linux 2.38.1 2.41
vagrant 2.3.4 2.3.7
vim 9.0.1378 9.1.1230
zsh 5.9 5.9

Misc unsorted

  • The asterisk package once again didn’t make it into trixie (see #1031046)
  • The new debian-repro-status package provides the identically named command-line tool debian-repro-status to query the reproducibility status of your installed Debian packages
  • The Grml live system project provided further of their packages into Debian. Available as of trixie are now also grml-keyring (OpenPGP certificates used for signing the Grml repositories), grml-hwinfo (a tool which collects information of the hardware ) + grml-paste (command line interface for paste.debian.net)
  • If you use pacemaker, be aware that its fence-agents package is now a transitional package. All the fence-agents got split into separate packages (fence-agents-$whatever). If you want to have all the fence-agents available, make sure to install the fence-agents-all package. If you have Recommends disabled, you definitely should be aware of this.
  • usrmerge is finalized (also see dpkg warning issue in release notes)
  • For an overview of the XMPP/Jabber situation in trixie see xmpp-team’s blog post
  • The curl package now includes the wcurl command line tool, being a simple wrapper around curl to easily download files

apt

The new apt version 3.0 brings several new features, including:

  • support for colors (f.e. green for installs/upgrades, yellow for downgrades, red for removals, can be disabled via ‐‐no-color, APT_NO_COLOR=1 or NO_COLOR=1 and customized via e.g. APT::Color::Action::Install “cyan”)
  • organizes output in more readable sections and shows removals more prominently
  • uses sequoia to verify signatures
  • includes a new solver
  • the new apt modernize-sources command converts /etc/apt/sources.list.d/*.list files into the new .sources format (AKA DEB822)
  • the new apt distclean command removes all files under $statedir/lists except Release, Release.gpg, and InRelease (it can be used for example, when finalizing images distributed to users)
  • new configuration option APT::NeverAutoRemove::KernelCount for keeping a configurable amount of kernels, f.e. setting APT::NeverAutoRemove::KernelCount 3 will keep 3 kernels (including the running, and most recent)
  • new command line option ‐‐snapshot, and configuration option APT::Snapshot, controlling the snapshot chosen for archives with Snapshot: enable
  • new command line option ‐‐update to run the update command before the specified command, like apt ‐‐update install zsh,
    apt ‐‐update remove foobar or apt ‐‐update safe-upgrade
  • apt-key is gone, and there’s no replacement for it available (if you need an interface for listing present keys)

systemd

systemd got upgraded from v252.36-1~deb12u1 to 257.7-1 and there are lots of changes.

Be aware that systemd v257 has a new net.naming_scheme, v257 being PCI slot number is now read from firmware_node/sun sysfs file. The naming scheme based on devicetree aliases was extended to support aliases for individual interfaces of controllers with multiple ports. This might affect you, see e.g. #1092176 and #1107187, the Debian Wiki provides further useful information.

There are new systemd tools available:

  • run0: temporarily and interactively acquire elevated or different privileges (serves a similar purpose as sudo)
  • systemd-ac-power: Report whether we are connected to an external power source
  • systemd-confext: Activates System Extension Images
  • systemd-vpick: Resolve paths to ‘.v/’ versioned directories
  • varlinkctl: Introspect with and invoke Varlink services

The tools provided by systemd gained several new options:

  • busctl: new option ‐‐limit‐messages=NUMBER (Stop monitoring after receiving the specified number of message)
  • hostnamectl: new option ‐j (same as ‐‐json=pretty on tty, ‐‐json=short otherwise)
  • journalctl: new options ‐‐image‐policy=POLICY (Specify disk image dissection policy), ‐‐invocation=ID (Show logs from the matching invocation ID), ‐I (Show logs from the latest invocation of unit), ‐‐exclude-identifier=STRING (Hide entries with the specified syslog identifier),‐‐truncate-newline (Truncate entries by first newline character), ‐‐list-invocations (Show invocation IDs of specified unit), ‐‐list-namespaces (Show list of journal namespaces)
  • kernel-install: new commands add‐all + list and plenty of new command line options
  • localectl: new option ‐‐full (Do not ellipsize output)
  • loginctl: new options ‐‐json=MODE (Generate JSON output for list-sessions/users/seats) + ‐j (Same as ‐‐json=pretty on tty, ‐‐json=short otherwise)
  • networkctl: new commands edit FILES|DEVICES… (Edit network configuration files), cat [FILES|DEVICES…] (Show network configuration files), mask FILES… (Mask network configuration files) + unmask FILES… (Unmask network configuration files) + persistent-storage BOOL (Notify systemd-networkd if persistent storage is ready), and new options ‐‐no-ask-password (Do not prompt for password), ‐‐no-reload (Do not reload systemd-networkd or systemd-udevd after editing network config), ‐‐drop-in=NAME (Edit specified drop-in instead of main config file), ‐‐runtime (Edit runtime config files) + ‐‐stdin (Read new contents of edited file from stdin)
  • systemctl” new commands list-paths [PATTERN] (List path units currently in memory, ordered by path), whoami [PID…] (Return unit caller or specified PIDs are part of), soft-reboot (Shut down and reboot userspace) + sleep (Put the system to sleep), and new options ‐‐capsule=NAME (Connect to service manager of specified capsule), ‐‐before (Show units ordered before with ‘list-dependencies’), ‐‐after (Show units ordered after with ‘list-dependencies’), ‐‐kill-value=INT (Signal value to enqueue), ‐‐no-warn (Suppress several warnings shown by default), ‐‐message=MESSAGE (Specify human readable reason for system shutdown), ‐‐image‐policy=POLICY (Specify disk image dissection policy), ‐‐reboot‐argument=ARG (Specify argument string to pass to reboot()), ‐‐drop-in=NAME (Edit unit files using the specified drop-in file name), ‐‐when=TIME (Schedule halt/power-off/reboot/kexec action after a certain timestamp) + ‐‐stdin (Read
    new contents of edited file from stdin)
  • systemd-analyze” new commands architectures [NAME…] (List known architectures), smbios11 (List strings passed via SMBIOS Type #11), image-policy POLICY… (Analyze image policy string), fdstore SERVICE… (Show file descriptor store contents of service), malloc [D-BUS SERVICE…] (Dump malloc stats of a D-Bus service), has-tpm2 (Report whether TPM2 support is available), pcrs [PCR…] (Show TPM2 PCRs and their names) + srk [>FILE] (Write TPM2 SRK (to FILE)) and new options ‐‐no-legend (Disable column headers and hints in plot with either ‐‐table or ‐‐json=), ‐‐instance=NAME (Specify fallback instance name for template units), ‐‐unit=UNIT (Evaluate conditions and asserts of unit), ‐‐table (Output plot’s raw time data as a table), ‐‐scale-svg=FACTOR (Stretch x-axis of plot by FACTOR (default: 1.0)), ‐‐detailed (Add more details to SVG plot), ‐‐tldr (Skip comments and empty lines), ‐‐image
    -policy=POLICY (Specify disk image dissection policy) + ‐‐mask (Parse parameter as numeric capability mask)
  • systemd-ask-password: new options ‐‐user (Ask only our own user’s agents) + ‐‐system (Ask agents of the system and of all users)
  • systemd-cat: new option ‐‐namespace=NAMESPACE (Connect to specified journal namespace)
  • systemd-creds: new options ‐‐user (Select user-scoped credential encryption), ‐‐uid=UID (Select user for scoped credentials) + ‐‐allow-null (Allow decrypting credentials with empty key)
  • systemd-detect-virt: new options ‐‐cvm (Only detect whether we are run in a confidential VM) + ‐‐list-cvm (List all known and detectable types of confidential virtualization)
  • systemd-firstboot: new options ‐‐image-policy=POLICY (Specify disk image dissection policy), ‐‐kernel-command-line=CMDLINE (Set kernel command line) + ‐‐reset (Remove existing files)
  • systemd-id128: new commands var-partition-uuid (Print the UUID for the /var/ partition) + show [NAME|UUID] (Print one or more UUIDs), and new options ‐‐no-pager (Do not pipe output into a pager), ‐‐no-legend (Do not show the headers and footers), ‐‐json=FORMAT (Output inspection data in JSON), ‐j (Equivalent to ‐‐json=pretty (on TTY) or ‐‐json=short (otherwise)) + ‐P ‐‐value (Only print the value)
  • systemd-inhibit: new option ‐‐no-ask-password (Do not attempt interactive authorization)
  • systemd-machine-id-setup: new option ‐‐image-policy=POLICY (Specify disk image dissection policy)
  • systemd-mount: new options ‐‐json=pretty|short|off (Generate JSON output) + ‐‐tmpfs (Create a new tmpfs on the mount point)
  • systemd-notify: new options ‐‐reloading (Inform the service manager about configuration reloading), ‐‐stopping (Inform the service manager about service shutdown), ‐‐exec (Execute command line separated by ‘;’ once done), ‐‐fd=FD (Pass specified file descriptor with along with message) + ‐‐fdname=NAME (Name to assign to passed file descriptor(s))
  • systemd-path: new option ‐‐no-pager (Do not pipe output into a pager)
  • systemd-run: new options ‐‐expand-environment=BOOL (Control expansion of environment variables), ‐‐json=pretty|short|off (Print unit name and invocation id as JSON), ‐‐ignore-failure (Ignore the exit status of the invoked process) + ‐‐background=COLOR (Set ANSI color for background)
  • systemd-sysext: new options ‐‐mutable=yes|no|auto|import|ephemeral|ephemeral-import (Specify a mutability mode of the merged hierarchy), ‐‐no-reload (Do not reload the service manager), ‐‐image-policy=POLICY (Specify disk image dissection policy) + ‐‐noexec=BOOL (Whether to mount extension overlay with noexec)
  • systemd-sysusers: new options ‐‐tldr (Show non-comment parts of configuration) + ‐‐image-policy=POLICY (Specify disk image dissection policy)
  • systemd-tmpfiles: new command ‐‐purge(Delete files and directories marked for creation in specified configuration files (careful!)), and new options ‐‐user (Execute user configuration), ‐‐tldr (Show non-comment parts of configuration files), ‐‐graceful (Quietly ignore unknown users or groups), ‐‐image-policy=POLICY (Specify disk image dissection policy) + ‐‐dry-run (Just print what would be done)
  • systemd-umount: new options ‐‐json=pretty|short|off (Generate JSON output) + ‐‐tmpfs (Create a new tmpfs on the mount point)
  • timedatectl: new commands ntp-servers INTERFACE SERVER (Set the interface specific NTP servers) + revert INTERFACE (Revert the interface specific NTP servers) and new option ‐P NAME (Equivalent to ‐‐value ‐‐property=NAME)

Debian’s systemd ships new binary packages:

  • systemd-boot-efi-amd64-signed (Tools to manage UEFI firmware updates (signed))
  • systemd-boot-tools (simple UEFI boot manager – tools)
  • systemd-cryptsetup (Provides cryptsetup, integritysetup and veritysetup utilities)
  • systemd-netlogd (journal message forwarder)
  • systemd-repart (Provides the systemd-repart and systemd-sbsign utilities)
  • systemd-standalone-shutdown (standalone shutdown binary for use in exitrds)
  • systemd-ukify (tool to build Unified Kernel Images)

Linux Kernel

The trixie release ships a Linux kernel based on latest longterm version 6.12. As usual there are lots of changes in the kernel area, including better hardware support, and this might warrant a separate blog entry. To highlight some changes with Debian trixie:

See Kernelnewbies.org for further changes between kernel versions.

Configuration management

For puppet users, Debian provides the puppet-agent (v8.10.0), puppetserver (v8.7.0) and puppetdb (v8.4.1) packages. Puppet’s upstream does not provide packages for trixie, yet. Given how long it took them for Debian bookworm, and with their recent Plans for Open Source Puppet in 2025, it’s unclear when (and whether at all) we might get something. As a result of upstream behavior, also the OpenVox project evolved, and they already provide Debian 13/trixie support (https://apt.voxpupuli.org/openvox8-release-debian13.deb). FYI: the AIO puppet-agent package for bookworm (v7.34.0-1bookworm) so far works fine for me on Debian/trixie. Be aware that due to the apt-key removal you need a recent version of the puppetlabs-apt for usage with trixie. The puppetlabs-ntp module isn’t yet ready for trixie (regarding ntp/ntpsec), if you should depend on that.

ansible is available and made it with version 2.19 into trixie.

Prometheus stack

Prometheus server was updated from v2.42.0 to v2.53, and all the exporters that got shipped with bookworm are still around (in more recent versions of course). Trixie gained some new exporters:

Virtualization

docker (v26.1.5), ganeti (v3.1.0), libvirt (v11.3.0, be aware of significant changes to libvirt packaging), lxc (v6.0.4), podman (v5.4.2), openstack (see openstack-team on Salsa), qemu/kvm (v10.0.2), xen (v4.20.0) are all still around.

Proxmox already announced their PVE 9.0 BETA, being based on trixie and providing 6.14.8-1 kernel, QEMU 10.0.2, LXC 6.0.4, OpenZFS 2.3.3.

Vagrant is available in version 2.3.7, but Vagrant upstream does not provide packages for trixie yet. Given that HashiCorp adopted the BSL, the future of vagrant in Debian is unclear.

If you’re relying on VirtualBox, be aware that upstream doesn’t provide packages for trixie, yet. VirtualBox is available from Debian/unstable (version 7.1.12-dfsg-1 as of 2025-07-20), but not shipped with stable release since quite some time (due to lack of cooperation from upstream on security support for older releases, see #794466). Be aware that starting with Linux kernel 6.12, KVM initializes virtualization on module loading by default. This prevents VirtualBox VMs from starting. In order to avoid this, either add “kvm.enable_virt_at_load=0” parameter into kernel command line or unload the corresponding kvm_intel / kvm_amd module.

If you want to use Vagrant with VirtualBox on trixie, be aware that Debian’s vagrant package as present in trixie doesn’t support the VirtualBox package version 7.1 as present in Debian/unstable (manually patching vagrant’s meta.rb and rebuilding the package without Breaks: virtualbox (>= 7.1) is known to be working).

util-linux

The are plenty of new options available in the tools provided by util-linux:

  • blkdiscard: new option ‐‐quiet (suppress warning messages)
  • blockdev: new options ‐‐getdiskseq (get disk sequence number) + ‐‐getzonesz (get zone size)
  • dmesg: new option ‐‐kmsg-file … (use the file in kmsg format), new ‐‐time-format … argument ‘raw’
  • findmnt: new options ‐‐list-columns (list the available columns), ‐‐dfi (imitate the output of df(1) with -i option), ‐‐id … (filter by mount node ID), ‐‐filter … (apply display filter) + ‐‐uniq-id … (filter by
    mount node 64-bit ID)
  • fstrim: new option -types …. (limit the set of filesystem types)
  • hardlink: new options ‐‐respect-dir (directory names have to be identical), ‐‐exclude-subtree … (regular expression to exclude directories), ‐‐prioritize-trees (files found in the earliest specified top-level directory have higher priority), ‐‐list-duplicates (print every group of duplicate files), ‐‐mount (stay within the same filesystem) + ‐‐zero (delimit output with NULs instead of newlines)
  • ipcmk: new options ‐‐posix-shmem … (create POSIX shared memory segment of size), ‐‐posix-semaphore … (create POSIX semaphore), ‐‐posix-mqueue … (create POSIX message queue) + ‐‐name … (name of the POSIX resource)
  • ipcrm: new options ‐‐posix-shmem … (remove POSIX shared memory segment by name), ‐‐posix-mqueue … (remove POSIX message queue by name), ‐‐posix-semaphore (remove POSIX semaphore by name) + ‐‐all=… (remove all in specified category)
  • lsblk: new options ‐‐ct-filter … (restrict the next counter), ‐‐ct … (define a custom counter), ‐‐highlight … (colorize lines matching the expression), ‐‐list-columns (list the available columns), ‐‐nvme (output info about NVMe devices), ‐‐properties-by … (methods used to gather data), ‐‐filter … (print only lines matching the expression), ‐‐virtio (output info about virtio devices)
  • lscpu: new options ‐‐raw (use raw output format (for -e, -p and -C)) + ‐‐hierarchic=… (use subsections in summary (auto, never, always))
  • lsipc: new options ‐‐posix-shmems (POSIX shared memory segments), ‐‐posix-mqueues (POSIX message queues), ‐‐posix-semaphores (POSIX semaphores), ‐‐name … (POSIX resource identified by name)
  • lslocks: new option ‐‐list-columns (list the available columns)
  • lslogins: new option ‐‐lastlog2 … (set an alternate path for lastlog2)
  • lsns: new options ‐‐persistent (namespaces without processes), ‐‐filter … (apply display filter) + ‐‐list-columns (list the available columns)
  • mkswap: new options ‐‐endianness=… (specify the endianness to use (native, little or big)), ‐‐offset … (specify the offset in the device), ‐‐size … (specify the size of a swap file in bytes) + ‐‐file (create a swap file)
  • namei: ‐‐context (print any security context of each file)
  • nsenter: new options ‐‐net-socket … (enter socket’s network namespace), ‐‐user-parent (enter parent user namespace), ‐‐keep-caps (retain capabilities granted in user namespaces), ‐‐env (inherit environment variables from tar get process) + ‐‐join-cgroup (join the cgroup of the target process)
  • runuser: new option ‐‐no-pty (do not create a new pseudo-terminal)
  • setarch: new option ‐‐show=… (show current or specific personality and exit)
  • setpriv: new options ‐‐ptracer … (allow ptracing from the given process), ‐‐landlock-access … (add Landlock access), ‐‐landlock-rule … (add Landlock rule) + ‐‐seccomp-filter … (load seccomp filter from file)
  • su: new option ‐‐no-pty (do not create a new pseudo-terminal)
  • unshare: new option ‐‐load-interp … ( load binfmt definition in the namespace)
  • whereis: new option -g (interpret name as glob (pathnames pattern))
  • wipefs: new argument option feature for ‐‐backup=… option to specify directory (instead of default $HOME)
  • zramctl: new option ‐‐algorithm-params … (algorithm parameters to use)

Now no longer present in util-linux as of trixie:

  • addpart (tell the kernel about the existence of a specified partition): use partx instead
  • delpart (tell the kernel to forget about a specified partition): use partx instead
  • last (show a listing of last logged in users, binary got moved to wtmpdb), lastb (show a listing of last logged in users), mesg (control write access of other users to your terminal), utmpdump (dump UTMP and WTMP files in raw format): see Debian release notes for details

The following binaries got moved from util-linux to the util-linux-extra package:

  • ctrlaltdel (set the function of the Ctrl-Alt-Del combination)
  • mkfs.bfs (make an SCO bfs filesystem)
  • fsck.cramfs + mkfs.cramfs (compressed ROM file system)
  • fsck.minix + mkfs.minix (Minix filesystem)
  • resizepart (tell the kernel about the new size of a partition)

And the util-linux-extra package also provides new tools:

  • bits: convert bit masks from/to various formats
  • blkpr: manage persistent reservations on a device
  • coresched: manage core scheduling cookies for tasks
  • enosys: utility to make syscalls fail with ENOSYS
  • exch: atomically exchanges paths between two files
  • fadvise: utility to use the posix_fadvise system call
  • pipesz: set or examine pipe buffer sizes and optionally execute command.
  • waitpid: utility to wait for arbitrary processes

OpenSSH

OpenSSH was updated from v9.2p1 to 10.0p1-5, so if you’re interested in all the changes, check out the release notes between those versions (9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9 + 10.0).

Let’s highlight some notable behavior changes in Debian:

There are some notable new features:

  • allow forwarding Unix Domain sockets via ssh -W
  • OpenSSH penalty behavior: visit my separate blog post for more details
  • add support for reading ED25519 private keys in PEM PKCS8 format. Previously only the OpenSSH private key format was supported.
  • the new hybrid post-quantum algorithm mlkem768x25519-sha256 (based on the FIPS 203 Module-Lattice Key Encapsulation mechanism (ML-KEM) combined with X25519 ECDH) is now used by default for key agreement. This algorithm is considered to be safe against attack by quantum computers, is guaranteed to be no less strong than the popular curve25519-sha256 algorithm, has been standardised by NIST and is considerably faster than the previous default.
  • the ssh-agent will now delete all loaded keys when signaled with SIGUSR1. This allows deletion of keys without having access to $SSH_AUTH_SOCK.
  • support systemd-style socket activation in ssh-agent using the LISTEN_PID/LISTEN_FDS mechanism. Activated when these environment variables are set, the agent is started with the -d or -D option and no socket path is set.
  • add a sshd -G option that parses and prints the effective configuration without attempting to load private keys and perform other checks. (This allows usage of the option before keys have been generated and for configuration evaluation and verification by unprivileged users.)
  • add support for configuration tags to ssh(1). This adds a ssh_config(5) “Tag” directive and corresponding “Match tag” predicate that may be used to select blocks of configuration.
  • add a “match localnetwork” predicate. This allows matching on the addresses of available network interfaces and may be used to vary the effective client configuration based on network location.
  • add a %j token that expands to the configured ProxyJump hostname
  • add support for “Match sessiontype” to ssh_config. Allows matching on the type of session initially requested, either “shell” for interactive sessions, “exec” for command execution sessions, “subsystem” for subsystem requests, such as sftp, or “none” for transport/forwarding-only sessions.
  • allow glob(3) patterns to be used in sshd_config AuthorizedKeysFile and AuthorizedPrincipalsFile directives.

Thanks to everyone involved in the release, looking forward to trixie + and happy upgrading!
Let’s continue with working towards Debian/forky. :)

Grml 2025.05 – codename Nudlaug

May 16th, 2025

Debian hard freeze on 2025-05-15? We bring you a new Grml release on top of that! 2025.05 🚀 – codename Nudlaug.

There’s plenty of new stuff, check out our official release announcement for all the details. But I’d like to highlight one feature that I particularly like: SSH service announcement with Avahi. The grml-full flavor ships Avahi, and when you enable SSH, it automatically announces the SSH service on your local network. So when f.e. booting Grml with boot option `ssh=debian`, you should be able to login on your Grml live system with `ssh grml@grml.local` and password ‘debian‘:

% insecssh grml@grml.local
Warning: Permanently added 'grml.local' (ED25519) to the list of known hosts.
grml@grml.local's password: 
Linux grml 6.12.27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.27-1 (2025-05-06) x86_64
Grml - Linux for geeks

grml@grml ~ %

Hint: grml-zshrc provides that useful shell alias `insecssh`, which is aliased to `ssh -o “StrictHostKeyChecking=no” -o “UserKnownHostsFile=/dev/null”`. Using those options, you aren’t storing the SSH host key of the (temporary) Grml live system (permanently) in your UserKnownHostsFile.

BTW, you can run `avahi-browse -d local _ssh._tcp –resolve -t` to discover the SSH services on your local network. 🤓

Happy Grml-ing!

HTU Bigband Konzert am 05.06.2025

May 16th, 2025

Plakat für das HTU Bigband-Konzert am 05.06.2025

Wie letztes Jahr schon, spielen wir auch heuer wieder mit der HTU-Bigband ein Konzert an der TU Graz.

Und zwar am Donnerstag, 5. Juni 2025! Das Konzert startet um 19:30 Uhr, bei Schönwetter im Innenhof der TU Graz (Alte Technik, Rechbauerstraße 12, 8010 Graz), und bei Schlechtwetter geht es an der gleichen Adresse in den Hörsaal 2. Wir sind über 25 Musikerinnen und Musiker und haben ein anspruchsvolles Programm, von Swing, über Soul, Funk und Latin bis Pop ist alles dabei. Es gibt über 2 Stunden Musik vom Feinsten, die Set-List ist spitze, und das Ganze bei freiem Eintritt.

Ich freue mich schon tierisch darauf und würde mich wieder über bekannte und gut gelaunte Gesichter freuen. Ich hoffe man sieht und hört sich! :-)

Lessons learned from running an open source project for 20 years @ GLT25

April 23rd, 2025

Time flies by so quickly, it’s >20 years since I started the Grml project.

I’m giving a (german) talk about the lessons learned from 20 years of running the Grml project this Saturday, 2025-04-26 at the Grazer Linuxtage (Graz/Austria). Would be great to see you there!

OpenSSH penalty behavior in Debian/trixie #newintrixie

April 13th, 2025

This topic came up at a customer of mine in September 2024, when working on Debian/trixie support. Since then I wanted to blog about it to make people aware of this new OpenSSH feature and behavior. I finally found some spare minutes at Debian’s BSP in Vienna, so here we are. :)

Some of our Q/A jobs failed to run against Debian/trixie, in the debug logs we found:

debug1: kex_exchange_identification: banner line 0: Not allowed at this time

This Not allowed at this time pointed to a new OpenSSH feature. OpenSSH introduced options to penalize undesirable behavior with version 9.8p1, see OpenSSH Release Notes, and also sshd source code.

FTR, on the SSH server side, you’ll see messages like that:

Apr 13 08:57:11 grml sshd-session[2135]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55792 ssh2 [preauth]
Apr 13 08:57:11 grml sshd-session[2135]: Disconnecting authenticating user root 10.100.15.42 port 55792: Too many authentication failures [preauth]
Apr 13 08:57:12 grml sshd-session[2137]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55800 ssh2 [preauth]
Apr 13 08:57:12 grml sshd-session[2137]: Disconnecting authenticating user root 10.100.15.42 port 55800: Too many authentication failures [preauth]
Apr 13 08:57:13 grml sshd-session[2139]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55804 ssh2 [preauth]
Apr 13 08:57:13 grml sshd-session[2139]: Disconnecting authenticating user root 10.100.15.42 port 55804: Too many authentication failures [preauth]
Apr 13 08:57:13 grml sshd-session[2141]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55810 ssh2 [preauth]
Apr 13 08:57:13 grml sshd-session[2141]: Disconnecting authenticating user root 10.100.15.42 port 55810: Too many authentication failures [preauth]
Apr 13 08:57:13 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55818 on [10.100.15.230]:22 penalty: failed authentication
Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55824 on [10.100.15.230]:22 penalty: failed authentication
Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55838 on [10.100.15.230]:22 penalty: failed authentication
Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55854 on [10.100.15.230]:22 penalty: failed authentication

This feature certainly is useful and has its use cases. But if you f.e. run automated checks to ensure that specific logins aren’t working, be careful: you might hit the penalty feature, lock yourself out but also consecutive checks then don’t behave as expected. Your login checks might fail, but only because the penalty behavior kicks in. The login you’re verifying still might be working underneath, but you don’t actually check for it exactly. Furthermore legitimate traffic from systems which accept connections from many users or behind shared IP addresses, like NAT and proxies could be denied.

To disable this new behavior, you can set PerSourcePenalties no in your sshd_config, but there are also further configuration options available, see PerSourcePenalties and PerSourcePenaltyExemptList settings in sshd_config(5) for further details.

Mein Lesejahr 2024

December 31st, 2024

Foto der hier vorgestellten Bücher

Mein Lesejahr 2024 war mit durchschnittlich einem Buch pro Woche ähnlich wie 2023. Mein Best-Of der von mir 2024 fertig gelesenen Bücher (jene die ich besonders lesenswert fand bzw. empfehlen möchte, die Reihenfolge entspricht dem Foto und stellt keinerlei Reihung dar):

Mein SuB bzw. Lesestapel für 2025 ist bereits gut gefüllt, ich freue mich aber trotzdem über etwaige Leseempfehlungen. Ebenso freue ich mich über Feedback, wenn jemand ein Buch aufgrund dieses Beitrags hier gelesen hat.

Grml 2024.12 – codename Adventgrenze

December 20th, 2024

Picture with metrics of three user profiles on GitHub.com, with many contributions especially in the last quarter of the year

We did it again™! Just in time, we’re excited to announce the release of Grml stable version 2024.12, code-named ‘Adventgrenze’! (If you’re not familiar with Grml, it’s a Debian-based live system tailored for system administrators.)

This new release is built on Debian trixie, and for the first time, we’re introducing support for 64-bit ARM CPUs (arm64 architecture)!

I’m incredibly proud of the hard work that went into this release. A significant amount of behind-the-scenes effort went into reworking our infrastructure and redesigning the build process. Special thanks to Chris and Darsha – our Grml developer days in November and December were a blast!

For a detailed overview of the changes between releases 2024.02 and 2024.12, check out our official release announcement. And, as always, after a release comes the next one – exciting improvements are already in the works!

BTW: recently we also celebrated 20(!) years of Grml Releases. If you’re a Grml and or grml-zsh user, please join us in celebrating and send us a postcard!

HTU Bigband Konzert am 04.06.2024

May 28th, 2024

Plakat für das HTU Bigband-Konzert am 04.06.2024

Am Dienstag den 4. Juni spielen wir ab 19:30 Uhr unser nächstes HTU-Bigband-Konzert. Das Konzert findet im Innenhof der TU Graz (Alte Technik, Rechbauerstraße 12, 8010 Graz) statt, bei Schlechtwetter geht es an der gleichen Adresse in den wunderbaren Hörsaal 2. Wir sind über 25 Musikerinnen und Musiker und haben ein anspruchsvolles Programm – von Swing, über Soul, Funk und Latin bis Pop ist alles dabei. Es gibt über 2 Stunden Musik vom Feinsten, und das Ganze bei freiem Eintritt.

Das Event gibt es auch auf Facebook und Instagram zum “Liken” und Weiterverteilen.

Ich freue mich schon tierisch darauf und würde mich über bekannte Gesichter freuen. Ich hoffe man sieht und hört sich! :-)

Being a Debian Developer since 15 years

May 28th, 2024

15 years ago I became an official Debian Developer. Incredible how time flies.

Vortrag: We got hacked: Lektionen aus realen Security-Vorfällen @ GLT24

April 7th, 2024

Auf den Grazer Linuxtagen 2024 (GLT24) war ich als Referent mit einem Vortrag zum Thema “We got hacked: Lektionen aus realen Security-Vorfällen” vertreten. In meinem Vortrag gibt es einen Einblick in reale Security-Incidents und welche Lektionen sich aus solchen Vorfällen mitnehmen lassen.

Es gibt den Vortrag dank des fantastischen c3voc-Teams bereits als Videomitschnitt online. Meine Vortragsfolien (2.1MB, PDF) stehen ebenfalls online zur Verfügung. Viel Spaß beim Anschauen!

Mein Lesejahr 2023

January 3rd, 2024

Foto der hier vorgestellten Bücher

Ich habe auch 2023 keine Bookdumps geschrieben (zu viel Aufwand), darum gibt es auch diesmal wieder (siehe Lesejahr 2022 für die letzte Ausgabe) eine Art Best-Of der von mir 2023 fertig gelesenen Bücher, also jene die ich besonders lesenswert fand bzw. empfehlen möchte (die Reihenfolge entspricht dem Foto und stellt keinerlei Reihung oder dergleichen dar):

  • Das glücklichste Volk: Sieben Jahre bei den Pirahã-Indianern am Amazonas, Daniel L. Everett. Diese Buch ist eine Empfehlung von Khaled Hakami, der u.a. im Erklär-mir-die-Welt-Podcast zum Thema Erklär mir Jäger und Sammle zu Gast war. Das Buch ist eine große Empfehlung speziell für all jene Leute, die sich für andere Kulturen interessieren. Es ist eines der horizonterweiterndsten Bücher, das ich in den letzten Jahren gelesen habe.
  • Kleine Probleme, Nele Pollatschek. Der letzte Tag des Jahres, eine To-do-Liste, unerledigte Dinge und die Sehnsucht nach Ordnung. Die Autorin schreibt aus Sicht von Lars, Familienvater und Endvierziger ein unterhaltsames Buch.
  • Gentleman über Bord, Herbert Clyde Lewis. Ein Roman aus dem Jahr 1937, in dem der Protagonist Henry bei einer Schiffsreise bei einem Missgeschick über Bord geht, und die restliche Besatzung des Schiffes jede Menge Ausreden für sich (er)findet, um das Verschwinden des Passagiers Henry zu entschuldigen. Bedeutungstief und zeitlos.
  • The Undoing Project: A Friendship That Changed Our Minds, Michael Lewis. Ein Buch das die Freundschaft und Lebensweg von Daniel Kahneman (bekannt u.a. für das Buch “Thinking, Fast and Slow”) und Amos Tversk beleuchtet, jene Herren die u.a. für die Prospect Theory bekannt sind. Ich wurde auf das Buch über die wunderbare Rezension von Kathrin Passig aufmerksam, und mir hat das Buch voll zugesagt (auch wenn ein Lesebuddy zurecht anmerkte, dass man kein Problem mit amerikanischen Journalisten als Autor wie auch ein bisserl Drama haben sollte, mich hat beides nicht gestört). Für mich war das Buch insgesamt sehr gut gemacht, es gab einige interessante Stellen und diente mir als Erinnerung, dass ich die Werke von Kahneman (wieder) mal (fertig)lesen sollte.
  • Mir geht’s gut, wenn nicht heute, dann morgen, Dirk Stermann. Eine wunderbare Mischung aus ernsten Themen und Schmäh.
  • Vati, Monika Helfer. Eine Fortsetzung ihrer eigenen Familiengeschichte, ich mag die schlichte und trotzdem berührende Sprache.
  • Eigentum, Wolf Haas. Ich habe Haas erst 2022 für mich entdeckt, er ist einer meiner Lieblingsautoren und ich bin seither auf dem Weg alles von ihm zu lesen. Auch Eigentum ist ein wunderschönes Buch, in dem Haas von seiner sterbenden Mutter schreibt. Sprachkünstler, Hilfsausdruck!
  • Arbeit und Struktur, Wolfgang Herrndorf. Los ging es mit dem Schließen einer Bildungslücke: Tschick vom selbigen Autor hat mich dermaßen reingezogen, dass ich auch endlich mal dessen Arbeit und Struktur angefangen habe, ein Buch das Clemens J. Setz im Podcast “Das Lesen der Anderen” empfohlen hat. Das Buch ist die Autobiografie der letzten Lebensjahre des Autors. Ursprünglich als Blog aufgesetzt nachdem der Autor die Diagnose Hirntumor bekommen hat, wurde sein digitales Tagebuch dann in Papier- und Buchform gebracht. Man möge sich von dem vielleicht etwas sperrigen Titel nicht aufhalten lassen. Ein beeindruckendes, bewegendes und großartiges Buch.
  • Die Herrenausstatterin, Mariana Leky. Ich bin ein Fan von Mariana Leky und habe über die letzten Jahre regelmäßig Bücher von ihr gelesen, so auch dieses wunderbare Buch über eine Dreiecksgeschichte.

Ich freue mich übrigens über Feedback, wenn jemand von euch ein Buch aufgrund dieses Beitrags hier gelesen oder selbst Lese-Empfehlungen für mich hat.

Postfix failing with “no shared cipher”

September 25th, 2023

I’m one of the few folks left who run and maintain mail servers. Recently I had major troubles receiving mails from the mail servers used by a bank, and when asking my favourite search engine, I’m clearly not the only one who ran into such an issue. Actually, I should have checked off the issue and not become a customer at that bank, but the tech nerd in me couldn’t resist getting to the bottom of the problem. Since I got it working and this might be useful for others, here we are. :)

I was trying to get an online banking account set up, but the corresponding account creation mail didn’t arrive me, at all. Looking at my mail server logs, my postfix mail server didn’t accept the mail due to:

postfix/smtpd[3319640]: warning: TLS library problem: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../ssl/statem/statem_srvr.c:2283:
postfix/smtpd[3319640]: lost connection after STARTTLS from mx01.arz.at[193.110.182.61]

Huh, what’s going on here?! Let’s increase the TLS loglevel (setting smtpd_tls_loglevel = 2) and retry. But how can I retry receiving yet another mail? Luckily, on the registration website of the bank there was a URL available, that let me request a one-time password. This triggered another mail, so I did that and managed to grab this in the logs:

postfix/smtpd[3320018]: initializing the server-side TLS engine
postfix/tlsmgr[3320020]: open smtpd TLS cache btree:/var/lib/postfix/smtpd_scache
postfix/tlsmgr[3320020]: tlsmgr_cache_run_event: start TLS smtpd session cache cleanup
postfix/smtpd[3320018]: connect from mx01.arz.at[193.110.182.61]
postfix/smtpd[3320018]: setting up TLS connection from mx01.arz.at[193.110.182.61]
postfix/smtpd[3320018]: mx01.arz.at[193.110.182.61]: TLS cipher list "aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH"
postfix/smtpd[3320018]: SSL_accept:before SSL initialization
postfix/smtpd[3320018]: SSL_accept:before SSL initialization
postfix/smtpd[3320018]: SSL3 alert write:fatal:handshake failure
postfix/smtpd[3320018]: SSL_accept:error in error
postfix/smtpd[3320018]: SSL_accept error from mx01.arz.at[193.110.182.61]: -1
postfix/smtpd[3320018]: warning: TLS library problem: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../ssl/statem/statem_srvr.c:2283:
postfix/smtpd[3320018]: lost connection after STARTTLS from mx01.arz.at[193.110.182.61]
postfix/smtpd[3320018]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 starttls=0/1 commands=1/2
postfix/smtpd[3320018]: connect from mx01.arz.at[193.110.182.61]
postfix/smtpd[3320018]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2

Ok, so this TLS cipher list “aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH” looked like the tls_medium_cipherlist setting in postfix, but which ciphers might we expect? Let’s see what their SMTP server would speak to us:

% testssl --cipher-per-proto -t=smtp mx01.arz.at:25
[...]
Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
SSLv3
TLS 1
TLS 1.1
TLS 1.2
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 256   AESGCM      256      TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 xc028   ECDHE-RSA-AES256-SHA384           ECDH 256   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 xc014   ECDHE-RSA-AES256-SHA              ECDH 256   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384
 x3d     AES256-SHA256                     RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA256
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 256   AESGCM      128      TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 xc027   ECDHE-RSA-AES128-SHA256           ECDH 256   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 xc013   ECDHE-RSA-AES128-SHA              ECDH 256   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256
 x3c     AES128-SHA256                     RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA256
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
TLS 1.3

Looks like a very small subset of ciphers, and they don’t seem to be talking TLS v1.3 at all? Not great. :(

A nice web service to verify the situation from another point of view is checktls, which also confirmed this:

[000.705] 	<-- 	220 2.0.0 Ready to start TLS
[000.705] 		STARTTLS command works on this server
[001.260] 		Connection converted to SSL
		SSLVersion in use: TLSv1_2
		Cipher in use: ECDHE-RSA-AES256-GCM-SHA384
		Perfect Forward Secrecy: yes
		Session Algorithm in use: Curve P-256 DHE(256 bits)
		Certificate #1 of 3 (sent by MX):
		Cert VALIDATED: ok
		Cert Hostname VERIFIED (mx01.arz.at = *.arz.at | DNS:*.arz.at | DNS:arz.at)
[...]
[001.517] 		TLS successfully started on this server

I got distracted by some other work, and when coming back to this problem, the one-time password procedure no longer worked, as the password reset URL was no longer valid. :( I managed to find the underlying URL, and with some web developer tools tinkering I could still use the website to let me trigger sending further one-time password mails, phew.

Let’s continue, so my mail server was running Debian/bullseye with postfix v3.5.18-0+deb11u1 and openssl v1.1.1n-0+deb11u5, let’s see what it offers:

% testssl --cipher-per-proto -t=smtp mail.example.com:25
[...]
Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
SSLv3
TLS 1
 xc00a   ECDHE-ECDSA-AES256-SHA            ECDH 253   AES         256      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 xc009   ECDHE-ECDSA-AES128-SHA            ECDH 253   AES         128      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
TLS 1.1
 xc00a   ECDHE-ECDSA-AES256-SHA            ECDH 253   AES         256      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 xc009   ECDHE-ECDSA-AES128-SHA            ECDH 253   AES         128      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
TLS 1.2
 xc02c   ECDHE-ECDSA-AES256-GCM-SHA384     ECDH 253   AESGCM      256      TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
 xc024   ECDHE-ECDSA-AES256-SHA384         ECDH 253   AES         256      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
 xc00a   ECDHE-ECDSA-AES256-SHA            ECDH 253   AES         256      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
 xcca9   ECDHE-ECDSA-CHACHA20-POLY1305     ECDH 253   ChaCha20    256      TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
 xc0af   ECDHE-ECDSA-AES256-CCM8           ECDH 253   AESCCM8     256      TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8
 xc0ad   ECDHE-ECDSA-AES256-CCM            ECDH 253   AESCCM      256      TLS_ECDHE_ECDSA_WITH_AES_256_CCM
 xc073   ECDHE-ECDSA-CAMELLIA256-SHA384    ECDH 253   Camellia    256      TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 xa7     ADH-AES256-GCM-SHA384             DH 2048    AESGCM      256      TLS_DH_anon_WITH_AES_256_GCM_SHA384
 x6d     ADH-AES256-SHA256                 DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA256
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 xc5     ADH-CAMELLIA256-SHA256            DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA256
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 xc05d   ECDHE-ECDSA-ARIA256-GCM-SHA384    ECDH 253   ARIAGCM     256      TLS_ECDHE_ECDSA_WITH_ARIA_256_GCM_SHA384
 xc02b   ECDHE-ECDSA-AES128-GCM-SHA256     ECDH 253   AESGCM      128      TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
 xc023   ECDHE-ECDSA-AES128-SHA256         ECDH 253   AES         128      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
 xc009   ECDHE-ECDSA-AES128-SHA            ECDH 253   AES         128      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
 xc0ae   ECDHE-ECDSA-AES128-CCM8           ECDH 253   AESCCM8     128      TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8
 xc0ac   ECDHE-ECDSA-AES128-CCM            ECDH 253   AESCCM      128      TLS_ECDHE_ECDSA_WITH_AES_128_CCM
 xc072   ECDHE-ECDSA-CAMELLIA128-SHA256    ECDH 253   Camellia    128      TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 xa6     ADH-AES128-GCM-SHA256             DH 2048    AESGCM      128      TLS_DH_anon_WITH_AES_128_GCM_SHA256
 x6c     ADH-AES128-SHA256                 DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA256
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 xbf     ADH-CAMELLIA128-SHA256            DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA256
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
 xc05c   ECDHE-ECDSA-ARIA128-GCM-SHA256    ECDH 253   ARIAGCM     128      TLS_ECDHE_ECDSA_WITH_ARIA_128_GCM_SHA256
TLS 1.3
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256      TLS_AES_256_GCM_SHA384
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256      TLS_CHACHA20_POLY1305_SHA256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128      TLS_AES_128_GCM_SHA256

Not so bad, but sadly no overlap with any of the ciphers that mx01.arz.at offers.

What about disabling STARTTLS for the mx01.arz.at (+ mx02.arz.at being another one used by the relevant domain) mail servers when talking to mine? Let’s try that:

% sudo postconf -nf smtpd_discard_ehlo_keyword_address_maps
smtpd_discard_ehlo_keyword_address_maps =
    hash:/etc/postfix/smtpd_discard_ehlo_keywords

% cat /etc/postfix/smtpd_discard_ehlo_keywords
# *disable* starttls for mx01.arz.at / mx02.arz.at:
193.110.182.61 starttls
193.110.182.62 starttls

But the remote mail server doesn’t seem to send mails without TLS:

postfix/smtpd[4151799]: connect from mx01.arz.at[193.110.182.61]
postfix/smtpd[4151799]: discarding EHLO keywords: STARTTLS
postfix/smtpd[4151799]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2

Let’s verify this further, but without fiddling with the main mail server too much. We can add a dedicated service to postfix (see serverfault), and run it in verbose mode, to get more detailled logging:

% sudo postconf -Mf
[...]
10025      inet  n       -       -       -       -       smtpd
    -o syslog_name=postfix/smtpd/badstarttls
    -o smtpd_tls_security_level=none
    -o smtpd_helo_required=yes
    -o smtpd_helo_restrictions=pcre:/etc/postfix/helo_badstarttls_allow,reject
    -v

[...]

% cat /etc/postfix/helo_badstarttls_allow
/mx01.arz.at/ OK
/mx02.arz.at/ OK
/193.110.182.61/ OK
/193.110.182.62/ OK

We redirect the traffic from mx01.arz.at + mx02.arz.at towards our new postfix service, listening on port 10025:

% sudo iptables -t nat -A PREROUTING -p tcp -s 193.110.182.61 --dport 25 -j REDIRECT --to-port 10025
% sudo iptables -t nat -A PREROUTING -p tcp -s 193.110.182.62 --dport 25 -j REDIRECT --to-port 10025

With this setup we get very detailed logging, and it seems to confirm our suspicion that the mail server doesn’t want to talk unencrypted with us:

[...]
postfix/smtpd/badstarttls/smtpd[3491900]: connect from mx01.arz.at[193.110.182.61]
[...]
postfix/smtpd/badstarttls/smtpd[3491901]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2
postfix/smtpd/badstarttls/smtpd[3491901]: master_notify: status 1
postfix/smtpd/badstarttls/smtpd[3491901]: connection closed
[...]

Let’s step back and revert those changes, back to our original postfix setup. Might the problem be related to our Let’s Encrypt certificate? Let’s see what we have:

% echo QUIT | openssl s_client -connect mail.example.com:25 -starttls
[...]
issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA384
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4455 bytes and written 427 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 384 bit
[...]

We have an ECDSA based certificate, what about switching to RSA instead? Thanks to the wonderful dehydrated, this is as easy as:

% echo KEY_ALGO=rsa > certs/mail.example.com/config
% ./dehydrated -c --domain mail.example.com --force
% sudo systemctl reload postfix

With switching to RSA type key we get:

% echo QUIT | openssl s_client -connect mail.example.com:25 -starttls smtp
CONNECTED(00000003)
[...]
issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5295 bytes and written 427 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit

Which ciphers do we offer now? Let’s check:

% testssl --cipher-per-proto -t=smtp mail.example.com:25
[...]
Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
SSLv3
TLS 1
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 x88     DHE-RSA-CAMELLIA256-SHA           DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 x84     CAMELLIA256-SHA                   RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 x9a     DHE-RSA-SEED-SHA                  DH 2048    SEED        128      TLS_DHE_RSA_WITH_SEED_CBC_SHA
 x45     DHE-RSA-CAMELLIA128-SHA           DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
 x96     SEED-SHA                          RSA        SEED        128      TLS_RSA_WITH_SEED_CBC_SHA
 x41     CAMELLIA128-SHA                   RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
TLS 1.1
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 x88     DHE-RSA-CAMELLIA256-SHA           DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 x84     CAMELLIA256-SHA                   RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 x9a     DHE-RSA-SEED-SHA                  DH 2048    SEED        128      TLS_DHE_RSA_WITH_SEED_CBC_SHA
 x45     DHE-RSA-CAMELLIA128-SHA           DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
 x96     SEED-SHA                          RSA        SEED        128      TLS_RSA_WITH_SEED_CBC_SHA
 x41     CAMELLIA128-SHA                   RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
TLS 1.2
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 253   AESGCM      256      TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 xc028   ECDHE-RSA-AES256-SHA384           ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x9f     DHE-RSA-AES256-GCM-SHA384         DH 2048    AESGCM      256      TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
 xcca8   ECDHE-RSA-CHACHA20-POLY1305       ECDH 253   ChaCha20    256      TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xccaa   DHE-RSA-CHACHA20-POLY1305         DH 2048    ChaCha20    256      TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xc0a3   DHE-RSA-AES256-CCM8               DH 2048    AESCCM8     256      TLS_DHE_RSA_WITH_AES_256_CCM_8
 xc09f   DHE-RSA-AES256-CCM                DH 2048    AESCCM      256      TLS_DHE_RSA_WITH_AES_256_CCM
 x6b     DHE-RSA-AES256-SHA256             DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 xc077   ECDHE-RSA-CAMELLIA256-SHA384      ECDH 253   Camellia    256      TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384
 xc4     DHE-RSA-CAMELLIA256-SHA256        DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256
 x88     DHE-RSA-CAMELLIA256-SHA           DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 xa7     ADH-AES256-GCM-SHA384             DH 2048    AESGCM      256      TLS_DH_anon_WITH_AES_256_GCM_SHA384
 x6d     ADH-AES256-SHA256                 DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA256
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 xc5     ADH-CAMELLIA256-SHA256            DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA256
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384
 xc0a1   AES256-CCM8                       RSA        AESCCM8     256      TLS_RSA_WITH_AES_256_CCM_8
 xc09d   AES256-CCM                        RSA        AESCCM      256      TLS_RSA_WITH_AES_256_CCM
 x3d     AES256-SHA256                     RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA256
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 xc0     CAMELLIA256-SHA256                RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256
 x84     CAMELLIA256-SHA                   RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc051   ARIA256-GCM-SHA384                RSA        ARIAGCM     256      TLS_RSA_WITH_ARIA_256_GCM_SHA384
 xc053   DHE-RSA-ARIA256-GCM-SHA384        DH 2048    ARIAGCM     256      TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384
 xc061   ECDHE-ARIA256-GCM-SHA384          ECDH 253   ARIAGCM     256      TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 253   AESGCM      128      TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 xc027   ECDHE-RSA-AES128-SHA256           ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x9e     DHE-RSA-AES128-GCM-SHA256         DH 2048    AESGCM      128      TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
 xc0a2   DHE-RSA-AES128-CCM8               DH 2048    AESCCM8     128      TLS_DHE_RSA_WITH_AES_128_CCM_8
 xc09e   DHE-RSA-AES128-CCM                DH 2048    AESCCM      128      TLS_DHE_RSA_WITH_AES_128_CCM
 xc0a0   AES128-CCM8                       RSA        AESCCM8     128      TLS_RSA_WITH_AES_128_CCM_8
 xc09c   AES128-CCM                        RSA        AESCCM      128      TLS_RSA_WITH_AES_128_CCM
 x67     DHE-RSA-AES128-SHA256             DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 xc076   ECDHE-RSA-CAMELLIA128-SHA256      ECDH 253   Camellia    128      TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256
 xbe     DHE-RSA-CAMELLIA128-SHA256        DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256
 x9a     DHE-RSA-SEED-SHA                  DH 2048    SEED        128      TLS_DHE_RSA_WITH_SEED_CBC_SHA
 x45     DHE-RSA-CAMELLIA128-SHA           DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 xa6     ADH-AES128-GCM-SHA256             DH 2048    AESGCM      128      TLS_DH_anon_WITH_AES_128_GCM_SHA256
 x6c     ADH-AES128-SHA256                 DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA256
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 xbf     ADH-CAMELLIA128-SHA256            DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA256
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256
 x3c     AES128-SHA256                     RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA256
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
 xba     CAMELLIA128-SHA256                RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256
 x96     SEED-SHA                          RSA        SEED        128      TLS_RSA_WITH_SEED_CBC_SHA
 x41     CAMELLIA128-SHA                   RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
 xc050   ARIA128-GCM-SHA256                RSA        ARIAGCM     128      TLS_RSA_WITH_ARIA_128_GCM_SHA256
 xc052   DHE-RSA-ARIA128-GCM-SHA256        DH 2048    ARIAGCM     128      TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256
 xc060   ECDHE-ARIA128-GCM-SHA256          ECDH 253   ARIAGCM     128      TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256
TLS 1.3
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256      TLS_AES_256_GCM_SHA384
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256      TLS_CHACHA20_POLY1305_SHA256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128      TLS_AES_128_GCM_SHA256

With switching our SSL certificate to RSA, we gained around 51 new cipher options, amongst them being ones that also mx01.arz.at claimed to support.

FTR, the result from above is what you get with the default settings for postfix v3.5.18, being:

smtpd_tls_ciphers = medium
smtpd_tls_mandatory_ciphers = medium
smtpd_tls_mandatory_exclude_ciphers =
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3

But the delay between triggering the password reset mail and getting a mail server connect was getting bigger and bigger. Therefore while waiting for the next mail to arrive, I decided to capture the network traffic, to be able to look further into this if it should continue to be failing:

% sudo tshark -n -i eth0 -s 65535 -w arz.pcap -f "host 193.110.182.61 or host 193.110.182.62"

A few hours later the mail server connected again, and the mail went through!

postfix/smtpd[4162835]: connect from mx01.arz.at[193.110.182.61]
postfix/smtpd[4162835]: Anonymous TLS connection established from mx01.arz.at[193.110.182.61]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
postfix/smtpd[4162835]: E50D6401E6: client=mx01.arz.at[193.110.182.61]
postfix/smtpd[4162835]: disconnect from mx01.arz.at[193.110.182.61] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7

Now also having the captured network traffic, we can check the details there:

[...]
% tshark -o smtp.decryption:true -r arz.pcap
    1 0.000000000 193.110.182.61 → 203.0.113.42 TCP 74 24699 → 25 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=2261106119 TSecr=0 WS=128
    2 0.000042827 203.0.113.42 → 193.110.182.61 TCP 74 25 → 24699 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=3233422181 TSecr=2261106119 WS=128
    3 0.020719269 193.110.182.61 → 203.0.113.42 TCP 66 24699 → 25 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=2261106139 TSecr=3233422181
    4 0.022883259 203.0.113.42 → 193.110.182.61 SMTP 96 S: 220 mail.example.com ESMTP
    5 0.043682626 193.110.182.61 → 203.0.113.42 TCP 66 24699 → 25 [ACK] Seq=1 Ack=31 Win=29312 Len=0 TSval=2261106162 TSecr=3233422203
    6 0.043799047 193.110.182.61 → 203.0.113.42 SMTP 84 C: EHLO mx01.arz.at
    7 0.043811363 203.0.113.42 → 193.110.182.61 TCP 66 25 → 24699 [ACK] Seq=31 Ack=19 Win=65280 Len=0 TSval=3233422224 TSecr=2261106162
    8 0.043898412 203.0.113.42 → 193.110.182.61 SMTP 253 S: 250-mail.example.com | PIPELINING | SIZE 20240000 | VRFY | ETRN | AUTH PLAIN | AUTH=PLAIN | ENHANCEDSTATUSCODES | 8BITMIME | DSN | SMTPUTF8 | CHUNKING
    9 0.064625499 193.110.182.61 → 203.0.113.42 SMTP 72 C: QUIT
   10 0.064750257 203.0.113.42 → 193.110.182.61 SMTP 81 S: 221 2.0.0 Bye
   11 0.064760200 203.0.113.42 → 193.110.182.61 TCP 66 25 → 24699 [FIN, ACK] Seq=233 Ack=25 Win=65280 Len=0 TSval=3233422245 TSecr=2261106183
   12 0.085573715 193.110.182.61 → 203.0.113.42 TCP 66 24699 → 25 [FIN, ACK] Seq=25 Ack=234 Win=30336 Len=0 TSval=2261106204 TSecr=3233422245
   13 0.085610229 203.0.113.42 → 193.110.182.61 TCP 66 25 → 24699 [ACK] Seq=234 Ack=26 Win=65280 Len=0 TSval=3233422266 TSecr=2261106204
   14 1799.888108373 193.110.182.61 → 203.0.113.42 TCP 74 10330 → 25 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=2262906007 TSecr=0 WS=128
   15 1799.888161311 203.0.113.42 → 193.110.182.61 TCP 74 25 → 10330 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=3235222069 TSecr=2262906007 WS=128
   16 1799.909030335 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=2262906028 TSecr=3235222069
   17 1799.956621011 203.0.113.42 → 193.110.182.61 SMTP 96 S: 220 mail.example.com ESMTP
   18 1799.977229656 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=1 Ack=31 Win=29312 Len=0 TSval=2262906096 TSecr=3235222137
   19 1799.977229698 193.110.182.61 → 203.0.113.42 SMTP 84 C: EHLO mx01.arz.at
   20 1799.977266759 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=31 Ack=19 Win=65280 Len=0 TSval=3235222158 TSecr=2262906096
   21 1799.977351663 203.0.113.42 → 193.110.182.61 SMTP 267 S: 250-mail.example.com | PIPELINING | SIZE 20240000 | VRFY | ETRN | STARTTLS | AUTH PLAIN | AUTH=PLAIN | ENHANCEDSTATUSCODES | 8BITMIME | DSN | SMTPUTF8 | CHUNKING
   22 1800.011494861 193.110.182.61 → 203.0.113.42 SMTP 76 C: STARTTLS
   23 1800.011589267 203.0.113.42 → 193.110.182.61 SMTP 96 S: 220 2.0.0 Ready to start TLS
   24 1800.032812294 193.110.182.61 → 203.0.113.42 TLSv1 223 Client Hello
   25 1800.032987264 203.0.113.42 → 193.110.182.61 TLSv1.2 2962 Server Hello
   26 1800.032995513 203.0.113.42 → 193.110.182.61 TCP 1266 25 → 10330 [PSH, ACK] Seq=3158 Ack=186 Win=65152 Len=1200 TSval=3235222214 TSecr=2262906151 [TCP segment of a reassembled PDU]
   27 1800.053546755 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=186 Ack=3158 Win=36096 Len=0 TSval=2262906172 TSecr=3235222214
   28 1800.092852469 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=186 Ack=4358 Win=39040 Len=0 TSval=2262906212 TSecr=3235222214
   29 1800.092892905 203.0.113.42 → 193.110.182.61 TLSv1.2 900 Certificate, Server Key Exchange, Server Hello Done
   30 1800.113546769 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=186 Ack=5192 Win=41856 Len=0 TSval=2262906232 TSecr=3235222273
   31 1800.114763363 193.110.182.61 → 203.0.113.42 TLSv1.2 192 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
   32 1800.115000416 203.0.113.42 → 193.110.182.61 TLSv1.2 117 Change Cipher Spec, Encrypted Handshake Message
   33 1800.136070200 193.110.182.61 → 203.0.113.42 TLSv1.2 113 Application Data
   34 1800.136155526 203.0.113.42 → 193.110.182.61 TLSv1.2 282 Application Data
   35 1800.158854473 193.110.182.61 → 203.0.113.42 TLSv1.2 162 Application Data
   36 1800.159254794 203.0.113.42 → 193.110.182.61 TLSv1.2 109 Application Data
   37 1800.180286407 193.110.182.61 → 203.0.113.42 TLSv1.2 144 Application Data
   38 1800.223005960 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5502 Ack=533 Win=65152 Len=0 TSval=3235222404 TSecr=2262906299
   39 1802.230300244 203.0.113.42 → 193.110.182.61 TLSv1.2 146 Application Data
   40 1802.251994333 193.110.182.61 → 203.0.113.42 TCP 2962 [TCP segment of a reassembled PDU]
   41 1802.252034015 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5582 Ack=3429 Win=63616 Len=0 TSval=3235224433 TSecr=2262908371
   42 1802.252279083 193.110.182.61 → 203.0.113.42 TLSv1.2 1295 Application Data
   43 1802.252288316 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5582 Ack=4658 Win=64128 Len=0 TSval=3235224433 TSecr=2262908371
   44 1802.272816060 193.110.182.61 → 203.0.113.42 TLSv1.2 833 Application Data, Application Data
   45 1802.272827542 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5582 Ack=5425 Win=64128 Len=0 TSval=3235224453 TSecr=2262908392
   46 1802.338807683 203.0.113.42 → 193.110.182.61 TLSv1.2 131 Application Data
   47 1802.398968611 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=5425 Ack=5647 Win=44800 Len=0 TSval=2262908518 TSecr=3235224519
   48 1863.257457500 193.110.182.61 → 203.0.113.42 TLSv1.2 101 Application Data
   49 1863.257495688 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5647 Ack=5460 Win=64128 Len=0 TSval=3235285438 TSecr=2262969376
   50 1863.257654942 203.0.113.42 → 193.110.182.61 TLSv1.2 110 Application Data
   51 1863.257721010 203.0.113.42 → 193.110.182.61 TLSv1.2 97 Encrypted Alert
   52 1863.278242216 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=5460 Ack=5691 Win=44800 Len=0 TSval=2262969397 TSecr=3235285438
   53 1863.278464176 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [RST, ACK] Seq=5460 Ack=5723 Win=44800 Len=0 TSval=2262969397 TSecr=3235285438
% tshark -O tls -r arz.pcap
[...]
Transport Layer Security
    TLSv1 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 152
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 148
            Version: TLS 1.2 (0x0303)
            Random: 4575d1e7c93c09a564edc00b8b56ea6f5d826f8cfe78eb980c451a70a9c5123f
                GMT Unix Time: Dec  5, 2006 21:09:11.000000000 CET
                Random Bytes: c93c09a564edc00b8b56ea6f5d826f8cfe78eb980c451a70a9c5123f
            Session ID Length: 0
            Cipher Suites Length: 26
            Cipher Suites (13 suites)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
                Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
                Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
[...]
Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Server Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 89
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 85
            Version: TLS 1.2 (0x0303)
            Random: cf2ed24e3300e95e5f56023bf8b4e5904b862bb2ed8a5796444f574e47524401
                GMT Unix Time: Feb 23, 2080 23:16:46.000000000 CET
                Random Bytes: 3300e95e5f56023bf8b4e5904b862bb2ed8a5796444f574e47524401
            Session ID Length: 32
            Session ID: 63d041b126ecebf857d685abd9d4593c46a3672e1ad76228f3eacf2164f86fb9
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
[...]

In this network dump we see what cipher suites are offered, and the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 here is the Cipher Suite Name in IANA/RFC speak. Whis corresponds to the ECDHE-RSA-AES256-GCM-SHA384 in openssl speak (see Mozilla’s Mozilla’s cipher suite correspondence table), which we also saw in the postfix log.

Mission accomplished! :)

Now, if we’re interested in avoiding certain ciphers and increase security level, we can e.g. get rid of the SEED, CAMELLIA and all anonymous ciphers, and could accept only TLS v1.2 + v1.3, by further adjusting postfix’s main.cf:

smtpd_tls_ciphers = high
smtpd_tls_exclude_ciphers = aNULL CAMELLIA
smtpd_tls_mandatory_ciphers = high
smtpd_tls_mandatory_protocols = TLSv1.2 TLSv1.3
smtpd_tls_protocols = TLSv1.2 TLSv1.3

Which would then gives us:

% testssl --cipher-per-proto -t=smtp mail.example.com:25
[...]

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
SSLv3
TLS 1
TLS 1.1
TLS 1.2
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 253   AESGCM      256      TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 xc028   ECDHE-RSA-AES256-SHA384           ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x9f     DHE-RSA-AES256-GCM-SHA384         DH 2048    AESGCM      256      TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
 xcca8   ECDHE-RSA-CHACHA20-POLY1305       ECDH 253   ChaCha20    256      TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xccaa   DHE-RSA-CHACHA20-POLY1305         DH 2048    ChaCha20    256      TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xc0a3   DHE-RSA-AES256-CCM8               DH 2048    AESCCM8     256      TLS_DHE_RSA_WITH_AES_256_CCM_8
 xc09f   DHE-RSA-AES256-CCM                DH 2048    AESCCM      256      TLS_DHE_RSA_WITH_AES_256_CCM
 x6b     DHE-RSA-AES256-SHA256             DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384
 xc0a1   AES256-CCM8                       RSA        AESCCM8     256      TLS_RSA_WITH_AES_256_CCM_8
 xc09d   AES256-CCM                        RSA        AESCCM      256      TLS_RSA_WITH_AES_256_CCM
 x3d     AES256-SHA256                     RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA256
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 xc051   ARIA256-GCM-SHA384                RSA        ARIAGCM     256      TLS_RSA_WITH_ARIA_256_GCM_SHA384
 xc053   DHE-RSA-ARIA256-GCM-SHA384        DH 2048    ARIAGCM     256      TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384
 xc061   ECDHE-ARIA256-GCM-SHA384          ECDH 253   ARIAGCM     256      TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 253   AESGCM      128      TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 xc027   ECDHE-RSA-AES128-SHA256           ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x9e     DHE-RSA-AES128-GCM-SHA256         DH 2048    AESGCM      128      TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
 xc0a2   DHE-RSA-AES128-CCM8               DH 2048    AESCCM8     128      TLS_DHE_RSA_WITH_AES_128_CCM_8
 xc09e   DHE-RSA-AES128-CCM                DH 2048    AESCCM      128      TLS_DHE_RSA_WITH_AES_128_CCM
 xc0a0   AES128-CCM8                       RSA        AESCCM8     128      TLS_RSA_WITH_AES_128_CCM_8
 xc09c   AES128-CCM                        RSA        AESCCM      128      TLS_RSA_WITH_AES_128_CCM
 x67     DHE-RSA-AES128-SHA256             DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256
 x3c     AES128-SHA256                     RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA256
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
 xc050   ARIA128-GCM-SHA256                RSA        ARIAGCM     128      TLS_RSA_WITH_ARIA_128_GCM_SHA256
 xc052   DHE-RSA-ARIA128-GCM-SHA256        DH 2048    ARIAGCM     128      TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256
 xc060   ECDHE-ARIA128-GCM-SHA256          ECDH 253   ARIAGCM     128      TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256
TLS 1.3
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256      TLS_AES_256_GCM_SHA384
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256      TLS_CHACHA20_POLY1305_SHA256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128      TLS_AES_128_GCM_SHA256

Don’t forget to also adjust the smpt_tls_* accordingly (for your sending side). For further information see the Postfix TLS Support documentation. Also check out options like tls_ssl_options (setting it to e.g. NO_COMPRESSION) and tls_preempt_cipherlist (setting it to yes would prefer the servers’ order of ciphers over clients).

Conclusions:

  • no matter what you change in your mail server settings, be aware that the type of your SSL certificate also matters for what ciphers are offered and used
  • there are mail servers out there that don’t support SSL certificates with ECDSA, using RSA for those ensure better compatibility (nowadays postfix supports parallel usage of ECDSA and RSA keys BTW, check out the smtpd_tls_eccert_file + smtpd_tls_eckey_file options)
  • testssl is a very useful tool, especially with its –cipher-per-proto -t=smtp option to check SMTP servers
  • if you’re uncertain what’s going on, consider capturing network data (tshark/tcpdump/… are your friends)
  • review your postfix configuration and logs every now and then :)

What to expect from Debian/bookworm #newinbookworm

June 11th, 2023

Bookworm Banner, Copyright 2022 Juliette Taka

Debian v12 with codename bookworm was released as new stable release on 10th of June 2023. Similar to what we had with #newinbullseye and previous releases, now it’s time for #newinbookworm!

I was the driving force at several of my customers to be well prepared for bookworm. As usual with major upgrades, there are some things to be aware of, and hereby I’m starting my public notes on bookworm that might be worth also for other folks. My focus is primarily on server systems and looking at things from a sysadmin perspective.

Further readings

As usual start at the official Debian release notes, make sure to especially go through What’s new in Debian 12 + Issues to be aware of for bookworm.

Package versions

As a starting point, let’s look at some selected packages and their versions in bullseye vs. bookworm as of 2023-02-10 (mainly having amd64 in mind):

Package bullseye/v11 bookworm/v12
ansible 2.10.7 2.14.3
apache 2.4.56 2.4.57
apt 2.2.4 2.6.1
bash 5.1 5.2.15
ceph 14.2.21 16.2.11
docker 20.10.5 20.10.24
dovecot 2.3.13 2.3.19
dpkg 1.20.12 1.21.22
emacs 27.1 28.2
gcc 10.2.1 12.2.0
git 2.30.2 2.39.2
golang 1.15 1.19
libc 2.31 2.36
linux kernel 5.10 6.1
llvm 11.0 14.0
lxc 4.0.6 5.0.2
mariadb 10.5 10.11
nginx 1.18.0 1.22.1
nodejs 12.22 18.13
openjdk 11.0.18 + 17.0.6 17.0.6
openssh 8.4p1 9.2p1
openssl 1.1.1n 3.0.8-1
perl 5.32.1 5.36.0
php 7.4+76 8.2+93
podman 3.0.1 4.3.1
postfix 3.5.18 3.7.5
postgres 13 15
puppet 5.5.22 7.23.0
python2 2.7.18 – (gone!)
python3 3.9.2 3.11.2
qemu/kvm 5.2 7.2
ruby 2.7+2 3.1
rust 1.48.0 1.63.0
samba 4.13.13 4.17.8
systemd 247.3 252.6
unattended-upgrades 2.8 2.9.1
util-linux 2.36.1 2.38.1
vagrant 2.2.14 2.3.4
vim 8.2.2434 9.0.1378
zsh 5.8 5.9

Linux Kernel

The bookworm release ships a Linux kernel based on version 6.1, whereas bullseye shipped kernel 5.10. As usual there are plenty of changes in the kernel area, including better hardware support, and this might warrant a separate blog entry, but to highlight some changes:

See Kernelnewbies.org for further changes between kernel versions.

Configuration management

puppet‘s upstream sadly still doesn’t provide packages for bookworm (see PA-4995), though Debian provides puppet-agent and puppetserver packages, and even puppetdb is back again, see release notes for further information.

ansible is also available and made it with version 2.14 into bookworm.

Prometheus stack

Prometheus server was updated from v2.24.1 to v2.42.0 and all the exporters that got shipped with bullseye are still around (in more recent versions of course).

Virtualization

docker (v20.10.24), ganeti (v3.0.2-3), libvirt (v9.0.0-4), lxc (v5.0.2-1), podman (v4.3.1), openstack (Zed), qemu/kvm (v7.2), xen (v4.17.1) are all still around.

Vagrant is available in version 2.3.4, also Vagrant upstream provides their packages for bookworm already.

If you’re relying on VirtualBox, be aware that upstream doesn’t provide packages for bookworm yet (see ticket 21524), but thankfully version 7.0.8-dfsg-2 is available from Debian/unstable (as of 2023-06-10) (VirtualBox isn’t shipped with stable releases since quite some time due to lack of cooperation from upstream on security support for older releases, see #794466).

rsync

rsync was updated from v3.2.3 to v3.2.7, and we got a few new options:

  • --fsync: fsync every written file
  • --old-dirs: works like –dirs when talking to old rsync
  • --old-args: disable the modern arg-protection idiom
  • --secluded-args, -s: use the protocol to safely send the args (replaces –protect-args option)
  • --trust-sender: trust the remote sender’s file list

OpenSSH

OpenSSH was updated from v8.4p1 to v9.2p1, so if you’re interested in all the changes, check out the release notes between those version (8.5, 8.6, 8.7, 8.8, 8.9, 9.0, 9.1 + 9.2). Let’s highlight some notable new features:

  • new system for restricting forwarding and use of keys added to ssh-agent(1), see SSH agent restriction for details)
  • switched scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default (see release notes for v9.0 for details
  • ssh(1): when prompting the user to accept a new hostkey, display any other host names/addresses already associated with the key
  • ssh(1): allow UserKnownHostsFile=none to indicate that no known_hosts file should be used to identify host keys
  • ssh(1): add a ssh_config KnownHostsCommand option that allows the client to obtain known_hosts data from a command in addition to the usual files
  • ssh(1), sshd(8): add a RequiredRSASize directive to set a minimum RSA key length
  • ssh(1): add a “host” line to the output of ssh -G showing the original hostname argument
  • ssh-keygen -A (generate all default host key types) will no longer generate DSA keys
  • ssh-keyscan(1): allow scanning of complete CIDR address ranges, e.g. ssh-keyscan 192.168.0.0/24

One important change you might wanna be aware of is that as of OpenSSH v8.8, RSA signatures using the SHA-1 hash algorithm got disabled by default, but RSA/SHA-256/512 AKA RSA-SHA2 gets used instead. OpenSSH has supported RFC8332 RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys will automatically use the stronger algorithm where possible. A good overview is also available at SSH: Signature Algorithm ssh-rsa Error.

Now tools/libraries not supporting RSA-SHA2 fail to connect to OpenSSH as present in bookworm. For example python3-paramiko v2.7.2-1 as present in bullseye doesn’t support RSA-SHA2. It tries to connect using the deprecated RSA-SHA-1, which is no longer offered by default with OpenSSH as present in bookworm, and then fails. Support for RSA/SHA-256/512 signatures in Paramiko was requested e.g. at #1734, and eventually got added to Paramiko and in the end the change made it into Paramiko versions >=2.9.0. Paramiko in bookworm works fine, and a backport by rebuilding the python3-paramiko package from bookworm for bullseye solves the problem (BTDT).

Misc unsorted

  • new non-free-firmware component/repository (see Debian Wiki for details)
  • support only the merged-usr root filesystem layout (see Debian Wiki for details)
  • the asterisk package didn’t make it into bookworm (see #1031046)
  • e2fsprogs: the breaking change related to metadata_csum_seed and orphan_file (see #1031325) was reverted with v1.47.0-2 for bookworm (also see #1031622 + #1030939)
  • rsnapshot is back again (see #986709)
  • crmadmin of pacemaker no longer interprets the timeout option (-t/–timeout) in milliseconds (as it used to be until v2.0.5), but as of v2.1.0 (and v2.1.5 is present in bookworm) it now interprets the argument as second by default

Thanks to everyone involved in the release, happy upgrading to bookworm, and let’s continue with working towards Debian/trixie. :)

HTU Bigband Konzert am 27.06.2023

June 2nd, 2023

Plakat für das HTU Bigband-Konzert am 27.06.2023

Die HTU Bigband ist zurück! Am 27. Juni 2023 findet im Innenhof der TU Graz (Alte Technik, Rechbauerstraße 12, 8010 Graz) das nächste Konzert statt (bei Schlechtwetter geht es in den Hörsaal 2, der ebenfalls an der gleichen Adresse ist). Mit einem fulminanten Programm von Swing, über Soul, Funk, Latin bis Pop ist alles dabei – es gibt über 2 Stunden Musik vom Feinsten, und das Ganze bei freiem Eintritt.

Für diejenigen mit Facebook-Account unter euch gibt es auch das passende Facebook-Event.

Ich bin als Schlagzeuger und Percussionist mit von der Partie und würde mich über bekannte Gesichter freuen, ich hoffe man sieht und hört sich! 8-)

Vortrag: Debugging für Sysadmins @ GLT23

April 16th, 2023

Auf den Grazer Linuxtagen 2023 (GLT23) war ich als Referent mit einem Vortrag zum Thema “Debugging für Sysadmins” vertreten. In meinem Vortrag gibt es einen Überblick, welche Tools und Strategien rund ums Debugging in der Toolbox von Sysadmins nicht fehlen dürfen.

Es gibt den Vortrag dank des wunderbaren c3voc-Teams bereits als Videomitschnitt online. Meine Vortragsfolien (1.2MB, PDF) stehen ebenfalls online zur Verfügung. Viel Spaß beim Anschauen!

BTW: weil ich schon mehrfach gefragt wurde, den Vortrag gibt es auch in längerer Workshop-Version, bei Interesse einfach bei mir melden.

Automatically unlocking a LUKS encrypted root filesystem during boot

March 22nd, 2023

Update on 2023-03-23: thanks to Daniel Roschka for mentioning the Mandos and TPM approaches, which might be better alternatives, depending on your options and needs. Peter Palfrader furthermore pointed me towards clevis-initramfs and tang.

A customer of mine runs dedicated servers inside a foreign data-center, remote hands only. In such an environment you might need a disk replacement because you need bigger or faster disks, though also a disk might (start to) fail and you need a replacement. One has to be prepared for such a scenario, but fully wiping your used disk then might not always be an option, especially once disks (start to) fail. On the other hand you don’t want to end up with (partial) data on your disk handed over to someone unexpected.

By encrypting the data on your disks upfront you can prevent against this scenario. But if you have a fleet of servers you might not want to manually jump on servers during boot and unlock crypto volumes manually. It’s especially annoying if it’s about the root filesystem where a solution like dropbear-initramfs needs to be used for remote access during initramfs boot stage. So my task for the customer was to adjust encrypted LUKS devices such that no one needs to manually unlock the encrypted device during server boot (with some specific assumptions about possible attack vectors one has to live with, see the disclaimer at the end).

The documentation about this use-case was rather inconsistent, especially because special rules apply for the root filesystem (no key file usage), we see different behavior between what’s supported by systemd (hello key file again), initramfs-tools and dracut, not to mention the changes between different distributions. Since tests with this tend to be rather annoying (better make sure to have a Grml live system available :)), I’m hereby documenting what worked for us (Debian/bullseye with initramfs-tools and cryptsetup-initramfs).

The system was installed with LVM on-top of an encrypted Software-RAID device, only the /boot partition is unencrypted. But even if you don’t use Software-RAID nor LVM the same instructions apply. The system looks like this:

% mount -t ext4 -l
/dev/mapper/foobar-root_1 on / type ext4 (rw,relatime,errors=remount-ro)

% sudo pvs
  PV                    VG     Fmt  Attr PSize   PFree
  /dev/mapper/md1_crypt foobar lvm2 a--  445.95g 430.12g

% sudo vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  foobar   1   2   0 wz--n- 445.95g 430.12g

% sudo lvs
  LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root_1 foobar -wi-ao---- <14.90g

% lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
[...]
sdd                     8:48   0 447.1G  0 disk
├─sdd1                  8:49   0   571M  0 part  /boot/efi
├─sdd2                  8:50   0   488M  0 part
│ └─md0                 9:0    0   487M  0 raid1 /boot
└─sdd3                  8:51   0 446.1G  0 part
  └─md1                 9:1    0   446G  0 raid1
    └─md1_crypt       253:0    0   446G  0 crypt
      ├─foobar-root_1 253:1    0  14.9G  0 lvm   /
[...]
sdf                     8:80   0 447.1G  0 disk
├─sdf1                  8:81   0   571M  0 part
├─sdf2                  8:82   0   488M  0 part
│ └─md0                 9:0    0   487M  0 raid1 /boot
└─sdf3                  8:83   0 446.1G  0 part
  └─md1                 9:1    0   446G  0 raid1
    └─md1_crypt       253:0    0   446G  0 crypt
      ├─foobar-root_1 253:1    0  14.9G  0 lvm   /

The actual crypsetup configuration is:

% cat /etc/crypttab
md1_crypt UUID=77246138-b666-4151-b01c-5a12db54b28b none luks,discard

Now, to automatically open the crypto device during boot we can instead use:

% cat /etc/crypttab 
md1_crypt UUID=77246138-b666-4151-b01c-5a12db54b28b none luks,discard,keyscript=/etc/initramfs-tools/unlock.sh

# touch /etc/initramfs-tools/unlock.sh
# chmod 0700 /etc/initramfs-tools/unlock.sh
# $EDITOR etc/initramfs-tools/unlock.sh
# cat /etc/initramfs-tools/unlock.sh
#!/bin/sh
echo -n "provide_the_actual_password_here"

# update-initramfs -k all -u
[...]

The server will then boot without prompting for a crypto password.

Note that initramfs-tools by default uses an insecure umask of 0022, resulting in the initrd being accessible to everyone. But if you have the dropbear-initramfs package installed, its `/usr/share/initramfs-tools/conf-hooks.d/dropbear` sets `UMASK=0077`, so the resulting /boot/initrd* file should automatically have proper permissions (0600). The cryptsetup hook warns about a permissive umask configuration during update-initramfs runs, but if you want to be sure, explicitly set it via e.g.:

# cat > /etc/initramfs-tools/conf.d/umask << EOF
# restrictive umask to avoid non-root access to initrd:
UMASK=0077
EOF
# update-initramfs -k all -u

Disclaimer: Of course you need to trust users with access to /etc/initramfs-tools/unlock.sh as well as the initramfs/initrd on your system. Furthermore you should wipe the boot partition (to destroy the keyfile information) before handing over such a disk. But that is a risk my customer can live with, YMMV.

Mein Lesejahr 2022

January 3rd, 2023

Foto der hier vorgestellten Bücher

Ich habe 2022 keine Bookdumps geschrieben, weil es mir einerseits zu viel Aufwand war, andererseits wollte ich mir auch nicht immer bzw. zeitnahe zu jedem Buch eine Meinung bilden (müssen). 2022 war aus verschiedenen Gründen intensiv, daher habe ich meine Lese-Gewohnheit von 2021 nicht ganz halten können, aber schlussendlich sind es doch 82 Bücher (ca. 19k Seiten) geworden.

Im Gegensatz zu den Vorjahren habe ich diesmal nicht auf das Verhältnis von Autorin zu Autor geachtet, und entsprechend sind es leider auch nur 27 Autorinnen zu 55 Autoren geworden. Ich bin leider noch immer ziemlich schlecht beim Abbrechen von Büchern, aber es waren fast alle Bücher gut. Daher hier nur eine kleine Auswahl jener Bücher, die ich besonders lesenswert fand bzw. empfehlen möchte (die Reihenfolge entspricht dem Foto und stellt keinerlei Reihung oder dergleichen dar):

  • Maksym, Dirk Stermann. Ein wichtiges Thema (Aufteilung der Kinderbetreuung) ist hier in eine unterhaltsam zu lesende Autofiktion verpackt. Das Buch hat meinen Humor im Sommerurlaub perfekt getroffen und wer mit kabarettistischem Humor kein Problem hat, sollte hier nicht enttäuscht werden.
  • Müll (Brenner #9), Wolf Haas. Ich hatte zuvor noch kein Buch von Haas gelesen, weil ich einmal den Spruch “wenn man nicht weiß was man lesen soll greift man zu Haas” aufgeschnappt habe und mich damit nicht angesprochen fühlte. Großer Fehler, Hilfsausdruck. Auf Empfehlung von Daniela Strigl hin habe ich mir den neuesten Brenner-Roman von Haas besorgt und fühlte mich total abgeholt. Ich habe darauf hin gleich weitere Bücher von Haas gelesen. Danke für den Stupser, Frau Strigl.
  • Eine runde Sache, Tomer Gardi. Zwei Geschichten in einem Buch die miteinander lose verbunden sind – zuerst eine Odyssee mit einem Schäferhund in gebrochenem Deutsch, dann die Lebensgeschichte des indonesischen Malers Raden Saleh von Java. Danke für die Empfehlung, Insa Wilke.
  • Das blinde Licht, Benjamín Labatut. In vier Geschichten erzählt Benjamín Labatut vom schmalen Grat zwischen Genie und Wahnsinn, von menschlicher Hybris und der zwiespältigen Kraft der Wissenschaft. Ein wunderbares Buch, das ich nur aufs Wärmste weiterempfehlen kann.
  • Die verschissene Zeit, Barbi Marković. Ein wunderbarer popkultureller Ausflug in das Belgrad der Neunziger.
  • Coventry: Essays, Rachel Cusk. Unaufgeregte Beobachtungen des Alltags die zum Denken anregen. Ich bin leider erst im Nachhinein drauf gekommen, dass die deutsche Übersetzung wohl um einige Kapitel gekürzt ist, sprachlich hat mich die deutsche Ausgabe trotzdem absolut abgeholt.
  • Das hier ist Wasser / This is water, David Foster Wallace. Ein 64 Seiten schlankes Buch, das im ersten Teil die deutsche Übersetzung und im zweiten Teil das englische Original beinhaltet. Inspirierende Gedanken rund um Bildung, Denken und Leben.
  • Kummer aller Art, Mariana Leky. Mir war noch Lekys “Was man von hier aus sehen kann” in guter Erinnerung, hatte die Autorin aber irgendwie aus den Augen verloren. Dieses Buch war ein Weihnachtsgeschenk an mich – und wow, was für ein Volltreffer. Ein fantastisches Buch, ich habe mir umgehend weitere Bücher von Mariana Leky besorgt. Klare Leseempfehlung.
  • Was ist der Mensch? Ein Gespräch über die Welt und Gott, Mark Twain. Ein philosophisches Zwiegespräch über den freien Willen des Menschen. Sehr anregend, danke für die Empfehlung, Darsha.
  • Herr Lehmann (Frank Lehmann #1), Sven Regener. Das Buch wurde 2001 veröffentlicht und 2003 verfilmt, hat es aber erst 2022 auf mein Buchregal geschafft. Für mich hat sich in der Sommerzeit mit diesem Buch ein wunderbarer Lesesog ergeben, ich habe daraufhin gleich weitere Bücher von Regener besorgt und gelesen.
  • Reise durch Ruinen, George Orwell. Orwell folgte als Kriegsberichterstatter den alliierten Streitkräften durch Deutschland und Österreich. Naturgemäß keine leichte Kost.
  • On Writing: A Memoir of the Craft, Stephen King. Dank eines Geburtstagsgeschenks (thx, Kathi + Karl!) habe ich 2022 endlich den Kosmos “Stephen King” betreten. Sprachlich hat mich in “Finderlohn“, der deutschen Ausgabe von “Finders Keepers” aber irgendetwas irritiert, ohne es wirklich benennen zu können. Im Zuge des Lesens von “On Writing: A Memoir of the Craft” habe ich stellenweise das englischsprachige Original mit dessen deutscher Übersetzung “Das Leben und das Schreiben” verglichen und festgestellt, dass ich den “englischen King” unvergleichlich lesenswerter empfinde. Dieses Buch gibt einen lesenswerten Einblick in den Werdegang von King und seinem Zugang zum Schreiben. Und ich möchte hier ganz klar für die englische Ausgabe dieses Buches werben.

RIP, Sven Guckes

February 26th, 2022

Die älteste mir zugängliche Mail von Sven Guckes direkt an mich stammt aus dem Jahr 2002, und startet seinerseits mit:

eine dokumentierte loesung – das ist prima! :-)

Natürlich ging es um: Vim. Viele Mails unter anderem zum Kunsthaus Graz, den Chemnitzer Linux-Tagen, den Grazer Linuxtagen (GLT) und seinem Geek Brunch sollten folgen.

Unvergessen bleibt mir unser Ausflug 2003 zu den Chemnitzer Linux-Tagen mit dem von ihm organisierten Bustransfer aus Berlin. Sven hat netterweise meine Freunde Karl, Thorsten und mich in seiner Wohnung in Berlin gehostet. Einzigartig war Svens Berlin-Führung, von der mir speziell sein mehrfaches “ist gleich ums Eck!” gut in Erinnerung geblieben ist. (Wer das nicht kennt: das war Svens Euphemismus für einen Spaziergang von mehreren Kilometern.)

Ich habe Sven dann weiterhin regelmäßig bei den Chemnitzer Linuxtagen, sowie besonders auf den – von mir mitorganisierten – Grazer Linuxtagen getroffen. Sven konnte dabei für Event-Organisatoren schon auch mal eine Herausforderung sein, er war aber trotzdem immer eine Bereicherung. Svens Sonntags-Brunch war ebenso legendär wie seine Feature-Shows (siehe z.B. Vim Feature Show), oder auch seine späteren “show+tell”-Events.

Technisch hatte ich mit Sven viele Anknüpfungspunkte, von centericq und irssi, über mutt und slrn, bis hin zu screen, vim und zsh. Wir haben zum Thema Texttools 2004 sogar gemeinsam Workshops auf den Grazer Linuxtagen und der KDE Community World Summit 2004 gehalten. Sein Enthusiasmus war im besten Sinne des Wortes ansteckend, seine Geduld und sein Umgang mit Newbies kann für uns alle nur ein Vorbild sein. Aber auch mit Nicht-Techies konnte sich Sven wunderbar unterhalten. Sven war neugierig und hilfsbereit, tolerant aber speziell auch verbindend und ein Menschenfreund. Ich habe viele Bekannte und Freunde dank Sven kennengelernt, und wenn ich mir heute alte Bildergalerien durchschaue, wird mir wieder bewusst, wie viel Einfluss Sven auf mich und mein Umfeld hatte.

Mein nach wie vor verwendetes Mailsetup mit mutt ist mittlerweile über 20 Jahre alt, und noch heute finden sich Schnipsel von Sven in meinen Konfigurationsdateien. Sven ist nach wie vor in der muttrc von Grml vertreten, und exakt am Abend des 20.02.2022 habe ich zur muttrc von Grml eine Mail von einem Anwender erhalten, kurz bevor Sven uns leider für immer verlassen hat. :( Meine vimrc hat noch immer jede Menge iab-Einträge, die ich damals gemeinsam mit Sven erstellt habe. Sven hatte großen Einfluss auf zsh-lovers, aber generell auch auf das gesamte Grml-Projekt (siehe dazu übrigens sein wunderbares Zsh-Buch gemeinsam mit Julius Plenz.)

Svens Leidenschaft für freie Software und die Arbeit mit der Kommandozeile haben mich geprägt, er hatte maßgeblichen Einfluss darauf, wer ich heute bin. Er bleibt mir als Lebenskünstler und Freigeist mit rotem Schal in Erinnerung.

Svens letzte Mail an mich endet übrigens so:

joa.. xundheit hat prio!
denn die beste krankheit nutzt nix. ;)

euch allen ebenfalls alles gute.
auf ein wiedersehen bei den glt22!

:wq

Revisiting 2021

January 12th, 2022

*

Uhm yeah, so this shirt didn’t age well. :) Mainly to recall what happened, I’m once again revisiting my previous year (previous edition: 2020).

2021 was quite challenging overall. It started with four weeks of distance learning at school. Luckily at least at school things got back to "some kind of normal" afterwards. The lockdowns turned out to be an excellent opportunity for practising Geocaching though, and that’s what I started to do with my family. It’s a great way to grab some fresh air, get to know new areas, and spend time with family and friends – I plan to continue doing this. :)

We bought a family season ticket for Freibäder (open-air baths) in Graz; this turned out to be a great investment – I enjoyed the open air swimming with family, as well as going for swimming laps on my own very much, and plan to do the same in 2022. Due to the lockdowns and the pandemics, the weekly Badminton sessions sadly didn’t really take place, so I pushed towards the above-mentioned outdoor swimming and also some running; with my family we managed to do some cycling, inline skating and even practiced some boulder climbing.

For obvious reasons plenty of concerts I was looking forward didn’t take place. With my parents we at least managed to attend a concert performance of Puccinis Tosca with Jonas Kaufmann at Schloßbergbühne Kasematten/Graz, and with the kids we saw "Robin Hood" in Oper Graz and "Pippi Langstrumpf" at Studiobühne of Oper Graz. The lack of concerts and rehearsals once again and still severely impacts my playing the drums, including at HTU BigBand Graz. :-/

Grml-wise we managed to publish release 2021.07, codename JauKerl. Debian-wise we got version 11 AKA bullseye released as new stable release in August.

For 2021 I planned to and also managed to minimize buying (new) physical stuff, except for books and other reading stuff. Speaking of reading, 2021 was nice — I managed to finish more than 100 books (see “Mein Lesejahr 2021“), and I’d like to keep the reading pace.

Now let’s hope for better times in 2022!

Mein Lesejahr 2021

January 4th, 2022

Auch 2021 habe ich wieder mittels Bookdumps versucht, kurze Reviews zu den von mir gelesenen Büchern festzuhalten (wobei sich das auf Belletristik- und Sachbücher beschränkt, also grundsätzlich keine Fachbücher und IT-Bücher, bzw. auch keine Kinderbücher :)). Dazu erschienen 2021 folgende Bookdumps:

In Summe habe ich 2021 mindestens 102 Bücher mit insgesamt 24266 Seiten gelesen, das durchschnittliche Buch hatte also 238 Seiten und ich kam auf durchschnittlich >66 Seiten pro Tag. Diesmal hab ich nicht speziell darauf geachtet, wie das Verhältnis von Autorin zu Autor ist, und mit 69 Autoren zu 32 Autorinnen (sowie 1x gemischt) ist es leider auch entsprechend unausgewogen ausgefallen.

Wie schon in “Mein Lesejahr 2020” geschrieben, helfen mir die Reviews beim Erinnern an die gelesenen Bücher sowie beim Austausch mit Lesekollegen. Die Reviews habe ich diesmal bewusst kürzer bzw. unaufwendiger gehalten und auf die vier Quartale aufgeteilt. Trotzdem passt das Verhältnis Aufwand vs. Nutzen für mich noch nicht ganz. Besonders, da ich den Diskurs mittlerweile in persönlichen Gesprächen mit meinen Lesebuddies suche (und dort eine andere Form von Notizen gefragt ist), die Bookdumps nicht bei der Vorab-Auswahl der Bücher helfen und ich auch viele Bücher ohne Lesebuddy lese. Für 2022 könnte ich mir daher das Format eines “Best Of” vorstellen – let’s see.

Was waren 2021 meine Lieblingsbücher? Es waren viele gute Bücher dabei, die folgenden sind mir aber besonders im Gedächtnis geblieben (keine Reihung):

Mein Bücherregal für 2022 ist bereits gut gefüllt, wer aber noch Empfehlungen hat oder Bücher “gemeinsam” mit mir lesen möchte: ich freue mich über Kommentare hier im Blog oder via Mail (bookdump at michael-prokop.at).