The fedora.us buildsystem

Enrico Scholz

Revision History
Revision 0.12003-11-28
Revision 0.22004-01-14
fixed/added some links

Table of Contents

Requirements on the build-system
Attacks from upstream author
Attacks from the packager
Effects of attacks
Summary
Components
mach
vserver
SELinux
User Mode Linux (UML)
QEMU/Bochs
vserver-djinni
The current buildsystem
The physical host
The buildmaster vserver
The buildslave environment
The buildroot(s)
Problems
Different rpm-database layouts and rpm-versions
disttag-replacement
Bibliography

Abstract

This paper gives an overview about ways to implement a buildsystem for rpm-packages and about the current fedora.us buildsystem. The official source of this document is http://www.tu-chemnitz.de/~ensc/fedora.us-build where it can be found in a HTML and PDF format.

Requirements on the build-system

There are two critical requirements on the build-system: it must build packages in a reliable manner, and it must be resistant against attacks. Builds without much overhead and buildmaster intervention are other requirements.

Attacks from upstream author

There are various ways for the upstream author to attack the buildsystem. Basically, it is to differ between attacks at package-buildtime, and these at the runtime of the package:

Runtime-attacks are the creation of backdoors in the programs (trojans), or to make them doing bad things at certain times (timebombs). On first glance, the runtime case seems to be uninteresting for the buildsystem itself, but as shown below this matters also.

Buildtime-attacks are possible since arbitrary code will be executed by the make-process. So:

  • local root exploits (e.g. backdoors introduced by other packages, overflows in suid'ed programs or daemons, kernel flaws) can be used to gain privileges and to spy out secrets (e.g. ssh keys), or to modify files (e.g. /etc/passwd or /lib/libc.so.6). Since chroot(2) is easy breakable[1], files of the host system can be compromised although the build happens in a chroot. Creating special devices (e.g. /dev/hda) and operating on them would by-pass chroots also; having access to /dev/kmem would allow to inject malignant code into the kernel.
  • processes which are running with the same uid can be ptrace'ed and killed. Such processes can be parallel builds of other packages, or -- in combination with local root-exploits -- system-processes like init or sshd.
  • a process could be spawned in the background and is killing or ptrace'ing processes of subsequent builds, or is modifying files of such builds.
  • network-resources of the build-machine could be used to open an hidden warez/porn server, or to run attacks against remote machines.

There are known cases where buildtime-attacks are triggered by the hostname of the build-machine, so that the build appears unsuspiciously on the machine of QA testers.

Altogether, since complex source-code will be compiled and complex code be executed by the make-system, it would require a full audit to preclude upstream attacks, so this kind can not be detected by the QA.

Attacks from the packager

Special crafted .src.rpm files

An attacker could create special crafted (.src).rpm files which are exploiting implementation or design flaws of the rpm-program. So, arbitrary code could be executed on the build-machine or on the machine of QA people, when the package is extracted or queried.

Doing the requested rpm-action after running such code would hide this kind attack effectively. QA will not protect against this kind of attack.

Malicious patches/modified upstream sources
An attacker could introduce code which would have the same effects like attacks from upstream authors. Modified sources should be detected by QA when there exist signed md5sums or signatures of upstream sources. Most patches are small and should be audited by QA also.
Malicious rpm-scriptlets or triggers
On installation of packages, these scriptlets are running as root and can do nearly everything. Malicious shell-code in such scriptlets can and should be detected by the QA, but since this shell-code executes binaries of the current package and/or other packages, it will be impossible to preclude attacks when programs of untrusted packages can be used in scriptlets.
SUID packaging

A packager could set the SUID bit of programs which were never designed to run with root privileges and untrusted input. Doing so, backdoors could be created.

Such an attack can and must be detected by QA and/or automatic QA tools like rpmlint.

Effects of attacks

There are two categories of attacks: those which are attacking the build-machine itself, and those which are modifying the content of packages. In the chapter about bad upstream sources, most of the attacks against the build-machine were mentioned already, and on a compromised build-machine it would be easy to do anything -- inclusive the modification of packages. Therefore, only the aspects of package-modifications will be discussed in this section.

To do reliable builds, certain dependencies must be fulfilled which will lead to the installation of other, untrusted packages. As mentioned in the section called “Attacks from upstream author”, limiting the requirements to packages of trusted packagers will not work since good src.rpms can create bad binary rpms.

At the installation of a package, rpm-scriptlets will be executed with root privileges, so that chroot's can be broken or files like libraries or programs be replaced. Since the replacement of files or the usage of chroot are common and valid operations for rpm, it is impossible to forbid these operations. Therefore, when the first untrusted package is installed, the build-root must be assumed as compromised.

Now, when a build happens in such an hostile environment, injected code can be executed (e.g. by calling a replaced make program or a program using a prepared /lib/libc.so.6). It will be possible, that the build-results can be modified in such a manner that the created binary package carries the same malicious code as the original code. So, entire dependency-trees can be infected. Yet worse, when using the same build-environment for every package, the entire distribution will be infected.

As an example, Figure 1, “Infection of build-tree vs. infection of build-environment” shows the build of the MagicPoint, kdebase, kdb, xine, ee and xemacs packages inclusive their dependencies (in this order). Malicious code which replaces /lib/libc.so.6 will be assumed in arts. The first image shows what happens when for each package an own build-system will be used, the second one shows the case when each build reuses the same build-system. The blue node marks the origin of the infection, the red ones the infected packages. The black arrows are symbolizing the BuildRequires:, the gray ones the build-order.

Figure 1. Infection of build-tree vs. infection of build-environment

Infection of build-tree vs. infection of build-environment

Creating the build-system from scratch for each build

Infection of build-tree vs. infection of build-environment

Reusing an already existing build-system

Summary

To prevent attacks and to have a reliable buildsystem, it must have the following properties:

Process-separation:
it MUST be impossible to kill or ptrace processes of other buildroots or of the system. Hiding of foreign processes SHOULD be provided.
Device/kernel protection:
Direct hardware access through /dev/* entries or modification of kernel parameters through /proc MUST be impossible. Forbidding the creation of such special files is one way to reach it, access restriction another one.
Unbreakable chroots:
it MUST be impossible for a process in a buildroot to have any kind of access on objects of the systems (e.g. ssh-keys), or write-access on other buildroots.
No buildroot-reusing:
each build MUST happen in an environment which can not be influenced by previous builds in this environment. This includes both filesystem-objects, and processes.
Resource-restrictions:
excessive resource-usage (memory, diskspace,...) of a build SHOULD be prevented. Usage of certain resources (e.g. network) MUST be prohibited
Good performance:
the buildsystem SHOULD should have only a small or non-existing impact on the performance.
Working environment:
building of common packages MUST succeed. This requires certain /dev entries, and a mounted /proc filesystem at least.
Mature userinterface:
the system SHOULD assist the buildmaster and automate the most tasks, so that the spent time will be reduced to a minimum.