Objectives

This post shows you what’s the current (mid-2021) story around “cross-prelink“. What is it supposed to do? Does it? What are the issues? What are the benefits?

Prerequisites

The latest version of the original prelink program is from 2013-05-03. Anyways this would only run on the target and not work in a cross-build environment like the Yocto Project and that’s why we use cross-prelink. The last change of the version we currently use is from 2018-10-12.

Features

Cross-prelink promises run-time linking optimizations which lead to faster load times and less memory used.

  • faster load times
  • less memory used (less memory relocation – copy on write)
  • less CPU used
  • faster boot times
  • less power consumption
  • optional Address Space Layout Randomization (ASLR)

Let’s forget about ASLR from “prelink” at the moment, since in 2021, we have more advanced methods to deal with ASLR. After running “prelink” or “cross-prelink” over an executable or a library it will slightly increase in size because a “.gnu.prelink_undo” is being added to be able to undo whatever “prelink” did. Keep that in mind, since I’m going to use this piece of information to determine whether “prelink” was able to modify a binary or not.

Except for bit-rotting? Initial tests show that “cross-prelink” does not do anything useful anymore, so it might be time to throw it out, just like many other distros did. A key phrase in the post of my friend Alex is this: “Recent tests have not shown any improvement over not pre-linking, and without a regression test it is very hard to say when and how this happened.[0][1]” Challenge taken. Especially if someone like Alex says it’s very hard. I will try to dig a bit deeper and see what’s going on.

There is no maintainer for it anymore, which leaves the (management) question open who will deal with bugs like this. Let me mention here, that I have an issue, which boils down to “cross-prelink” destroying third-party binaries (grafana), but my fix for this is just to exclude them from “cross-prelink“.

PIE

Address Space Layout Randomization (ASLR) was introduced in the Linux kernel around 2001. ASLR makes it possible that with every reboot and program start the address spaces where the program is loaded into are not anymore predictable but randomized. This makes it harder to write exploits.

First, there was position independent code (PIC). Shared objects “.so” were built with PIC. This means ASLR is possible, but only on libraries. Not on the executable.

Then, there was position-independent executable (PIE) mode, which allows not only libraries but also executables to be placed at randomized addresses in memory.

Keep in mind, that “PIE mode” was created to allow for ASLR.

Have a look at Prelink and address space randomization for more details, but for now just remember, that position-independent executables are not prelink-able. Or in other words, “cross-prelink” will not be able to operate on binaries built with PIE mode.

The Yocto Project and PIE mode

“PIE mode” was introduced around 2017 and was even mentioned in the release notes of Yocto Project 2.4 (rocko 18.0.0) (search for “knob”). Since then it is the default on most architectures. Four years and 11 Releases later we need to re-evaluate “prelink“, which, even in theory is not supposed to work with “PIE mode”. Apparently, people don’t really care too much about what “prelink” has to offer. Does it?

Tests

Root file systems

We need to build four root file systems here:

Variant“image-prelink”PIE mode
prelinked-with-pie
(default)
yesyes
no-prelink-with-pienoyes
prelinked-no-pieyesno
no-prelink-no-pienono

The “image-prelink” class can be enabled/disabled e.g. in local.conf like this:

# enable image-prelink
#USER_CLASSES ?= "buildstats image-prelink"
# disable image-prelink:
USER_CLASSES ?= "buildstats"

“PIE mode” is enabled by default. To disable it I did something like this:

# --> remove pie
GCCPIE = ""
GLIBCPIE = ""
SECURITY_CFLAGS_remove = "${SECURITY_PIE_CFLAGS}"
SECURITY_CFLAGS_pn-libgcc = ""
# <-- remove pie

To determine if files on the root file system are “pre-linked” or not I extract the rootfs tarball on the host and run this script:

#!/bin/sh

# That's what's in prelink.conf:

# -l /usr/local/sbin
# -l /sbin
# -l /usr/sbin
# -l /usr/local/bin
# -l /bin
# -l /usr/bin
# -l /usr/X11R6/bin
# -l /usr/games
# -l /usr/local/lib{,32,64,x32}
# -l /lib{,32,64,x32}
# -l /usr/lib{,32,64,x32}
# -l /usr/X11R6/lib{,32,64,x32}

# I assume there needs to be a prelink section to indicate that prelink was running

counter=0

if [ -f /tmp/elffiles ]; then
   rm -f /tmp/elffiles
fi

DIRS="usr/local/sbin sbin usr/sbin usr/local/bin bin usr/bin usr/X11R6/bin usr/games usr/local/lib lib usr/lib usr/X11R6/lib"

for directory in ${DIRS}; do
   if [ -d $directory ]; then
      # only do things if dir is not empty
      if [ "$(ls -A ${directory})" ]; then
         echo ">> ${directory}"
         find ${directory} -type f | xargs file | grep ELF | cut -f1 -d':' \
>> /tmp/elffiles
         echo "<< ${directory}"
      else
         echo "${directory} is empty"
      fi
   fi
done

for file in $(cat /tmp/elffiles) ; do
   #set -x
   readelf -S $file 2>/dev/null | grep -q prelink
   #set +x
   if [ $? -eq 0 ]; then
      echo "$file is prelinked"
      counter=$(($counter+1))
   fi
done

echo "${counter} files are prelinked"

prelinked-with-pie

$ ../../tests/check-prelink.sh 
>> sbin
<< sbin
>> usr/sbin
<< usr/sbin
>> bin
<< bin
>> usr/bin
<< usr/bin
usr/games is empty
>> lib
<< lib
>> usr/lib
<< usr/lib
usr/bin/grafana-cli is prelinked
usr/bin/grafana-server is prelinked
lib/libdl-2.33.so is prelinked
lib/ld-2.33.so is prelinked
lib/libpthread-2.33.so is prelinked
lib/libc-2.33.so is prelinked
6 files are prelinked

Please note, that “usr/bin/grafana-cli” and “usr/bin/grafana-server” are third party binaries, which were most likely destroyed by “cross-prelink” and therefore will not run but give a segmentation fault.

$ ../../tests/check-prelink.sh 
>> sbin
<< sbin
>> usr/sbin
<< usr/sbin
>> bin
<< bin
>> usr/bin
<< usr/bin
usr/games is empty
>> lib
<< lib
>> usr/lib
<< usr/lib
0 files are prelinked

prelinked-no-pie

$ ../../tests/check-prelink.sh 
>> sbin
<< sbin
>> usr/sbin
<< usr/sbin
>> bin
<< bin
>> usr/bin
<< usr/bin
usr/games is empty
>> lib
<< lib
>> usr/lib
<< usr/lib
sbin/vipw.shadow is prelinked
sbin/fstab-decode is prelinked
sbin/mkfs.ext3 is prelinked
sbin/mkfs.ext4 is prelinked
sbin/halt.sysvinit is prelinked
sbin/mkfs.ext2.e2fsprogs is prelinked
sbin/sysctl.procps is prelinked
sbin/udevd is prelinked
sbin/nologin.shadow is prelinked
sbin/sulogin.util-linux is prelinked
sbin/init.sysvinit is prelinked
sbin/killall5 is prelinked
sbin/bootlogd is prelinked
sbin/shutdown.sysvinit is prelinked
sbin/runlevel.sysvinit is prelinked
sbin/mke2fs.e2fsprogs is prelinked
usr/sbin/chroot.coreutils is prelinked
usr/sbin/groupmod is prelinked
usr/sbin/groupadd is prelinked
usr/sbin/powerdebug is prelinked
usr/sbin/groupmems is prelinked
usr/sbin/start-stop-daemon.dpkg is prelinked
usr/sbin/grpconv is prelinked
usr/sbin/iw is prelinked
usr/sbin/avahi-daemon is prelinked
usr/sbin/lsof is prelinked
...
usr/lib/libxcb-dri2.so.0.0.0 is prelinked
usr/lib/libgio-2.0.so.0.6800.3 is prelinked
usr/lib/libxshmfence.so.1.0.0 is prelinked
usr/lib/libmicrohttpd.so.12.58.0 is prelinked
usr/lib/libprocps.so.8.0.3 is prelinked
usr/lib/libtirpc.so.3.0.0 is prelinked
usr/lib/libcairo.so.2.11600.0 is prelinked
usr/lib/libcrypt.so.2.0.0 is prelinked
usr/lib/libICE.so.6.3.0 is prelinked
usr/lib/libperl.so.5.34.0 is prelinked
usr/lib/libxcb-sync.so.1.0.0 is prelinked
usr/lib/libunwind-arm.so.8.0.1 is prelinked
usr/lib/libxcb-present.so.0.0.0 is prelinked
usr/lib/libunwind.so.8.0.1 is prelinked
usr/lib/libXau.so.6.0.0 is prelinked
usr/lib/libpython3.9.so.1.0 is prelinked
usr/lib/libasm-0.185.so is prelinked
usr/lib/libxcb-xfixes.so.0.0.0 is prelinked
usr/lib/libreadline.so.8.1 is prelinked
usr/lib/libXcursor.so.1.0.2 is prelinked
usr/lib/libgdk-3.so.0.2404.25 is prelinked
usr/lib/libelf-0.185.so is prelinked
usr/lib/libX11.so.6.4.0 is prelinked
usr/lib/libXrandr.so.2.2.0 is prelinked
usr/lib/libglapi.so.0.0.0 is prelinked
usr/lib/libgmodule-2.0.so.0.6800.3 is prelinked
usr/lib/libatspi.so.0.0.1 is prelinked
usr/lib/libbfd-2.36.1.20210209.so is prelinked
usr/lib/libopcodes-2.36.1.20210209.so is prelinked
usr/lib/libdaemon.so.0.5.0 is prelinked
usr/lib/libXft.so.2.3.3 is prelinked
usr/lib/libxml2.so.2.9.12 is prelinked
usr/lib/libctf.so.0.0.0 is prelinked
usr/lib/libatk-bridge-2.0.so.0.0.0 is prelinked
usr/lib/libgtk-3.so.0.2404.25 is prelinked
usr/lib/libdw-0.185.so is prelinked
usr/lib/libXi.so.6.1.0 is prelinked
usr/lib/liblzma.so.5.2.5 is prelinked
usr/lib/libglib-2.0.so.0.6800.3 is prelinked
usr/lib/libgobject-2.0.so.0.6800.3 is prelinked
usr/lib/libsolv.so.1 is prelinked
usr/lib/libcairo-gobject.so.2.11600.0 is prelinked
usr/lib/libell.so.0.0.2 is prelinked
473 files are prelinked
$ ../../tests/check-prelink.sh 
>> sbin
<< sbin
>> usr/sbin
<< usr/sbin
>> bin
<< bin
>> usr/bin
<< usr/bin
usr/games is empty
>> lib
<< lib
>> usr/lib
<< usr/lib
0 files are prelinked

Variant“image-prelink”PIE modeprelinked files
prelinked-with-pie
(default)
yesyes6
at least 2 of them corrupt
no-prelink-with-pienoyes0
prelinked-no-pieyesno473
2 of them corrupt
no-prelink-no-pienono0

Conclusion

In our specific case theory and practice still seem to match. It’s not very surprising that “cross-prelink” does not show much of a difference if it can not operate on files compiled with PIE.

Appendix

If you want to learn how Embedded Linux works have a look here. To learn more about the Yocto Project® have a look here. For some more test and measurements with “cross-prelink” have a look here.

Upcoming Events

Our 3 points

of differentiation

We provide host and target hardware during onsite and remote training.

Three or more people from the same company? We provide private customized training – consulting included.

Subject matter experts develop high quality, job-related, up-to-date, authentic courseware.