Objectives

This is a follow-up to this post, which was the theory of a cross capable “prelink” (“cross-prelink” or “prelink-cross”), brought to you by the Yocto Project®. What we will see here is: What is “prelink” good for? This article shows to measure various things to see if “cross-prelink” holds its promises. This hopefully helps to clarify a few things around “prelink“.

Prerequisites

It might help to better understand the measurements performed here after reading the first part.

Summary of our last post

How many pre-linked files are inside our four root file systems?

Variant“image-prelink”PIE modeprelinked files
prelinked-with-pie
(default)
yesyes6
at least 2 of them corrupt
no-prelink-with-pienoyes0
prelinked-no-pieyesno473
at least 2 of them corrupt
no-prelink-no-pienono0

Load times

To measure load times first of all we need an executable that loads many shared objects “.so“. I decided “/usr/bin/evim“, which is a symlink to “/usr/bin/vim.vim” might be a good choice to start with. Check here how many “.so” files are required by “vim.vim“:

ldd /usr/bin/evim
        linux-vdso.so.1 (0xbea8c000)
        libgtk-3.so.0 => /usr/lib/libgtk-3.so.0 (0xb6834000)
        libgdk-3.so.0 => /usr/lib/libgdk-3.so.0 (0xb6787000)
        libpangocairo-1.0.so.0 => /usr/lib/libpangocairo-1.0.so.0 (0xb676d000)
        libpango-1.0.so.0 => /usr/lib/libpango-1.0.so.0 (0xb6727000)
        libcairo.so.2 => /usr/lib/libcairo.so.2 (0xb6657000)
        libgdk_pixbuf-2.0.so.0 => /usr/lib/libgdk_pixbuf-2.0.so.0 (0xb6629000)
        libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0xb65de000)
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb64ed000)
        libSM.so.6 => /usr/lib/libSM.so.6 (0xb64d7000)
        libICE.so.6 => /usr/lib/libICE.so.6 (0xb64b6000)
        libXt.so.6 => /usr/lib/libXt.so.6 (0xb6469000)
        libX11.so.6 => /usr/lib/libX11.so.6 (0xb6374000)
        libm.so.6 => /lib/libm.so.6 (0xb630e000)
        libtinfo.so.5 => /lib/libtinfo.so.5 (0xb62e3000)
        libacl.so.1 => /usr/lib/libacl.so.1 (0xb62cd000)
        libdl.so.2 => /lib/libdl.so.2 (0xb62ba000)
        libc.so.6 => /lib/libc.so.6 (0xb61bf000)
        /lib/ld-linux-armhf.so.3 (0xb6fbe000)
        libXrandr.so.2 => /usr/lib/libXrandr.so.2 (0xb61a8000)
        libXcursor.so.1 => /usr/lib/libXcursor.so.1 (0xb6191000)
        libXext.so.6 => /usr/lib/libXext.so.6 (0xb6176000)
        librt.so.1 => /lib/librt.so.1 (0xb6160000)
        libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0xb614d000)
        libXi.so.6 => /usr/lib/libXi.so.6 (0xb6132000)
        libXcomposite.so.1 => /usr/lib/libXcomposite.so.1 (0xb611f000)
        libXdamage.so.1 => /usr/lib/libXdamage.so.1 (0xb610c000)
        libXfixes.so.3 => /usr/lib/libXfixes.so.3 (0xb60f8000)
        libcairo-gobject.so.2 => /usr/lib/libcairo-gobject.so.2 (0xb60e2000)
        libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0 (0xb60b9000)
        libatk-bridge-2.0.so.0 => /usr/lib/libatk-bridge-2.0.so.0 (0xb6086000)
        libxkbcommon.so.0 => /usr/lib/libxkbcommon.so.0 (0xb6041000)
        libwayland-cursor.so.0 => /usr/lib/libwayland-cursor.so.0 (0xb602a000)
        libwayland-egl.so.1 => /usr/lib/libwayland-egl.so.1 (0xb6018000)
        libwayland-client.so.0 => /usr/lib/libwayland-client.so.0 (0xb5fff000)
        libepoxy.so.0 => /usr/lib/libepoxy.so.0 (0xb5f3a000)
        libfribidi.so.0 => /usr/lib/libfribidi.so.0 (0xb5f11000)
        libgio-2.0.so.0 => /usr/lib/libgio-2.0.so.0 (0xb5ddc000)
        libpangoft2-1.0.so.0 => /usr/lib/libpangoft2-1.0.so.0 (0xb5dbd000)
        libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0xb5d13000)
        libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0xb5cd4000)
        libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0xb5c5a000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb5c35000)
        libpixman-1.so.0 => /usr/lib/libpixman-1.so.0 (0xb5bab000)
        libpng16.so.16 => /usr/lib/libpng16.so.16 (0xb5b78000)
        libxcb-shm.so.0 => /usr/lib/libxcb-shm.so.0 (0xb5b65000)
        libxcb.so.1 => /usr/lib/libxcb.so.1 (0xb5b3b000)
        libxcb-render.so.0 => /usr/lib/libxcb-render.so.0 (0xb5b21000)
        libXrender.so.1 => /usr/lib/libXrender.so.1 (0xb5b0a000)
        libz.so.1 => /lib/libz.so.1 (0xb5aea000)
        libGL.so.1 => /usr/lib/libGL.so.1 (0xb5a8d000)
        libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0xb5a4a000)
        libffi.so.8 => /usr/lib/libffi.so.8 (0xb5a2f000)
        libpcre.so.1 => /usr/lib/libpcre.so.1 (0xb59cf000)
        libattr.so.1 => /usr/lib/libattr.so.1 (0xb59bb000)
        libdbus-1.so.3 => /usr/lib/libdbus-1.so.3 (0xb5978000)
        libatspi.so.0 => /usr/lib/libatspi.so.0 (0xb5942000)
        libmount.so.1 => /lib/libmount.so.1 (0xb58f4000)
        libresolv.so.2 => /lib/libresolv.so.2 (0xb58d4000)
        libexpat.so.1 => /usr/lib/libexpat.so.1 (0xb58a9000)
        libuuid.so.1 => /usr/lib/libuuid.so.1 (0xb5893000)
        libXau.so.6 => /usr/lib/libXau.so.6 (0xb5880000)
        libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0xb586b000)
        libglapi.so.0 => /usr/lib/libglapi.so.0 (0xb582c000)
        libdrm.so.2 => /usr/lib/libdrm.so.2 (0xb580d000)
        libxcb-glx.so.0 => /usr/lib/libxcb-glx.so.0 (0xb57eb000)
        libX11-xcb.so.1 => /usr/lib/libX11-xcb.so.1 (0xb57d9000)
        libxcb-dri2.so.0 => /usr/lib/libxcb-dri2.so.0 (0xb57c5000)
        libXxf86vm.so.1 => /usr/lib/libXxf86vm.so.1 (0xb57b1000)
        libxcb-dri3.so.0 => /usr/lib/libxcb-dri3.so.0 (0xb579d000)
        libxcb-present.so.0 => /usr/lib/libxcb-present.so.0 (0xb578a000)
        libxcb-sync.so.1 => /usr/lib/libxcb-sync.so.1 (0xb5775000)
        libxshmfence.so.1 => /usr/lib/libxshmfence.so.1 (0xb5763000)
        libxcb-xfixes.so.0 => /usr/lib/libxcb-xfixes.so.0 (0xb574d000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb5724000)
        libblkid.so.1 => /lib/libblkid.so.1 (0xb56dd000)

The test case which tries to measure load times looks like this:

#!/bin/sh
TEST="/usr/bin/evim"
set -x
md5sum ${TEST}
ldd ${TEST}
readelf -S ${TEST}
sync; echo 3 > /proc/sys/vm/drop_caches
perf stat sh -c "LD_BIND_NOW=true LD_DEBUG=statistics /usr/bin/evim 2>&1 > /dev/null"
set +x

If we put the results of our four root file systems in a table it looks like this:

prelinked-with-pie (default)no-prelink-with-pieprelinked-no-pieno-prelink-no-pie
number of relocations1756517565717558
number of relocations from the cache100410045701004
number of relative relocations3820638206030517
the final number of relocations1756517565717558
the final number of relocations from the cache100410045701004
task-clock/msec172.64172.6669.64188.76
CPUs utilized0.1770.1790.0640.142
context-switches215211187209
cpu-migrations0001
page-faults799793668771
seconds time elapsed0.9746170.9657213321.0832598871.333485667
seconds user0.0889420.1136970.0105160.116912
seconds sys0.0885230.0638030.0628570.076899
Pss: is the proportional set size, which is the amount of memory shared with other processes, divided by the number of processes sharing each page

Uss: is a unique set size, which is the amount of memory that is private to the process and is not shared with any other

Free memory after boot

This is the test case I run to check for used memory:

#!/bin/sh
set -x
cat /proc/meminfo
procrank
cat /proc/slabinfo
set +x

The results from the test above when you run them on all four root file systems are this:

prelinked-with-pie (default)no-prelink-with-pieprelinked-no-pieno-prelink-no-pie
MemTotal kB1013656101365610136561013656
MemFree kB820064820212816092820644
MemAvailable kB939412939696940344941216
total K1013656101365610136561013656
free K819572820112815840820092
buffers K4016398050803956
cached K116960116240118788116892
shmem K240240256228
slab K33352334523343633040
Pss K116853115725120277117877
Uss K113520112420116740114364
Pss: is the proportional set size, which is the amount of memory shared with other processes, divided by the number of processes sharing each page

Uss: is a unique set size, which is the amount of memory that is private to the process and is not shared with any other

MemFree

MemFree” (kB) was obtained by “cat /proc/meminfo” and describes free physical memory in both user and kernel space.

prelinked-with-pie (default)no-prelink-with-pieprelinked-no-pieno-prelink-no-pie
prelinked-with-pie (default)-1483972-580
no-prelink-with-pie1484120-432
prelinked-no-pie-3972-4120-4552
no-prelink-no-pie5804324552

Interpretation of relative Numbers

We’ll look at whether there is more or less physical memory consumed. Less physical memory consumed is better, although we typically care more about virtual than physical memory in Linux.

more physical memory consumed (kB)less physical memory consumed (kB)bybiggest negative impactbiggest positive impact
prelinked-with-pie/no-prelink-with-pie148prelinked-with-pie6
prelinked-with-pie/prelinked-no-pie3972prelinked-with-pie3
prelinked-with-pie/no-prelink-no-pie580prelinked-with-pie4
no-prelink-with-pie/prelinked-with-pie148no-prelink-with-pie6
no-prelink-with-pie/prelinked-no-pie4120no-prelink-with-pie2
no-prelink-with-pie/no-prelink-no-pie432no-prelink-with-pie5
prelinked-no-pie/prelinked-with-pie3972prelinked-no-pie3
prelinked-no-pie/no-prelink-with-pie4120prelinked-no-pie2
prelinked-no-pie/no-prelink-no-pie4552prelinked-no-pie1
no-prelink-no-pie/prelinked-with-pie580no-prelink-no-pie4
no-prelink-no-pie/no-prelink-with-pie532no-prelink-no-pie5
no-prelink-no-pie/prelinked-no-pie4552no-prelink-no-pie1

3rd worst or 3rd best, however you want to see it, is currently our default setting in the Yocto Project®. Please keep in mind, that most libraries are not “prelinked” but some crucial ones (lib/libdl-2.33.so, lib/ld-2.33.so, lib/libpthread-2.33.so, lib/libc-2.33.so) are, which somewhat implies that they are not compiled in “PIE mode” and hence a security issue.

Interpretation of absolute Numbers

Let’s say that more free physical memory is better, although we care more about virtual memory.

MemFree (kB)WinnerLooser
prelinked-with-pie (default)82006432
no-prelink-with-pie82021223
prelinked-no-pie81609241
no-prelink-no-pie82064414

MemAvailable

MemAvailable” (kB) was obtained by “cat /proc/meminfo” and is an estimate of how much virtual memory is available after reclaiming it (caches, buffers, slab,…) without getting “swap” involved.

prelinked-with-pie (default)no-prelink-with-pieprelinked-no-pieno-prelink-no-pie
prelinked-with-pie (default)-284-932-1804
no-prelink-with-pie284-648-1520
prelinked-no-pie932648-872
no-prelink-no-pie18041520872

Interpretation of relative Numbers

We’ll look at an estimate of whether there is more or less virtual memory available after it’s being reclaimed. More virtual memory available is better.

more virtual memory available (kB)less virtual memory available (kB)bybiggest positive impactbiggest negative impact
prelinked-with-pie/no-prelink-with-pie284prelinked-with-pie6
prelinked-with-pie/prelinked-no-pie932prelinked-with-pie3
prelinked-with-pie/no-prelink-no-pie1804prelinked-with-pie1
no-prelink-with-pie/prelinked-with-pie284no-prelink-with-pie6
no-prelink-with-pie/prelinked-no-pie648no-prelink-with-pie5
no-prelink-with-pie/no-prelink-no-pie1520no-prelink-with-pie2
prelinked-no-pie/prelinked-with-pie932prelinked-no-pie3
prelinked-no-pie/no-prelink-with-pie648prelinked-no-pie5
prelinked-no-pie/no-prelink-no-pie872prelinked-no-pie4
no-prelink-no-pie/prelinked-with-pie1804no-prelink-no-pie1
no-prelink-no-pie/no-prelink-with-pie1520no-prelink-no-pie2
no-prelink-no-pie/prelinked-no-pie872no-prelink-no-pie4

Interpretation of absolute Numbers

Let’s assume more virtual memory available after reclaiming is better.

MemAvailable (kB)WinnerLooser
prelinked-with-pie (default)93941241
no-prelink-with-pie93969632
prelinked-no-pie94034423
no-prelink-no-pie94121614

Memory Consumption of a specific executable after boot

This is the file we’ll look into:

~# ldd /usr/sbin/ofonod
        linux-vdso.so.1 (0xbedb4000)
        libudev.so.1 => /lib/libudev.so.1 (0xb6f33000)
        libell.so.0 => /usr/lib/libell.so.0 (0xb6ede000)
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb6ded000)
        libdbus-1.so.3 => /usr/lib/libdbus-1.so.3 (0xb6daa000)
        libdl.so.2 => /lib/libdl.so.2 (0xb6d97000)
        libc.so.6 => /lib/libc.so.6 (0xb6c9c000)
        /lib/ld-linux-armhf.so.3 (0xb6f5a000)
        libpcre.so.1 => /usr/lib/libpcre.so.1 (0xb6c3c000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb6c17000)

The results from the test above when you run them on all four root file systems are this:

prelinked-with-pie (default)no-prelink-with-pieprelinked-no-pieno-prelink-no-pie
Vss K5136513250365004
Rss K3360332433483324
Pss K2106207720462008
Uss K1828179617281676
Vss: also called VIRT and VSZ is the total amount of virtual memory of the process has mapped, regardless of whether it has been committed to physical memory

Rss: also called RES and RSS, is the amount of physical memory being mapped

Pss: is the proportional set size, which is the amount of memory shared with other processes, divided by the number of processes sharing each page

Uss: is a unique set size, which is the amount of memory that is private to the process and is not shared with any other

Conclusion

We can clearly see that “cross-prelink” can only work on binaries that were not compiled in “PIE mode”. You might turn off “PIE mode” and make your system more vulnerable to attacks, or selectively compile certain libraries and executables with or without “PIE mode” to give “cross-prelink” a chance. I guess, at least network-facing apps should be compiled in “PIE mode” for security reasons.

Appendix

Keep in mind, that we still have the initial issue, that “cross-prelink” runs on certain third-party binaries and destroys them. How to selectively disable “cross-prelink” on those specific files might be discussed in another article.

If you want to learn how Embedded Linux works have a look here. To learn more about the Yocto Project® have a look here.

Upcoming Events

Our 3 points

of differentiation

We provide host and target hardware during onsite and remote training.

Three or more people from the same company? We provide private customized training – consulting included.

Subject matter experts develop high quality, job-related, up-to-date, authentic courseware.