Cross-compilation

1. Introduction

About 

This document describes how to setup a cross-compiler for PowerPC on a x86 for use in a distributed compilation system. The system used is Gentoo Linux. Most information will be Gentoo-centric, however, some information can be generally applied to other Linux systems and to non-Linux systems.

Some information comes by examining the Gentoo crossdev script. Unfortunately, I was unable to build successfully a cross-compiler for my system configuration by simply using the script. The approach taken was to read the script and go step by step, fixing a few things for my system. Later, cross-compilation support had been completed in portage and further information was added about this. Finally, I received several contributions from readers that I added, making this documentation more complete.

Document History 

2005-04-20: Started reorganizing documentation and using Gentoo doc stylesheets.

2005-04-19: Added info on building minimal toolchain using Gentoo portage.

2005-02-23: Updated documentation.

2. Options

Overview 

Four options are presented. The Gentoo-related options make full use of Gentoo tools such as portage. The non-Gentoo options can be generalized to several systems and require more work. The full toolchain options builds a full cross-compiler toolchain. With a full toolchain, full applications can be cross-compiled on the system provided all the dependencies are cross-compiled. With the minimal toolchain, applications can not entirely be cross-compiled on the system, however, this toolchain is sufficient for distributed cross-compilation with distcc.

The options are:

1. Gentoo Minimal Toolchain
2. Gentoo Full Toolchain
3. General Minimal Toolchain
4. General Full Toolchain

Full Toolchain 

It consists of all the tools with glibc. It is suitable to cross-compile full applications on the cross-compiler host.

Distcc-Only Toolchain 

It consists of the minimal amount of tools required for distributed cross-compilation with distcc. In particular, only the assembler (as) and the compilers (gcc, g++, ...) are required. All preprocessing and linking are done by the distcc client.

Note: Advanced users might want to try out the nolibgcc patch in this section. On a Gentoo system, that patch allow an easy cross-compiler installation in two steps: binutils and then gcc. It might also work on a non-Gentoo system.

3. Cross-Compilation Support in Gentoo

Gentoo Minimal Toolchain 

A reader contributed a patch and some instructions on how to build a minimal cross-compiler toolchain (binutils and gcc only). This toolchain is sufficient for cross-compilation with distcc. He used it for a x86 cross-compiler on an amd64. Following his instructions, I built successfully and easily an x86 cross-compiler on PowerPC as follows:

# mkdir -p /usr/local/cross-portage-minimal/cross-i686-pc-linux-gnu
# cd /usr/local/cross-portage-minimal/cross-i686-pc-linux-gnu
# ln -sf /usr/portage/sys-devel/binutils
# cp -ax /usr/portage/sys-devel/gcc .
# PORTDIR_OVERLAY=`pwd`/.. emerge -av cross-i686-pc-linux-gnu/binutils
Add this user-contributed patch to gcc/files/3.4.3
Add an epatch line to the src_unpack() function in the gcc ebuild:
epatch ${FILESDIR}/3.4.3/gcc-3.4.3-nolibgcc.patch
# PORTDIR_OVERLAY=`pwd`/.. emerge -av cross-i686-pc-linux-gnu/gcc

Gentoo Full Toolchain 

Read this HOWTO to build a full toolchain on a Gentoo system.

Note that I was unable to a PowerPC cross-compiler by following directly the instructions from the HOWTO. I had to do these two modifications:

1. Build the kernel headers before gcc. In order to do this, change make prepare ... for make include/linux/version.h in the headers ebuild.
2. Add --with-sysroot=/ to the gcc configure options in toolchain.eclass.

Example of Use 

This document explains how to cross-compile SableVM on a Gentoo system.

4. General Build Information

Overview 

Although this information is Gentoo-centric, it explains how to manually configure and build the various components and it can be applied to several systems. Information on package names and on getting sources is Gentoo-centric. Users of non-Gentoo systems will have to learn how to get the sources for their systems for the packages involved.

Configuration 

Both systems run Gentoo Linux. Distributed compilation is accomplished with distcc.

The host is a AMD Athlon (i686-pc-linux-gnu).

The target is a PowerPC 750FX (G3) (powerpc-unknown-linux-gnu)

The cross-compilation toolchain was built from the following packages and versions:

• sys-kernel/linux26-headers-2.6.7-r4
• sys-devel/binutils-2.15.90.0.3-r3
• sys-libs/glibc-2.3.4.20040808
• sys-devel/gcc-3.4.1-r2

Space Requirements 

The full toolchain installation takes 190MB. The distcc-only toolchain takes only 12MB after some cleanup.

Note the build requirements are much higher.

Installation - Preparation 

Throughout this document, I will refer to the following variables:

• DEST: Destination directory
• CROSS_CHOST
• MY_CHOST
• MY_CFLAGS
• CROSS_CFLAGS
• MIN_KV
• CROSS_GCCLANG: Languages to support.
  export CROSS_GCCLANG="c,c++"
  or
  export CROSS_GCCLANG="c,c++,ada,f77,objc"
• POWERPC_HOSTNAME: Hostname of your PowerPC machine.

I used these values:

export DEST=/home/crossdev/ppc
export CROSS_CHOST=powerpc-unknown-linux-gnu
export MY_CHOST=i686-pc-linux-gnu
export MY_CFLAGS="-O2 -pipe -march=i686 -fomit-frame-pointer"
export CROSS_CFLAGS="-O3 -mcpu=750 -pipe"
export MIN_KV=2.6.7
export CROSS_GCCLANG="c,c++"

export HEADERS_VER=2.6.7-r4
export BINUTILS_VER=2.15.90.0.3-r3
export GLIBC_VER=2.3.4.20040808
export GCC_VER=3.4.1-r2

export POWERPC_HOSTNAME=ibou

export PATH=${DEST}/bin:${PATH}

Step 0 - Preparation 

Create a destination directory:

# mkdir -p ${DEST}
# chmod a+rx ${DEST}

Unpack the linux-headers, binutils, glibc and gcc sources. Note that these should correspond to your target system, in particular the gcc versions should match in you are considering distributed compilation with distcc.

On Gentoo, this can be done with the ebuild command as follows:

(By default, sources are unpacked in /var/tmp/portage/package_name.)
# ebuild /usr/portage/sys-kernel/linux26-headers/linux26-headers-${HEADERS_VER}.ebuild unpack
# ebuild /usr/portage/sys-devel/binutils/binutils-${BINUTILS_VER}.ebuild unpack
# ebuild /usr/portage/sys-libs/glibc/glibc-${GLIBC_VER}.ebuild unpack
# ebuild /usr/portage/sys-devel/gcc/gcc-${GCC_VER}.ebuild unpack

You are now ready to configure/compile/install each component. For each of the next 5 steps, go into the source root directory of the component to build before executing the commands.

Step 1 - Installation of kernel-headers 

Do:

# mkdir ${DEST}/include
# make ARCH=ppc CROSS_COMPILE=powerpc-unknown-linux-gnu- mrproper

Optionnally copy the .config from your PowerPC before continuing:

# scp ${POWERPC_HOSTNAME}:/usr/src/linux/.config .

Then, continue the installation:

# make ARCH=ppc CROSS_COMPILE=powerpc-unknown-linux-gnu- oldconfig
# make include/linux/version.h
# cp -axf include/linux ${DEST}/include
# cp -axf include/asm-ppc ${DEST}/include/asm-ppc
# cp -axf include/asm-generic ${DEST}/include/asm-generic
# ln -sf ${DEST}/include/asm-ppc ${DEST}/include/asm

Step 2 - Installation of binutils 

Configure binutils:

# CFLAGS=${MY_CFLAGS} \
./configure \
  --target=${CROSS_CHOST} \
  --host=${MY_CHOST} \
  --prefix=${DEST} \
  --enable-shared

Build it:

# make

Install it in one of two ways:

1. For distcc-only toolchain, install only the assembler and its libraries:

   Code listing 9: Build binutils - Minimal Toolchain

   # (cd gas && make install) # (cd opcodes && make install) # (cd bfd && make install)

2. For a full toolchain:

   Code listing 10: Build binutils - Full Toolchain

   # make install

Update your path to use the binutils just built:

# export PATH=${DEST}/bin:$PATH

Step 3 - Installation of gcc (bootstrap) 

For some reasons, headers are installed in ${DEST}/include, however, gcc does not include that directory but it does look in the non-existing ${DEST}/powerpc-unknown-linux-gnu/include directory. To fix this, I do the following symlink:

# ln -s ../include ${DEST}/powerpc-unknown-linux-gnu/include

Prepare, configure, build, and install by doing one of these two alternatives:

1. For a distcc-only installation:

   Code listing 13: Building gcc (Bootstrap) - Minimal Toolchain

   # mkdir buildfullhere-ppc # cd buildfullhere-ppc # CFLAGS="${MY_CFLAGS}" \ ../configure \ --prefix=${DEST} \ --host=${MY_CHOST} \ --target=${CROSS_CHOST} \ --disable-multilib \ --disable-nls \ --enable-shared \ --enable-languages="${CROSS_GCCLANG}" \ --without-included-gettext \ --with-system-zlib \ --enable-threads=posix \ --enable-long-long \ --enable-symvers=gnu \ --disable-checking \ --enable-cstdio=stdio \ --enable-clocale=generic \ --enable-__cxa_atexit \ --enable-c99 \ --enable-version-specific-runtime-libs \ --with-local-prefix=${DEST}/local \ --with-sysroot=/ # make CFLAGS_FOR_TARGET="${CROSS_CFLAGS} -DHAVE_SYS_ERRLIST" CXXFLAGS_FOR_TARGET="${CFLAGS_FOR_TARGET}"

   Since powerpc-unknown-linux-gnu-ar was not built, this build error should occur:

   Code listing 14: Build Error

   make[2]: powerpc-unknown-linux-gnu-ar: Command not found make[2]: *** [libgcc.a] Error 127 make[2]: Leaving directory `/var/tmp/portage/gcc-3.4.1-r2/work/gcc-3.4.1/buildfullhere-ppc/gcc' make[1]: *** [stmp-multilib] Error 2 make[1]: Leaving directory `/var/tmp/portage/gcc-3.4.1-r2/work/gcc-3.4.1/buildfullhere-ppc/gcc' make: *** [all-gcc] Error 2

   It is not required to build any PowerPC libraries (such as libgcc, etc.). Simply force the compilation of the remaining components with the -k option:

   Code listing 15: Finishing Build with make -k

   # make -k CFLAGS_FOR_TARGET="${CROSS_CFLAGS} -DHAVE_SYS_ERRLIST" CXXFLAGS_FOR_TARGET="${CFLAGS_FOR_TARGET}" (...) make[1]: Target `all' not remade because of errors. rm gcc.pod make[1]: Leaving directory `/var/tmp/portage/gcc-3.4.1-r2/work/gcc-3.4.1/buildfullhere-ppc/gcc' make: *** [all-gcc] Error 2 make: Target `all' not remade because of errors.

   Not everything was built. It is okay, simply install the compiler files:

   Code listing 16: Installing Compiler

   # (cd gcc && make -k install)

   More errors but the compiler should have been installed:

   Code listing 17: More Errors

   make: Target `install' not remade because of errors.

   Congratulations! At this point you should have a working cross-compiler toolchain that is sufficient for distcc use. Go to Testing the Cross-compiler Toolchain.

2. For a full installation:

   Code listing 18: Configuring gcc (Bootstrap) - Full Toolchain

   # mkdir buildboothere-ppc # cd buildboothere-ppc # CFLAGS="${MY_CFLAGS} -Dinhibit_libc -static-libgcc" \ ../configure \ --prefix=${DEST} \ --host=${MY_CHOST} \ --target=${CROSS_CHOST} \ --with-newlib \ --disable-shared \ --disable-threads \ --enable-languages="c" \ --disable-multilib \ --disable-nls \ --enable-symvers=gnu \ --enable-__cxa_atexit \ --with-sysroot=/ \ --enable-static

   Note: We use the --with-sysroot / option rather than the deprecate --with-headers /usr/include option. Unlike the --with-headers /usr/include, the --with-sysroot option does not copy the system headers to ${DEST}/powerpc-unknown-linux-gnu/sys-include.

   Build and install:

   Code listing 19: Building and Installing gcc (Bootstrap) - Full Toolchain

   # make CFLAGS_FOR_TARGET="${CROSS_CFLAGS}" CXXFLAGS_FOR_TARGET="${CFLAGS_FOR_TARGET}" # make install

Step 4 - Installation of glibc 

Apply the following patch:

--- Makeconfig  2004-09-08 02:03:53.755231360 -0400
+++ Makeconfig  2004-09-08 02:23:03.817395336 -0400
@@ -522,16 +522,16 @@
 
 ifndef gnulib
 ifneq ($(have-as-needed),yes)
- libgcc_eh := -lgcc_eh
+ libgcc_eh :=
 else
  libgcc_eh := -Wl,--as-needed -lgcc_s$(libgcc_s_suffix) -Wl,--no-as-needed
 endif
 ifneq ($(have-cc-with-libunwind),yes)
  gnulib := -lgcc $(libgcc_eh)
- static-gnulib := -lgcc -lgcc_eh
+ static-gnulib := -lgcc
 else
  gnulib := -lgcc $(libgcc_eh) -lunwind
- static-gnulib := -lgcc -lgcc_eh -lunwind
+ static-gnulib := -lgcc -lunwind
 endif
 endif
 ifeq ($(elf),yes)

Configure, build and install glibc by using one of these alternatives.

1. For linuxthreads, do:

   Code listing 21: Building glibc - with linuxthreads

   # mkdir buildppc # cd buildppc # BUILD_CC="gcc" \ CC="${CROSS_CHOST}-gcc ${CROSS_CFLAGS}" \ AR="${CROSS_CHOST}-ar" \ RANLIB="${CROSS_CHOST}-ranlib" \ ../configure \ --prefix=${DEST} \ --host=powerpc-unknown-linux-gnu \ --build=${MY_CHOST} \ --without-tls \ --without-__thread \ --enable-add-ons=linuxthreads \ --enable-clocale=gnu \ --enable-kernel=${MIN_KV} \ --without-gd \ --without-cvs \ --disable-profile \ --disable-debug \ --with-headers="${DEST}/include" # make # make install

2. For nptl, do:

   Code listing 22: Building glibc - with nptl

   # mkdir buildppc # cd buildppc # cp ../nptl/sysdeps/pthread/pthread.h ${DEST}/include # mkdir ${DEST}/include/bits # cp ../nptl/sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h ${DEST}/include/bits # echo "libc_cv_forced_unwind=yes" > config.cache # echo "libc_cv_c_cleanup=yes" >> config.cache # BUILD_CC="gcc" \ CC="${CROSS_CHOST}-gcc ${CROSS_CFLAGS}" \ AR="${CROSS_CHOST}-ar" \ RANLIB="${CROSS_CHOST}-ranlib" \ ../configure \ --prefix=${DEST} \ --host=powerpc-unknown-linux-gnu \ --build=${MY_CHOST} \ --with-tls \ --with-__thread \ --enable-add-ons=nptl \ --enable-clocale=gnu \ --enable-kernel=${MIN_KV} \ --without-gd \ --disable-profile \ --disable-debug \ --with-headers="${DEST}/include" \ --cache-file=config.cache # make # make install

   Note: The differences from linuxthreads are shown in italic.

Step 5 - Installation of gcc (full) 

Build and install gcc (full):

# mkdir buildfullhere-ppc
# cd buildfullhere-ppc
# CFLAGS="${MY_CFLAGS}" \
../configure \
  --prefix=${DEST} \
  --target=${CROSS_CHOST} \
  --host=${MY_CHOST} \
  --disable-multilib \
  --disable-nls \
  --enable-shared \
  --enable-languages="${CROSS_GCCLANG}" \
  --without-included-gettext \
  --with-system-zlib \
  --enable-threads=posix \
  --enable-long-long \
  --enable-symvers=gnu \
  --disable-checking \
  --enable-cstdio=stdio \
  --enable-clocale=generic \
  --enable-__cxa_atexit \
  --enable-c99 \
  --enable-version-specific-runtime-libs \
  --with-local-prefix=${DEST}/local \
  --with-libs="${DEST}/lib" \
  --with-headers="${DEST}/include"

# make CFLAGS_FOR_TARGET="${CROSS_CFLAGS} -DHAVE_SYS_ERRLIST" CXXFLAGS_FOR_TARGET="${CFLAGS_FOR_TARGET}"

# make install

Congratulations! At this point you should have a working cross-compiler toolchain.

5. Testing the Cross-compiler Toolchain

Setup 

We use the hello world program as an example:

curly root # cat > /tmp/hello.c

#include <stdio.h>

int main(void) {
  printf("allo!\n");
}

We are compiling on a x86:

curly root # uname -a
Linux curly 2.6.7-rc3 #1 Tue Jun 15 02:26:52 EDT 2004 i686 AMD Athlon(tm) Processor AuthenticAMD GNU/Linux
curly root #

Full Toolchain 

We cross-compile the HelloWorld program and we get a PowerPC executable:

curly root # ${DEST}/bin/powerpc-unknown-linux-gnu-gcc hello.c
curly root # file a.out
a.out: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), for GNU/Linux 2.6.5, dynamically linked (uses shared libs), not stripped
curly root # 

We can send the executable to the PowerPC and execute it:

root@curly ~ # ssh ibou 'uname -a ; cd /tmp ; cat > hello ; chmod a+x hello; ./hello' < a.out
Linux ibou 2.6.7-gentoo-r14 #1 Sat Sep 4 18:51:22 EDT 2004 ppc 750FX PowerBook4,3 GNU/Linux
allo!
root@curly ~ # 

Minimal Toolchain 

If a full toolchain was not build, a full executable cannot be built. Although the code can be compiled, it cannot be linked. If the preprecessor installed for the cross-compiler, we can compile the program to assembly code or to object file. If the preprocessor was not installed, the source will need to be processed on the PowerPC first. It can then be transfered to the x86 and it can be used instead of the C source to perform the tests:

# gcc -E hello.c > hello.i

To compile to assembly, do:

root@curly /tmp # powerpc-unknown-linux-gnu-gcc -S hello.c

The output file hello.s should contain PowerPC assembly code:

        .file   "hello.c"
        .section        .rodata
        .align 2
.LC0:
        .string "allo!\n"
        .section        ".text"
        .align 2
        .globl main
        .type   main, @function
main:
        stwu 1,-32(1)
        mflr 0
        stw 31,28(1)
        stw 0,36(1)
        mr 31,1
        lis 9,.LC0@ha
        la 3,.LC0@l(9)
        bl printf
        mr 3,0
        lwz 11,0(1)
        lwz 0,4(11)
        mtlr 0
        lwz 31,-4(11)
        mr 1,11
        blr
        .size   main, .-main
        .section        .note.GNU-stack,"",@progbits
        .ident  "GCC: (GNU) 3.4.1 20040803 (Gentoo Linux 3.4.1-r2, ssp-3.4-2, pie-8.7.6.5)"

To compile to an object file:

root@curly /tmp # powerpc-unknown-linux-gnu-gcc -c hello.c
root@curly /tmp # file hello.o
hello.o: ELF 32-bit MSB relocatable, PowerPC or cisco 4500, version 1 (SYSV), not stripped

The object file can be linked and tested on the PowerPC:

root@curly /tmp # ssh ibou 'uname -a ; cd /tmp ; cat > hello.o ; gcc -o hello hello.o; ./hello' < hello.o
Linux ibou 2.6.7-gentoo-r14 #1 Sat Sep 4 18:51:22 EDT 2004 ppc 750FX PowerBook4,3 GNU/Linux
allo!
root@curly /tmp # 

6. Extra Cleanup for Distcc-Only Toolchain

If a minimal toolchain has been manually installed (i.e. not through portage), unnecessary files can be removed.

The distcc-only toolchain takes 28MB. With some cleanup, I reduced it to only 12MB. A full toolchain (without any cleanup) is 190MB.

We can remove: include files, unneeded executables, unneeded static libs, unused libs, documentation files (info and man pages), and locales.

# rm -rf ${DEST}/include \
  ${DEST}/powerpc-unknown-linux-gnu/include \
  ${DEST}/i686-pc-linux-gnu/powerpc-unknown-linux-gnu/include

# rm ${DEST}/bin/powerpc-unknown-linux-gnu-{cpp,gcc-3.4.1,gccbug,gcov}

# rm -f ${DEST}/i686-pc-linux-gnu/powerpc-unknown-linux-gnu/lib/*.{a,la}

# rm -rf ${DEST}/lib

# rm -rf ${DEST}/info ${DEST}/man

# rm -rf ${DEST}/share

We removed the preprocessor... To do a test, use preprocessed input:

root@curly /home/crossdev/ppc # gcc -E -o /tmp/hello.i /tmp/hello.c

root@curly /home/crossdev/ppc # powerpc-unknown-linux-gnu-gcc -o /tmp/hello.o -c /tmp/hello.i ; file /tmp/hello.o
/tmp/hello.o: ELF 32-bit MSB relocatable, PowerPC or cisco 4500, version 1 (SYSV), not stripped
root@curly /home/crossdev/ppc # 

7. Cross-compiling with distcc

Install distcc 

Install distcc. On Gentoo it can be done as follows:

emerge distcc
(Follow the setup instructions.)

Configuration for cross-compilation 

Add ${DEST}/bin (and optionnally ${DEST}/powerpc-unknown-linux-gnu/bin) to the distcc daemon path configuration. This is done by modifying /etc/init.d/distccd:

start() {
        rm -f ${DISTCCD_PIDFILE}
        ebegin "Starting distccd"
        TMPDIR="${TMPDIR}" \
        PATH="/home/crossdev/ppc/bin:/home/crossdev/ppc/powerpc-unknown-linux-gnu/bin" \
        /sbin/start-stop-daemon --start --quiet --startas ${DISTCCD_EXEC} \
        --pidfile ${DISTCCD_PIDFILE} -- \
        --pid-file ${DISTCCD_PIDFILE} -N ${DISTCCD_NICE} --user distcc \
        ${DISTCCD_OPTS}

        eend $?
}

If you do not add this directory to the path, you might encounter some problems for some packages that do not respect the $CC and $CXX variables. If you add this directory however, you will run into problems if you do distributed compilation for the host architecture (x86 in our case) as distcc will use the cross-compiler instead of the native compiler.

Compiling 

If you added ${DEST}/powerpc-unknown-linux-gnu/bin to the distccd path, you can compile as usual:

# emerge vi

Otherwise, you need to set the CC and CXX environment variables to specify the compiler to use. For example:

# CC=powerpc-unknown-linux-gnu CXX=powerpc-unknown-linux-gnu emerge vi

An approach suggested here can be used to select the appropriate compiler without manually setting CC and CXX before emerge. It consists of setting these variables to the full ${CHOST}-gcc and ${CHOST}-g++ names in the compiler configuration files located in /etc/env.d/gcc. This is especially useful when the same distccd servers are used for multiple architectures.

8. Adding ccache Support

Note that with distributed compilation, the code is cached on the client side. If you intend to cross-compile code directly on the host with the cross-compiler, you can add ccache support with the command:

# ccache-config --install-links powerpc-unknown-linux-gnu

9. Resources

This section will contain various links to web pages and forums that discuss cross-compilation for a variety of operating systems and architectures.

10. About this Document

Note that several code segments are coming from the crossdev script. The crossdev script is licensed under GPL-2.

This document uses a modified XSL stylesheet from the Gentoo project: guide.xsl.

Send your comments, suggestions, or corrections to: