[RISU,v5,05/13] build-all-archs: support cross building via docker

Message ID 20170619104655.31104-6-alex.bennee@linaro.org
State Superseded
Headers show
Series
  • RISU record/replay patches
Related show

Commit Message

Alex Bennée June 19, 2017, 10:46 a.m.
If we want to link to any other libraries we might find using simple
cross toolchains doesn't work so well. One way around this is to use a
dockerised cross-toolchain which then won't clash with your host
system. If the user specifies --use-docker the obvious will be done.

By default we use the QEMU projects qemu:debian-FOO-cross images as
RISU hackers are likely to be QEMU developers too. However any docker
tag can be passed on the command line.

If none of the docker images have usable compilers we fall back to
checking the host path.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>


---
v5
  - swapped with --static patch so this can be dropped if desired
---
 build-all-archs | 46 +++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 39 insertions(+), 7 deletions(-)

-- 
2.13.0

Comments

Peter Maydell June 20, 2017, 2:09 p.m. | #1
On 19 June 2017 at 11:46, Alex Bennée <alex.bennee@linaro.org> wrote:
> If we want to link to any other libraries we might find using simple

> cross toolchains doesn't work so well. One way around this is to use a

> dockerised cross-toolchain which then won't clash with your host

> system. If the user specifies --use-docker the obvious will be done.

>

> By default we use the QEMU projects qemu:debian-FOO-cross images as

> RISU hackers are likely to be QEMU developers too. However any docker

> tag can be passed on the command line.

>

> If none of the docker images have usable compilers we fall back to

> checking the host path.

>

> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>


$ ./build-all-archs --use-docker
Got permission denied while trying to connect to the Docker daemon
socket at unix:///var/run/docker.sock: Get
http://%2Fvar%2Frun%2Fdocker.sock/v1.27/images/json?filters=%7B%22reference%22%3A%7B%22qemu%22%3Atrue%7D%7D:
dial unix /var/run/docker.sock: connect: permission denied

but maybe my local docker setup is just broken?

thanks
-- PMM
Alex Bennée June 20, 2017, 2:57 p.m. | #2
Peter Maydell <peter.maydell@linaro.org> writes:

> On 19 June 2017 at 11:46, Alex Bennée <alex.bennee@linaro.org> wrote:

>> If we want to link to any other libraries we might find using simple

>> cross toolchains doesn't work so well. One way around this is to use a

>> dockerised cross-toolchain which then won't clash with your host

>> system. If the user specifies --use-docker the obvious will be done.

>>

>> By default we use the QEMU projects qemu:debian-FOO-cross images as

>> RISU hackers are likely to be QEMU developers too. However any docker

>> tag can be passed on the command line.

>>

>> If none of the docker images have usable compilers we fall back to

>> checking the host path.

>>

>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

>

> $ ./build-all-archs --use-docker

> Got permission denied while trying to connect to the Docker daemon

> socket at unix:///var/run/docker.sock: Get

> http://%2Fvar%2Frun%2Fdocker.sock/v1.27/images/json?filters=%7B%22reference%22%3A%7B%22qemu%22%3Atrue%7D%7D:

> dial unix /var/run/docker.sock: connect: permission denied

>

> but maybe my local docker setup is just broken?


Generally you need to add yourself to the "docker" group to have access.
This isn't recommended in production but the alternative is to preface
all docker calls with sudo and set up permissions that way. The official
docs discuss it here:

  https://docs.docker.com/engine/installation/linux/linux-postinstall/

The advice that "only trusted users should be allowed to control your
Docker daemon." is valid but irrelevant to developers who generally have
complete control of their own box.

>

> thanks

> -- PMM



--
Alex Bennée

Patch hide | download patch | download mbox

diff --git a/build-all-archs b/build-all-archs
index 581a1b4..63918e5 100755
--- a/build-all-archs
+++ b/build-all-archs
@@ -11,9 +11,6 @@ 
 # Contributors:
 #     Peter Maydell (Linaro) - initial implementation
 
-# So we notice risugen failing even though it's in a pipeline
-set -o pipefail
-
 # Simple usage
 usage() {
     cat <<-EOF
@@ -21,7 +18,10 @@  usage() {
 
         Options include:
             --static               build a static binary
+            --use-docker[=tags]    use docker cross compile
 
+        If specifying docker the default will be to use the any
+        qemu:debian-FOO-cross targets available on your system.
 EOF
     exit 1
 }
@@ -37,6 +37,14 @@  while [[ "$1" = -* ]]; do
         --static)
             CONF="--static"
             ;;
+        --use-docker)
+            if [ -z "$arg" ]; then
+                default_tags=$(docker images qemu --format "{{.Repository}}:{{.Tag}}" | grep "\(arm\|power\).*cross$")
+                docker_tags=$(echo $default_tags | sed 's/\n/\s/g' )
+            else
+                docker_tags="$arg"
+            fi
+            ;;
         --help)
             usage
             ;;
@@ -48,10 +56,24 @@  done
 
 # Debian stretch and Ubuntu Xenial have cross compiler packages for
 # all of these:
-# gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu gcc-m68k-linux-gnu
-# gcc-powerpc64le-linux-gnu gcc-powerpc64-linux-gnu
+#   gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu gcc-m68k-linux-gnu
+#   gcc-powerpc64le-linux-gnu gcc-powerpc64-linux-gnu
+# If docker is enabled we just brute force the various images until we
+# can set the one that has a workable cross compiler.
+
+DOCKER_RUN="docker run --rm -t -u $(id -u) -v $(pwd):$(pwd) -w $(pwd)"
 
 program_exists() {
+    if [ ! -z "$docker_tags" ]; then
+        use_docker_tag=""
+        for tag in $docker_tags; do
+            if ${DOCKER_RUN} ${tag} /bin/bash -c "command -v $1 >/dev/null"; then
+                use_docker_tag=$tag
+                return
+            fi
+        done
+    fi
+
     command -v "$1" >/dev/null 2>&1
 }
 
@@ -62,19 +84,29 @@  for triplet in aarch64-linux-gnu arm-linux-gnueabihf m68k-linux-gnu \
     if ! program_exists "${triplet}-gcc"; then
         echo "Skipping ${triplet}: no compiler found"
         continue
+    else
+        echo "Building ${triplet} on ${use_docker_tag:-host}..."
     fi
 
     # Do a complete rebuild from scratch, because it's cheap enough.
     rm -rf build/${triplet}
     mkdir -p build/${triplet}
 
-    (cd build/${triplet} && CROSS_PREFIX="${triplet}-"  ../../configure ${CONF})
-    make -C build/${triplet} EXTRA_CFLAGS=-Werror
+    CONFIGURE="cd build/${triplet} && CROSS_PREFIX="${triplet}-"  ../../configure ${CONF}"
+    MAKE="make -C build/${triplet} EXTRA_CFLAGS=-Werror"
 
+    if [ -z "$use_docker_tag" ]; then
+        /bin/bash -c "${CONFIGURE}"
+        ${MAKE}
+    else
+        ${DOCKER_RUN} $use_docker_tag /bin/bash -c "${CONFIGURE}"
+        ${DOCKER_RUN} $use_docker_tag /bin/bash -c "${MAKE}"
+    fi
 done
 
 # Now run risugen for all architectures
 mkdir -p build/risuout
+set -o pipefail # detect failures in pipeline
 
 for f in *.risu; do
     echo "Running risugen on $f..."