diff mbox

[Xen-devel,OSSTEST,v2,09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs

Message ID 1398681696-2773-9-git-send-email-ian.campbell@citrix.com
State New
Headers show

Commit Message

Ian Campbell April 28, 2014, 10:41 a.m. UTC 
This patch introduces ts-debian-di-install which can install Debian from a
netboot (PXE) debian installer image. By default it installs from the d-i image
used by osstest (using the special Xen PV guest enabled flavour where
necessary) but it can also install the current release versions from Squeeze
onwards (up to and including Sid) and the d-i daily builds. This is controlled
by runvars {Guest}_diver = osstest|current and {Guest}_dist = squeeze|...|sid.
The resulting guests boot the distro kernel using pygrub (pvgrub will follow).

The distros flights differ substantially from the existing flights. Introduce
make-distros-flight using the functionality previously refactored into
mfi-common. The new flight tests all versions of Debian from Squeeze onward as
an amd64, i386 and armhf guests (armhf from Jessie onwards only) using the
usual smoke tests.

Add the new cases to sg-run-job

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-distros-flight  | 101 +++++++++++++++++++++++++++++++
 sg-run-job           |  11 ++++
 ts-debian-di-install | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 279 insertions(+)
 create mode 100755 make-distros-flight
 create mode 100755 ts-debian-di-install

Comments

Ian Jackson May 2, 2014, 11:46 a.m. UTC | #1 
Ian Campbell writes ("[PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
...
> +if [ x$buildflight = x ]; then
> +
> +  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
> +     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
> +  fi

This has been clone-and-hacked from make-flight and presumably belongs
in mfi-common (perhaps a subroutine defined there).

...
> +    if ( $di_ver eq "osstest" ) {
...
> +	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
> +	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
> +    } else {
...
> +	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
> +	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);

Had you not noticed that you'd written out the store_runvars lines
twice ?

> +    target_cmd_root($ho, <<END);
> +rm -rf /root/di
> +mkdir /root/di

You should put the flight and job in this filename in case of future
host-sharing.

> +    my $cmdline = join(" ", (
> +			   "debian-installer/exit/always_halt=true",
> +			   "--",
> +			   "console=hvc0",
> +			   "auto-install/enable=true",
> +			   "hostname=$gho->{Name}",
> +			   "domain=$c{TestHostDomain}",
> +			   "url=$ps_url",
> +			   "DEBIAN_FRONTEND=text",
> +			   "netcfg/dhcp_timeout=150",
> +			   "netcfg/choose_interface=eth0",

I think you want to use Osstest::Debian::di_installcmdline_core.

> +    my $blcfg = <<END;
> +bootloader = "pygrub"
> +END

I'm not sure why you've broken this out.  (I just mention this in case
it's not deliberate.)

Thanks,
Ian.
Ian Campbell May 2, 2014, 12:28 p.m. UTC | #2 
On Fri, 2014-05-02 at 12:46 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
> ...
> > +if [ x$buildflight = x ]; then
> > +
> > +  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
> > +     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
> > +  fi
> 
> This has been clone-and-hacked from make-flight and presumably belongs
> in mfi-common (perhaps a subroutine defined there).

Ack.

> ...
> > +    if ( $di_ver eq "osstest" ) {
> ...
> > +	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
> > +	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
> > +    } else {
> ...
> > +	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
> > +	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
> 
> Had you not noticed that you'd written out the store_runvars lines
> twice ?

The $netboot_foo are local in scope to within the two halves of the
if/else.

> > +    target_cmd_root($ho, <<END);
> > +rm -rf /root/di
> > +mkdir /root/di
> 
> You should put the flight and job in this filename in case of future
> host-sharing.

Ack.

> > +    my $cmdline = join(" ", (
> > +			   "debian-installer/exit/always_halt=true",
> > +			   "--",
> > +			   "console=hvc0",
> > +			   "auto-install/enable=true",
> > +			   "hostname=$gho->{Name}",
> > +			   "domain=$c{TestHostDomain}",
> > +			   "url=$ps_url",
> > +			   "DEBIAN_FRONTEND=text",
> > +			   "netcfg/dhcp_timeout=150",
> > +			   "netcfg/choose_interface=eth0",
> 
> I think you want to use Osstest::Debian::di_installcmdline_core.

I think I do too, thanks for the pointer.

To what extent are $ho and $gho interchangeable? Is it ok to call e.g.
get_host_property on a guest (as that function would do).?

> > +    my $blcfg = <<END;
> > +bootloader = "pygrub"
> > +END
> 
> I'm not sure why you've broken this out.  (I just mention this in case
> it's not deliberate.)

It saved some refactoring noise in a later patch when I add support for
pvgrub.

Ian.
Ian Jackson May 2, 2014, 2:14 p.m. UTC | #3 
Ian Campbell writes ("Re: [PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
> On Fri, 2014-05-02 at 12:46 +0100, Ian Jackson wrote:
> > Had you not noticed that you'd written out the store_runvars lines
> > twice ?
> 
> The $netboot_foo are local in scope to within the two halves of the
> if/else.

You can move declare them (with my) outside the if to fix this
problem.  That's IMO much better than writing out those runvar stores
twice.

> > I think you want to use Osstest::Debian::di_installcmdline_core.
> 
> I think I do too, thanks for the pointer.
> 
> To what extent are $ho and $gho interchangeable? Is it ok to call e.g.
> get_host_property on a guest (as that function would do).?

Hrm.  Arguably according to its name, get_host_property should look
into the $ho->{Host} but it doesn't and that's not what you want here.

Is whether we should do that a function of the property, or a function
of the enquiring call site ?  The code at the moment implies the
former: there is explicit machinery for doing the indirection for
(e.g.) DhcpWatchMethod.

I think until we discover otherwise, we can declare that it's fine to
call get_host_property on a guest object, provided that we understand
that the answer is always going to be "no such property set".  Perhaps
this should be documented.

> > > +    my $blcfg = <<END;
> > > +bootloader = "pygrub"
> > > +END
> > 
> > I'm not sure why you've broken this out.  (I just mention this in case
> > it's not deliberate.)
> 
> It saved some refactoring noise in a later patch when I add support for
> pvgrub.

So I now see.

Thanks,
Ian.
Ian Campbell May 2, 2014, 3:19 p.m. UTC | #4 
On Fri, 2014-05-02 at 15:14 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
> > On Fri, 2014-05-02 at 12:46 +0100, Ian Jackson wrote:
> > > Had you not noticed that you'd written out the store_runvars lines
> > > twice ?
> > 
> > The $netboot_foo are local in scope to within the two halves of the
> > if/else.
> 
> You can move declare them (with my) outside the if to fix this
> problem.  That's IMO much better than writing out those runvar stores
> twice.

OK.

> > > I think you want to use Osstest::Debian::di_installcmdline_core.
> > 
> > I think I do too, thanks for the pointer.
> > 
> > To what extent are $ho and $gho interchangeable? Is it ok to call e.g.
> > get_host_property on a guest (as that function would do).?
> 
> Hrm.  Arguably according to its name, get_host_property should look
> into the $ho->{Host} but it doesn't and that's not what you want here.
> 
> Is whether we should do that a function of the property, or a function
> of the enquiring call site ?  The code at the moment implies the
> former: there is explicit machinery for doing the indirection for
> (e.g.) DhcpWatchMethod.
> 
> I think until we discover otherwise, we can declare that it's fine to
> call get_host_property on a guest object, provided that we understand
> that the answer is always going to be "no such property set".

I'm happy to do that.

> Perhaps this should be documented.

I'll try and remember to slip something in.

Ian.
diff mbox

Patch

diff --git a/make-distros-flight b/make-distros-flight
new file mode 100755
index 0000000..abc24ff
--- /dev/null
+++ b/make-distros-flight
@@ -0,0 +1,101 @@ 
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. ap-common
+. cri-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+if [ x$buildflight = x ]; then
+
+  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
+     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
+  fi
+
+  WANT_XEND=false REVISION_LINUX_OLD=disable
+
+  create_build_jobs
+
+else
+
+  bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+  if [ "$xenarch" = "i386" ]; then return 1; fi
+  return 0
+}
+
+test_matrix_branch_filter_callback () {
+    :
+}
+
+test_do_one_netboot () {
+  job_create_test test-$xenarch$kern-$dom0arch-$domU-$dist-netboot \
+    test-debian-di xl $xenarch $dom0arch                        \
+      kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
+      debian_arch=$domU                                         \
+      debian_dist=$dist                                         \
+      debian_method=netboot                                     \
+      debian_diver=current                                      \
+      all_hostflags=$most_hostflags
+}
+
+test_matrix_do_one () {
+  case ${xenarch} in
+  amd64) domUarches="amd64 i386";;
+  armhf) domUarches="armhf";;
+  esac
+
+  for domU in $domUarches ; do
+    for dist in squeeze wheezy jessie sid daily ; do
+      case ${domU}_${dist} in
+      armhf_squeeze) continue;; # No armhf in Squeeze
+      armhf_wheezy) continue;; # No armhf guest support in Wheezy
+      *) ;;
+      esac
+
+      test_do_one_netboot
+
+    done
+
+  done
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/sg-run-job b/sg-run-job
index 3d44331..789315a 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -246,6 +246,17 @@  proc run-job/test-debian {} {
     test-guest debian
 }
 
+proc install-guest-debian-di {} {
+    run-ts . = ts-debian-di-install
+    run-ts . = ts-guest-start + debian
+}
+
+proc need-hosts/test-debian-di {} { return host }
+proc run-job/test-debian-di {} {
+    install-guest-debian-di
+    test-guest debian
+}
+
 proc need-hosts/test-freebsd {} { return host }
 proc run-job/test-freebsd {} {
     run-ts . = ts-freebsd-install
diff --git a/ts-debian-di-install b/ts-debian-di-install
new file mode 100755
index 0000000..a71a336
--- /dev/null
+++ b/ts-debian-di-install
@@ -0,0 +1,167 @@ 
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+$gn ||= 'debian';
+
+our $ho= selecthost($whhost);
+
+our $ram_mb=    512;
+our $disk_mb= 10000;
+
+our $guesthost= "$gn.guest.osstest";
+our $gho;
+
+sub prep () {
+    target_install_packages_norec($ho, qw(lvm2));
+
+    $gho= prepareguest($ho, $gn, $guesthost, 22,
+                       $disk_mb, 40);
+
+    prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
+
+    target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
+
+sub setup_netboot($$$)
+{
+    my ($didir, $arch, $suite) = @_;
+
+    my $di_ver= $r{"$gho->{Guest}_diver"} || "osstest";
+
+    if ( $di_ver eq "osstest" ) {
+	my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
+
+	my $netboot_kernel = "$di_path/vmlinuz-xen";
+	my $netboot_initrd = "$di_path/initrd.gz-xen";
+
+	target_putfile_root($ho, 60, $netboot_kernel, "$didir/kernel_${suite}_${arch}");
+	target_putfile_root($ho, 60, $netboot_initrd, "$didir/initrd_${suite}_${arch}");
+
+	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
+	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
+    } else {
+	my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+
+	my $di_url = $suite eq "daily" ?
+	    "http://d-i.debian.org/daily-images/$arch/daily/netboot/xen" :
+	    "$mirror/dists/$suite/main/installer-$arch/$di_ver/images/netboot/xen";
+
+	my $netboot_kernel = "$di_url/vmlinuz";
+	my $netboot_initrd = "$di_url/initrd.gz";
+
+	target_fetchurl($ho, $netboot_kernel, "$didir/kernel_${suite}_${arch}");
+	target_fetchurl($ho, $netboot_initrd, "$didir/initrd_${suite}_${arch}");
+
+	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
+	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
+    }
+
+    return <<END;
+kernel      = "$didir/kernel_${suite}_${arch}"
+ramdisk     = "$didir/initrd_${suite}_${arch}"
+END
+}
+sub ginstall () {
+    my $arch= $r{"$gho->{Guest}_arch"};
+    my $method= $r{"$gho->{Guest}_method"};
+
+    target_cmd_root($ho, <<END);
+rm -rf /root/di
+mkdir /root/di
+END
+
+    my ($method_cfg, $ps_url, $extra_disk);
+
+    if ( $method eq "netboot" )
+    {
+	my $suite= $r{"$gho->{Guest}_dist"};
+	logm("$method $suite/$arch");
+
+	$method_cfg = setup_netboot("/root/di", $arch, $suite);
+
+	$suite = "sid" if $suite eq "daily";
+
+	$ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+
+	$extra_disk = "";
+    }
+    else
+    {
+	die "$method";
+    }
+
+    my $cmdline = join(" ", (
+			   "debian-installer/exit/always_halt=true",
+			   "--",
+			   "console=hvc0",
+			   "auto-install/enable=true",
+			   "hostname=$gho->{Name}",
+			   "domain=$c{TestHostDomain}",
+			   "url=$ps_url",
+			   "DEBIAN_FRONTEND=text",
+			   "netcfg/dhcp_timeout=150",
+			   "netcfg/choose_interface=eth0",
+		       ));
+
+    my %install_xopts = (
+	OnPowerOff => "preserve"
+    );
+
+    prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
+$method_cfg
+extra       = "$cmdline"
+#
+disk        = [
+            $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+            ]
+END
+
+    my $cmd= toolstack()->{Command}." create ".
+        $r{ $gho->{Guest}.'_'. toolstack()->{CfgPathVar} };
+    target_cmd_root($ho, $cmd, 300);
+
+    guest_checkrunning($ho, $gho) or die "$gho->{Name} not running";
+
+    guest_await_shutdown($ho,$gho,2000);
+    guest_destroy($ho,$gho);
+
+    my $blcfg = <<END;
+bootloader = "pygrub"
+END
+
+    prepareguest_part_xencfg($ho, $gho, $ram_mb, {}, <<END);
+$blcfg
+#
+disk        = [
+            'phy:$gho->{Lvdev},xvda,w'
+            ]
+END
+    return;
+}
+
+prep();
+ginstall();