Migrating from lava-deployment-tool to packages

Migrating from lava-deployment-tool to packages

Please read the section on LAVA Components for details of how the LAVA packaging is organised. In particular, note the section on Daemon renaming.

Only Debian Jessie is supported for new migrations.

  1. Replacing precise with Debian Jessie

Either involves moving the (possibly large) amount of files in the LAVA log directories, so a backup of those files would be useful.

Warning

A backup of the entire server is also recommended as there is no support for downgrading from packaging back to lava-deployment-tool.

The best method for any one installation will depend on the local admins. A fresh install is often faster than an upgrade from one LTS to another, as long as the database export process is reliable. If there is a lot of data on the machine besides LAVA content, there will need to be some consideration of how a kernel and OS upgrade will affect those other services.

Due to limitations within lava-deployment-tool and lava-manifest, there is no way to migrate to the packaging based on django1.6 using the tools already installed within the existing LAVA instance.

Note

A default LAVA install from packages supports http://, not https:// in the Apache configuration. If your existing instance uses https://, ensure you have a copy of the apache configuration and remember to port those changes to apache2.4 in /etc/apache2/sites-available/lava-server.conf.

Exporting the postgres database

Most admins will have custom ways to get a dump of the postgres database and any script which can dump the data and import it successfully into a fresh, upgraded, install will be suitable.

lava-deployment-tool would have read variables from the instance.conf and used a call based on:

$ pg_dump \
     --no-owner \
     --format=custom \
     --host=$dbserver \
     --port=$dbport \
     --username=$dbuser \
     --no-password $dbname \
     --schema=public \
     > "$destdir/database.dump"

A new install will need the database user created:

sudo -u postgres createuser \
    --no-createdb --encrypted \
    --login --no-superuser \
    --no-createrole --no-password \
    --port $dbport $dbuser
sudo -u postgres psql --port 5432 \
    --command="ALTER USER \"lavaserver\" WITH PASSWORD '$dbpass';"

lava-deployment-tool would attempt a restore from this dump by using the variables from instance.conf and calls based on:

sudo -u postgres dropdb \
    --port $dbport \
    $dbname || true
sudo -u postgres createdb \
    --encoding=UTF-8 \
    --locale=en_US.UTF-8 \
    --template=template0 \
    --owner=$dbuser \
    --port $dbport \
    --no-password \
    $dbname
sudo -u postgres createlang \
    --port $dbport \
    plpgsql \
    $dbname || true
sudo -u postgres pg_restore \
    --exit-on-error --no-owner \
    --port $dbport \
    --role $dbuser \
    --dbname $dbname \
    "${1}" > /dev/null

Tip

If your database is very large, consider adding the --jobs option to pg_restore to parallelise the postgresql workload. See the postgresql documentation (man 1 pg_restore) for the best value to pass as the number of concurrent jobs to use.

Whatever method is chosen, verify that the dump from postgresql-9.1 can be successfully imported into postgresql-9.3 then check the migration by connecting to the new database using the username, database name and password specified in instance.conf and check for the relevant tables. e.g.:

sudo su postgres lavaserver
psql
 psql (9.3.4)
 ...
postgres=# \l
 lava-production | lava-production | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 ...
postgres=# \dt
  ...
   public | dashboard_app_attachment                   | table | lavaserver
  ...
postgres=# \q

Assumptions

  1. LAVA is already installed using lava-deployment-tool on Ubuntu Precise Pangolin 12.04LTS in /srv/lava/instances/

  2. postgresql9.1 is installed and running on port 5432:

    ls -a /var/run/postgresql/
    
  3. there are idle devices or possibly running test jobs

  4. any local buildouts are either removed or merged back to master and updated. (This is a precaution to ensure that there are no development changes like database migrations which exist only in the buildout and not in master.)

Requirements

To copy the test job log files to the new location, it can be useful to have rsync installed on each machine, it is not always part of a LAVA install.

The only parts of the existing LAVA instance which will be retained are:

  • The test job log output, bundles and attachments:

    /srv/lava/instances/<INSTANCE>/var/lib/lava-server/media/
    
  • The database (master instance only) See Exporting the postgres database.

  • The device configuration files:

    /srv/lava/instances/<INSTANCE>/etc/lava-dispatcher/devices/
    /srv/lava/instances/<INSTANCE>/etc/lava-dispatcher/device-types/
    
  • The lava-server instance.conf file:

    /srv/lava/instances/<INSTANCE>/etc/lava-server/instance.conf
    

Other configuration files are ported or generated by the packaging.

Preparing for the upgrade

  1. Declare a maintenance window for scheduled downtime.
  2. Take all devices offline using the Django admin interface. Wait for any devices in status GoingOffline to complete the test job or cancel the test job if necessary.
  3. Ensure suitable backups exist for the database, device configuration, test job output files and the instance.conf.
  4. Ensure the machine has enough free space for a large set of package downloads. Ensure that the master instance also has enough free space for a copy of the test job output directories.
  5. Incorporate into the plan for the upgrade that the master will need to be upgraded but then work will need to concentrate on all the remote_worker_upgrade tasks before restarting the lava-server service on the master instance or putting any devices back online.
  6. Exit out of all shells currently using the /srv/lava/instances/<INSTANCE>/bin/activate virtual environment settings.
  7. Ensure that any local buildouts are either removed or merged back to master and updated. (This is a precaution to ensure that there are no development changes like database migrations which exist only in the buildout and not in master.)

Select the upgrade path:

Only Debian Jessie is supported as the upgrade path for LAVA releases after 2015.9

Upgrading LAVA to Debian Jessie

See Installing on Debian Jessie.

The recommended method to upgrade LAVA to Debian is to backup critical data on the Ubuntu Precise machine and then install a fresh Debian install. See Requirements.

It is possible to upgrade from Ubuntu to Debian but it is not recommended as it may end up with a mix of package setups and an unexpected final configuration.

Most of the steps are similar to the Ubuntu upgrade steps and these instructions also cover if you choose to make a fresh install of Ubuntu Trusty Tahr 14.04LTS.

The data needed off the old Precise instance will be:

  1. The test job data:

    /srv/lava/instances/<INSTANCE>/var/lib/lava-server/media/*
    
  2. The database (except for remote workers) See Exporting the postgres database.

    • The device configuration files:

      /srv/lava/instances/<INSTANCE>/etc/lava-dispatcher/devices/
      /srv/lava/instances/<INSTANCE>/etc/lava-dispatcher/device-types/
      
  3. The instance configuration:

    /srv/lava/instances/<INSTANCE>/etc/lava-server/instance.conf
    

To switch the OS, it may be best to retire the old machine / VM and put it onto a different network address and hostname. Then dump the postgres database and create a backup of the test job data.

The choice between using Jessie and Sid is entirely down to you. There is no particular reason to upgrade to jessie as a route to unstable, you can just go from wheezy to unstable, especially with a server-based install without a graphical user interface.

Installing a LAVA master instance on Debian

The process does not differ greatly from the standard installation instructions for Debian-based distributions. The extra stages occur between installation of the base system and installation of the LAVA packages.

  1. Download an ISO for Debian 7.5 Wheezy from http://www.debian.org/

  2. Install on required machine - no need for a desktop environment and the database installation is best left until after the upgrade to Jessie. openssh-server would be useful.

  3. Edit the apt sources list to point at jessie instead of wheezy:

    $ sudo vim /etc/apt/sources.list
    
  4. update, upgrade and then dist-upgrade:

    sudo apt-get update
    sudo apt-get upgrade
    sudo apt-get dist-upgrade
    sudo apt-get autoclean
    

    Avoid making manual changes between the upgrade and dist-upgrade steps - glibc will be upgraded and some daemons will need to be restared, this is best done automatically when prompted by debconf.

    The upgrade will bring in a new kernel, so a reboot is recommended at this point.

    Tip

    apt has migrated to version 1.0 in Jessie, which means that some commands can now be run as just apt as well as the previous apt-get. See man 1 apt after the upgrade.

  5. Clean-up after the upgrade.

    Apache has been upgraded to 2.4, so apache2.2 is one of many packages which can be safely removed:

    sudo apt-get --purge autoremove
    
  6. Add the LAVA packaging repository.

    All packages are in the Jessie release, this step is no longer required.

  7. Migrate the instance configuration to the packaging location.

    The packages will respect an existing LAVA configuration, if the relevant files are in the correct location /etc/lava-server/instance.conf. Copy the instance.conf from the precise box to the new Debian machine and put into place. e.g.:

    sudo mkdir -p /etc/lava-server/
    sudo cp /tmp/instance.conf /etc/lava-server/instance.conf
    

    Convert the LAVA_PREFIX in /etc/lava-server/instance.conf to the FHS (Filesystem Hierarchy Standard) compliant path:

    LAVA_PREFIX="/var/lib/lava-server/"
    

    Some settings are no longer used by the packaging but these will simply be ignored by the packaging.

  8. Migrate the instance logfiles to the packaging location.

    The permissions on these files will be fixed once lava-server is installed. Depending on how the files were copied from the Ubuntu machine, the files can be decompressed directly into the new location.

  9. Import the postgres database dump.

    Use the values in the /etc/lava-server/instance.conf to import the postgres data with the correct username, password and database access.

  10. Install LAVA from packages:

    sudo apt update
    sudo apt install lava-server
    

    The install will prompt for the instance name, you can specify the same instance name as the original lava-deployment-tool instance but this no longer affects where files are actually installed, nor does it affect the database name or database user. The instance name becomes a simple label with the packaging upgrade.

  11. Enable the lava-server apache configuration:

    sudo a2dissite 000-default
    sudo a2ensite lava-server
    sudo service apache2 restart
    
  12. Check your tftp support - see TFTP support requirement.

  13. Restart daemons affected by the installation:

    sudo service tftpd-hpa restart
    
  14. Ensure all devices remain offline.

  15. Configure the master to work with a remote worker.

See Remote databases and Postgresql configuration. Remember to use the LAVA_DB_USER and LAVA_DB_NAME from the instance.conf on the master. e.g.:

host    lava-playground    lava-playground    0.0.0.0/0    md5
  1. Pause to Install a LAVA remote worker using Debian.
  2. Run forced healthchecks on devices.
  3. Return devices to Online status.
  4. Complete scheduler maintenance.

Install a LAVA remote worker using Debian

The process does not differ greatly from the standard installation instructions for Debian-based distributions. The extra stages occur between installation of the base system and installation of the LAVA packages.

  1. Download an ISO for Debian 7.5 Wheezy from http://www.debian.org/

  2. Install on required machine - no need for a desktop environment, openssh-server would be useful.

  3. Change apt sources to point at jessie instead of wheezy:

    /etc/apt/sources.list
    
  4. update, upgrade and then dist-upgrade:

    sudo apt-get update
    sudo apt-get upgrade
    sudo apt-get dist-upgrade
    sudo apt-get autoclean
    

    Avoid making manual changes between the upgrade and dist-upgrade steps - glibc will be upgraded and some daemons will need to be restared, this is best done automatically when prompted by debconf.

    The upgrade will bring in a new kernel, so a reboot is recommended at this point.

    Tip

    apt has migrated to version 1.0 in Jessie, which means that some commands can now be run as just apt as well as the previous apt-get. See man 1 apt after the upgrade.

  5. Clean-up after the upgrade.

    Apache has been upgraded to 2.4, so apache2.2 is one of many packages which can be safely removed:

    sudo apt-get --purge autoremove
    
  6. Add the LAVA packaging repository.

    All packages are in the Jessie release, this step is no longer required.

  7. Migrate the instance configuration to the packaging location.

    The packages will respect an existing LAVA configuration but still ask the questions, so keep a terminal window open with the values. Copy the instance.conf from the precise box to the new Debian machine and put into place. e.g.:

    sudo mkdir -p /etc/lava-server/
    sudo cp /tmp/instance.conf /etc/lava-server/instance.conf
    

    Convert the LAVA_PREFIX in /etc/lava-server/instance.conf to the FHS (Filesystem Hierarchy Standard) compliant path:

    LAVA_PREFIX="/var/lib/lava-server/"
    

    Some settings are no longer used by the packaging but these will simply be ignored by the packaging.

  8. Do not migrate the instance logfiles to the packaging location.

    There is no rsync operation on a remote worker - the files are on an sshfs from the master. Ensure that /var/lib/lava-server/default/media is empty and that there is no current sshfs mount.

  9. Install LAVA from packages:

    sudo apt update
    sudo apt install lava-server
    

    The install will prompt for the instance name, you can specify the same instance name as the original lava-deployment-tool instance but this no longer affects where files are actually installed, nor does it affect the database name or database user. The instance name becomes a simple label with the packaging upgrade.

  10. Configure the remote worker

    See Configuring remote worker to setup the SSH key, the fuse configuration and lava-coordinator.

    Restart the lava-server daemon once done and check that the SSHFS mount operations has worked. See Mounting the SSHFS.

  11. Enable apache on the remote worker.

    This is used to serve modified files to the devices:

    sudo a2dissite 000-default
    sudo a2ensite lava-server
    sudo service apache2 restart
    
  12. Check your tftp support - see TFTP support requirement.

  13. Restart daemons affected by the installation:

    sudo service tftpd-hpa restart
    
  14. Return to Installing a LAVA master instance on Debian.