Upgrading GitLab from 6x to 10x

Recently at my day job, the client tasked me with bringing their development stack up to date. The (fun) difficultly here was that these servers were about 4 years out of date and included Jenkins, Sonar, GitLab), and Nexus. Jenkins was the jumbled mess you'd expect with unsupported plugins, new configurations breaking builds, and more fun, Sonar was so out of date I started from scratch after dropping the old databse, but GitLab was the one I was looking forward to, and while it was enlightening, it wasn't much fun! First I'll say that I'm a big fan of GitLab, it's the only thing I'd want to use internally at a company, and the community edition has a ton of great features. The main issue with my setup was that it was so out of date and required us to step up one major verision at a time; so 6.x up to 7.x, 7.x up to 8.x, 8.x up to 9.x, and finally, 9.x up to 10.2.x.

The Plan

After trying, failing, trying again, and failing again to upgrade GitLab, I came up with some tricks that helped me (eventually) succeed.

Getting started

Install GitLab on the DEV server

First get GitLab installed on DEV, as mentioned make sure it is the same version/type as you have installed in PROD. Get it up and running, don’t worry about configuring it too much because that’s all going to get overwritten, just make sure it’s up and working.

Backup the PROD database

gitlab-rake gitlab:backup:create SKIP=repositories,uploads

NOTE This didn’t work for me, I don’t know if it’s because we were using such an outdated version or what, but while the above command worked (it didn’t give an error, but also didn’t SKIP the things I asked it to), the real command for this step was:

gitlab-rake gitlab:backup:create
scp /var/opt/gitlab/backups/1511809717_gitlab_backup.tar svcansible@DEV:/tmp
scp -r /etc/ssh /etc/gitlab svcansible@DEV:/tmp
rsync -avz /var/git-data/repositories/ git@domain-dev.com:/var/git-data/repositories
mv /etc/ssh /etc/ssh-dist
cp -R ~/new/ssh ~/new/gitlab /etc
chgrp ssh_keys /etc/ssh/*_key
external_url "https://domain-dev.com/"
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.key"
cp /tmp/1511982248_gitlab_backup.tar /var/opt/gitlab/backups/
chown git:git /var/opt/gitlab/backups/1511982248_gitlab_backup.tar
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
# Verify those two processes are Down
gitlab-ctl status
gitlab-rake gitlab:backup:restore BACKUP=1511809717

at the end you’re asked:

This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
Do you want to continue (yes/no)? 

Answer with yes and let it rebuild the authorized_keys file for consistency sake

NOTE if you’re running 6.9.2 like I was, due to a bug in the code you first need to edit a file to allow systemctl to be used in place of initctl in the upstart.rb file

vi /opt/gitlab/embedded/cookbooks/runit/recipes/upstart.rb

replace any initctl with systemctl

gitlab-ctl reconfigure

if it completes sucessfully, start GitLab

gitlab-ctl start

Using RPM for further upgrades

Now we’re going to start upgrading the GitLab code to bring it up to the current release, 10.2.2, and will switch to use RPM/yum to pull directly from the packagemanager instead of pulling individual RPMS

yum install pygpgme yum-utils
[gitlab_gitlab-ce]
name=gitlab_gitlab-ce
baseurl=https://packages.gitlab.com/gitlab/gitlab-ce/el/7/$basearch
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey
       https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

[gitlab_gitlab-ce-source]
name=gitlab_gitlab-ce-source
baseurl=https://packages.gitlab.com/gitlab/gitlab-ce/el/7/SRPMS
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey
       https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
sudo yum -q makecache -y --disablerepo='*' --enablerepo='gitlab_gitlab-ce'

Upgrade GitLab to 7.x

NOTE old GitLab packages weren’t signed, so we need to skip that here)

yum install --nogpgcheck gitlab-ce-7.14.3-ce.0.el7.x86_64

NOTE output from the install script

# If you just upgraded from GitLab 7.9 or earlier, please run the following
gitlab: command:
gitlab:
gitlab: sudo ln -sf   /opt/gitlab/bin/gitlab-ctl   /opt/gitlab/bin/gitlab-rake   /opt/gitlab/bin/gitlab-rails   /opt/gitlab/bin/gitlab-ci-rake   /opt/gitlab/bin/gitlab-ci-rails  /usr/bin/
ln -sf   /opt/gitlab/bin/gitlab-ctl   /opt/gitlab/bin/gitlab-rake   /opt/gitlab/bin/gitlab-rails   /opt/gitlab/bin/gitlab-ci-rake   /opt/gitlab/bin/gitlab-ci-rails  /usr/bin/
gitlab-ctl reconfigure
gitlab-ctl restart

Upgrade GitLab to 8.x

NOTE again old GitLab packages weren’t signed, skip that for this one

yum install --nogpgcheck  gitlab-ce-8.17.7-ce.0.el7.x86_64

NOTE follow the output and update postgres data

gitlab: GitLab now ships with a newer version of PostgreSQL (9.6.1), and will be used
gitlab: as the default in the next major release. To upgrade, RUN THE FOLLOWING COMMANDS:

gitlab-ctl pg-upgrade
gitlab-ctl pg-upgrade

Upgrade GitLab to 9.x

yum install gitlab-ce-9.2.10-ce.0.el7.x86_64
gitlab-ctl reconfigure
gitlab-ctl restart

Upgrade to 10.x

Now we should be able to just install the latest by calling:

yum install gitlab-ce
### base
external_url "https://domain-dev.com/"

### web
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.key"

### ldap
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load 
main:
  label: 'LDAP'
  host: 'ldap.domain.com'
  port: 636
  uid: 'sAMAccountName'

  bind_dn: 'CN=Service GitLab,OU=ServiceAccounts,OU=Users,OU=NonManaged,DC=domain,DC=com'
  password: '***********'

  encryption: 'simple_tls'
  verify_certificates: false
  ca_file: ''
  ssl_version: ''

  timeout: 10
  active_directory: true
  allow_username_or_email_login: true
  block_auto_created_users: false
  base: 'DC=domain,DC=com'

  user_filter: ''
  attributes:
    username: ['uid', 'userid', 'sAMAccountName']
    email:    ['mail', 'email', 'userPrincipalName']
    name:       'cn'
    first_name: 'givenName'
    last_name:  'sn'
EOS
gitlab-ctl reconfigure
gitlab-ctl restart

Errata

Optional steps if you’re ready to go live

gitlab-ctl stop
rsync -avz /var/git-data/repositories/ git@domain-dev.com:/var/git-data/repositories
external_url "https://domain.com/"
gitlab-ctl reconfigure
gitlab-ctl start