We're currently running OpenStack in production (well, almost) on the Icehouse release. We used to build our own packages but currently switching to RDO and get rid of the legacy OVS neutron plugin. So I tried to pre-perform the whole upgrade thing on packstack for testing in advance, just in case.

Here are some notes on how to database migrations did or did not work for me.

I always experience problems with MySQL and UTF8 on every upgrade. So be sure to convert all relevant tables to UTF8. You'll get foreign key errors (errno 150) if you don't do. So I run the following script prior to every upgrade:

#!/usr/bin/env python

import MySQLdb

skip_databases = ['mysql', 'information_schema', 'performance_schema',]
conn = MySQLdb.connect(read_default_file="~/.my.cnf")

c = conn.cursor()
dbs = c.execute("SHOW DATABASES")
databases = c.fetchall()

for database in databases:
    database = database[0]
    if database not in skip_databases:
        r = c.execute("SHOW TABLES")
        tables = c.fetchall()
        print "Working on %s" % database
        for i in tables:
            statement = "ALTER TABLE `%s` CONVERT TO CHARACTER SET 'utf8'" % i[0]
            print "Executing " + statement

Keystone, Glance, Cinder

Nothing noticible yet.


The migration itself worked. But i rather tend to say the whole nova database schema is broken. I can't archive deleted instances as i get an foreign key constraint error. I will have a closer look on that.


There is a module which converts all OVS stuff to ML2. As mentioned we used the OVS plugin since the beginning and did not get it working on Icehouse.

$ python -m neutron.db.migration.migrate_to_ml2 openvswitch --tunnel-type=gre --release=icehouse --save-tables ${mysql_connection}

Now for the upgrade itself:

$ neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade kilo
   Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 238, in main
    CONF.command.func(config, CONF.command.name)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 106, in do_upgrade
    do_alembic_command(config, cmd, revision, sql=CONF.command.sql)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 72, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/alembic/command.py", line 165, in upgrade
  File "/usr/lib/python2.7/site-packages/alembic/script.py", line 382, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/site-packages/alembic/util.py", line 242, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/site-packages/alembic/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 109, in <module>
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 100, in run_migrations_online
  File "<string>", line 7, in run_migrations
  File "/usr/lib/python2.7/site-packages/alembic/environment.py", line 742, in run_migrations
  File "/usr/lib/python2.7/site-packages/alembic/migration.py", line 305, in run_migrations
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/versions/2b801560a332_remove_hypervneutronplugin_tables.py", line 132, in upgrade
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/versions/2b801560a332_remove_hypervneutronplugin_tables.py", line 124, in _migrate_port_bindings
    op.execute(ml2_port_bindings.insert(), ml2_bindings)
  File "<string>", line 7, in execute
  File "/usr/lib/python2.7/site-packages/alembic/operations.py", line 1270, in execute
  File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 108, in execute
    self._exec(sql, execution_options)
  File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 104, in _exec
    conn = conn.execution_options(**execution_options)
TypeError: execution_options() argument after ** must be a mapping, not list

Whoops, that did go wrong. Ok, that's just HyperV-stuff that does something HyperV-specific magic which we don't use and deletes the obsolete tables. I'll will do some deeper investigation on that. It pretty much looks like that a specific release did something nasty (we migrated from Folsom and Grizzly packaged from Ubuntu to selfpackaged Icehouse skipping Havana using RDO/Packstack for tests). So let's skip that for now.

$ neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp 2b801560a332

That stamps the database to the revision that went wrong and we can continue with the migration. Hooray! neutron-db-manage current says kilo.