Upgrade steps

Contents

1   Introduction

Upgrade steps are the standard way within CPS to manage upgrade of data, such as documents, workflow history, user information if not from an outside source, in short everything's that stored within a portal.

The upgrade steps API is divided in two parts: non-persistent (category and step registrations), and persistent, keeping notably track of applied steps, current level of upgrade for a given CPS portal. The persistent part is provided by CPS' Setup Tool.

This document is a doctest, meaning that the examples are actually checked as part of the test suite. This can make for a special writing style.

It will be convenient for examples to have a pretty printer:

>>> from pprint import PrettyPrinter
>>> pretty_print=PrettyPrinter(width=50).pprint

2   Categories registration

Upgrade steps are divided in categories. These are pure markers that morally correspond to sets of products (base distributions, groups of extensions, custom projects, etc.). The CPSCore product has already registered some categories, notably the main one (cpsplatform):

>>> from Products.CPSCore.upgrade import _categories_registry
>>> pretty_print(_categories_registry['cpsplatform'])
{'description': '',
 'floor_version': '3.2.0',
 'is_applicable': None,
 'portal_attr': 'last_upgraded_version',
 'title': 'CPS Platform'}

Let's add one as if we were managing a side component:

>>> from Products.CPSCore.upgrade import registerUpgradeCategory
>>> registerUpgradeCategory('my_app',
...                 title='My Application',
...                 floor_version='0.5',
...                 description='My very cool app on top of CPS',
...                 portal_attribute='my_app_up_version')
>>> pretty_print(_categories_registry['my_app'])
{'description': 'My very cool app on top of CPS',
 'floor_version': '0.5',
 'is_applicable': None,
 'portal_attr': 'my_app_up_version',
 'title': 'My Application'}

We can also specify a reference product for the upgrade category by means of the ref_product keyword argument. The current code version will be stored as well.

We need to cheat a bit to avoid the example to rely on CPSCore's version at the time of this writing:

>>> import os
>>> from Products.CPSCore import upgrade
>>> upgrade.VERSION_FILE = os.path.join('tests', 'data', 'VERSION.sample')

Now let's go:

>>> registerUpgradeCategory('my_app2',
...                 title='My Application',
...                 floor_version='0.1',
...                 ref_product='CPSCore',
...                 description='My very cool app on top of CPS',
...                 portal_attribute='my_app2_up_version')
>>> pretty_print(_categories_registry['my_app2'])
{'code_version': '19.0.0',
 'description': 'My very cool app on top of CPS',
 'floor_version': '0.1',
 'is_applicable': None,
 'portal_attr': 'my_app2_up_version',
 'ref_product': 'CPSCore',
 'title': 'My Application'}
>>> del _categories_registry['my_app2']

code_version can be used, e.g., by site factories to set the database version at site creation time. It is normally read from the VERSION file that should sit at the topmost level of the reference product. Without the above monkey patching, we would have gotten CPSCore's version.

3   Upgrade steps registration

3.1   In python

We need here to provide a configuration context for the Zope Component Architecture:

>>> class SimplestConfigurationContext:
...     def action(self, discriminator, callable=None, args=()):
...         """Do it now!"""
...         if callable is not None:
...             callable(*args)

>>> context = SimplestConfigurationContext()
>>> from Products.CPSCore.upgrade import upgradeStep, _upgrade_registry

We have a few predefined sample handlers in the context of this doctest. Of course a real-life handler should not directly print to the standard output. These ones do nothing else:

>>> portal = None
>>> up_my_app_1_0_1_1(portal)
my_app: 1.0 -> 1.1
>>> up_my_app_1_1_1_2(portal)
my_app: 1.1 -> 1.2
>>> up_my_app_1_1_1_2first(portal)
my_app: 1.1 -> 1.2 (first)
>>> up_my_app_1_2_2_0(portal)
my_app: 1.2 -> 2.0
>>> up_my_app_2_0_3_0(portal)
my_app: 2.0 -> 3.0
>>> up_noop(portal)

So, let's register a step for 'My Application':

>>> upgradeStep(context, 'Test Step 1', up_my_app_1_0_1_1,
...             category='my_app',
...             source='1.0',
...             destination='1.1', sortkey=10)

Now, let's see if we can find it in the list of proposed steps for 'My Application':

>>> from Products.CPSCore.upgrade import listUpgradeSteps
>>> listed = listUpgradeSteps(portal, 'my_app', (1, 0))
>>> len(listed)
1
>>> listed[0]['title']
'Test Step 1'

We'll keep this step object for further examples:

>>> my_step = listed[0]['step']

One can disable the step by handler lookup:

>>> from Products.CPSCore.upgrade import disableUpgradeSteps
>>> my_step.isProposed(portal, 'my_app', (1,0))
True
>>> disableUpgradeSteps(up_my_app_1_0_1_1)
>>> my_step.isProposed(portal, 'my_app', (1,0))
False

This is actually implemented by changing the checker to a uniform False:

>>> my_step.checker(portal)
False

There's also an API to change the checker for an upgrade step:

>>> from Products.CPSCore.upgrade import updateStepsChecker
>>> def new_checker(portal):
...     return 'this is the new checker'

>>> updateStepsChecker(up_my_app_1_0_1_1, new_checker)
>>> my_step.checker(portal)
'this is the new checker'

One can change a step's checker conditionnally, according to the step's version:

>>> my_step.version
1
>>> updateStepsChecker(up_my_app_1_0_1_1, new_checker,
...                    max_version=1)
>>> my_step.checker(portal)
'this is the new checker'
>>> updateStepsChecker(up_my_app_1_0_1_1, None)
>>> my_step.checker is None
True
>>> my_step.version = 2
>>> updateStepsChecker(up_my_app_1_0_1_1, new_checker,
...                    max_version=1)
>>> my_step.checker is None
True
>>> updateStepsChecker(up_my_app_1_0_1_1, new_checker,
...                    min_version=1)
>>> my_step.checker(portal)
'this is the new checker'
>>> updateStepsChecker(up_my_app_1_0_1_1, None,
...                    min_version=1, max_version=2)
>>> my_step.checker is None
True
>>> updateStepsChecker(up_my_app_1_0_1_1, new_checker,
...                    min_version=3, max_version=5)
>>> my_step.checker is None
True

Getting back to absence of checker for further examples, no matter what::
>>> updateStepsChecker(up_my_app_1_0_1_1, None)

3.2   In ZCML

There's a dedicated XML namespace for CPS ZCML directives::
<configure xmlns:cps="http://namespaces.nuxeo.org/cps">

Use the <cps:upgradeStep> element. Mandatory attributes 'title' and 'handlers' corrspond to positional args of the upgradeStep function (except `context`, which is for ZCML handling internals), while others correspond to kwargs (see also various examples within CPS configuration)

3.3   Listing available steps

Let's register a few more steps:

>>> upgradeStep(context, 'Test Step 3', up_my_app_1_1_1_2,
...             category='my_app',
...             source='1.1',
...             destination='1.2', sortkey=10)

>>> upgradeStep(context, 'Test Step 2', up_my_app_1_1_1_2first,
...             category='my_app',
...             source='1.1',
...             destination='1.2', sortkey=6)

>>> upgradeStep(context, 'Test Step 4', up_my_app_1_2_2_0,
...             category='my_app',
...             source='1.2',
...             destination='2.0', sortkey=10,
...             requires='cpsplatform-3.4.5')


>>> steps = listUpgradeSteps(None, 'my_app', (1, 0))

Let's remove some human non friendly data and display the ordered list of proposed steps:

>>> for step in steps:
...     del step['id']
...     del step['step']
>>> pretty_print(steps)
[{'dest': (1, 1),
  'proposed': True,
  'source': (1, 0),
  'title': 'Test Step 1'},
 {'dest': (1, 2),
  'proposed': True,
  'source': (1, 1),
  'title': 'Test Step 2'},
 {'dest': (1, 2),
  'proposed': True,
  'source': (1, 1),
  'title': 'Test Step 3'},
 {'dest': (2, 0),
  'proposed': True,
  'requires': ('cpsplatform', (3, 4, 5)),
  'source': (1, 2),
  'title': 'Test Step 4'}]

It's also possible to bound the destination version from above (see #2250):

>>> steps = listUpgradeSteps(None, 'my_app', (1, 0), max_dest=(1, 2))
>>> [step['title'] for step in steps]
['Test Step 1', 'Test Step 2', 'Test Step 3']

Listing by handlers can be useful for scripts/jobs. This also demonstrates that no signature check is done on the handler (the one we use is really stupid):

>>> upgradeStep(context, 'Test by handler', upgradeStep,
...             category='my_app',
...             source='2.0',
...             destination='2.5', sortkey=10,
...             requires='cpsplatform-3.5.2')
>>> from Products.CPSCore.upgrade import listUpgradesByHandler
>>> [s.title
...  for s in listUpgradesByHandler('Products.CPSCore.upgrade.upgradeStep')]
['Test by handler']

This works also by passing the handler callable itself:

>>> [s.title for s in listUpgradesByHandler(up_my_app_1_2_2_0)]
['Test Step 4']

4   With persistence : Setup Tool API

Let's instantiate the setup tool and prepare some context:

>>> class FakePortal:
...     pass
>>> portal = FakePortal()
>>> class FakeUrlTool:
...     def getPortalObject(self):
...         return portal
>>> portal.portal_url = FakePortal()
>>> from Products.CPSCore.setuptool import CPSSetupTool
>>> tool = CPSSetupTool()
>>> tool.portal_url = FakeUrlTool()

The version is kept as a property on the portal object. If missing, the category's floor version is used:

>>> tool._getCurrentVersion('my_app')
(0, 5)

The tool provides an API to update the version number.
>>> tool._setCurrentVersion('my_app', (1, 0))
'1.0'
>>> tool._getCurrentVersion('my_app')
(1, 0)

>>> pretty_print(tool._getUpgradeCategoryDisplayInfo('my_app'))
{'description': 'My very cool app on top of CPS',
 'id': 'my_app',
 'title': 'My Application',
 'version': '1.0'}

Views have to use the following method:

>>> listed = tool.listUpgradeCategories()
>>> pretty_print([l for l in listed  if l['id'] == 'my_app'])
[{'description': 'My very cool app on top of CPS',
  'id': 'my_app',
  'title': 'My Application',
  'version': '1.0'}]

The listing of categories is affected by the 'is_applicable' checker. In this example, we only get the cpsplatform category:

>>> _categories_registry['my_app']['is_applicable'] = lambda x: False
>>> 'my_app' in [c['id'] for c in tool.listUpgradeCategories()]
False

But the applicability filtering usually really involves the portal:

>>> def is_my_app(portal):
...    return getattr(portal, 'is_my_app', False)
>>> _categories_registry['my_app']['is_applicable'] = is_my_app
>>> 'my_app' in [c['id'] for c in tool.listUpgradeCategories()]
False
>>> portal.is_my_app = True
>>> 'my_app' in [c['id'] for c in tool.listUpgradeCategories()]
True

Actual applicability checkers may rely on interfaces, meta_types, the import of a given profile having set a property, the presence of a tool. In any case for further examples, let's get back to the absence of check:

>>> _categories_registry['my_app']['is_applicable'] = None

4.1   Upgrades proposals

Let's ask the tool for available upgrades:

>>> list_ups = tool.listUpgrades(category='my_app')
>>> [up['title'] for up in list_ups]
['Test Step 1', 'Test Step 2', 'Test Step 3']

Our test step 4 was not proposed because it requires (the upgrade of) CPSPlatform to 3.4.5. Let's see what info is provided for each step:

>>> keys = list(list_ups[0]); keys.sort(); pretty_print(keys)
['dest',
 'done',
 'haspath',
 'id',
 'proposed',
 'sdest',
 'source',
 'ssource',
 'step',
 'title']

If the applicable checker returns False, then the list is empty.

>>> _categories_registry['my_app']['is_applicable'] = lambda x: False
>>> tool.listUpgrades(category='my_app')
[]
>>> _categories_registry['my_app']['is_applicable'] = None

Now, let's do as if the portal were upgraded to 1.1: step 1 shouldn't be presented anymore:

>>> tool._setCurrentVersion('my_app', (1, 1))
'1.1'
>>> list_ups = tool.listUpgrades(category='my_app')
>>> [up['title'] for up in list_ups]
['Test Step 2', 'Test Step 3']
>>> tool._setCurrentVersion('my_app', (1, 2))
'1.2'
>>> tool.listUpgrades(category='my_app')
[]

Now we pretend that CPSPlatform has been upgraded to the required version for step 4:

>>> tool._setCurrentVersion('cpsplatform', (3, 4, 5))
'3.4.5'
>>> list_ups = tool.listUpgrades(category='my_app')
>>> [up['title'] for up in list_ups]
['Test Step 4']

The result would be the same for a later version:

>>> tool._setCurrentVersion('cpsplatform', (3, 5))
'3.5'
>>> list_ups = tool.listUpgrades(category='my_app')
>>> [up['title'] for up in list_ups]
['Test Step 4']

Dependency requirement is implicit for all steps after one that has some:

>>> upgradeStep(context, 'Test Step 5', up_my_app_2_0_3_0,
...             category='my_app',
...             source='2.0',
...             destination='3.0')
>>> tool._setCurrentVersion('cpsplatform', (3, 4, 4))
'3.4.4'
>>> tool.listUpgrades(category='my_app')
[]

4.2   Launching the upgrades

One has to call the listing API to retrieve ids first to feed them to the general launcher:

>>> tool._setCurrentVersion('my_app', (1, 1))
'1.1'
>>> list_ups = tool.listUpgrades(category='my_app')
>>> ids = [up['id'] for up in list_ups]
>>> tool.doUpgrades(ids, 'my_app', do_commit=False)
my_app: 1.1 -> 1.2 (first)
my_app: 1.1 -> 1.2

The stored version was updated:

>>> tool._getCurrentVersion('my_app')
(1, 2)

The tool keeps track of applied steps and filters them, now:
>>> tool._getAppliedStepsIds('my_app') == tuple(ids)
True
>>> tool.listUpgrades(category='my_app')
[]

Another scenario where one applies only the first of those upgrades:

>>> tool._resetAppliedSteps()
>>> tool._setCurrentVersion('my_app', (1, 1))
'1.1'
>>> list_ups = tool.listUpgrades(category='my_app')
>>> ids = [list_ups[1]['id']]
>>> tool.doUpgrades(ids, 'my_app', do_commit=False)
my_app: 1.1 -> 1.2

In that case, the version has not been updated, because there remains a step to be done:

>>> tool._getCurrentVersion('my_app')
(1, 1)

and the remaining step is still in the list:

>>> tool.listUpgrades(category='my_app') == list_ups[:1]
True

If a step version changes (with due registration again), it's not considered as done anymore:

>>> upgradeStep(context, 'Test Step 3', up_my_app_1_1_1_2,
...             category='my_app',
...             source='1.1',
...             destination='1.2', sortkey=10,
...             version=2)
>>> list_ups = tool.listUpgrades(category='my_app')
>>> len(list_ups)
2
>>> [info['title'] for info in list_ups]
['Test Step 2', 'Test Step 3']
>>> step_info = list_ups[1]
>>> step_info['done']
False
>>> step_info['step'].version
2

There's a special rule for the cpsplatform category. Development or release candidate versions are never marked as reached. Let's see that, but first let's pretend the code is at CPS 3.3.17-devel (we're using versions that will never exist intentionnaly to avoid mixing up with real upgrade steps):

>>> from Products.CPSCore.portal import CPSSite
>>> saved = CPSSite.cps_version, CPSSite.cps_version_suffix
>>> CPSSite.cps_version = ('CPS', 3, 3, 17)
>>> CPSSite.cps_version_suffix = 'devel'
>>> tool._setCurrentVersion('cpsplatform', (3, 3, 15))
'3.3.15'

Let's register upgrade steps to CPS 3.3.16 and 3.3.17:

>>> upgradeStep(context, 'Test Platform 3.3.16', up_noop,
...             source='3.3.15', destination='3.3.16')
>>> upgradeStep(context, 'Test Platform 3.3.17', up_noop,
...             source='3.3.16', destination='3.3.17')
>>> list_ups = tool.listUpgrades();
>>> ids = [up['id'] for up in list_ups
...       if up['sdest'] in ('3.3.16', '3.3.17')]
>>> len(ids)
2
>>> tool.doUpgrades(ids, 'cpsplatform', do_commit=False)
>>> tool._getCurrentVersion('cpsplatform')
(3, 3, 16)

Same with the suffix for release candidates

>>> tool._resetAppliedSteps()
>>> tool._setCurrentVersion('cpsplatform', (3, 3, 15))
'3.3.15'
>>> CPSSite.cps_version_suffix = 'rc'
>>> tool.doUpgrades(ids, 'cpsplatform', do_commit=False)
>>> tool._getCurrentVersion('cpsplatform')
(3, 3, 16)

Now without any version suffix:

>>> tool._resetAppliedSteps()
>>> tool._setCurrentVersion('cpsplatform', (3, 3, 15))
'3.3.15'
>>> CPSSite.cps_version_suffix = ''
>>> tool.doUpgrades(ids, 'cpsplatform', do_commit=False)
>>> tool._getCurrentVersion('cpsplatform')
(3, 3, 17)

Finally, we restore the current code version, in order to avoid side effects in other tests:

>>> CPSSite.cps_version, CPSSite.cps_version_suffix = saved