Factory for multi profiles setups

Revision: $Id$


1   Introduction

Fully customized site configuration can get quite complicated, despite the great improvements that were brought by GenericSetup.

Indeed, some profiles have to be ran in a precise order, it may happen that a few configuration steps have to be handled in ZMI. Finally some parameters may change from an instance to the other (SMTP host, LDAP bind password) and have nothing to do in a shared version controlled XML file. We call these in this text external parameters

This factory is an automated solution for the three use cases above. Profiles to be imported are grouped in meta profiles. Each meta profile has pre and post hooks. Registered parameters appear on the site creation form. A method to replay profiles for upgrades is also provided. It takes care not to override external parameters.

Everything is done by the following class:

>>> from Products.CPSDefault.metafactory import CPSSiteMetaConfigurator
>>> conf = CPSSiteMetaConfigurator()

It will be convenient for examples to have a pretty printer:

>>> from pprint import PrettyPrinter
>>> pretty_print=PrettyPrinter(width=50).pprint

We'll use a fake setup tool, too:

>>> from Products.CPSDefault.metafactory import FakeSetupTool
>>> from Products.CPSDefault import factory as default_factory
>>> true_setup_tool = default_factory.CPSSetupTool
>>> default_factory.CPSSetupTool = FakeSetupTool

2   Meta profiles

A meta profile is made of a tuple of profiles to load, together with parameters that are exposed to the user at creation form stages and some additional helpers. They are kept in the meta_profiles attribute. The list of meta profiles to load is kept in meta_orders. Let's put a simple configuration there:

>>> conf.meta_profiles = {
...      'LDAP': {'title' : 'LDAP Members directory',
...               'extensions' : ('CPSLDAPSetup:default',),
...               'parameters' : {'properties' : ('ldap_bind_dn'),
...                             'class': 'Products.CPSDefault.metafactory.'
...                                      'SampleTool',
...                             'rpath': 'dtool/the_backing',
...                             },
...               'optional' : False,
...               'predelete_tool' : 'some_tool'
...              },
...      'Base': { 'extensions': ('SomeExt', 'SomeOther'), 'optional': True,
...                'title': "Base title"},
...     }

>>> conf.metas_order = ('Base', 'LDAP',)
>>> conf.form_heading = 'Add a Sample Site'
>>> conf.post_action = 'addConfiguredSampleSite'

In real life, subclassing is recommended, so that it's easy to ensure that re-runs are made with the same configuration as for a fresh site.

3   Form generation

The method responsible for the form rendering receives a bunch of keyword args. The underlying dict is altered by the overloaded prepareOptions() right before being passed. We'll demonstrate here what this method does.

Let's start with the basics: the method adds forwards two attributes of the configurator in the options dict, to be used as heading and 'action' attribute in the form element:

>>> options = {'otherinfo': 'hungry'}
>>> conf.prepareOptions(options)
>>> options['form_heading']
'Add a Sample Site'
>>> options['post_action']

The main addition is s a new meta_profiles key to the dict (other entries are kept untouched):

>>> options['otherinfo']
>>> 'meta_profiles' in options

The passed value is a list of dicts, one for each meta profile specified in self.metas_order, in the correct order, if the user has input to provide. Let's see what we get at a minimum:

>>> base_info = options['meta_profiles'][0]
>>> pretty_print(base_info)
{'id': 'Base',
 'optional': True,
 'title': 'Base title'}

Some metas get additional parameters as well. They can be used in a form and have to correspond to properties of the provided class. This way, we can reuse labels, typing and so on:

>>> ldap_info = options['meta_profiles'][1]
>>> ldap_info['id']
>>> pretty_print(ldap_info['parameters'])
[{'id': 'LDAP-ldap_bind_dn',
  'label': 'Sample tool bind',
  'mode': 'w',
  'type': 'string'}]

A non-existing property is simply dropped:

>>> params = conf.meta_profiles['LDAP']['parameters']
>>> save = params['properties']
>>> params['properties'] = ('  non ex',)
>>> conf.prepareOptions(options)
>>> len(options['meta_profiles'][1]['parameters'])

You can specify additional attributes that aren't managed as Zope 2 properties. In that case, the attribute id will also play the role of the label. There will be not type checking. In use cases, the attribute is handled via a Zope 3 schema, but this isn't leveraged yet:

>>> params['attributes'] = ['an_attr']
>>> conf.prepareOptions(options)
>>> pretty_print(options['meta_profiles'][1]['parameters'])
[{'id': 'LDAP-an_attr', 'label': 'an_attr'}]
>>> params['properties'] = save

A required meta profile that has no parameters won't be mentionned on the form:

>>> conf.meta_profiles['Base']['optional'] = False
>>> conf.prepareOptions(options)
>>> len(options['meta_profiles'])
>>> options['meta_profiles'][0]['id']

This does not happen if there are parameters, as we have actually already seen with LDAP

4   Meta Profile import internals

4.1   Preparations

Let's give the configurator a folder to work in:

>>> from Products.CMFCore.PortalFolder import PortalFolder
>>> conf.dispatcher = PortalFolder('app')

Now we build a bare site with a setup tool:

>>> conf.addSite('the_site')
>>> conf.site

System Message: ERROR/3 (./CPSDefault/doc/metafactory.txt, line 174)

Inconsistent literal block quoting.

<CPSDefaultSite at app/the_site> >>> conf.addSetupTool() >>> conf.site.portal_setup <FakeSetupTool at app/the_site/portal_setup>

4.2   Class Lookup

The classes referred to in meta_profiles are looked up like this:

>>> from Products.CPSDefault.metafactory import _resolveDottedName
>>> _resolveDottedName('Products.CPSDefault.metafactory.CPSSiteConfigurator')
<class 'Products.CPSDefault.factory.CPSSiteConfigurator'>

Note that if a class is directly accessible from a module placed
higher in the hierarchy, you must use this latter module to access
it. This comes from usage of ``__import__`` (the following example would
need python-ldap to be executed, that's why it isn't)::

>> _resolveDottedName('Products.CPSDirectory.LDAPBackingDirectory)
<class 'Products.CPSDirectory.CPSDirectory.LDAPBackingDirectory'>
>> _resolveDottedName(
... 'Products.CPSDirectory.LDAPBackingDirectory.LDAPBackingDirectory')
Traceback (most recent call last):
AttributeError: ... no attribute 'LDAPBackingDirectory'

But this doesn't fail::

>> from Products.CPSDirectory.LDAPBackingDirectory import LDAPBackingDirectory

4.3   Importation

_importMetaProfile just launches profiles in correct order:

>>> conf._importMetaProfile(conf.meta_profiles['Base'])
Imported 'profile-SomeExt'
Imported 'profile-SomeOther'

importMetaProfiles() is the main entry point in this factory's specifics. It reads requested metas from its kwargs, which eventually come from the form, launches pre and post import hooks and takes care of parameters:

>>> conf.metas_order = ('Base', 'Opt')
>>> conf.meta_profiles['Base']['extensions'] = ('SomeExt',)
>>> conf.meta_profiles['Base']['optional'] = False
>>> conf.meta_profiles['LDAP']['optional'] = True
>>> conf.meta_profiles['Opt'] = {'extensions' : ('OptExt1', 'OptExt2',),
...                              'optional' : True,}
>>> conf.importMetaProfiles()
Imported 'profile-SomeExt'
>>> conf.importMetaProfiles(requested_metas=['Opt'])
Imported 'profile-SomeExt'
Imported 'profile-OptExt1'
Imported 'profile-OptExt2'

The list of all imported meta_profiles is stored as a property of the portal_object (to be used by meta profiles aware updaters):

>>> conf.site.getProperty('meta_profiles')
('Base', 'Opt')

conf.metas_order prevails over ordering in the request:

>>> conf.meta_profiles['Base']['extensions'] = ('SomeExt',)
>>> conf.importMetaProfiles(requested_metas=['Opt', 'Base'])
Imported 'profile-SomeExt'
Imported 'profile-OptExt1'
Imported 'profile-OptExt2'

Non registered metas are safely ignored:

>>> conf.importMetaProfiles(requested_metas=['Base', 'Dream'])
Imported 'profile-SomeExt'

Let's see the hooks in action:

>>> def before(site, **kw):
...    print "Before: site %s\n  keys %s" % (site, kw.keys())
>>> def after(site, **kw):
...    print "After: site %s\n  keys %s" % (site, kw.keys())
>>> conf.meta_profiles['hooks'] = {'extensions': ('HookedExt',),
...                               'before_import': before,
...                               'parameters': {},
...                               'after_import': after,}
>>> conf.metas_order = ('hooks'),
>>> conf.importMetaProfiles(requested_metas=['hooks'])
Before: site <CPSDefaultSite at the_site>
  keys ['requested_metas']
Imported 'profile-HookedExt'
After: site <CPSDefaultSite at the_site>
  keys ['requested_metas']

4.4   Parameters

Since we dont' actually import extension profiles in this examples, we'll have to build an object to set properties on:

>>> conf.site._setObject('dtool', PortalFolder('dtool'))
>>> dtool = conf.site.dtool
>>> from Products.CPSDefault.metafactory import SampleTool
>>> dtool._setObject('the_backing', SampleTool('the_backing'))
>>> dtool.the_backing
<SampleTool at app/the_site/dtool/the_backing>

Now let's put back some parameters in the LDAP meta_profile. The meta_profile's id is used to prefix them so that to instances of the same class can be configured from the same form:

>>> params = {'LDAP:ldap_bind_dn': 'a_valid_dn'}
>>> conf.metas_order = ['LDAP']
>>> conf.importMetaProfiles(requested_metas=['LDAP'], **params)
Imported 'profile-CPSLDAPSetup:default'
>>> dtool.the_backing.ldap_bind_dn

An empty valued parameter is left unchanged. Therefore the corresponding property will have the same value as was set by profile, or at class level:

>>> params = {'LDAP:ldap_bind_dn': ''}
>>> conf.importMetaProfiles(requested_metas=['LDAP'], **params)
Imported 'profile-CPSLDAPSetup:default'
>>> dtool.the_backing.ldap_bind_dn

Finally, let's try the support of non-property attribute. Let's first instanciate an object that doesn't support properties and write a meta_profile that sets an attribute on it.

>>> from OFS.SimpleItem import SimpleItem
>>> conf.site._setObject('simple_item', SimpleItem())
>>> conf.meta_profiles['NonProp'] = {'title' : 'LDAP Members directory',
...            'extensions' : ('Dumb:default',),
...            'parameters' : {'attributes' : ('an_attr',),
...                             'class': 'OFS.SimpleItem.SimpleItem',
...                             'rpath' : 'simple_item',},
...            'optional' : False,
...            }
>>> conf.metas_order = ['NonProp']

Applying the meta profile, we get:

>>> conf.meta_profiles['NonProp']['parameters']['attributes'] = ['an_attr']
>>> params = {'NonProp-an_attr': 'some_value'}
>>> conf.importMetaProfiles(requested_metas=['NonProp'], **params)
Imported 'profile-Dumb:default'
>>> conf.site.simple_item.an_attr

5   Upgrades

After the initial site creation, the list of imported meta_profiles is kept as a property on the portal object. The configurator's replayMetaProfiles replays the importations, but keeps the associated external parameters as the user set them (be it directly in ZMI or through the configurator's form), whatever value is set in the profiles.

In order to demonstrate this, we had our fake setup tool reset one of its attributes at each import.

>>> site = conf.site
>>> site.portal_setup.witness = 1
>>> site.portal_setup.runAllImportStepsFromProfile('profile-example')
Imported 'profile-example'
>>> site.portal_setup.witness

Now let's make a fresh configurator. Since we'll need a second instance later with the same configuration, it's simpler to subclasss:

>>> class ReplayableConfigurator(CPSSiteMetaConfigurator):
...     meta_profiles = {
...      'ReplayedMeta': {'title' : 'Replayable & sets attr on portal_setup',
...                       'extensions' : ('SomeExt:default',),
...                       'parameters' : {
...                             'properties' : ('witness',),
...                             'class': 'Products.CPSDefault.metafactory.'
...                                      'FakeSetupTool',
...                             'rpath': 'portal_setup',
...                             'undisclosed' : ('witness',),
...                             },
...               'optional' : True,
...              },
...      'Other': { 'extensions': ('SomeOther'), 'optional': True,
...                'title': "Other title",
...                'parameters' : {'properties' : ('prop', 'password',),
...                                'class': 'Products.NoSuch.tool',
...                                'rpath': 'shadow_tool',
...                                'undisclosed' : ('password',),
...                                },
...                },
...      'NoParam': {'extensions': ('Nopar'), 'optional': False,
...                  'title' : ''},
...      'MissingTool': { 'extensions': ('SomeMissing',), 'optional': True,
...                'title': "Other title",
...                'parameters' : {'properties' : ('bind_dn',
...                                                'bind_password',),
...                                'class':  'Products.CPSDirectory.'
...                                          'LDAPServerAccess.'
...                                          'LDAPServerAccess',
...                                'rpath': 'ldap_sa',
...                                },
...                },
...               }
...     metas_order = ('Other', 'ReplayedMeta', 'MissingTool')
>>> conf = ReplayableConfigurator()
>>> conf.dispatcher = PortalFolder('app')
>>> conf.addSite('new_site')
>>> site = conf.site
>>> conf.addSetupTool()
>>> class FakeTool:
...    prop = ''
...    password = ''
>>> site.shadow_tool = FakeTool()

Now, let's launch the first import:

>>> conf.importMetaProfiles(requested_metas=['ReplayedMeta'],
...                         **{'ReplayedMeta-witness': 3})
Imported 'profile-SomeExt:default'
>>> site.portal_setup.witness

and let's do the replay as an external method would:

>>> rconf =  ReplayableConfigurator(site=site)
>>> rconf.replayMetaProfiles()
Imported 'profile-SomeExt:default'
>>> site.portal_setup.witness

The configurator can tell which parameters shouldn't be disclosed by the external method. For this, it uses the 'undisclosed' key in the 'params' dict and doesn't check anything:

>>> undis = rconf.getUndisclosedParams()
>>> undis.sort()
>>> undis
['Other-password', 'ReplayedMeta-witness']

Let's check consistency of undisclosed ids with the general list:

>>> set(undis).issubset(set(rconf.paramsSnapshot(['Other', 'ReplayedMeta'])))

It's also possible to replay only a few import steps. This is especially useful to avoid steps that require lengthy operations, like update of the catalog.

>>> rconf.replayMetaProfiles(steps=('layouts', 'schemas'))
Purge and import step profile 'profile-CPSDefault:default', step 'layouts'
Purge and import step profile 'profile-CPSDefault:default', step 'schemas'
Import step profile 'profile-SomeExt:default', step 'layouts'
Import step profile 'profile-SomeExt:default', step 'schemas'

5.1   External method

The site creation ends with the setting of an external method whose definition is controlled by the ``replay_external_method `` attribute:

System Message: WARNING/2 (./CPSDefault/doc/metafactory.txt, line 445); backlink

Inline literal start-string without end-string.
>>> rconf.replay_external_method['module']

System Message: ERROR/3 (./CPSDefault/doc/metafactory.txt, line 450)

Inconsistent literal block quoting.


You can override all the parameters in your configurator. If the attribute doesn't evaluate to True, no External Method will be made

5.2   Internals

The list of played metas and the configurator import path have been kept on the site object:: >>> site.meta_profiles ('ReplayedMeta',) >>> site.configurator '__builtin__.ReplayableConfigurator'

A snapshot of existing parameters is made:

>>> pretty_print(rconf.paramsSnapshot(['ReplayedMeta']))

System Message: ERROR/3 (./CPSDefault/doc/metafactory.txt, line 468)

Inconsistent literal block quoting.

{'ReplayedMeta-witness': 3}

Robustness tests:

>>> rconf.paramsSnapshot(['NoParam'])

System Message: ERROR/3 (./CPSDefault/doc/metafactory.txt, line 473)

Inconsistent literal block quoting.

{} >>> pretty_print(rconf.paramsSnapshot(['ReplayedMeta', 'NoParam'])) {'ReplayedMeta-witness': 3}

The replay does not break if the object bearing parameters is absent, as the list of meta profiles could have just been updated with a profile that creates precisely this object (ticket #2263):

>>> site.meta_profiles = ('ReplayedMeta', 'MissingTool')
>>> rconf.replayMetaProfiles()
Imported 'profile-SomeExt:default'
Imported 'profile-SomeMissing'
>>> site.meta_profiles = ('ReplayedMeta', ) # back to normal

6   Cleaning

We need to restore the default Setup Tool, otherwise layer tests won't run:

>>> default_factory.CPSSetupTool = true_setup_tool