HOWTO use meta profiles and switch an existing site to them

Author: Georges Racinet

Contents

1   What are meta profiles

Configuration of a CPS instance can get really complicated. Besides, in the typical instance, it is actually made of local changes or additions to the default configuration that ships with CPS standard products. This is a major source of complexity in long term maintainance, in which upgrading the default configuration is both required and desireable.

1.1   Profiles

With the introduction of GenericSetup based configuration in CPS 3.4, most of what can be done in ZMI is saved in so-called "profiles". These are hierarchies of XML files describing all the configuration objects that make up a CPS instance. They are organised in "steps", corresponding more or less to the various tools (e.g., `portal_layouts`) found in the instance.

There are powerful import/export capabilities that effectively allow to reproduce a configuration from one instance to the other.

Profiles can be replayed (reimported) and each step can also be exported, so that one may create new profiles or entire site snapshots.

1.2   Base and extension

There are two types of profiles: a 'base' profile is thought as an initial configuration, while an 'extension' profile is thought as an add-on.

CPSDefault comes with a base profile, that's loaded during of the site creation procedure. Most high level CPS products (e.g, CPSSubscriptions) come with an extension profile that's meant to be imported on top of CPSDefault's base profile. Real life sites have typically known the import or repeated import of a bunch of extension profiles.

Importing a base profile reinitializes all the tools and wipes their existing content, while importing an extension profile actually merges the configuration. This allows for instance to change just one property of a widget. In CPSCourrier, for instance, applying the `lucene` profile switches the widgets for all mailbox views to those special widgets that request a Lucene Catalog instead of the standard ZCatalog. In theory, that profile should change only that.

The potential for decorrelation brought by GenericSetup is great, but not infinite. It is easy to be in the situation where the order of import does matter. In the example above, you better import the `CPSCourrier:lucene` profile after `CPSCourrier:default`. In other words, profile imports are noncommutative operations.

1.3   Meta profiles

Although the flexibility of the profiles system is great, once you push it a bit, you come to a human limit if each update means reimporting a dozen profiles, in the right order, and if you want to be sure of the result, you better start again from the base profile, because in a noncommutative world, abab can be different from ab even though aa = a and bb = b.

Meta profiles are precisely meant to overcome that difficulty. A meta profile is nothing but an ordered set of profiles.

CPSDefault's metafactory is a factory for a CPS instance based on the meta profile concept. One declares several meta profiles, and the order between them. One can then automatically have a ZMI site creation page with each meta profile being an option for the administrator. The most interesting feature is that meta profiles are replayable in a single shot, and replaying them guarantees to reproduce the configuration exactly. Therefore you can work incrementally on your development instance, do the final testing after replaying the profiles and be pretty sure of the result on the production instance.

2   Declaring and using meta profiles

2.1   Creating a factory

TODO There is a rather complete example in CPSCourrier/factory.py

2.2   The site creation page

TODO check CPSCourrier/__init__.py to get the declarations

2.3   Replaying profiles

register an External method with module `CPSDefault.replay_meta_profiles` and function: `replay` Call it, and reindex the catalog, the portlets catalog and rebuild the tree caches.

A much better alternative is to use cpsjobs:

bin/zopectl run Products/CPSDefault/jobs/replaymetaprofiles.py -u  <user_id> <portal_id>
bin/zopectl run Products/CPSDefault/jobs/resync.py -u <user_id> --all

where user_id is the id of a user with Manager role on the portal. It can be either a toplevel Zope user or a CPS user.

Using a cpsjob is better if you have console access to the server with enough rights, because you don't need any prior registration, you don't need any password which also means that it's easily included in deployment scripts, and it's better suited for long runs.

cpsjobs must always be run on a shutdown ZEO client or monolithic instance.

2.4   Why resync and how to do it faster

Once the meta_profiles have been reloaded either by an external method or the first cpsjob above, it is really necessary to to reindex a few things, such as the catalog (used for all kind of searches, either user-driven or internal), the trees caches (used notably by navigation portlets) and the portlets catalog (used by the portlets lookup system).

This is usually very long, and can become really impractical, especially the catalog part.

In cases where you are sure there is no change in configuration of the catalog, you may use an exclude option on the replay job : excluding the catalog step allievates the need to resync the catalog:

bin/zopectl run Products/CPSDefault/jobs/replaymetaprofiles.py -u <user_id> <portal_id> --exclude=catalog
bin/zopectl run Products/CPSDefault/jobs/resync.py -u <user_id>  --portlets-catalog --trees

This is in most cases enough to make the length of the process acceptable.

Similarly, you can exclude the trees step and get rid of the trees resynchronization (if no change in the configuration of trees cache is expected) and even the portlets to get rid of portlets catalog reindexing if no change in profiles managed portlets have been made. Run the resync job with --help to get the full list of options.

3   Switching an existing instance to meta profiles

If starting from scratch, the normal process would be to declare the meta profiles, use them to create the site, and let the system handle the upgrades automatically.

Now, some older production instances would benefit from the system, but it's of course unacceptable to recreate them. Here we explain how to alter an existing instance so that meta profile replaying works. You can think of the process as tricking the portal into believe it has been created by the meta profile based creation page.

3.1   Create the factory

This is exactly the same as for a new site.

You should create the factory class as above and check that it works, i.e., that you can use it to make a new, fresh portal with it.

3.2   Mark the portal with the factory class

Go to a fresh portal created with the new factory class. Look, in the introspection tab to the `configurator` attribute. Create a new property on the portal you want to switch to meta_profiles with the same value (without the quotes of course).

3.3   Store the list of loaded meta profiles

Similar, but easier: look for the `meta_profiles` property on the fresh portal, and reproduce it in the one to upgrade.

3.4   Check the outcome and correct

Now you can replay the meta profiles ! Check that your configuration hasn't changed by performing snaphots before and after, and using the diff facility of `portal_setup`. Most probably, you'll have to alter your most custom profile to achieve the wished result. Check the next section for big divergences.

4   Fixing big configuration divergences

In this part we assume that your portal is managed through meta profiles, but that its configuration has diverged from what's in the meta profiles, and you want to resync. This is a cumbersome job, but it can be done.

For the sake of example and clarity, we assume also that the meta profile is made of one profile in a custom product, on top of a bunch of CPS standard profiles.

In case the goal is an upgrade of CPS generic products, we strongly advice to resynchronize before applying the upgrade (see the dedicated section of this document for the upgrade itself).

In short, it's all about comparing what the meta profile does with the reality, in a wash/rinse/repeat sort of way.

4.1   Preparations

  1. Work on a development instance. If you don't have one, it's about time to make one.
  2. Make an export of the configuration, using the "export all steps" button that's on the "Export steps" tab of portal_setup in ZMI. You'll get a tarball.
  3. Unpack the tarball in a directory called saved_export and apply the profiles_remove_irrelevant.sh script. (TODO provide this script). Indeed, the profiles export has a few but repeated source of irrelevant changes, such as creation dates for portlets.

4.2   Incremental adjusting

  1. Replay the meta profiles on the development instance.
  2. Export all steps, unpack in a directory called current_export, and apply the cleaning script as above in there.
  3. Compare the two exports, by issuing diff -ruw saved_export current_export.
  4. Report the differences in you custom profile. You may decide to discard some, just remember that it'll keep showing in diffs. Sometimes you'll have to create the XML file in which the difference sit in the custom profile. In that case, use the export as a guideline (name/path of the file). Keep in mind that you should not report all properties of XML elements, but rather the minimal path in the XML tree to the value that is different or new. TODO give an example with a widget property.
  5. Each time you have to interrupt, save your work (commit if you use a VCS) and resume at step 1.

In case there are several custom products/profiles, you of course have to decide how to dispatch the changes between them, according to the separation logic of these profiles.

5   Upgrading generic products in meta profiles situation