$Home|Elec|Tech|Misc|Pics

Plone v3.2.2 upgrade

20 August 2016
Although my University days (as are my days living in the UK) are now distant past, I still run BristolShooting which is a Plone-based site I still host for Bristol's clay shooting club. I eventually lost patience with the VPS it was hosted on, so decided to upgrade & migrate it to a new host. The original plan was to upgrade to Plone 4, but as is detailed below I gave up on this as it was throwing up a lot of complications. It was someone else who designed & skinned the Plone site in the first place, and in the end it looked like conversion would not be far off a complete rebuild from scratch.

This is a write-up I drafted back in 2014, but for various reasons deferred publication. Apart from some link fixups it is minimally updated...

Prerequisites

Looking at timestamps of various files the installation was built early-2010, but I suspect this was an upgrade of a Plone instance built early 2008. Plone v3.2.2 is one of the earlier buildout based versions which allows automated upgrades, but I was getting errors like the ones below trying to do anything with my backed-up Plone instance, which I suspected was down to bit-rot:
../Python-2.4/bin/python: bad interpreter: No such file or directory
Plone hard-wires bang-paths to use the built-in copy of Python, which may well be incorrect when you copy the Plone installation en-masse to a new system. Workaround is to edit them manually, but this is a pain.
Download error: unknown url type: https -- Some packages may not be found!
In my case this occurred alongside “getting distribution for 'Cheetah>1.0'.” and is caused because SSL was not built into Python during the original installation. This seems to be a flaw in the Plone v3.x.x packaging of Python (v2.4) rather than the absence of SSL development files on the build system.
As a starting point I will assume that within the directory $BASE there are the following subdirectories & files, and for all shell snippets I will assume $BASE is the initial current directory:
Plone-3.2.2
The Plone installation copied across from the old server.
Plone-3.3.6-UnifiedInstaller
The v3.3.6 Unified Installer, downloaded and unpacked.
Plone-4.3.3-UnifiedInstaller
The v4.3.3 Unified Installer, downloaded and unpacked.
The following Plone hotfix zipfiles:
You will not actually need all of the above files, but this keeps all the download pointers in one place. Whether the upgrade is to v.3.3.4 or v4.3.3 is really dependent on the ulterior motive of just patching up an old site, or whether there's intention to actively add extra components to it.

From Plone v3.2.2 to Plone v3.3.6

As far as I was aware only bundled plugins were used, so the approach I took was to graft the database into a fresh v3.3.6 install:

cd Plone-3.3.6-UnifiedInstaller ./install.sh standalone -target=$BASE/Plone-3.3.6 cd ../Plone-3.3.6/zinstance cp ../../Plone-3.2.2/zinstance/var/filestorage/Data.fs var/filestorage/ ./bin/buildout ./bin/plonectl start

If you point a web-browser at http://127.0.0.1:8080/Plone you should see the Plone landing page (or your Plone site, depending on your old setup). It might take a while to load first time, even on a local system. There have been several security advisories for Plone v3.3.6, but only five of them include hotfixes. Apply them using the following:

cd Plone-3.3.6/zinstance ./bin/plonectl stop cd products unzip ../../../plonehotfix20130618-1-3.zip unzip ../../../PloneHotfix20131210.zip unzip ../../../Products.PloneHotfix20150910-1.0.zip unzip ../../../PloneHotfix20151208-1.0.zip unzip ../../../PloneHotfix20160419-1.0.zip cd .. ./bin/instance fg

It may take a while, but in the output look for something similar to the following:

2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied catalog patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied cb_decode patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied dataitems patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied get patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied imports patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied in_portal patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied linkintegrity patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied mail_password patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied member_portrait patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied objectmanager patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied principiaredirect patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied pts patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied publish patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied request patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied sendto patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied spamProtect patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied traverser patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied traverseName patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied typeswidget patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied wysiwyg patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Applied zip patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20130618 Hotfix installed 2014-05-24 18:42:10 INFO Products.PloneHotfix20131210 Applied browser_id_manager patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20131210 Applied catalog patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20131210 Applied factory patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20131210 Applied image patch 2014-05-24 18:42:10 INFO Products.PloneHotfix20131210 Hotfix installed

If these appear, you can Ctrl-C this foreground Plone instance and restart the background one using plonectl.

Forgotten admin password

It had been at least 4 years since I personally logged into (as opposed to visited) any of the Plone sites in the zinstance, so I had long forgotten what the passwords were. The solution is to add a new admin user via the command-line:

cd Plone-3.3.6/zinstance ./bin/plonectl stop ./bin/instance adduser root s3kret ./bin/plonectl start

This fails silently of the user already exists, so it has to be a new username. Browse to http://127.0.0.1:8080/acl_users/users/manage_users to reset existing Plone users and delete the temporary admin user.

Forgotten portal passwords

Password reset under Users and Groups within site portals does a mailout, which will fail if like me you tend not to install mail delivery software on your server. To change these passwords, do the following: I am presuming you are using an admin password reset previously to get here. I have never quite worked out how Plone's not-quite-segregated user management works, as portals seem to have their own user databases, but they are not quite entirely independent from each other and the central user database.

Database packing

Much of the content of my Plone portals was pretty dated, so I decided it was not worth keeping around old revisions of pages. Packing the database file can be done as follows: The existing database is backed up to Data.fs.old, which in my case I simply deleted. On the grand scale of things I think packing is only really worthwhile if a lot of image files have been deleted.

Lighttpd to Nginx

I have compared Lighttpd & Nginx in the past, although from last year's rebuild of my main web-server, my lasting impression of Lighttpd is that it is a one-hit-wonder optimised for single-domain sites that use CGI/FastCGI. In itself this isn't a bad thing, but ever since I stopped using Apache I have been standardising on Nginx for all my systems. The Lighttpd snippit I used on my old server is shown below, which forwards all requests to Plone using the magic prefix/suffix that Plone's Virtual Host Monster uses. A redirect is also used if the www prefix is missing, as Plone uses absolute URLs in its generated HTML.

$HTTP["host"] == "www.bristolshooting.org" { url.rewrite-once = ( "^/(.*)$" => "/VirtualHostBase/http/www.bristolshooting.org:80/shooting/VirtualHostRoot/$1" ) proxy.server = ( "/VirtualHostBase/" => ( ( "host" => "127.0.0.1" , "port" => 8080 ) ) ) } $HTTP["host"] == "bristolshooting.org" { url.redirect = ( "^/(.*)" => "http://www.bristolshooting.org/$1" ) }

Although sample snippits tend to use the upstream directive, I see little value in this for single-server use-cases. Using proxy_pass in a location subsection is all that really needs to be done for Nginx:

server { server_name www.bristolshooting.org; location / { proxy_pass http://127.0.0.1:8080/VirtualHostBase/http/www.bristolshooting.org:80/shooting/VirtualHostRoot/; } } server { server_name bristolshooting.org; return 301 $scheme://www.bristolshooting.org$request_uri; }

The $scheme keyword evaluates to ‘http’ or ‘https’ depending on whether SSL is being used.

Plone init.d startup script

I used the longer of the two init.d scripts given on the Plone site to startup Plone on server reboot. The only difference is that I used update-rc.d to setup all the S20plone/K20plone scripts that Ubuntu 14.04 insists on scattering everythere, as insserv didn't seem to be installed:

update-rc.d plone defaults

I've never been particular fond of the SysV-style startup scripts used by most Linux distributions, and my preference for BSD-style ones is a major reason I've historically used Slackware.

From Plone v3.3.6 to Plone v4.3.3

This is much the same as upgrading to v3.3.6, namely installing from the Universal Installer and copying across the datafiles from v3.3.6:

cd Plone-4.3.3-UnifiedInstaller ./install.sh standalone -target=$BASE/Plone-4.3.3 cd ../Plone-4.3.3/zinstance cp ../../Plone-3.3.6/zinstance/var/filestorage/Data.fs var/filestorage/ ./bin/instance fg

However once this has been done you need to point your web browser at http://127.0.0.1:8080 which should list all the Plone sites within the Plone installation, even if you previously set the front landing page to empty. It will tell you which ones need upgrading, as well as providing a helpful Upgrade.. button. Click away and before long everything should be running under Plone version 4. Unfortunately any customised templates are probably going to be completely hosed, as Plone 4 changed various template directives.

Plone v4 has changed a lot compared to v3, so a v3 to v4 upgrade is likely going to involve at least some manual steps. In particular it seems that styling & theming in Plone v4 is gravitating towards using Diazo in a segregated plain-to-themed transformation pass in favour of the previous template system, that latter of which which in addition to changing looks like it is also heading for deprecation. In the longer-run this looks like a good thing, but it also means that any robust v3 to v4 upgrade is going to be at least a partial rebuild.

AttributeError 'Title' in folder_listing

If folder_listing.pl is throwing an AttributeError with value 'Title' regarding Expression: <PathExpr standard:u'item_icon/html_tag'> (Line 80, Column 28) this is probably caused by stale database entries that newer versions of Plone are less tolerant of. Rebuild it using the following:

NotFound in unrestrictedTraverse errors

Plone v4 seems to have changed how images are stored, so it is possible that during upgrade, you get the following failure:

2014-05-24 19:41:31 ERROR plone.app.upgrade Upgrade aborted. Error: Traceback (most recent call last): File "/home/remy/Plone-4.3.3/buildout-cache/eggs/Products.CMFPlone-4.3.3-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 179, in upgrade step['step'].doStep(setup) File "/home/remy/Plone-4.3.3/buildout-cache/eggs/Products.GenericSetup-1.7.4-py2.7.egg/Products/GenericSetup/upgrade.py", line 140, in doStep self.handler(tool) File "/home/remy/Plone-4.3.3/buildout-cache/eggs/plone.app.upgrade-1.3.6-py2.7.egg/plone/app/upgrade/v40/betas.py", line 130, in convertToBlobs output = migrateATBlobImages(context) . . . File "/home/remy/Plone-4.3.3/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/OFS/Traversable.py", line 317, in restrictedTraverse return self.unrestrictedTraverse(path, default, restricted=True) File "/home/remy/Plone-4.3.3/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/OFS/Traversable.py", line 300, in unrestrictedTraverse raise e NotFound: MSG2.jpg

This is caused by stale indexing of deleted files, and the workaround is to rebuild the portal's catalogue while it is still hosted under v3.x, which can be done using the instructions above. Once this is done, copy the database datafiles over to the Plone v4 installation and it should upgrade fine.

KeyError errors

After doing an upgrade and browsing to your site, you may see errors such as the following in your browser window:
KeyError('language',) (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: language)
This is likely caused by a custom main_temple as a lot of template directives were changed between Plone version 3 and version 4. Problem is that it is not an easy fix, as by the time you've deleted all the dodgy directives the effort needed to fix the template up is likely not that far off doing a new template from scratch.

First thing is to get to the error log so you can find out the line that caused the error:

You also need to get at the affected file: The main_template which will be shown to you in an editor window.

Impression of Plone

I last played around with Plone back in 2010, and my lasting impression of Plone's template system was that it was a bit of a beast. In this sense the addition of Diazo in Plone 4 to separate out appearance from Plone internals was a bullet that needed biting, but apart from completely trashing any real hope of easy Plone 3 to Plone 4 upgrades, it makes the system a lot easier to deal with. Now that I am much more familiar with the ins-and-outs of Python as I was back in 2008-2010, Plone also does not seem as unwieldy as it did back then. Plone's buildout system means that a lot of stuff like upgrading is almost entirely automated, and I was pleasantly surprised at how little manual intervention was actually required.

If I was to use Plone as opposed to Django, it would be for sites where logged in users are predominantly administrators such as news authors and editors, rather than end-viewers. Plone feels very much like the gold standard in information management rather than something I would use for something more interactive such as a eCommerce site, and compared to Django it feels much more geared towards the large-scale corporate sites that would need to span multiple servers.