The Plone content management system has been around for several years, and the core set of publishing features is very stable and reliable. However, if you require features not included in the core, you will need to either create custom templates and content types (see the Plone documentation) or use third-party software packages, which are called 'products' in the Plone world. 

As I've worked with Plone over the last year or so, I've found that very few of these products are reliable enough or well-documented enough to be deployed without significant research and testing. I needed a way to systematically decide what works, what doesn't and what is worth deploying.  Here are a few of the questions, guidelines and tactics I've used in that evaluation process.

Planning

Much has been written, on the web and in print, about the process of defining requirements, so I won't cover this in much detail, other than to reiterate the importance of outlining a list of desired benefits and features, as well as the limitations of the project and organization, before looking at the specifics of available solutions.

I don't think you need to be overly detailed in this first phase, and needs often change during the research phase. But your initial guesses can act as an anchor that will keep you from drifting too far in the direction of an attractive solution that might not be worth the trip.

This is also a good time to define a few use cases: who needs to do which tasks, and what steps will they take to complete those tasks?  The answers will be used later, during the testing phase.

Finding Plone Products

You can find Plone products at these locations:

• The products section of plone.org (This is quickly becoming the only place you really need to check.)
  
• Zope and Plone Products at contentmanagementsoftware.info
  
• Plone's Subversion Repositories
  
• The Plone Collective
• Ingeniweb
  

Initial Assessment

• Identify features and benefits offered by the product that seem to match your defined needs.  Also make note of any that may conflict with your intended use cases.
  
• Identify dependencies: which versions of libraries and other products are required?  Often, required products will have their own requirements, so you need to follow the dependency chain back as far as it goes.  Also make note of the sequence in which all dependencies must be installed.  Are there any major documented incompatibilities?
• Check the product's bug tracker/collector.  How many bugs have been resolved?  Were they resolved quickly?  Which bugs haven't been resolved?  Are they critical?
• How old is the oldest significant bug? All software has bugs, but you should check the collector or bug tracker carefully for any that affect the specific features you need. 
  
• If you find a bug that might be important to you: Has the bug been tasked to a developer? Is there any information on how it might be fixed?  If there is a fix, has it been assigned to a pending release or is it simply in the code repository?
  
• What is the release frequency?  Don't be impressed or discouraged by version numbers.  There are fully-functional open source software packages with version numbers like 0.2, and there are also useless packages labeled as 3.1 or higher.  Version numbers are not used in a consistent and standardized way, so they are only a relative indicator of product maturity.
• Is the product 'plug and play' or does it require extensive customization?  TeamSpace, for example, is described as a 'framework' product - it is a 'toolkit' and not a finished product.  While this kind of solution provides additional power and flexibility, it also means you will need to budget time and resources for designing, coding, testing, etc. Using this kind of product may require different skills that are well beyond those needed for standard Plone operation and management.
• Who created the product?  Do they continue to support it?
• Review other resources like the Plone newsgroups, the #plone IRC channel, etc.  Try to find out who else might be using the product, and what kind of experience they've had.
  
• Download the product and read the included documentation. Does it match the currently-released version? How far behind is it? You may not be able to determine this until the testing phase.
• Do different releases of the product have well-documented migrations?  Are migration scripts included, or is there an assumption that users of the product will write their own, if needed?
  
• You may want to contact the developers/maintainers of the code directly and establish a relationship if it is a product that will become central to your project.
  

If you research all those angles, and the product still looks promising, it is time to test.

Testing

Based on my experiences with third-party products, I approach all products with the assumption that they are essentially untested, even those with releases or versions labeled beta, release candidate (aka RC) or final. 

Another rule of thumb:  I won't even go to a test phase on products that have not been released.  Unreleased software is often hard to test because it is a moving target. Any testing you do may be invalidated the next time a maintainer checks new code into the repository. If you want to help the developers move their product along by testing and reporting bugs, that's a great way to participate in open source.  In this context, I'm assuming the goal is to assess software for actual, near-term deployment.

In general, I think most non-profits should avoid deploying unreleased software into a production environment. If a product is extremely critical, or if your organization or project has the resources to test and re-test, then it might be worth it to explore unreleased code. If you do decide to deploy unreleased code, you should have a very good reason, a well-tested recovery plan, and a long-term maintenance and migration plan.

Describing a full testing process is beyond the scope of this document, but here are a few tips to get you started:

• Set up a test environment.  Although Plone is cross-platform, it's probably best to use the same operating system as your production server.  The testing environment must have exactly the same Zope and Plone versions as your production server for testing to be effective.
• Learn how to quickly create new Zope instances (this varies among the different platforms) and set up Plone instances within them.
  
• For each product you want to test, first test it in its own Zope/Plone instance, with no other modifications.  Again, details vary depending on your operating system, but you can use this how-to as a model. You will, of course, have to install all the dependencies for a product first, and the order of installation is often critical.
• Once the product is installed, step through a few of the use cases that you defined in the planning stage.  Try to include a wide variety of actions that users with different roles might take.  Deliberately use the tool incorrectly or outside the design norms of the product to see how it responds.  Try to break it - that's why you set up the test instance in the first place!
• If possible, have a few of your real-world end users try it, and see what they think.
  
• Verify that standard Plone functions still work as expected.  For example, does the 'email forgotten password' feature still work?
• Check configuration panels, both in Plone and the ZMI.  Can you make important changes here, or do you have to edit configuration files on the file system and restart the Zope server?
  
• If you are satisfied with the performance of the product, install it in combination with all other products that you would like to use, and re-test.  Are there any dependency conflicts? Do they all work together? 
  
• Test the migrations if possible.  Although you could test a product that works now with, for example, a beta release of Plone 2.5, it is still difficult to predict whether it will work with Plone 3.0 a year from now.

Making a Decision

• Which promised features and benefits actually work?
• Is it maintainable in the long-term?
• Will it play nice during future migrations?
• Can you un-install?  For example, CMFMember is a one-way trip - there is no way to remove it from a Plone site once you have migrated to it.
• What risks or harms are introduced?  Do any Plone features stop working? Are those features less important than the beneficial features of the product? Can you define workarounds? How much will they complicate your existing processes?
• What are the alternatives, either in terms of defined processes or other products? Is there another, lighter way to provide the same functionality?
• Is it better than competing packages which try to solve the same problems?  I would compare the user experience to other non-Plone products, because your users will. They are going to compare it to their other experiences on the web, and won't be satisfied if it's only good in comparison to other Plone products.