When trying to adhere to established best practices like avoiding singletons, registry, static properties and base controllers, how I can populate my layout(and partials used by the layout) with data that is only used by the layout and common across all actions?
The typical scenario is menu which is built on variable data, like an database. Keeping separation of concerns in mind, the views/layout should never talk to an backend directly, but rather be told what to contain.
Using a front controller plugin is simply not possible without using the singleton "feature" in Zend_Layout. The plugin only knows the request and response object, neither has access to controllers, views or the layout.
Zend's action helpers have init/preDispatch/postDispatch methods. One can add action helpers to the HelperBroker (ex. using the bootstrap) and these will be executed in the normal application flow.
Using the init-method to inject data into the view is not possible since it's trigged before the controller/view is ready. preDispatch/postDispatch is possible, but not perfect, since these methods are always triggered when executing a controller action.
That means that all uses of the Zend_Controller_Action::_forward() will also execute the preDispatch/postDispatch in all action helpers. This doesn't have any big implications for the code except speed, I really do not want to be setting a view (or an view helper) variable several times. One can code around this issue using some sort of $firstRun variable, but I really don't want to track this in my own code.
Another method is doing this in the bootstrap, but in my opinion it really does not belong there.
So, how can I populate my layout/view helper with data from the database, doing it only once and still keep a good separation of concerns?
You could go with Action Helpers as stated in this answer (take a look at the question too, its pretty much similar to yours)