New hostmaster feature : Queue Dispatcher
I've built a new API for use in hm2, which allows you to define queues that would like to get triggered. I've built the api so that defining 2 functions hook_hosting_queues and then the queue handler, gives you an impressive amount of free process accounting, and also configurability. 
What the queue dispatcher does, is it keeps a list of queues, when last they were triggered (dispatched), how frequently they need
to dispatch, how long have they taken (are they hung?) and exactly what they are busy with. The primary example of a queue
is the task queue, which translates all changes on the front ends, down to the back ends.
Additionally, the dispatcher supports two types of queues, your typical serial queue (process x items every y seconds), but also
the far more complex batching type of queue. Batch queues are like the cron queue. You need to ensure that all your sites are
cronned every hour for instance, but it is unfeasible to try and cron all your sites every hour at a certain time.
What batching does, is it knows how many items it needs to process, and in what amount of time it needs to do so. It balances the amount
of items being processed in one instance, until the amount of items per instance become too great, then they add another instance.
As an example, if you have 100 sites that need to be cronned in an hour, it will cron 20 sites every 10 minutes, but once the number of sites per cron reaches 100, it will add another instance, until it reaches it's maximum amount of instances per interval. At 1000 sites,
it will have grown to do 100 sites every 5 minutes, keeping the load even the whole time.
This will allow you the flexibility of configuring how frequently sites are backed up, statistics gathered, system monitoring take place,
and the api handles configuration, error handling and all the rest.
It could even be possible (though most php internals people i've spoken to don't recommend it) to run the dispatcher as a daemon,
for times when you need quick turnover of tasks -> sites.
| Attachment | Size |
|---|---|
| queues - 2 - queue admin.png | 34.18 KB |

