"Large" Multisite / Domainaccess Hosting & Deployment

macdev_drupal's picture

We are running about 50 Drupal 7 Multisites + 1 Domainaccess Site with 10 Domains on a 2x 64GB 16 Core Cluster Nodes with Apache mod_php with apc and mysql stack.

The multisites are utilizing all the same codebase and just differ in usage of content and node types.
At the moment there are 3 mysqld running on the host.
We are thinking about splitting the 50 multisites in 2 halfs and run another mysqld for that.

One of our main problems is, that deployments with changes to many features could take several hours at the moment.
So we are looking for solutions to speed up drush rr -y && drush updb -y && drush cc all -y && drush fra -y
Sometimes we already revert only dedicated features, disabeld the clearing of the entity and features cache on drush cc all.
disabled Cron during deployment...

I think splitting up the multisites to 2 or even more mysqld could help us by running drush parallel at the same time (running drush in parallel mode against the same db-daemon lead to locks.)

As we are driven to go virtual this year this is a real show stopper. We already evaluate using docker or alternatives to get smaller deployment units.

Do you have any other recommendations?
Could e.g. clearing the cache via mysql-queries directly speed up the process? Are there any tweaks speeding up drush fr ?

Comments

Used to have a similar setup

mikeytown2's picture

We had a similar setup on D6, 1.3k domains via Multisites + Domain access (direct competitor to http://patch.com/). We were using bare metal; 4 MySQL boxes, 7 Apache boxes, and 3 Memcache boxes. You can virtualize the apache boxes but we found that MySQL worked best on bare metal; memcache was always the weak link for us. The best you can do is run 2-3 drush deployment scripts in parallel per MySQL box. This was before drush had parallel mode so we used bash and exit codes to keep track of the currently running jobs.

3 mysqld running on the host

Put each mysqld instance on its own box; this will allow you to have 9 deployments going on in parallel. If you are using the core database layer for the cache you're going to have a bad time with metadata locks; if this is the case you can only do one drush deployment per mysql box. If you're not ready to switch to Redis/Memcache checkout LCache/APDQC.

The only way you can speed up your deployments is to get a cachegrind of one and fix the bottlenecks. You can do tricks via parallel execution but to truly speed it up you'll need to patch core most likely.

Agree w/ Mikeytown2. Some

btopro's picture

Agree w/ Mikeytown2. Some other considerations:
- https://www.drupal.org/project/dslm DSLM can be used w/ a structured routine / bash script to systematically flip symlinks per site, then run the commands you list above, then do the next and the next and the next so you reduce donwtime of each site individually
- We use APDQC which really speeds up cache read/write for things like this
- php-fpm is WAY faster then mod_php to speed up how those commands are processed. Obviously php 7 will give you gains to decrease processing time too.
- I can't do concurrent drush above 3 and still see a benefit. Granted we don't cluster anything but on a dual-core 5.7 fast dell blades we max out CPU if we 3 concurrently. 2 concurrent is fine for about 90% of a 100+ site multisite rolling through in a fashion you lay out.
- drush rr -y (clears cache) && drush updb -y (clears cache) && drush cc all (clears yet more caches) -y && drush fra -y (also clears cache) just so you know, you don't need that drush cc all in there.
- I wouldn't do fra and would instead opt for hook_update style features_revert_module('MODULE_NAME'); calls to the specific features that need reverted. This eliminates additional cache clearing calls as well as calls for forcibly reverting things that don't need to sit there processing to begin with.
- Some cache re-seeders worth checking out:
https://www.drupal.org/project/drush_ecl - cache all entity types (doesn't build pages, just primes objects)
https://www.drupal.org/project/httprl_spider - spider paths via httprl front end calls
https://www.drupal.org/project/xmlrpc_page_load - works with httprl_spider to prime calls, faking a specific user (useful if pages are auth'ed, plentiful, and need rebuilt)

ELMS:LN handles it's spider upgrades via a drush recipe (https://drupal.org/project/drush_recipes) that RR's, takes site offline, runs updb, brings site online, runs cron, runs ecl drush @sites cook dr_safe_upgrade --y