How do I create modifications to an Octopus existing platforms and install profiles?

Posted by wickwood on June 28, 2011 at 9:08pm

I originally posted this set of questions at http://drupal.org/node/1186524 and rewrote them once there because after reading my original post I realized that I had done a pretty poor job of creating the post. I wasn't in the best of moods when I created the first post, so I hope anyone that reads it forgives me for that.

Omega8cc suggested that I repost the questions here for discussion now that this group is open. I would have done it sooner, but I've been dealing with a pinched nerve and it's putting a "crimp" in my workflow and getting things done.

Also, since I'm reposting and re-reading, I've made some changes from what I posted at http://drupal.org/node/118652

Anyways, here is the main question at hand:
How do I create modifications to an Octopus existing platforms and install profiles?

I have BOA installed and all of the distributions are very nice, but I need to make my own modifications to them to achieve the desired results. This includes adding non-site specific modules and themes, having as many sites working off of the same code bases as possible to minimize the work required during upgrades, and I also want to be able to upgrade these distributions on my own schedule and not be dependent on the release of BOA updated scripts.

This has created several questions for me, which might in many ways be the same question being asked several different ways:

Q1. In Aegir, why do we pick the Install Profile first and then select the Platform?

Perhaps with more experience of creating Platforms and Install Profiles this will make sense to me, but I find this really confusing because, as I understand it, a Platform in Aegir is a Drupal Code Base with a set of associated Install Profiles. But when I go to create a new website, I select the install profile first and the list of available Platforms changes.

What is wrong with my understanding here, because intuitively this is backwards to me. I suspect I have fundamental misunderstanding of how things are working.

Q2. Correspondingly, are all of the distributions released with Octopus that are based on Drupal 6 or Drupal 7 share the same Drupal code base respectively?

As far as I can tell, the answer to this No, but I might be wrong about this.

I've been trying to understand the file structure here and the Platform/Site interface in Aegir, and it appears to me that each of the Octopus distributions, even if based on the same version of Drupal 6, each have their own copy of Drupal to make up their own platforms. Again, I expect that with more Aegir experience this will begin to make sense to me, but it doesn't yet! ;-)

It also seems to me that unless a particular platform has altered the Drupal Code Base in some way, that all the Install Profiles for Drupal 6 would be associated with the Drupal 6 Platform and similarly for Drupal 7.

Q3. Related to Question 1 and 2, where do the Platform Install Profiles live on the server?

I know I have to spend more time exploring the file structure of BOA and this will probably/eventually make sense to me. As I stated before, I probably have a fundamental misunderstanding of something and once that is cleared up everything will make sense to me. But as of yet, I'm still struggling with these issues that seem critical to understand if I am to be able to achieve my goals with this.

Q4. Am I suppose to create a Install Profile for every site I want to create?

It would seem that the answer to this is Yes, but I don't yet understand why.

As I understand it, an Install Profile along with a copy of Drupal make up a Platform or what Mig5 calls a Platform Build in his tutorial at
http://greenbeedigital.com.au/content/drupal-deployments-workflows-versi...

So, I guess that when I create a new version of the site with new modules and/or themes I will have a modified Install Profile/Platform Build, to migrate the old site to.

I understand that these new Install Profiles and Platform Builds are created with Drush Make files which I keep in Git version control.

So I'm probably asking the same question again in just a different way here, but based on this wouldn't I just need to upload Install Profiles to a given Drupal Code base to create a new profile for a site (or set of sites) if they are all using the same version of Drupal?

Q4. How do I, or can I, add an Install Profile to an existing Platform on Aegir like any of the Distributions preinstalled with BOA or any previously existing platform?

This was my original question and is what I'm trying to actually do, but as I've been trying to figure this out, these other questions took over.

Right now, I think the answer to this is you can't and it is probably my own inexperience with Aegir and doing things the "Aegir Way" that confuses me as to why I can't or wouldn't want to do this.

I think the answer to this question is here, Migrating sites from one Aegir platform to another with a different install profile, but I haven't yet had the opportunity to explore this.

Also, Omega8cc replied to my original post and said, "As for the sites migration between different install profiles - there is a very simple, already built-in in Aegir method I explained on the IRC a few days ago: http://hefring.mig5.net/bot/log/aegir/2011-06-06#T79665 - look for old_short_name trick there." I haven't yet had the opportunity to explore this yet either.

Q5. Are there Drush Make files for the Octopus distributions some where so that they can be modified?

Maybe for some, but from what I see in the Octopus script I don't think so. I think if I want to make my own modifications to these Distributions I will need to just create my own Drush Make files that pull the source codes for the distributions like the Octopus Script does.

Thanks again and any guidance from anyone will be greatly appreciated.

Steve

Comments

Profile creation challenged

Posted by ericrdb on June 29, 2011 at 7:12am

Thanks for your questions and thoughts wickwood.

I'm thrilled about Barracuda and Octopus and find myself stuck trying to add profiles.

I picked a profile I want to deploy: http://openchurchsite.com, added the .make file as a new platform and it verified (after I declared pressflow [core] in their makefile).
From here, I created the profiles/openchurch directory and uploaded their openchurch.profile there. I re-verified the site, and I'm not seeing the profile added to the original list from Octopus.
I am seeing a new platform when I select the Pressflow profile. Based on what I read, I expected to see Open Church as a new profile option and make no platform selection, as my newly created platform should be the only option for that profile.

I read the articles noted above, as well as http://bryanhirsch.com/node/7, as referenced in http://drupal.org/node/118652. Also, http://omega8.cc/import-your-sites-to-aegir-in-8-easy-steps-109 (nice redesign, btw). If I understand, I should be on the right track.

What am I missing for creating a new profile?

cat /var/aegir/config/includes/barracuda_log.txt

Tue Apr 5 23:04:22 UTC 2011 / Ubuntu.lucid i686 / Aegir 1.0-boa-T / Barracuda 1.0-boa-T / Nginx 0.9.5 / PHP 5.2.17
Wed Apr 6 03:14:55 UTC 2011 / Ubuntu.lucid i686 / Aegir 1.0-boa-T-2 / Barracuda 1.0-boa-T-2 / Nginx 0.9.5 / PHP 5.2.17
Sat Apr 9 08:25:54 UTC 2011 / Ubuntu.lucid i686 / Aegir 1.0-boa-T-5 / Barracuda 1.0-boa-T-5 / Nginx 0.9.5 / PHP 5.2.17
Fri Jun 3 16:44:18 UTC 2011 / Ubuntu.lucid i686 / Aegir 1.0-boa-T-8.7 / Barracuda 1.0-boa-T-8.7 / Nginx 1.0.2 / PHP 5.2.17

Answer to Q1

Posted by omega8cc on July 7, 2011 at 8:57am

Q1. In Aegir, why do we pick the Install Profile first and then select the Platform?

A1. There was a reverse order of Platforms and Install Profiles before, but it was much, much more confusing and hard to manage, so it has been changed to the current design to fix all previously existing problems.

I will explain it a bit: first, the App in Drupal (and in Aegir) is not a Platform, it is the Install Profile, while the Platform is simply an environment, where the Install Profile lives.

Next, the same Install Profile can be used (and is used) in many different Platforms, each based on either different Drupal core version or different Distro (like Open Atrium) version.

This means that the list of Platforms can grow out of control and the form will be a giant and hard to use, because you will have to find a Platform you wish to use first and then again choose the Install Profile in this Platform below, which was totally confusing and people always was curious wtf?! they need to choose the Install Profile after they already picked the Platform?

Now the list is limited to the list of Install Profiles and not Platforms first, and when you click on any Install Profile, the list of Platforms including this Install Profile will be displayed dynamically. This is in fact far better than the previous logic, so this is why it is done that way.

Answer to Q2

Posted by omega8cc on July 7, 2011 at 9:02am

Q2. Correspondingly, are all of the distributions released with Octopus that are based on Drupal 6 or Drupal 7 share the same Drupal code base respectively?

A2. No, no code is shared between Platforms.

This part of Octopus description: "It allows to install new versions of platforms with clean directory structure, with code shared between all created instances, so one vanilla Octopus instance is using only ~18 MB, while most of the code, which is almost 400 MB total, is shared." is about sharing the Platforms code between Octopus Satellite Instances created on the same server and not between platforms in the same Octopus Instance.

Answer to Q3

Posted by omega8cc on July 7, 2011 at 9:05am

Q3. Related to Question 1 and 2, where do the Platform Install Profiles live on the server?

A3. In the /data/all directory tree, where from they are symlinked in every Octopus Instance directory tree.

Answer to Q4

Posted by omega8cc on July 7, 2011 at 9:17am

Q4. Am I suppose to create a Install Profile for every site I want to create?

A4. Absolutely no.

There is no reason to create separate Install Profiles per site, unless every site is absolutely unique, but then you could either create separate Platform per unique site or yes - use site-specific Install Profile in the same Platform.

The problem with this approach (Install Profile per site in the same Platform) is that you are locking yourself in the chicken/egg syndrome - you can't modify/update safely any module or theme (or Install Profile!) in the Platform /sites/all space without changing also all other Install Profiles on the same Platform, and when you think you can safely separate the code between sites using the Install Profile specific /profiles/name/modules space, you are doing it wrong, because you are only complicating things further and duplicating the code in the same Platform. Why not to maintain it in a separate Platforms then instead?

Answer to Q5

Posted by omega8cc on July 7, 2011 at 9:25am

Q5. How do I, or can I, add an Install Profile to an existing Platform on Aegir like any of the Distributions preinstalled with BOA or any previously existing platform?

A5. You can either add any Install Profile to any pre-installed Platform, living in the /data/all space and then re-verify the Platform in Aegir to be able to use the added Install Profile, or you can use the built-in Aegir feature and migrate the site from your imported first, custom Platform with custom, old Install Profile, as explained in my how-to:

http://omega8.cc/migrate-sites-between-install-profiles-easily-111

Answer to Q6

Posted by omega8cc on July 7, 2011 at 9:30am

Q6. Are there Drush Make files for the Octopus distributions some where so that they can be modified?

A6. Yes, all makefiles we use in Octopus are available, so you can modify them and use to build your custom platforms: http://drupalcode.org/project/octopus.git/tree/HEAD:/aegir/makefiles.

Note that we some Platforms are created from downloaded tarballs, even if they include their custom makefiles, so you would have to look into corresponding Install Profiles directories to find their makefiles to modify and re-use.

@ericrdb

Posted by omega8cc on July 7, 2011 at 9:38am

Correctly created makefile will create fully functional Platform for you, so you shouldn't create any directories or upload any files manually. If the newly created Platform has its own, unique Install Profile, it should be displayed on the list of Install Profiles in the Create Site form.

If it isn't available, delete this Platform node in Aegir (don't use the Delete task) and add the Platform again, using its known system path.

The how-to for deleting Platforms or sites nodes: http://omega8.cc/aegir-task-fails-or-spins-forever-126

Thank you for the detailed answers...

Posted by wickwood on August 28, 2011 at 1:38am

Thank you for the detailed answers Omega8cc, and I'm very sorry for the long time it has taken for me to respond with my appreciation.

I should have commented right away, but I wanted to digest your answers and try to post thoughtful replies. However, health problems followed an August full of activities and events prevented me from finding the time to work on this.

Now I'm back at this and trying to learn how to use my BOA again. Unfortunately, the time away hasn't helped me at all.

I'm sure I will have some follow up questions because this sure isn't making sense to me yet, but I'll wait to post until I can formulate my questions more coherently. Confusion (and frustration) sometimes makes it hard to be articulate.

Thanks again, and I hope you have had a good summer.
Steve

Here's where I think I'm stuck...

Posted by wickwood on August 28, 2011 at 4:40am

Here's where I think I'm stuck and not understanding how to best use this. Let me outline an example to help clarify things.

I want to build many sites using the platforms that comes with BOA. In this example let's assume the set of sites will be based on the OpenPublic platform.

First thing that I want to do is add the SecureSite module to the OpenPublic platform so that I can password protected development sites and make them inaccessible to search engines and other crawlers but still allow access to certain users. Also this first site will have it's own custom theme.

Question Set 1:
Does dev.OPSite1.org have to be on it's own Platform?
Or could this SecureSite Module just be added to the BOA version OpenPublic Platform so that the 2 sites are sharing the same code?
If the SecureSite Module can be added to the BOA version OpenPublic Platform to customize it, how should I do this?
Can I just added it /data/all/002/openpublic-beta1-build100/sites/all/modules ?
What about adding the custom theme?

Assuming I do need to create a separate OpenPublic Dev Platform, I would create it with a version controlled Platform Makefile that would pull the same version of the OpenPublic, the D7 version of the SecureSite module and the custom theme.

Now I create the first site, let's call it dev.OPSite1.org, on the OpenPublic Dev Platform and when I'm done, I clone it to BOA version OpenPublic Platform create the production version of the site and name it www.OPSite1.org.

Question Set 2:
If dev.OPSite1.org does have to be on it's own Platform do I need to add the SecureSite module to the BOA OpenPublic?
And if so, can I automatically turn it off when I clone dev sites to the production version
or will this always have to be reconfigured manually?

Next I want I want create the second site, dev.OPSite2.org, on the OpenPublic Dev Platform, but this site requires a some unique additional functionality from a custom module and another custom theme. When it's ready, I would then clone it to a production site called www.OPSite2.org.

Question Set 3:
How do I add this site specific module and second custom theme without creating another whole new set of platforms?

I can see the number of platforms quickly getting out of control here, and since I need to create new platforms to perform upgrades to Drupal, OpenPublic, and all the Contrib modules, even if everything is managed with version controlled Drush Make files the workflow seems to be spiraling out of control and we only have 2 productions sites with 2 corresponding development sites.

Question Set 4:
I just don't understand how I'm suppose to add site specific modules and themes and/or customizations to default platforms without creating new platforms for every single site.
And if you are creating a new platform for everysite, how is the code ever going to be shared between the site?

Am I understanding the process correctly or am I missing something here? It really seems like I'm missing something because this just doesn't seem like an efficient use of code or time.

And if I can't add modules to an Octopus pre-installed platform either on a site by site basis or to create a new platforms, I don't understand the purpose of installing any of Octopus' platforms.

Thanks again for all your help and support, and of course, patience! One day I will get this!
Steve

Ooops.. I missed your reply

Posted by omega8cc on September 22, 2011 at 2:30am

Ooops.. I missed your reply for some reason. I will try to respond soon.