I have written a module that writes views programmatically, and my mymodule.views_default.inc file is enormous. Currently there are 150 individual views, and 780 will be the target (this is the initial target).
But, even with the test group of 150 several things are happening.
1) when enabling the module via drush, php memory usage goes through the roof. It climbs into the gigs (TOP command), but it finally loads.
2) but, when I then try to view the site I get WSOD, and the top command shows apache2 climbs into the gigs for memory usage until the Apache process actually crashes. Of course there is not enough memory to load it, and it's of course unrealistic to put php memory limit in the gigs.
What's going on here? How is the code in mymodule.views_default.inc actually handled once it's loaded? Where is it being loaded to? Is it loaded into memory for each Apache process? If I can actually get it loaded will my Apache process footprints actually shrink back down to something normal (they were around 60M before)?
Can this file be divided up into individual files, and would this even help? Is it better to someway put them into the database instead of using the file at all?
In other words, how can I get almost 1000 views loaded and continue doing business as usual?
note, if you are curious, this is a census data project. I don't want to use filters or arguments as each set of data needs to to have its own individual page URL and other criteria unique to that data set.
I'm developing on a LAMP stack - ubuntu, 8 gigs of ram, mod_php
Comments
Hi there. What do you mean
Hi there.
What do you mean with "module that writes views programmatically" ? Is it that you create the views in code rather than they are stored in the DB? I have done something similar and have never experienced any issues with this. However, I have each view in its individual file in a dedicated directory and load them in in hook_views_default_views() :
have some abstract code in D6.
function MYMODULE_views_default_views() {
$views = array();
// see: see http://shellmultimedia.com/articles/managing-your-views-code
$path = drupal_get_path('module', 'MYMODULE') . '/views';
$files = drupal_system_listing('.inc$', $path, 'name', 0);
// loop over all the files and include them
foreach ($files as $file) {
// each file returns an object: $view
include_once $file->filename;
$views[$view->name] = $view;
}
return $views;
}
This keeps the footprint of mymodule.views_default.inc really low, except when you flush the cache and views comes along to execute MYMODULE_views_default_views() to pick them all up.
If I remember correctly, what views does is that it stores the loaded data in its cache-table for later use (NOTE: this is not the same table as when you create a view using the GUI which is then stored in the database as well).
I can't remember out of my head if mymodule.views_default.inc is always loaded with each pageload though. But if so, and it is a rather big file like in your case, then this of course has an impact.
Personally I would not want to write a 1000 individual views if many of them can indeed be parameterised (either filters or arguments/contextual filters). You will end up in maintenance hell. If you indeed need dedicated paths, then I would use hook_menu() to create a big pile of callbacks which will redirect the user to the right view and its parameters. Maintaining the callbacks and drupal_goto() to the correct paths is for sure less painful than maintaining so many views. What are the other unique criteria? Is that something which can be changed with one of the views-hooks before rendering?
Interesting idea about the
Interesting idea about the menu callbacks, I'll definitely look into that.
The way my current module works is this:
There is an existing database, separate from Drupal, that has all the data in it that we need to add to Drupal. The module reads this data and feeds it to a "views_create" function. The views have a very similar structure, but, depending on what data is or is not available at the time the views are created, certain fields, filters, sort criteria change to accommodate. Whether or not certain data is available also determines if a page or block is or is not created programmatically as part of the view (as, I figure, it would be inefficient to have 100s of blocks or pages created that simply have no content.) As this all follows typical patterns based on the architecture of the first database, it's very easy to on-the-fly views based on what data is or is not available.
So, after the script runs it populates mymodule.views_default.inc with all the views in a huge foreach loop. All of this works great, the thing is just getting it to work performance wise.
It would be very easy to adjust the script to separate those out and write them to their own .inc files, but I was under the impression everything had to go in that one particular file. I am very curious now, I will play with separating those out as well as the hook menu option. Thanks for the tips and code!
The line for
The line for drupal_system_listing() should be:
<?php$files = drupal_system_listing('/.inc$/', $path, 'filename', 0);
?>
Otherwise you may get warnings about the regex being incorrect (at least in D7).
Memory Leaks
I've fixed a memory leak in views before
http://drupal.org/node/988680
There could be others.
Back story:
Came about that leak because of the 6.x Boost code. Boost 6.x can run every view adding in a condition for that exact node id. It takes about 5ms per view to run, but after destroying the view the memory usage kept on going up; which means we have a memory leak.
Your case:
I agree with michaelraasch. I would look at rebuilding the views cache using drush as that will not cause apache to eatup tons of ram.
How can I rebuild the Views cache with Drush?
I am familiar with clearing the views cache with Drush, of course, but how can I rebuild it with Drush?
I was in the process of writing a tiny script using CURL to loop through the urls. Better way to handle this?
Thanks for the info on the memory leak. I'll check more into that. The current data / file set comes to 100M. I was able to finally able to make enough RAM available (8G) for mymodule.view_default to load. As I mentioned the PHP process peaked out at a high 7G before it actually loaded. What detective work did you use to pinpoint the memory leaks in your thread? How can I test for that?
Drush
To prime the cache I would use php-eval and try running views_get_all_views().
drush php-eval "views_get_all_views(TRUE);"When one destroys (unset) a view object in memory you should get all of that memory back. If you do not then odds are there is a memory leak. Fixing memory leaks is hard; but making a small repeatable test case showing the leak is the first step to fixing it.