Hi
We develop/run several sites for our clients and some of then includes live centers where users (mostly anonomous) can get live stats from current events.
For example right no we have a site(http://www.wffc2013.com/) that show live statistics from 8 different sectors(one block each, all on one page) that refreshes every 5 seconds(javascript) and where we need to geoblock on mobile devices for some visitors.
We run apache worker with fast cgi. And during peak periods we run 8 cpus cloud server with with more ram then we need so far.
We receive all data from xml uploaded to the server. As soon as we have more then 100 people at the time we see huge CPU loads spiking the server. (parsing multiple xmls all the time seems to do this :)
We have currently APC installed in the server but cant seem to get drupal APC module to run with some of the modules installed so don't run that yet but dint seem to get that much different to add boot_strap to APC.
We add currently the result of the parsing in two layers to apc memory (after parsing the whole file and then after converting it to 8 tables).
Is the only way to do this by buying multiple servers and load balance?
We have had some success with converting the blocks with tables to their own static html files and then update the javascript pointing to them. But this seems like a harsh way to go with Drupal.
Any thought/feedback welcome.
Comments
Try separate VPS/server for xml processing
We had worked on a site where we had to do a lot of feed processing which used to bog down our server. We solved the problem by moving the feed processing to cron and then moved the cron on a VPS which continuously ran the cron but without any end users. That might work in this case. Which is the page which refreshes every 5 sec? The rest of the site looks like it can do with boost for anonymous users.
Thanks
Anoop
Check out our Free Drupal site review
Thx
Sorry should have been more clear. Its the the live center and the button to get there can be found on the front page in the top of the right sidebar.
//the event is now finished so the we dont have that mutch visitors now.
//
Niklas Nilsson, PiJa Media & Management AB
Apputvecklare, iPhone, Android och HTML5 - PiJa din apputvecklare!
ESI/AJAX
I would probably solve this using ESI/AJAX to get the content from the server, and use something like HTTPRL to keep the content cache fresh. It won't be super easy to do, but the ESI module will give you a URL that contains just that blocks content.
Having a dedicated cron
Having a dedicated cron machine is something we do for large sites with busy cron as well. We also relegate other CPU intensive tasks so this machine and run them using drush.
You can also look into more advanced cron management tools like Elysia Cron or Ultimate Cron (just google the modules). They help you organize cron tasks and schedule them appropriately.
Especially since the normal system tasks cron actually run a cache flush when it finishes. That one we always run as infrequently as possible (in some cases once a day).
For instance, if you're using Feeds, perhaps using drush to process the feed updates.
--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone
Use NodeJS
Do you think to use NodeJs to perform this task? With Nodejs and socket.io you can open communications channels over a single TCP connection, against Ajax method that you do refreshes every 5 seconds nodejs open a single channels, and if you have new data at your database him are sent in realtime.
NodeJS
Socket.io
I not experienced with node
I not experienced with node js but I think It will be the best choice for your sub-system, instead of just using a ajax to parse some XML, I believe It would be faster than anything else.
This back me up, and I
This back me up, and I believe there is a module to integrate node js with drupal node js
Yeah there is. I don't think
Yeah there is. I don't think it would really be needed for this scenario though. Even if they AJAX the results, it's much simpler to write a simple module or use the theme to put the javascript in, than using the module.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
yes, maybe you are right, I
yes, maybe you are right, I know Javascript but I didn't have the chance to work on node js to decide, but I think in general it's all about performance,
performance over simplicity is required on some system.
I just did a quick test with
I just did a quick test with Node using the xml2js module. Reading in and parsing the xml list of all Drupal module releases (almost 15mb in size) took about 15 seconds on a single core and used 330mb ram. There is another module made for processing big xml files (says it can do 1gb+ and keep low memory):
https://github.com/jahewson/node-big-xml
Node's kind of like Drupal in the sense that there are so many ways to do things, it's just finding the right tool. At least those tools are all called modules in Drupal and Node, so that keeps some braincells keeping things in order LOL.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
Node.JS
I agree 100% with node.js. I've been moving a lot of stuff out of Drupal and to node.js for a client. Mainly things like view and online counters and an internal shoutbox. It has taken a huge load off the server and the server running node has not seen the load go over 0.10 once, even at one period with over 15,000 concurrent sockjs connections (the real amazing part is that's on a CentOS 5 server, which really isn't up to par for Node).
If you know Javascript, Node isn't that hard to get into. The biggest obstacle is in your mind. You need to start thinking event driven programming; letting callbacks fire when processes complete.
Even if you don't want to live update the data via sockets, etc., you can still use it to produce the items and use ESI to get them into the page as Mikey suggests. That or just have node put the rendered data into a table for the Drupal install, then Drupal query the data and inject it into a block, view, whatever.
For parsing XML in Node, there's a couple of options. The easiest is the xml2js module, which converts XML documents into Javascript objects. If you need more in depth parsing then go with the libxmljs.
A lot is going to depend on the size of your XML documents. That applies to any technology you handle this in. One nice benefit of Node is that it isn't threaded by default. You control the threading using the cluster module, which is included with Node. Then you spawn workers from your master process to handle the processing. Just one warning if you go this route; watch the messages you send between processes. For example, don't have a worker parse and convert a big XML document into a JS object, then pass it to the master via the interprocess messaging system in cluster. Instead store it in Redis, a database, or even a file, then use interprocess messaging to say "hey - go ahead and get this file". That will save a ton on the load.
My suggestion is to get a cloud and try it out on there. You can write the stuff on a local machine (I use Node Eclipse, but have also used Cloud9), but just use a cloud to test it out for your live processing. Keep an eye on the server usage during it. You might find it's low enough that you can run node on the same server as Drupal.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
Message Queues + worker daemons
Instead of using cron, you could also try using Message queues + worker daemons to process your XMLs and do the backend data munging.
Try the combination of Advanced queue and Beanstalkd along with Supervisord to keep the workers alive.
Perhaps also try data replication once the data is processed. A simple example would be saving data into Memcache and allowing memcache to replicate data into different instances which different load balanced web servers connect to. Although there may be a slight replication lag, you would not get load spikes.
The node.js option seems nice too, although I haven't worked on it.
Thx for all the replys
Hi and thx for all feedback, ill will start looking into seperate cron server and node js (think this is a bigger step for me as im not that good with js).
//
Niklas Nilsson, PiJa Media & Management AB
Apputvecklare, iPhone, Android och HTML5 - PiJa din apputvecklare!