Posted by jaydul on January 30, 2015 at 11:25pm
We are using a custom CMS for our newspaper.We ware thinking to move drupal.here will be 1 million article.Its post all news 1 day then its create a day folder and then index then all are html file here.We cannot update 2 ore more a days that the problem.
I want move drupal.but I need to know its possible all article cache and when user visit article here will be no query on database.I want put as all articles as static.
After news post we will run cron to make some changes and it will added new post as static and change homepage too.now a days need to post articles in 3/4 hour regular interval.
if possible please let me know some idea.Then we will start our work with drupal.
Comments
It is possible
https://www.drupal.org/project/boost sounds like a good starting point for what you're trying to do.
https://www.drupal.org/project/varnish is also something to consider using.
This really depends on the
This really depends on the scale of the site. I'm generally averse to Boost on anything remotely smelling like it's large. Most sites can benefit from Varnish configured with a good VCL and no Drupal module is required.
Different cache times may be set using https://www.drupal.org/project/ape
Hi i am facing lots of 500 server error on drupal site
Hi I am facing too much slow down with my website http://www.touristsafari.com. Whenever i click on edit link of taxonomy urls, i see 500 server error. Any help?
Thanks and Regards,
Keshav Solanki
http://www.touristsafari.com/
Bug in code
More than performance issue, this seems like bug in code somewhere. Did you check web server error log?
Token Tweaks
This helped me with something similar.
Id give it a try. Set the value to 2 or even try it with one.
It limits how many levels of tokens are loaded.
https://www.drupal.org/project/token_tweaks
Bug in code
More than performance issue, this seems like bug in code somewhere. Did you check web server error log?
Not yet
Hi neeravbm, i am checking all possible resolutions. Checking logs on server now. I am using Godaddy Shared hosting.
Thanks and Regards,
Keshav Solanki
http://www.touristsafari.com/
One common problem is running
One common problem is running out of memory. You would have to look in the raw error logs for that problem. I'm not sure if GoDaddy will allow you to bump up PHP's memory limit or not, if that is the problem.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
Could also inject into CPS
Hi you could also try to use CPS module in combination with boost, to automatically permanently cache everything that is published on the site.
Thanks,
Fabian
But still my site mysql
But still my site mysql query.I need after post all will be static as html.Because here is 1 million post and DB is too big.
You can also try the
You can also try the https://www.drupal.org/project/static or https://www.drupal.org/project/mag modules to create static mirrors of your site.
Thanks,
Fabian
1 million rows for mysql is
1 million rows for mysql is not a big value, it is designed to store huge amount of data. If you have problems with database performance, try to configure replication (in case of issues with "read" queries) or provide sharding support (in case of issues with "write" queries).
Also you may think of switching your storage engine to MariaDB or Percona - it is mysql-compatible databases which provide a lot of nice tools to increase performance. But of course, you have to digg into documentation and adjust configurations to cover your case.
To get some help with tuning database configs consider using mysqltuner - it gives a good overview what could be configure better to increase performance of your db. Just be aware that this tool needs from mysql server to work at least one day to give you useful tips.
Also another quick solution to your problem could be changing your hardware (if possible). Just switch your storage from HDD to SSD and you will see the difference. That is a simple advice, but somewhy developers usually forget about it :)
MySQL Tuning
If you want to get the most out of MySQL, using the latest 5.6 branch of it is HIGHLY recommended. The InnoDB engine was greatly improved. I agree with Spleshka, 1M rows isn't that big of a deal as long as you have enough ram to keep the DB from swapping.
mysqltuner is a great resource. If you want an alt that is a module https://www.drupal.org/project/apdqc has a pretty comprehensive checklist on the status report page; if you actually use APDQC as the cache backend, it should prevent any deadlocks & metatdata locks on the cache tables in the DB.
I want use as HTML5 CSS3 as
I want use as HTML5 CSS3 as static front end And back end Drupal for articles posting.
Thank you for your help now I
Thank you for your help.Now I m thinking move drupal+elasticsearch+disqus+Static Generator.
I want All visitor see my
I want All visitor see my html raw file don't wanna use DB
Nginx + PHP-FPM + Memcache +
Nginx + PHP-FPM + Memcache + Varnish + mariaBD +Drupal 7+ elasticsearch+disqus+Static Generator all from HTML file I just just from a 2 core 4 GB RAM 80 GB SSD Server Hope it will good for 1 m post
I don't think you need both
I don't think you need both nginx and varnish. Just choose what you want to use. Personally I'd prefer to use nginx + memcached for deliver cached pages (using memcache_storage module for Drupal). If you will store cached pages in memcached, then don't forget to gzip them: you have only 4gb of RAM, and you may feel lack of memory if you will not care about it.
Also keep in mind that all search pages (no matter which search engine are you going to use, elasticsearch or others) will have to bootstrap Drupal anyway. Even if you store html output of searched content. Of course, you could try to cache the complete search page and embed search output with ssi/esi, and then request for data from elasticsearch within ssi/esi blocks. This may help you only to avoid database connection, but only if you will avoid drupal full bootstap as well.
HI Spleshka thank you for
HI Spleshka
thank you for comments.What about if I use
Nginx + Memcache + mariaBD +Drupal 7+drupal solrsearch+disqus+Static Generator.
If possible I want my html Static Generator front-end server.It,s possible?
Thank you
Nginx + Memcache + mariaBD
php-fpm is missing here, the rest is okay.
Choose any search engine you like most. Personally I like Solr or Elasticcsearch, but it is up to you to decide. Module Search API will help you with Drupal integration.
If you will use nginx + memcached + memcache_storage module, then you may not need Static Generator module. You can configure nginx to deliver pages from memcached even for authenticated users (if this is what you really want).
Hello Thank you for quick
Hello Thank you for quick reply.Still I m thinking use Static Generator and back-end want keep admin.domain.com a server and another one maindomain.com
Its possible?
Thank you
It seems like it could be a good idea.
Just be careful about using the Static module because from what I can tell, the HTML website copy it produces has ZERO dynamic features, which means no login, no permission based block control etc etc.
But if you're happy to serve the exact same site to every user, then it seems like it may be a good idea.
If you need to add in dynamic features later on, it could be an issue, or you may be able to work around it by having AJAX queries to bring in dynamic data and overlay/edit the static HTML accordingly.
I would advise testing out a few different configurations, and use a tool like NewRelic to get performance figures from each. Keep in mind that the workflow for producing new articles, and the routines for site maintenance are also very important - no point having a snappy website if it's a nightmare to run and maintain!
I just need HTML site here
I just need HTML site here user can visit site and I will post daily 3000 articles after generate Static site it will go to live server.I dont need dynamic data and overlay from front end.
If I want use Nginx + PHP-FPM
If I want use
Nginx + PHP-FPM + Memcache + Varnish + Redis + Drupal 7
It will be good?
No, I already told you that
No, I already told you that using nginx together with varnish is redundant. The same with usages of memcached and redis: choose only one tool.
I don't think you can make
I don't think you can make blanket statements about that. Nginx, for example, can have multiple uses. I've seen it used as:
Now it can fulfill each of these three roles potentially independently -- though I confess I might question using it as an SSL terminator with another reverse proxy, I would probably recommend Pound instead. But, for example, there is functionality that Varnish provides that nginx doesn't, e.g. bans along with some VMods that would be somewhat more challenging to implement similar functionality directly in nginx. At the same time nginx provides arguably more consistent performance under load than Apache 2.2 -- or so our sysadmins tell me. Redis and memcached are probably somewhat redundant, but depending on the functionality I could see it. Say, for example, you wanted to use Google Pagespeed and not store the objects on the file system, you could use memcached -- it doesn't support Redis. But you might also want Redis to store the cache objects from Drupal due to the better interface for finding and iterating over keys or persistence.
Redis is also better at
Redis is also better at wildcard flushes from what I heard at performance talks but also the big win for me was memcache item is limited by default to 1MB and views data, menu and features cache like to grow items bigger than that... so throwing things in to Redis doesn't have this issue.
Nginx and Varnish are not
Nginx and Varnish are not necessarily redundant, as others have said. One common setup is to setup Nginx to act as the webserver, fronted by Varnish on port 80, with Nginx terminating SSL in front of Varnish on port 443. Would Pound be better in that case? Perhaps from a raw performance standpoint (no, I don't have numbers), but it would introduce yet another point of maintenance.
Some point to Varnish as being duplicative of Nginx microcaching. It depends on the use case. Varnish uses more CPU than Nginx in my experience, but caching for complex sites (such as e-commerce) is simpler to manage.
By the way, we could throw both Memcached and Redis out and still use Memcache tooling. MySQL 5.6 does have a Memcache plugin that emulates the Memcache API. Performance is roughly halfway between Memcached and MySQL, with the key positive being cache persistence.
Hmm...
How many articles are likely to be added each day? It doesn't sound to me like the actual dynamic part of the site is under that much of a load, so maybe Fabian's suggestion of using the Static module to create an HTML only version of the site is the best option?
Serve that off a decent platform with nginx and it should handle the load no worries? nginx can be hard to run Drupal off if you're a bit of a noob like me (I'm using Pantheon though, so that's taken care of now) so perhaps a strategy could be something like:
"Backend" website like system.mysite.com - running Drupal off a standard LAMP stack, no tricky speed-ups necessary. Add and edit articles etc. and then the Static module pushes an HTML only version of the site once a day (I'm not sure how Static works?)
"Front End" website like mysite.com - runs on nginx and serves HTML only, uses some non-Drupal related search system.
It's an interesting scenario. I'd never heard of the Static module before today, so now pondering use cases :P
Drupal pre-bake
I've seen a number of schemes to convert Drupal to a "pre-bake" CMS. There are invariably tradeoffs and edge cases that you bump into -- things like filterable lists, search, contextual content, etc. Greatly depends on the functionality you are trying to expose. For the majority of sites I work on, content rich sites with lists of types of content with faceted browsing, featured content, etc. I haven't been very comfortable with those tradeoffs. It's generally been easier to use Varnish with a high TTL and purging when content changes.
Some good comments here
You might think about caching strategy as well. You could cache articles permanently and only invalidate the cache if the article is updated.
This might take a bit of coding with hook_node_update and calls to cache_clear_all but would prevent the server from having to process articles once they are cached.
Typically caching in Drupal is set for a ttl time of 10 minutes one hour or some such. The above strategy lets you cache permanently until there is an update event which can really speed things up. The caching backend could be Varnish, memcache, Redis - almost doesn't matter.
You could even set it up that upon saving a new article or updating an exisiting one you insert the article into the cache as well - this way everything is always cached.
Hits / Vistors ... concurrent verse per day?
@jaydul
Just wondering if you could give a little bit more info about what you are expecting in terms of hits and concurrent visitors?
You've mentioned posting 1m articles a day ... did I read that right?
Honestly, going with
Honestly, going with something like Varnish or NGinx + memcache is the way to go. If you're going to have 1m+ articles, it's going to take a lot of work and time to generate all those static pages. Maintenance also becomes a huge issue. What happens if you want to put ad codes on all your pages, or even change something in the template? You're going to have to regenerate everything.
1M+ rows for MySQL is not huge by any means. If you get a properly tuned stack, you'll be amazed what can be done dynamically, including dynamic caching solutions. I run a personal site on a Linode 2GB ($20/month) server and got hit a couple years back with something being on the front page of Reddit. I was getting over 40,000 page views an hour and the server never broke a sweat. That was also back when that package on Linode was the 512mb, so I had 1/4 the memory and only 1 core instead of 2, and not on SSD (Stack wise, I was running Apache+Varnish. I've since moved to NGinx, so have even better performance now).
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
After read Spleshka & Jamie I
After read Spleshka & Jamie I m now thinking use dynamic site with dynamic caching solutions.
Xeon E3 1230 Server with 16 GB RAM 2x80 GB SSD HDD I m now using liquid web server with cpanel.they install Varnish.I will move
Nginx+PHP-FPM+Memcache+Memcache_storage module.
Main point is Database what Database We can use.
Can you recommended MySQL mariaDB or mangoDB(Note:cpanel only allow MYSQL).Also need to know about
Our news portal have a bulk upload system Every our We upload 10,000 news over 20 category.
I want best cache system where only fast visitor hit database then all visitor via cache.If just post new news it will dynamic upload.Here is any Caching System can do that?
Please recommend
Thank you
Jaydul
In your case you can use any
In your case you can use any database. MySQL, MariaDB and MongoDB are designed to store huge amount of data, so you won't meet the limits. I can see you are not going to stress your database, because most of requests will be delievered by nginx from memcached. That means you shouldn't worry about database performance so much. Just use MySQL 5.6 (as Mike recommended) and make sure you have enough memory for database. That's it.
Normally any database is able to handle 10,000 inserts per hour easily. But if you will expirience problems, consider using bulk inserts. For example insert 1,000 rows per one insert request. In this case you should care about reliability of inserts and catch possible problems during inserts - otherwise you may loose your inserted data.
We are now using those
We are now using those module.I want use Minimal Module so site can be Blazing Fast please let me know what can must use and add some recommend Disable module (Database Logging) like and add module memcached
Dashboard
Database Logging
Field
Field SQL storage
Field UI
File
Filter
Image
List
Menu
Node
Number
Options
Path
RDF
Search
System
Taxonomy
Text
Toolbar
User
Pathauto
Block Class
Blog
Contextual links
Disqus
Entity tokens
Facebook Likebox
Field UI
Forum
Google Analytics
jCarousel
jQuery plugin
jQuery Update
List
Newsletter
Number
Poll
Quicktabs Styles
ShareThis
TagClouds
Taxonomy menu
Update manager
Views Slideshow: Cycle
Views UI
Thank you
Adding my 2 cents here, but
Adding my 2 cents here, but it seems all of this is a bit new for you. The big variable here isn't necessarily the size of your db, but rather your traffic volume. If you'll have some decent traffic numbers, I highly suggest going the cloud way with a preconfigured hosting environment. It will work a lot better for you, as building a platform from scratch isn't an easy thing, especially if you're asking yourself if you should use Boost, Varnish or Nginx.
We already have site with
We already have site with large visitor Is has 40,000-50,000/hour visitor in and millions of content.We are using own CMS all file create with all static page when need upload new news need to start from resume and it make all static file from and upload it upload as day keep file day by day.Thinking to move drupal from here.Because many big site running with drupal.MY CMS Run with a Cloud no problem but no SEO no live editing.
I want run my site Dedicated server with Drupal
Thank you
bulk Import XML feeds
bulk Import XML feeds -Content from xml systems category base
Hello I have custom CMS we are moving to Drupal.Our custom CMS has bulk post system via XML feed.
We post 1000 articles/News via xml per hour on our CMS.
We need to know how to import bulk xml as Categories (Taxonomy) base.
Here is our xml file below
<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel>
<Category>Sports</Category>
<title>Title</title>
<description>Image Location</description>
<item>
<title>Bulgaria votes in parliamentary poll</title>
<description>Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Lorem ipsum dolor sit amet, consectetur adipisici elit, sed eiusmod tempor incidunt ut labore et dolore magna aliqua. Lorem ipsum dolor sit amet. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi.Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. At vero eos et accusam et justo duo dolores et ea rebum. Sanctus sea sed takimata ut vero voluptua. Consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat.</description>
<alt>tag1,tag2,tag3</alt>
<pubDate>2/15/2015 8:26:01 AM</pubDate>
</item>
<item>
<Category>Sports</Category>
<title>Youssou Ndour awarded Polar Prize</title>
<description>At vero eos et accusam et justo duo dolores et ea rebum. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. Excepteur sint obcaecat cupiditat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Sanctus sea sed takimata ut vero voluptua.Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi.</description>
<alt>tag1,tag2,tag3</alt>
<pubDate>2/15/2015 8:26:01 AM</pubDate>
</item>
</channel></rss>