We are in the process of using drupal with our new game server, this server will read and write a lot of the users game data, We have chosen to go with the basic pressflow and minimal modules enabled, some custom ones that work with the services api (its how we communicate with the client and server with JSON).
Someone mentioned to me that and im still trying to figure out the formula of well with a pure drupal site optimized you can only have so many concurrent users hitting the db before apache/php bottle neck comes in.
Btw is their a good formula or a way to test what you can push before apache/php start to eat up the memory and start to get close to blowing up or responding slowly?
Someone mentioned using a event driven framework such as node.js or twisted could end up taking up some of the heavy lifting of hundreds of thousands concurrent users to another db while working along side with Drupal.
Anyhow anyone have experience of off loading some load on Drupal with another framework?
Comments
Can't
speak of frameworks. AFAIK nodejs aside there's nothing that can be pigeonholed as such. But there are many other open options. A very idiosyncratic short list:
Using ZeroMQ.
Doing a light bootstrap of the site and using the above or Lua with Nginx to talk directly to the DB.
Any of the above using a key-value storage.
Just random thoughts.
Nginx directly with Redis
http://wiki.nginx.org/HttpRedis2Module
Sure
Also any type of microcaching for authenticated users may alleviate authenticated user load.
Cheers guys.. here is what we
Cheers guys.. here is what we are forecasting so maybe this gets into a bit more detail but I would like to try to get as informed as possible and potentially work with someone to make sure we can scale.
We expect to have somewhere in the order of 5 million daily users (DAU), users will be logging into our game client iOS, Android and other at various times a day doing various things in the game, the communications layer im using right now is services with json and this seems to work well in our small tests client and server are working together.
Things that will happen during a users game session.
Get the users latest game info and player state data
The user might buy or sell various items during this session
The user might complete a few missions or other tasks
So with potentially having 5 million people doing this daily on the server I wanted to make sure that we could scale and drupal can handle this if not off load the most extensive stuff to another lightweight server but keep the drupal end for user auth, and some service calls that wont be a big hit each time they log in.
Granted we dont have a huge team of web developers so we will probably go into the AWS system with ec2 , load balancing etc, the db side of things not quite sure on and thats my next biggest fear, RDS i heard is slow, the new AWS DynamoDB which is a nosql db, running on ssd drives and such.. http://aws.amazon.com/dynamodb/
anyhow I would appreciate any help with this and if anyone is looking for contract work we will be needing some additional help.
cheers
Get a developer, start architecting, start benchmarking
You should probably prototype a few different options and benchmark them as a first step. Your target isn't 5M daily users, it's whatever the peak throughput per-second is. Calculate that and work towards it.
Find out what kind of requests per second your current Drupal setup can push. Find out how many reads vs writes. Find out how much you can cache.
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Agree you need some numbers
I agree with Josh that you need to take some numbers. At a minimum, go with the rule of thumb that for this type of application the database will be the bottleneck (because you can add webservers that all connect to the same database), and find out how many queries a typical few minutes of game interaction causes. Multiply that by your expected use and try to estimate if you are in the ballpark of being able to handle it.
The other comments here about using bare metal servers and SSDs for the database are also good.
I would certainly run away
I would certainly run away from any of the Amazon services.
If you need scalability, have a decent architecture team. But Drupal can certainly handle this with proper management and architecture.
Again, IMHO, I'd run away from these shiny "cloud" servers. They're worth nothing.
are you suggesting using bare
are you suggesting using bare metal servers, and unfortunately we dont have a huge team for this and well to be frank non of us had to tackle something like this.. hence the need to also hire someone to help us out.
Get help
If nobody on your team has experience, get help before spending any money.
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Bare metal servers with
Bare metal servers with proper scalability techniques like DB sharding, memcache clustering, etc.
You need to find a good provider. Don't go with Amazon or Rackspace, they're just too expensive for what they offer.
But yes, you need someone to help you out with this. This will be critical for your business since you're expecting to get millions of users/day.
Drupal or not, you're gonna need someone capable.
Nothings beats
bare metal. VMs will give you ease of deployment at the cost of an intermediate layer. If you're planning that type of traffic. You need to benchmark and plan. That won't guarantee that every thing will be smooth sailing, but it will allow you to endure rough sea with confidence.
Going for a centralized service like EC2 or Rackspace or anything in that line might be appealing and might work most of the time. But if you have a mission critical system that's not very wise since centralization introduces additional fragility.
Remember that Amazon's ecommerce business logic is not on EC2 but rather on a dedicated infrastructure AFAIK. I would suggest to have EC2 as a failover system never as primary if uptime is critical for your site.
Redundancy is the name of the game.
Also you don't need a huge team, you need a competent team. A well setup infrastructure almost runs itself once up and running.
Don't Rule Out the Cloud
If time is not a factor, then it is arguable that nothing beats barebones. Unfortunately in most scenarios this is not the case. I am a long-time fan of Rackspace's Cloud Servers, but also have extensive experience architecting bare metal based infrastructures. A system based on cloud servers is scalable both horizontally and vertically. Vertical scaling with cloud servers is virtually seamless while horizontal scaling on the cloud has very low risk - you can fire up new servers starting at $11/mo, practically a negligible cost.
Thanks to the tools at Rackspace (and Amazon?), redundancy is not an issue if you know what you are doing. Cloud-based infrastructures are capability of supporting high-availability operations.
Perusio is correct in terms of how time is allotted to these projects - they're very front-loaded. Regardless of what form of infrastructure you use, the vast majority of the work will be upfront. This why companies like Acquia are able to flourish. You work with them to build a system the right way from the ground up and they give you (your staff) the means to maintain and extend it. Contacting Acquia would probably be the most useful thing you could do in your situation, at the very least to just get an educated opinion of your specific situation.
The only thing bare metal gets you that the cloud does not is absolute assurance of how much resources you have at your disposal at any given moment. This comes at a great price when compared to alternatives - it's up to you whether or not it's worth it.
Make sure you do your homework and talk (verbally) to the right people (Acquia) before spending any money.
Regards,
Ben Koren
Rochester MN Web Design