high scalability

I am working on a web application which will primarily be used by authenticated users. The web application imports records from CSV files using the Feeds module, MongoDB was chosen to store this data because it is much faster to do the writes rather than in MySQL. The application is importing thousands of records every day really fast. The records are referenced to the users and when they log in, they can consult their information. The application is also used as an API for the mobile app.

I have a 17gb AWS EC2 Instance as the server.

We frequently use GlusterFS ( http://www.gluster.org ) in load balanced clusters to synchronise the /files directory between the web heads. That works great but we haven't found (or investigated fully) a method that the Boost module could be used easily alongside? I'm hoping someone's already doing it and can provide a proven working example.

Boost in this scenario is being used to complement memcache and standard page caching to provide static page delivery for high volumes of anonymous users, open to other equivalent suggestions appropriate to the environment too.

Hi,
I have a question about scalability issue of buddylist. Suppose I have a site with 1 million or more users and admin(or any user) have near about 200000 buddies or more. My question is can buddylist module handle the access of buddylist of admin specially when buddylist calls the function buddylist_get_buddies and execute a query :

$sql = 'SELECT b.buddy, u.name, u.mail, u.uid FROM {buddylist} b
INNER JOIN {users} u ON b.buddy = u.uid
WHERE b.uid = %d';

I'm new to the High Performance group and I'm interested to find out if people here have run a site with the following stats:

• 4,000 simultaneous visitors, up to 500 who are logged in at once
• approaching 2 million comments
• 200,000 forum posts
• 30,000 registered users

we're mostly friendly. and mostly fun. but we're pretty geeky. in a cool way, of course. but this month we're a bit melancholy on the news Crystal is leaving us for Seattle. :-/

the good news is that we taped our meeting. ya, real geeky. but LA area is HUGE. and sometimes making it to the meeting in person can be difficult. so there you go... we're geeky but thinking about everyone that wanted to join us, but couldn't.

As information over the Internet becomes more and more revolved around communities and as the need for fast, dynamic content increases, simple caching can't do it anymore. In order to really stand out, a new application logic layer, which would interface with the database abstraction layer and provide tools for easy database sharding would be great. The idea is to be able to split data across several tables, but at the same time keep the caching features and the code intact(meaning that in order to use sharding features, one would only have to plot some rules for the data logic and that's it, no need to redo parts of the code or anything). The idea is inspired by the hibernate sharding feature, and by the ever increasing need for scalability.
Later edit: The nicest thing about sharding is that you can have parts of the data in different databases, spread across multiple servers.