Hello,
We are developing a drupal site that needs to logs users activity from an external source. Imagine thousands (more than 800,000) users.
I was thinking of using XMPP not for a chat client but for data synchronization. In other words: An external source (separate from the drupal implementation) sending data about the user's activity to drupal, and drupal logging all this data (using the heartbeat module). My first approach was to use Atom feeds and consume them but the system wont scale that way (imagine trying to pull thousands of feeds on a cron run, even with the batch implementation). So I want to give it a try using XMPP.
What do you think? Any suggestions based on your experience that may help me starting this project?
Comments
use database layer
Although I am no expert in Drupal, my past experience with data transfer indicates that data feed in batch mode of this magnitude should be done at the database layer (Mysql), not at the application layer (Drupal).
If the data traverses the internet, then it is best for the data to be deposited in a staging area (by FTP?) and then for an ETL-type script to transform the data and load directly into the database itself.