Migration of large amount of data - worried about `node` overhead and performance

Events happening in the community are now at Drupal community events on www.drupal.org.
Mike_Waters's picture

I need to write a quick web frontend for an existing gui app that just displays information about a large quantity of image data; this data is in a myslq db currently. I would like to use Drupal if I can.

The image data model would work fine translated as nodes, I am just worried about query performance due to the overhead of storing this data as nodes; there will be 250K and a million images.

Additionally affecting performance, I believe I will need to use two content types for each image - one for static data (ex. 'name'), one for dynamic data (project-specific stuff), with a nodereference from static to most recent revision of the dynamic.
However, the traffic will not be high at all, maybe 50 users, and I have a decent 2U server with lots of ram and a T1.

Is it worth it for me to put a migration scheme into place to script the existing data into the drupal db as nodes, or should I just write a web service to connect to the existing database and not use drupal's content system at all (just use it for it's infrastructure, and spit the query results into $page or whatever)? Or is Drupal not suited for this at all, and I should go learn django and stop bothering you? ;)

As I write this, I realize that my question could be distilled into this:
"Will drupal's stable infrastructure, caching, and ease of development make up for the overhead that the content system will create when a huge amount of data is mapped/stuffed into a non-simplistic content type structure?"

On a side note, I wonder, now that Drupal is gaining popularity, if some ORM code will pop up to help people use Drupal in this manner.

Thanks for your help!

Comments

to be or not to be :)

Alexander Kosarev's picture

I think, that you as developer need to read more drupal documentation, specially for modules API.

Also you need to review popular Drupal modules, like views, cck and cache related. Modules can make your job extremely easy.

Is Drupal suitable or not - is your decision. I am sure peoples will have different answers for your questions, but for me you ask because you are not sure. I use Drupal since version 4 and had build very different application based on Drupal.

As example for "complicate" situation:
- module define new node type and attach data to base node from FireBird database. New nodes create via cron job based on information in FireBird database.
- data from pervasive database used for views to show this nodes

Sure I miss "details", but I used Drupal as platform in many projects and I am sure that I will continue to use it - it is my decision

Thanks Alexandr, I am

Mike_Waters's picture

Thanks Alexandr, I am familiar with the docs.
I suppose that the reason I asked in this particular group is to draw on the experience of others who may have used Drupal for this purpose, as somewhat of a relational-object-mapper, fitting an existing object/data model into Drupal's content structure and experiencing success even on a large scale.
This is using Drupal as a platform, not just as what is referred to as a 'CMS'.

@Mike_Waters your use case

dalin's picture

@Mike_Waters your use case seems like something that Views would work well with. I'm interpreting your requirements as "We have data entered into this other application, and I'd like to use Drupal to do lists of images, reporting, etc." The Views module has the ability to perform queries against any SQL table, not just Drupal tables. So you would write a small chunk of code to tell Views about your tables and then you have all the power of Views and its various extensions to do some powerful things quite quickly.

--


Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his

That's not a bad idea, I

Mike_Waters's picture

That's not a bad idea, I hadn't considered using Views against my Db.
One thing I was hoping to be able to leverage was the Drupal cache... I suppose there's always memcached

This is a classic post on

kyle_mathews's picture

This is a classic post on importing/accessing remote data with Drupal from Larry Garfield:
http://www.palantir.net/blog/remote-data-drupal-museums-and-web-2009

Kyle Mathews

Wow, thank you. That article

Mike_Waters's picture

Wow, thank you. That article has really turned me in the right direction.

High performance

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: