Is drupal suitable for bigdata project

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
yuseferi's picture

I'm a pro drupal developer, I'm a project that is something like facebook ,there is huge data here( 10,000,000,000,000 graph node, data stored in graphical database, in current state our team used Laravel as a middle-ware layer, I mean handle some feature and task with Laravel and mongodb and data getting fetch from graph database, I researching about is that drupal8 suitable solution for projects like this? or it's better instead of using drupal we keeping on on our laravel solution, in current solution laravel is middle ware that fetch data from graphdatabase and in Front-end layer we use Reactjs that fetch data from middle-ware with REST, user management,ACL and etc handled by REACT and REST and LARVEL. appreciate any help or your idea and opinion about this situation,

Comments

Graph database

miromarchi's picture

Very interesting. I'm a big fan of Drupal and also of graph databases. I believe you will be better off using one of those. Have you had a look at neo4j connector drupal module to use neo4j graph db as a search engine?
https://www.drupal.org/project/neo4j_connector

Or you could leverage the graphaware neo4j php client yourself https://github.com/graphaware/neo4j-php-client

Also, if you want to use a totally different approach you could try structr graph cms, built on neo4j.
https://structr.org

Graph database

miromarchi's picture

Very interesting. I'm a big fan of Drupal and also of graph databases. I believe you will be better off using one of those. Have you had a look at neo4j connector drupal module to use neo4j graph db as a search engine?
https://www.drupal.org/project/neo4j_connector

Or you could leverage the graphaware neo4j php client yourself https://github.com/graphaware/neo4j-php-client

Also, if you want to use a totally different approach you could try structr graph cms, built on neo4j.
https://structr.org

Thanks for your attention,but

yuseferi's picture

Thanks for your attention,but I want using drupal 8 and also we didn't use jeo4j,anyway , can you tell me the true method structure of how handle big data with drupal, as example
all content except user manangment and ACL handled with drupal, create a middleware layer to get data from grpah database pass it to front,I don't know what is the true solution and instruction for achieve our job.
I write down my idea in following but I don't know is it correct,

our story is "we want create a website like facebook,user can share status, video, audio,articles etc, also we have a big data in graph database, we should save user posts in graph data ( so we need a middleware layer to pass data to graph) and also fetch data from graph ( so we need another middleware to fetch data)", what method we should use? which database is suitable for us ? postgres,mongo,redis, ...,
our website designed for number of huge visitor per day .
what is your suggestion ?

suggestions

martin_andersn's picture

mongoDB with Elasticsearch ?

https://www.drupal.org/project/mongodb
https://www.drupal.org/project/elasticsearch_connector

(processing.js is could be looked into to as a way to go for some of the graph database interactions? should revist the above comments about neo4js module )

//*
maybe idealy it sounds like you probbly be better of with a combo of:

Cassandra
(Java, Store huge datasets in "almost" SQL, Apache, CQL3)
When you need to store data so huge that it doesn't fit on server, but still want a familiar interaction with it.
http://cassandra.apache.org/

+

ElasticSearch
(Java, Advanced Search, Apache, JSON over HTTP, memcached)
When you have objects with (flexible) fields, and you need "advanced search" functionality.

eks:dating service that handles human interaction, geographic location, thumbs up and downs, etc. Or a leaderboard system that depends on many variables.

https://wiki.apache.org/cassandra/API
OrientDB: For graph-style, rich or complex, interconnected data.
Java
Document-based graph database
User functions in SQL or JavaScrip

MongoDB

darrell_ulm's picture

Following up with a question instead of solution.

Is this indicating something like the MongoDB connector?
https://www.drupal.org/project/mongodb

Also a thread here: https://groups.drupal.org/node/286763
and I'm interested in Big Data / Hadoop / Apache Spark tie-ins to Drupal as well, so if anyone has some more info, would also like to see it.

Big Data Drupal

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: