Just in the process of planning a web project with a lot of fields the mongoDB integration came back to my attention.
When people speak about the integration they tend to point to a massive performance boost when writing and reading. But what is the real world factor here ?
Even more important is the data access. I know there is the EntityFieldQuery which give a way to integrate with Views. But is this production proofed; how does this work in a real world application ?
Next is search ! MongoDB has its interesting (and pretty much SQL near search-syntax); is this been used somehow ? The only thing I found is the a mongodb-backend for the Search API or is this fully done by EntityFieldQuery at the end ?
Hope to find a bit of light in the darkness; any help appreciated.
Comments
Can you quantify "a lot of
Can you quantify "a lot of fields"? I'm the development lead on https://www.dixonvalve.com which uses a MongoDB based backend. We use a few pieces of the MongoDB module. In particular, we use the queue backend, the session backend, the caching backend, and the field storage backend with much success. That said, we have hundreds of thousands of entities with 200+ fields attached to them (which is why we chose MongoDB in the first place). I don't have comparison numbers between a straight SQL backend and a MongoDB backend but everyone has been happy with the performance of a MongoDB backend. I will say that loading an entity with 200+ fields only requires 1 MongoDB query whereas loading that field from a SQL backend would require 200+ (or A LOT of joins).
EntityFieldQuery is fairly robust. However, the Views integration is challenging. We ended up pretty much abandoning Views and just doing everything in code with EntityFieldQuery. I believe this is the approach other large sites that use MongoDB have taken as well but I don't want to speak for them. The only "gotcha" with EntityFieldQuery is that there is no way to join a condition with an "OR" clause. We end up using a straight Mongo query at that point (which some of my developers hate but I equate it to doing a straight SQL query which they don't mind). Doing a straight Mongo query is unfortunate but not the worst thing I've seen in a Drupal site.
In terms of search, we pretty much dropped the Apache SOLR module in and let it run without any issues. However, the more complex search (product search) was ported from a different system so it is largely custom SOLR queries with a bit of EFQ.
Getting down to the million dollar question, you need to seriously look at your requirements to see if MongoDB is the right choice. Is a large portion of your data relational? If so, MongoDB is going to be a bad choice because it has no way to do relationships (you end up doing them in code rather than joining in a database). How many fields will you really end up with? Are you prepared to spend extra time developing because you need to use EntityFieldQuery instead of Views (although, after using EFQ for a bit my team has gotten just as fast at writing EFQ as they have with creating Views)?
Let me know if you have any other questions. I'm glad to share as much knowledge as I can.
yes, MongoDB+Drupal is production-ready
The last site I worked on w/ MongoDB used sessions, watchdog, and field_storage and did the EFQ Views route. I worked w/ Mike Crittenden on the project, and he wrote a post about MongoDB which covers Views in pretty good detail--http://mikecr.it/ramblings/converting-drupal-data-to-mongo . On this project, we already had Apache Solr set up on the D6 site and continued to use it on the D7 site (nothing fancy there though). I know we had to write some custom code to get things to work w/ MongoDB w/ the client's setup; but it was very minimal.
This particular site did not have a ton of node types or fields, but LOTS of traffic (anonymous and authenticated) and a good amount of nodes. We were primarily concerned w/ sessions (Memcache didn't look like a viable D7 solution for them) and getting load off of the MySQL databases.
I think it goes without saying that benchmarks can be deceiving... I only benchmarked one specific aspect of the site--a counter that was going to be hit on most requests; so it used a very minimal bootstrap and ran an $inc--it was very fast and very lightweight. I did not do a benchmark against a MySQL or Memcache alternative, but I believe they would have required two ops (read and write). MongoDB has some clever tricks like this that do not show up in benchmarks. I'm not sure you can get away with anything other than a full implementation in comparing them--both systems will cache data (and Drupal caches data all over the place too); so running a PHP SQL statement vs a MongoDB one is not going to illustrate how the systems will behave in production w/ cold caches or warm caches or under load or a large dataset or a diverse one.
A couple of things we saw w/ MongoDB:
1) authentication was disabled by default--I haven't installed it recently, so I'm not sure if it is still the default tho
2) watch the space/storage. I believe you need double the space of the database to enable "repairing"/reclaiming deleted space--so if you wait until the drive is half full, it's already too late.
You may want to look at pinging Moshe or chx in IRC. I know Moshe did at least some benchmarks w/ MongoDB and chx has really been a driving force in getting MongoDB into the Drupal community. They are also nice guys :) And on this front, I would consider hiring a consultant to either do a feasibility study or plan the implementation.
Cheers,
tim
PS I thought this was an interesting read-- http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/. I am not endorsing the conclusions, but it does underscore that there are tradeoffs, design considerations, good use cases and bad ones.
Loudon & Company Consulting
Prepare for a lot of work
We had a requirement to feed out a lot of JSON data out and decided originally to do it with custom entities, MongoDB field storage and Views, through EFQViews.
Well, EFQViews is, as rvtraveller said, 'challenging', meaning you can't really do a whole lot with it. There's a lot of problems out there and even though our team patched some of those modules along the way, the end result was still that we could maybe serve the JSON out, but couldn't use Views for any administration views.
The JSON output, even on MongoDB, proved very slow, due to Drupal touching and fiddling with every field going out (e.g. for theming) and thus if proved unsatisfactory performance-wise. We ended up indexing out data to an external MongoDB in usable form and serving that out with a Node.JS application. That setup is very, very fast.
I'd say use MongoDB field storage only if you're ready to cope with limited Views and a lot of manual work with queries. If you need a lot of Views and helper modules for those, skip the MongoDB.
Eventually we ditched the MongoDB field storage and went with regular MySQL storage, since we were able to move all the high-traffic stuff outside to the Node.JS stack.
MongoDB field storage might be production ready, but prepare for a fight get all your stuff working. EFQViews isn't ready, at least to the extent you'd expect it to be.