A "no-database" option (using Flat Files)

ddmdllt's picture

Hi,

I'd like to present an idea for the 2008 google summer of code (and in a more general way an idea for Drupal).
The idea is about using flat files instead of a database. After googling a little, it seems that the idea is not new (http://drupal.org/node/124353 and http://drupal.org/node/216485). However, it seems that no visible results are here now.

Implementing a flat file system for drupal is not an easy thing. In particular what I mean is that this could obviously lead to a loss of performance for users choosing the flat file system. So, I suppose the best to do is to remind some of the advantages of a flat file system:
* It is easier to use for the final user : no need to create a database (though this is not a hard task, for some people an easier system would be better; one reason of the popularity of CMS are that they are easy to use).
* Some people don't have a hosting service providing a correct mysql version (or providing no sql at all), and they may prefer not to invest money in a better hosting service. Sometimes they may have problem with database quotas too... One example of hosting service that does not provide mysql is the 20GP offer from ovh for French residents (you can see yourself if you are ok with French at http://www.ovh.com/fr/particulier/produits/20gp.xml).

If I suggest this idea, it is because I really think this is something sensible. Some quite popular projects (like pmwiki, for example) do use a flat file system. Although they were thought from the beginning with the flat file model in the head, they clearly shows that a "no database" solution is something that we could at least think about.

I think there is a page that should be read before discussing about the possibility of a flat file model in Drupal: it is http://www.pmwiki.org/wiki/PmWiki/FlatFileAdvantages from pmwiki (it contains a lot of informations, arguments, ideas about results). In particular if you think a flat file system for Drupal is impossible, you really should read this page.

A final note about performance: it is always possible to think about a Drupal version with some functionalities disabled (like internal search, even if it is useful to have something always up-to-date for searches)

For the rest, and about the way Drupal was coded, I don't think that it is necessary an easy job to implement a flat file system. I just think it would be both interesting and useful.

Any feedback is welcome.

Regards,

David Dallet (ddmdllt)

Comments

Flat file == SQLite

jpetso's picture

Drupal depends so much on database stuff that not using SQL is simply not an option. Given a lot of work, one might be able to add "write to a flat file" code as alternative to SQL statements in Drupal core, but that's neither maintainable nor has it any chance to work with most (or even many) modules in contrib.

So the only feasible option to get flat file support is having an SQL database that writes and reads to flat files, and it's most likely this database is SQLite. (Never heard about the Gladius DB that's mentioned in the second link, but it doesn't seem to have a lot of development activity and will likely be seriously slower than SQLite.) Thus, any sustainable effort to get flat file support in Drupal needs two things:

  • An SQLite database backend, and
  • Compatibility of the used SQL queries to SQLite. That has been made easier with the Schema API in Drupal 6, but might still not cover all cases.

Performance might be an issue, but we'll never know without trying. Anyways, I was surprised to see a "flat file" proposal without even one single mention of SQLite.

Re: Flat file == SQLite

ddmdllt's picture

Hi and thanks for your feedback.

The problem I see with SQLite is that it uses C (http://www.sqlite.org/selfcontained.html). Even if ANSI C is not a really exaggerated requirement in general, I think it is to much for the people I planned to target. The real target of my proposal is all people that need one thing they just need to upload on a hosting service just supporting PHP (though it may help people not knowing how to create a database).

Although, if there are other real reasons to use flat files that the reasons I thought about, SQLite may be a solution in this case, but the fact is that I did my proposal for some reasons I'm able to see, not just for using flat files.

If I quickly raised the attention to Gladius DB, it's not in the goal to use it. In fact I wanted to point that using flat files has been proposed before (even if I found this post because I was thinking about using flat files for Drupal, it's not the post speaking about Gladius DB that suggest me the proposal).

For me writing alternative to SQL statements does not really means take each line and then write a flat file replacement in PHP. Even if it could be necessary in some places, I think it would be better to write some functions able to deal with many cases, and then to use it in places where it helps.

As you pointed that's right it would be hardly maintainable if we want to keep the full power of modules. I think it could be more feasible by choosing a restricted set of features supported by the "flat file version" of Drupal, even if it the whole project would appear not as nice as a full featured standard version of Drupal.

But the idea was almost about making something working when it couldn't work otherwise (i.e. in some quite limited hosting services or for final users with a quite limited knowledge and a quite limited ability to get the relevant documentation).

Hoping that my proposal is a little more understandable,

Regards,

David Dallet

PHP5 and SQLite

kbahey's picture

Just flat files can have some use cases (e.g. serverless environment).

But, I think SQLite will be a far more useful option. It is already bundled in PHP5, so no need to muck around with C if that is your concern.

If you go the SQLite route, one can have a complete development environment or training environment for Drupal without having to have MySQL installed, nor even knowledge on how to handle it.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

php5.2 basically means sqlite is always supported

greggles's picture

hosting service just supporting PHP

If a hosting service "just supports php5.2" then you can be fairly certain that they will support sqlite as well. So, I don't think that going to flat file is the best way. Instead perhaps working on the sqlite support would be good (though chx is already planning to do that, I believe...). Given that this code would go into Drupal7 at the earliest and given that Drupal7 will require PHP5.2, I don't see a lot of value in providing a flat file port even if it were possible/reasonable for other reasons.

--
Open Prediction Markets | Drupal Dashboard

Re : php5.2 basically means sqlite is always supported

ddmdllt's picture

The fact is that some hosting service does not necessarily don't provide some features because they are not up-to-date but just propose different hosting plans (at different rates).

Some hosting services providers have for example a very low cost solution with no database and some feature voluntarily disabled (probably just because they provide more expensive solutions). These hosting services may not represent a large part of all hosting services but they exists and may have very cheap offers.

Something even more frequent is hosting services providing a quite acceptable space for standard files and providing a MySQL database but with a very small quotas. If it is possible to provide a flat file system that is still quite good in term of efficiency allowing to contain more data that such quotas, it may still be possible to make something justifying the effort needed for development. Moreover, the user may want to keep available place in his/her database in order to be able to use other applications requiring space in the database.

I have to admit that I wouldn't be the ideal target of such hosting plans: I already have a quite good hosting service (not the cheaper but still quite good one rather oriented for businesses), and if I would like to cut down the prices I could always use the multi ftp (and ssh) account feature to share safely the service with friends, but for some (non geek) people, cheap hosting plans may be the choice (and they may think only after about the consequences of choosing something without a database).

That was my point of view about the interest of a flat file support without SQLite. After, one think to really think about twice before is the set of consequences for code maintenance that could come with such flat file support. Even if the job is done in a proper way, it would still depend on how often changes are made to the existing base, what are the expectations for the flat file version (i.e. adding each new feature as soon as it appears or choosing to have a quite limited version), etc...

David

This is not suitable for an SoC task

chx's picture

SQL implementations in a flat file are readily available and writing a driver is not hard. Maintaing it, making sure all core works with as core progresses is the challenge.

Edit: http://www.c-worker.ch/txtdbapi/index_eng.php http://www.txtsql.com/index.php http://gladius.sourceforge.net/ http://sourceforge.net/projects/fsql and I bet there are even more.