Oracle Support in Full Gear

Note:I welcome others to join the effort and I will provide some tablespace on an Oracle 10g database server to test and implement an Oracle driver for Drupal upon request

I have begun Oracle Support in full gear. After this, I plan to do the same with MS-SQL. I shall start to release my findings here. I am starting with some code from this ( http://drupal.org/node/39260 ) drupal thread. I'm testing against Oracle 10g.

I have installed Drupal 4.7.3, and I am using the database.oracle.inc, and database.oracle.sql very well. I have full funcitnality thus far.

My next plan is to start enabling modules, and start an internal _db_oracle_rewrite function in the database.oracle.inc file to format any MySQL specific SQL statments to Oracle sepcific statements.

I then plan on making a _db_oracle_optimize function that checks sql statments for where certain Oracle specific code can be used to help speed things up (such as binding when you have simple quries like:
SELECT * FROM users WHERE user = 1
SELECT * FROM users WHERE uid = 381
SELECT * FROM users where uid = 19493

My list of modules I'm going to test are as follows (I tried to think of what most people use, and a few complicated ones):

  • upload.module
  • stats.module
  • tracker.module
  • taxonomy.module
  • events.module
  • views.module
  • og.module (and a few related og modules)
  • book.module
  • urlalias.module
  • search.module
  • throttle.module
  • profile.module
  • comment.module
  • contact.module
  • block.mdoule
  • aggregator.module
  • devel.module

If anyone can think of more, please comment :).
Comments are welcome.

Groups:
Login to post comments

First Quirk, Paging

Souvent22 - Wed, 2006-08-23 05:34

The first problem I'm getting is in paging. Examine this query:

SELECT sub."uid" FROM(
SELECT w.*, u.name, w."uid"
FROM watchdog w INNER JOIN users u ON w."uid" = u."uid" ORDER BY w.wid DESC) sub

If we didn't use reserved words, this would/does run fine. The problem is, you get a "Abiguously Defiend Column" error. I am still looking into why. However, this query does work:

SELECT sub."uid" FROM(
SELECT w.*, u.name
FROM watchdog w INNER JOIN users u ON w."uid" = u."uid" ORDER BY w.wid DESC) sub

Notice that the reserved work is used on the outer Select. Anyone think of a good workaround?


watchdog sql

Souvent22 - Wed, 2006-08-23 06:10

Fixed by fixing the watchdog sql statmenet. watchdog table already has uid in it, thus the ambigoius statement.

Line: 111
$sql = "SELECT w.*, u.name, u.uid FROM {watchdog} w INNER JOIN {users} u ON w.uid = u.uid";

to: $sql = "SELECT w.*, u.name FROM {watchdog} w INNER JOIN {users} u ON w.uid = u.uid";


HEAD

moshe weitzman's picture
moshe weitzman - Wed, 2006-08-23 12:27

i strongly recommend you use HEAD as your basis. The SQL files were all broken up by module. You will have smaller chunks to work with. Also, your patches will be suitable for review by our committers. They don't look at 4.7 only patches.

Great that you are doing this and MSSQL. Thanks.


HEAD

Souvent22 - Wed, 2006-08-23 17:56

Moshe,

Good Idea. I will start to do that.


CLOB Discussion

Souvent22 - Wed, 2006-08-23 18:00

Ok, it's time to start the dreaded CLOB discussion.

For a brief and high-level overview, CLOB is Oracle's equivlent to TEXT (both 4GB long).
But, CLOB has some, what they call 'minor' cavets. There's only 1 which really effects Drupal:
1. You can't have a CLOB in a SELECT DISTINCT..SELECT UNIQUE, GROUP BY, or WHERE clause..

So, this causes isses for thing like the comment module. So, anyone know of an Oracle GURU who could
give some insight as to how to progrmatically alter SQL statements so that we can use CLOB's?

Here's an example statement. c."comment" is a CLOB, this will return in error in Oracle:

SELECT sub.*, rownum AS line FROM (
            SELECT c.cid as cid, c.pid, c.nid, c.subject, c."comment", c.format,
            c.timestamp, c.name, c.mail, c.homepage, u."uid", u.name AS registered_name,
            u.picture, u.data, c.score, c.users, c.thread, c.status
           FROM comments c INNER JOIN users u ON c."uid" = u."uid"
           WHERE c.nid = 1 GROUP BY c.cid, c.pid, c.nid, c.subject, c."comment", c.format, c.timestamp,
            c.name, c.mail, u.picture, c.homepage, u."uid", u.name, u.picture, u.data, c.score, c.users, c.thread, c.status
             ORDER BY c.thread DESC


Possible CLOB wormhole?

bshensky's picture
bshensky - Fri, 2007-07-27 14:48

PL/SQL limits character strings to 32k in length. You could write a lightweight PL/SQL function that encapsulates the CLOB. You should be able to use the function in conjunction with DISTINCT, UNIQUE and/or GROUP BY features.

Just my initial knee-jerk reaction to the problem...

-b


it should no longer a

hswong3i's picture
hswong3i - Fri, 2007-07-27 18:08

it should no longer a problem: within latest version of oracle driver implementation (http://drupal.org/node/39260), we split LOBs handling into separated API (according to LOBs patch: http://drupal.org/node/147947). we should never use LOBs for DISTINCT/UNIQUE/GROUP BY/ORDER BY directly :)

on the other hand, it is too silly if using LOB (Large OBject) for such query handling (since we map "clob:big" as "longtext" within MySQL implementation, which support such query clause)... it is too large, and not query friendly :)


Is it the CLOB or the "Lazy SQL" ?

Souvent22 - Thu, 2006-09-07 14:41

Ok, so I now have Drupal running Oracle very nicely. It is definatley a nice system thus far. I did a few bench tests for run by importing 2 million nodes and items, and used ab and a script to simulate some users....and I'll post more on that later. Anyway, the reason for this post...CLOB&undefined;s. I only have Oracle running nicely after some modifications to some modules and their queries. This becomes a philisophical question; and the question is:

If you have a large string (e.g. body field), when would you ever use it in a GROUP BY clause?

This question has been plauging me for a while as I went through the core and a few other modules fixing them. For example, on a comment on this thread you can see the code in comment.module:

    SELECT sub.*, rownum AS line FROM ( SELECT c.cid as cid, c.pid, c.nid, c.subject, c."comment", c.format, c.timestamp, c.name, c.mail, c.homepage, u."uid", u.name AS registered_name, u.picture, u.data, c.score, c.users, c.thread, c.status FROM comments c INNER JOIN users u ON c."uid" = u."uid" WHERE c.nid = 1 GROUP BY c.cid, c.pid, c.nid, c.subject, c."comment", c.format, c.timestamp, c.name, c.mail, u.picture, c.homepage, u."uid", u.name, u.picture, u.data, c.score, c.users, c.thread, c.status ORDER BY c.thread DESC

Now, why oh why do we have a GROUP BY clause in this SQL statement? There are no aggregate functions contained in the SELECT portion, so what function does this group by have? Let us take another example, aggregator.module:

//$result = db_query('SELECT f.fid, f.title, f.description, f.image, MAX(i.timestamp) AS last FROM {aggregator_feed} f LEFT JOIN {aggregator_item} i ON f.fid = i.fid GROUP BY f.fid, f.title, f.description, f.image ORDER BY last DESC, f.title');

  $result = $result = db_query('SELECT f.fid, f.title, f.description, MAX(i.timestamp) AS last FROM {aggregator_feed} f LEFT JOIN {aggregator_item} i ON f.fid = i.fid GROUP BY f.fid, f.title, f.description ORDER BY last DESC, f.title');

Now, the one commented out is the original SQL. The latter is the modified sql. I took out the f.image column because it was not used anywhere in the function aggregator_page_sources, not by any of the functions that it called inside if itself.
There are also some examples where a f., or was used, and thus inadvertantly a CLOB column was included and "GROUPED".

So, right now, it is a combination of some general "lazy" SQL (and who doesn't do lazy SQL, SELECT * :)), and Oracle's CLOB preferences that are giving the headaches for the Oracle Driver implementation.

Currently, to make the driver more "universally compatible", I and considering breaking an incoming SQL statement down if it has a GROUP BY clause, and removing and fields that are CLOBS from the SELECT and the GROUP BY clause (I would also have to fully spell out any table_alias.* fields).

So, not I am pre-plexing myself with this question:
In what situation would I be doing a grouping that include a large data set of random and for all intents and purposes, unique?

E.g. In what situation would you include the body field? I mean, it's bascially like trying to group on a raw binary image? Thoughts?


Patches

Dries's picture
Dries - Mon, 2006-09-04 06:38

Make sure to submit patches against CVS HEAD. Oracle support in core would be great. :)


oracle

Souvent22 - Thu, 2006-09-07 14:40

I am just finishing up splitting up the install SQL statements. I got sidetracked with Timesheet and Ponotmail. Look for a commit soon.


Sounds good

tachyonxv - Mon, 2006-09-04 19:58

This does sound like it could have some uses. I tested this recently and am unable to loginto Drupal, though that maybe a type or bad setup. There is a sqlite port in development which is the opposite extreme, though the alternative databases are cool. Look at the postgres specific code instead as its more similar to Oracle. Have fun porting the forum module. :)


Code

megalex - Wed, 2006-09-06 21:18

Where can we obtain the latest database.oracle.inc and database.oracle.sql. I am also trying to get full oracle support and will be able to provide some feedback.


Oracle is a knockout feature

robertDouglass's picture
robertDouglass - Tue, 2006-10-03 15:34

The European Union agency for which I did Drupal work is an Oracle shop, and they would feel much more comfortable running Drupal on Oracle than on MySQL, if for no other reason than they have Oracle expertise in-house. Great work on this. Here's the Digg link to the Drupal.org front page story. Your Digg can help get it on the Digg front page... and in front of a lot more Oracle developers who may now check out Drupal for the first time:

http://digg.com/software/Finally_Drupal_runs_on_Oracle


Handbook

Souvent22 - Tue, 2006-10-03 18:21

robert,

great to know there is more interest. I have started a "Oracle Handbook", and I will make a "MSSQL" one too. It's a step-by-step on running Drupal on Oracle. I would like to have some critques from people to get this handbook included in Drupal.org and to get it polished off:

http://eb3it.com/node/7


Drupal work for the EU

homo_sapiens's picture
homo_sapiens - Wed, 2007-10-31 16:15

Hi,

can you tell more about your Drupal works for the European Union agency. It might be interesting since in the government-areas there is indeed much more in-house expertise with Oracle.

So if you have the idea to use Drupal in a government-environment after a few minutes of discussion about Drupal someone will ask you, if Drupal does support Oracle.

Peter


IF you need some Oracle OCP ideas

Christoph C. Cemper's picture
Christoph C. Cemper - Mon, 2007-03-05 22:27

Hi Souvent22

  • cool to see somebody's porting Drupal to Oracle.
    I'm an OCP (oracle certified professional DBA) by the way,
    and VERY experienced with Oracle RDBMS + perf tuning,
    so if you come across some questions or need help with sth
  • let me know and I might help!

cheers
Christoph C. Cemper


Exactly what I needed

Souvent22 - Wed, 2007-03-07 06:39

Christoph,

Thanks! you're exactly the person I was looking for. Just someone to bounce ideas off of. No worries though, I won't be throwing a million questions at you. I try and figure out most things myself; but to get the Oracle port really finished, it's nice to have someone such as yourself in the community as a reference. :). Hopefully I can get back to the Oracle port sometime soon.. Right now, I'm knee-deep in the SQL Server one. Hope all is well.


Ready for Testing: New Oracle Driver for Drupal

hswong3i's picture
hswong3i - Wed, 2007-07-25 05:46

After over 1.5 years of development (since 29/11, 2005), plus number of developers contribution, and help of Drupal 6.x schema API, a new Oracle driver for Drupal 6.x is now ready for testing.

If you are interesting about this topic, please feel free to test it out, and send back your feedback, bugs, and suggestions.

Drupal 6.x LOBs patch issue: http://drupal.org/node/147947
Drupal 6.x Oracle driver issue: http://drupal.org/node/39260

What do you hope from my personal blog??