I want to make a 2.0 version of the Date module into a real date api (it will still include a contrib folder with the CCK date field, but it is really intended to be an API). The current module is messy and inefficient and not optimized at all, it was originally designed to do one thing and has evolved into something completely different. On top of that, it makes no use at all of the new PHP 5 date handling capabilities. So it needs a real overhaul.
The new date and timezone handling for PHP 5 is killer! Practically nothing has been written about the new methods, but not only will they properly handle timezones, they will also properly handle historical dates, as far back as you could ever want to go. They can find things like the 'next Wednesday' after March 1, 2050 or the second Tuesday of October 1506.
I've been doing a lot of research on how to use the new functions (and believe me, it's hard to find much documentation!) I found the key is to quit using date(), gmdate(), mktime(), etc completely -- they have limitations even in PHP 5 on how well they work and how far back they can go. The new date_create(), date_modify(), etc functions can do almost anything you can imagine with dates with just a couple simple functions, once you understand how to use them.
I want to make an API with a nice, light set of basic date functions -- translated lists of month, day, and timezone names, functions to convert dates from one type to another (like unix to iso) and to parse the date parts from any type of date. It will handle pre-1970 and post-2038 dates no matter what system or PHP version you are using. Less commonly used functions will be available as options, included only when needed. Examples of that are things like date calculations to find the first Tuesday in the month, or whatever.
The most important thing is that I want to incorporate PHP 5 date handling into the api in a way that everything will still work in PHP 4. We'd need this even if Drupal required PHP 5 because the new date handling didn't work reliably and isn't always available until PHP 5.2. I'll handle this by using include files for PHP 4 functions so that the PHP 4 code is not parsed when not needed and it can just drop out later. My plan is to use my rewrites of the ADODB date library and the PEAR Date Calc functions to do most of the date wizardry for PHP versions before PHP 5.2.
I've got a lot of coding already done on this and plan to start to moving this into the HEAD branch of the Date module as I get things working. Then, obviously, I'll rewrite the CCK date field and the Calendar module to use the new API.
Comments
How to help?
Sounds like a fantastic road map, Karen. How can we help / support you in this? Want a ChipIn widget?
Didn't mean to ignore this
Didn't mean to ignore this post! While I'd love some $upport, I really needed to get this working to help me get Date and Calendar cleaned up, so I didn't want to wait and see what might happen with a Chipin, because I really need these changes :)
Karen, Is your above
Karen,
Is your above description instead of the drupal.date_.patch you posted elsewhere? Or is the patch one and the same? Please clarify. I'm reviewing that patch today. I'll post comments on that other thread.
Since that patch didn't make
Since that patch didn't make it into core, I'm doing it this way instead. I planned to use a lot of parts from that patch.
Oh, got it. Was it a time
Oh, got it. Was it a time issue that was the reason it didn't get in? Code freeze was the end of June, right?
Seriously, let us know how we can help. I realize that it's probably easier for you to keep track of this yourself since you are so intimate with the code, but there must be something we can do. One example is research. We could find links that point you to topics you need more info on and we could comment on which parts we think you'd be most interested in. We can also review code, debug, test, optimize code, etc. I don't think anyone here would want to take your place as architect of all of this. You're doing quite a good job. And this new route you're taking is much more favorable for my situation as I'll be able to use this with Drupal 5.1, right? I have chimed in on a couple of other discussions on events/calendar/etc, and I've been afraid that some paths discussed would make it difficult for me to create the modules I need. This API path, is very good for all potential modules in my opinion.
You may recall that I favor the idea of using node revisions for repeating events and I think if you aren't interested in that approach, that at least this API will allow me to pursue it.
So thank you very much. It's coming to a community filled with people like you that make newcomers like me join right in with the module creation/sharing.
Is there some place you are keeping the code that we can check out? And again, let me reiterate: is there anything we can do for you?
You may want to look at my
You may want to look at my recent rewrite of event module. I've rewritten it to use SQL date types and event to handle DST. The module still works with php4.
I feel that SQL date types are the most natural way to store dates.
sql data types
hi karen. i also suspect that sql data types are needed for best performance. here is an excerpt of a query generated by Views from observer.com:
SELECT fooFROM node node
WHERE (( REPLACE(node_data_field_publish_date.field_publish_date_value,'T',' ') + INTERVAL (-14400) SECOND) <= (NOW() + INTERVAL (-14400) SECOND))
ORDER BY node_data_field_publish_date_field_publish_date_value
DESC LIMIT 0, 4
this craziness in the WHERE clause does not let mysql use its indexes properly. to make this perform better, and still work before 1970, i would think SQL date types are needed. but i'm not too knowledgeable here. i just know a crazy query when i see one :)
Well, my queries aren't much
Well, my queries aren't much better (well, they are since they don't need the REPLACE and they are using actual dte fields which I expect to be somewhat optimized).
SELECT n.nid, n.uid, n.title, n.type, e.event_start AS event_start_orig, e.event_end AS event_end_orig, e.timezone, tz.offset AS offset, tz.offset_dst AS offset_dst, tz.dst_region, tz.is_dst, e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND AS event_start_utc, e.event_end - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND AS event_end_utc, e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 7200 SECOND AS event_start, e.event_end - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 7200 SECOND AS event_end, e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 7200 SECOND AS event_start_user, e.event_end - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 7200 SECOND AS event_end_user, e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL -14400 SECOND AS event_start_site, e.event_end - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL -14400 SECOND AS event_end_site, tz.name as timezone_name FROM node n INNER JOIN event e ON n.nid = e.nid INNER JOIN event_timezones tz ON tz.timezone = e.timezone WHERE n.status = 1 AND ((e.event_start >= '2007-06-30 00:00:00' AND e.event_start <= '2007-08-01 23:59:59') OR (e.event_end >= '2007-06-30 00:00:00' AND e.event_end <= '2007-08-01 23:59:59') OR (e.event_start <= '2007-06-30 00:00:00' AND e.event_end >= '2007-08-01 23:59:58')) HAVING (event_start >= '2007-07-01 00:00:00' AND event_start <= '2007-07-31 23:59:59') OR (event_end >= '2007-07-01 00:00:00' AND event_end <= '2007-07-31 23:59:59') OR (event_start <= '2007-07-01 00:00:00' AND event_end >= '2007-07-31 23:59:59');:)
oh gosh
that query makes me love our unix timestamps again :)
The advantage of the rewrite
The advantage of the rewrite is certainly not "easier SQL". The advantage is "easier handling of dates". For example, we can now simply store the date as the user entered it and don't need to modify it prior to saving it. That's a hige advantage if people ask you "why is my event shopwing up at the wrong time in the calendar?". Now you only need to look at the path from the database to output and not input to database too.
When you say date stored
When you say date stored as-entered, is that the case? or would they be stored in GMT so they could be sorted in the db?
By the way, i thought it interesting that PHP 5 doesn't have a way to create a DateTime object directly from a timestamp (at least not that I found). In my format_date() patch I first created a string from the timestamp:
<?php$date_time = date_create(gmdate('c', $timestamp));
date_timezone_set($date_time, timezone_open($timezone ? $timezone : 'UTC'));
?>
When I say "stored as
When I say "stored as entered" then I mean that, yes. Sorting by GMT does then require some adding and subtracting timezone shifts, but I let SQL deal with that. Hence the complicated query.
Is there any advantage to
Is there any advantage to this vs. simply storing a GMT date-time? The latter would be more efficient for sorting a number of events in chronological order. Also, if you want to find all the rows within a certain timeframe, for "stored as entered" you have to do the timezone math on each and every row, for the latter you convert only the timeframe to GMT. At least with php 5.2, there's no difficulty converting any date/time back and forth to GMT. So I don't see the advantage to "storing as entered", vs. GMT date + timezone.
For me the advantage is that
For me the advantage is that I conserve the time the user entered. This cuts down on debugging time when some user claims that his tie would be incorrectly displayed. Once you have converted a time to GMT, you can't be sure which time the user actually entered. The event module issue queue is full of such support requests or bug reports.
Sorting by GMT is indeed a bit more complicated since you need to subtract the GMT offset from the entered time. However, by storing the original time, you can actually compute the GMT time taking DST into account.
Incosistent formats
You will want to be careful going down this route since the different database vendors store date/time data in so many different formats. I haven't been down this road for a while, but Oracle at least has several different date formats in its field types, many of which are not ANSI SQL compliant. I believe MS SQL Server does not have an ANSI SQL compliant date format at all. I know these databases are not the focus of the Drupal community at the moment, but you shouldn't lock them out either. Focusing the efforts in PHP is much more maintainable and portable (although I understand everyone's frustration with it.)
You are right about
You are right about databases being inconsistent. I have now database include files for each of the supported database backends. I still think this is the better approach since the queries shouldn't change too often.
The event module's queries
The event module's queries may not change that often, but I need things that work in Views, where the queries change all the time. A part of the current Date API (the date_sql() function) is an attempt to normalize date query sub-parts so I can string them together like lego blocks for the query I need without needing to write a lot of database-specific code.
And absolutely, creating portable date sql is enormously hard, every database does it differently. For that matter, every VERSION of every database does it differently.
But the bigger issue is that many times we need a way to do date manipulation on dates that are not (or not yet) in the database -- and PHP 5 is the best way to do that.
While I agree that views
While I agree that views queries do change a lot, the sub-parts (SELECT, WHERE, ...) of them tend to be more or less the same.
I have also so far not encountered specific problems with version dependend SQL. I have however upped the requirements to MySQL 4.1.1 in order to make one query easier.
I agree that using php5.2(!) will make event module (and all date applications) somewhat leaner, but I've so far been reluctant to sprinkle the code with a lot of if-thens. Maybe for the D6 version.
No if...then needed
It won't require any if..then code. My plan is that the code (irregardless of PHP version, will just look like:
<?php$date = date_create('2007-05-01 15:30', timezone_open('US/Central'));
date_timezone_set($date, timezone_open('UTC'));
print date_format($date, 'm/d/Y H:i');
?>
In other words, everything written to use PHP 5 date functions. Behind the scenes will be an include file for PHP 4 that will have a PHP 4 version of those functions. Then when PHP 4 is no longer supported, the include file disappears and there is no need to make any changes in the code.
I obviously won't be able to replicate all the PHP 5 functions in PHP 4, but it should be enough to do things like create a date, adjust it to a new timezone, format the output, find things like the day of the week or the dates of the next two Thursdays after that date. There will be several extra included files in PHP 4, like my rewritten adodb date library and my rewritten PEAR Date Calc and some version of the old event.timezones.inc file to do all the dirty work behind the scenes. Then all those extra files can disappear when we get to Drupal 7.
As to sql differences, one difference between MYSQL 4.2 and MYSQL 5.x is what the database uses as a default timezone adjustment when you use functions that have implicit timezone adjustments, like NOW().
The API would not store
The API would not store dates, it would just provide helper and transformation functions to other modules, and the other modules could use whatever parts of it they need and do whatever they like about storing the results. I'm talking about a library of functions to handle pre-1970 and post-2038 dates and parse values out of dates, and adjust timezones, and find the first and third Tuesday of the month, or what day of the week March 10, 1960 was on, and things like that. It would be a basic Date API that it can be shared by other modules, including ones that don't need all the overhead of CCK or Event but still need some time and date manipulation capabilities. It would have to support all types of dates -- the unix dates we are still using throughout core, and the sql date format used by the Event module, and formats that might be encountered in iCal and RSS, and the ISO dates used by CCK.
Most importantly, I want it to use real PHP 5 date and timezone handling instead of our bastardized PHP 4 methods (and just use our old methods on systems that don't have PHP 5 available).
We also need to look at database storage issues, but that's not what I was talking about.
A README for the project
I have quite a bit done on this now, but it's not far enough along to commit anything. So to help give people the flavor of what I'm trying to do, here is the current version of the README file for the API. Also, I'm creating Simpletest tests for many of these things as I develop them, and I'm using WAMP which has a nice easy method that allows me to switch back and forth between PHP 4 and PHP 5 to test things as I go.
INFORMATION FOR DEVELOPERS
Once the Date API is installed, all functions in date_api.module are available to be used
anywhere. The system site timezone selector and the user timezone selectors are overwritten
to allow the selection of timezone names instead of offsets. Proper timezone conversion
requires knowledge of those timezone names, something that is not currently available in
Drupal core, and the change in selectors makes it possible to track it.
PHP 4 substitutions are made for PHP 5 date handling when needed. All files needed for PHP 4
substitutions are in date/includes/ and are ignored and can be removed if no PHP 4 substitution
is required. No action is required to substitute PHP 4 date handling, it will be done
automatically by the Date API module when necessary.
Simpletest tests for these functions are included in the package.
Available functions include the following (more documentation is provided in the files):
============================================================================
A version of Drupal's format_date()
============================================================================
This reworked version of the Drupal function will handle pre-1970 and post-2038 dates
using timezone names in both PHP 4 and PHP 5.
date_format_date();
============================================================================
Arrays of translated date parts
============================================================================
date_month_names();
date_month_names_abbr();
date_week_days();
date_week_days_abbr();
date_years();
date_hours();
date_minutes();
date_seconds();
date_timezone_names();
date_ampm();
date_short_formats();
date_medium_formats();
date_long_formats();
============================================================================
Miscellaneous date manipulation functions
============================================================================
These functions will handle pre-1970 and post-2038 dates in both PHP 4 and PHP 5, in any OS.
date_is_valid();
date_part_is_valid();
date_part_extract();
date_last_day_of_month();
date_day_of_week();
date_day_of_week_name();
date_unix2iso();
date_unix2datetime();
date_iso2unix();
date_datetime2unix();
date_iso2datetime();
date_datetime2iso();
date_array2iso();
date_array2datetime();
date_array2unix();
date_iso2array();
date_datetime2array();
date_unix2array();
============================================================================
Helpers for portable date SQL
============================================================================
date_sql();
date_server_zone_adj();
date_sql_concat();
date_sql_pad();
============================================================================
Date forms and validators
============================================================================
date_select_input();
date_text_input();
date_jscalendar_input();
date_select_validate();
date_text_validate();
date_jscalendar_validate();
date_select_process();
date_text_process();
date_jscalendar_process();
============================================================================
Wrapper functions for native php date functions.
============================================================================
These functions will handle pre-1970 and post-2038 dates in both PHP 4 and PHP 5, in any OS.
Uses native PHP 5 functions when available and degrades automatically for PHP 4 to use
substitutions like those provided in ADODB's date library.
date_getdate();
date_gmgetdate();
date_date();
date_gmdate();
date_mktime();
date_gmmktime();
============================================================================
PHP 5 date handling functions
============================================================================
Substitutions for the PHP 5 date functions are supplied. Use the PHP 5 functions in your code as
they would normally be used and the PHP 4 alternatives will be automatically be substituted
in when needed.
You cannot do everything with these functions that can be done in PHP 5, but you can create
dates, find timezone offsets, and format the results. Timezone handling uses native PHP 5
functions when available and degrades automatically for PHP 4 to use substitutions like those
provided in previous versions of the Date and Event modules.
Read the documentation in date/includes/date_php4.inc for more information about using the
functions in ways that will work in PHP 4.
Simpletest tests for the PHP 4 equivalent functions are included in the package.
date_create()
date_date_set()
date_format()
date_offset_get()
date_timezone_set()
timezone_abbreviations_list()
timezone_identifiers_list()
timezone_name_from_abbr()
timezone_offset_get()
timezone_open()
For example, the following will create a date for the local value in one timezone, adjust it
to a different timezone, then return the offset in seconds in the new timezone for the
input date; The offset will be adjusted for both the timezone difference and daylight
savings time, if necessary:
$date = create_date('2007-03-11 02:00:00', 'America/Chicago');
$chicago_time = date_format('Y-m-d H:i', $date);
print 'At '. $chicago_time .' in Chicago, the timezone offset in seconds was '. date_offset_get($date);
date_timezone_set($date, timezone_open('Europe/Berlin');
print 'It was '. date_format($date, 'Y-m-d H:i') .' in Berlin when it was '. $chicago_time .' in Chicago.';
print 'At that time in Berlin, the timezone offset in seconds was '. date_offset_get($date);
I was wonderinig if you have
I was wondering if you have all of these functions already coded? Are there any parts you'd like to have help with? Maybe some functions that still need coded?
Almost done, actually. I'm
Almost done, actually. I'm adding a bunch of simpletest tests to be sure things are working right and clearing out a few bugs, then I'm going to commit the new API to the HEAD of the Date module.
What I'd love then would be to have people put it through the wringer to be sure things are working right.
PHP 5 date handling odds and ends
After working with this a while, I'm a total convert to the PHP 5 date handling functions. They do in a couple steps things that take lots more code in PHP 4, and they do it with amazing accuracy.
Timezone offsets are always right, even going back to obscure things like changes in the way that the US handled daylight savings time during WWII. Accurate DST rules go back to the 1920's!
Convert a local date to UTC? No databases, no include files, no fussing around with ENVIRONMENT or other tricks to get timezones working, just two lines of code (and it always seems to be right):
<?php$date = date_create('2007-03-01 10:30', timezone_open('US/America'));
date_timezone_set($date, timezone_open('UTC'));
print the UTC value for 2007-03-01 10:30 is '. date_format($date, 'Y-m-d H:i');
?>
The ability to modify dates and do date calculations will blow you away. You can use any phrase that strtotime() understands as a way to modify your dates, except that strtotime() won't work on dates older than 1970 and the PHP 5 date handling will work on any date. What's the last Tuesday of July 1807? Two lines of code:
<?php$date = date_create('July 31, 1807', timezone_open('UTC'));
date_modify($date, '-1 Tuesday');
print 'The last Tuesday of July, 1807 was '. date_format($date, 'm/d/Y');
?>
You can create and manipulate extremely old dates, like find out what day of the week historical events happened, on and everything I've checked is always right, even an obscure adjustment to the calendar made in the 1500's comes out correctly.
Rewriting my modules to use PHP 5 date handling is whacking out huge chunks of code (and making things far easier to debug).
This is why I am so bent on getting this API working, with a PHP 4 alternative for those functions anyone can go ahead and code date handling modules to use these great tools.
I've spent some time looking
I've spent some time looking at the PHP5 date functions today. While I agree that they are a great improvment, they eem to be only halfway there. For example, it seem to me that to answer the question "When will be the next DST change for a given timezone?" I actually need to get all the changes from 1918 to 2038 and then loop over them... I may be wrong there, but if I am not, this lack of a direct lookup disappointing.
However, it is of course much better than having its own list of DST change dates that event.module currently maintains.
With the datetime and
With the datetime and timezone object handling you shouldn't have to work directly with the concept of DST, offsets, etc.; it's all handled for you automatically in the background. If you are actually looking at DST change dates manually, you may be doing something you don't need to (although I'm not sure what it is you're trying to do).
Yes, that is probably the
Yes, that is probably the only way to get that, but I wouldn't say that is something that would be needed in most use cases. You really don't need to worry about when dst happens to use the functions, it all gets taken care of behind the scenes. If php is going to automatically correct for dst, why would most people need to even think about it? So I'm not sure I'd consider it poor design if there is no automatic way to identify the next time dst will change.
Between the fact that your dates are now going to be human-readable and that PHP 5 date functions will make converting a date to GMT pretty painless, I don't think we'll have the problems storing date in GMT in PHP 5 that we've had with PHP 4. I'm re-writing Date and Calendar to completely get rid of mktime() and date() and gmdate() and use the new functions instead, and I think it's going to be a big step forward in simplifying the code and making things work better. Lots of the problems in the past are related to trying to pass a timestamp around to a series of functions using mktime() and date() without doubling up on or missing timezone conversions. Problems with dst just made everything worse.
The biggest problem is the lack of documentation. I keep finding little undocumented tricks. Like, to get a timestamp into date_create, just prefix it with '@' so the system knows not to treat it as a string, like:
$date = '567890211';
date_create("@$date");
The only place I found that referenced was in a bug report :-)
nice one! that would
nice one! that would simplify the format_date patch.
Timestamps
I was looking for that, too, and if you follow the link in the PHP manual to http://www.gnu.org/software/tar/manual/html_node/tar_109.html, you will find that section 7.8 (http://www.gnu.org/software/tar/manual/html_node/tar_117.html#SEC117) talks about that.
Any new updates?
How's this coming along? It's been over a month since we've heard. I'd love to get my hands on this.
I got diverted to working on
I got diverted to working on the CCK 6.x port, so I had to put this on hold for a time. It's been a good exercise, though, since I can make sure these changes will smoothly upgrade from 5.x to 6.x.
I'm anxious to get this done, too. As soon as I finish working on the CCK FAPI port, I'm back to this project.
Thanks for the update. I'm
Thanks for the update. I'm hoping it will happen soon.
Thank you Karen...
...for all that you've done and continue to do with your important module development. I'm really looking forward to the release! :)
Walt Esquivel, MBA; MA; President, Wellness Corps; Captain, USMC (Veteran)
$50 Hosting Discount Helps Projects Needing Financing
Walt Esquivel, MBA; MA; President, Wellness Corps; Captain, USMC (Veteran)
$50 Hosting Discount Helps Projects Needing Financing
Early version coming along
I just checked a bunch of changes into HEAD that gets this partially working. It is absolutely positively NOT ready for production, but you can start to see how it will work.
I'm adding in a number of things that have been long-requested, including
1) a jquery popup calendar (very experimental) that has separate date and time input selectors, as a substitute for the odd behaviour of the jscalendar time entry.
2) the ability to define default values as 'blank', 'now' or 'custom', where custom might be something like +90 days
I figure it's about time for
I figure it's about time for my once-a-month pestering to see how this is coming along. Any updates for us?
Has progress stalled? I see
Has progress stalled? I see that the 2.x version is still in HEAD and there are quite a lot of old, unresolved bugs, tasks, feature requests, etc. in the queue. I suppose it's a matter of real life getting in the way, but I see that you've been able to work on other projects during this time (unless I'm thinking of someone else), so I'm wondering how much of a priority this is, how far from an official release it is, etc. Any news?
Not at all, I'm just about
Not at all, I'm just about ready to make a 5.2 branch and issue a dev release. If you look at the commit log you'll see where I've been spending all this time. I've also been addressing issues in both the Calendar and Date queue, so I haven't neglected them either, I just haven't gotten to everything, especially issues that I hope will be resolved by the new version.
It's working fairly well now, but needs other eyes looking at it and testers. I even have a date repeat module added in that makes it possible to automatically repeating dates. The last thing I need to do is get the Calendar module updated to use the new API, and that is partly done.
I had to stop and work on the CCK D6 port since we're getting a new RC this week and CCK needs to be ready.
Thanks for the update and
Thanks for the update and your work. I have been increasingly frustrated with the existing event_repeat and related modules (despite having come up with some patches that seem to have fixed most of the problems) and have anticipated yours for quite some time. I look forward to a release. I have hesitated testing HEAD simply because my time is limited. But if you release something you think is good enough for a dev release, I can definitely put that on my test server and give feedback - and possibly patches; I don't mind getting my hands dirty :-)
I'll try to remember to check commit logs in the future. It never even occurred to me.
subscribing