Wondering if this has been built yet, or if I should start from scratch.
I'm looking for a CCK text field that takes a "nice" value for location such as "Montreal, QC" and converts it in the back-end to "45.518376,-73.56308" so that it can be displayed in a Google Map, or searched via proximity or etc. I've attached a mock-up so you know what I mean. Ignore the form_alter stuff at the top; this was when I was thinking it was going to be its own form element but Eaton advised me to go with CCK field instead (user form would be possible w/ something like Bio module).
I basically need to ping $geo_service on form submission with whatever fancy API they offer and then store the entered text, longitude, and latitude. Then display the output through $mapping_service. It seems like the "convert the address to long/lat" and "show me a map of this long/lat" services should be separate, and configurable/pluggable.
I saw http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/geolocation/ although it doesn't have a project page associated with it, so I wasn't sure how good that was as a starting base. Any other suggestions or pointers on where to start would be appreciated. :)
| Attachment | Size |
|---|---|
| location_field.png | 177.83 KB |
Comments
Geocoding snippet
Here's a code snippet that andremolnar posted in #drupal to help w/ the geocoding stuff
<?php
//<em>sigh</em> geonames is simply not predictable enough with its results to give us a correct city
//we fall back to our good friend google
//some stuff required if we use google maps API to do our geocoding
//Google documentation found at http://www.google.com/apis/maps/documentation/services.html#Geocoding_Di...
$googleurl = "http://maps.google.com/maps/geo?";
//we're gonna assume that gmap module is installed and configured
$key = variable_get('googlemap_api_key', '');
//we'll want xml output
$output = 'xml';
$lookup = urlencode($lookup);
$url = $googleurl . 'q=' .$lookup . '&output=' . $output . '&key=' . $key;
//get the xml contents
$xml = file_get_contents($url);
//turn that into simpleXML
$sxml = simplexml_load_string($xml);
//If we have a good response code and an accuracy of 4 (city level) or better we use the data
//particularly bad responses are 600 level
if ($sxml->Response->Status->code == 200 && $sxml->Response->Placemark->AddressDetails['Accuracy'] >= 4) {
$coord = $sxml->Response->Placemark->Point->coordinates;
$dividepoint = strpos($coord, ',');
$lat = substr($coord, $dividepoint+1, strpos($coord, ',', $dividepoint)-1);
$long = substr($coord, 0, $dividepoint);
$address_details = $sxml->Response->Placemark->AddressDetails;
$country = (string) $address_details->Country->CountryNameCode;
$state = (string) $address_details->Country->AdministrativeArea->AdministrativeAreaName;
if ($address_details->Country->AdministrativeArea->SubAdministrativeArea) {
$city = (string) $address_details->Country->AdministrativeArea->SubAdministrativeArea->Locality->LocalityName;
}
else {
$city = (string) $address_details->Country->AdministrativeArea->Locality->LocalityName;
}
}
?>
Location Field
I started an issue about this for the Location Module a while ago. It would be very handy to have a CCK field manage the information instead of how it's currently managed... Mikesmullin posted a module for it, but I haven't had the time to check it out yet....
Excellent, thanks!!
I'll take a look.
+1
+1 Subscribed
This would be awesome. But
This would be awesome. But please make sure it's not U.S.-specific (not sure if it could be, but just saying it up front to be sure).
Definitely not US-specific
That's something that annoyed the living heck out of me as I was going through the existing options. >:\ Looks like the choices in international geocoding are somewhat limited though.
three pieces - largely built
So, there are really three pieces here in my mind - and I just built this, so I feel like I understand it fairly well ;):
1) cck text field to enter the location information (or you could use the cck address field as well, if you want to be more specific).
2) a field to hold the lat/lon
3) the web service code that can read in #1 and stick the geocoded data into #2
For #2 you have two options as far as I know. The first and recommended in my opinion is the geo module's cck field. The module is still not perfect, but it is getting a lot closer. The second is the cck_latlon module that I wrote which is very naive in its storage of data, but which works :) You would just need to hide the entry of this field from users using form alter or similar. I would love to get rid of the cck_latlon module, but it's kind of on hold now until my project is finished and the geo module matures.
If you don't use the geo cck field you should at least consider using the geo WKT storage API. If we all standardize on that then it makes it easier to build the rest of the infrastructure for these kinds of modules.
For #3, well, I wrote that for a service that currently only includes US data but it is still under development/review before it can be released. Given that it's for a commercial service there's no capability in the geocoding/webservice module I built to make it extensible to other service providers. Contact me directly if you want more details on becoming a beta tester for that module or have any other questions on it :)
For your question and goals webchick, I just wanted to point out that your module only needs to do the webservice integration piece and that the rest already exists. I know that the whole "form alter away this one field" thing is not as cool as providing your own set of fields, but
--
Knaddisons Denver Life | mmm Chipotle Log | The Big Spanish Tour
knaddison blog | Morris Animal Foundation
Thanks a lot!
As I was researching the various location-related modules, it seems to me that many are using Location module for its schema only (in much the same way as all voting modules use VotingAPI - so they can all talk to each other in the same language). So is Geo basically to become that new "unified means of storage", or is that the end goal? I agree that standardizing on something like is preferable to a cck_latlon type of module, although that'd certainly work for something quick.
I tried Geo module a couple months ago and <a href="http://drupal.org/node/168864>literally couldn't even install it, so that made me shy away from it some. ;) I'll try and will myself to give it another shot. ;)
If you'd be willing to share even skeleton code for how you implemented #3, I would be SO appreciative, but if you can't I of course totally understand that too.
Thanks for responding! I knew you were working on something like this but you never drop by IRC anymore :( and I also never respond to e-mails so. :P
yes, that's the ticket
My understanding is that due to the way that Geo stores data it is technically superior to location module. It uses db-native location storage which will beat lat/lon for most any purposes. So, yes, I think that Geo should become the one storage engine to rule them all.
There were some critical but relatively easy to fix issues with Geo that should have been ironed out. If you find any new ones (or anyone else does) feel free to ping me about them. I want to see that module get to a 5.x-1.0 release!
About irc...well...sometimes hotels block irc. Sometimes like the last week of my life :(
EDIT: Of course, geo module lacks many features of location module (it's not intended to be a replacement) and is not yet a fully working module and of course Location could be migrated to use db-native geolocative storage and I'm sure someone will point out how location's storage is superior and I shouldn't make blanket statements like that, but...geo makes more sense to me right now as the module to be rallying around so I'm rallying.
--
Knaddisons Denver Life | mmm Chipotle Log | The Big Spanish Tour
knaddison blog | Morris Animal Foundation
Music to my ears...
/me witholds comment on location module. ;)
Ok cool. I'll give Geo another shot and file patches for anything I come across. Thanks a lot for your input!
I started to do this a long
I started to do this a long time ago, but ran out of time and never got anything usable finished. Here's what I was thinking was needed:
Create a location API. The current location module actually has lots of usable parts for an API. For instance, there are functions that can do a SQL search to find location proximity to a lat/lon point using the lat/lon fields in the location file. I would throw out the parts that actually store data and the zip code table (which is the part that is too US-centric) and focus on creating an API that can create flexible location forms, produce lists of country and province names,( especially dependent dropdown lists where the province names are limited to those in the selected country), create sql to do proximity searches, create 'map this' links that work in various map services, etc.
CCK fields can work perfectly here as the place to store the data, using the API to create the widgets. With CCK you can decide how many addresses you want on a node and if the address should include a phone number, and how many street address lines you need, things that are not flexible enough in the current location module.
Create a location service API that can access various location services to geocode an address. Ideally, it would be flexible enough to use various services, since some services work better than others for geocoding data from different countries.
Create a map service API that will let you pick a point on a map and store the geocodes, again ideally using different mapping services.
I think many of these elements exist in whole or in part in various modules (location, GMap, YMap, address, geo, etc), but they don't work together or create sharable APIs.
As to storing the data in native geo fields, keep in mind that this may have a negative impact on database portability, since not only will the storage be different, but the SQL to retrieve the info from the storage will be different, too, so you will end up with lots of database-specific code to make that work. Also, CCK doesn't have a native geo field as an option in D5, and Schema API doesn't have one in D6, so you couldn't use CCK or Schema API to manipulate your data if you go that route.
clarify
Can you clarify that? I'm using the geo-cck field with Drupal5 to store lat/lon data so it is an option in the geo module package of modules. Or do you just mean that the cck project itself doesn't have one? That would be good news since there are two already.
Your point about SchemaAPI in D6 is an interesting one, but I think that even if it comes to "run this script manually" that is a hurdle that people who want/need Geo module will be able to get over - heck we all did that back in the barbaric days of 4.6.
--
Knaddisons Denver Life | mmm Chipotle Log | The Big Spanish Tour
knaddison blog | Morris Animal Foundation
Lowest common denominator
What Karen is saying is that both CCK in D5 and Schema in D6 have a "lowest common denominator" approach to the fields that are allowed. You get ints and strings and not much else. Geo.module overcomes this by handling its own storage instead of giving that job to CCK. THis is really a shame since it requires a lot of effectively duplicated code in Geo, but it is the way things work right now.
On whole, I find Karen's post a little pessimistic. There is no intrinsic reason why we can't have a geometry field supported by CCK/Schema, it's just nobody has taken the time to include it. On the whole the SQL is fairly standardized around the OGC standard for DB storage. For geo.module bdragon by and large just copied my postgresql/postgis code for the mysql backend. We could probably factor alot of code into a common .inc file.
Note: here we are talking about the fields support at the database layer, not Drupal fields - which are a different entity completely.
CCK uses Schema API in D6,
CCK uses Schema API in D6, so only field types that Schema API supports can be used as CCK fields. De-coupling CCK from Schema API would be a huge effort that no one has time or interest in doing -- Schema API allowed us to rip hundreds of lines of code out of CCK and we don't want to go back :)
So even if we added it to the D5 version of CCK (which is more work than you may realize and the reason why other types of fields weren't added), it won't be supported in D6.
And that means if you want to store data in that kind of field, you'll have to create and manage your own tables rather than using CCK to do it.
And none of that addresses my other point, which is that it is not just the data storage that is different, but the SQL you use to read it, so it requires extra work any way you look at it to create and maintain something that is portable across databases.
The question is whether the advantages of native db storage are significant enough that it's worth all that. I don't know about that one way or another. I'm just pointing out the issues.
one custom storage vs. another
Among the three choices for lat/lon data (location, cck_latlon, geo)
1) Location requires 100% it's own code to manage the db
2) cck_latlon I'm saying is probably not worth using or upgrading to D6
3) geo requires its own code to manage the db
So within the D6 time-frame, all of the solutions have this weakness. It's a bummer, but not a dead-end.
And thanks to both of you for clarifying this. Makes much more sense now.
--
Knaddisons Denver Life | mmm Chipotle Log | The Big Spanish Tour
knaddison blog | Morris Animal Foundation
I think the issue is really
I think the issue is really how to design this so it will work the best possible way going forward. In D6 you can still put lat/lon info in a varchar field and be able to manipulate it to do proximity searches, etc. That's not what the geo module has done, but that's the way the location module does it, and it does work. If you did that, you have something that you can use in CCK in D5 and D6.
It really comes down to how significant the advantages are of native db storage, since that's what won't be supported until at least D7 (and possibly not even then if it doesn't make it into Schema API). I've run into this kind of thing with the date module (native date storage vs using integer and varchar fields for that purpose) and although that sounds like a slam dunk (it seems like native date storage must be best) it's quite a bit more complicated than that because the SQL to manipulate native date fields varies not just between MYSQL and Postgres, but even from one version of MYSQL to another, including differences not just between major versions but also minor versions of each database. My conclusion in that case was that the work of keeping everything working the same way across all versions of all database types overwhelms whatever advantages there are in using a native date storage field. And even if you get them working right now, what happens when something changes (again!) in the next version of MYSQL?
That may or may not be as much of an issue with native geo fields, I don't know anything about that, but that's what you have to examine.
First, let's define this
First, let's define this problem to be as simple as possible. So forget about D5, let's concentrate on D6.
Second, I really can't get behind the dumb databases argument. It breaks down really quickly when it's proposed as a solution for maintainability. The argument is that if we manage everything in PHP then it is easier to change and upgrade over time. We are not beholden to the database and not beholden to it's bugs/faults.
Sounds good, but let's look at Drupal modules that take this tack: Flexinode (giant Entity-Attribute tables, yuck!), location (have you looked at this code base? There was a reason I started geo.module), the list goes on. I'm going to stop before I insult more people.
I'm all for a little tough love when it comes to databases. I think Drupal has been far too permissive in allowing people to sit on old DB versions. While I won't stand in the way of a working mysql component for geo, I'm not holding my breath either. If you have to have postgis/postgesql I'm fine with that.
That said, I encourage people to write ANSI SQL as often as possible, and use tools like CCK, Views and Schema to push code for handling database inconsistencies to the lowest level. (Frankly, I'm falling out of love with RDBMSs lately, but that's a topic for another post).
I think Drupal is being crushed under the weight of maintaining legacy code. We should do one of two things: a) call a universal feature freeze and just spend our time working out the bugs and making drupal stable, or b) burn our bridges and move forward, embracing new technology and leaving some code/databases/people behind.
Anyway, I've wander away from the topic of this post, so I'll end my tirade here. Plus, I've written squat for code lately, so I have no credit in my rant account.
+1 to Karen who is always thinking the big thoughts and looking at the big picture. :-) If we have time to argue over this or that, it's only because we agree on so much more.
Cheers,
-Mark
Geo looks like a great fresh start
I'm very impressed with the start you've made in Geo module. We needed this fresh start and the approach feels right: build modularly basing support on established standards. Nice work.
Open source database support (PostGIS and MySQL spatial) for Open Geospatial Consortium (OGC) standards has been a major step forward, enabling the sorts of complex and powerful geographic analysis that it would be painful to contemplate in PHP. Several years ago I wrote PHP code for converting OGC WKT (well-known text feature geometry representations) to and from various formats. That's one of many problems I'm very happy now to leave to the native support of geographic databases. Let alone problems like creating buffers, doing spatial overlays, handling projections....
Mark, you've written lots of
Mark, you've written lots of great stuff, I have a great appreciation for your ideas :)
As to supporting legacy databases and old versions, lots of that comes down to supporting people who can't control those things, like on shared hosts. And that includes me at the moment since I have a host that doesn't provide support for many things I wish I could do. I plan to move to a better host some day, but that is a huge time-consuming project that I don't have time for right now, so for now I'm stuck.
Look at all the ranting and raving there has been even over the decision to go PHP5-only in D7, and lots of people (including me) would like to see Drupal support more databases, not fewer ones, and I think that's where we're heading. Dropping support for the oldest versions of these databases makes sense, but will still depend on what's actually available on shared hosts, and they don't move very fast.
Anyway, if this kind of data storage is really important, push to get it into Schema API in D7.
The glory of contrib: raising the bar
If there isn't already, there should be lines in Drupal module .info files for declaring required / supported database types and versions (MySQL 5, PostgreSQL) just like modules can declare minimum PHP requirements.
This way if a better database simply is the way to go for advanced geolocations, a module can lead the way for those who must have the best, without requiring all of Drupal to ditch the rest of us.
benjamin, Agaric Design Collective
benjamin, agaric
minimum requirements
drupal6 added a minimum php in the .info file. all other requirements can be expressed by hook_requirements
distressing
it is a bit distressing to hear that a hotshot new module like geo-cck is simply impossible in port to D6 given the tight coupling of CCK and schema API. how can two great giants thwart the underdog like this? what to do?
Disconnecting CCK from
Disconnecting CCK from Schema API is just not going to happen, it would require a huge re-write and lots of additional code. This is not a case of a 'giant' trying to thwart an underdog :) Barry asked what kinds of fields needed to be supported in Schema API and there was a long discussion about that on g.d.o. and no one ever mentioned this type of field.
The solutions are to get support for this field type into Schema API, or write the sql code for this field the old-fashioned way (not using Schema API). You can still use CCK for everything except the storage and handling of that one field (the widgets and formatters and the field setup forms and everything else should work fine), so it should be possible to make something work. It just won't be as seamless as it would if CCK supported that data type.
ok, relieved
now that sounds better ... though i've just noticed location.module has been revived.
Schema API and database types
I haven't read most of this thread yet but I gather that Schema API does not define a database type that some module needs. The easy solution is to use mysql_type and/or pgsql_type, ie:
<?php$schemea['table']['fields']['foo'] = array('type' => 'varchar', 'mysql_type' => 'MAGIC_GEO_TYPE', pgsql_type => 'whatever');
?>
This is supported in D6 though it seems not to be documented at http://drupal.org/node/146843. Hmmm.
Moshe sent me an email asking me to comment here. The subject was "module defined column types in schema api" and I think he is suggesting something like hook_element() where modules could define their own Schema API types for different database engines. That sounds like a great idea but is pretty clearly too late for D6.
Just for others that find
Just for others that find this thread, the length variable is not db type specific Drupal 6, so to allow both field types you need to manually unset the length for a fallback varchar element.
<?php$schema['node_time_terms'] = array(
'fields' => array(
'nid' => array('type' => 'serial', 'unsigned' => TRUE, 'not null' => TRUE),
'delta' => array('type' => 'int', 'unsigned' => TRUE, 'not null' => TRUE, 'default' => 0),
'type' => array('type' => 'varchar', 'length' => 63, 'not null' => TRUE, 'default' => 'rangedtimeterm'),
'start_datetime' => array('type' => 'datetime', 'not null' => TRUE),
'end_datetime' => array('type' => 'datetime', 'not null' => TRUE),
'flags' => array('type' => 'int', 'not null' => FALSE, 'default' => 0),
'start_time' => array('type' => 'varchar', 'mysql_type' => 'time', 'mysqli_type' => 'time', 'length' => 8, 'not null' => TRUE),
'end_time' => array('type' => 'varchar', 'mysql_type' => 'time', 'mysqli_type' => 'time', 'length' => 8, 'not null' => TRUE),
),
'primary key' => array('nid', 'delta'),
'indexes' => array(
'nid' => array('nid'),
'start_datetime' => array('start_datetime'),
'end_datetime' => array('end_datetime'),
),
);
if ($GLOBALS['db_type'] == 'mysql' || $GLOBALS['db_type'] == 'mysqli') {
unset($schema['node_time_terms']['fields']['start_time']['length']);
unset($schema['node_time_terms']['fields']['end_time']['length']);
}
?>
From left field: lat-lon attached to place taxonomy terms
For the 2008 World Social Forum (actions all over the world) I made a module that use taxonomy terms to represent the town, optional province/state, and country of places. The advantage to this approach is that, as their own entities, each place term itself has a latitude and longitude. I used some service (I don't remember which, maybe Yahoo) to look up the coordinates for each place as it got created.
It's mostly in a submodule (and only module) of the Place project, which would like to be part of an ecosystem of location modules that work together (and could be home to a good chunk to ease the dependency download pain).
I didn't get as far as I wanted with it even for this one project, but if anyone's interested give me a heads up so I can make sure the last of our enhancements make it into the module on d.o.
benjamin, Agaric Design Collective
benjamin, agaric
yessir
By all means, please check it into CVS or make a release so others may take a look. Using taxonomy terms for each location makes sense to me, and I remember being very interested when I saw the Places project page appear.
Are there any updates on this?
I'm trying to create/re-use something similar combined with Domain Access module where there would be one default location (domain) and more specific locations for various content within the domain.