Abstracting Drupal Mapping

Oh, please note that I am not thinking that this should be a standard API for creating maps, like what Mapstraction (the library ) tries to do, but instead creating a standard architecture for mapping libraries to integrate with Drupal.

I think this makes sense. I don't know which JS library is going to ultimately win out, but with the direction that visualization tools are about to go, I would expect for a burst of creativity to happen, and result in many JS libraries each with cool effects. An abstracted mapping GUI would withstand this creative activity.

I will help you. Let's get started.

It's probably necessary at some point to have an easier transition from OpenLayers to other libraries, as polymaps is its first real competitor and will fit certain use cases far better than OL has. However, designing a common interface to mapping libraries is tantamount to defining a new mapping library - which is no small order. The concern is mainly that the pattern that we've been familiar with in OpenLayers - essentially an implementation of Java-style coding practices and configuration by simple objects - is not what polymaps uses and is not likely going to be used by other mapping libraries anytime soon. The fact that OL tries desperately to change javascript is one of the big reasons why it's verbose and, even in 'minimal' distributions, very large. In contrast, polymaps seems to imitate jQuery in using monads often and avoiding pseudo-namespaces.

So, it's an honorable goal, but it shouldn't be understated, and we shouldn't overrate the similarities in API between these mapping libraries - a wrapper that transparently handles OpenLayers, polymaps, and cloudmade would have to deal with three vastly different code styles and moving targets.

Well put, @tmcw; I totally agree. Maybe I should clarify more of what I would assume this abstraction layer would be.

Well, first, what it is not. It is not what Mapstraction is. It is not trying to provide a common API to many different libraries. I definitely do think this would fail and would not be worth the effort to maintain. I also don't think this abstraction module would have any serious amount of JS except to just initiate some things.

I also don't think that this abstraction layer would necessarily allow someone to "switch out" maps. Like there is no reason you could change a setting form OpenLayers to PolyMaps and expect it to work (or even have that option).

What I imagine this to be would be a layer that provides the CTools (or other valid plugin architecture) plugins for the common objects of maps. It is very easy to see this as far as maps, layers, and behaviors, but I am still thinking on how styles/markers are best handled in this realm. Think about the current plugins for OpenLayers, but then adding a Map Type plugin on top.

Also, past setting up plugins and plugin API functions, I see this as a common way to set up a UI for building maps. A similar structure would be the Views interface, but instead of display plugins (or primary tables), it would be map types, like OpenLayers or PolyMaps, and from there the settings and plugins available would change.

My goal would be to take out the common architecture which is basically defining plugins and the UI basics and put this in another module so it does not need to be repeated, and can provide common integration points where available, like maybe Views data layers.

So in then end, a module like OpenLayers would just be a set of plugins for OpenLayers maps.

Thinking about how different map types could interact, in theory, one could write a behavior plugin that could work across multiple maps, as a behavior is actually a very thin idea for adding JS to interact with a map.

(obviously code would help here a bit, but let me know if I can go into more details)

I think that an abstraction API is a pretty off-the-tracks means of handling the discrepancies between the different input/output requirements. As tmcw implies, there are a lot of nuances to account for

The logical solution is to identify a common means of communicating GIS data. A gentleman's agreement that most of the major building block modules will convey their available data and their means of displaying data in one or more agreed-upon formats. Fundamentally, this is what the field system in Drupal does ( widgets for input, fields for storage, formatters for display). But many requirements transcend fields (e.g. views), and the same tools do not apply. For example, the input mechanism of typing a zipcode to have it geocoded into a field is not the same solution that would allow you to use it as an exposed search filter.

Also within the current system is the need to "hunt around" for data and fields. In CCK, a widget or a display formatter must declare which field type(s) it's willing to work with. Thus, if it wants to work with Geo's data, it must be aware of Geo and add 'geo' to its 'field types' definition. But Geo is an API, and other modules (e.g. Postal) might also have geo fields. Must openlayers and all other formatters be sent a memo whenever a new field that supplies geo data comes to exist? No, ideally, there should be a hook that declares what GIS-related fields exist and what formats they contain. Functionality that surrounds what is done with that data should be at the discretion of the consuming module.

Similarly, views functionality must grope around in the dark to include data from disparate GIS storage providers. Is Location installed? Look for fields supplied by the Location module. The fallback is having the user fumble around manually until either a lat/lon combo or something that provides WKT (which means what?! to a user) can be cobbled into a field source. Rather than all of this, it would be preferable to ask something along the lines of, "are there any fields in this view that are declaring themselves as points, linestrings or polygons? Assuming that the GIS modules all agreed on how to convey this, they can all remain loosely coupled, transcend a specific API, and still share data effectively.

Thus the problem is not that we need some giant abstraction layer, it's that we need modules to become aware of the data and display formatters provided by other modules. I have put a lot of time and thought into this, and the Geo/Geocode/Etc. tools solve this problem by using hooks that declare available GIS data, available fields, and matching up appropriate data types (e.g. "this display formatter will work with points but not polygons") I've spent a lot of time attempting to have this conversation on what appear to be dead ears. No matter what the ultimate solution is, it will have to be one that is agreed upon by multiple developers. My experience with trying to work with other developers in this spase has been disheartening to say the least, so the first problem to address will be how to work together fruitfully and productively.

I'd like to back up what Allie is saying here. I think this makes a tremendous amount of sense as something for the various developers involved in the Drupal GIS space to pursue, and really it shouldn't be that difficult of an effort to undertake.

If we can agree on a standard set of hooks for declaring the availability to provide or consume GIS data in standard ways, it becomes so much easier for module developers to work together and allow for the use of different storage or presentation solutions. We don't need to abstract everything and attempt to create our own Mapstraction or Openlayers equivalent, just starting with a set of agreed upon and documented practices for communicating GIS data would be a huge leap forward.

I think this also underscores a need for the developers in this space to themselves work on how we communicate with each other. All of these projects would benefit greatly from a tighter relationship between the individuals involved. Drupal would benefit tremendously as well.

@Allie Micka and @jerdavis, thanks for the comments. I do agree with most of what you are saying and trying to do, but I think its a separate issue than what I am trying to discuss here. Maybe I am just not describing my idea well.

I think the key concept here is that I want to build a simple Mapping Architecture for Drupal, not a mapping API for Drupal.

I am just interested in creating a mapping architecture layer that brings together the common objects that go into making a map from a Drupal perspective (not JS), because, IMO, it is the same for all the different libraries that are currently out there. I am not interested in making a shared API across multiple libraries.

No drupal_zoom_map('map_id'), but something like drupal_get_map('openlayers', 'map_id'); in order to just load a basic object and then that specific module will make everything happen.

I think this is a positive step, especially when looking towards Geo or other modules, because this sort of abstraction means that whatever integration with Geo or other module happens here, then it can translate to any mapping module that builds from it.

What we're talking about is more in line with what you're discussing, except we're talking about starting on a lower level. Call it an abstraction layer, an API, an Architecture - whatever you want. But the first and easiest piece to tackle with what you're describing is figuring out how the different pieces at play in a mapping or GIS pipeline within Drupal communicate with each other without each module maintainer having to specifically rebuild the wheel every time.

Think of it as ctools for Drupal GIS.

Although it's likely not even something you have to include as code like you do with ctools. Rather it's documented principles on how to communicate GIS information or announce that you have information you can make available in standard formats.

So a module that handles storing GIS data can make it known that it has some valuable stuff it can give to the modules responsible for handling presentation of GIS data (IE: Maps). Similarly a module that implements some input mechanism can easily hook into any module that might provide a storage option for that type of data.

I think that an abstraction API is a pretty off-the-tracks means of handling the discrepancies between the different input/output requirements.

Indeed: it's on different tracks. They're related but separate goals.

I don't have much time today to join this discussion, but I'll add one note: the OpenLayers module communicates via WKT and lat/lon pairs stored in any arbitrary data. It will soon communicate with maps exclusively via GeoJSON. Data standards like WKT and simple norms like storing everything as EPSG:4326 make this interaction rather simple, and thus several other modules integrate with OpenLayers on this level. There are a few design decisions in Geo which have been criticized, but the most obvious in this case is that this kind of modularity - integrating with any module that provides WKT via views - is not supported by it, but rather one must include Geo and invoke its own functionality in order to get data. The rationale - converting WKB to WKT, if I remember correctly - is decent, but could it not be handled by a field handler in Geo itself? Essentially the problem is that OpenLayers doesn't integrate with Geo because it integrates with Views, and Geo doesn't.

@tmcw Actually, Geo does integrate extensively with Views. With the ability to do some really complex stuff. It also can be asked to provide information as WKT, and you CAN use it with OpenLayers 2.x. It just was far more intuitive in 1.x for users to set something up. That support was removed from OpenLayers as a result of this thread: http://drupal.org/node/632316.

This type of misunderstanding of what Geo is capable of, while talking from a place of authority as principal contributor behind OpenLayers is quite destructive.

I'd encourage you and other developers in the space to look at Geo, and to ask questions or post issues if you run into problems. We'll work to do the same as we build solutions with both Geo and OpenLayers.

A principle point from this thread is figuring out how to communicate back and forth. I think we need to build this not only at the module level, but on the developer level as well. If we can find common ground and work together to understand and build out the landscape of GIS in Drupal we'll all benefit greatly.

I have a lot of input to provide to this discussion. I am coming from a different direction than "just maps", and I would argue that such insight is valuable and should just be disregarded out of hand. However, it's counterproductive to have a conversation founded on unsubstantiated assumptions.

The design goals and implementation requirements of Geo are not correctly described in this comment, but the way these assertions are presented - as fact - not as an attempt to gain more understanding - rules out the idea that there might be input or requirements that fits outside of a single use case.

Geo does not require you to call its api functions in order to use its data. If you want to communicate in strictly WKT, you can use the input/output descriptors I'm describing and completely leave Geo out of the picture. But WKT is profoundly limited, inefficient if not impossible to work with for complex data, and completely opaque to the common user. (Most people think Well Known Text means "Cincinnati, OH"). Storing and displaying WKT should be part of an overall solution, not "problem solved." Thus, communicating data types and formats internally for loose coupling is the goal, NOT getting everyone to go through Geo.

If there's a question about this, I would love to have a discussion that results in a solution that meets everyone's needs. But in the absence of such a discussion, an implementation based on assumptions leaves a lot of people with no viable solution.

If what I'm writing is destructive and incorrect, then I'm glad that it'll be corrected very quickly right here. Geo certainly has a high ratio of investment to adoption, which makes it an extremely sensitive topic.

I'll take a second look at Geo - my opinions are based too much on what geo integration used to look like in OpenLayers - what you can see is a tour de force of terrible code that was the 'integration.'

If I'm not mistaken, there are no other things that the OpenLayers module could be doing to integrate with Geo.

Geo does not require you to call its api functions in order to use its data. If you want to communicate in strictly WKT, you can use the input/output descriptors I'm describing and completely leave Geo out of the picture.

This is great to hear - I'm glad that I was incorrect in assuming that the WKB internal storage that Geo provides (or does it use WKB anymore?) needs manual conversion.

But WKT is profoundly limited, inefficient if not impossible to work with for complex data, and completely opaque to the common user.

Indeed. I can't really support the claim that it's limited or opaque to the common user - since, after all, whatever format geodata is stored in is going to be opaque to the common user. WKB, GeoJSON, etc., are not terribly human-readable.

Certainly - it is inefficient, and I can think of one use case that requires efficient polygon storage in Drupal - determining whether points intersect with polygons. This one's hobbled, though, since MySQL spatial only supports determining whether points are in minimum bounding boxes. Thus I've come across few cases in which people really want things in Geo (note there's the other obvious use case - displaying polygons on maps - in which browsers break before data formats do).

Yup, this sounds great. Reminds me of how WYSIWYG (http://drupal.org/project/wysiwyg) replaced the individual fckeditor and tinymce module. Now media.module provides a wysiwyg plugin which is equally usable with ckeditor and tinymce because wysiwyg.module provides the necessary abstraction. This is exactly the same as zzolo is talking about: Providing an abstraction layer between different mapping solutions, so that you have to define your map tooltip plugin only once for all of them.