The main goal for this project is to create a new framework to implement multi user edition using AJAX into Drupal. Although there are some tools to do collaborative edition so far, almost all of them are proprietary software and the presence in the open source world is still scarce. In addition, none of them has been intended to integrate with any content management system yet, probably until Google decides to use writely everywhere.
Another important challenge for the project is implement new prototypes with COMET technology and use them where appropriate. COMET uses the http streaming pattern to push information from server to client, providing real synchronous communication between client and server whereas AJAX pulls the information with periodic refreshes to emulate the same.
This group will focus on discussions and feedback to support the Google Summer of Code project. The accepted project description at http://www.ernestdelgado.com/wiki/index.php/Drupal
Project page: http://drupal.org/project/collaborative_editor
CVS repository viewer: http://cvs.drupal.org/viewcvs/drupal/contributions/modules/collaborative_editor/
IMPORTANT: If you want to test the module it's better to read this before -> How to test the module
Refactor Collaborative Editor
Based on the Refactor revisions to use a version control system proposal changed, I post this proposal.
The problem
Core only allows the first user that sumbit the node form to save the changes; so the others have to see the saved node again and manually make the changes, hoping no other user save it before he does it. Clearly, it's a really slow way to make a contributed document.
Read more
Statement of Support
Hi Ernest,
I want to express my hope that this project makes it to a functional state. In light of the recent code update, I tried the project out, and it doesn't seem to be working. I think this project is very cool, and I would love to help out. But, since I don't know any of the technology that you are using, I probably won't be able to help until it is functional.
Any updates as to where you are at with the project are eagerly appreciated.
Dan
Read more
Google Docs - a good example?
This looks like where we want to go. I haven't had a chance to test the SoC module yet, but this one seems like a winner.
-M
Game Over
Yes, the SoC is over :(
The different aims written in the initial draft of the project have been achieved except to do more prototypes using the comet implementation
Luckily the project doesnt end here. My supervisor Vlado and I have a lot of ideas to add and improve the collaborative edition in drupal. This SoC has been the perfect starting point and the collaborative editors and development in this field is still scarce on internet. Therefore it will be great continue such development
Many points listed in the initial specification have had to be changed when we had to find the means or the technical resources to implement them. For example, in order to follow the xhtml strict specification (as drupal does) we decided to use textareas instead of iframes (as tinymce does). Using textareas we are not able to implement rich text edition immediatly but it never was an initial aim
Read morejQuery
There are many drupal developers eager to see and use jQuery (maybe too much?)
Now there is another js (drupal.collaborative_editor_ui.js) file with those functions that show the visual elements in the collaborative editor. For now, they use the typicall-old dom functions to manage the nodes. So, the very next step will be substitute them with the jQuery syntax and use some of the fancy jQuery features
No node functions
The following hooks: insert, nodeapi, form, node_info are not used anymore.
The aim is to extend the use of the collaborative edition to a more general types of forms
Once the module is activated we create any type of content (story, page, etc). Then if we go to the edit form of this content the collaborative edition will be active automatically
For now, it works with paths with arg(1) as a numeric value. And the collaborative editor is only active for the first textarea of the form. Both criteria will be improved to make the module even more general
Compatability with rich text editors?
Has anyone tested this with a rich text editor like TinyMCE?
Read moreAdding some UI elements
It's just to let you realize when every autoupdate happens or see which users are editing the document. It uses pure DOM functions to add the visual elements and there are still some bugs
Read moreCNodeId
All bugs commented in the last post are already fixed. Now there shouldnt be any problem when leaving the editor. The control of which users are currently editing the document is fixed and there's no need to edit the array manually as the previous version.
Internally there are two places where the node content is saved: the node table where the different revisions we 'submit' are saved as we always have done with any type of nodes, and ce_node table where the asyncronous calls save the content on every heartbeat reload
The new code is also in the cvs. Check the install.txt file and the .mys
Creating nodes
I've just uploaded a new version to www.ernestdelgado.com/drupal so that you can test a more consistent method to do collaborative edition
Now when you add new content you have to introduce the title first and then you can go to the edit tab to do collaborative edition. Right now, I'm adding some events (onbeforeunload) so the user is dropped from the current editors list when he/she leaves the document. Thus, you can get some popups in the wrong moment asking you to submit your changes before you leave
Once these are fixed I'll upload the code properly commented to the cvs repotitory
Therefore be ready to expect weird behaviour during the next hours.
Read moreHow to test the module
The easiest way is going to http://www.ernestdelgado.com/drupal
Use any of the following users: test1 (login: test1, password: test1), test2 (login: test2, password: test2) or test3
Create any type of content (story, page). Once the content is created go to the edit section to do collaborative edition
Notes:
If you experiment weird behaviour at the beginning press F5 and start again.
Different things you can test:
Read moreCarriage return BUG
Everybody knows internet explorer adds \r\n instead of \n everytime we hit return
I've tried in several ways to delete the \r before and after the asynchronous call both in server and client side:
<?php
$new_content = str_replace("\r\n", "\n", $new_content);
?>currentContent = currentContent.replace(/\r\n/g, "\n" );But IE still add the "invisible" character at the end of the line. As a result, if we delete the end of the line (i.e. merging two lines in one) there will be a wrong replacement length value (i.e. 4 instead of 3) and the content in both users will be different
Read moreCollision detection
There are two possible cases of collision:
Let's take the follosing parameters:
insert_A: where the user A inserts content
length_A: the content length inserted by user A
insert_B: where the user B inserts content
length_B: the content length inserted by user B
Important: this is only processed in case of replacement or deletion. Not in normal inserts
if (insert_B <= insert_A) {
// This is true if caret of user A is in between replacements or deletions done by user B
if ((insert_B + length_B) > insert_A) {
}
// the other way around
// This is true if caret of user B is between replacements or deletetions done by user A
DB optimization
The collaborative editor module uses three differenta tables: ce_node, ce_pchanges, and ce_users
As I consider myself a non expert in db performance and scalability I some points of the application so the rest of developers can improve the performance for future versions of the module
Besides different common techniques like catching, it will also be necessary to reduce how many times the data is read from the database and, if needed, to find alternatives like store the data in memory, xml, etc
It's necessary to see the workflow for each use case and avoid to pass parameters that are not used later either in client or in server side. In some cases, if a user hasn't uploaded new content it should be unnecessary to read the revision id from the database, unless there are changes from others. For now, there is an access to the database on every asynchronous call. So it has to be optimized the frequency of the asynchronous calls depending on the user actions (see HeartBeat pattern).
Line detection
A good advice from my supervisor was to use line locking for the concurrent editor. This would change radically the point of view on how to program the application but technically uses the same resources as the idea I had about set a pointer image to indicate what line are other users editing. I'll explain what is this common resource and what limitations or obstacles I see in its implementation, mainly cross-browser limitations:
In order to control the editing permissions a user would have in a specific line we need to know in what line is the user at the moment. There is the wrap attribute textarea when it is set to "hard" the text is sent wrapped to the server. I haven't tried a test to see if it helps me to find out at least how many lines currently are in the textarea, though.
Read moreStop typing
There is a second issue mentioned in Caret position and line detection we need to implement if we want to give the user a non obtrusive user experience in the collaborative editor. See the following scenario:
1. We are typing some content
2. The periodic refresh get what we have typed and send it to the server
3. The server checks what use case is going on and send the proper response back
4. During all this proces on the server side, we havent stopped to introduce new content on the client side
5. The functions that insert the response into the textarea dont take into account if the client is typing or not
Caret position
While a user is writing content, we must find a means to avoid to interfere with them in every update of the content of the textarea. There are two main issues to solve to achieve this.
a) If a caret of a user is in a concrete position, it should remain in the same one even if new content is inserted before or after it.
b) if a user is writing at the moment it cannot be interrupted whatever the insertion is
The last point is explained in more detail in Stop typing
Here, we will talk about the first one. To solve this point we need to use consistent crossbrosser functions. A possible scenario could be the following:
Use Cases
Based on the principle explained in pending changes we consider different use cases that could happen in the collaborative edition. The server distinct the use cases depending on two parameters:
a) if the user has done any modification on the client side (add or delete content)
b) if the revision id of the document uploaded by the client is different than the revision id stored in the server, which means that somebody else has also updated the server content before.
Thus there are the following use cases (with plain descriptions):
Pending changes
A means used in the server side to manage the concurrency is the use of pending changes list. I'm sure any top coder can improve (even laught at) the code so I'll explain how it works so it can be improved in the future:
Usually when one hears about collaborative edition one thinks the most difficult part is to handle when two users are writing content at the same time. And this is not true. The user who is accessing the server is going to read the data is stored even if another user is accessing 0,1 seconds later to store new data. The problem of concurrency is that a user should work on the last common version of the document even if the last version has been uploaded by another user 20 seconds before.
Read moreGet the diff
There are two moments in the application where we need to find the difference between the old content and the new content of the textarea where the users are writing. Actually, the techniques used to do this are infinites but all of them share the same principle of delta compression (or delta encoding or differential compression). There is a lot of research in this field. Since the 70's there has been better algorithms to do this aim, improving the way to find the difference between two files (or delta). Delta compression is concerned with many files transmisions compressions, control version systems and remote file synchronization in order to increase the efficiency of data transfers, reduce space requirements on the receiving device, etc
Read more