We've been talking about maintaining a contrib module as a group for a little while now. I have an idea for one, which I would like to propose: a Unicode filter module. On every CMS site I've ever built I've had a problem with users copying and pasting in content from Microsoft Word or other web pages that contains curly quotes, em dashes, and other non-ASCII characters that invalidate the HTML output and break RSS feeds. We're dealing with that issue at HDR right now, and I've finally come up with a workable solution: I've written an input filter that simply maps Unicode characters to their corresponding HTML entities (e.g. an em dash to —). It's a very simple module, but it serves, in my opinion, a tremendous need, and it has lots of potential for growth. Right now it only handles a handful of characters (of which there are hundreds, if not thousands), so there's room for all of us to contribute, and a lot of it is very simple—no module coding experience required (while, for those of us who are more experienced, there's need for actual coding as well as CVS work, etc.). I think the module is a great candidate for our group project. Would everyone chime in and be prepared to discuss at our next meetup? Thanks!!
Comments
Fab
I like this idea. I've had plenty of sites that would have benefitted.
Agreed
I don't run into this quite as often, but it is because we go out of our way to train users to not do this.
Sure would be nice to not have to worry about it though.
WYSIWYG considered?
I don't know the background of your issue, but have you ruled out the WYSIWYG editors, e.g. tinymce (http://drupal.org/project/wysiwyg)? Tinymce and some others have a 'paste form word' feature that works most of the time to clean up the garbage and possibly correct characters. They also solve other problems that end-users have with editing their content by providing much of what people have come to expect in editing text. They can also do things like clean up markup. However they do come with their own pitfalls and we avoid them where we can, but they have proven necessary from time to time.
Actually...
Thank you for the suggestion, but WYSIWYGs cause the problems this module proposes to solve more often than they fix them. For example, put a trade mark symbol (
™) into FCKeditor or CKeditor in source code view. Now switch to preview mode and back. The WYSIWYG has converted the HTML entity into a Unicode character, which is how it will be saved in the database. So even if your content providers are very savvy and put the correct code into the system manually, these two WYSIWYG's will break it. This conversion is unavoidable, as far as I know. As to "Paste from Word" features, I haven't found one that works acceptably. I still think this module is necessary.I see. These are complicated
I see. These are complicated problems and sometimes they were fixable through settings in the Web page or Web server char sets (I think sometimes the problem is in the way the browser's textarea works, especially I.E.), or through editor settings such as FCKConfig.ProcessHTMLEntities (I'm not sure--perhaps even these do not solve all the issues, would have to play with it!) The problem you've seized upon is that even if one can solve it, lots of people are going to run into it, and it's hard to solve. The module would make it easy for everyone!
I like it too...
especially since you have already written the module. Though you hard coded the unicode characters that we needed for expediency, right? It probably should have a setting page for selecting the unicode characters to replace (or strip). Also, how did you implement the replacement - string replacement or regex?
Update: A brief search for reports of this issue w/ WYSIWYG editors
TinyMCE: entity_encoding raw removes HTML entities
http://drupal.org/node/373542
Preventing CKEditor from converting HTML entities
http://drupal.org/node/803562
wysiwyg broken ckeditors protectedSource feature.
http://drupal.org/node/775076
An option to not convert accents to HTML entities
http://drupal.org/node/734816
Unicode characters on FCKEditor or TinyMCE - unanswered since Dec 2008
http://drupal.org/node/344136
Enabling Unicode with Editors
http://drupal.org/node/346508
[SOLVED] UTF-8, UNICODE or foreign character showing up as ????
http://drupal.org/node/412184
Utterly confused about Full HTML and Unicode in TinyMCE
http://drupal.org/node/306608
—M
Michael Joyce
"Design is a good idea.
Good Research
Good research, Michael. It looks like there are plenty of people looking for such a solution as I'm proposing. I've got a mostly functional module written at this point. It just needs to have the rest of the Unicode characters entered into it for an initial release. We'd like to make a decision to pursue this or not at the next Drupal meetup, so if anyone has any objections or competing ideas, please get them out there now so we can be prepared to discuss them then.
Drupal is UTF8, which means
Drupal is UTF8, which means you should be able to put in a unicode character and it should all work.
Check a few things.
\status
Do you see:
Server characterset: latin1
Db characterset: latin1
Client characterset: latin1
Conn. characterset: latin1
show create database
my_drupal_database_name;Do you see:
+-----------------+----------------------------------------------------------------------------+
| Database | Create Database |
+-----------------+----------------------------------------------------------------------------+
| live_ahc_drupal | CREATE DATABASE
my_drupal_database_name/*!40100 DEFAULT CHARACTER SET latin1 */ |+-----------------+----------------------------------------------------------------------------+
Well you've found your problems.
Try setting up everything to use UTF8 in the future :-)