Tablesorting on taxonomy terms within a node view

I have a view which displays nodes. There are a bunch of terms attached to those nodes, all single-select vocabularies. I display the terms by using the "Taxonomy: All Terms" field in three separate columns and limiting each field to its particular vocabulary.

The display plugin is a table, and I want the user to be able to tablesort these taxonomy term fields.

As far as I know, there are two ways to display terms associated with a node within a node view - "Taxonomy: All Terms" and "Taxonomy: Term".

The problem with "Taxonomy: All Terms" is that it displays all terms associated with the node in list form in a single row. Thus sorting by taxonomy term can't really work.

"Taxonomy: Term" produces a new row for every term associated with the node. You can tablesort this one, but you can't limit to a certain vocabulary like you can with "Taxonomy: All Terms".

What I want is a field that is like "Taxonomy: All Terms" in that you can limit the term to a certain vocabulary, and it doesn't produce a new row for each term like "Taxonomy: Term". Of course I also want the column to be tablesortable.

I understand that this wouldn't really work unless the vocabulary is single-select. I'm not sure if the field can be toggled between tablesortable and not tablesortable on the fly. I guess a 'Limit by vocabulary' option on the "Taxonomy: Term" field would work.

Is this something I need to make my own module for, or should I make it an issue on the Views project page?