Excuse the hyperbole, but...
I really think the biggest problem from a designer/developer/admin's perspective is:
Not enough module specificity. By which I meant that module's are not specific (ie small) enough.
I'll illustrate by way of example:
I built an online classified web site (www.bandbswapshop.com), and had some difficulties with one particular aspect. I wanted the admin to be able to create a user and a Classified Ad at the same time.
In order to make this happen, I had to contract someone to program the php in the ed_classifieds module to create that form and hook it into the user signup module.
In and of itself, not that big a problem. But it made me realize that what Drupal needs is a bunch of very small modules that do one very specific thing. I mean, I can think of lots of other people who might want to create content and a user at the same time. Unfortunately, that bit of code is not stuck in ed_classified, not in a separate module. That is just one example.
I can think of several other examples where this happens. Where there is one aspect of a module that i'd like to plug into some other content in Drupal.
What I'd like to see is a concerted effort to keep things small. Keep things fine-grained.
The ship may have set sail on Drupal as a CMS to be fine-grained, but modules are certainly still being developed and still able to be influenced.
FWIW I am a usability specialist (MS in Human Factors Psych) who fell into Drupal development. I have now developed a dozen or so sites completely. So, I've been on all sides of this issue.
Thanks
JR