The menu system has a very old history in Drupal. The system as we knew it, began its life in 4.5 (2004 September, actually) when JonBob added back menu caching. As not all menu items were cacheable some code defined parts were in cache, some parts were not. To make it customizable, again some information was kept in a normalized database table when menu module was enabled. The complex structure kept every menu link for every page load. This was a performance and a memory problem. Also, the primary index was the path. From 4.6 the primary links was also handled by the menu module which meant that almost all sites began to use menu module. It was quite a logical wish to have a Login primary link which disappeared when you are logged in but as the login page also needed a menu entry in itself, this was not solvable cleanly. 403 and 404 pages were another huge problem -- it turned out to be totally impossible to have a 403/404 page in the non cached menu (maybe it is possible, but the then menu maintainer Richard Archer and Heine, kkaefer and chx could not find a way).
Another question was "how can I disable / change an already defined menu item"?
Unfixable bugs, plausible but impossible (or at least cleanly impossible) feature requests, scalability problems meant that old system has reached the end of its useable life. So consider this a process similar to the introduction of Forms API - i.e. an API conversion that will take time and effort to get perfect but which we expect to be rewarding in the long term.