Should "system name" or "machine name" be used for non-human readable names?

As any of the names already suggest, humans should not have to be confronted with these labels at all. Keep them out of sight as much as possible. This is part of the below-the-surface plumbing, not of the UI-surface.

Yes, but in some cases Drupal requires administrators to enter a system/machine name. I am trying to find a general name for this, so core and contrib can be consistent in how they call it.

"Machine" sounds very 19th century don't you think?

I uaually replace the text with "unique identifier", followed by the valid patterns, but system sounds more reasonable than machine IMO.