how does Quiz work?

I am new to using Quiz as well but I'll answer part of your question and maybe someone more familiar with it will add to it.

First the Quiz output to the user is a form. Meaning that what the student answering a Quiz sees in the browser is a form, and I believe Quiz refers to that in the code as "Answer Form". And yes that form is created by the Drupal Form API. So it can be altered by your own custom module(s) using available form hooks.

The different question types that Quiz offers such as Multichoice, Long Answer and so on, are each sub-modules that extend Quiz. And because Quiz uses some classes for handling Quiz questions/answers, those question types mostly inherit and add to the classes from the Quiz core. So one of those class function is used to create the answer form shown to the student when taking the test.

As far as the next part of your question, well I think first you have to explain what your trying to do with Quiz. Because what knowledge you need depends on what you want to achieve. If you wanted to change basic theming for example, you might simply need to add CSS styles to your theme stylesheet, or if you want to change layout you might need to override Quiz theme templates. If you want to make a structural change to how something works, then you would probably need to make a new module that adapts/extends Quiz.

As an example of something you can do with Quiz, if you know PHP and module development fairly well, is to create a custom Question Type. My site for example needed a Multichoice style question type, but with an image field and an audio track. So the student is supposed to listen to the audio, view the image, then answer the question. I was able to fairly easily copy the existing Multichoice question type, change the module name to make it my own custom module, then add those custom fields to it.

On my project, the next challenge is how do we change the rendering of the quiz questions enough to actually show 3 different questions at once, which is one unusual requirement certain English tests have.