Direct database querying: pros and cons

This is the right place, and it is a great debate to have. In fact, your point about whether node/user load return already themed values is a really clear-headed tip on how one can vet code for bad smells.

Meh, I didn't get back to you directly, it being a wiki with no notification and all, and I forgot to check back after a few days. And now IE doesn't persist cookies on this groups site (?) Anyway, we're here now.

I appreciate your decisions I guess, if you are thinking more about bulk imports. I don't see what the danger from other modules really is, but I guess you deserve to be able to pick and choose which processes you apply and which you don't. If you are doing your own process() or node_invoke() type thing when needed, it should be fine.
Some of them really are quite useful, like path or menu. Sure, you CAN rebuild the work they do, but I found it easier (in my import_html.module) to just set $node->path and then node_submit($node) . Remaking the DB calls for that job would have been so wrong.
Other ones, that really modify the content may be unwanted, but I don't work with any of those at the moment. Surely most of them run as filters anyway?
I imagined that node_load() and node_save() just did validation and serialization. It's node_prepare() and node_show() you need to avoid. OTTOMH.

When you start thinking in bulk, you need to also think about creating your navigation on-the-fly also, not to mention cross-referencing between sibling files you might be importing simultaneously (or even asynch). I had fun with that on my attempt. Have you given thought to those issues? finding relative images and links when it arrives on the other side?

.dan.

At the moment, I haven't got any solid ideas for how to handle attachments, so any brainwaves that you (or other people) have is most welcome! However, my thinking so far is that 'user files' (i.e. attachments, images, audio, etc) will require their own 'get' and 'put' engine (e.g. 'userfile'). In fact, this is the main reason why I have introduced the 'get and put API' into the module - because files are not stored in the database - they're stored on the filesystem - and so they require their own engines separate to the database engines. Since files in Drupal are also tracked in the 'files' table in the DB, however, they'll also need a corresponding entity whose primary engine IS the DB.

Like I said, a UI is technically out of scope for this project. However, I'm thinking that ultimately, the default UI for exporting attachments would consist of a 'download' link for each file; alternate UIs could do more advanced things, such as packing multiple attachments into a tarball that could then be downloaded (but which would probably require non-standard PHP libs/extensions, or 3rd party libraries - don't know, haven't researched such things). Similarly, the default UI for importing attachments would consist of a series of 'upload' file form fields for each file; and alternate UIs could do things such as accept a tarball and then unpack it.

BTW, I would recommend that you wait until I have written the definition for the base 'node' entity, and for the various core node types, before you start writing node import/export things that depend on this API. I haven't written all the core definitions yet, and for good reason too: the API is not yet stable enough! But when I do write the 'node' definition, I'm planning for 'node' to be one entity (which is not actually defined through hook_def(), but which is just defined in a stand-alone function), and for each node type to be a separate 'node_typefoo' entity (which grabs the base definition from the stand-alone function, adds to it, and then passes its definition along through hook_def). Nodeapi-like modifying of node type fields will be done, of course, through hook_def_alter().