Re: As big a problem is the missing holy book
> I despair when I look at most software projects with documentation automatically generated from the code because all it tells me is how the software developer chose to decompose the problem into units of code
Glad you said "most", not "all", because it *is* possible to use something like Doxygen to generate documentation that runs the gamut from end-user to bug-fixing or feature enhancing dev. I *think* I managed it once (once!)[1]: well, the client liked it at the long handover meeting and nobody ever came back to grab me for a quick shout in the following years...
BUT using (in this case, Doxygen) like that was, I admit, very much going against the grain of how Doxygen itself is presented (and the same is true of similar tools I've looked at): they go into detail about how to mark up your code *but* there is little, if any, emphasis on writing SEPARATE files that are just documentation - to the extent I've been asked "what was the point of these dot-dox files, they only make the build system more complicated?"! Bangs head on table. "Why not use Word?" Um, version control; keeping materials together; regenerating and fixing references as the code matures; being able to drop "TODO: write this dox" into the same tracker as we drop "TODO: write this function"; the docs are broken into easy chunks so you aren't constantly looking at hundreds of pages of Word content in a disheartening and never ending task; auto-generating variants (End User Manual is done in PDF without hyperlinks to any code; Installer's Manual in PDF hyperlinks the User features to the INI files, Dev's Manual in HTML has all the same, but now hyperlinks to where the INI settings are acted upon in code...).
Getting just devs to write docs in "the code markup tool" is hard enough, but getting management to allow it, yikes. You can fudge things, sometimes[2] but, again, the tools (that I've seen, so far) don't make it easy.
And "by easy" I don't mean "write another bleeping WYSIWYG IDE that does it all"! You can sensibly allow "I am not a dev" authors to use, say, a WYSIWYG editor for writing Markdown that can then be pulled into the dot-dox, that is good (great, even). But you also need to be able to drive the generation from a build system (GUI begone!), so encourage long and meaningful file name extensions[3]; you need to pull in externals (images, auto-generated lists of CLI options, ...)[4], must support multiple programming languages (without still being so obviously a Python tool that got hacked for C++ and Bison that has those guys foaming at the mouth).
Most important of all:
When you go to the website for the tool (looking at you, Sphinx, Doxygen - well, everyone) and the "how to use this", please, please, START by showing how to write the User docs, the Overview Of My Library, even The Next Great American Novel, as organised prose; output into lots of language lovely End User and Management-friendly formats. THEN segue over to cross-referencing your example code. THEN into adding details into the bulk of your code! That last bit really, really is the least important, honest. Management may leave love to see a table of all the parameters of flump19() but if you haven't got that far yet (as it is last on your list!) a direct copy of just the declaration is a usable standin for anybody who'll actually use that level of the docs! Go back to the history of *when* that struct has five fields, not three, and the arguments that'll at least mean you can say "well, you were warned" when Mr SuperNewbie "optimises" your code and it all stops working!
[1] important note: I had time *scheduled* on this project for a write-up *and* management wasn't demanding it be done in Word (phew), so there was an opportunity.
[2] my little utility for a particular End User doc generates Word files, as per Management demands. Well, to be honest, it generates HTML that looks like the stuff (an old) Word spat out and gives it a dot-dot extension...
[3] seriously, I encourage the use of fred.markdown.txt, or even better, fred.github_markdown.txt! *Any* user can double-click and read it in Notepad, any *sensible* Markdown editor can open them (AND prompt for that as the extension when saving a file! Your's can't? Then it is in the wrong!) Make loves those names, gets all the options set correctly.
[4] sigh, ok, I'll add a pass through another preprocessor into the build - getting further away from one nicely documented document-writing language as we go