Thursday, June 21, 2007

Thoughts on Documentation: Quality

Recently I've been working on my Google Summer of Code project dcov (a Ruby documentation analyzer) a lot. It's basically broken into two large functions: analyzing your documentation and reporting on it. I don't have to do much on reporting (thanks Ruport!), but on the analysis side I'm basically making it up as I go. The first part of the analysis is quantity: is something documented? Right now, this works famously: it can give you a precise precentage of your functional unit documentation coverage. This is easy, of course, because all it has to do is check for the presence of a comment. The second part is quality, and that's where the analysis gets a little hairy.

There are no documentation standards for Ruby/Rails really (I'm working on an entry on this...so bare with me), and so I'm sort of left to my own devices when analyzing quality. The plan is to allow users to write their own analyzers and thus let them create their own quality standards, but it would be nice to include a "default" quality analyzer. The problem though is not what it should do, but how?

Quality is so subjective, that if we're to do a midline, objective analysis, a standard needs to be developed or some sort of criteria needs to be met. This entry is little more than a plea to you guys and gals: help me create one! What do you think makes good quality API documentation? I'll reconcile your responses into something workable in a future entry and that will become dcov's standard criteria set (until we decide to change it ;)). Here are a few of my ideas:

  • Parameters should at least be mentioned.

  • Options hash must be documented if present.

  • Default values must be explained.

  • A return value description should also be present, if there is a return value and it is not apparent.

  • Examples! Examples! Examples!


Let me know what you'd like to see.

5 comments:

Jamie said...

I think there are 2 kinds of consumers for documentation: future maintainers, and library users. Future maintainers benefit greatly from code that reads like pseudocode; library users don't see the source and need lots of extra words basically explaining that the unseen source does.

So, keep in mind that code can be its own documentation for some folks, not for others. Comments not extracted by rdoc only benefit some folks.

The BDD folks will also point out that tests (a whole nuther file to parse and consider when talking about the docuemented-ness of a source file) are documentation of what the code they're calling is supposed to do. Again this is less useful if you can't read the BDDified test source code, or a documentation format generated from that code.

manveru said...

I've been working on a way to do doctests in ruby, the code itself isn't documented (my apologies) since i only did it as a kind of proof-of-concept, but it has quite some appeal to me as a way to ensure docs really match reality when it comes to examples.
See the source here: testdoc source
The specs explain how to use it.

I have a newer version on my notebook that supports setting up contexts (kind of 'before :all do') before running the testdocs. Will push that a bit later.

Al Hoang said...

Don't forget a simple README and introduction that explains how to start using the project.

I'm continuously appalled at most documentation being a link to some blog post somewhere else rather than having a decent README that explains what the library is and a nice trivial example to help get you started playing with the codebase.

Jan Aerts said...

We've made an effort in the last year to try and document the bioruby package as good as possible. With a standardized way of documenting. The README.DEV in the package explains what format to use to document classes, methods, return values, ... See here.

dwilkins said...

I think that eventually, the tool will need to maintain a documentation quality metric. This can probably be tied to the number of times the documentation is updates / number of times the source is updated. Code commit analysis can probably yield the raw changes.

The real problem (IMHO) with documentation quality is that it's at it's highest after the code is written initially. The documentation frequently gets ignored as the code is maintained.