documenting objects

[stefanseefeld]

BPO	35449
Nosy	@rhettinger, @terryjreedy, @stefanseefeld, @stevendaprano, @tirkarthi

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-12-09.17:41:42.454>
labels = ['type-feature', '3.8']
title = 'documenting objects'
updated_at = <Date 2019-03-23.19:09:33.155>
user = 'https://github.com/stefanseefeld'

bugs.python.org fields:

activity = <Date 2019-03-23.19:09:33.155>
actor = 'xtreak'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2018-12-09.17:41:42.454>
creator = 'stefan'
dependencies = []
files = []
hgrepos = []
issue_num = 35449
keywords = []
message_count = 10.0
messages = ['331443', '331453', '331454', '331455', '331457', '331458', '331461', '331474', '331501', '331868']
nosy_count = 5.0
nosy_names = ['rhettinger', 'terry.reedy', 'stefan', 'steven.daprano', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35449'
versions = ['Python 3.8']

[stefanseefeld]

On multiple occasions I have wanted to add documentation not only to Python classes and functions, but also instance variables. This seems to involve (at least) two orthogonal questions:

1. what is the proper syntax to associate documentation (docstrings ?) to objects ?
2. what changes need to be applied to Python's infrastructure (e.g., the help system) to support it ?

I have attempted to work around 1) in my custom code by explicitly setting an object's __doc__ attribute. However, calling help() on such an object would simply ignore that attribute, and instead list the documentation associated with the instance type.

Am I missing something here, i.e. am I approaching the problem the wrong way, or am I the first to want to use object-specific documentation ?

[stevendaprano]

Minor note on terminology: classes and functions are themselves objects.

I think that help() (or in particular PyDoc in general) should support any instance with a __doc__ attribute. Its failure to do so is causing pain, see bpo-12154.

[stefanseefeld]

Exactly ! I'm fully aware of the ubiquity of objects in Python, and it is for that reason that I had naively expected pydoc to simply DoTheRightThing when encountering an object containing a __doc__ attribute. rather than only working for types and function objects.

OK, assuming that this is a recognized bug / limitation, it seems easy to address.

Is there any discussion concerning what syntax might be used for docstrings associated with objects ? (There seem to be some partial solutions added on top of the Python parser (I think epydoc offered one), but it would be nice to have a built-in solution to avoid having to re-invent wheels.

[stevendaprano]

Is there any discussion concerning what syntax might be used for
docstrings associated with objects ?

I don't know about PyDoc in general, but I would expect help(obj) to
just use obj.doc which will return the instance docstring if it
exists, and if not, the type docstring (if it exists). No new syntax is
required, the standard help(obj) is sufficient.

(There seem to be some partial
solutions added on top of the Python parser (I think epydoc offered
one), but it would be nice to have a built-in solution to avoid having
to re-invent wheels.

Are you suggesting we need new syntax to automatically assign docstrings
to instances? I don't think we do.

I expect that if you want to set a custom instance docstring, you would
just say instance.__doc__ = "The doc string" after creating the
instance, or self.__doc__ = "..." inside the init method.

[stefanseefeld]

On 2018-12-09 18:35, Steven D'Aprano wrote:

Steven D'Aprano [email protected] added the comment:

Is there any discussion concerning what syntax might be used for
docstrings associated with objects ?
I don't know about PyDoc in general, but I would expect help(obj) to
just use obj.doc which will return the instance docstring if it
exists, and if not, the type docstring (if it exists). No new syntax is
required, the standard help(obj) is sufficient.

That's why I distinguished between points 1) and 2) in my original mail:
The syntax is about how certain tokens in the parse tree are associated
as "docstring" with a given object (i.e., point 1), while the pydoc's
behaviour (to either accept any __doc__ attributes, or only those of
specific types of objects) is entirely orthogonal to that (thus point 2).

I now understand that the current pydoc behaviour is considered
erroneous, and it sounds like a fix would be simple and focused in scope.

(There seem to be some partial
solutions added on top of the Python parser (I think epydoc offered
one), but it would be nice to have a built-in solution to avoid having
to re-invent wheels.
Are you suggesting we need new syntax to automatically assign docstrings
to instances? I don't think we do.

No, I'm not suggesting that. I'm suggesting that within the current
syntax, some additional semantic rules might be required to bind
comments (or strings) to objects as "docstrings". For example:

foo = 123

"""This is foo's docstring"""

might be one convention to add a docstring to a variable.

foo = 123

# This is foo's docstring

might be another.

None of this is syntactically new, but the construction of the AST from
the parse tree is. (I have seen both of these conventions used in custom
tools to associate documentation to variables, which of course requires
hacking into the parser internals, to add the given docstring to the
object's __doc__ attribute.

It would be great to establish a convention for this, so in the future
tools don't have to invent their own (non-portable) convention.

[tirkarthi]

There was a related proposal in https://www.python.org/dev/peps/pep-0258/#attribute-docstrings

[stefanseefeld]

On 2018-12-09 19:48, Karthikeyan Singaravelan wrote:

There was a related proposal in https://www.python.org/dev/peps/pep-0258/#attribute-docstrings

Right, but that was rejected (for unrelated reasons). The idea itself
was rejected by Guido
(https://www.python.org/dev/peps/pep-0224/#comments-from-our-bdfl), and
I'm not aware whether anyone has addressed his concerns by proposing a
different syntax.

It's sad, as right now there doesn't appear to be any way to address
this need...

Stefan

--

  ...ich hab' noch einen Koffer in Berlin...

[stevendaprano]

I asked:

> Are you suggesting we need new syntax to automatically assign docstrings
> to instances?

Stefan replied:

No, I'm not suggesting that.

And then immediately went on to suggest new syntax for automatically
binding a string to objects as docstrings. I am amused :-)

Whether you want to call it "new semantics for existing syntax" or "new
syntax" is a matter of terminology. The point is, you are suggesting
something that requires dedicated support from the interpreter, as
opposed to merely writing some Python code.

I'm suggesting that within the current
syntax, some additional semantic rules might be required to bind
comments (or strings) to objects as "docstrings".

To my mind, it makes sense to have dedicated docstring syntax for
classes, modules and functions: I expect that they will make up in
excess of 95% of use-cases for docstrings. In this case, special cases
*are* special enough to change the rules.

But for the rare(?) cases of wanting to add docstrings to arbitrary
instances, I don't think it is justified to have dedicated syntax to do
it. It's easy enough and more than flexible enough to just do an
instance attribute assignment:

    instance.__doc__ = """This is the instance docstring."""

I strongly oppose any suggestion that *comments* be treated as code:

foo = 123

This is foo's docstring

Deleting comments shouldn't make have any runtime effect on the code,
but in this case it would. (I'm willing to make an exception for the
optional encoding cookie at the beginning of modules, as a rather
special case.)

So I think there are three related but separate issues here:

1. help(obj) and possibly PyDoc in general ought to support per-instance
   docstrings. I think that is uncontroversial and we just need somebody to
   do the work to make it happen.

2. Add special syntactic sugar to automatically associate a string with
   arbitrary instances as a docstring. I think that's overkill and
   unnecessary, but I'd be willing to be convinced otherwise.

3. Your examples suggest that even built-in immutable objects like ints
   should be able to take docstrings. I don't think that idea is going to
   fly, but I'm also not sure how serious you are about that. It runs into
   the problem that small ints and certain strings are cached, so your
   docstring could clobber my docstring. It is also going to require the
   addition of an extra attribute slot to every int, str, etc for the
   docstring, even if 99.999% of them never use it.

It would be great to establish a convention for this, so in the future
tools don't have to invent their own (non-portable) convention.

instance.__doc__ = "docstring" seems pretty portable to me :-)

[stefanseefeld]

ad 3) sorry, I picked a bad example - I didn't mean to suggest that immutable objects should in fact become mutable by modifying their __doc__ attribute.

ad 1) good, glad to hear that.

ad 2) fine. In fact, I'm not even proposing that per-instance docstring generation should be "on" by default. I'm merely asking whether the Python community can't (or even shouldn't) agree on a single convention for how to represent them, such that special tools can then support them, rather than different tools supporting different syntax / conventions.

[terryjreedy]

The reason that modules, classes, and functions need a special rule for assigning the .__doc__ attribute is that one cannot get a reference to the module, class, or function within the body of its definition. And putting the docstring at the top of a file or after a header is usually the best place.

For modules, the body is the file, leaving nowhere else to put the docstring. For classes and functions, the alternative of an assignment elsewhere, after the object is created, remains.

>> def f(): pass

>>> f.__doc__ = 'Docstring outside f body'
>>> help(f)
Help on function f in module __main__:

f()
    Docstring outside f body

This alternative is used in functools.partial and decorators that wrap functions with a function and then copy the original docstring to the wrapper.

I think object.doc = 'docstring' is sufficient for other objects and that PEP-224 was wrong to propose otherwise and should have been rejected.

So I think that this issue should propose what Steven said: pydoc/help() should be simplified to fetch object.__doc__ with normal lookup, instead of bypassing object if not one of the special types. Stefan Seefeld, can you try patching pydoc to do this?

If one wants to add docstrings to builtins, a subclass can work.

>> class Docint(int): pass

>>> i,j = Docint(1), Docint(2)
>>> i+j
3
>>> i.__doc__ = 'one'

We just need help to print i.__doc__ instead of int.__doc__.

[iritkatriel]

This was done here: #84438.