Skip to content

Overhaul Lifetimes Section (was "Help Needed: Extend lifetimes section") #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 13 commits into from

Conversation

vext01
Copy link

@vext01 vext01 commented Mar 16, 2017

Hi,

As mentioned in #2, I'm trying to better document the lifetimes system of Rust, as it's the part all beginners trip on.

I was trying to combine the collective knowledge of what the nomicon already tells us about desugraing to see implicit lifetimes, and the intuition given by the answer of my stack overflow question (http://stackoverflow.com/questions/42807679/clarify-the-meaning-of-binding-two-references-to-differently-scoped-referents-to/42808207#42808207).

All was going well until I tried to show a counter example where the lifetime constraints imposed by a function signature can't be satisfied (the last rust code block added in this PR). At this point something has gone wrong, since I am unable to come to the same conclusion as the rust compiler. The compiler rejects the program, the intuitive method I am trying to explain within this PR does not reject the program!

Can an expert take a read of the stack overflow question, then the change in this PR and comment on where I have gone wrong?

  • Am I de-sugaring correctly?
  • Are the rules for converting lifetimes correct?
  • Are the rules for detecting an inconsistency correct?

I really wonder if there is a better way to express how the compiler is assigning and checking lifetimes, as it's not very intuitive as it stands. Before reading the nomicon, I've always though of the lifetime of a reference as the lifetime of the referent, but the nomicon seems to explain it in another way.

Cheers

@vext01
Copy link
Author

vext01 commented Mar 16, 2017

(excuse the typos in the commit messages, I will squash this all into one nice commit at the end)

@steveklabnik
Copy link
Member

(excuse the typos in the commit messages, I will squash this all into one nice commit at the end)

No worries at all!

@vext01
Copy link
Author

vext01 commented Mar 16, 2017

I think the problem here is the rules for "converting" lifetimes. I don't think it helps to think in that way.

Given the de-sugared code:

fn main() {
    'a {
        let s1 = String::from("short");
        'b {
            let res: &'b str;
            'c {
                let s2 = String::from("a long long long string");
                'd {
                    // Annonymous scope for the borrows of s1 and s2
                    // Assigning to the outer scope causes s1 and s2 to have 'b
                    res: &'b = shortest(&'d s1, &'d s2);
                    println!("{}", res);
                }
            }
        }
    }
}

fn shortest<'k>(x: &'k str, y: &'k str) -> &'k str {
    ...

The way my colleague see this is, the lifetimes are inconsistent because:

  • The sig of shortest() tells us the return value has the same lifetime as the two args.
  • The args were marked &'d by de-sugaring, so we must show that the return value lifetime, &'d, is consistent with the lifetime we annotated earlier: &'b.
  • To be consistent, &'d would have to live longer than &'d, which it does not, so after the call res may refer to a freed value.

Therefore the program is inconsistent. Makes sense to me, but there might be a more intuitive way of thinking about this.

@vext01
Copy link
Author

vext01 commented Mar 17, 2017

Alright. Can someone with jedi rust skills check the most recent version.

I've borrowed (no pun intended) the style of this post, which makes a great deal of sense to me.

If you guys like this, then I would then need to go back and revise the prior sections to mark references with the lifetimes of their referents, and I would probably want to adjust the flow a bit too.

There's also the issue of (I think) move semantics giving rise to additional gotchas (since they limit the lifetimes of values), but I think I'd like to tackle that as a separate PR.

@steveklabnik Are you able to review this, or do you know someone who could?

Thanks.

@vext01 vext01 changed the title Help Needed: Extend lifetimes section Overhaul Lifetimes Section (was "Help Needed: Extend lifetimes section") Mar 17, 2017
@vext01
Copy link
Author

vext01 commented Mar 17, 2017

(I'm unable to test under mdbook, due to ogham/rust-ansi-term#27, so I've been just pushing and viewing in GH ;) )

@steveklabnik
Copy link
Member

@vext01 I will try to soon, but in addition, @aturon and I have been talking about having lang team / docs team coordination to work on the reference; we may want that for the nomicon as well.

@vext01
Copy link
Author

vext01 commented Mar 17, 2017

OK great. Let me know if I can be of any help, as I'm keen to learn about this stuff.

@vext01
Copy link
Author

vext01 commented Mar 27, 2017

Sorry to persist, but has anyone at moz had a chance to review this?

@steveklabnik
Copy link
Member

steveklabnik commented Mar 27, 2017

I have not, sorry :( I should be able to at least give it a cursory review soon; still have to figure out those meetings

@aturon
Copy link
Member

aturon commented Mar 27, 2017

cc @gankro

src/lifetimes.md Outdated

Consider the following program:

```rust,ignore
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason why this is ignore? Seems like it should be compilable and runable, no?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same with some of the other examples here

src/lifetimes.md Outdated

```rust,ignore
print_shortest(&s1, &s1);

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: unnecessary newline here

src/lifetimes.md Outdated
}
```

Again, at the call-site of `shortest` the comipiler needs to check the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: comipiler

src/lifetimes.md Outdated

Again, at the call-site of `shortest` the comipiler needs to check the
consistency of the arguments in the caller with the signature of the callee.
The signature of `shortest` fisrt says that the two reference arguments have
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: fisrt

Copy link
Member

@steveklabnik steveklabnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some nits

src/lifetimes.md Outdated
}
```

`print_shortest` simply prints the shorter of its two pass-by-reference
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"simply" should be taken out

src/lifetimes.md Outdated
s1` with `&'s2 s2`. After this both arguments are of lifetime `&'s2` and the
call-site is consistent with the signature of `print_shortest`.

[More formally, the basis for the above rule is in *type variance*. Under this
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this would be all in >s, not [].

src/lifetimes.md Outdated
variance isn't strictly required to understand the Rust borrow checker. We've
tried here to instead to explain using intuitive terminlolgy.]

# Inter-procedural Borrow Checking of Function Return Values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be a smaller heading than #

@vext01
Copy link
Author

vext01 commented Apr 5, 2017

Thanks everyone for the comments.

Formatting aside, is what I have written correct? Notice how, when de-sugaring, I initialise references with the lifetime of the referent, whereas the previous examples do something slightly different, often using a narrower lifetime.

Of course, a narrower lifetime can be fine (and probably beneficial, if it can be shown valid), but I wonder if that can be considered a separate "optimisation" that we don't mention here. I think it's important that we have a single intuitive narrative for explaining this to new users. I can't understate this. Lifetimes are what make Rust unique, but are also very hard to grasp (I'm still grasping myself to be honest).

So assuming my de-sugaring is correct (I hope someone checked, because I'm really very new to this), should I go back and revise the earlier examples to fit this style too?

Thanks

@vext01
Copy link
Author

vext01 commented May 6, 2017

Anyone? I was hoping this kind of PR would be prioritised, since the borrow checker is the one thing that no newcomer understands for a long time.

@Mark-Simulacrum
Copy link
Member

Sorry about that! We'll try to get you a review soon.

cc @steveklabnik @frewsxcv

@steveklabnik
Copy link
Member

steveklabnik commented May 6, 2017 via email

@steveklabnik
Copy link
Member

steveklabnik commented May 15, 2017

Hey @vext01;

Sorry again about the extreme delay here; it has nothing to do with you and everything to do with me.

so, i was talking with @nikomatsakis and/or @aturon about how we can work through things like this; I think the plan will be to eventually go through everything with them, rather than looking at individual PRs. So, I don't think we need actual lang team signoff here.

I think that this is all correct, but it's not my area of specialty. It's unclear to me who should actually sign off on it for that reason :/ Maybe @gankro himself?

@Gankra
Copy link
Contributor

Gankra commented May 16, 2017

So I'm a bit concerned that there seems to be some redundancy with other sections. For instance https://doc.rust-lang.org/nomicon/subtyping.html discusses this merging behaviour a bit already (though it comes later).

I need to re-read this chapter to see what gets discussed where.

src/lifetimes.md Outdated
let s2 = String::from("a long long long string");
print_shortest(&s1, &s2);
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing }

src/lifetimes.md Outdated
let s2 = String::from("a long long long string");
print_shortest(&'s1 s1, &'s2 s2);
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing }

src/lifetimes.md Outdated
*caller* (`main`) are consistent with the lifetimes in the signature of the
*callee* (`print_shortest`).

The signature of `print_shortest` simply requires that both of it's arguments
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kill all "simply"s

@vext01
Copy link
Author

vext01 commented May 17, 2017

I've fixed the comments on my new prose.

Outstanding topics we need to discuss (quoting my earlier comments):

Notice how, when de-sugaring, I initialise references with the lifetime of the referent, whereas the previous examples do something slightly different, often using a narrower lifetime.

There's also the issue of (I think) move semantics giving rise to additional gotchas (since they limit the lifetimes of values), but I think I'd like to tackle that as a separate PR.

We might decide to do these in separate PRs, as this PR will start to bitrot.

I'm glad the rust team are starting to think about how to deal with docs. I can't emphasize enough the importance of documenting rust, since it has radical new ideas which even lifelong programmers (including myself) are struggling to put to work. You need to make it as easy as possible to learn rust, and documentation is all you've got! :)

If you keep me in the loop, I am happy to review/comment of PRs. I hope I can offer the "newbie's-eye-view".

@vext01
Copy link
Author

vext01 commented May 17, 2017

Fixed build.

@steveklabnik
Copy link
Member

@gankro when you're happy with this, I think I'm happy with this 😄

src/lifetimes.md Outdated
(`s2`).

> Formally, function return values are said to be *contravariant*, the opposite
> of *covariant*.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the argument/return position might be a bit of a red herring here.

I wouldn’t use the term contravariant to refer to return types, at least not in this sense. They appear so because a return type was compared to an argument type, but if you compared a return type to another return type they would appear covariant just as well!


The lifetime coercion (subtyping) rule is simply &'a T -> &'b T provided that 'a outlives 'b. Argument/return type doesn’t play a role here.* The compiler is allowed to shrink the lifetime of a reference, but never extend it. Formally speaking, reference types are said to be “lifetime-covariant”.

Another rule is that an object must always outlive its binding. This means you can’t, in a variable of a larger scope, store a reference to an object of a smaller scope:

'r_scope {
    let r: &'r_ref;
    'inner {
        let inner = String::new();
        r = &'r_ref inner; // doesn't work because 'r_ref ≥ 'r_scope > 'inner
                           // yet 'r_ref ≤ 'inner
    }
}

From the first example, you learned that 'k has to be 's2 (or shorter). This remains true for the second example.

In the second example, the returned reference has lifetime 'k. When it gets stored in res, the compiler may shrink 'k if necessary. Therefore the lifetime 'res_ref of the stored reference must be 'k or shorter. Here, I used 'res_ref to distinguish between the lifetime of the reference res: &'res_ref str, and the lifetime of the scope itself 'res_scope { let res; ... }.

From the structure of the code itself, we know that the scope of s2 is contained fully within that of res. Because an object must always outlive its binding, it follows that 'res_ref must outlive 'res_scope, which is strictly longer than 's2.

If we combine these three facts together,

  • 'k must be 's2 or shorter
  • 'res_ref must be 'k or shorter
  • 'res_ref must be strictly longer than 's2

we see that there can be no solution that satisfies all three constraints :(

There is a way to fix it though, by making 's2 and 'res_scope equal:

fn main() {
    let s1 = String::from("short");
    let (res, s2);
    s2 = String::from("a long long long string");
    res = shortest(&s1, &s2);
    println!("{}", res);
}

--

* Due to variance, the rule changes for more complicated for composite types like Cell<&'a ()> or fn(&'a ()) -> &'b () but that doesn't apply here.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn’t use the term contravariant to refer to return types, at least not in this sense. They appear so because a return type was compared to an argument type, but if you compared a return type to another return type they would appear covariant just as well!

I'm not sure I understand what you mean. Can you expand on this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compare these two statements:

Rule I:

A function argument of type &'p T can be coerced with an
argument of type &'q T if the lifetime of &'p T is equal or longer than &'q T.

Rule II:

The return value of a function &'r T can be converted to an argument &'s T if the lifetime of &'r T is equal or shorter than &'s T.

Rule I compares arguments vs argument. Rule II is return value vs argument. If you compare things in opposite positions, you will always the opposite result, so Rule II is just a corollary to Rule I.

Moreover, I don't think "co-/contravariant" is the appropriate term for this. Formally, we say F is covariant in its parameter if and only if A ⊆ B implies F<A> ⊆ F<B>, and contravariant if and only if A ⊆ B implies F<A> ⊇ F<B>. There is no mention of argument vs return types at all.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the language "The return value of a function &'r T can be converted to an argument" feels a bit odd.

However, I personally think that the type co/contra-variance is a useful thing to point out to people a) because it's surprisingly unintuitive (indeed, there are real programming languages that have got this wrong in their type systems) b) because that can be used as a hook to explain the relationship to lifetimes. However, one doesn't necessarily need to use those words provided one gets the concept across somehow. What I think the document is trying to say is "if a function takes a parameter of type T and returns a type U, you can pass a (non-strict) supertype T' and receive back a (non-strict) supertype U' " and then making the link between that and the length of lifetimes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean "if a function takes T and returns U, you can pass a subtype T' and supertype U'"?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite correct -- mea culpa!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's my concrete suggestion for this PR:

  1. Rewrite Rule I (L289-290) to say something along the lines of "A reference can always be shrunk to one of a shorter lifetime. In other words, &'a T can be implicitly converted to &'b T as long as 'a outlives 'b."

  2. Replace "for function arguments to be co-variant" (L299) with "a reference &'a T is always covariant in its lifetime parameter 'a."

  3. Replace Rule II and adjacent paragraphs (L348-372) with something like:

    Again, at the call-site of shortest the compiler needs to check the consistency of the arguments in the caller with the signature of the callee. The signature of shortest says that all three references must have the same lifetime 'k, so we have to find a lifetime 'k such that:

    • &s1: &'s1 str can be converted to the first argument &'k str,
    • &s2: &'s2 str can be converted to the second argument &'k str, and
    • the return value &'k str can be converted to res: &'res str
    res: &'res = shortest::<'k>(&'s1 s1, &'s2 s2);
              //                ^^^^^^^
              //               's1 -> 'k ^^^^^^^
              //                        's2 -> 'k
              // ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              //           'k -> 'res

    This leads to three requirements:

    • 's1 must outlive 'k,
    • 's2 must outlive 'k, and
    • 'k must outlive 'res.

    However, 'res strictly outlives 's2 by virtue of being a larger scope that encloses s2, so it's impossible to satisfy these conditions. The borrow checker rightfully rejects our program because we are making a reference (res) which outlives one of the values it may refer to (s2)."

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I'll make these edits (and see if it's understandable to a beginner) when i find some time.

Thanks for your comments Rufflewind.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vext01 any chance you can find the time sometime soon?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to find time this week.

I've learnt more Rust since I last looked at this, so I may have some edits of my own.

@vext01
Copy link
Author

vext01 commented Nov 27, 2017

I've implemented Rufflewind's recommended changes, and read the whole thing over.

I shortened a fair amount of my prose, as I felt I had been rambling a bit.

Let me know what you think.

@vext01
Copy link
Author

vext01 commented Nov 27, 2017

btw, I killed the stuff on variance, as I felt it was confusing and vague.

@Rufflewind
Copy link
Contributor

I think the revised version look great!

@vext01
Copy link
Author

vext01 commented Nov 28, 2017

Shall I squash?

@vext01
Copy link
Author

vext01 commented Dec 5, 2017

ping

@vext01
Copy link
Author

vext01 commented Jan 5, 2018

I think someone should make it their new year's resolution to merge this :)


> A reference can always be shrunk to one of a shorter lifetime. In other
> words, `&'a T` can be implicitly converted to `&'b T` as long as `'a`
outlives `'b.`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this line have a > on it too? (Github's rendering adds it to the blockquote just fine, but i'd rather have it all neat. >_>)

Copy link
Contributor

@Gankra Gankra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So first off: I apologize greatly for having kicked this ostensibly simple review down the road for so long. I've been a mess for the last year and this is one of the things I avoided as a result.

Ultimately I've avoided this for a few reasons:

  • I just don't understand the motivation for this change; or at least it doesn't feel like the nomicon is the right place for this.
  • It's extending the scoped lifetime illustration which has never been great
  • All discussions of lifetimes are a mess as a result of non-lexical lifetimes, and this entire section probably needs to be rewritten to focus on the new system (and this addition just exacerbates the situation, I think)

As such I'm inclined to cut our losses and just close this. It absolutely sucks that you have been strung along this long by my negligence, but it happened. I'm sorry.

@@ -1,6 +1,6 @@
# Lifetimes

Rust enforces these rules through *lifetimes*. Lifetimes are effectively
Rust ensures memory safety through *lifetimes*. Lifetimes are effectively
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels too strong, (lifetimes are a single piece of the puzzle that comes together for mem safety), but this is probably pedantry.

@@ -213,3 +213,142 @@ totally ok*, because it keeps us from spending all day explaining our program
to the compiler. However it does mean that several programs that are totally
correct with respect to Rust's *true* semantics are rejected because lifetimes
are too dumb.

# Inter-procedural Borrow Checking
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've never seen this hyphenated; drop it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also maybe a Bad Title, because borrowchecking is specifically not interprocedural.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it's the wrong term, but I was referring to the way that the arguments at call-sites must be checked against a function definition, so in some way lifetimes do cross the function boundary. If you see what I mean?

```

(for brevity, we don't show the third implicit scope that would be introduced
to limit to lifetimes of the borrows in the call to `print_shortest`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the" lifetime.

Regardless, It seems like this can either be cut, or is Actually Very Important?

they refer to were introduced in different scopes. At the call site of
`print_shortest` the compiler must now check that the lifetimes in the
*caller* (`main`) are consistent with the lifetimes in the signature of the
*callee* (`print_shortest`).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not how the lifetime system works. It's a constraint solver; they never have "wrong" lifetimes, and the lifetimes aren't checked if they're "right". It just tries to solve the system of constraints and either reports success or failure.

@vext01
Copy link
Author

vext01 commented Jan 17, 2018

Hi @gankro!

Thanks for looking at my PR.

I just don't understand the motivation for this change; or at least it doesn't feel like the nomicon is the right place for this.

I started working on this because I didn't understand how lifetimes work behind the scenes. Although I've gotten better at Rust since then, there are still times where I "brute force" my way past cryptic lifetime errors offered up by the compiler. And it's not just me, I occasionally see my friends and colleagues (experienced programmers by the way) struggling in the same way.

I think this is indicative of a problem with the mental model which non-expert Rust programmers have when writing Rust code. I also think that we need to find an intuitive way to describe what is really going on here, because guessing how to fix these weird errors just doesn't satisfy my curiosity.

It's extending the scoped lifetime illustration which has never been great.

Yes agreed.

All discussions of lifetimes are a mess as a result of non-lexical lifetimes, and this entire section probably needs to be rewritten to focus on the new system (and this addition just exacerbates the situation, I think)

Yeah, I've seen the RFCs about non-lexical lifetimes.

I apologize greatly for having kicked this ostensibly simple review down the road for so long.
...
It absolutely sucks that you have been strung along this long by my negligence, but it happened. I'm sorry.
...
As such I'm inclined to cut our losses and just close this.

Don't sweat it! I completely understand. Sometimes these things don't work out. And in this instance I'd rather we get it right than cling on pretending that my precious code is helping :)

Before I close this, can I recommend that we make an issue on this (or another) repo that basically says "find a more intuitive way to explain lifetimes to users". If you let me know which repo, I will happily raise the issue.

You might consider describing the constraint solver in this repo though? I am happy to help reviewing if you can write the prose. [I have experience with linear constraint solving and SAT].

Thanks 😄

@vext01
Copy link
Author

vext01 commented Feb 9, 2018

Ping?

@vext01
Copy link
Author

vext01 commented Mar 15, 2018

I'm going to close this as there seems to be a lack of interest/time.

@vext01 vext01 closed this Mar 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants