I liked the following: ↷

An Unfortunate Experience with Rust

<style><br>/*pre {white-space: pre-wrap};*/<br>span.bold {font-weight: bold;}<br>span.red {color: red;}<br>span.green {color: green;}<br>span.blue {color: blue;}<br>span.cyan {color: cyan;}<br></style><br><br><p>I consider myself to be pretty experienced in Rust. I’ve been using Rust in personal projects since late 2016 and professionally since early 2021. I’ve prepared internal presentations to teach coworkers about borrow checking and about async programming. However, even I still occasionally run into issues fighting the Rust compiler, and the following is a series of particularly unfortunate issues I ran into at work this week.</p><br><br><h3 id="background">Background</h3><br><br><blockquote><br> <p><strong>Note:</strong> The following code examples are simplified in order to demonstrate the essence of the problem. Obviously, our actual code is a lot more complicated.</p><br></blockquote><br><br><p>We have an async trait <code class="language-plaintext highlighter-rouge">Foo</code> with a method <code class="language-plaintext highlighter-rouge">update_all</code> to process a list of items, which then calls the per-implementation <code class="language-plaintext highlighter-rouge">update</code> method on the trait.</p><br><br><div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[async_trait(</span><span class="err">?</span><span class="nd">Send)]</span><br><span class="k">trait</span> <span class="n">Foo</span> <span class="p">{</span><br> <span class="k">async</span> <span class="k">fn</span> <span class="nf">update</span><span class="p">(</span><span class="o">&amp;</span><span class="k">self</span><span class="p">,</span> <span class="n">item</span><span class="p">:</span> <span class="nb">String</span><span class="p">)</span> <span class="k">-&gt;</span> <span class="n">Result</span><span class="o">&lt;</span><span class="nb">String</span><span class="p">,</span> <span class="n">Error</span><span class="o">&gt;</span><span class="p">;</span><br><br> <span class="k">async</span> <span class="k">fn</span> <span class="nf">update_all</span><span class="p">(</span><span class="o">&amp;</span><span class="k">self</span><span class="p">,</span> <span class="n">items</span><span class="p">:</span> <span class="nb">Vec</span><span class="o">&lt;</span><span class="nb">String</span><span class="o">&gt;</span><span class="p">)</span> <span class="k">-&gt;</span> <span class="n">Result</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="nb">String</span><span class="o">&gt;</span><span class="p">,</span> <span class="n">Error</span><span class="o">&gt;</span> <span class="p">{</span><br> <span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="n">items</span><span class="p">)</span><br> <span class="nf">.map</span><span class="p">(</span><span class="k">move</span> <span class="p">|</span><span class="n">item</span><span class="p">|</span> <span class="k">async</span> <span class="k">move</span> <span class="p">{</span> <span class="k">self</span><span class="nf">.update</span><span class="p">(</span><span class="n">item</span><span class="p">)</span><span class="k">.await</span> <span class="p">})</span><br> <span class="nf">.buffered</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><br> <span class="nf">.try_collect</span><span class="p">()</span><br> <span class="k">.await</span><br> <span class="p">}</span><br><span class="p">}</span><br></code></pre></div></div><br><br><p>We then have several implementations of <code class="language-plaintext highlighter-rouge">Foo</code>, here represented by <code class="language-plaintext highlighter-rouge">FooImpl</code>, and another type <code class="language-plaintext highlighter-rouge">FooWrapper</code> which makes use of a boxed <code class="language-plaintext highlighter-rouge">dyn Foo</code> to do various things.</p><br><br><div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">FooImpl</span><span class="p">;</span><br><span class="nd">#[async_trait(</span><span class="err">?</span><span class="nd">Send)]</span><br><span class="k">impl</span> <span class="n">Foo</span> <span class="k">for</span> <span class="n">FooImpl</span> <span class="p">{</span><br> <span class="k">async</span> <span class="k">fn</span> <span class="nf">update</span><span class="p">(</span><span class="o">&amp;</span><span class="k">self</span><span class="p">,</span> <span class="n">item</span><span class="p">:</span> <span class="nb">String</span><span class="p">)</span> <span class="k">-&gt;</span> <span class="n">Result</span><span class="o">&lt;</span><span class="nb">String</span><span class="p">,</span> <span class="n">Error</span><span class="o">&gt;</span> <span class="p">{</span><br> <span class="nd">println!</span><span class="p">(</span><span class="s">"{:?} Updating {}"</span><span class="p">,</span> <span class="nf">time</span><span class="p">(),</span> <span class="n">item</span><span class="p">);</span><br> <span class="nn">tokio</span><span class="p">::</span><span class="nn">time</span><span class="p">::</span><span class="nf">sleep</span><span class="p">(</span><span class="nn">tokio</span><span class="p">::</span><span class="nn">time</span><span class="p">::</span><span class="nn">Duration</span><span class="p">::</span><span class="nf">from_millis</span><span class="p">(</span><span class="mi">100</span><span class="p">))</span><span class="k">.await</span><span class="p">;</span><br> <span class="nf">Ok</span><span class="p">(</span><span class="n">item</span><span class="p">)</span><br> <span class="p">}</span><br><span class="p">}</span><br><br><span class="k">struct</span> <span class="n">FooWrapper</span> <span class="p">{</span><br> <span class="n">foo</span><span class="p">:</span> <span class="nb">Box</span><span class="o">&lt;</span><span class="n">dyn</span> <span class="n">Foo</span><span class="o">&gt;</span><span class="p">,</span><br><span class="p">}</span><br><span class="k">impl</span> <span class="n">FooWrapper</span> <span class="p">{</span><br> <span class="k">async</span> <span class="k">fn</span> <span class="nf">do_all_the_things</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span> <span class="k">self</span><span class="p">)</span> <span class="k">-&gt;</span> <span class="n">Result</span><span class="o">&lt;</span><span class="p">(),</span> <span class="n">Error</span><span class="o">&gt;</span> <span class="p">{</span><br> <span class="k">let</span> <span class="n">items</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">10</span><span class="p">)</span><span class="nf">.into_iter</span><span class="p">()</span><span class="nf">.map</span><span class="p">(|</span><span class="n">i</span><span class="p">|</span> <span class="nd">format!</span><span class="p">(</span><span class="s">"Item {}"</span><span class="p">,</span> <span class="n">i</span><span class="p">))</span><span class="nf">.collect</span><span class="p">();</span><br> <span class="k">self</span><span class="py">.foo</span><span class="nf">.update_all</span><span class="p">(</span><span class="n">items</span><span class="p">)</span><span class="k">.await</span><span class="o">?</span><span class="p">;</span><br> <span class="nf">Ok</span><span class="p">(())</span><br> <span class="p">}</span><br><span class="p">}</span><br></code></pre></div></div><br><br><h3 id="the-task">The task</h3><br><br><p>The change I needed to make was to call some extra logic in <code class="language-plaintext highlighter-rouge">FooWrapper</code> right before each item is processed in the loop inside <code class="language-plaintext highlighter-rouge">Foo::update_all</code>, which is here represented by the <code class="language-plaintext highlighter-rouge">Updates</code> struct and its <code class="language-plaintext highlighter-rouge">notify</code> method. Note that this is a <em>mutable</em> method.</p><br><br><div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">Updates</span><span class="p">;</span><br><span class="k">impl</span> <span class="n">Updates</span> <span class="p">{</span><br> <span class="k">fn</span> <span class="nf">notify</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span> <span class="k">self</span><span class="p">,</span> <span class="n">item</span><span class="p">:</span> <span class="nb">String</span><span class="p">)</span> <span class="p">{</span><br> <span class="nd">println!</span><span class="p">(</span><span class="s">"{:?} About to update {}"</span><span class="p">,</span> <span class="nf">time</span><span class="p">(),</span> <span class="n">item</span><span class="p">);</span><br> <span class="p">}</span><br><span class="p">}</span><br><br><span class="k">struct</span> <span class="n">FooWrapper</span> <span class="p">{</span><br> <span class="n">foo</span><span class="p">:</span> <span class="nb">Box</span><span class="o">&lt;</span><span class="n">dyn</span> <span class="n">Foo</span><span class="o">&gt;</span><span class="p">,</span><br> <span class="n">updates</span><span class="p">:</span> <span class="n">Updates</span><span class="p">,</span><br><span class="p">}</span><br></code></pre></div></div><br><br><h3 id="part-1-adding-a-callback-parameter">Part 1: Adding a callback parameter</h3><br><br><p>The obvious approach is to pass a callback into the <code class="language-plaintext highlighter-rouge">Foo::update_all</code> method, and call that before each update. Since <code class="language-plaintext highlighter-rouge">Foo</code> is a dyn trait, the callback needs to be dyn as well, and that usually means boxing, so I went ahead and added a boxed function parameter:</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> trait Foo {<br> async fn update(&amp;self, item: String) -&gt; Result&lt;String, Error&gt;;<br> <br><span class="gd">- async fn update_all(&amp;self, items: Vec&lt;String&gt;) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br></span><span class="gi">+ async fn update_all(<br>+ &amp;self,<br>+ items: Vec&lt;String&gt;,<br>+ cb: Box&lt;dyn FnMut(String)&gt;,<br>+ ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br></span> stream::iter(items)<br> .map(move |item| async move { self.update(item).await })<br> .buffered(3)<br></code></pre></div></div><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> impl FooWrapper {<br> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; {<br> let items = (0..10).into_iter().map(|i| format!("Item {}", i)).collect();<br><span class="gd">- self.foo.update_all(items).await?;<br></span><span class="gi">+ let mut cb = |item| self.updates.notify(item);<br>+ self.foo.update_all(items, Box::new(cb)).await?;<br></span> Ok(())<br> }<br> }<br></code></pre></div></div><br><br><p>This results in our first misleading lifetime error message:</p><br><br><pre><span class="bold red">error</span><span class="bold">: lifetime may not live long enough<br> </span><span class="bold blue">--&gt; </span>src/main.rs:60:36<br> <span class="bold blue">|<br>57 | </span> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; {<br> <span class="bold blue">| - let's call the lifetime of this reference `'1`<br>...<br>60 | </span> self.foo.update_all(items, Box::new(cb)).await?;<br> <span class="bold blue">| </span><span class="bold red">^^^^^^^^^^^^ cast requires that `'1` must outlive `'static`<br> </span><span class="bold blue">|<br></span><span class="bold cyan">help</span>: to allow this `impl Trait` to capture borrowed data with lifetime `'1`, add `'_` as a bound<br> <span class="bold blue">|<br>57 | </span> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt;<span class="green"> + '_</span> {<br> <span class="bold blue">| </span><span class="green">++++<br><br></span></pre><br><br><p>Most compiler errors in Rust are clear and easy to fix, but this one is different. This error points to <em>the completely wrong part of the code</em> and the suggested fix <em>isn’t even valid Rust</em>, as we can confirm if we try to perform the suggested fix anyway:</p><br><br><pre><span class="bold red">error[E0404]</span><span class="bold">: expected trait, found enum `Result`<br> </span><span class="bold blue">--&gt; </span>src/main.rs:57:46<br> <span class="bold blue">|<br>57 | </span> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; + '_ {<br> <span class="bold blue">| </span><span class="bold red">^^^^^^^^^^^^^^^^^ not a trait<br> </span><span class="bold blue">|<br></span><span class="bold cyan">help</span>: `+` is used to constrain a "trait object" type with lifetimes or auto-traits; structs and enums can't be bound in that way<br> <span class="bold blue">--&gt; </span>src/main.rs:57:66<br> <span class="bold blue">|<br>57 | </span> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; + '_ {<br> <span class="bold blue">| ----------------- </span><span class="bold cyan">^^ ...because of this bound<br> </span><span class="bold blue">| |<br> | expected this type to be a trait...<br></span><span class="bold cyan">help</span>: if you meant to use a type and not a trait here, remove the bounds<br> <span class="bold blue">|<br>57 </span><span class="red">- </span> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt;<span class="red"> + '_</span> {<br><span class="bold blue">57 </span><span class="green">+ </span> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; {<br> <span class="bold blue">| <br><br></span><span class="bold red">error[E0782]</span><span class="bold">: trait objects must include the `dyn` keyword<br> </span><span class="bold blue">--&gt; </span>src/main.rs:57:46<br> <span class="bold blue">|<br>57 | </span> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; + '_ {<br> <span class="bold blue">| </span><span class="bold red">^^^^^^^^^^^^^^^^^^^^^^<br> </span><span class="bold blue">|<br></span><span class="bold cyan">help</span>: add `dyn` keyword before this trait<br> <span class="bold blue">|<br>57 </span><span class="red">- </span> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; + '_ {<br><span class="bold blue">57 </span><span class="green">+ </span> async fn do_all_the_things(&amp;mut self) -&gt; <span class="green">dyn </span>Result&lt;(), Error&gt; + '_ {<br> <span class="bold blue">| <br><br></span><span class="bold">Some errors have detailed explanations: E0404, E0782.<br>For more information about an error, try `rustc --explain E0404`.<br></span></pre><br><br><p>Now I should be clear - <strong>Rust is my favorite language</strong>. Rust really stands out for the high quality of its compiler error messages, which is why I was surprised to see such a broken error here. I’m pointing this out because I hold Rust to a higher standard, and so that people can fix this error and make it even better. In many programming languages, inscrutable and misleading compiler error messages are a routine fact of life, not something notable enough to be worth spending my whole Saturday writing a blog post about.</p><br><br><h3 id="1b">1b</h3><br><br><p>Fortunately, this error is easily fixed, in spite of the broken error message. We know that we just added a boxed trait object to the signature of <code class="language-plaintext highlighter-rouge">update_all</code>, and that our callback captures a reference to <code class="language-plaintext highlighter-rouge">FooWrapper</code> and that we probably need to indicate that in its type somehow.</p><br><br><p>We need to add a lifetime bound which is tied to the future that <code class="language-plaintext highlighter-rouge">update_all</code> returns, but it wasn’t clear to me how to do that, given that this is an async trait method. I decided to just add a <code class="language-plaintext highlighter-rouge">'_</code> to see what would happen.</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> async fn update_all(<br> &amp;self,<br> items: Vec&lt;String&gt;,<br><span class="gd">- cb: Box&lt;dyn FnMut(String)&gt;,<br></span><span class="gi">+ cb: Box&lt;dyn FnMut(String) + '_&gt;,<br></span> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br> stream::iter(items)<br> .map(move |item| async move { self.update(item).await })<br></code></pre></div></div><br><br><p>Fortunately, this time, Rust tells us exactly what we need to do. Apparently, the <code class="language-plaintext highlighter-rouge">async_trait</code> macro generates a <code class="language-plaintext highlighter-rouge">'async_trait</code> lifetime that we can use.</p><br><br><pre><span class="bold red">error[E0621]</span><span class="bold">: explicit lifetime required in the type of `cb`<br> </span><span class="bold blue">--&gt; </span>src/main.rs:26:37<br> <span class="bold blue">|<br>25 | </span> cb: Box&lt;dyn FnMut(String) + '_&gt;,<br> <span class="bold blue">| --------------------------- help: add explicit lifetime `'async_trait` to the type of `cb`: `Box&lt;(dyn FnMut(String) + 'async_trait)&gt;`<br>26 | </span> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br> <span class="bold blue">| </span><span class="bold red">_____________________________________^<br></span><span class="bold blue">27 | </span><span class="bold red">| </span> stream::iter(items)<br><span class="bold blue">28 | </span><span class="bold red">| </span> .map(move |item| async move { self.update(item).await })<br><span class="bold blue">29 | </span><span class="bold red">| </span> .buffered(3)<br><span class="bold blue">30 | </span><span class="bold red">| </span> .try_collect()<br><span class="bold blue">31 | </span><span class="bold red">| </span> .await<br><span class="bold blue">32 | </span><span class="bold red">| </span> }<br> <span class="bold blue">| </span><span class="bold red">|_____^ lifetime `'async_trait` required<br><br></span><span class="bold">For more information about this error, try `rustc --explain E0621`.<br></span></pre><br><br><h3 id="1c">1c</h3><br><br><p>I later realized that the boxing isn’t necessary in the first place. Boxing is a Rustacean’s first instinct for passing around <code class="language-plaintext highlighter-rouge">dyn Trait</code> objects, but in this case, it makes more sense to pass the callback by mutable reference instead. In fact, the <code class="language-plaintext highlighter-rouge">Box</code> doesn’t actually add anything here since the closure is not <code class="language-plaintext highlighter-rouge">'static</code> anyway - the closure is effectively just a <code class="language-plaintext highlighter-rouge">&amp;mut Updates</code> pointing into the parent <code class="language-plaintext highlighter-rouge">FooWrapper</code>.</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> async fn update_all(<br> &amp;self,<br> items: Vec&lt;String&gt;,<br><span class="gd">- cb: Box&lt;dyn FnMut(String) + '_&gt;,<br></span><span class="gi">+ cb: &amp;mut dyn FnMut(String),<br></span> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br> stream::iter(items)<br> .map(move |item| async move { self.update(item).await })<br><br> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; {<br> let items = (0..10).into_iter().map(|i| format!("Item {}", i)).collect();<br> let mut cb = |item| self.updates.notify(item);<br><span class="gd">- self.foo.update_all(items, Box::new(cb)).await?;<br></span><span class="gi">+ self.foo.update_all(items, &amp;mut cb).await?;<br></span> Ok(())<br> }<br> }<br><br></code></pre></div></div><br><br><p>Switching to <code class="language-plaintext highlighter-rouge">&amp;mut</code> dodges the lifetime issues we had with the <code class="language-plaintext highlighter-rouge">Box</code> approach above. If I’d thought of that the first time around, it would have saved a lot of trouble. On the other hand, this way I discovered a broken error message to report, so perhaps it was for the best.</p><br><br><h2 id="part-2-calling-the-callback">Part 2: Calling the callback</h2><br><br><p>Now comes the <em>hard</em> part - actually calling the callback parameter that we added to <code class="language-plaintext highlighter-rouge">update_all</code>.</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br> stream::iter(items)<br><span class="gd">- .map(move |item| async move { self.update(item).await })<br></span><span class="gi">+ .map(move |item| async move {<br>+ cb(item.clone());<br>+ self.update(item).await<br>+ })<br></span> .buffered(3)<br> .try_collect()<br></code></pre></div></div><br><br><p>The naive approach predictably fails:</p><br><br><pre><span class="bold red">error[E0507]</span><span class="bold">: cannot move out of `cb`, a captured variable in an `FnMut` closure<br> </span><span class="bold blue">--&gt; </span>src/main.rs:28:41<br> <span class="bold blue">|<br>25 | </span> cb: &amp;mut dyn FnMut(String),<br> <span class="bold blue">| -- captured outer variable<br>...<br>28 | </span> .map(move |item| async move {<br> <span class="bold blue">| </span><span class="bold red">___________________</span><span class="bold blue">-</span><span class="bold red">______________________^<br> </span><span class="bold blue">| </span><span class="bold red">| </span><span class="bold blue">__________________|<br> | </span><span class="bold red">|</span><span class="bold blue">|<br>29 | </span><span class="bold red">|</span><span class="bold blue">| </span> cb(item.clone());<br> <span class="bold blue">| </span><span class="bold red">|</span><span class="bold blue">| -- move occurs because `cb` has type `&amp;mut dyn FnMut(String)`, which does not implement the `Copy` trait<br>30 | </span><span class="bold red">|</span><span class="bold blue">| </span> self.update(item).await<br><span class="bold blue">31 | </span><span class="bold red">|</span><span class="bold blue">| </span> })<br> <span class="bold blue">| </span><span class="bold red">|</span><span class="bold blue">| </span><span class="bold red">^<br> </span><span class="bold blue">| </span><span class="bold red">|</span><span class="bold blue">|_____________</span><span class="bold red">|<br> </span><span class="bold blue">| </span><span class="bold red">|______________</span><span class="bold blue">captured by this `FnMut` closure<br> | </span><span class="bold red">move out of `cb` occurs here<br><br></span><span class="bold">For more information about this error, try `rustc --explain E0507`.<br></span></pre><br><br><p>Unfortunately, the <em>non-naive</em> approaches also all fail here. There’s just no way to do what we want to do here in Rust. The problem is that we’re creating a bunch of futures, and in order to call the callback, Rust requires them to <em>each</em> have a reference to <code class="language-plaintext highlighter-rouge">cb</code> that is exclusive for the <em>entire</em> duration of the stream.</p><br><br><p>Usually when you run into an intractable borrow problem in Rust, it means that either a) there’s a bug in your code or b) you need a self-referential type, which requires unsafe code. (There are a number of libraries that aim to fill that gap, such as <code class="language-plaintext highlighter-rouge">owning_ref</code> and <code class="language-plaintext highlighter-rouge">ouroboros</code>, but <a href="https://github.com/noamtashma/owning-ref-unsoundness">their soundness is rather dubious</a>).</p><br><br><p>Here, on the other hand, we have a “type system false positive” of a different sort. Our callback is synchronous, and the <code class="language-plaintext highlighter-rouge">update_all</code> future can only ever poll a single child future at once, so the callback can only ever be called once at a time. What we need is a reference that is only exclusive <em>while the future is being polled</em>, but Rust provides no facility for this.</p><br><br><p>The usual approach to solve this in Rust is to just have a <em>single</em> exclusive (i.e. <code class="language-plaintext highlighter-rouge">&amp;mut</code>) reference that we thread through the stream and pass into each future just while it is being polled (or pass around a <code class="language-plaintext highlighter-rouge">GhostCell</code> token if passing the data itself around is too cumbersome for some reason). Unfortunately, <code class="language-plaintext highlighter-rouge">Future</code> and <code class="language-plaintext highlighter-rouge">Stream</code> have fixed interfaces and there is no way to pass extra data through them.</p><br><br><h3 id="2b">2b</h3><br><br><p>Given that there is no “proper” way to solve this in Rust, we just have to sigh and go with an imperfect solution. We can work around the issue by wrapping the callback in an unnecessary <code class="language-plaintext highlighter-rouge">RefCell</code>:</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> items: Vec&lt;String&gt;,<br> cb: &amp;mut dyn FnMut(String),<br> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br><span class="gi">+ let cb = RefCell::new(cb);<br></span> stream::iter(items)<br> .map(move |item| async move {<br><span class="gd">- cb(item.clone());<br></span><span class="gi">+ cb.borrow_mut()(item.clone());<br></span> self.update(item).await<br> })<br> .buffered(3)<br></code></pre></div></div><br><br><p>Since our <code class="language-plaintext highlighter-rouge">map</code> closure is <code class="language-plaintext highlighter-rouge">move</code>, we also need to store a (shared) reference in a local variable so it can be moved into the closure. Note that in this example, there’s no particular reason for the closure to be <code class="language-plaintext highlighter-rouge">move</code>, but the real world code which inspired this post had other stuff going on which required it.</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cb: &amp;mut dyn FnMut(String),<br> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br> let cb = RefCell::new(cb);<br><span class="gi">+ let cb = &amp;cb;<br></span> stream::iter(items)<br> .map(move |item| async move {<br> cb.borrow_mut()(item.clone());<br></code></pre></div></div><br><br><p>… and it works. Yay!</p><br><br><div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>381ns About to update Item 0<br>10.431µs Updating Item 0<br>31.824µs About to update Item 1<br>36.807µs Updating Item 1<br>49.812µs About to update Item 2<br>54.461µs Updating Item 2<br>101.421368ms About to update Item 3<br>101.444199ms Updating Item 3<br>101.477413ms About to update Item 4<br>101.494382ms Updating Item 4<br>101.513491ms About to update Item 5<br>101.522133ms Updating Item 5<br>202.929102ms About to update Item 6<br>202.946752ms Updating Item 6<br>202.972324ms About to update Item 7<br>202.97918ms Updating Item 7<br>202.992133ms About to update Item 8<br>203.004573ms Updating Item 8<br>304.513564ms About to update Item 9<br>304.544859ms Updating Item 9<br><br></code></pre></div></div><br><br><h3 id="2c">2c</h3><br><br><p>Again, it should be noted that as much as I complain about having to add an unnecessary <code class="language-plaintext highlighter-rouge">RefCell</code> here, I think this is really a testament to Rust and how far it can be pushed. Rust makes it easy to track ownership and aliasing invariants through many layers of abstraction and across large codebases, to the point where people get upset in the rare cases where this <em>isn’t</em> possible. Meanwhile, in other languages programmers slather the code with defensive copies and refcounts and gratuitous runtime checks for far less than this (and in the case of C++, almost certainly <em>still</em> get it wrong and have tons of mysterious crashes).</p><br><br><h2 id="part-3-making-it-optional">Part 3: Making it optional</h2><br><br><p>The previous code works, but it has the problem that <em>every</em> caller of <code class="language-plaintext highlighter-rouge">update_all</code> must now provide a callback. Let’s try to make the callback parameter optional instead.</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> async fn update_all(<br> &amp;self,<br> items: Vec&lt;String&gt;,<br><span class="gd">- cb: &amp;mut dyn FnMut(String),<br></span><span class="gi">+ cb: Option&lt;&amp;mut dyn FnMut(String)&gt;,<br></span> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br> let cb = RefCell::new(cb);<br> let cb = &amp;cb;<br></code></pre></div></div><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> async fn do_all_the_things(&amp;mut self) -&gt; Result&lt;(), Error&gt; {<br> let items = (0..10).into_iter().map(|i| format!("Item {}", i)).collect();<br> let mut cb = |item| self.updates.notify(item);<br><span class="gd">- self.foo.update_all(items, &amp;mut cb).await?;<br></span><span class="gi">+ self.foo.update_all(items, Some(&amp;mut cb)).await?;<br></span> Ok(())<br> }<br> }<br></code></pre></div></div><br><br><p>Now, how do we actually make this work inside <code class="language-plaintext highlighter-rouge">update_all</code>? The natural approach is to just use a default no-op callback if one isn’t provided.</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> items: Vec&lt;String&gt;,<br> cb: Option&lt;&amp;mut dyn FnMut(String)&gt;,<br> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br><span class="gi">+ let cb = cb.unwrap_or(&amp;mut |_| ());<br></span> let cb = RefCell::new(cb);<br> let cb = &amp;cb;<br> stream::iter(items)<br></code></pre></div></div><br><br><p>Unfortunately, this doesn’t work:</p><br><br><pre><span class="bold red">error[E0716]</span><span class="bold">: temporary value dropped while borrowed<br> </span><span class="bold blue">--&gt; </span>src/main.rs:27:36<br> <span class="bold blue">|<br>27 | </span> let cb = cb.unwrap_or(&amp;mut |_| ());<br> <span class="bold blue">| </span><span class="bold red">^^^^^^ </span><span class="bold blue">- temporary value is freed at the end of this statement<br> | </span><span class="bold red">|<br> </span><span class="bold blue">| </span><span class="bold red">creates a temporary which is freed while still in use<br></span><span class="bold blue">28 | </span> let cb = RefCell::new(cb);<br> <span class="bold blue">| -- borrow later used here<br> |<br> = </span><span class="bold">note</span>: consider using a `let` binding to create a longer lived value<br><br><span class="bold">For more information about this error, try `rustc --explain E0716`.<br></span></pre><br><br><p>I thought the problem might have to do with it being a closure, so I tried converting it to a regular function.</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> items: Vec&lt;String&gt;,<br> cb: Option&lt;&amp;mut dyn FnMut(String)&gt;,<br> ) -&gt; Result&lt;Vec&lt;String&gt;, Error&gt; {<br><span class="gi">+ fn default_cb(_: String) {}<br>+ let cb = cb.unwrap_or(&amp;mut default_cb);<br></span> let cb = RefCell::new(cb);<br> let cb = &amp;cb;<br> stream::iter(items)<br></code></pre></div></div><br><br><p>Surely taking a reference to a plain function would work. After all, functions are stateless and static - there’s no way this could fail a borrow check…</p><br><br><pre><span class="bold red">error[E0716]</span><span class="bold">: temporary value dropped while borrowed<br> </span><span class="bold blue">--&gt; </span>src/main.rs:28:36<br> <span class="bold blue">|<br>28 | </span> let cb = cb.unwrap_or(&amp;mut default_cb);<br> <span class="bold blue">| </span><span class="bold red">^^^^^^^^^^ </span><span class="bold blue">- temporary value is freed at the end of this statement<br> | </span><span class="bold red">|<br> </span><span class="bold blue">| </span><span class="bold red">creates a temporary which is freed while still in use<br></span><span class="bold blue">29 | </span> let cb = RefCell::new(cb);<br> <span class="bold blue">| -- borrow later used here<br> |<br> = </span><span class="bold">note</span>: consider using a `let` binding to create a longer lived value<br><br><span class="bold">For more information about this error, try `rustc --explain E0716`.<br></span></pre><br><br><h3 id="3b">3b</h3><br><br><p>To be honest, I’m still not sure why that didn’t work. Fortunately, there’s a plan B - we can just keep the <code class="language-plaintext highlighter-rouge">Option</code> inside the <code class="language-plaintext highlighter-rouge">RefCell</code> and check whether the callback exists in the inner loop every time we call it, which is probably the cleaner and more principled approach anyway.</p><br><br><p>It’s hard to keep track of the billions of helper methods on <code class="language-plaintext highlighter-rouge">Option</code> and <code class="language-plaintext highlighter-rouge">Result</code>, but this is a pretty simple case. We just want to run some code on the contained value (i.e. callback) if present, and do nothing if not present, which is normally done via <code class="language-plaintext highlighter-rouge">Option::map</code>.</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> let cb = &amp;cb;<br> stream::iter(items)<br> .map(move |item| async move {<br><span class="gd">- cb.borrow_mut()(item.clone());<br></span><span class="gi">+ cb.borrow_mut().map(|cb| cb(item.clone()));<br></span> self.update(item).await<br> })<br> .buffered(3)<br></code></pre></div></div><br><br><p>Unfortunately, this results in the <em>second</em> broken error message I ran into.</p><br><br><pre><span class="bold red">error[E0507]</span><span class="bold">: cannot move out of dereference of `RefMut&lt;'_, Option&lt;&amp;mut dyn FnMut(String)&gt;&gt;`<br> </span><span class="bold blue">--&gt; </span>src/main.rs:31:17<br> <span class="bold blue">|<br>31 | </span> cb.borrow_mut().map(|cb| cb(item.clone()));<br> <span class="bold blue">| </span><span class="bold red">^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ move occurs because value has type `Option&lt;&amp;mut dyn FnMut(String)&gt;`, which does not implement the `Copy` trait<br> </span><span class="bold blue">|<br></span><span class="bold cyan">help</span>: consider borrowing the `Option`'s content<br> <span class="bold blue">|<br>31 | </span> cb.borrow_mut().map(|cb| cb(item.clone()))<span class="green">.as_ref()</span>;<br> <span class="bold blue">| </span><span class="green">+++++++++<br><br></span><span class="bold">For more information about this error, try `rustc --explain E0507`.<br></span></pre><br><br><p>This error message is clearly nonsense, because our callback returns <code class="language-plaintext highlighter-rouge">()</code>, i.e. nothing, and there is no reason you would ever want to call <code class="language-plaintext highlighter-rouge">.as_ref()</code> on an optional <em>nothing</em> - there’s no lifetimes contained in the result to dodge in the first place!</p><br><br><p>If we humor the Rust compiler anyway, it just repeats the same nonsense suggestion in an infinite loop:</p><br><br><pre><span class="bold red">error[E0507]</span><span class="bold">: cannot move out of dereference of `RefMut&lt;'_, Option&lt;&amp;mut dyn FnMut(String)&gt;&gt;`<br> </span><span class="bold blue">--&gt; </span>src/main.rs:31:17<br> <span class="bold blue">|<br>31 | </span> cb.borrow_mut().map(|cb| cb(item.clone())).as_ref();<br> <span class="bold blue">| </span><span class="bold red">^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ move occurs because value has type `Option&lt;&amp;mut dyn FnMut(String)&gt;`, which does not implement the `Copy` trait<br> </span><span class="bold blue">|<br></span><span class="bold cyan">help</span>: consider borrowing the `Option`'s content<br> <span class="bold blue">|<br>31 | </span> cb.borrow_mut().map(|cb| cb(item.clone()))<span class="green">.as_ref()</span>.as_ref();<br> <span class="bold blue">| </span><span class="green">+++++++++<br><br></span><span class="bold">For more information about this error, try `rustc --explain E0507`.<br></span></pre><br><br><p>This error took quite a bit of trial and error, and some red herrings, to resolve. I noticed that <code class="language-plaintext highlighter-rouge">RefMut</code> (the type of the lock guard) has its own <code class="language-plaintext highlighter-rouge">map</code> method and thought it might be calling the wrong one, so I tried using <a href="https://web.mit.edu/rust-lang_v1.25/arch/amd64_ubuntu1404/share/doc/rust/html/book/first-edition/ufcs.html">UFCS</a> to call <code class="language-plaintext highlighter-rouge">Option::map</code> explicitly, but that didn’t work.</p><br><br><p>I ended up having to split the line up so that each call was a separate variable on a separate line, and then added explicit type annotations to each one (which also required constantly adding and removing imports to the top of the file) in order to narrow down the source of the error. In retrospect, my mistake was obvious, but I’m disappointed that the Rust compiler provided little assistance in discovering it.</p><br><br><p>The problem is that our refcell has type <code class="language-plaintext highlighter-rouge">RefCell&lt;Option&lt;T&gt;&gt;</code> (where <code class="language-plaintext highlighter-rouge">T</code> is <code class="language-plaintext highlighter-rouge">&amp;mut dyn FnMut(String)</code>). This means that borrowing it results in a guard which derefs to <code class="language-plaintext highlighter-rouge">&amp;mut Option&lt;T&gt;</code>. However, what we <em>want</em> is an <code class="language-plaintext highlighter-rouge">Option&lt;&amp;mut T&gt;</code>, so that we can then unwrap it and call the contained callback. Therefore, the solution is to just add an <code class="language-plaintext highlighter-rouge">as_mut</code> call:</p><br><br><div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> let cb = &amp;cb;<br> stream::iter(items)<br> .map(move |item| async move {<br><span class="gd">- cb.borrow_mut().map(|cb| cb(item.clone()));<br></span><span class="gi">+ cb.borrow_mut().as_mut().map(|cb| cb(item.clone()));<br></span> self.update(item).await<br> })<br> .buffered(3)<br></code></pre></div></div><br><br><h2 id="conclusion">Conclusion</h2><br><br><p>…and that’s how what I expected to be a simple change ended up taking the better part of two days. Even as a long-time user of Rust, I still sometimes get frustrated by errors, and sometimes quite frustrated indeed. However, I think this also shows how far Rust has come, in that such cases are the exception rather than the norm, and I hope by highlighting these issues that it will continue to improve.</p>
• posted archived copycurrent