Distillations About

Stress testing variadic zip

23 May 2020 ⇐ Notes archive

(This is an entry in my technical notebook. There will likely be typos, mistakes, or wider logical leaps — the intent here is to “let others look over my shoulder while I figure things out.”)

Yesterday, Daniel Williams and I messaged through a crasher he ran into when using CombineExt.Collection.zip (similarly with .Collection.combineLatest).

For the uninitiated, Combine ships with zip (and combineLatest) overloads up to arity four in the Publisher namespace.

But, if you want to zip arbitrarily many publishers, you’re kind of stuck and as more Combine code gets written, folks are quickly realizing this. That’s why we’ve been heads down filling in gaps with an extensions package to sit next to Combine proper.

Daniel was attempting to first ping https://hacker-news.firebaseio.com/v0/topstories.json for an array of Hacker News story IDs and then hydrate each by hitting the https://hacker-news.firebaseio.com/v0/item/:id.json endpoint. The former returns on the order of 500 entries and that turned out to be enough to push variadic zip beyond its limits.

We can reduce the scenario down with convenience publishers for a closer look.

(Gist permalink.)

(You might need to tweak count to trigger the crash.)

The stack trace is a head scratcher.

And the repeated Zip, Map, and PublisherBox frames hint at the issue.

CombineExt’s variadic zip and combineLatest are “composed” operators — they’re built up from existing Publisher methods instead of a more dedicated conformance. While this simplifies things and lets each implementation measure out to ~15 lines, it also introduces intermediate runtime overhead.

Let’s take a look at why (in shorter form, here’s the fuller implementation).

(Gist permalink.)

  1. The only way to line up seed’s type with reduce’s accumulator is to erase — or, at least I tried without in hopes of preserving fusion and got type checked into a corner.
  2. This and the following two lines are the source of the Zip, Map, and PublisherBox stack trace dance. As we approach thousands of publishers, we’re triply nesting for each.

Can we fix this?

Yep — by writing a specialized ZipCollectionType à la RxSwift’s! But with WWDC around the corner, it’s probably best to hang tight and see if the Combine team will address the gap.

Until then, and if you want to read more about variadic zipping, an older entry has your back.