← Producer Manual
Producer Manual · Chapter 1 · NBA Top Shot & Disney Pinnacle

Your job is growth

Every producer owns one number: weekly buyers. Start with the problem and the number, not the idea. Come up with several ways to move it, rank them by how much you expect each to move it, and try the best one first. When it misses, you still learn, because you wrote down what you expected.

Two ways to work

Same team, same amount of work. The order is what changes.

Idea first

  1. Fall in love with an idea
  2. →
  3. Spec out the plan
  4. →
  5. Find a number for it
  6. →
  7. Build it
  8. →
  9. It doesn't work
Start all over again, with no clear lesson.

Data first

  1. Pick the problem
  2. →
  3. Pick the number that shows it
  4. →
  5. Come up with several solutions
  6. →
  7. Rank them by expected impact
  8. →
  9. Try the top one
  10. →
  11. Compare the result to your estimate
Even a miss makes your next estimate, and your next pick, better.

Three kinds of number

One you're judged on. The ones you move to get there. The ones that tell you to stop.

The KPIWhat you're judged on
Weekly buyers

Everyone who bought something this week. It moves late: by the time it changes, the week is over. Chapter 12 covers the weekly scoreboard.

▲ FEED THE KPI ▲
Leading indicatorsWhat you can move this week
Priority ordersSets completed and lockedQuest and challenge completionsPacks openedGames playedPoints redeemedMarketplace buys that complete a setMessages that lead to a purchase
Health metricsAlso called guardrails: they tell you when to stop
Unsubscribes and notification opt-outs climbingSellers outnumbering buyers after a dropTop tiers selling out slowerLocks and trade-ins falling behind new supplyRevenue concentrating in a handful of collectorsComplaints and support tickets

Floor price is none of these. We don't control it, and it doesn't tell you whether more people are buying.

The loops feed each other

Every part of the product feeds the others, not just itself. Tap a loop to see what it feeds, what feeds it, and the number to watch.

What it feedsWhat feeds it

Top Shot and Pinnacle share this shape. The names differ: on Pinnacle, trivia is a game, and the exchange turns pins you don't want into points for the store.

Who you're building for

Three lenses, for three different jobs.

Lens 1 · Designing the collectibles, sizing a problem, reading a result

Spend tiers, from new collector to XL

Weekly buyers breaks into spend tiers. Every drop needs something built for each tier, and every problem and result should be read tier by tier, never as one total.

Weekly buyers=New+S+M+L+XL
Lens 2 · Marketing

Five archetypes, and why each one buys

Marketing speaks to motivation. The same Moment or pin is a grail to one collector and a missing piece to another. Chapter 4 covers each in full.

Spend tiers are average weekly spend.

Lens 3 · Notifications

Many dimensions at once

Every message is aimed using all of these together, and changes with what the collector did last. Chapter 5 has the message plan.

ActivityOpened the app todayQuiet for 30 days
BehaviourRips packsFinishes setsPlays games
OwnershipOne piece from a setHolds last season's set
StatusThe tier they've earnedClose to the next
AccessHas first shotIn the priority drawMain draw only
Combined, that's hundreds of groups and dozens of message types, each with its own next message.

Set up, and ask good questions

All our data lives in BigQuery. You don't need to write queries: our agents do it for you.

MagicAsk it anything about NBA Top Shot.
PersiiAsk it anything about Disney Pinnacle.

If an agent gets something wrong or can't answer, send the question and its answer to Roham, who gets it to Matt and Alan.

The six parts of a good question

Weekly buyers among M-tier collectors who completed last season's rookie set, for each of the last 8 weeks, compared with M-tier collectors who didn't complete it. Count a buyer as anyone who bought a pack or a Moment that week, and leave out refunds. Give me a table: weeks as rows, the two groups as columns.

The number. One, named exactly.
Who. Tier, set, lock or cohort.
When. A fixed window.
Compared with. Without it, a number means nothing.
Your definitions. The agent can't guess them.
The shape of the answer. Table, chart or one number.
"Why did sales drop?"
→
"Weekly buyers by spend tier for the last 8 weeks, as a table. Which tier fell most compared with the 4 weeks before?"
"How did the drop do?"
→
"Priority orders by hour since the doors opened, this drop against the last two, split by which lock earned the order."
"Are people completing the set?"
→
"Of everyone who bought into [set], how many completed it within 30 days, by spend tier? Of those, how many listed any piece for sale?"
Before a drop

"How many collectors are one or two pieces from completing each set that earns first shot at the next drop, by spend tier?"

"How many collectors hold each qualifying lock, against the size of each priority allocation?"

During a drop

"Priority orders so far, against the same hour of the last drop."

"Every message sent today, by type: sent, opened, clicked, bought, and the same for the held-back group."

After a drop

"Weekly buyers by tier for the 4 weeks after the drop, against the 4 weeks before."

"Set completions 7 and 30 days after the drop, split into collectors who bought pieces and collectors who sold them."

Then go one level deeper

  • "Which group drove that?"
  • "Show me the same thing for Pinnacle" (or for last season).
  • "What else could explain this, and what would prove it wrong?"

Check the answer before you use it

  • Ask the agent to explain, in plain words, exactly what it counted.
  • Check one total you already know, like a drop's published supply.
  • Ask for the same number a second way. If they differ, use neither.
  • Check the parts add up: the tiers should sum to the total.

How to do it

Five steps, every time, in this order.

  1. 1

    Pick the problem, and the number that shows it

    Start in the data, not with an idea. Look at weekly buyers by tier, and at the leading indicator for each loop. Find the gap: the tier or loop that's weakest against last month or the last drop. That gap is the problem. The leading indicator that shows it is your number.

  2. 2

    Come up with several solutions, and rank them

    Never bring one idea. For each, estimate how many collectors it can reach and what share of them you expect to act. Multiply the two, rank, and try the top one. Change the numbers below and watch the order move.

    IdeaCan reach% who act

    Practice numbers, made up for the exercise. A small group asked to take a step they already want to take often beats a big group asked for something new.

  3. 3

    Write the bet down before you build

    One card. If you can't fill it in, you're not ready to build.

    The bet
    Problem
    The gap you found, in one line
    Number
    The leading indicator you'll move, and the tier it shows up in
    Who
    Tier, archetype and the exact group
    The pick
    The idea you ranked first, and the ones below it
    Estimate
    How much it moves the number, and by when
    Stop if
    The health metric that tells you to pull it
    Held back
    The small group that doesn't get it, so you can compare
  4. 4

    Run every lever this week, and measure each one

    A week isn't one nudge. It's every lever running at once, each aimed at its own groups:

    Drop beatsPacks on saleQuestsChallengesGamesPoint storeMarketplaceMessages

    Messages alone are dozens of types going to hundreds of groups, by activity, behaviour, ownership, status and access. Before anything goes live, check every automation is tracked end to end, with a held-back group:

    Sent→Opened→Clicked→Bought→Bought again

    The leading indicators move this week. Weekly buyers confirms it later.

  5. 5

    Read the result honestly, and get better at estimating

    Read it by tier and by cohort, never as one blended number. A cohort is a group that started at the same time, like everyone whose first purchase was in June. Read across a row to see one group fade. Read down a column to see whether newer groups are holding on better.

    An illustration. Darker means more of the group is still buying. Hatched cells haven't happened yet.

    Then check the four ways a number fools you:

    It feels goodbut nothing got better for collectors.
    The number got chasedinstead of what it stood for.
    It was newso people tried it once.
    Who was buying changednot what anyone did.

    Last, put the result next to your estimate. How far off you were is what makes the next ranking better.

What it looks like

Examples of the method, not plans. [N] marks numbers you'd get from the data.

NBA Top Shot · M-tier collectors buying less
Problem

Since the last drop, M-tier weekly buyers are down against the month before. XL is steady.

Number

Set completions among M-tier collectors, the leading indicator for that tier.

Ranked
  1. A set challenge for collectors one or two pieces from finishing
  2. Messages showing each collector the exact pieces they're missing, and where to find them
  3. A spotlight pack built for the M tier, featuring last season's set
  4. A bigger chase in the open pack
The bet

The set challenge. Estimate: [N] more M-tier weekly buyers within two weeks. Stop if sellers outnumber buyers among the collectors finishing. [N]% of eligible collectors held back.

Read

Compare M-tier weekly buyers against the held-back group, cohort by cohort, then against the estimate. Ideas two and three are next in line.

Disney Pinnacle · newcomers who stop after one pack
Problem

Most newcomers who open a first pack don't buy a second.

Number

The share of newcomers who buy again within [N] days of their first pack.

Ranked
  1. Right after the first pack, show the collection that pin belongs to, the lore behind it, and how close they are to finishing
  2. A small starter collection a newcomer can finish cheaply
  3. Daily trivia that earns points toward the next pack
  4. A reason to invite a friend
The bet

The collection screen after the first pack. Estimate: second purchases up [N] points within a month. Stop if unsubscribes climb among new collectors. A small group held back.

Read

Compare each month's newcomers against the held-back group, then against the estimate, before trying the starter collection.

Words to know

You'll hear these every week.

KPIThe number you're judged on. Ours is weekly buyers.
Leading indicatorA number you can move this week that feeds the KPI.
Health metricAlso called a guardrail. Tells you when to stop.
Spend tierNew, S, M, L or XL, by average weekly spend.
ArchetypeWhy someone buys. There are five (Chapter 4).
CohortA group that started at the same time. It never changes.
Held-back groupA small group that doesn't get the change, kept to compare against.
EstimateWhat you expect an idea to move, written before you build it.
CalibrationGetting better at estimates by comparing them to results.

When it breaks

A hypothetical, inside our own products. Revenue jumps the week a big chase pack ships, and the team calls it a win. Read by tier, the jump is XL collectors buying more of the chase. The M-tier collectors it was pitched to win back never moved. Nobody wrote down an estimate, so there's nothing to compare against and nothing to learn. The next idea is another guess.

The fix: fill in the bet card before building, read the result by tier, and put it next to the estimate.

Questions for your drop

  1. What's the problem, and which tier and which loop does it show up in?
  2. Which leading indicator will you move, and which health metric tells you to stop?
  3. What other solutions did you consider, how did you rank them, and what's your estimate for the one you picked?
  4. Is every lever running this week tracked from sent to bought, with a held-back group?
  5. Does the result hold up tier by tier and cohort by cohort, and how far was it from your estimate?