Platform comparisons rankings are the wrong shape: three honest alternatives

The argument against ranking platforms is usually made as a complaint about bias. That is the weaker argument. The stronger one is about shape: a ranked list is a total order, and the thing being described has no total order, so the format is wrong before anyone gets to the question of who paid for it.

This page is about output shapes. What a comparison can honestly produce, which shapes carry how much information, and how to give a reader something more useful than a number one without pretending you have no view.

What to take away

Why a total order is the wrong shape

Ranking assumes a single scale. Platforms differ on axes with no exchange rate between them: durability against reach, portability against discovery, moderation against openness. There is no defensible number of units of discovery that one unit of portability is worth, and any weighted score has invented one.

The moment a score exists, a second problem starts. A published score becomes something to optimize toward, and the properties that are easy to move are not the ones that matter. That is Goodhart's law, and it applies to catalog rankings as reliably as to any other measure that acquires consequences.

The reader's cost is the real one. A rank tells someone that the first entry is better than the second without telling them on what, so a reader whose priorities differ from the editor's gets a confident wrong answer instead of a usable one.

Shape one: the undominated set

The most information-dense honest output. A candidate is dominated if some other candidate is at least as good on every axis you measured and better on at least one. Drop every dominated candidate and publish what is left.

This is a Pareto frontier, and it does real work. It usually eliminates half the field without any weighting at all, which is the whole point: the eliminations are defensible to anyone, whatever their priorities, because no set of weights would rescue a dominated candidate.

What remains is a set of genuine trade-offs. Present it as a table: axes as rows, survivors as columns.

Say plainly that each is better than the others on something. Which one is right for you depends on the row you care about most. That is the honest conclusion of almost every platform comparison ever written.

Shape two: partition by job

Instead of one order, several short lists, each headed by a job stated in a sentence: for reaching strangers, for keeping an existing audience, for work that has to be findable in five years.

The partition is only honest if the jobs were defined before the candidates were scored and if a candidate can appear under more than one. Jobs invented afterwards to give every candidate a heading are a ranking with better manners.

Where the jobs come from is a real question, and the answer is the same as for a comparison index: from what readers are actually trying to do, tested against whether the job admits more than one candidate. The rule for deciding which pairs are worth comparing at all builds the same sets for the same reason.

Shape three: bands, with the differences named

Where readers genuinely need something ordered, use a small number of bands rather than positions, and name what separates the bands.

Three bands are usually enough: meets every requirement, meets them with a stated exception, and fails a requirement. Inside a band, order alphabetically and say so. A band boundary is a claim you can defend with a sentence; a position is a claim you cannot defend at all.

Bands also survive small measurement errors, which positions do not. Two candidates separated by a rounding difference sit in the same band, which is the truth, and appear at ranks four and five, which is not.

What to do with the request for a number one

It will come, and refusing it flatly is unhelpful. Three replies work.

Give them the filter. A short sequence of eliminating questions that produces their own shortlist. This is the answer that is actually theirs, and the sort-key contract for a catalog sets out how to publish the orders you do offer without any of them implying merit.

Give them a worked example. One reader with stated requirements, walked through to a shortlist, every elimination explained. The worked scoring of a messaging choice shows the arithmetic version, including the sensitivity check that tells you when a winner is not a winner.

Give them your own view, labeled. State the job you assume, the weights you used, and the date. A signed opinion with its assumptions visible is a legitimate thing to publish. It is only dishonest when it is dressed as a measurement.

Reading someone else's ranking

The same shape argument tells you what to look for in a list you did not write.

Ask what the order is computed from, and whether the page says. Ask whether any two entries are separated by something that could be stated in a sentence, or only by a score.

Ask whether the list contains anything that cannot pay a commission, since a pool selected by who can pay is a pool selected before any judging happened, which the anatomy of a ranked list covers in detail.

Look for the tell that no testing happened: a list where every entry is described in its own marketing vocabulary, with no trade-offs and no losers.

However well written it is, treat a single vivid account of someone's experience as anecdotal evidence, not a finding. One account on one platform generalizes to nothing.

What the catalog publishes instead

The six structural axes are filled from testing, per service, with dates. A reader who wants an order can produce one from them in ten minutes with their own weights.

That order will beat anything an editor could publish because it contains their requirements, not someone else's. The method for reading a service against the axes is in the six-axis comparison. The important property of that output is that it does not order anything.

Common questions

Is the undominated set ever the whole field?

Occasionally, in a mature category where every service has specialized. That is a finding worth stating plainly: there is no dominated option here, and the choice is entirely about which trade suits you.

Bands still put something at the top. Is that not a rank?

The top band is a set, not a position, and its members are ordered by a key that carries no judgment. The difference matters because a reader can see that four candidates are equivalent for their purpose, which is usually true and which a numbered list denies.

What if the editor genuinely knows one is better?

Then say so in a signed piece with the assumptions on the page, and keep it out of the catalog's ordering. Opinions and records age differently and should be corrected differently.

Do readers accept this?

The ones who use the filter come back, because the shortlist was theirs and it held up. A number one that turned out wrong for someone does not bring them back, and it is the most likely outcome, since the number one was computed for a reader who does not exist.