Salma Mayorquin PRO

Get Outrider: github.com/remyxai/outrider

Replay is a good idea to estimate FNR.
But I'm willing to pass on an idea if it helps keep the FPR low.

The FPR is a real concern for experimentation leads, check out Kohavi's post on how false positives carry the risk of steering your program in the wrong direction: https://www.linkedin.com/pulse/cost-false-positive-ab-tests-ron-kohavi-mkswc/

So we're helping teams triage which candidate ideas to try next, gating on evidence and external validation.

There is a natural mechanism for identifying false negatives, that's related work in the literature.
If scientists keep producing extensions and replications, we can surface these variants for testing.

The audit process you've laid out for understanding the false positive rate makes sense.

But I also agree that it may be a tough sell, many teams are building and making deployment decisions using different eval methods, and can't tell you about their own success rates, because few are performing this audit.

We build and advocate for humans to review the diffs.
And you can determine your own false positive rates to instrument the gates.

Outrider is designed for teams to progressively ramp up their investment in an idea by filtering candidates through cheap evaluation gates before paying the greater cost of maintainer attention or expensive testing, like online controlled experiments.

Appreciate the attention to false positives and negatives in these filters. In a recent chat with head of experimentation for a top 100 tech company, I heard:

"When code is almost free, when ideas are almost free, how do you make sure that the thing actually sticks. The part that keeps me up at night is, if we get to the point where we shift less than 10% of the treatment arms that we try, then we're probably shipping more false positives than we are true positives. And at that point, we've gone into a thrash machine."

So it seems that with AI coding in practice, people are more willing to accept false negatives than pay the higher cost of maintaining false positives. But method papers are not i.i.d. and truly good ideas will keep popping up in the literature through ablations and extensions in related work.

We're interested in helping teams close the loop in their AI engineering and believe understanding the causal mechanisms behind why one method helps or hurts is key to efficiently exploring the candidate space.

Hey @dipankarsarkar
That's interesting, I know many teams understand A/B testing as the gold standard in evaluating their software changes. But they are looking for cheap offline methods like benchmarks and judges to help predict what they'll find in the slower, more expensive online tests.

"Mutation testing" resonates, we're talking about how tranferability of a method and its robustness to perturbation is important to understanding its value and impact in your own application.

With Outrider, we aim to abstract to the level of helping teams apply all sorts of testing methods to gate the launch/no-launch decision. Layering cheap filters early to prune the candidate search space before progressively ramping up to the online tests.

For most organizations "the oracle" is impact on a business metric that the team cares about. See how Zuckerberg describes AI evaluation and their "north star" at Meta (https://www.dwarkesh.com/p/mark-zuckerberg-2)

Outrider is modeled around these software engineering best practices. A feature proposal is measured against the baseline to make decisions around merging.