An opaque pipeline is not a technical choice. It is a pricing strategy. The price you cannot see is the price you pay later.
The renegotiation clock
Every black box has a clock on it. The moment the vendor finishes delivery, the clock starts. It runs against you in three predictable ways.
The first call after delivery. Something behaves oddly. You cannot see the prompt, the model version, the eval set, or the cost-per-call. Only the vendor can diagnose it — and that diagnostic time is now billable at a rate not in the original scope (Class C — observable in the change-order paperwork on almost every black-box engagement we have audited).
The first dependency change. A model is deprecated. An API price doubles. A library goes end-of-life. If the pipeline is opaque, you do not know which one until something breaks. The vendor knows. That asymmetry is the renegotiation.
The first attempt to leave. You decide to move in-house or to a different vendor. The export is a screenshot. The "documentation" is a sales deck. A re-implementation quote arrives, conveniently from the same vendor, framed as the cheap option compared to starting over.
None of these events are accidents. They are the return on the opacity that was sold to you as "we handle the complexity so you don't have to."
Why the market produces this
Opacity is rational for the vendor. A documented pipeline can be operated by someone else. An undocumented one cannot. The vendor who hands over a complete runbook has, from a pure-revenue standpoint, just deleted next year's invoice. The vendor who hands over a UI and a login has just secured it.
This is not a moral failing of individual vendors. It is what the incentive structure of the AI services market currently selects for. The fix is not to find a more virtuous vendor. The fix is to make opacity contractually expensive — to write the engagement so the vendor's economic interest aligns with yours.
The hiring side of this market tells the same story from the other direction. People who can ship and explain a working artifact end-to-end are paid more than people who can only operate a vendor's interface (Class E — consistent with hiring-marketplace data, as we read it). If end-to-end transparency is what the labor market rewards, it is also what a client should be buying.
The transparent-pipeline counter-pattern
Our standard engagement makes the pipeline auditable by clause, not by promise. Four artifacts are non-negotiable.
A versioned recipe. Every prompt, model identifier, parameter setting, and orchestration step is in a versioned file the client owns from day one. When something changes, the diff is visible. When something breaks, the suspect line is findable.
A named eval set. Before we ship, we agree on a fixed set of inputs and the outputs we consider correct or acceptable. The eval set ships with the pipeline. Anyone — including your future internal team or a competing vendor — can rerun it and see how the system behaves today.
A cost-and-dependency sheet. Every external service, API key, model version, and library the pipeline depends on is listed with a monthly cost estimate and a substitution note: if this provider doubles its price or disappears, the replacement is X, cost delta Y. Clients have a right to know how their thing will die.
A runbook a stranger can run. The acceptance test is that someone outside the engagement can take the recipe, the eval set, the dependency sheet, and the runbook, and operate the pipeline from scratch — without our help. If they cannot, the artifact is not done. We have failed that test before, which is why it is the test.
These four artifacts are not extras. They are the deliverable. The pipeline is the thing they describe.
What this changes about your buying position
Once a pipeline is transparent in the sense above, three things flip. You can shop the dependencies — the cost sheet is a document you can take to another supplier. You can fire the vendor — the runbook test guarantees it; the engagement that cannot be fired is the engagement that costs the most. And you can ask harder questions: "show me last quarter's eval results" means something when the eval set exists, and nothing against a UI and a login.
The questions to ask, in writing
If you are evaluating a pipeline build — ours or anyone else's — these are the four to put on paper:
- Where is the versioned recipe? May I see the diff between last month and this month?
- What is the eval set, and may I rerun it without your help?
- What does this depend on that I am not being shown, and what is the substitution plan if any of it fails?
- Can a person outside this engagement operate the pipeline from a runbook? When was that last tested?
A vendor whose contract cannot answer those four is selling you the renegotiation, not the pipeline.