Picture this: a fine-tuning job for a customer-support chatbot at a mid-sized SaaS company. The team that built it was three engineers, all of whom left for a competitor in the same quarter. The job has been running for eight months on a reserved GPU cluster, producing checkpoints that no one has looked at since the week they left. The lease on that cluster runs for another sixteen months, and the termination penalty is steep enough that finance would rather eat the cost than cancel early. The job is still in the scheduler, still consuming power, still generating a steady stream of loss curves that flatten into a straight line. This is the quiet reality of machine learning infrastructure in 2026: the most durable artifact isn't the model, it's the lease.

The Lease Outlives the Model

Think about the lifecycle of a machine learning project. A team forms, picks a base model, writes a fine-tuning pipeline, and reserves GPU capacity. The reservation might be a committed use contract, a spot pool with a minimum spend, or a reserved instance that the finance department signed off on because it was cheaper than on-demand. The model gets trained, evaluated, and deployed. Six months later, a better base model appears, and the team fine-tunes that instead. The old model is archived, the new one takes over. The lease, however, is still ticking.

The disconnect is structural. Hardware procurement operates on a different clock than software development. A GPU cluster is a capital expense, amortized over years. A model is a software artifact, iterated in weeks. When the two clocks misalign, the slower one wins. The lease outlives the model, and the fine-tuning job that was supposed to be temporary becomes a permanent resident of the cluster.

Capacity planning, which used to be about predicting how much compute a project would need, has become a form of archaeology. Engineers dig through old tickets to figure out why a certain node pool exists, what workload it was sized for, and whether that workload still matters. In many cases, the answer is no. The pool was sized for a model that got replaced, but the lease renewal was automatic, and no one in the current org even knows the contract exists.

The 2026 reality is that hardware outlives software by a wide margin. A typical GPU lease might run for one to three years, while a state-of-the-art model might be superseded within six months. That means the average lease will see at least two, maybe three, generations of models. The engineers who signed the lease are long gone by the time the third generation ships. The job of managing that hardware, and the fine-tuning jobs that run on it, falls to whoever inherits the cluster.

When the Team Leaves, the Job Stays

The most common scenario in 2026 is the orphaned job. A team builds a fine-tuning pipeline, gets it working, and then the team dissolves. Maybe the company pivoted, maybe the lead engineer left for a competitor, maybe the project was deprioritized but not cancelled. The job stays in the scheduler, consuming GPUs, producing checkpoints that no one looks at, and incurring costs that no one questions. The runbook that explains what the job does was written by someone who no longer works there, and it was written for an audience of one: the author.

I've seen this happen at three companies. The pattern is always the same. A farewell email goes out, the departing engineer hands over a wiki page with a diagram and a few commands, and then the job sits there. The scheduler shows it as running, the metrics dashboard shows loss curves that have flattened into a straight line, and the alerting system, if it was ever configured, fires occasionally. No one responds, because no one knows what the alert means.

The cost center never sleeps. A single fine-tuning job on a modest cluster can burn through tens of thousands of dollars a month, and if the job is no longer producing anything useful, that's pure waste. But shutting it down requires someone to take responsibility. That means understanding what the job does, whether the checkpoints are valuable, and whether the lease can be released without penalty. Most engineers, faced with that uncertainty, choose to leave it running. It's easier to explain a steady cost than to explain why you deleted something that might have been important.

This is the quiet tragedy of long-lived leases. The job that outlives its team becomes a liability, not an asset. It consumes resources, both financial and cognitive, and it blocks the cluster for work that actually matters. The only way to avoid it is to write for the successor, to document the lease and the job as if you expect to be hit by a bus tomorrow. Most people don't.

API Drift, Wire-Level Reality

Even when the team stays, the API layer drifts. Inference APIs change faster than models. A provider might deprecate an endpoint, change the request schema, or alter the pricing model with a week's notice. The fine-tuning job that calls that API is now broken, but the lease is still running. The job might fail silently, or it might retry with exponential backoff, burning cycles on a request that will never succeed.

Protocols are the only stable contract. The wire format, the HTTP headers, the authentication scheme, those are what actually bind your job to the cluster. If the provider changes the protocol, your job breaks. If the provider keeps the protocol but changes the semantics, your job breaks in subtler ways. The training data you're sending might be interpreted differently, the loss function might behave unexpectedly, and the checkpoints you produce might be worthless.

Versioned endpoints are a partial answer. Many providers now offer explicit versioning, so you can pin to a known-good API. But versioning doesn't help if the underlying model changes. A fine-tuning job that was written for a specific base model will produce garbage if the base model is swapped out underneath it. The version number on the endpoint is a lie, because the model is a moving target.

The wire is the real documentation. If you want to know what a job actually does, you have to read the traffic. The code that calls the API is a story about intent, but the wire is the truth. That's why the engineers who inherit orphaned jobs spend their first week sniffing the network, not reading the README. The README was written by someone who wanted to move on. The wire doesn't care about your feelings.

But let's be fair: versioning and pinning can mitigate drift. Providers like OpenAI and Anthropic offer versioned endpoints that are stable for months, and many teams pin to a specific version and test before upgrading. For internal APIs, you can enforce a contract with schema validation and backward-compatible changes. The catch is that these measures only work if you're diligent about upgrading and testing. When the team is gone, that diligence goes with them. The pinned version becomes a frozen artifact, and the inevitable drift happens anyway, just slower. The wire is still the truth, but it's a truth you can sometimes postpone.

The Tragedy of the Shared Cluster

Shared clusters are the commons of the ML world, and they're overgrazed. When a cluster is shared across teams, each team has an incentive to grab as much capacity as possible, because capacity is free at the margin. The lease is already paid for, so the marginal cost of running a job is zero. The result is a queue that's months long, with jobs that were submitted by people who have since left the company, sitting next to jobs that are critical to a launch that's happening next week.

Flock-style monitoring, the kind that watches the whole flock of jobs rather than any individual, is the only way to see the problem. But even with monitoring, the incentives are misaligned. No one pays for idle capacity, so no one has a reason to free it. The team that's waiting for their job to run is the team that's most likely to notice the waste, but they can't do anything about it, because the orphaned jobs belong to someone else.

The commons get overgrazed because the cost of a single job is invisible. A manager sees a utilization report that shows 80% of the cluster is busy, and that looks good. But 60% of that utilization might be jobs that are stuck in a retry loop, or jobs that are producing checkpoints that will never be used. The utilization number is a lie, because it doesn't distinguish between useful work and noise.

Some companies have tried to fix this with chargeback systems, where each team pays for its share of the cluster. That helps, but it only works if the teams are still around to be charged. When a team dissolves, the chargeback goes to a cost center that no one owns, and the jobs keep running. The tragedy of the commons is that the commons are infinite, but the people who use them are not.

What Your Successor Needs to Know

If you're writing a fine-tuning job today, you have a responsibility to the person who will inherit it. That person might be you, six months from now, when you've forgotten everything. Or it might be a stranger who found your job in the scheduler and is trying to figure out whether to kill it. Write the README for that person. Include the lease terms, the cost, the expected lifetime, and the kill switch.

Leave a kill switch for your own code. A simple environment variable that, when set, stops the job gracefully and exits. It sounds trivial, but most orphaned jobs don't have one. They have a complex web of dependencies, and the only way to stop them is to cancel the lease, which has a penalty. A kill switch is the difference between a clean exit and a messy one.

Document the lease, not the model. The model will be replaced. The lease is the thing that outlives you. Include the contract ID, the renewal date, the termination penalty, and the contact for the provider. If your successor wants to cancel the lease, they need that information. Without it, they're stuck.

Pin dependencies, but also pin intent. The requirements.txt file is not enough. Write down what the job is supposed to do, what success looks like, and what the checkpoints are for. That way, your successor can decide whether the job is still relevant. If the intent is unclear, they'll assume it's important, and the job will run forever.

Hedging the Numbers You Can't Verify

Fleet sizes vary wildly. I've seen clusters with ten GPUs and clusters with a hundred thousand. The numbers are hard to pin down, because companies don't publish them, and the ones that do are often marketing. Some estimates put the largest training runs in the range of tens of thousands of GPUs, but that's a guess. The point is that the scale doesn't matter as much as the lifecycle.

Hourly rates for GPU compute are in the single digits, but they vary by provider, by region, and by commitment level. A reserved instance might cost a fraction of on-demand, but it comes with a lock-in. The lease terms are the real cost driver, not the sticker price. A one-year commitment on a node that's idle is more expensive than a three-month commitment that you actually use.

Contract lengths are typically months to years, but the trend is toward longer commitments as providers offer discounts for multi-year deals. That's great for the provider, but it means the lease is even more likely to outlive the model. The longer the contract, the more generations of models it will see, and the more likely it is that the team that signed it will be gone.

Assume churn, not stability. The people who work on a machine learning project are a temporary feature. The hardware is permanent. Plan for the worst case, where the team leaves, the model is deprecated, and the lease is still running. Hope for the best case, where the model ships, the team stays, and the lease is renewed on better terms. But build for the worst case.

The Human Cost of Long-Lived Leases

The human cost is the part that doesn't show up in the capacity report. Engineers burn out on maintenance, on babysitting jobs that were written by someone else, on fighting a cluster that's full of ghosts. The institutional memory, the knowledge of why a job exists and what it's for, is the real asset. When that memory walks out the door, it's gone. The lease is still there, but the knowledge is not.

Your career outlives the model, too. The fine-tuning job you write today will be running after you've moved on, and someone will have to deal with it. That someone might be a junior engineer who's been given the task of cleaning up the cluster, and they'll have to reverse-engineer your intent from the code and the logs. Make it easy for them.

The team is a protocol, not a group. A group of people is a temporary arrangement. A protocol is a set of rules that persist. The handoff between teams is a protocol, and it's the most important one in the system. If the protocol is broken, the knowledge is lost, and the cost is paid by the next person who inherits the job.

Build for the person who inherits your work. That person is not your friend, not your colleague, and not you. They're a stranger who will have to make decisions based on what you left behind. If you leave them a clear picture of what the job does, why it exists, and how to stop it, they'll be grateful. If you leave them a mystery, they'll curse your name. In 2026, the lease is the only thing that's guaranteed. Make sure the knowledge outlives the team, even if the job doesn't.

Here's a concrete example of a successful handoff. At a fintech startup, a data scientist named Priya built a fine-tuning job for a fraud-detection model. She knew she was leaving, so she spent her last week writing a runbook that included the lease contract ID, the provider's support phone number, a diagram of the data flow, and a one-page explanation of why the job existed. She also set up a cron job that would automatically cancel the lease if the model's accuracy dropped below a threshold for two consecutive weeks. When she left, the job ran for another six months, then the accuracy dropped, the cron job fired, and the lease was cancelled. The company saved $40,000. That's the kind of handoff you want to enable.

So, do this: before you move on to your next project, take an hour to write the runbook you'd want to inherit. Include the kill switch, the lease details, and the intent. Your successor will thank you. And if you're the one inheriting, don't be afraid to kill the job. It's easier to explain why you stopped a job than to explain why you let it run for a year.