Skip to main content
Said Rahal
Menu
11/04/2026

AI product teams: from planning to experimenting

AIProductDevelopment

This article started with a conversation. On the evening of October 29, when I got back from the gym, I hopped on voice chat with Leonardo, my AI assistant in Grok Bot, and we got into a debate about how my line of work is changing. A lot came up, but one thing got me thinking more than anything else. Why do we plan the way we do? And the short answer is money.

Building is no longer the expensive part

For years, building software was expensive. Every feature took several people weeks or months, and what a company wanted, quite reasonably, was to know for sure what it would cost. That’s where estimates, roadmaps, sprint planning, and a lot of the ceremonies come from. All of it is there so you know how much you’re going to spend before you spend it. But now developing with AI is much cheaper. And when building is no longer the expensive part, the model changes. We go from planning to experimenting.

From planning to experimenting. When building stops being the expensive part, the question changes.

From planning to experimenting. When building stops being the expensive part, the question changes.

Look at what that means. If the question used to be “How much will this cost us to build?”, now it’s “Is it worth trying?” A variant that used to fill a sprint can be ready in an afternoon, so what makes sense now isn’t spending weeks arguing over whether an idea is good, but putting it in front of users and seeing what happens.

I have to be honest here. I don’t have metrics that show we’re moving faster, and the ones we do have don’t make it easy for me. A 2025 controlled study by METR found that experienced developers took 19% longer with AI. But in an update in February 2026, the authors themselves explained that the initial result was already out of date and that their new experiments had serious selection bias because many developers flat-out refused to work without AI tools. Everything points to the real effect being different today and probably making us faster, though it’s getting harder to measure with traditional methods. What I am sure of is that development costs are coming down. And my impression, even if I can’t measure it, is that having several agents working in parallel makes us faster. Something else is obvious, too. A prototype isn’t the same as taking something to production. What really gets cheaper is trying things out, while anything that stays in production still needs the same care as before.

There’s another important point. Cheap doesn’t mean free. The tools cost money, someone has to review what they generate, and above all, there’s the gap between the real cost of developing with AI and subsidized prices. Today’s subscriptions are miles apart from what you pay for API usage at scale. Even if Anthropic or OpenAI change their plans and limits, at large companies the cost is tied directly to actual usage. The real question isn’t whether it looks cheap today for one person, but whether those economics can sustain continuous experimentation at scale. I don’t have a definitive answer, and I don’t think anyone does yet. But even with that doubt, the direction seems clear to me.

A team in two circles

If the model is experimentation, the team can’t be organized the same way it was when the model was planning. I think it needs to reorganize into two circles. The first includes the stakeholders, product, the designer, the manager, and the most senior engineers. Their job is to decide what’s worth trying, formulate the hypothesis, choose the metric that should move, and write down why. The rest of the team is in the second circle. They’re the ones who do the work, turning that hypothesis into a change in production, now with agents writing much of the code, and shipping it ready to be measured.

The two circles. One decides what to try and why. The other runs it with agents and returns a measurable result.

The two circles. One decides what to try and why. The other runs it with agents and returns a measurable result.

What the two circles share isn’t a list of tasks, but a hypothesis with the reasoning behind it. They use decision records for that, short notes with the context, the decision they made, and its consequences. What comes back is a measurable result. The point of the model is to judge the team by the actual impact of the experiment.

And how does a hypothesis get from one circle to the other without getting stuck waiting? Certainly not with sprints. If a task keeps going back and forth between circles, sprints lose their point. Sprints were created to give us predictability when coding was expensive, but these days making a finished change wait until the end of the iteration just slows down learning. What makes sense is to deliver continuously, using a board where each hypothesis moves at its own pace and limiting work in progress so half-finished experiments don’t pile up. That doesn’t go against Agile, but gets back to what Agile is about. The Agile Manifesto puts responding to change ahead of following a plan. Meetings don’t go away, but their purpose changes. Now they’re for agreeing on what to try and writing down why.

I know this way of working will draw criticism, and it’s worth taking it seriously. Some will say that separating a circle that decides from one that does the work sounds like waterfall in disguise. The objection makes sense, but I think it confuses the shape of the process with its purpose. Waterfall has an end goal set in advance, and what it develops is a work plan to get there. Here there’s no fixed end goal and no plan to follow. What passes from the first circle to the second is a hypothesis, and what we’re after when we test it is learning. So the second circle doesn’t get orders. It gets a hypothesis and decides how to test it. You experiment, you measure, and what you learn becomes the knowledge that decides the next step. If the experiment goes badly, the next step isn’t what you wrote down, but what the data tells you to do. Waterfall has no way back. This model is precisely the way back.

Waterfall follows a fixed plan, Scrum iterates on a plan one sprint at a time, and the experiment cycle lets the metric decide the next step.

Waterfall follows a fixed plan, Scrum iterates on a plan one sprint at a time, and the experiment cycle lets the metric decide the next step.

Judgment gets passed on

Decision records try to capture something the code never will. The why. What the code gives you is a record of the result. Imagine an incident, a problem in production. There was a process behind the fix. Discovery, debate, a solution, a mistake, another solution. That process lives in the team’s shared experience, not in the code. And without proper documentation it won’t be in the model’s memory either, because in the end the model’s memory is nothing more than documentation.

You could rightly argue that AI could reconstruct much of that reasoning by reading commits, tickets, and chat threads. Models can navigate Git history better than many developers these days. I’ll grant that it could reconstruct it. But it wasn’t part of the conversation when it happened. It has no memory of it. Reconstructing isn’t the same as having been there. It’s like walking into a renovated house. You see the finished wall, but you don’t know that a pipe burst twice behind it or why the plumber rerouted it. That’s why people’s experience on a team, their time there, and their know-how are still essential.

That brings me to what I think is the most important idea in this whole article. Passing on judgment. You build judgment through your own experiences, the decisions you make, and the mistakes you make, but you can also pick it up from others. You pick it up by being in the conversation where something was decided, listening to why someone ruled out an option, and watching how someone who’s been there for years thinks it through. And when the agent writes much of the code, passing on judgment stops being something that happens by chance and becomes something you have to work at deliberately. That goes for the second circle, too. If they’re never in the conversation where people decide what to try, they’ll never pick up that judgment. This matters even more for junior developers, and there’s already data confirming that there’s less hiring in the areas most exposed to AI. That’s a debate for another article, but I’m sure their big challenge will be showing why the human side matters so they can prove their value. I’m convinced that making a point of passing on judgment is one of the most important things companies need to start doing now. They need to develop it in their employees, not just hire people who already bring it, and of course keep the people who already have it.

Metrics decide

The experiment cycle only closes if you measure. The work doesn’t end when you merge the change, but when you’ve checked its impact. You also need to set the success criteria before you start, including the metric to move, the smallest change that matters, and how long to wait. Stopping the experiment the moment the dashboard shows a “significant” result is fooling yourself with the data.

Measuring changes how we look at spending. Trying something that doesn’t work isn’t wasting money. At Microsoft, they found that only a third of their ideas improved the expected outcome, another third changed nothing, and the rest made it worse. What’s really expensive is shipping without measuring. It took Microsoft eight months to detect a change in Office Online that cut responses by more than 80%. On the other hand, a small idea that sat on the shelf at Bing for six months raised revenue by 12% when they tested it, more than $100 million a year in the United States. Learning from the result justifies the investment.

The multipotential developer

Coming back to where I started, all of this makes me rethink the developer’s role. And here’s the big question. Does extreme specialization still make sense? Backend, frontend, mobile, each in its own box. For years, the market rewarded people who shut themselves away in their own silo and mastered it. Today, if an AI agent does much of the grunt work, what’s left is judgment, and judgment grows out of knowing a little about a lot of things.

Just because judgment matters more doesn’t mean code stops mattering. Good practices are still what separates a product that holds up from one that breaks on the second iteration. What changes is where you get hands-on. You give the work to AI, but you still care about the code, its complexity, and its tests, and that’s still programming. Reviewing what an agent generates takes knowing more, not less.

Knowing more, and about more things. It’s like going back to a much older kind of role. Aristotle wrote about logic, physics, biology, ethics, politics, and poetry, and nobody asked him to pick a single discipline. Newton invented calculus, explained gravity and the composition of light, spent years on alchemy and theology, and ended his life running England’s Royal Mint. For centuries, knowledge advanced thanks to people who crossed disciplines, until specialization came along and brought silos with it. That’s why I think the developer in the experimentation model tends to be multipotential, quite different from a full-stack developer. It’s someone who understands what the designer is saying, what the stakeholder is worried about, and what it costs to build. That’s who I think Anne-Laure Le Cunff’s Tiny Experiments was written for, a book that proposes swapping linear goals for a circular model where you try things, learn, and adjust. Which, oddly enough, is the same thing we’re asking a team to do now.

Learning to make decisions with proactive AI

Mind you, covering that much ground and taking a wider view means rethinking how we learn. These days it’s all too easy to shut yourself away with an AI model and stay there, losing touch with the team and the reality of the business. But the tools play a very different role in how we learn depending on how they work and how we approach them.

A reactive tool like Claude Code waits for you to ask it to do something and then does it. Its main value is that it builds on the technical judgment you already have, letting you take the decisions you already know how to make much further and at a speed you couldn’t have reached before. A proactive bot, on the other hand, works completely differently. It takes the initiative, brings you finished proposals, and suggests next steps. That’s exactly what happens to me with Leonardo, the assistant that started this discussion. It brings me work it’s already done and concrete ideas, and I have to weigh them up and decide what gets approved and what gets dropped.

At first glance, you could argue that supervising someone else’s work makes you rusty on the technical side. It’s an old warning and a very fair one. When someone starts watching over the machine, they risk losing their touch for doing the work themselves. But that objection assumes a passive operator. What a proactive bot tests is the other side of being senior, making decisions, managing risk, setting priorities, and exercising strategic judgment about the business. Your ability to lead and think things through doesn’t atrophy when you delegate the work. You actively train it every time you accept or reject a proposal. Meanwhile, you keep exercising and refining your finer technical skills by working with passive tools that wait for your direct instructions.

Ultimately, the real risk isn’t delegating the task, but delegating responsibility for the judgment. If we let AI decide without testing its proposals against our own judgment, the learning stops. So learning as a developer today isn’t just about memorizing syntax or code patterns, but about making critical oversight a daily habit. Question what you see, check it against the context, and keep practicing making decisions so the judgment stays unmistakably human. And to do that, you often have to step out of the conversation with the model and back into the one with the team.

Certainty moves

All this sounds great to people who enjoy learning this way, but not everyone is comfortable living in a cycle. Some people need certainty, in life and at work, so they know what they have to do tomorrow. And that’s a legitimate need, not a flaw. The planning model gave it to them in the form of a roadmap, a sprint, a list of tasks with dates. If that goes away, what are they left with?

My answer is that certainty doesn’t disappear. It moves. It’s no longer in a roadmap or an end goal set in advance, but in the task itself. Your job is to iterate on the task until you move the metric or show that it won’t move, however you get there. You know what you have to achieve and you know how it will be measured. You decide how to get there, and you can change course as often as you need to. It’s the same for the second circle. It doesn’t get orders. It gets a hypothesis and decides how to get there. I think that’s the certainty experimentation gives you, and for a lot of people it can be more reassuring than a plan, because it doesn’t depend on the plan being right. In a way, what changes is when you expect the result, no longer at the end of the plan, but with each test. What changes is the business’s time preference.

I started by asking myself why we plan the way we do, and the short answer was money. But in the long run it goes further. What we’re living through isn’t just a new tool, but a paradigm shift, and you don’t adopt those, you inhabit them. I imagine companies in the near future with virtual teammates, agents that propose, build, and test while we sleep. In that world, teams are no longer organized by what each person knows how to do, but by what they know how to decide. The developer’s role doesn’t disappear, it expands. It’s less like that of a specialist and more like that of Aristotle, who connected different disciplines to understand the whole. I’m still learning to work this way with forms of intelligence that are reshaping my line of work. That’s why I ask myself the same question from time to time, and I’ll put it to you. If half your team were virtual, what part of your work would still be yours, and what would you have to become to keep adding value?