Marie Aiello, President, ContinuumGlobal, an engagement marketing services company.
I spend a ridiculous amount of my time talking about AI. Which model should we use? Should this be an agent? Is this something we should build ourselves? How much autonomy should we give it? Why did something that worked perfectly last week suddenly give us a completely different answer?
And somewhere in the middle of one of these conversations, it occurred to me that choosing AI is a lot like dating. Everyone wants to find “the one.” You make your list of requirements. You do your research. Other people tell you how great they are. You start thinking this could really work.
Then you actually spend some time together, and suddenly there are things you didn’t see coming. The thing everyone else loves about them drives you crazy. Something you thought mattered turns out not to matter at all. And occasionally the one you almost dismissed surprises you.
AI is starting to feel very similar.
The ‘best’ model isn’t necessarily the best for you.
I get asked all the time which LLM companies should be using. It’s a completely reasonable question, and I understand why everyone wants a definitive answer. I just don’t think there is one.
At my company, we’ve tested different models against the same work and gotten very different results. A model that’s fantastic at one task can be surprisingly average at another. Sometimes the model with the best reputation isn’t the one that performs best for what we’re trying to accomplish.
Performance isn’t the only consideration. There’s also cost, speed, security, context, integration and model interaction requirements to consider. There are also plenty of situations where we discover that we don’t need an LLM at all. A traditional automation or API can perform part of the work more reliably and inexpensively.
That’s not nearly as exciting as saying everything needs to be agentic, but we’re trying to solve the problem, not win an award for using the most AI.
Like dating, everyone can tell you they’re a great catch or that an AI model is the best. That’s nice, but at the end of the day, you’re the one who has to have dinner with them.
A great first date is not enough information.
Time and time again, I’ve seen companies fall in love with an AI demo very quickly. And some demos are amazing. But think of a demo as a first date. Everyone’s prepared. The environment is controlled. The conversation is good. Nobody has met the family yet. You definitely haven’t tried to assemble furniture together. Production, on the other hand, is when you find out what the relationship is actually like.
Now give the agent incomplete information. Connect it to your actual systems. Introduce the strange exception nobody remembered to mention during requirements gathering. Run the workflow thousands of times instead of 10. Let people who weren’t involved in building it start using it. That’s when you learn whether you have something.
Don’t marry your LLM.
I get nervous when companies design an entire AI architecture around the assumption that one model will always be the answer. This market is moving way too quickly for that.
The model that is best for a particular workflow today may not be the one we want a year from now. At the rate things are moving, it may not even be the one we want six months from now.
That doesn’t mean switching every time someone releases a new model. That would be the enterprise equivalent of constantly checking the dating apps to see if someone better showed up, which doesn’t seem like a particularly healthy strategy either. What it does mean is avoiding unnecessary dependency.
When a model changes, we test. Understand whether the output still performs at the same level and whether the prompting, context or other parts of the workflow need to change. When a new model becomes available, evaluate whether it offers a meaningful improvement rather than rebuilding everything because the industry has moved on to the next shiny thing.
I’ve learned to be skeptical of the word “forever” in both technology and dating.
Know what you want.
Having spent the last 25 years working in marketing, operations and client services, I know how much work happens between the boxes on a process diagram. I know that “we’ll just integrate it” can represent six months of work. And I know how many marketing processes exist because “that’s how we’ve always done it.”
That context matters enormously when evaluating AI. If you don’t understand the work, every impressive technology looks like an opportunity.
Once you understand the work, you ask better questions. Where are people wasting time? Where is something routed through a step it doesn’t actually need, just because that’s how the process has always run? Which decisions genuinely require expertise? Where is the information someone needs to make that decision? What would happen if this step took five minutes instead of five hours? Would anything downstream actually change?
That’s when AI becomes much more useful because you’re no longer looking for somewhere to put it. You’re solving a problem.
We’re all figuring this out.
The enterprise world sometimes has a hard time admitting that we don’t have it all figured out. There’s pressure right now to have an AI strategy, an agent strategy, a model strategy and a three-year roadmap for technology that may look completely different three months from now.
Of course companies need a strategy. But part of that strategy has to be the ability to learn and change your mind. We test. We compare. We put things into real workflows. We see where they break. We change the architecture. We retest when models change. We keep what works and get rid of what doesn’t.
There are going to be frogs; probably a lot of them. The trick is getting good enough at knowing what you’re looking for that you recognize the right fit when you find it.
In AI, unlike dating, I’m perfectly comfortable keeping my options open.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?