How to add LLMs to your product? Start by not adding them

LLMs (Large Language Models) are dominating the conversation. Social media is buzzing, investors are intrigued, everyone is looking for AI [insert job title] and leadership is pushing you to integrate them into everything. However, as a product manager, your role is not to follow the trend. It’s to deliver valuable, viable products that solve real user problems. Sometimes, that means saying “no”— at least until you’ve done some homework with your team.

As with any other technology, LLMs are a means to an end, not the end itself. They might transform your product, or they might just add complexity without real value. The key is ensuring any decision aligns with your user’s needs and business objectives. Simply put, your job is to identify the right solution, not to chase the latest technology.

Start with the Right Questions

Before diving into LLMs, step back and ask:

1. What problem are we solving?

At Whitesmith, when collaborating with partners like RaceCrew—an AI-powered racing companion—, we prioritise gaining a deep understanding of their users and unique challenges before proposing any solutions, whether AI-based or not. Without a clear grasp of the problems we aim to solve, even the most advanced tools can fall short or become inefficient overkill. Before considering tools, engage with users to identify their needs and establish initial requirements.

1. What does success look like?  

In Product Management, success operates on at least two dimensions: the company’s goals (viability) and the users’ needs (value). Is your business optimising for growth, engagement, or monetisation? Meanwhile, are your users optimising for speed, precision, cost savings, or efficiency? These dimensions are deeply interconnected, as technical decisions influence business outcomes and vice versa. For example, while an LLM may excel in prototype demos with impressive accuracy, it might fail to scale effectively in production due to high costs, ultimately misaligning with the company’s need for growth and the users’ demand for quick, cost-efficient results. Understanding and aligning success criteria for your business and your users is crucial.

1. What are the options?  

At this stage, you clearly understand the problem and the expectations and success criteria for both the business and its users, and you’re ready to do the needed research to develop your solution. This is the stage where we typically return to the drawing board, analysing potential solutions and their specific trade-offs. It’s also the time to iterate with users, gathering feedback to refine your approach and deepen your understanding of their perception of value. 

Once again, LLMs—or, more broadly, AI and machine learning models—may or may not be the right solution for the problem you’re addressing. It’s essential to invest the time to understand this. Especially when the hype around these technologies is at its peak, my role as a product manager is to ask critical questions and thoroughly evaluate options before committing to any solution or, worse, create expectations for your users, investors and leadership.      

Why LLMs Might Not Be the Answer

While LLMs offer undeniable potential—and don’t get me wrong, I use them daily and love their versatility—they come with significant limitations depending on the use case: 

- Specialized Use Cases:  

General-purpose models like ChatGPT can struggle with industry-specific terminology or niche tasks. In a UX project, we discovered that combining rule-based systems with tailored data models was far more effective at ensuring task accuracy.

- Uncontrolled Inputs:  

User-facing applications often deal with unpredictable inputs. For instance, in a recruitment solution, allowing candidates to provide unrestricted responses introduced a wide range of edge cases—something a deterministic system managed more effectively than an LLM.

- Accuracy vs. Creativity:  

In an AI-powered content project, balancing creativity with factual accuracy proved challenging. While LLMs excel at generating ideas, they often fall short in scenarios where precision is paramount, such as compliance-heavy industries.

These are just a few examples we encountered. Naturally, we explored ways to integrate LLMs and adjust our system to address the blockers or inefficiencies they presented. Still, it was not worth the investment, primarily because of the development costs compared with the value it would bring to users and our business goals. 

That said, LLMs are a powerful and versatile tool but not a one-size-fits-all. Instead of having “add an LLM” as a requirement, focus on understanding the problem and defining what success looks like. Ultimately, your success as a business depends on your ability to align the solution with the problem and the goals of your users and the business.  

From Hype to Impact: Lessons from the Field

In a recent project, we encountered a critical tradeoff: accuracy versus volume. Initially, the instinct was to maximise input volume—analysing more data seemed like a win-win, delighting users and driving business growth. However, during product discovery, we realised that the actual value for users lay in accuracy, even if it required drastically reducing the volume of data processed. Users preferred a system where non-critical data could be quickly reviewed manually, while an LLM model generated highly accurate reports for critical and complex cases. This delicate balance was where the real value emerged for them.

With this crucial insight, we pivoted to a solution that prioritised accuracy, completely transforming our approach and tools. This wasn’t merely a technical adjustment but a product decision deeply rooted in understanding user needs and aligning with business goals. 

The Core Principles Remain the Same

Yes, AI is transformative. Yes, LLMs are exciting. But the principles of great product management haven’t changed:

• Understand the problem: Invest the time to deeply explore user needs, just as we do in our AI engineering partnerships.  
• Define clear goals: Success metrics guide everything from feature prioritisation to tech choices.  
• Iterate to find the right solution: Whether it’s a state-of-the-art LLM or a lightweight, specialised model, the goal is value, not complexity.   

Next time someone asks, “How do we add LLMs to our product?” start with: “How does that help us solve the problem?” 
Technology is a tool, not the answer. By focusing on outcomes, you’ll create products that matter—whether or not they include the latest AI trend.