I’ve been in product management for well over a decade & have been using/building with AI/ML since 2015. Here’s my high-level framework for shipping your first LLM-powered feature:
- Identify a problem worth solving that’s a good fit for Gen AI
- Start by solving the problem with prompting (i.e., have a conversation with your favorite frontier model)
- Define the overall UX (chatbot, under the hood, click-to-generate, etc.)
- Define your core evals (i.e., fast & slow automated checks for result Quality)
- Model selection (select the right model for your unit economics & latency needs)
- Create your minimum viable solution on your target model (get from conversation to single prompt, iterating until you pass your fast evals)
- Protip: don’t rush into fine-tuning or even RAG
- Finalize & implement your slow evals with metrics & monitoring (slow evals incl. things like user feedback and user retention)
- Protip: leverage automation & define rituals for reviewing metrics
- Get it in front of users for learning & validation!
- Ship it! 🚢 🎉
Shipping GenAI features isn’t as hard as it might seem, so get out there and build something your customers will love.
1. Identify a problem worth solving that’s a good fit for Gen AI
- What problems are LLMs good at solving
- Describing/explaining
- Filtering
- Sorting/prioritization
- Comprehension/Understanding/Extracting meaning
- Search
- User Interface / Chat
- Decision Making / Evaluation
- Action taking (agentic)
- Reasoning (but proceed with caution)
- Summarization
- Content generation (but don’t fall into the trap of forgetting all the other use cases!)
- What problems are LLMs not so great at
- Doing math and calculations – even though this is improving all the time, these types of problems have exact answers, and it’s easy for a language model to miss or hallucinate – consider doing the calculation outside the LLM and providing it as a ‘tool’ for the agent to interact with
- Reasoning – while this is listed above, it’s also a challenge to get right consistently enough to make it into a product feature.
- Predicting – this is generally better tackled with machine learning and statistical techniques
- Precision – LLMs are inherently probabilistic, which makes their output approximate rather than precise and unpredictable. That’s part of the magic, but it rules out many use cases and presents some challenges when building and testing product features (but we’ll cover much of that!)
- Ways to find LLM-friendly ideas
- Consider both improvements to existing features & net-new features
- Just like non-AI features, using other products and seeing how they solve problems and taking inspiration works great
- Focus first on understanding the art of the possible, explore and understand various ways LLMs are used in other products, and combine that with your existing customer knowledge and the problems they need to solve
- Look at repetitive tasks in your existing product flow. LLMs can often help automate or augment them. Think about things with lots of steps, lots of clicks, lots of (mundane) typing, and even lots of reading. This isn’t always the highest value use, but it can be a great place to get started!
- Start with the data – LLMs are great at working with high-quality data (just like any other AI tech), so start with high-quality, unique, and continuously growing data and think about how you could put it to work
- Consider internal problems as well as external problems. LLMs can be great for internal products and processes, too!
- Start with user problems – start with problems worth solving and think about how LLMs might help solve the problem. You can even use an LLM like Claude or ChatGPT to help with that part
- Don’t forget to collaborate with others. LLMs are on many people’s minds right now, so harness their unique perspective and creativity by facilitating a brainstorming session or otherwise collecting their ideas. And even mediocre ideas can spark great ideas!
- Consider how making your product proactive instead of passive. Consider how you might use an LLM to preempt the needs of your user.
Other sections coming soon =)