Lessons from building Competely AI Agent
Key insights and costly mistakes from building my first AI agent
Hey, Lior here! In my newsletter, I’ll share insights on how to build and grow profitable online products. Subscribe here, and be sure to follow me on 𝕏 / Twitter and LinkedIn.
After months of hard work, I've just completed building my first AI agent-based product! 🎉
Competely AI Agent is officially live on Product Hunt today, and I'd greatly appreciate your support today - your vote or comment makes a huge difference:
While I've built several AI-powered products before, this is my first AI agent. The past few months have been fascinating, challenging, and filled with valuable lessons. Let’s dive in!
What is Competely, really?
If you’re not familiar, Competely is an AI-powered competitive analysis platform, designed to automate and simplify researching your competitors, saving entrepreneurs, marketers, and product managers days of tedious manual research. Competely helps you understand your competitive landscape and outsmart your competitor in 3 simple steps:
Why I decided to rebuild Competely
I first launched Competely back in December 2023 as an AI-powered competitive analysis tool. Since then, the AI landscape has evolved dramatically. Although the original version of Competely was successful and provided substantial value to many customers, I knew there was room for improvement. Competitive analysis involves extensive, unpredictable research, precisely the kind of task AI agents excel at. Recognizing this opportunity, I decided to rebuild Competely from the ground up using a modern AI agent architecture.
What are AI agents?
There are many definitions for AI agents, but I particularly like the one provided by Anthropic in their excellent article.
Traditional AI products typically use deterministic workflows - meaning their steps and outcomes are predefined and predictable. You ask a specific question or provide a task, and the AI follows a clear, linear path to give you a set response or outcome. For example, a typical AI tool may be programmed to always perform the same actions in response to certain inputs, following a set script or recipe.
AI agents, on the other hand, are less deterministic. Instead of following rigid workflows, they operate more autonomously, deciding their actions based on real-time data and context. They independently plan, reason, and adapt to changing situations. This flexibility makes AI agents particularly effective for dynamic, complex tasks that require ongoing decision-making and adaptability, like competitive analysis, market research, or exploring new and unpredictable areas of knowledge.
In short, while traditional AI tools follow predefined rules, AI agents actively decide what to do next, making them powerful for tasks where adaptability and continuous learning are key.
How AI agents improved Competely
Building Competely with agentic architecture allowed me to significantly enhance Competely’s capabilities. Here are a few examples of how it helped improve Competely:
🔍 Deeper competitor discovery
Our AI agent rapidly crawls the web, performs targeted searches, and analyzes multiple sites within 1-2 minutes. It builds several relevant Google queries using advanced reasoning, and starts crawling and analyzing dozens of pages in parallel. The agent goes much deeper than our traditional competitor search, identifying hidden or less obvious competitors that the traditional workflow missed.
📊 Customized metrics for your industry
Instead of a one-size-fits-all competitive analysis with a rigid set of metrics/fields to compare, Competely's AI Agent dynamically assesses and selects the most relevant metrics several times throughout the analysis. It takes into account your own product, competitors’ products, and everything it has learned so far in the analysis.
🕸️ Smarter crawling capabilities
Our AI Agent intelligently determines which web pages to crawl, and how deep we need to go, based on everything it has learned so far and the type of data that we are trying to extract.
📩 Summarizing changes
Competely AI agent now does automated, autonomous re-analysis of your competitors. The agent compares new results to previous data, highlighting competitors’ changes, trends, risks, and opportunities via clear, personalized summary emails.
🔗 Transparent data sourcing
Competely's AI Agent explicitly provides source URLs for every data point collected while crawling the web. Not only is this helpful for customers, making data verification more straightforward, but this also grounds the model, ensuring it doesn’t hallucinate as much and needs to provide “proof” for any claim it makes.
Key insights from building an AI Agent
I've made plenty of mistakes over the past few months while building Competely AI Agent. Here are three hard-learned lessons I want to share, in case you decide to build your own AI agent:
1. AI costs can run wild with agents
Even though AI costs are dropping, it's crucial to closely track your expenses when it comes to autonomous AI agents. Traditional AI apps typically have predictable resource usage, but AI agents' greater autonomy means their resource consumption can vary significantly. Without careful monitoring, your AI expenses can quickly escalate to hundreds or even thousands of dollars.
Most AI toolkits and SDKs don’t include built-in cost tracking, so you'll likely need to build your own solution. For Competely, I wrapped every API call to precisely calculate the cost of each task - such as finding competitors or doing a complete competitive analysis - based on token pricing from AI providers. Initially, Competely’s AI agent costs were very high, threatening the product’s unit economics. Meticulously tracking these expenses allowed me to identify inefficiencies and optimize usage, ensuring Competely’s pricing ($9-29/mo) still remains viable.
Both OpenAI and Anthropic APIs offer email alerts when usage exceeds certain thresholds. I strongly recommend enabling these alerts at the global and per-project levels. Early in development, even before going to production, I received an alert from OpenAI notifying me of hundreds of dollars in API fees! This highlighted the importance of proactive cost monitoring.
2. Caching is great - until it's not
In most of my previous projects, caching significantly improved performance, user experience, and reduced costs associated with AI model usage, computing, and proxies. Naturally, I implemented a basic caching layer for Competely using a Postgres database. However, this quickly proved problematic due to the enormous volume of data handled by AI agents.
When processing large sitemaps containing thousands of URLs, and crawling deep into dozens of websites, database connections rapidly reached their limits. Handling multiple parallel requests to cache each resource overloaded servers, exhausted connection pools, and impacted database performance. Additionally, the sheer volume of cache operations quickly grew to millions of database entries, complicating maintenance even during beta testing.
Interestingly, the cache hit rate remained low, and fetching fresh data directly from most websites was often just as fast. This realization prompted me to reconsider whether caching truly benefits AI agent workflows. Evaluate carefully the trade-offs between user experience, cost, complexity, and speed. For Competely, getting rid of caching altogether improved both stability and efficiency.
3. Context windows are sneaky
When building AI agents handling large volumes of data, paying close attention to AI models' context window limits is critical.
Although modern models offer large context windows, it's easy to exceed these limits when processing extensive amounts of data. Therefore, carefully selecting what information to include in each prompt becomes essential.
Initially, it may seem beneficial to maximize the usage of the context window provided by the model. However, longer prompts slow down queries, increase costs, and lead to occasional errors, as the AI struggles to pinpoint relevant information within the vast amount of data provided.
For Competely, I started by feeding a prompt with trimmed-down HTML content from multiple pages into an AI model. Occasionally, context windows overflowed, causing errors. My first inclination was to engineer complex solutions, such as sophisticated prompt trimming or parallelizing queries. However, I ended up with a simpler and more effective workaround: whenever a standard model (128K token limit) overflowed, I simply re-ran the exact same query using a larger context model (GPT-4.1 mini with a 1M token limit, which is slightly more expensive). Sometimes the simplest solution is the best one.
Additionally, consistent tracking of input and output tokens became crucial. I built a wrapper around the AI models to track token usage and limits, which helps me dynamically build prompts that balance speed, cost, and reliability.
The future
Building my first AI agent was incredibly rewarding and allowed me to significantly improve Competely. Given the rapidly evolving AI landscape, there's still much to explore, but this experience has made me excited to build more AI agent-based products, so stay tuned!
Meanwhile, if you'd like to show your support, please visit Competely on Product Hunt and show some love - every bit counts:
Until next time,
- Lior
Here are some of my products that can help you:
Competely: get a competitive analysis for your product or idea within minutes, with the power of a diligent AI agent
TailoredRead: learn faster with an AI-crafted book tailored to your goals, interests, and background
BookAuthority: level up your skills and knowledge with book recommendations by experts and thought leaders
Congrats on the launch! Completely looks awesome.