This is not another AGI discussion (well, it’s a monologue). I do not believe the evil artificial intelligence will threaten us all in the near future. However, if you continue reading past this point, expect a long article with diagrams and (mostly wrong) opinions on AI, employees, and cofounders.

A tale of an impossible AI cofounder

I do not understand things very well until I can visualize the way they work. In tech, the easiest way to understand how things work is to try to recreate them or use them to build something.

From the first moment I tried using LLMs, it was clear that they are amazing and have huge potential. Lav and I kept sending each other amazing demos we saw on Twitter. Then we tried recreating them just for fun and learning purposes. Most ended up with facepalming and “of course, it does not work except for this specific case in the demo!” But we learned a lot.

Then, one day, we decided to try building a product using AI (like building one product is not hard enough). And to make things even more interesting, we decided to use AI as a cofounder and create something without any employees or external help (while still running and working on Vacation Tracker full-time). This decision set us on a journey of 36 working days with CofounderGPT (over 5 months), which was fully documented here: https://knowlo.co/blog/day-1-startup-adventure-with-cofoundergpt/. And then, we just stopped. From the very beginning, it was clear that this experiment had failed. It was impossible to have a real AI cofounder. However, we continued because it was fun and we learned a lot.

But why was it impossible? Were we too early? Would it be different today with GPT 5 Pro, Claude Opus 4.1, Cursor, reasoning models, and all the other amazing tools we have access to? It would be different for sure. And we would have an illusion that our experiment might succeed for a long time.

There were and still are many problems with the potential AI cofounder. Some are very obvious, such as the context window (the amount of data an AI can keep in its memory), reasoning (the ability to think through and analyze problems), and available tools (the things an AI cofounder can use to do its job). All these things are getting better and better. Just look at Cursor, Claude Code, OpenAI Codex, and other coding AI tools today. The improvement of AI coding abilities over the past few years is so big that it’s not even measurable anymore.

However, a cofounder is more than a set of skills. Cofounders have visions, motivation, passion, and many other qualities that an AI cannot replicate. They make decisions (which is easy for an AI) and accept the potential consequences of these decisions (which is not even close to possible with LLMs and the current state of AI). And they have emotions and gut feelings.

To be honest, a true AI cofounder would not be possible from a legal perspective at the moment. However, the focus was on cofounder skills, not legal matters (they are not fun anyway).

Ok, let’s say that cofounders are a bit out of reach of the AI in its current stage. But what about employees? Will AI replace developers and other employees in the near future?

To understand that, let’s rewind a bit and try to understand how people do things.

A case of AI employees

I enjoy using AI for coding, research, and product management. Writing code or exploring and building a feature prototype has never been easier. I feel a lot more productive. Everything moves so fast! New tools, new models, better practices. Does that mean that we’ll have full-time AI developers replacing actual people soon?

If you compare the number of data center vs office space construction in the US, it seems so. The trend of building space for GPUs and our future digital employees is rising rapidly.

Source: https://x.com/JosephPolitano/status/1951740903925715126.

But, of course, it’s not that simple. To understand why, we need to go back to the core goal of each company: to make money. Whatever a company does, it needs to generate revenue to survive. Even hot new startups that focus on market capture instead of profits need to make money through investments.

How do companies make money? By selling products or services (providing value) to entities that can pay. So, companies hire employees to do things that lead to creating and selling products or services (directly, such as sales, programming, or indirectly, such as management, HR, etc.).

So, our oversimplified value chain could look similar to the following.

1. Companies need to make money.
2. Companies make money by creating and selling products or services (value) to entities capable of purchasing them.
3. Companies need employees to create and sell products or services.

Obviously, it’s more complex than that. But if we focus on the employees, we can simplify even more and say:

1. Companies need to make money.
2. Companies make money by doing something (creating, selling, etc.).
3. Companies need employees to be able to do that “something.”

But what type of employees are we talking about here? It’s quite different when we talk about factory workers or product managers and software developers. Or is it? In a timeline long enough, all jobs will either become obsolete and disappear or become fully automated.

But let’s focus on what we call knowledge workers. “Knowledge workers are professionals whose primary asset is their mental skills and expertise, used for critical thinking, problem-solving, and generating new information rather than just performing manual tasks.”

How do knowledge workers do their jobs? The answer is obvious: it depends on the actual job. Their main asset are their mental skills and expertise, but they use various tools to accomplish more and amplify their skills.

Initially, knowledge workers did everything manually, but eventually, they or someone else created a variety of tools to automate and simplify parts of their jobs, enabling them to complete more tasks and reduce the number of mistakes. For example, software developers have frameworks, linters, and many other tools. But we now have AI, too! Where do Cursor, Claude Code, OpenAI Codex, and other tools fit?

Let’s try to visualize this evolution. First, we’ll take our company value chain and add another axis to it, which represents an evolution of all these components (an X-axis). We’ll get something similar to the diagram below. Actually, this diagram is a special type of map, known as a Wardley Map. The Y-axis represents a value chain with a user (or an anchor) at the top (Company) and all of its needs (company needs, interaction channels, activities, etc.) below. Needs higher on the vertical axis are more visible and important in the context of the map than those lower in the value chain.

Wardley Maps define the following stages of the evolution:

1. Genesis - Novel, uncertain, constantly changing
2. Custom Built - Growing, becoming more defined
3. Product - Stable, well-defined, widely available
4. Commodity - Ubiquitous, standardized, utility-like

It’s a bit hard to fit employees in these evolution stages. We do not think about human evolution in the same way. However, we could fit the work they do.

Initially, we had knowledge workers doing everything manually, which was the genesis stage, because it was novel. Then knowledge workers started using tools to amplify their skills, which was (and still is) a custom build stage. This may be a bit confusing because the actual tools they (or we) use are often in the product or even commodity stage, but the way we use them is still custom, and it’s slowly becoming more defined. Finally, we see these amazing AI tools that are still not stable enough, but they are slowly moving knowledge workers to the product stage: an employee + AI automation will become well-defined and widely available over time. But what’s next? What would be the actual commodity version of a knowledge worker? Fully automated (AI) employee, I guess.

Scary, right?

It is! Luckily, it’s as true as the Windows updater telling you it’s 99% done.

So, is AI coming for our jobs? And if it does, when?

Is AI coming for our jobs?

The only proper way to answer this question is to define what our job actually is. As I already said, if we look far enough in the future, all currently known jobs will eventually become either obsolete or fully automated. But we don’t really care about thousands of years from now. We care about the relatively near future (decades, I guess).

Let’s take software developers as an example. If you are a software developer and you think your job is to write code, then I have bad news.

Luckily, businesses rarely care about code (yet some care even about GitHub stars, I know). Ok, but what are the actual responsibilities of a software developer, then? Companies need to make money, so it must be something related to earning more money or spending less money. They need to use their mental skills and expertise to:

• Intent & prioritization: Decide which problems are worth solving, define success metrics (revenue, retention, service levels), identify constraints, and set sequencing.
• Boundaries & contracts: Shape the domain model and API/data contracts to keep change cheap and risk contained.
• Code & implementation: Turn intent into working software: write code and tests, integrate with existing systems, and document expected behavior.
• Verify & operate (reliability & maintenance): Verify changes using tests and controlled rollouts, run the system in production with monitoring, alerting, error tracking, incident response, rollbacks, etc.
• Security: Define trust boundaries, access controls, and data‑handling rules; manage vulnerabilities, exceptions, and risk acceptance.
• Delivery & pipelines: Automate build/test/deploy/rollback, with quality and compliance checks, ensuring changes ship safely and predictably.
• Long‑term strategy & evolution: Choose a target architecture and evolve it in alignment with the business strategy; manage tech debt and total cost of ownership.

Obviously, these depend on the software engineer’s expertise. It’s not the same for frontend and backend engineers, juniors and seniors, etc. It also depends on the person who is preparing the list. Finally, it depends on the LLM and model we use to analyze the list of essential software engineer activities. Yes, of course, we use LLMs to assist us with almost everything today.

AI affects most of these activities, but some have a high potential to be fully (or mostly) replaced by AI, and other activities will be augmented by AI. Let’s see activity by activity:

• Intent & prioritization: AI can help you find relevant information faster and help you evaluate everything you need to know (you can run potential scenarios through brainstorming-like sessions, etc.). It could help with KPIs, ROI, documentation, and many other things. But, it can’t replace your judgment, among other important things. AI could help, but it’s, again, up to you to use and apply that help and come up with something that makes sense in each specific case (so, it’s still in the custom built phase).
• Boundaries & contracts: API schemas, payloads, examples, scaffolds, and similar things are (or soon will be) a commodity with AI. These will be mostly automated. However, you’ll still need to define domain boundaries and understand the domain. However, this will get more well-defined over time, so I think it’s closer to the product phase.
• Code & implementation: This one seems obvious, AI already writes code well enough. Over time, you’ll write code only in specific cases, such as very novel logic, examples of tricky integrations, or in cases when you need to gain a very high level of performance in your area of expertise (and when I say expertise, I mean you are one of the best, not just good). So, this goes to the commodity stage.
• Verify & operate (reliability & maintenance): You still need to make decisions, but everything gets easier to decide (AI does triage, correlates logs and traces, sends PRs, suggests decisions). I would say that this goes to the product phase and gradually shifts to the commodity phase over the years.
• Security: Something similar might happen to security. It’s still crucial for you to make a decision and have the final word, but applications are continually improving. You make decisions, and the app provides what you need. Well-defined, widely available => product phase.
• Delivery & pipelines: This is another area that we can see easily dominated by AI in the relatively near future. You might still need to think about the release strategy and to give final approvals in some cases, but CI/CD, versioning, changelogs, canaries, rollbacks, and similar are strong candidates for the full AI automation (shifts to the commodity stage).
• Long‑term strategy & evolution: Similarly to the intent and prioritization, AI will be able to give you all the critical information and assistance you might need. For example, codebase analytics, refactor opportunities (or even PRs), potential impact analysis, and different potential paths. However, it’s still up to you, your understanding of the goal, judgment, and a team to use these and come up with something that makes sense for the current and desired state of the business you are working on (or for). This stays in the custom built phase.

Here’s the updated map:

So, what do you say, is AI coming for software engineer jobs? AI has high potential to eliminate the typing aspect of the job really fast, but a software developer’s job is not typing code. It’s problem-solving through thinking, judgment, communication, and writing instructions for computers to follow.

We are going in circles. Now that we can get the code faster, we started talking about better specifications, then we’ll focus on tests, and so on. The only important difference is that, as my good friend said recently, these circles are not actually circles; they are a big spiral, and everything goes faster and faster.

Focused and recharged super employee

If AI automates most of the typing aspect of the software engineering job, we can expect something similar for other knowledge workers sooner or later. However, that does not sound so scary if you remember that the main asset of knowledge workers is their mental skills and expertise, used for critical thinking, problem-solving, and generating new information rather than just performing manual tasks.

We will have more time to devote to the essential things, such as thinking and judgment. Ok, that sounds a bit scary.

However, there’s one problem with that: you can write code 14 hours per day, but you can’t really make good decisions and process large amounts of high-quality information that long.

You need to be focused and rested.

Are you a cofounder, software developer, marketing manager, product manager, or do you have some other role? It does not matter. What matters is that you have enough focus and mental power to review all generated code, prepare prototypes, write instructions (prompts), orchestrate all agents, and decide what matters. These activities burn your energy faster than AI burns tokens. You can’t do that and have 10 meetings a day. Or work long hours and weekends. You need your battery at 100%.

How do you recharge and get focused? That’s different for each of us. However, skipping meetings that do not provide value, replying to non-urgent Slack messages or emails the next working day, closing your laptop after you finish your workday, and taking PTO are definitely helpful.

I don’t want to scare you, but someone on Twitter said that “most jobs will become obsolete” in less than 10 years. McKinsey agrees (they say AI Agents will replace 70% of office work), and Goldman, too.

So, I guess our clock is ticking. We don’t have much time. It’s probably better to take a woodworking course or something similar.

But I am not that good at woodworking. So, let’s try to understand how AI Agents work and if they are that scary.

If you read Twitter or Linkedin, AI Agents look like special agents that can do everything. The demos that they share look amazing.

However, these agents don’t feel that special when you use them. They are helpful for specific cases, just like travel agents. If you give enough details about your desired journey and budget to travel agents, they can find you the vacation you want and plan the entire trip. Just like Cursor! If you give enough details and specific instructions, vibe coding feels like magic (parts of the application start assembling in front of you). In other cases, Cursor feels as smart and useful as Alexa or Siri.

So, AI Agents can be extremely helpful, especially if you understand how they work. But before we can understand AI Agents, we need to understand LLMs.

How LLMs work

Large Language Models or LLMs are very good at predicting the next best set of words based on your input (questions, part of the text they need to complete, or detailed instructions), their training data (all the text that creators of the LLM you use were able to use for training, such as books, websites, your private data [just kidding, or am I?], and other datasets), context (the previous conversation flow or documents you attached), and specific configurations (such as weights, which prioritize certain word patterns, and settings like temperature, which control the randomness of predictions).

Let’s use the same example I used in the “5 Prompt Engineering Tips for Developers” article! If you ask an LLM to finish the following sentence: “I am speaking at,” it’ll probably say something such as “a business conference,” “a tech meetup,“or “a community forum.” There’s almost zero chance it would say, “A Martian picnic?” Or “a space farmer’s market.”

However, if we add a few sentences to the beginning of the instructions (or prompt, as we call it when talking to LLMs) that tell an LLM that it is a playful, chatty cartoon character named"Space Bunny,” the LLM would not finish the sentence with “a tech meetup,” or similar, but with something more similar to a Martian picnic.

When you talk to an LLM, your question or a set of instructions is called a prompt. So, prompts are just instructions. You tell an LLM what you want, and it tries to reply based on your input and context, as well as its training set and configuration. If your instructions are clear, there’s a higher chance you’ll get a helpful reply. However, an LLM will reply even if your instructions do not make sense. In that case and in some other cases, its replies might not be based on truth (we call that hallucinations). Everything related to hallucinations is improving fast, so whatever I write here will probably not be true in a few months.

So, you give your instructions (or write your prompt if you want to sound smarter), LLM takes these instructions, spins up some GPUs, burns a small forest, “eats” some of your tokens, and you get an unexpected wisdom or a hallucination. In the world of LLMs, tokens = money. You burn them like the Monopoly money, but the key difference is that LLM tokens are connected to your credit card.

But how does an LLM know how to reply to your prompt?

Computers are not that great with words. They prefer numbers. So, an LLM will split your instructions into tokens (yeah, these are the tokens I mentioned above). A token is a set of characters that is sometimes equal to a word, sometimes to the part of the word, and sometimes to a set of letters and other characters such as spaces, dots, commas, etc. The exact number of tokens your instructions have depends on the algorithm the LLM uses. You can see the visualization of OpenAI’s tokenizer in the image below or here: https://platform.openai.com/tokenizer. You’ll get slightly different results based on the model you select.

But tokens are still words! A tokenizer represents each of these tokens as a set of numbers (so each token becomes an array of numbers). These numbers are vectors that can be placed in a multidimensional space. The entire training set of an LLM is also transformed into tokens and then vectors and put in the same multidimensional space. The major power of LLMs is their ability to put related words (based on their vast training sets) close to each other in this space.

For a quick visual example, imagine that each token converts to an array of two numbers (two-dimensional space is easy to visualize). Then, we would be able to put our dots in this space similar to the following image:

Now that an LLM converts your instructions to a set of vectors (an array of arrays of numbers!) and puts them in its multidimensional space, it can use its algorithm to find the closest vectors that might be a good answer to your instructions. LLMs are Large Language Models, meaning they are trained on a massive set of data, which helps them put these vectors in the correct places in the multidimensional space and offer a meaningful answer.

Luckily, LLMs are products, and like other products, they evolve with user feedback and misusage. So we got many useful features that were not initially available, such as system prompts (parts of the prompts that are more important than the rest of the conversation with an LLM), better coding and JSON skills, etc.

An undercover agent

Meet my friend Claude. I ask it many weird questions every day. Claude is nice, so it tries to answer each question in detail politely.

One day, I asked Claude what the weather was like in Belgrade. I ask way more weird questions to both Claude and ChatGPT. But this question is special!

It’s special because Claude can’t answer it. It told me politely that it had no access to real-time weather information. Ah, I forgot that ChatGPT can search the internet, but Claude can’t do it yet!

It makes sense that Claude does not have an answer to my question because it takes months to train an LLM model. I could ask another LLM to answer my question or simply check my phone. But I like Claude! Can I do something to help it to answer this kind of question?

Can I do a quick Google search for Claude when it needs some real-time data?

It’s a weird idea, but let’s try it! I’ll tell Claude that it should let me know when it needs me to search the internet. Claude can be a bit chatty, so I’ll make sure to tell it to provide an exact search phrase I should use. For example, telling me Google:weather in Belgrade today would be ideal.

It seems that my friend Claude likes this game. Let’s ask again, “What’s the weather like in Belgrade today?”

It worked! Claude provided an exact search query so I can do a Google search and provide a screenshot. The reply was more detailed than I needed it to be, but it did not matter; I understood my assignment.

I copied the search phrase, opened my browser, and googled it. Then, I took a screenshot of the result and sent it to Claude. And Claude replied with useful information about the current weather in Belgrade!

Claude definitely liked this game.

But while I did this just for fun, I accidentally did one more thing – I just created an AI agent!

I know it’s not a very useful agent, as I could just read the weather data from Google. But it’s still an agent.

I also know ChatGPT can search the internet, so I could use it instead of Claude. But ChatGPT is also an agent! It’s just an undercover agent that looks like a plain old LLM. To be fair, Claude is also an agent. Just ask it to draw a diagram or create a webpage for you, and you’ll see some superpowers that LLMs do not have.

How AI Agents work

LLMs are amazing! They really are. But like many other tools, they are good at some things but not so good at others.

For example, LLMs are excellent at picking the best set of tokens to continue the set of tokens we provided. Or, in a human-understandable language, they are very good at answering questions, completing sentences, writing text, etc.

You ask a question. LLM replies. Sometimes, it’s helpful information; sometimes, you must ask a follow-up question. But it always replies.

However, not all of these replies are based on truth. Sometimes, an LLM replies with false information that we call hallucinations. That’s because it tries to find the closest set of tokens to your tokenized instructions (or your question) and always finds something.

LLMs do not really care about the truth. They care about the closest tokens to your tokenized instructions, their training sets, their configuration, and some additional parameters.

But what makes an LLM an agent?

Agents are LLMs with something that provides missing information or capabilities to help LLMs answer our questions. If we call these things “tools,” agents are LLMs with tools.

However, to be an agent, LLM must orchestrate these tools and decide when it has enough information to answer our questions or complete our tasks. If we orchestrate the tools with predefined code, LLM is just a tool in our code, and our code is not an AI agent.

The diagram above looks familiar. It looks like a while loop!

So, I guess we can say that an AI agent is like a “while loop” that keeps asking available tools to provide additional information or capability until it has all it needs to complete the task or answer the question.

Anything can be a tool that provides missing capabilities or information to LLMs as long as LLMs have an easy way to use that tool.

For example, I was a tool that my friend Claude used to find information about the current weather in Belgrade! But that made our “while loop” expensive because it used both LLM tokens and my time.

These “while loops” are generally expensive. They are not expensive because of the big O notation and code complexity but because in each iteration, LLM evaluates whether it can complete the task and uses tokens (and our money).

Being expensive depends on the value it provides, but it’s always a good idea to be careful. You can be careful by setting the billing alarms and spending limits, making sure that the LLM does not iterate indefinitely (by limiting the number of iterations), picking the right model for your task (sometimes cheaper models can also complete your tasks), and configuring monitoring, error tracking, and alarms.

While loops and where to write them

So, if agents are while loops with LLMs and some additional tools, where do we write these while loops to create an agent?

The answer is almost anywhere. While creating an AI agent using pen and paper could be a thing, it’s not really a practical way of making an agent. Another cost-inefficient and unhelpful way of creating a while loop is using a person to act as one. However, you can write this “while loop” wherever you need it. For example, it can be inside the app you are working on, in your terminal, on a server (using any backend language you prefer), in a browser, etc. As long as you are careful and you do not leak your LLM secret key and other similar secrets.

To write an agent “while loop,” you need to do the following:

1. Choose an LLM model that fits your needs and your budget (which can be $0 or whatever other number).
2. Define a system prompt with a clear explanation of all the tools you want to provide (including when and how to use them).
3. Ask an LLM to reply in a strict JSON format or any other structure you prefer.
4. Make sure you parse and validate the reply correctly.
5. Handle errors and set the maximal number of iterations, billing budget, and alerts.

Remember, LLMs are good at talking to humans (actually imitating human interaction), but human language is not easy to parse in the code. If you worked with LLMs and tried to get a JSON reply and nothing else, I am sure you, at least once, got the reply similar to: “Here’s your JSON: {...}.” Yelling at LLM and telling it to reply with JSON works sometimes, but in some cases, even 3 exclamation points at the end do not help. Even a simple agent we built inside Claude.ai replied with a sentence in front of the search phrase:

You can either pick a format that is easier to parse or use a simple trick explained in my previous article: provide the beginning of the reply and let an LLM continue. You can see the code example here: 5 Prompt Engineering Tips for Developers.

But, while understanding how these LLM “while loops” work is good, you do not need to write your own while loops. There are many existing tools and frameworks you can use.

AI Agent tools and frameworks

AI Agent tools and frameworks are like JavaScript frameworks – we get many new ones every day. Pick any word that comes to your mind. The chance to find a JavaScript package with that name in NPM and a .ai domain with that name is higher than the latest US-to-China tariff percentage.

For example, LangChain was the AI Agent framework a while back. Today, we have LlamaIndex and many other popular tools besides it. Big players like Microsoft have their own open-source takes, such as AutoGen. And, of course, services such as Amazon Bedrock Agents. There are many other examples, from tools for non-coders and open-source tools to enterprise-grade tools.

It’s hard to pick the best one. If you want to check just one that works with JavaScript or TypeScript, you can start with LlamaIndex.

LlamaIndex sounds similar to the Meta Llama models. But it’s not the same. Actually, LlamaIndex supports the Meta Llama model and many others (including OpenAI models, Anthropic models, open source models, Amazon Bedrock, Azure OpenAI, etc.).

Another interesting thing about LlamaIndex is that they focused on the AI Agent memory issue as an important problem to focus on. If you have worked with AI Agents, you know what I am talking about. If not, read on.

Remember, remember… the conversation we had yesterday

As I already mentioned, LLMs are limited by their training set, configuration, your instructions, and a few other things. One of their most important limitations is their context size.

The context size represents the maximum number of tokens an LLM can have in a single conversation (through the API, UI, or any other way you interact with it). It’s a hard stop. Once you fill the context with tokens, an LLM will explode. Well, not literally. But it’ll stop working. If you have used LLMs from the early days, you might remember that after a certain number of messages, LLM seems to forget what you were talking about. That’s because the context was filled, and an LLM removed the initial messages to make space for your new messages. Luckily, LLM then made system prompts, sticky parts of the conversation that always stay in context and allow you to provide the instructions.

If you manage to fill the context, LLMs will most likely do one of the following:

1. Remove some messages from the beginning of your conversation, but keep the system prompt so it still follows the instructions). This can cause an LLM to forget some parts of your conversation.
2. Summarize some parts of the conversation and replace N messages with the summary (well, LLMs are good with summarization). The quality of the remaining conversation depends on the way an LLM summarizes the conversation.
3. Block you from sending more messages (most likely if you are using an API).

Luckily, the context size is increasing fast (Claude has a 200k token context, Gemini 1M context, and Llama now has a context size with up to 10M tokens). However, a larger context size can decrease the ability of an LLM to find specific items in it. Also, we want to fit larger items in the context. We started with simple spreadsheets and PDFs, and now we want to embed whole knowledge bases, books, project documentation, etc.

Again, luckily, many smart people work with LLMs, so they quickly came up with an effective way to make the most of the (at that time very) limited LLM context size. However, naming things is hard (ask OpenAI and Anthropic or simply read the names of their models), so they called this approach Retrieval-Augmented Generation (RAG).

While RAG sounds complicated and is still one of the most misunderstood terms related to LLM, it is quite a simple but powerful concept.

In short, instead of putting all documents in the system prompt, you can wait for the user question, then tokenize it before replying and do a vector search against your knowledge base to find a few closest matches. Then, you take these pieces and tell an LLM to respond to the user’s question in the context of the provided pieces of your knowledge base.

Before you do a vector search, you need to split your knowledge base into reasonable chunks (i.e., articles, sections of the articles, or even paragraphs in some cases) and create vectors from each piece.

And when I say vector search, I mean something similar to the vector search that LLMs use under the hood. Remember the following image?

You can use a vector DB to do a vector search, but that’s not required, as you can do vector searches in some of the popular databases (such as PostgreSQL, ElasticSearch, etc.) or store vectors almost anywhere and create your own vector search function (Amazon S3, for example).

Writing your own vector search (or actually vector similarity) function also sounds complicated, but luckily, you can ask an LLM to write that function for you, and it can look similar to the following one:

// Calculate cosine similarity between two vectors
function cosineSimilarity(vector1, vector2) {
  // Calculate the dot product of the two vectors
  const dotProduct = vector1.reduce((sum, a, i) => sum + a * vector2[i], 0)
  
  // Calculate the magnitude of both vectors
  const magnitude =
        Math.sqrt(vector1.reduce((sum, val) => sum + val * val, 0))
        * Math.sqrt(vector2.reduce((sum, val) => sum + val * val, 0))
  
  // Return the cosine similarity
  return dotProduct / magnitude
}

This function returns a number that represents the similarity percentage. You can run the same function against each of the knowledge base article vectors and pick the two top ones with more than 80% similarity or something similar.

A simple function like the one I showed above would work fine for smaller databases. However, you should use a more efficient search for large data sets.

AI Agents use many tokens, and often need vast knowledge bases. LlamaIndex helps with more efficient vector search and allows you to create agents that are not like they came from the Memento movie.

However, explaining RAG and LLM memory in detail requires more than a few paragraphs. All these explanations would convert this article into a short book. So, let’s leave that for another article and get back to AI Agents.

Let’s build an AI Agent!

LLMs are products. Products evolve and add features based on users’ feedback and usage patterns. Well, using tools is one of the important features that most LLMs now support natively. Some LLMs call this feature tools (for example, tools in Claude), some call it functions (for example, OpenAI functions), but it’s the same thing that allows us to build AI Agents.

Built-in tools have a few clear benefits, such as replies in a strict JSON format, a well-defined format, and an easier ability to stream responses via HTTP. They also have less surface area for errors because they are now built into the LLM itself. But, as always, there are many different standards, and if you want to switch to some other LLM, you’ll probably need to define tools in a slightly different format. However, a simple abstraction (or, even better, a hexagonal architecture) makes this problem easier to manage.

Let’s build a simple agent using built-in tools! You can pick any LLM you like. I’ll use Claude Sonnet 3.7 on Amazon Bedrock. The example below would work fine with other models. I use Amazon Web Services (AWS) every day, so Bedrock is a natural choice (despite its limits, especially in European data centers).

So, where do we start? With our “while loop,” of course! Remember this?

I want to build a simple agent for my product, Vacation Tracker. It will be very simple because otherwise, I would need to write a book to show all the details. I want my AI Agent to be able to do the following 3 things only:

1. Help users to request leave, such as PTO.
2. Let users see which coworkers are not working today and who will be off this or next week.
3. Answer some basic questions about our product using our knowledge base.

With these 3 features, my “while loop” would look like the following diagram.

There are so many different ways to build this AI Agent using AWS. For example, we could create a simple serverless solution like the diagram below, with the following components:

• I would use an Amazon API Gateway to expose the API endpoint for my AI Agent.
• My AI Agent “while loop,” or business logic, would be in a Lambda function that defines the specification of the tool, invokes the LLM, and talks to the Vacation Tracker API and storage for our knowledge base and vectors.
• I would use the Claude Sonnet 3.7 model on Amazon Bedrock.
• I could store the vectors and parts of our knowledge base in the S3 bucket. This is not an ideal long-term solution, but it would work fine for the MVP version.

By using API Gateway, we get all the benefits this service offers, including easy setup for rate limits, Web Application Firewall (WAF), etc. However, as Austen Collins suggested, API Gateway has some significant downsides when building AI Agents. For example, the API Gateway timeout is limited to 29 seconds (AWS allows us to change the timeout now, but changing the timeout can affect account-level throttle limits, etc.), which could be a serious limitation for more complex production-ready agents that can do some longer tasks. Also, we can’t stream the response from our Lambda function, so we need to wait for our agent to generate the whole long reply before we can start showing it to the user. Streaming would allow us to show the response as LLM generates it. This is especially helpful for long responses, as our agent starts responding to the user faster and keeps adding text as the user reads (the effect is similar to typing).

Luckily, there’s an alternative! AWS Lambda supports Lambda function URLs. It’s basically a simple HTTP endpoint in front of your Lambda function. The main benefits of the function URL over API Gateway are that it offers timeouts of up to 15 minutes (it’s a Lambda timeout, not an API Gateway timeout anymore) and support for streaming responses. Just what we needed!

However, it comes with many downsides, too. You do not get all the features of the API Gateway, such as built-in rate limits, authorization support, etc. It does not support WAF, either. There are no custom domains for Lambda function URLs. However, you can mitigate some of these downsides by putting a CloudFront in front of the Lambda function, as shown in the diagram below.

Is this an ideal setup? It depends on your use case. It’s a good start. There are many other alternatives. For example, we could keep the initial setup, and instead of waiting for the reply with an open HTTP connection, we could send a message to a background job and tell the frontend application that the message is received and that we’ll send a reply via WebSockets. There’s no out-of-the-box streaming for this setup, too, but it can give you more flexibility and some benefits from both approaches.

In production, we would need to think about our use case, WAF, rate limiting (for our app, for LLM we are using, and for other services we are using), securing the API endpoint (auth token, API key, etc.), monitoring, error logging and handling, storage for the conversation (we do not want to send the whole conversation from the frontend when a user sends a new message), and many other things.

But let’s keep things simple.

Show me some code!

The initial idea of this article was to show the code. But here we are, 4000 words later, and I barely explained how agents work. I’ll show the most important parts of the code here. I might do a “part 2” article with a code deep dive if anyone reads the article past this point and thinks a complete coding example might be beneficial.

The most important part of the code is a definition of the available tools. The format depends on a model or a service you use, but for Claude Sonnet 3.7 on AWS Bedrock using the Converse API, you can do something similar to the following:

const toolConfig = {
  tools: [
    // We need a similar object for each of our tools
    {
      // The name and the description of our tool
      // A clear description is important because it helps an LLM to select the right tool
      name: 'request_leave',
      description: 'Request leave (such as PTO, sick day, etc.). The leave request will be submitted to the Vacation Tracker application, and sent to your manager (approver).',
      inputSchema: {
        // It accepts JSON, I told you developers love JSON!
        json: {
          type: 'object',
          properties: {
            // And it expects the following properties (see the type and the description for each).
            // For simplicity, I'll show the most important parts only
            startDate: {
              type: 'string',
              description: 'The date when the leave starts. Format: YYYY-MM-DD',
            },
            endDate: {
              type: 'string',
              description: 'The date when the leave ends. Format: YYYY-MM-DD',
            },
            leaveType: {
              type: 'string',
              description: 'The type of leave. For example, "vacation", "sick", etc.',
            },
            reason: {
              type: 'string',
              description: 'The reason for the leave request. For example, "Vacation in Greece", "I am not feeling well today," etc.',
            },
          },
          required: ['startDate', 'endDate', 'leaveType'],
        },
      },
    },
    // Define the other 2 tools here
  ],
}

It’s important to provide clear descriptions as that would help an LLM pick the right tool when needed.

I am a big fan of hexagonal architecture (or ports and adapters), and I would use it in a production code. Our business logic does not care about the Claude Sonnet 3.7 model. It does not care about Amazon Bedrock, either. So, I would put the LLM logic in some kind of repository, initialize it, and use it to send a message when needed. That would make my code cleaner to read, easier to test, and allow me to try other models (i.e., Open AI models, which are not available in Amazon Bedrock) without changing the business logic.

However, I’ll just show the simplest code example without hexagonal architecture to keep things simple.

// Stripped down example of using the AWS Bedrock SDK to create a simple AI Agent
import { BedrockRuntimeClient, ConverseCommand } from '@aws-sdk/client-bedrock-runtime'
import { toolConfig } from './tool-config.js'

// Initialize the Bedrock client and specify the region and a model ID
const client = new BedrockRuntimeClient({ region: 'us-east-1' })

const modelId = 'anthropic.claude-3-7-sonnet-20250219-v1:0'

// Start a conversation with the user message.
const conversation = [
  {
    role: 'user',
    content: [{
      // The user messsage is hardcoded for this example, but in a real application, you would get it from the user input
      // Also, in practice, users would probably tell us that they want to go on vacation in the first week of July, and the agent would need to ask for the start date
      text: 'I want to go on vacation on the first week of July. Full week, starting June 30th.',
    }],
  },
]

// Create a command with the model ID, the message, and a basic configuration
const command = new ConverseCommand({
  modelId,
  messages: conversation,
  system: [{
    // We would need to provide a more detailed system message in a real application
    text: 'You are Vacation Tracker assistant and you help users to request leaves, see who else from their team is off, check their leave balance and learn about the Vacation Tracker app functionalities. When a user asks to go off for a longer period, assume whole week. Here is the list of available leave types: `PTO`, `SickDay`. Try assuming the correct leave type from the input.',
  }],
  // We pass our tool configuration to Bedrock
  toolConfig: toolConfig,
})


async function run() {
  try {
    // Send the command to the model and wait for the response
    const response = await client.send(command)

    console.log('Response:', JSON.stringify(response))
  } catch (err) {
    console.log('ERROR', err)
  }
}

// We wrap the call in the run function just to be able to use it in a terminal without the deployment
// The actual code would define a Lambda function handler
run().then(console.log).catch(console.error)

To run this code example, you’d need an AWS account with Claude Sonnet 3.7 enabled in the Amazon Bedrock (in the us-east-1 region). When you run it, the response should be similar to the following JSON (inline comments make this JSON invalid, but I added them for easier understanding):

{
  // Just metadata, you can ignore this part at the moment
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "bc806712-b1b2-40eb-8488-f0085237ebcf",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  // Metrics can be useful, but let's ignore that too at the moment
  "metrics": {
    "latencyMs": 4191
  },
  // Claude Sonnet 3.7 response
  "output": {
    "message": {
      "content": [
        {
          // A message that we can show to our users if we want to
          "text": "Certainly! I'd be happy to help you request leave for your vacation during the first week of July. Based on the information you've provided, I'll submit a leave request for you using the Vacation Tracker application. Let me go ahead and process that for you."
        },
        {
          // Claude Sonnet 3.7 tells us to use a tool!
          "toolUse": {
            // Request params, as defined in the tool configuration
            "input": {
              "startDate": "2023-06-30" ,
              "endDate": "2023-07-07",
              "leaveType": "PTO",
              "reason": "Vacation for the first week of July"
            },
            // The name and ID of a tool we should use
            "name": "request_leave",
            "toolUseId": "tooluse_JaORLwrHSSGifTRMApUrGA"
          }
        }
      ],
      // This is a response from an agent, we need to pass this with the response
      // from a tool to continue the conversation
      "role": "assistant"
    }
  },
  // Claude tells us that it stopped because it needs a tool to continue
  "stopReason": "tool_use" ,
  // Useful metrics for the number of used tokens
  "usage": {
    "inputTokens": 630,
    "outputTokens" : 183,
    "totalTokens": 813
  }
}

As you can see in the JSON above, Claude tells us that it needs a tool to be able to reply (stopReason: 'tool_use'). It gives us a nice message that we can display to our users if we want to, but keep in mind that the agent still works at this point, so users should not send new messages yet. Finally, it gives us the details about the tool we need to use and the parameters we should send to our tool to get the response.

After receiving this response, we should send the API request to the Vacation Tracker API with the provided start and end dates, leave type, and reason. But before that, we probably need to validate the data and get the authentication token for the API.

Before we continue, let’s talk about one more thing: request duration. Remember the 29-second timeout in the API Gateway? Well, this request took 4.2 seconds out of these 29. If we add a few hundred milliseconds for the Lambda overhead and a request to a database to get the previous messages in this conversation, we are probably around 5 seconds.

But that’s just the first part of this request. Before we reply, we need to call the Vacation Tracker API, parse the response, and call the Amazon Bedrock again. If we are lucky, our agent will need just one tool to be able to reply, so we’ll be at 10 or 15 seconds, including saving the conversation to the DynamoDB.

What happens if we have a complex agent that could use multiple tools in one request? We can easily get close or over the 29-second timeout, which could break our agent.

Ok, so, we got the ID of the tool we need to use and the parameters to send to the tool. In this case, an agent wants us to request a leave. In production, we would ask the user to confirm the request details first, but for this simple example, we can send the request straight to the Vacation Tracker API.

Once the Vacation Tracker API responds, we probably want to process the response to make it clear to our LLM because most API responses are not 100% LLM or human friendly. For example, we can add a description, change the field names to be more descriptive, etc.

We could send a new request to Amazon Bedrock with the following conversation:

[
  // The initial user message
  {
    "role": "user",
    "content": [
      {
        "text": "I want to go on vacation on the first week of July. Full week, starting June 30th."
      }
    ]
  },
  // The unchanged assistant message from the previous example
  {
    "role": "assistant",
    "content": [
      {
        "text": "Certainly! I'd be happy to help you request leave for your vacation during the first week of July. Based on the information you've provided, I'll submit a leave request for you using the Vacation Tracker application. Let me go ahead and process that for you."
      },
      {
        "toolUse": {
          "input": {
            "startDate": "2025-06-30",
            "endDate": "2025-07-06",
            "leaveType": "PTO",
            "reason": "Vacation for the first week of July"
          },
          "name": "request_leave",
          "toolUseId": "tooluse_JaORLwrHSSGifTRMApUrGA"
        }
      }  
    ]
  },
  // Tool response
  {
    // Role needs to be "user"
    "role": "user",
    "content": [
      {
        // We provide the tool result object
        "toolResult": {
          // Tool ID
          "toolUseId": "tooluse_JaORLwrHSSGifTRMApUrGA",
          "content": [
            {
              // Response is a JSON object with a custom payload
              "json": {
                // For example, the API tells us that the request was successfully sent
                "success": true,
                // We have the request details
                "request": {
                  "id": "request_12345",
                  "status": "pending",
                  "startDate": "2025-06-30",
                  "endDate": "2025-07-06",
                  "leaveType": "PTO",
                  "reason": "Vacation for the first week of July"
                },
                // And also the remaining quota
                "quota": {
                  "remaining": 10,
                  "used": 5,
                  "total": 15,
                  "unit": "days"
                }
              }
            }
          ]
        }
      }
    ]
  }
]

When we send this request to Claude, the reply will probably be similar to the following JSON:

{
  // Metadata, ignore
  "$metadata": {
    "httpStatusCode": 200,
    "requestId" : "de4dad99-1d57-46bd-ad71-c69ebe853756",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  // It took almost 9 seconds to reply!
  "metrics": {
    "latencyMs": 8985
  },
  // The actual response object
  "output": {
    "message": {
      "content": [
        {
          // The text we want to show to the user
          "text": "Great news! I've successfully submitted your leave request for your vacation. Here are the details of your request:\n\n- Leave Type: PTO (Paid Time Off)\n- Start Date: June 30, 2025 (Monday)\n- End Date: July 6, 2025 (Sunday)\n- Reason: Vacation for the first week of July\n- Status: Pending approval\n \nYour request has been submitted to the Vacation Tracker application and sent to your manager for approval. The request ID is request_12345.\n\nAdditionally, I can see from the response that your current leave balance is as follows:\n- Total PTO: 15 days\n- Used: 5 days\n- Remaining: 10 days\n\nThis request will use 6 working days of your PTO (assuming a standard Monday to Friday work week). Please note that your request is currently pending approval from your manager. You should receive a notification once it's been reviewed. \n\nIs there anything else you'd like to know about your leave request or the Vacation Tracker application?"
        }
      ],
      "role": "assistant",
    }
  },
  // Claude says that the request was successfully fulfilled
  "stopReason": "end_turn",
  // Usage data
  "usage": {
    "inputTokens": 901,
    "outputTokens": 236,
    "totalTokens": 1137
  }
}

To make it a bit more readable, here’s the AI Agent response:

Great news! I’ve successfully submitted your leave request for your vacation. Here are the details of your request:

• Leave Type: PTO (Paid Time Off)
• Start Date: June 30, 2025 (Monday)
• End Date: July 7, 2025 (Friday)
• Reason: Vacation for the first week of July
• Status: Pending approval

Your request has been submitted to the Vacation Tracker application and sent to your manager for approval. The request ID is request_12345.

Additionally, I can see from the response that your current leave balance is as follows:

• Total PTO: 15 days
• Used: 5 days
• Remaining: 10 days

This request will use 5 working days of your PTO (assuming a standard Monday to Friday work week). Please note that your request is currently pending approval from your manager. You should receive a notification once it’s been reviewed.

Is there anything else you’d like to know about your leave request or the Vacation Tracker application?

As you can see, our AI Agent is a bit more chatty and friendly. It made up some parts of the reply (for example, we never mentioned the notification), but the response generally looks good.

In production, we probably want an LLM to reply in a structured way so we can display a predefined UI for the leave request along with a short, friendly message. Also, we need to be careful with dates and other numbers, as LLMs do not care about numbers. Numbers are close in the vector space. For an LLM, 4 is similar to 5 or even 42, but 4 or 5 used PTO days make a big difference for our users.

And that’s it! We built an AI Agent again. This time with code. It’s a simple agent, more like a proof of concept. But it’s still an agent.

Summary

After a long article, I’ll keep the summary short.

Before we do a quick summary, you should check 2 relatively fresh things related to AI Agents.

First, my Twitter feed has been buzzing about MCP servers in the past few weeks. Model Context Protocol (MCP) is an open-source protocol that standardizes how applications provide context (and tool specification) to LLMs, and you can read more about it here: https://docs.anthropic.com/en/docs/agents-and-tools/mcp.

Second, OpenAI has some new tools for building agents. Read more about these tools here: https://openai.com/index/new-tools-for-building-agents/.

To end this long article, I just want to remind you that AI Agents sound very complicated, but they are actually like LLM “while loops” with tools.

You have all the skills you need to build these tools. So, go, build tools and agents, and have fun!

However, it’s hard to give clear instructions. Not just to you, your manager, or your friends. It’s hard to give clear instructions even if you are a person whose main job is to provide clear instructions.

A sentence can destroy Fleets

It was July, 1914. The beginning of WW1 was just a few days away. SMS Goeben, an Imperial German Navy battlecruiser built for speed and power, dominated the Mediterranean Sea, accompanied by a light cruiser, SMS Breslau. Goeben was significantly larger and better armored than the British Navy battleships, but it was experiencing issues with its boilers, which limited its speed and operational efficiency.

SMS Goeben encountered the British battlecruisers HMS Indefatigable and Indomitable and four additional vessels.

Winston Churchill, the political head of the British Royal Navy at that moment, ordered the British Navy to “shadow, not to engage” with SMS Goeben until the war was officially declared. And they did it.

The war was declared just a few days later. However, Churchill’s order contained the following sentence:

“… do not at this stage be brought to action against superior forces.”

For Churchill, the superior forces were the Austrian Fleet. However, this part of unclear instructions allowed SMS Goeben and SMS Breslau to escape and reach Constantinople, where they eventually became part of the Ottoman Navy, playing a key role in drawing the Ottoman Empire into the war and leading naval operations in the Black Sea against Russia.

Giving clear instructions to computers

Human interactions are complicated. While I hope your unclear instructions never had an impact as significant as the one described above, you probably have many stories.

Luckily, talking to computers is easier, as they have more predictable behaviors and no emotions. Programming languages make communication with computers predictable and repeatable, which makes our job easier. However, Large Language Models (LLMs) are still different beasts. Currently, there’s a lot of magic and mysticism involved in LLM prompting.

But all the magic aside, we know enough about LLMs to get helpful information from them.

Large Language Models are good at guessing the next best set of words based on your input (instructions, questions, or part of the text), their training data (text from many books, websites, and other datasets), context (documents or previous conversation flow), and specific configurations (such as weights, which prioritize certain word patterns, and settings like temperature, which control the randomness of predictions).

Let’s say you asked an LLM to finish the following sentence: “I am speaking at”

Will it continue with “a Martian picnic?” Or “a space farmer’s market?” Highly unlikely. You’ll get a reply similar to the following: “a business conference,” “a tech meetup,” or “a community forum.”

But what happens when you change the context? Answers can now be the complete opposite!

Let’s say you add the following at the beginning of your prompt: “You are a playful, chatty cartoon character called “Space Bunny” selling sandwiches on Mars near the intergalactic highway.” It’s more likely that an LLM continues with “a Martian picnic” or “a space farmer’s market” now.

I already said “prompt” a few times. But prompts are not magic. They are just instructions. You tell an LLM what you want, and it tries to reply based on its training, your instructions are a few additional parameters and configurations. Simple!

It works similarly to the instructions you give to humans. The better you explain what you want, the higher chance you’ll get a useful reply. But the main difference is (or is it?) that an LLM always replies. Not all of its replies are based on truth.

So, prompts are just instructions. But how do LLMs work?

You give your instructions. An LLM eats a lot of tokens, and it spills some unexpected wisdom. Or hallucination.

Wait, what are tokens?

Money. Tokens are money. I am not kidding. Most of the time, you pay for using an LLM based on the number of tokens it eats and spills out. So, for you, tokens = money.

So, less tokens = less money out of your pocket = better? Well, not really. There’s another side to that story.

For you, tokens are money, but for an LLM, tokens are the only way to understand and follow your instructions. LLMs are computers, and you might think that computers are good with text because you interact with them using text and clicks, but they are not. They like numbers more (don’t tell anyone 🤫). When you enter a text, a computer converts that to numbers, does some operations, converts numbers back to text, and displays that text on the screen. It’s an oversimplification, of course.

LLMs do the same. What a surprise, right?

First, they convert your instructions to tokens. Tokens are words or parts of the words that help LLMs to match patterns. There’s no universal way for LLMs to convert words and sentences to tokens. There are many tokenizers. However, one generalization is still close to the truth: around 750 words roughly equal 1000 tokens.

There are also many visual explanations for tokenized prompts. For example, you can try OpenAI’s tokenizer here: https://platform.openai.com/tokenizer. You’ll get slightly different results based on the model you select.

So, an LLM first splits your instructions into chunks we call tokens. And then (this is oversimplification, again) it converts each of these tokens to a series of numbers (array of integers). These arrays of integers are often called vectors. You can also think about them as dots in a multidimensional space. But best of all, these dots are not randomly put in the space; they are close to other dots with a similar meaning. For example, a vector representation of the word “potato” is close to a “tomato” but not close to a “business conference.” And that’s one of the most important sources of the LLMs’ magic.

Let’s imagine that each token converts to a simple array of two numbers (two-dimensional space) to simplify the visual representation.

Anatomy of a prompt

Remember when I said that prompts are just instructions? I lied. Instructions are the most useful subset of prompts, but in general, any text (or just a single letter or character) can be a prompt. If you type a single character, such as “$,” that’s a valid prompt. Just remember that the quality of a reply depends on the quality of your prompt.

However, prompts are not just written words. With multimodal LLMs (a fancy name for LLMs that understand different types of input), you can use your voice or upload a document, an image, or even a song. You can go further and combine these inputs. These are all valid prompts. I often upload a spreadsheet or some other document and provide a few sentences of textual instructions.

So, asking, “Who is faster: Godzilla or T-Rex?” is a valid prompt. Let me save you some tokens: the answer is T-Rex.

Asking an LLM to “Write a 500-word article about the bad influence of Amazon’s RTO policy on Lambda cold starts” is also a valid prompt. It took ChatGPT 13 seconds to figure this one out. I was afraid to ask Claude. It’s hosted on AWS, and I don’t want it to be part of the next round of layoffs.

So, almost anything could be a prompt. However, not all prompts are useful. Luckily, LLMs are products, and like with many other products, LLM users (we) did many crazy things (“lizards,” as my friend would call us) and came up with some helpful tricks. Products evolve with user needs and feedback. LLMs are not different.

One of the early tricks was telling an LLM to pretend to be an expert in the field related to the prompt. This worked incredibly well until you hit the context limit (the number of tokens or characters an LLM can keep in “memory”). Once you reach the context limit, the conversation often takes a weird direction. Imagine talking to Dory Fish on LSD.

People behind LLMs figured out that some parts of our instructions are more important than others and that we often repeat that part across multiple LLM conversations. And system prompts were born. A system prompt is a repeatable part of the prompt with higher priority than the rest of your conversation, which always remains in the context (oversimplification, again).

Repeatability and more control are often essential when you work with LLMs. The system prompts help, but we (software developers, at least) already know how to talk to computers to get repeatable and controlled responses. We can get amazing results by combining code with LLM interaction through API.

How do we get these “amazing results?” Let’s see the 5 habits of highly effective prompters! Oops, I forgot this is not Linkedin. I mean 5 simple tricks for better prompting results.

Five simple tricks for better prompting results

This article is already too long, so there’s no need for a cliffhanger. Here are the tricks I want to tell you about:

1. Hint the beginning of the answer.
2. Give examples.
3. Think step by step.
4. Use tools.
5. Ask an LLM to improve the prompt.

Let’s go through them (step by step).

Hint the beginning of the answer

I told you to combine code (JavaScript, Python, or anything else; I don’t judge) and LLM for better responses. But to do so, you need to get a structured response from an LLM.

While there are countless ways to structure a response, we really love JSON. So, we ask an LLM politely to answer with JSON and nothing else. But an LLM is also polite, so the conversation from time to time looks like this:

User: …some super long and smart instructions… Always answer with valid JSON and nothing else.

Assistant: Here’s your JSON:

```json

{

“some”: “valid”,

“JSON”: true,

}

```

Then your JSON parser starts yelling at you, and you do the same to the LLM. Replying, “I said JSON only!!!” in capslock works sometimes.

But here’s one more trick you can try: write the beginning of the assistant’s reply in the API call!

LLM APIs expect a conversation (user -> agent -> user -> agent, and so on). If you finish your API call with the user’s message, LLM will reply with the assistant’s message. However, if you finish your API call with the assistant’s message, LLM will treat it as the beginning of its reply and simply continue.

Below is the code example with AWS Bedrock using AWS Node.js SDK and Bedrock Converse API. Other LLMs support something similar.

import { BedrockRuntimeClient, ConverseCommand } from '@aws-sdk/client-bedrock-runtime'

// Create a Bedrock Runtime client in the AWS Region you want to use.
const client = new BedrockRuntimeClient({ region: "eu-central-1" })

// Set the model ID, i.e., Claude 3.5 Sonnet
const modelId = 'anthropic.claude-3-5-sonnet-20240620-v1:0'

// Start a conversation with the user message.
const userPrompt = 'Return the result of all games of the NBA 2023 finals as a short structured JSON.'
const conversation = [
  {
    role: 'user',
    content: [{ text: userPrompt }],
  },
  {
    role: 'assistant',
    content: [{ text: '{"' }], // <---- Assistant's reply starts with `{"`
  },
]

// Create a command with the model ID, the message, and a basic configuration.
const command = new ConverseCommand({
  modelId,
  messages: conversation,
})

try {
  // Send the command to the model and wait for the response
  const response = await client.send(command)

  // Extract, parse and print the response text, but prepand `{ "`
  const responseText = '{ "' + response.output.message.content[0].text
  const parsedJson = JSON.parse(responseText)
  console.log(parsedJson)
} catch (err) {
  console.log('ERROR', err)
}

Give examples

We can use a similar approach for one more trick.

You probably use ChatGPT or Claude at least a few times per week. Do you always get a perfect reply after your initial prompt? I don’t. I often ask follow-up questions, point out mistakes, or give a few examples to get the answer I am looking for. I know many other people do the same.

However, we often send just one detailed command via the API and expect LLMs to deliver a perfect response. If someone tries to look smart, they’ll probably refer to this as “zero-shot prompting.” It’s just a name for sending direct commands without examples.

Why does it need the name? It doesn’t. But as you can expect, there’s a fancy name for the opposite, too. If you provide examples and simulate conversation, you are using “few-shot prompting.”

For example, you can ask an LLM to label a review comment with a “positive” or “negative” label and return it in JSON format. You can do that by sending a prompt similar to the following:

Here’s a review comment. Return a JSON with the label “positive” or “negative” for this comment:

“Super helpful, worth it!”

This example is simple, so the “zero-shot prompting” works perfectly fine. But let’s say you want to label neutral comments as “negative.” LLM will not know that. You can improve your prompt or even add a detailed system prompt, which might work well. But you can also do the following:

Label review comments and return a JSON with the label “positive” or “negative.” Here are a few examples:

• “Great product 10/10” => { "label": "positive" }
• “Didn’t work very well” => { "label": "negative" }
• “Not sure” => { "label": "negative" }

Label the following comment based on the examples above:

“Super helpful, worth it!”

This technique will improve your chance to get more precise labeling. It’s like a mini in-prompt training.

But there’s a cleaner way to do this using the same approach we used in the first trick: simulate the conversation by passing multiple user and agent messages.

Let’s see the code example!

// Same as above
import { BedrockRuntimeClient, ConverseCommand } from '@aws-sdk/client-bedrock-runtime'

const client = new BedrockRuntimeClient({ region: "eu-central-1" })

const modelId = 'anthropic.claude-3-5-sonnet-20240620-v1:0'

// Start a conversation with the user message.
const systemPrompt = [
  {
    text: 'Label review comments and return a JSON with the label "positive" or "negative."',
  }
]
const conversation = [
  {
    role: 'user',
    content: [{ text: "Great product 10/10" }], // Rating example
  },
  {
    role: 'assistant',
    content: [{ text: '{"label":"positive"}' }], // Label example
  },
  {
    role: 'user',
    content: [{ text: "Didn't work very well" }], // Rating example #2
  },
  {
    role: 'assistant',
    content: [{ text: '{"label":"negative"}' }], // Label example #2
  },
  {
    role: 'user',
    content: [{ text: "Not sure" }], // Rating example #3
  },
  {
    role: 'assistant',
    content: [{ text: '{"label":"negative"}' }], // Label example #3
  },
  {
    role: 'user',
    content: [{ text: "Super helpful, worth it!" }], // <---- ACTUAL REVIEW
  },
  // You can even hint the begin of the reply here
]

// Create a command with the model ID, the message, and a basic configuration.
const command = new ConverseCommand({
  modelId,
  system: systemPrompt,
  messages: conversation,
})

try {
  // Send the command to the model and wait for the response
  const response = await client.send(command)

  // Extract, parse and print the response text
  const responseText = response.output.message.content[0].text
  const parsedJson = JSON.parse(responseText)
  console.log(parsedJson)
} catch (err) {
  console.log('ERROR', err)
}

Easy, right? Just remember that this approach eats more tokens. And tokens = money.

Think step by step

The previous trick works well with humans, too. If you give examples when you delegate a task, there’s a higher chance you’ll get the task done.

But imagine you could use the following trick in conversation with humans, too! Let me give you an example.

You might have a boss or a client. One of the superpowers of bosses and clients is that they can come up with completely unreasonable ideas they want you to do now. Trust me, I was on both sides of that process.

Imagine you could ask them, “Let’s think step by step. How would that feature help our users?”

In practice, this scenario rarely works because people have complex emotions. Instead of them thinking through the process step by step and explaining every detail, you could end up with a new green LinkedIn badge around your profile photo.

Luckily, LLMs have no emotions. Yet. So, adding “Let’s think step by step” or a similar phrase uses more tokens for the response, but it often improves the response.

It actually works so well that OpenAI based its new O1 models on a similar concept. They said the following in the announcement:

We’ve developed a new series of AI models designed to spend more time thinking before they respond. They can reason through complex tasks and solve harder problems than previous models in science, coding, and math.

As the O1 models already use something similar under the hood, you may not want to try this trick if you use them.

Use tools

LLMs are powerful. We use them for everything. And, as Maslow said a long time ago, “it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”

But LLMs suck at some specific tasks. These tasks depend on the model we use, but most models struggle with their math skills, working with dates, etc. The previous tricks might help with these skill issues in some situations but are not real solutions.

Luckily, all major LLMs now support tools. So, instead of using a hammer for everything, we can bring our tools for specific cases.

But what are tools, and why are they useful? Here’s an example!

I asked Claude: “What’s the weather like in Montreal today?” Even if you don’t have the current weather info, you can bet it’s cold in Montreal in November. But Claude tries to be helpful, and it answers the following:

I need to clarify something: I don’t have access to real-time weather data or current conditions. While I can discuss Montreal’s typical weather patterns and climate, I cannot tell you today’s specific weather conditions. For current weather information in Montreal, I’d recommend:

1. Checking a weather website or app
2. Looking up Environment Canada’s weather service
3. Checking local Montreal news stations’ weather reports

Would you like to know about Montreal’s typical weather patterns for this time of year instead?

Instead of giving up, we can give an LLM, in this case, Claude, the skill to check the current weather conditions in the specified location.

How do we do that?

Let’s see the code example! I’ll use AWS Bedrock Converse API with Node.js and Claude Sonnet 3.5 as a model, but this is just an example. You can do something similar with any popular model.

// Same as above
import { BedrockRuntimeClient, ConverseCommand } from '@aws-sdk/client-bedrock-runtime'

const client = new BedrockRuntimeClient({ region: "eu-central-1" })

const modelId = 'anthropic.claude-3-5-sonnet-20240620-v1:0'

// Start a conversation with the user message.
const conversation = [
  {
    role: 'user',
    content: [{ text: "What's the weather like in Montreal?" }],
  },
]

// Define available tools
const toolConfig = {
  tools: [{
    toolSpec: {
      // The name and the description of our tool
      name: 'get_weather',
      description: 'Get weather conditions for a specific location',
      inputSchema: {
        // It accepts JSON, I told you we love JSON!
        json: {
          type: 'object',
          properties: {
            // And it expects the "location" property (see the type and the description)
            // For simplicity, we'll skip the date part. Our tool will always answer with today's weather
            location: {
              type: 'string',
              description: 'The location for which you want the weather conditions. Location can be any city, country, etc.',
            },
          },
          required: [ 'location' ],
        },
      },
    },
  }],
]

// Create a command with the model ID, the message, and a basic configuration.
const command = new ConverseCommand({
  modelId,
  messages: conversation,
  toolConfig: toolConfig, // Pass the tool configuration to Bedrock
})

try {
  // Send the command to the model and wait for the response
  const response = await client.send(command)

  // We'll see this part in the next example
} catch (err) {
  console.log('ERROR', err)
}

That’s the first part of the magic. Claude now knows it can use a tool. However, it still needs to learn how to answer the question. Let’s add the actual skill.

AWS Bedrock Converse command will reply with a JSON object similar to the following:

{
    "output": {
        "message": {
            "role": "assistant",
            "content": [
                {
                    "toolUse": {
                        "toolUseId": "tooluse_kZJMlvQmRJ6eAyJE5GIl7Q",
                        "name": "get_weather",
                        "input": {
                            "location": "Montreal"
                        }
                    }
                }
            ]
        }
    },
    "stopReason": "tool_use"
}

Now that we know that, we can update the code with the following (this is a very simple example):

// Code from above
try {
  // Send the command to the model and wait for the response
  let response = await client.send(command)
  
  // Check if the stop signal is "tool_use" and if our tool was used
  if (response.stopReason === 'tool_use' && response.output.message.content[0].toolUse.name === 'get_weather') {
    // Extract location and toolUseId
    const location = response.output.message.content[0].toolUse.input.location
    const toolUseId = response.output.message.content[0].toolUse.toolUseId
    // And call a function that might call some 3rd party API to get the current weather conditions
    const weatherConditionsResponse = await getWeatherConditions(location)
    
    // Then invoke the model again with the tool_use message and the reply
    const updatedCommand = new ConverseCommand({
      modelId,
      messages: [
        ...conversation, // Previous message
        response.output.message, // Assistant response
        // Tool response
        {
    			role: 'user',
    			content: [
        		{
            	toolResult: {
                toolUseId: toolUseId,
                content: [
                  {
                    // Let's assume that weatherConditionsResponse contains JSON an LLM can understand
                    json: weatherConditionsResponse,
                  },
                ],
              },
            },
          ],
        },
      ],
      toolConfig: toolConfig, // Pass the tool configuration to Bedrock
    })
    response = await client.send(updatedCommand)
  }

  // Continue parsing the code
} catch (err) {
  console.log('ERROR', err)
}

And that’s it. You have a tool now! Remember that the tool can be any function in your preferred programming language (even Java, I don’t judge you).

You probably heard people talking about AI agents. Sounds magical, right? But most of the time, agents are just 3 raccoons (pardon, tools) in a trench coat.

Ask an LLM to improve the prompt

Can an LLM write a prompt? Remember, prompts are just commands. Often textual. And LLMs are good with text. So, in theory, yes, it can.

But that’s theory. What about practice?

You should try it! It’s simple. Just paste your initial prompt to ChatGPT or Claude and ask it to review and improve it. Guess what? The better instructions you provide, the better result you may expect.

Both OpenAI and Claude now have their prompt generator tools. These generators are a good start.

But we do not need to reinvent the wheel. Giving commands and instructions evolved with humans. Many people wrote about it. For example, the military has many techniques for giving commands and communicating intent.

One of the useful concepts is the Commander’s Intent Statement, which aims to explain the desired end state. An effective Commander’s Intent statement should contain the following:

• Purpose: Explain why personnel must complete the assignment.
• Task: Explain the objective or goal.
• End state: Explain what the result should look like.

An easy way to improve your current prompt is to paste it in ChatGPT or Claude and ask an LLM to rewrite it to follow the Commander’s Intent Statement. You can tell an LLM to ask you if any important details are missing.

However, the Commander’s Intent Statement is just one example. I recently re-read an excellent book called “Sources of Power: How People Make Decisions” by Gary Klein. The book contains a lot of useful information. While the book does not talk about LLMs, some concepts apply to both human conversations and LLMs. For example, the “Considerations in Communicating intent” section presents a good “framework” to review the quality of the command or intent statement.

Gary mentions that a person could present seven types of information to help the people receiving the request understand what to do. Here’s the list of all seven types:

1. The purpose of the task (the higher level goals)
2. The objective of the task (an image of the desired outcome)
3. The sequence of steps in the plan
4. The rationale for the plan
5. The key decisions that may have to be made
6. Antigoals
7. Constraints and other considerations

You do not need to have all seven types of information for an effective prompt. For example, you need a strong sequence of steps only when the task’s purpose and objective are not strong enough. However, the list provides good questions to ask when writing a complex prompt.

Summary

Let’s quickly recap the five tips mentioned above:

• Hint the beginning of an answer, LLM will continue.
• Give examples (or use the few-shot prompting technique if you want to sound smart).
• Ask an LLM to think step-by-step.
• Help an LLM with tools. Remember that tools are just functions.
• Ask an LLM to help you improve the prompt.

And, before you go, I just want to state the obvious: writing effective prompts requires thinking and writing. You’ll need to write a lot. I know! You hate writing. But you wrote thousands of lines of JavaScript or Python last month. That’s also just a bunch of text and instructions.