From Prompt Chains to Agent Graphs: How Our Agent Architecture Evolved

Stage 1 - Build chatbot using prompt chaining workflow

The first iteration began in early 2025 because during that time we repeatedly had to diagnose batch-ingestion failures, where we need to diagnose batch ingestion failure. So I was thinking maybe we can build a chatbot (that time the term “agent” was not yet as widely used) and help reduce the human effort.

So I built a prompt chaining workflow using CrewAI and picked Llama 3 served by Ollama:

During that time, the model isn’t very intelligent, therefore, in order to reduce hallucination, we have to make the workflow deterministic.

Stage 2 — LangGraph-Based Tool Orchestration

My second journey started in October 2025, the background is we have 100+ tables stored in DataBricks Unity Catalog related to payment (contract, license, order, offer … etc), and we want to build a chatbot for answering question for our customer.

Initially, we wanted to adopt Genie, for those who doesn’t use Genie before, Genie is a DataBricks feature that allows business teams to interact with their data using natural language. You can simply create a Genie space and fill in the table and some instruction and example SQL queries which helps Genie generate a better sql query.

However, after I did some exploration, I found a few problems:

• we can provide some instructions to help Genie understand our business logic, but genie starts to hallucinate as the semantic scope and instruction set became more heterogeneous.
• We have so many tables (100+) and it becomes very difficult to help Genie differentiate between them.

The solution is simple, Rather than putting heterogeneous business domains behind a single generalist Genie space, we partitioned the system into domain-specialized spaces.

In our internal testing, domain specialization reduced table-selection errors, but introduced a new problem: how to route each request to the appropriate Genie space.

Inspired by Cursor that time, where you can host an mcp server, and cursor handles the orchestration. So I am thinking, can we also do the same thing on our end, basically wrapped our Genie api into tools and building an agent which do the orchestration, and route to the tools.

This is what our architecture looks like:

Example code:

from langgraph.graph import END, StateGraph
workflow = StateGraph(AgentState)

workflow.add_node("agent", RunnableLambda(call_model))
workflow.add_node("tools", ToolNode(tools))

workflow.set_entry_point("agent")
workflow.add_conditional_edges(
  "agent",
  should_continue,
  {
     "continue": "tools",
     "end": END,
  },
)
workflow.add_edge("tools", "agent")

return workflow.compile()

Afterwards, we also integrate with our slack channel so that our customer can simply ask question in slack:

Building agent with skill-based agent loop

Starting from early 2026, the term “skill” has become increasing common, we decided to migrate based on several reasons:

• We want to enhance our agent to not just generating sql, but also triage on-call alerts, monitoring lag, create schedule and report. So we need a centralized place to manage our knowledge.
• Inspired by Andrej Karpathy’s llm-wiki, we build a similar thing: llm will incrementally persist / update a wiki while engineer is coding without additional effort. And this will act as the skill - for our agent to use. Therefore our architecture need to reflect these changes.

Our architecture looks like:

We made a few changes:

• Our knowledge will be stored in skill, and the skill.md looks like: Our smallest granularity is skill, which defines the boundary for the llm. In the agent loop, LLM will only orchestrate with the tool listed in the skill.
• We refactor all our mcp tools into cli, in filesystem format. We did it since Models are great at navigating filesystems. Presenting tools as code on a filesystem allows models to read tool definitions on-demand, rather than reading them all up-front.
• We remove the agent framework such as LangGraph, since they often create extra layers of abstraction.

Shifting from loop to graph

We decided to migrate our harness from loop to graph because we find that sometimes there is a dependency via tool selection, but don’t want to hard-code everything in skill.md.

Let me explain more, for example, when we want to query a table, we want our agent to describe the table first, check the schema before actual query the table. Oncall is a more complex usecase, we want our agent to first lookup runbook, check cortex, pods to get more knowledge before doing more heavy lifting work such as querying splunk, replay the api call …

So there are actual dependencies via tool, but we don’t want to hard-code it in our skill. Because we want to give llm more freedom (llm is getting more and more intelligent), and hard-code everything means if anything changes, we have to update the skill, which is also time-consuming.

Therefore, we decided to migrate our agent harness from loop to graph, so instead of defining the allowed_tools in agent.md file, we will also provide the dependency in the skill:

And we also setup some rules:

• If tool A depends on tool B, then tool B will have to be execute first.
• If all the predecessor has executed, the tool can be use.
• Every layer of the graph cannot have tool which have predecessor tool not processed yet.

Instead of hard-coding the sequence in skill, we only set the tool dependency in skill, in that way, llm can reuse the tool as long as it meet with our rules, since each layer it doesn’t have tool dependency, each layer can run tool asynchronously.

So this will translate into a DAG, where each node represents a tool, and each edge represents the dependency of the tool. Then a very popular algorithm came into my mind: Topological sort. We will use Kahn's topological sort algorithm, where a tool will be released for llm where all the predecessor has been used.