Agentic Engineering: 30 Concepts Are Useful, but Knowing When to Use an Agent Matters More

Agentic AI is quickly becoming one of the most discussed topics in software and data engineering.

Agents. Tools. Memory. Planning. MCP. Multi-agent systems. RAG. Guardrails. Observability.

The terminology is growing almost as quickly as the frameworks implementing it.

But after experimenting with LLMs in real data-engineering workflows, I have started thinking that there is an even more fundamental question we need to answer:

Does this problem actually need an AI agent?

That question sounds simple.

In practice, it may be one of the most important architectural decisions we make while building AI-enabled systems.

I recently came across an excellent article covering 30 core concepts in Agentic Engineering:

30 Core Agentic Engineering Concepts Every Developer Should Know

Understanding these concepts is important.

But I think the next step is understanding how they fit together in real engineering systems - especially in data engineering.

---

1. First: What Is an Agent?

At the simplest level, an agent can be thought of as a loop.

Goal -> Think -> Act -> Observe -> Repeat

A traditional program usually follows instructions we define ahead of time.

For example:

Read file
Validate schema
Load table
Run reconciliation
Generate report

The developer has already decided the sequence.

An agent operates differently. We might instead tell it:

Investigate why yesterday's pipeline failed and determine the most likely root cause.

The system may then decide:

Check pipeline logs
        |
Inspect failed task
        |
Query target table
        |
Compare source and target counts
        |
Inspect recent schema changes
        |
Run another validation
        |
Determine likely root cause
        |
Recommend remediation

The important difference is not that an LLM is involved.

The important difference is: the next step is determined dynamically based on what happened in the previous step.

That is where the workflow starts becoming agentic.

---

2. Prompt vs Workflow vs Agent

This is one of the most useful distinctions when designing AI systems.

Not every problem needs an agent. There are roughly three levels.

Level 1 - Prompt

Suppose I have a Snowflake query and want an explanation.

I provide a SELECT statement and ask: Explain what this query does.

The LLM produces an answer. There is no reason for an autonomous agent.

Input -> LLM -> Output

Done.

Level 2 - Deterministic Workflow

Now imagine I have an STTM containing 500 target columns. For every column I want to generate null checks, datatype checks, default-value checks, referential-integrity checks, duplicate checks, and reconciliation SQL.

At first this may look like an excellent AI use case. But much of it is deterministic.

If metadata says:

COLUMN_NAME = CUSTOMER_ID
DATA_TYPE = NUMBER
NULLABLE = NO
DEFAULT = -98

I can generate the validation SQL with Python. I don't need an LLM to "think" 500 times.

A metadata-driven Python generator might be faster, cheaper, repeatable, testable, and deterministic.

This is an important principle:

Just because an LLM can generate something doesn't mean an LLM should generate it.

---

3. When Does the Same Problem Become Agentic?

Now change the requirement.

Instead of: Generate 500 validation queries.

I ask: Validate whether this implementation correctly follows the STTM and investigate anything suspicious.

Now the system may need to:

• ›Read the STTM
• ›Understand source-to-target mappings
• ›Inspect source DDL
• ›Inspect target DDL
• ›Understand joins
• ›Understand filtering conditions
• ›Identify transformations
• ›Generate appropriate tests
• ›Execute those tests
• ›Analyze failures
• ›Determine whether the problem is data, mapping, code, or metadata
• ›Generate another test based on the result
• ›Recommend remediation

That is fundamentally different.

Imagine the first validation finds:

Source rows: 5,400,000
Target rows: 5,230,000
Difference:    170,000

The system now has to decide: What should I investigate next?

Maybe it checks business keys. It discovers the same business key maps inconsistently through an alternate hierarchy. Then it decides to inspect the hierarchy join. It discovers that some records return a valid dimension ID while others receive a default ID.

That feedback loop is where an agent becomes useful.

---

4. Tools Are What Turn an LLM Into an Agent

An LLM by itself can reason about information we provide. But enterprise systems require interaction.

An agent may need tools for Snowflake, GitHub, Databricks, Jira, data catalogs, APIs, pipeline orchestrators, observability platforms, and documentation repositories.

Consider a data-quality agent. Its available tools might include:

get_table_schema()
run_sql()
get_pipeline_logs()
get_git_diff()
read_sttm()
get_data_quality_results()

The LLM becomes the reasoning layer. The tools become its ability to act.

This separation is extremely important. The model should not directly control everything. Instead, we expose carefully designed capabilities.

---

5. Tool Design Is More Important Than It Looks

Giving an agent a generic function like execute_any_sql(sql) may be convenient. It is also dangerous.

A better design may expose:

run_read_only_query()
get_table_row_count()
get_column_statistics()
get_schema()
compare_source_target()

Why? Because agent reliability depends heavily on limiting the action space. The more unrestricted the tool, the more opportunities exist for unexpected behavior.

In enterprise systems: good tool design becomes part of AI governance.

---

6. Context Is the Agent's Working Memory

Suppose we ask an agent to investigate a pipeline. It may need the STTM, source DDL, target DDL, transformation code, pipeline logs, data-quality results, previous investigation steps, and business rules.

Putting everything into one giant prompt is tempting. It is also inefficient.

Large context creates several problems: higher token usage, higher cost, slower responses, irrelevant information, context dilution, and harder debugging.

The better approach is usually: retrieve only what the agent needs for the current step.

This is why context engineering is becoming as important as prompt engineering.

---

7. Memory Is Different From Context

Context answers: What does the agent need to know right now?

Memory answers: What should the system remember across interactions?

For example, a data-engineering agent may learn:

• •DIM_CUSTOMER_ID = -98 means source mapping missing
• •DIM_CUSTOMER_ID = -99 means not applicable
• •Machine transactions use a different transformation path
• •YTD measures reset at the beginning of the year
• •ITD measures carry forward

The important architectural question is: What should the model remember, and what should remain authoritative enterprise metadata?

I strongly prefer authoritative business rules to remain outside the LLM. The agent should retrieve them. It should not become the system of record.

---

8. Planning

Complex tasks often require decomposition.

Goal: Validate implementation against STTM.

The agent might generate a plan: parse STTM, extract mappings, compare schemas, validate joins, validate filters, validate transformations, generate test queries, execute tests, analyze discrepancies, produce findings.

Planning is useful because it makes the reasoning process more structured. But plans should not necessarily be rigid. If step 4 reveals a major problem, continuing blindly through all ten steps may waste time.

A good agent should be capable of replanning.

---

9. Reflection

One interesting agentic pattern is reflection.

Instead of immediately accepting its answer, the system asks: Did the evidence actually support this conclusion? What assumptions did I make? Is another validation required?

For example:

Agent hypothesis: The 170K-row difference is caused by hierarchy mapping.

Reflection: Do all 170K missing rows share the hierarchy issue?

The agent runs another query. Maybe only 130K do. Now the hypothesis changes.

This is much closer to how engineers investigate production problems.

---

10. Multi-Agent Systems

Multi-agent architectures are attractive:

STTM Agent -> SQL Validation Agent -> Data Quality Agent -> Root Cause Agent -> Documentation Agent

It looks elegant. But we should be careful.

Every additional agent adds another prompt, another context, another model call, more tokens, more latency, more failure possibilities, and more observability requirements.

Sometimes five agents are useful. Sometimes five Python functions would be better.

The question should never be: How many agents can we build?

The question should be: Where is independent reasoning actually required?

---

11. The Cost Problem

Agentic workflows can become surprisingly expensive.

Imagine an agent analyzing 500 columns. For every column it reads metadata, reasons, generates SQL, executes validation, reads the result, reasons again, and writes an explanation.

If every step requires model calls, token consumption can explode.

Instead we could design:

Metadata
    |
Python Generator
    |
500 deterministic SQL checks
    |
Execute in Snowflake
    |
Collect failures only
    |
LLM analyzes exceptions

This is a very different architecture. The LLM handles ambiguity. Python handles repetition. Snowflake handles computation. That separation can dramatically improve the system.

---

12. The Agent Should Not Be the Compute Engine

Suppose we need to determine whether 500 million records contain duplicates.

The agent should not process 500 million records. Snowflake should.

The agent should generate:

SELECT business_key, COUNT(*)
FROM fact_table
GROUP BY business_key
HAVING COUNT(*) > 1;

Snowflake performs the computation. The agent interprets the result.

Think of the architecture as:

LLM      = Reasoning
Python   = Deterministic logic
Database = Computation
Metadata = Knowledge
Agent    = Orchestration

Each component should do what it does best.

---

13. Guardrails

Now imagine an autonomous data agent connected to production. It identifies what it believes is incorrect data.

Should it execute DELETE FROM production_table ...?

Absolutely not by default.

A safer workflow:

Detect issue
     |
Generate evidence
     |
Recommend fix
     |
Generate SQL
     |
Human approval
     |
Execute

This introduces an important concept: human-in-the-loop.

Autonomy should depend on risk. Reading metadata may require no approval. Running SELECT queries may require minimal controls. Changing production data should require strong approval.

---

14. Observability

Traditional applications need logs. Agentic applications need much more.

We need to know: what goal was given, what context was retrieved, which model was used, which tools were called, what queries were executed, how many tokens were consumed, how much the workflow cost, which decisions were made, what failed, what was retried, and who approved the action.

Without this information, debugging an agent can become extremely difficult.

Instead of "Agent failed," we need a full trace - from goal through every tool call, retry, and decision to the final root cause. That trace becomes critical for enterprise adoption.

---

15. MCP and Standardized Tool Access

One reason MCP has received so much attention is that agents need standardized ways to interact with external systems.

Instead of building custom integration logic every time, standardized interfaces can expose databases, files, Git repositories, documentation, APIs, and developer tools.

But MCP does not magically make an application agentic. It provides connectivity.

The architecture still needs to decide: when should the tool be called, which tool, with what permissions, what data should be returned, and what happens after the result.

Connectivity and intelligence are separate problems.

---

16. RAG and Agents Solve Different Problems

RAG answers: What information should I retrieve to answer this question?

Agents answer: What action should I take next to achieve this goal?

They can work together:

Agent receives investigation goal
       |
RAG retrieves business rule
       |
Agent generates validation
       |
Snowflake executes SQL
       |
Agent analyzes result
       |
RAG retrieves another mapping rule
       |
Agent continues investigation

RAG supplies knowledge. The agent controls the workflow.

---

17. Failure Recovery

Real systems fail. Queries time out. APIs return errors. Files are missing. Schemas change. Permissions expire.

A useful agent must understand how to recover.

For example, if a query fails because POLICY_ID does not exist, a brittle workflow stops. A more capable agent may inspect the latest schema, find POLICY_IDENTIFIER, check mapping metadata, regenerate the query, and retry.

This is one area where agentic systems can provide genuine value. The path cannot always be predicted ahead of time.

---

18. Agents Need Boundaries

A production agent should know what it CAN do, what it CANNOT do, when it needs approval, when it should stop, and when it should escalate.

For example:

• •Allowed: Read metadata, run SELECT queries, generate validation SQL, analyze logs
• •Approval required: Modify pipeline configuration, create pull request
• •Not allowed: Delete production data, modify security policies, expose sensitive data

This should be designed into the architecture - not buried in a prompt saying "please be careful."

---

19. Deterministic Engineering Still Matters

There is a danger that we start replacing good software engineering with LLM calls.

IF datatype = NUMBER THEN generate numeric validation - that is deterministic. We don't need reasoning.

A useful design principle: push deterministic logic outside the LLM whenever practical.

The result is usually cheaper, faster, easier to test, easier to audit, and more predictable.

---

20. A Hybrid Architecture for Data Engineering

The architecture I find increasingly interesting:

              USER GOAL
                 |
          ORCHESTRATOR
                 |
    +------------+------------+
    |                         |
DETERMINISTIC ENGINE      LLM REASONING
    |                         |
 Python                  Investigation
 Rules                   Interpretation
 Metadata                Planning
    |                         |
    +------------+------------+
                 |
               TOOLS
                 |
   +-------------+-------------+
   |             |             |
Snowflake      GitHub       Catalog
   |             |             |
   +-------------+-------------+
                 |
              RESULTS
                 |
           LLM ANALYSIS
                 |
         HUMAN APPROVAL

This is different from simply wrapping everything in an agent. The agent becomes the coordinator of intelligence. It doesn't replace the engineering platform.

---

21. Example: An STTM Validation Agent

Let's make this concrete.

Suppose an enterprise project contains an STTM, source tables, target tables, transformation SQL, business rules, and 500 target columns.

The goal: Determine whether the implementation correctly follows the STTM.

Step 1 - Parse metadata: Python extracts target column, source column, transformation, join condition, filter, default value, and datatype. No LLM required. Step 2 - Detect deterministic differences: Python compares STTM target columns vs. actual table columns. Again, no LLM required. Step 3 - Understand complex transformations: Now the LLM examines transformations like:

CASE
WHEN TRANSACTION_TYPE = 'PREMIUM'
AND ACCOUNT_CODE IN (...)
THEN AMOUNT * SIGNAGE
END

Here semantic reasoning becomes useful.

Step 4 - Generate targeted tests: The agent determines which validations are required. Step 5 - Execute: Snowflake performs the calculations. Step 6 - Analyze failures: Instead of sending 500 successful tests to the LLM, we might send only the 13 that require investigation. The model concentrates on those. Step 7 - Investigate: The agent may dynamically generate additional SQL. Step 8 - Produce evidence: Final output includes mapping, expected behavior, implemented behavior, test executed, observed result, potential issue, confidence, and recommended action.

Now the LLM is being used where reasoning actually matters.

---

22. The 95/5 Architecture

A useful mental model is something I call the 95/5 architecture.

This doesn't mean the percentages must literally be 95 and 5. The principle is:

Let deterministic systems handle the majority of predictable work. Use AI for the smaller portion requiring interpretation, ambiguity, and reasoning.

For example:

500 columns
475 straightforward validations -> Metadata + Python + SQL
 25 complex transformations      -> LLM analysis

The exact ratio may become 80/20 or 60/40 depending on the problem. But the architecture should intentionally ask: Where is intelligence actually required?

---

23. Don't Measure Agent Sophistication by the Number of Agents

Six agents do not automatically create a better architecture than one agent.

In fact:

More Agents -> More Context -> More Model Calls -> More Tokens
           -> More Latency -> More Failure Paths -> More Governance

Multi-agent architecture should solve a real separation-of-responsibility problem. Not become an architectural fashion statement.

---

24. Evaluation Is Different for Agents

Testing an ordinary function is straightforward: Input -> Expected Output.

Agents are harder. The same goal may produce different valid paths.

Therefore evaluation may need to measure: Did the agent achieve the goal? Did it use appropriate tools? Did it avoid unsafe actions? Was the conclusion supported by evidence? How many steps were required? What did it cost? Did it unnecessarily call the LLM?

This last question is especially interesting. Efficiency itself should become an agent evaluation metric.

---

25. Cost Should Become an Architectural Metric

For traditional systems we monitor CPU, memory, storage, network, and query cost.

For agentic systems we should also monitor tokens per task, LLM calls per task, cost per successful task, tool calls, retry count, context size, and latency.

Imagine two agents produce the same answer:

• •Agent A: 47 model calls, $3.20, 90 seconds
• •Agent B: 5 model calls, $0.08, 12 seconds

Accuracy alone is not enough to evaluate them.

---

26. Model Routing

Not every task requires the most powerful model.

An architecture could route:

Simple classification          -> Small/local model
SQL explanation                -> Medium model
Complex root-cause investigation -> Advanced reasoning model
Deterministic generation       -> Python
Large aggregation              -> Snowflake

This can dramatically change the economics of enterprise AI.

The future may not be: Which LLM are we using?

It may be: How intelligently are we routing work between models, code, and compute engines?

---

27. Agentic Engineering Is Really Systems Engineering

This may be the biggest lesson.

Building production agents is not primarily about writing clever prompts. It combines software engineering, data engineering, distributed systems, security, AI/LLMs, observability, governance, cost engineering, and human workflow design.

The LLM is only one component.

---

28. Start With the Workflow, Not the Framework

Before selecting LangChain, LangGraph, CrewAI, AutoGen, or another framework, I would first draw the workflow.

Ask:

• •What is the goal?
• •What information is required?
• •What steps are deterministic?
• •Where does uncertainty exist?
• •Which decisions require reasoning?
• •Which systems must be accessed?
• •What actions require approval?
• •What happens when something fails?
• •How do we measure success?
• •How much should one execution cost?

Only then choose the framework. Framework-first architecture often leads to unnecessary complexity.

---

29. A Simple Decision Framework

Before building an agent, I now like to think about the problem this way:

That decision process can prevent a lot of unnecessary architecture.

---

30. The Most Important Agentic Engineering Skill

I think developers absolutely should understand agents, memory, context, tools, planning, RAG, MCP, reflection, multi-agent architectures, guardrails, and observability.

But there is another skill underneath all of them: engineering judgment.

Knowing when to use a prompt vs. Python vs. SQL vs. workflow vs. RAG vs. agent vs. multi-agent system may ultimately matter more than knowing every AI framework.

Because the best agentic architecture may sometimes contain fewer agents. The best LLM workflow may sometimes contain fewer LLM calls. And the best AI solution may sometimes be a Python script.

---

Final Thoughts

We are moving from the era of prompt engineering toward agentic engineering. That is an exciting shift.

But production Agentic AI will require more than giving an LLM tools and putting it inside a loop.

We need systems that are intelligent enough to reason, deterministic where possible, observable by design, governed by risk, and economical enough to operate at scale.

For data engineering specifically, I believe the winning architecture will increasingly be hybrid:

Metadata for knowledge
+
Python for deterministic automation
+
Snowflake/Databricks for computation
+
LLMs for reasoning
+
Agents for dynamic orchestration
+
Humans for high-risk decisions

That is where Agentic AI becomes more than a demo. It becomes engineering.

---

Reference: "30 Core Agentic Engineering Concepts Every Developer Should Know"