In this lecture, Frank Coyle explains that LLM and agent probabilistic reasoning alone make it difficult to prevent errors in actual work. He places ontology and logical verification layer outside the agent and presents neurosymbolic AI as a solution, which combines the flexible judgment of a neural network with the strict constraints of a rule-based system. The key is "Check the type at the entrance with Pydantic, and let the agent act only after verifying the result with Ontology."

1. Philosophy of education and practice in the AI era

The speaker introduces himself as an educator teaching at UC Berkeley and someone who has studied and taught computer science for about 30 to 35 years. He says this is a very important time for computer science students. In the past, you could get a stable job with just a computer science degree, but now with the advent of AI, that guarantee has become weaker. Nevertheless, the fact that about 5,000 people gathered in the lecture hall emphasizes that AI and agents are an important direction moving forward.

The question he asks is how to best utilize the new world of AI. In today's lecture, he says he will explain how agents and ontology can be combined as one of the answers.

But before getting into the main topic, he introduces his educational philosophy. The philosophy is said to have originated from the words of poet and artist Sister Corita Kent, and was made widely known through avant-garde musician John Cage.

"Nothing is a mistake. There is no victory. There is no failure. There is only what you make."

Coyle says what's especially important today is not just reading books, but making and trying things yourself. This means that you learn by making it yourself, even if the result is not perfect.

He also values ​​writing and the act of recording by hand. He explains that his early career was in neuroscience, and that he is returning to that field as agent AI brings back cognitive science. When you type on a keyboard, your brain focuses on the letters on the keyboard, but when you write on paper with a pen or pencil, more sensory systems, including hand movements, vision, and senses, work together.

"Use your senses. Get a notebook, pen, and pencil. Draw and write."

He begins his lecture by saying that writing and drawing by hand can lead to faster and deeper learning than simply typing.

2. Two lineages: agent and ontology

Coyle explains that agents and ontologies each have a long history and lineage.

Agent's Genealogy

The concept of agents did not appear suddenly recently. In the research of early artificial intelligence researchers such as John McCarthy and Marvin Minsky, the idea that artificial intelligence could become a system that recognizes its surroundings, makes judgments, and acts like humans already emerged.

The term artificial intelligence appeared in 1956 when several researchers gathered to discuss future computing and intelligence. Afterwards, the concept of agent gradually developed into the following structure.

1. Aware of your surroundings
2. After making a decision based on the situation
3. Actually acting

The AI agents that people talk about today also follow this structure. This is how LLM interprets information, decides which tools to use, and influences the outside world through tool invocation.

Genealogy of ontology

Ontology is an older concept. Coil traces its origins back to the ancient Greek philosopher Aristotle. Aristotle attempted to systematically classify "what existence is" and organized the categories of existence.

This way of thinking is connected to today's knowledge representations, graph databases, and knowledge graphs. Later, philosopher Willard Van Orman Quine and Tom Gruber, who presented the famous definition of ontology in 1993, further formalized this concept.

Ontology can be understood as follows.

"A formal specification of a shared conceptualization"

In other words, it organizes how a specific organization or field understands the world in the form of entities, properties, relationships, and rules. Providing an ontology to an agent means clearly informing the agent of the domain in which the agent operates and the conceptual structure within it.

3. Neurosymbolic AI: Building logic outside of probabilistic models

Nowadays, probabilistic agents and LLM are developing rapidly. At the same time, more formal and logical representations such as ontologies, rule-based systems, and knowledge graphs also exist. Coyle calls the combination of the two Neurosymbolic AI.

Neurosymbolic AI consists of two major parts.

• Neural networks and LLM: Understand language, understand context, and reason flexibly.
• Symbolic AI and logic systems: Apply rules, types, and constraints and block incorrect conclusions.

The point of Coyle's argument is not to abolish the LLM. Rather, the advantage of LLM, which is flexibility and creativity, must be utilized, but logical guardrails must be installed outside it.

"Neurosymbolic AI is the way to keep LLM within guardrails."

LLM is inherently probabilistic. Rather than logically proving that a certain sentence is true and outputting it, it generates a token with a high probability of coming next based on the context so far. Therefore, hallucinations are not just an exception, but a natural phenomenon of how LLM works.

Coyle also compares this to human imagination. Humans also imagine things that do not exist, and turn those imaginations into reality. LLM also works in that way, allowing you to come up with flexible ideas, but it can be just as dangerous when applied to actual tasks such as payment, refunds, and delivery.

"LLMs are inherently probabilistic. People worry about hallucinations, but that's actually a characteristic of large-scale language models."

Therefore, it is necessary not only to trust the internal judgment of LLM, but also to have an explicit logic layer externally and a structure that checks the results before action.

4. What is ontology and how to create it

Coyle explains that although ontology may sound like a difficult and grand philosophical concept, it is fundamentally simple. Ontology is a structure that expresses the relationships between entities and the properties they possess.

For example, a purchasing/customer support system may have the following entities:

• order
• Customer
• Payment
• Refund
• Customer support representative
• Delivery status

And there is the following relationship between them.

• The customer places an order.
• Payment is linked to the order.
• Orders can be refunded.
• Customer support staff handle customer inquiries.
• Orders are given a delivery status.

Graph database and ontology

Traditional relational databases store data in tables and columns. Adding new information often requires creating new columns and changing the structure of a table.

On the other hand, a graph database connects objects and relationships, so new properties or relationships can be added relatively flexibly.

"In a graph database, you can simply connect different items, properties, or relationships."

This structure is advantageous when continuously expanding the organization's business concept or connecting different data sources.

Two approaches to creating an ontology

Methods for creating ontology are largely divided into top-down and bottom-up.

In a top-down approach, experts in the field gather together to analyze the domain. For example, in a purchasing system, first organize what entities exist, what properties each entity has, and how they are connected to each other.

"What do you have? You have purchase orders, you have customers, you have customer support representatives. You structure these, you define attributes and relationships."

The bottom-up approach starts with real data and use cases. While examining customer inquiries or transaction records, we discover entities and relationships that appear repeatedly and add them to the ontology.

Both approaches are useful, and in practice, it is more effective to utilize a combination of expert conceptual models and data accumulated in the field.

5. Lessons learned from past expert systems and utilization of existing classification systems

Coyle explains that the top-down approach to creating ontologies resembles the expert systems of the 1980s. At the time, symbolic AI and rule-based systems were considered the future of artificial intelligence.

Many companies invested in expert systems, and in Japan, a national future technology project was also promoted in the late 1980s. However, expert systems did not solve the scalability problem. Rules had to keep being added, and as the system became more complex, it became more difficult to maintain.

"But they didn't scale. They didn't scale, and we ended up going into AI winter."

Afterwards, neural networks received attention again, but initially there was not enough computing power. Neural network research has existed since the 1960s, but there was a lack of hardware to train it on a large scale. Coyle explains that advances in GPUs, especially those developed for gaming graphics, have enabled the revival of neural networks.

"We didn't have Nvidia back then. We couldn't make neural networks a reality because we didn't have GPUs."

Today's AI is made possible by the combination of neural networks and large-scale computing resources, but Coyle argues that it is time to revisit the advantages of symbolic AI.

Reusing an already existing ontology

Not all organizations need to create an ontology from scratch. This is because various classification systems and ontologies have already been created in various fields over the past 15 to 20 years.

Representative examples are as follows:

• schema.org: Defines objects and relationships used in web documents and services.
• FOAF(Friend of a Friend): Used to express social relationships with people.
• Dublin Core: Describes metadata of papers, books, research materials, etc.
• DBpedia: Expresses Wikipedia information as a structured knowledge graph.

Coyle says that by utilizing these existing resources, there is no need to reinvent the wheel.

"Use what already exists. You don't have to recreate everything from scratch."

The explanation is that when searching on Wikipedia, it does not just look for simple text documents, but also utilizes the huge graph structure and knowledge representation in the background.

6. Inference and constraints provided by RDFS and OWL

Ontology does not end with expressing entities and relationships. If you add inference rules and constraints here, the system can derive information that is not explicitly written down or find incorrect data.

Coyle introduces technologies such as RDFS and OWL for this purpose. These techniques lie next to the graph, expanding the meaning of the ontology and applying conditions and rules of relationships.

Domains and scopes in RDFS

For example, let's say there is a relationship called 가르친다(teaches). The subject, or domain, of this relationship can be defined as the teacher.

Then, when we have the following sentence:

"Bob teaches scootering."

The system can infer that Bob is a teacher. If all teachers are defined as people, we can automatically obtain the information that Bob is a person.

Conversely, if we define the target, or scope, of 가르친다 as a student, we can also derive the fact that the scooter is a student.

In this way, ontology does not simply store input sentences, but infers new information based on the meaning of predefined relationships.

Transitive relationships and functional properties in OWL

In OWL, more diverse logical relationships can be defined. For example, let's say the relationship 조상이다(ancestor of) is transitive.

• Sue is Mary's ancestor.
• Mary is Anne's ancestor.

Then we can automatically infer that Sue is Anne's ancestor.

You can also define functional properties. A functional property means that an object should have only one value for a particular property. For example, you could have a rule that a person can only have one father.

If Bob and BB are both recorded as Jim's father, the system can infer the possibility that Bob and BB are in fact different representations of the same person.

These rules are not facts stored directly within the graph, but are an additional layer of logic applied to the graph. It is this layer that can be utilized to inspect the agent's behavior.

7. The power and danger of agent loops

Coyle now explains the core of the agent system: the loop. In programming, it has long been discussed that a program consists of three basic elements.

1. Sequential execution: Execute A, then execute B and C.
2. Conditional statement: Performs different tasks depending on conditions.
3. Repeat statement: Continuously repeats a specific task.

In 1966, Böhm and Jakopini explained that if these three elements are in place, the programming language can express anything that can be computed. As loops are added to the agent system, the agent can repeat the following tasks instead of just answering once.

• Analyze the situation.
• Call the tool.
• Check the results.
• Decide the next action.
• If necessary, call other tools or ask people for help.

This repetitive structure makes the agent much more powerful. But loops are also a risk factor.

"Loops can be broken. Every programmer has encountered an infinite loop. It's not good."

An agent may repeat infinitely, or may deviate from its initial goal as multiple agents talk to each other. This can be seen as drift. Additionally, the longer the repetition, the greater the token usage and the greater the cost.

"Loops can be costly. As the loop continues, the number of tokens goes up."

Therefore, the agent loop requires not only a simple prompt, but also intermediate validation and termination conditions.

8. Put a verification layer in the Claude tool usage loop

Coyle demonstrates a simple agent loop example using Claude. The code contains a loop in the form of while true, and the model, prompt, and tool are connected.

The LLM itself cannot act directly in the outside world. What LLM does is predict the next token with a high probability based on the given context.

"LLM can't do anything directly. All it can do is suggest the next word with a high probability."

However, if you provide the tools, LLM can decide how to use those tools. For example, set the tool's input parameters and return a tool call of the following form:

• Decide which tool to use.
• Configure the required input parameters.
• Returns a tool call request.
• The actual tool execution is handled by external code.

LLM does not execute the tool directly, but generates the call information necessary for execution.

If the use of a tool is necessary, the termination reason for the response is displayed as tool use, and the program checks the signal and then executes the actual tool. When the tool is run, the results are sent back to LLM, and the agent decides the next steps based on the results.

Coil explains that an ontology-based verifier can be installed at this point after tool execution.

1. LLM proposes a calling tool.
2. The program runs the tool.
3. Pass the tool's results to the verifier.
4. The verifier checks the rules and constraints of the ontology.
5. If the results are reasonable, proceed to the next step.
6. If there is a problem, have LLM correct it again or ask a person to make a judgment.

"The key is to surround your input with checks."

It is a structure that does not unconditionally execute the agent's judgment, but installs a logical checkpoint between the action and the result.

9. Pydantic at the entrance, Ontology at the ledger

Coyle says Python developers need to take a look at Pydantic. Since Python is basically a language whose types are not strictly fixed, there is no major problem if you put numbers and strings in the same variable.

For example, it is possible to first enter the number 20 into one variable, and then enter the string "hello". Pydantic gives these inputs an explicit type and structure.

So before the agent calls the tool, Pydantic can check the following:

• Is the parameter in a required format?
• The value that should have been a number was not entered as a string.
• Are any required fields missing?
• Check that an unauthorized value has not been entered.

Then, the tool execution results are verified with ontology.

"Pydantic is at the entrance, ontology is at the ledger."

This expression clearly shows the difference in roles. Pydantic checks the type and format of the request, and Ontology checks whether the result is logically valid within the business domain.

Coyle also emphasizes that agents must minimize side effects. It is dangerous to change the database or execute payment before the agent is verified.

"Agents should have as few side effects as possible."

This means that decisions and tool calls must be constructed first, and only after passing ontology verification can actual data changes or external actions be allowed.

10. Errors that are difficult to prevent with only English instructions

In the final part of the talk, Coyle shows what OWL's logical constraints can actually catch. The examples he presented are problems that are difficult to prevent with only explanations or prompts, but can be relatively clearly detected with ontology.

Refunding the same order twice

If you define a rule that an order can only be refunded once, the moment a second refund is attempted for the same order, it can be treated as an error.

"Giving a second refund for the same order is a problem. Ontology can catch this, but explaining it in English is very tricky."

You can explain to an LLM that "you only get a refund once," but in real-life situations, accurately tracking order identifiers and transaction statuses and preventing duplicate requests is difficult to ensure with simple sentence instructions.

Payment to customer support representative on behalf of buyer

In some cases, it is necessary to distinguish whether the payment recipient is the buyer or a customer support representative. By defining 고객 and 고객지원 담당자 as different types of entities in the ontology, incorrect requests to support personnel can be blocked.

"Customers and customer support representatives are different entities."

This can be expressed as constraints on mutually exclusive object types in OWL, such as disjoint property or disjoint class.

Fictional states such as 'probably delivered'

LLM can sometimes create undefined states in the system. For example, the order status could be returned like this:

"Probably delivered"

However, if we limit the allowable values of order states in the ontology to the following three, the problem becomes immediately apparent:

• Payment completed
• Delivery completed
• Refund completed

Undefined values such as "아마 배송됨" are not allowed.

"The status must be one of the following: paid, delivered, or refunded. Nothing else."

In this way, ontology can logically specify list of allowable values, distinction between entities, actions that can only be done once, required relationships, etc.

11. Conclusion: Using Probabilistic Reasoning and Logical Control Together

Coyle's conclusion is not to not trust LLMs and agents. Rather, the argument is that the probabilistic and flexible reasoning capabilities of LLM should be utilized, but important actions must pass through an external logical system.

Using technologies such as Ontology, RDFS, and OWL, the following roles can be performed.

• Clearly define the entities and relationships used by the agent.
• Limit allowable values ​​and types.
• Find logical contradictions between data.
• Check whether the tool call result meets the business rules.
• If there is a problem, have the agent retry or pass it on to a human.
• Prevent changes to the database or payment system without verification.

Ultimately, the structure proposed by the lecturer can be summarized as follows.

"Keep probabilistic reasoning inside, logic outside."

Agents are allowed to think freely and suggest tool use, but only allow actual action after passing type verification and ontology verification. At the end, he again emphasizes the educational philosophy he introduced at the beginning.

"Nothing is a mistake. There is no victory. There is no failure. There is only what you make."

The speaker concludes the presentation by introducing his website and contact information, and adds that the name of his project, which deals with both code and music, was inspired by John Coltrane's jazz album 〈A Love Supreme〉. The lecture ends with applause from the audience.