This summary details how Cerebras built its in-house knowledge base. In just three months, the system has become one of the most widely used tools within the company, with employees answering more than 15,000 questions every day to address information dispersion. In particular, we focused on providing accurate and useful information by utilizing a hybrid search approach and a reranking technique that integrates various data sources while minimizing changes to existing behavior and efficiently handles unstructured data such as Slack conversations and code repositories.

1. Background of building an in-house knowledge base 💬

Cerebras is a company active in various fields, including data center operations, chip design, hardware, education, inference, and cloud platforms. With hundreds of new employees joining each year, questions like "Where can I find X?", "Who are the experts in Y?", and "What is Z?" The same questions started flooding our internal communication channels. 😮 To solve these problems, we built a knowledge base called Cerebras Knowledge. The goal of this system was to help people easily connect to information and systems.

2. Integration of distributed data 🌐

Finding information is a really difficult task inside an organization. Because your data is spread across multiple tools. Proposals like "let's record all information on one platform!" come up periodically, but in reality, the dream of a "single source of truth" is rarely realized. This is because information is created on platforms optimized for each purpose, such as Slack threads, GitHub code, and Jira state metadata. Taking this into account, Cerebras focused on designing a system that minimizes changes to existing behavior. In other words, we chose to extract data directly from each platform.

3. Structure of the knowledge base 🏗️

Cerebras Knowledge provides three main functions:

1. Platform that collects and stores internal data
2. Platform to query data
3. Layer that enforces authentication and authorization and provides auditing and analytics

The core of the system is a single Postgres table containing embeddings, raw summaries, and metadata from various sources. This table continuously collects data from across the company and maintains a queryable data store.

The data interface is designed to be simple and work with most data formats. We also make it easy for other developers to create custom connectors. As a result, all sources (from Slack threads to netlists) are stored in the same embedding table, and all data in this table can be instantly queried through the same interface.

Each data source defines the type of data, how to connect to it, and how often the data should be retrieved. These generated embedding lines follow the same interface whether they come from Slack, a code repository, a document system, or a custom database.

4. How Slack data is processed 🗣️

Slack was one of our most important data sources. This is because it is where the most active and up-to-date engineering discussions take place within the company.

4.1. Unstructured Slack conversation processing process 📝

At first we tested whether simply embedding the raw text would be sufficient, but we soon realized that it would be difficult to match all relevant data using vector searches alone. There were the following problems with Slack messages:

• Information density varies greatly. The message is both "Okay, okay Mike" and a detailed kernel description.
• Message lengths varied, and short messages often ranked higher in cosine similarity than long messages.
• The meaning of the message often varied depending on the surrounding conversation.

So I decided a hybrid approach was needed. We built the Slack collection system so that all threads can be searched simultaneously using multiple search techniques, each of which compensates for the weaknesses of the other.

• Full-text search: Finds exact tokens (error strings, flag names, host names) that the embedding can blur. When engineers paste literal error messages, an exact lexical match is almost always the best evidence.
• Embedding search: Find paraphrases. A person asking "Restore stops after manifest load" and someone answering "Checkpoint stops on NFS mount" may not use the same vocabulary. Vector similarity serves to connect questions and answers written in different words.
• Inverse document frequency (IDF): Separates signals from unnecessary information. Short messages consisting of rare tokens (e.g. little-known configuration flags) should be ranked higher. "Okay, thank you!" is close to many queries in the embedding space, but when term rarity is taken into account, the score becomes almost 0.
• Age decay: Encodes that Slack answers expire over time. Two threads may answer the same question, but the thread from 6 months ago may describe infrastructure that no longer exists. If they are of equal relevance, newer threads will rank higher.

No single score can be trusted on its own. Each technique produces its own ranked view of the same corpus, and these views are merged at query time.

4.2. Real-time data collection using Socket Mode ⚡

To collect data in real time, I installed a Slack bot in my workspace and ran it in Socket Mode. Slack pushed all message events to Cerebras over a persistent WebSocket, allowing us to get real-time updates without polling the API and hitting rate limits.

When events arrive, they are immediately acknowledged, duplicates are removed using stable event IDs, and messages are marked for processing by the ingestion consumer.

Collection consumers do not store new messages alone. Instead, it determines which thread the message belongs to and pulls back the entire conversation from the Slack API, including the parent message and all replies. And we rewrite the entire thread as one line. Therefore, replies to existing threads bring back parent and all sibling messages, ensuring that saved content, participant list, and last activity timestamp always reflect the complete conversation.

Every Slack channel in the system has its own data source. This allows fine-grained control over data freshness. For example, a team may choose to collect frequently used incident channels more frequently.

4.3. Threads and message handling 🔄

Raw Slack text maintains a Postgres full-text (GIN) index of the content, making it keyword searchable as soon as it arrives. However, some additional processing was required to enable useful vector searches.

During the distillation process, LLM extracts structured data from the entire thread.

• One-line questions that engineers will actually search for**
• BRIEF SUMMARY
• Solution (resolution)
• Mentioned System and Code Reference

Embed these data points and write them to a shared embedding table. The original conversation log is not directly embedded. Cerebras' experiments show that accuracy improves significantly when threads are normalized into a consistent format. Additional metadata provides more useful signals for semantic matching.

4.4. Bursting 💥

Slack search was fine at this point, but I was still running into the problem that important messages within long threads weren't always well represented in thread-level summaries.

Uses Bursting to strengthen the signal of individual messages. A burst is a set of messages sent consecutively from the same author. Embed individual bursts by adding the thread topic as context. Because sometimes the answer may be one or two stray messages that aren't included in the thread summary. Burst embedding allows you to find that message alone.

To prevent low-signal data from reaching the database, each burst is scored based on a weighted combination of signals and must pass a certain threshold before being embedded.

• Contains tokens that are relatively rare across the corpus and must have an IDF of 4.0 or higher.
• The combined burst must be at least 200 characters long.
• One or more messages within a burst contain reactions, providing a social boost.

After distillation, the qualified bursts are embedded and stored in the embedding table along with their thread-level history.

5. Code repository handling 💻

At first we discussed whether it was necessary to embed a code repository. With the advent of Claude Code and other command line tools, it felt counterintuitive to create code embeds when you thought "grep is enough." But after talking to others in the industry and reading Cursor's research on semantic search in large codebases, I decided to give it a try.

Cerebras had a lot of large internal storage, over 40GB, and keeping it up to date efficiently was a major concern.

5.1. Maintaining code embeddings with CocoIndex 🚀

After many experiments, I ended up using CocoIndex, an open source document embedding framework specialized in codebase vectorization.

For each repository, we split the code using language-specific regular expression boundaries. This splitter first tries high-level boundaries such as classes, method boundaries if the resulting chunk is still too large, and then divides it into smaller blocks. Embed the resulting chunk and write the vector to Postgres. A single file can generate multiple embeddings at different levels of detail, such as file-level and function-level history.

CocoIndex tracks synchronization metadata in Postgres. Instead of recalculating the entire repository for each commit, only the chunks of code that have changed are re-embedded and exported. This was especially effective because the sync state and embedding repository were in the same database.

As the number of codebases grew, we moved repository onboarding to configuration files that teams could submit directly, including allowlists and denylists at the file path level.

6. Custom data sources 🗄️

Some teams already had their own databases and didn't want to move their data to Slack or a document system to participate in the knowledge base. They wanted the same query interface for existing tables.

To support this, we've treated custom sources as plugin scripts. The team opens a pull request with a small Python module that reads from their system, exports rows in something like an embedding table, and adds matching data source items.

As long as the script writes to the shared database using the same schema as all other embedding rows, the rest of the stack will work without change. The data becomes queryable alongside Slack, code, and documents, with no special processing required anywhere else in the system.

7. Make a plan and distribute tools 🗺️

For every query, LLM first goes through a short planning phase to determine which tools and data sources will be important. The main tools are as follows:

• subsystem_index: LLM summary by file
• search: A unified vector pipeline that integrates and internally reranks Slack, wikis, code, and other indexed sources.
• search_slack: Direct Slack search
• search_code: ripgrep for source repository.
• recent_prs: Latest pull request related to your question
• who_knows: A person who has demonstrated expertise in a specific topic

The planner works based on a concise description of the indexed content. This includes what projects exist, what sources are available for each project, and what questions each source is useful for answering. Considering the user's query and the current scope, the planner emits a selection of tools that the executor distributes in parallel, executes, normalizes into a common proof format, and then passes to the final synthetic LLM.

8. Reranking ⚖️

A document may appear at the top simply because it answers a different question but shares vocabulary with the query. So before re-ranking, we combine the searchers' incompatible result lists with Reciprocal Rank Fusion (RRF). For each document, we add 가중치 / (60 + 순위) to each list that appears, with a default weight of 1.0 and a smoothing constant of 60.

The smoothing constant makes consensus more important than a single strong vote. A document that ranks highly in multiple browsers may receive a higher score than a document that ranks first in a single browser. We then merge the duplicate chunks back into one source, limiting the number of results each file can contribute to get a more diverse top 20 results.

We send the original query and these candidates to a small reranker model. This model gives each document a score from 0 to 10 and keeps the top 10.

Once the rankings are finalized, we add context back to the winners. For example, matching wiki sections brings together two adjacent sections, ensuring that titles, prerequisites, and notes that would have otherwise been lost due to chunking are not lost. This gives readers a complete snippet instead of a lonely paragraph missing important context.

So the result of the search is a rich packet of evidence. That is, results that are fused from different searchers, deduplicated at the source level, reranked against the actual question, and then expanded with surrounding context.

9. MCP (Micro Code Platform) 🛠️

The MCP integration exposes the search component as a direct tool rather than hiding it behind a single "answer this question" endpoint. These tools are intentionally simple and do not rely on LLM as much as possible, allowing clients to query quickly and inexpensively.

Each MCP tool corresponds to one basic search primitive, such as search_slack, search_code, search, or who_knows. The tool's inputs and outputs are bounded, structured, and stable so that they can be easily invoked by any client or agent without embedding additional orchestration logic within the tool itself.

Most tools run one query pipeline, such as vector search, vocabulary search, or ripgrep, apply a lightweight scoring heuristic, and return raw evidence rows.

A Claude Code or MCP compatible agent becomes the orchestration engine. You decide which tools to call, in what order, and how to combine the results into a final answer or code edit. The search layer itself does not rely on these LLM decisions to process requests.

10. Web UI 💻

In the web UI, the same tools exist, but are connected to a complete query pipeline that runs from start to finish for every user question. The UI agent owns the planner and executor stages.

• Planner: Lightweight LLM pass examines queries and active projects, then selects which search tool to invoke: search, search_slack, subsystem_index.
• Executor: The system distributes these tool calls in parallel, collects the results, and normalizes them into a shared evidence schema containing score, recency, and source hints.
• Synthesis: The final LLM pass takes the bundle of typed evidence and the original questions and produces answers that will be displayed in the UI. This includes citations, caveats, and cross-source synthesis.

From a user perspective, the web UI appears to simply "ask a question and get an answer," but internally it executes the same Planner → Executor → Composer pattern that MCP clients can explicitly reproduce.

11. Get organized 📊

As corpora grow, "searching everything, anywhere" quickly becomes useless. Compiler team engineers didn't want infrastructure runbooks included in search results, and vice versa. So Projects have become the main way to make search relevant by default.

11.1. Project and Scoping Search 🔎

Cerebras introduced projects as the primary way to organize the workspace where queries are run. A project is a named bundle of data sources, such as a specific Slack channel, code repository, internal database, and document space associated with a team or initiative.

The project was intentionally designed to be lightweight. The same data source, such as a shared incident channel or central platform repository, can be referenced in multiple projects without duplication.

11.2. Onboarding and setting defaults 🤝

During onboarding, users are guided to select or create a default project that matches the way they work, such as ML training infrastructure, compilers, and data center operations.

This default project is stored in your user profile and automatically scopes your queries. New engineers can get high-signal answers without having to first learn which Slack channels, repositories, or document spaces are important.

12. Closing ✨

In conclusion, Cerebras' knowledge base works by meeting people where the information already exists, rather than forcing everything into one rigid system. By combining various search techniques, you can quickly find evidence. The result is a search experience that is flexible enough for real company data, yet structured enough to remain useful as Cerebras continues to grow.