This article introduces Google's newly announced foundation model called TabFM. TabFM dramatically simplifies table data classification and regression tasks, enabling high-accuracy predictions without the complex manual work and hyperparameter tuning required by existing models. This model is expected to solve many of the difficulties experienced by data scientists, especially by utilizing zero-shot prediction and in-context learning (ICL) methods.

1. A new approach to table data modeling 🚀

Just as there have been major changes in the field of time series forecasting since the launch of TimesFM, Google is now introducing a new foundation model called TabFM, applying the same "zero-shot" logic to table data. This model aims to simplify classification and regression workflows for tabular data.

Tabular data forms the core of enterprise data infrastructure and powers many of the most important predictive machine learning applications, such as predicting customer churn or identifying financial fraud. For a long time, supervised learning-based tree algorithms such as AdaBoost, XGBoost, and Random Forest have been mainstream, showing strong performance on such structured data.

However, deploying these traditional models presented significant bottlenecks. Applying an XGBoost model to a new dataset was a complex process beyond simply going through the .fit() step. Data scientists had to spend countless hours performing extensive hyperparameter optimization and domain-specific feature engineering to extract reliable signals from the raw data. 🤯

On the other hand, recent developments in the field of machine learning, especially the evolution of Large-Scale Language Models (LLMs), have changed the way we interact with new tasks. LLM has demonstrated remarkable zero-shot prediction capabilities through in-context learning (ICL). This technique allows a pre-trained model to learn new tasks by providing examples and instructions in the input context without updating the underlying model weights.

And finally, TabFM, a foundation model specifically designed for table data classification and regression, has been released! TabFM reframes table predictions as an ICL problem, eliminating the need for manual model training, hyperparameter tuning, and complex feature engineering. This approach allows users to generate high-quality predictions in just one forward pass, even for tables they have never seen before. TabFM is currently available in the Hugging Face and GitHub repositories. 🚀

2. How TabFM works 🧐

Existing machine learning paradigms rely on updating model parameters to match the distribution of a given dataset. In contrast, the ICL paradigm completely skips this process. Rather than going through traditional training steps for each new task, TabFM treats the entire dataset—including both historical training examples and target test rows—as a single, unified prompt. The model learns to interpret the relationships between columns and rows directly from this context during inference.

However, applying ICL to table data is not as simple as tokenizing natural language. Standard language models deal with one-dimensional, ordered sequences, but tables are fundamentally two-dimensional and inherently unordered. That is, swapping two rows or two columns does not change the fundamental meaning of the data.

To effectively handle these diverse table structures while enabling scalable zero-shot predictions, TabFM has integrated the strengths of architectures such as TabPFN and TabICL into a new hybrid design. This architecture, visualized below, relies on three core mechanisms:

• Cross row and column attention: First, the original table is processed through a multi-layer attention module. Similar to TabPFN, this step applies cross-attention across both columns (features) and rows (examples). By continuously attending to these two dimensions, the model learns a rich representation that naturally captures complex feature interactions and dependencies. This deep contextualization effectively frees data scientists from the tedious task of manually creating complex features.
• Row Compression: After this contextualization, the rich, cross-attentioned information for each individual row is compressed into a single, dense vector representation.
• In-Context Learning (ICL): Finally, a dedicated transformer operates on the compressed embedding sequence. We adopt TabICL's highly efficient approach to dramatically reduce computational cost by performing attention on these compressed row vectors rather than the original uncompressed grid. This allows the prediction step to remain highly efficient even with much larger datasets.

3. Training on large synthetic datasets 📊

A common way to build a foundation model is to train a high-capacity neural network on massive amounts of diverse data. However, a major obstacle to tabular machine learning is that high-quality, diverse, and large-scale tabular datasets—especially the massive tables needed to reflect real-world industrial data analysis—are extremely lacking in the open source space. Industrial tables often contain proprietary schemas and sensitive information, making extensive pre-training difficult to access. 😥

Because synthetic tables can be generated arbitrarily large, they are effectively the only viable option for pretraining foundation models at these scales. As a result, TabFM is trained entirely on hundreds of millions of synthetic datasets. These datasets are dynamically generated using a structured causal model (SCM) that incorporates a variety of random functions. This massive synthetic generation captures the diverse distributions and complex feature relationships prevalent in real tabular data. As a result, the model generalizes well even to previously unseen real-world tables, as shown in the benchmark below. 👍

4. Excellent performance and benchmarking results 🏆

To rigorously test TabFM against existing state-of-the-art methods, Google evaluated TabFM on a real-time benchmark system called TabArena. TabArena calculates your Elo score based on head-to-head win percentage. This comprehensive evaluation includes 38 classification datasets and 13 regression datasets of varying sizes ranging from 700 to 150,000 items.

As you can see in the performance graph, Google performed benchmarks on two configurations of the model:

• TabFM: This represents the out-of-the-box functionality of the model. Predictions are generated in just one forward pass, with no tuning or cross-validation required.
• TabFM-Ensemble: This configuration further improves performance by integrating intersection features and Singular Value Decomposition (SVD) features. Compute optimal weights for a 32-way ensemble using a nonnegative least squares solver. For classification tasks, this variant also includes Platt scaling as an additional correction step.

More detailed information about the TabArena benchmark results—including detailed metrics for each fold and head-to-head win ratios for specific baseline models—can be found on Google's GitHub page. 📈

Conclusion 🎉

By reformulating table prediction into an in-context learning problem, TabFM leverages a hybrid attention architecture and massive synthetic training data to naturally capture complex feature interactions. This approach successfully eliminates the traditional bottlenecks of manual feature engineering, hyperparameter optimization, and iterative model training, and consistently outperforms highly tuned industry-standard supervised learning algorithms. TabFM brings the out-of-the-box convenience of modern foundation models directly to table ML workflows, enabling practitioners to generate highly accurate predictions in just one forward pass.

TabFM will be integrated directly into Google BigQuery for increased accessibility. In the coming weeks, users will be able to perform advanced regression and classification in BigQuery using a simple AI.PREDICT SQL command. No machine learning expertise required! I'm really looking forward to it! 😄

Thank you 🙏

This project is a collaboration with Erez Louidor Ilan, Taman Narayan, Shuxin Nie, Rajat Sen, Yichen Zhou, Joe Toth, Deqing Fu and Samet Oymak. We would also like to thank Kimberly Schwede for her graphic design.