Tabular Classification Models

Tabular classification models predict a discrete label, or a probability distribution over labels, from rows of a table. Columns may be numeric, ordinal, binary, or categorical, and each column can have a different scale and meaning. Unlike pixels or words, columns usually have no useful universal ordering. Missing values, class imbalance, duplicated entities, temporal drift, and target leakage are therefore central parts of the modeling problem rather than incidental cleanup.

Regularized logistic regression, random forests, and gradient-boosted decision trees remain important baselines. Specialized neural networks can be useful when training must be end-to-end, representations must be shared across tasks, or tabular features are combined with text or other modalities. Since 2023, pre-trained tabular foundation models, including TabPFN and TabICL, have become strong additional candidates in the data regimes for which they have been evaluated. Comparative studies do not support a universal winner: rankings depend on the datasets, tuning budget, preprocessing, metric, and resource constraints.^[1][2]

Part of the Tabular Models hub. For continuous targets, see Tabular Regression Models.

Problem definition

For single-label classification, a training set contains pairs $(x_i, y_i)$, where each row $x_i$ has $d$ features and $y_i$ belongs to one of $K$ classes. Binary classification has two classes. Multiclass classification has more than two mutually exclusive classes. Multilabel classification assigns any subset of labels to a row and normally needs a multilabel-aware loss or reduction rather than an ordinary multiclass softmax.

A useful classifier should be judged against the decision it supports. Accuracy can be adequate when classes and error costs are balanced. Precision and recall, average precision, and the F1 score expose different tradeoffs when the positive class is rare. The area under the ROC curve measures ranking across thresholds, but a precision-recall curve often communicates practical performance more clearly under strong imbalance because its baseline changes with prevalence.^[3] Log loss and Brier score assess probabilistic predictions, while a business or clinical system may ultimately require an explicit cost function and operating threshold.

The split comes before the model. A random split is unsuitable when rows from the same customer, patient, device, site, or time period can appear on both sides, or when a feature contains information that would not exist at prediction time. Preprocessing learned from all rows can also leak test information. Pipelines should fit encoders, imputers, feature selectors, and calibrators only on the appropriate training folds.^[4]

Main model families

Scroll sideways for more →

Classical baselines

Among linear models, logistic regression models class log-odds as a linear function of the inputs. Regularization and a well-defined feature pipeline make it a strong reference for both binary and multiclass problems.^[5] It is especially useful when inference must be cheap, the feature count is large, or a nonlinear model must justify its added complexity. Calibration and discrimination still need to be measured on held-out data.

Decision trees partition the feature space with a sequence of rules. One tree is easy to inspect but unstable. Bagging reduces variance by averaging trees trained on resampled data. Leo Breiman's random forest adds random feature selection at each split and supplies an internal out-of-bag estimate.^[6] Random forests are useful baselines when interactions matter and training time is limited, although modern boosting often achieves better predictive scores after tuning.

Naive Bayes, discriminant analysis, nearest neighbors, and support vector machines remain worth testing when their assumptions match the data. A sparse text-derived table may favor a linear model or naive Bayes. A small, scaled dataset with a meaningful distance can favor nearest neighbors. A high-dimensional small-sample problem may favor a regularized linear classifier or an SVM. Dataset size alone does not determine the answer.

Gradient-boosted decision trees

Gradient boosting builds an additive model by fitting each new learner to improve a differentiable objective. Friedman's formulation connected boosting with stage-wise optimization and remains the foundation for modern GBDT systems.^[7]

XGBoost, presented by Tianqi Chen and Carlos Guestrin at KDD 2016, combines second-order optimization with a regularized tree objective, a sparsity-aware split algorithm, weighted quantile sketching, and systems work on cache access, compression, and sharding.^[8] Its competition history should be stated with a date: the paper examined public descriptions of 29 winning Kaggle solutions from 2015 and found that 17 used XGBoost, with eight using it as the sole model. That is evidence of substantial adoption at the time, not proof that one library wins most present-day competitions.

LightGBM, developed at Microsoft, introduced Gradient-based One-Side Sampling and Exclusive Feature Bundling. GOSS retains examples with large gradients and samples those with smaller gradients; EFB combines sparse features that are rarely nonzero at the same time. The authors reported large training-speed gains over conventional GBDT in their experiments while maintaining similar accuracy.^[9]

CatBoost, developed at Yandex, was designed around ordered boosting and categorical-feature statistics. Both techniques address a form of target leakage and prediction shift that can arise when the target is used to construct categorical encodings or boosting residuals on the same observations.^[10] This makes CatBoost a natural baseline for tables with many categorical columns, but it does not remove the need for honest validation or schema control.

Scikit-learn includes HistGradientBoostingClassifier, a histogram-based implementation inspired by LightGBM. Current documentation describes native missing-value and categorical-feature support, subject to its documented category and bin limits.^[11] XGBoost also has supported interfaces for categorical splits, but capabilities and model-IO requirements have changed across releases. Its documentation explicitly distinguishes one-hot and partition-based splits and warns that input category encodings must remain consistent.^[12]

These libraries share the boosting idea but are not interchangeable. Their tree growth, categorical algorithms, missing-value policies, GPU paths, objectives, and serialization formats differ. A production comparison should pin software versions and measure at least predictive quality, fit time, peak memory, batch and single-row latency, artifact size, and behavior on unseen categories.

Neural approaches

General tabular deep learning includes ordinary MLPs, residual networks, attention models, differentiable trees, and retrieval-based methods. A neural model is often most compelling when its embeddings must be learned jointly with another component, when millions of rows make GPU training efficient, or when the table contains inputs already represented by a neural encoder.

TabNet, developed by Sercan Arik and Tomas Pfister at Google, applies sequential attention to choose features at each decision step. Its paper also studied self-supervised pre-training by reconstructing masked features.^[13] NODE uses differentiable oblivious trees. TabTransformer, developed by researchers at Amazon, contextualizes categorical embeddings with transformer layers. SAINT attends across both features and rows and studies contrastive pre-training.^[14]

The NeurIPS 2021 study Revisiting Deep Learning Models for Tabular Data established two particularly useful neural baselines: a ResNet-like MLP and FT-Transformer. FT-Transformer tokenizes numerical and categorical features and processes them with a standard transformer. Under the study's common protocol, it was the strongest neural model on most tasks, but the authors found no universally superior solution between deep models and GBDT.^[15]

TabM, published at ICLR 2025, returns to the MLP. It trains several implicit MLPs together while sharing most parameters, then averages their predictions as an ensemble. The authors' public-benchmark evaluation found a favorable accuracy-efficiency tradeoff relative to more elaborate tabular neural architectures.^[16] This supports using TabM or a well-regularized ResNet as a neural baseline before assuming that attention is necessary.

Models from recommender systems, such as factorization machines, Wide and Deep, DeepFM, and cross networks, are also tabular classifiers. They are particularly relevant to high-cardinality sparse fields and click-through prediction, but results from that setting should not automatically be generalized to small mixed-type business tables.

Tabular foundation models

Tabular foundation models amortize learning across many pre-training tasks. At inference, an in-context model receives labeled training rows as context and predicts labels for query rows without ordinary gradient training on the new table. This resembles in-context learning in GPT, but the data representation, objective, and scale constraints are specific to tables.

The original TabPFN, published at ICLR 2023, was pre-trained on synthetic classification tasks to approximate Bayesian posterior prediction under a chosen prior. It targeted small tables and produced predictions in a forward pass without per-dataset hyperparameter tuning.^[17]

The substantially expanded TabPFN commonly called v2 was published in Nature in 2025. The peer-reviewed evaluation covered 29 classification and 28 regression datasets with at most 10,000 samples, 500 features, and 10 classes. It added regression, categorical data, and missing-value support and reported strong out-of-the-box results against tuned tree baselines in that regime.^[18] The same paper lists important limitations: inference can be slower than optimized CatBoost, memory grows linearly with table size, and scaling beyond the evaluated regime needed further study.

TabICL, published at ICML 2025, uses a two-stage column-then-row architecture. It was pre-trained on synthetic datasets with up to 60,000 samples and was reported capable of handling 500,000 samples. Across 200 TALENT classification datasets, the authors reported performance on par with TabPFN v2 while running up to ten times faster; on the 53 datasets above 10,000 samples, it surpassed both TabPFN v2 and CatBoost.^[19]

The field changed again after those peer-reviewed papers:

Scroll sideways for more →

The last three rows should not be read as a single neutral leaderboard. They use different versions, task collections, hardware, tuning budgets, and product paths. Some features are available through local weights, some through an API, and access or license terms can differ. Before deployment, verify the exact checkpoint, package, license, data-governance path, supported class count, and memory behavior.

What comparative studies show

The statement that trees always beat neural networks is too broad, as is the reverse. Two benchmark studies provide a more useful picture.

Grinsztajn, Oyallon, and Varoquaux compared tree models and neural models on 45 datasets, with a large common hyperparameter search. On the study's medium-sized tables, around 10,000 samples, tree models remained strongest even before accounting for their speed. The authors identified three challenges for tabular neural networks: resisting uninformative features, preserving the meaningful orientation of the original feature axes, and learning irregular functions.^[1] This is not a claim that trees are robust to arbitrary rotation. Axis-aligned trees benefit from the original column orientation and generally lose that advantage when features are rotated.

The TabZilla study compared 19 algorithms across 176 datasets. It found that on many datasets the neural-versus-GBDT difference was negligible, or that light GBDT tuning mattered more than choosing between those families. GBDT performed particularly well on skewed, heavy-tailed, or otherwise irregular features. The released TabZilla benchmark is the 36 hardest datasets from the larger analysis, not a 176-dataset or 196-dataset suite.^[2]

TabPFN v2's Nature results add a third point: a pre-trained in-context model can be exceptionally competitive on the small and medium tables covered by its evaluation.^[18] TabICL extends the peer-reviewed evidence to larger classification tables.^[19] The ICML 2026 TabICLv2 study and the 2026 TabPFN-3 technical report suggest the scale frontier is moving quickly, but they do not erase the need to test a tuned boosting baseline.

The practical conclusion is to compare families under the same split, preprocessing boundary, metric, and resource budget. A small average rank difference across public datasets may be irrelevant if one candidate violates a latency, memory, licensing, or calibration requirement.

Data preparation and evaluation

Validation design

Choose the test split to simulate deployment:

• Use a temporal split when predicting future events.
• Keep all rows for an entity or site in one side of a grouped split.
• Fit preprocessing only inside each training fold.
• Reserve a final test set that is not used for feature selection, threshold choice, or hyperparameter search.
• Report uncertainty across folds, time windows, sites, or seeds when those sources of variation matter.

Cross-validation is an estimation procedure, not a cure for a mismatched split. Random stratification can preserve class ratios while still leaking identity or future information.

Categorical and missing values

One-hot encoding is a transparent default for low-cardinality nominal features and works well with linear models. It can become wide for high-cardinality columns. Integer or ordinal encoding is appropriate when categories have a real order. Giving unordered categories arbitrary integer codes and then applying ordinary numerical threshold splits imposes an artificial order and can change the result when the codes are permuted.

Target encoding replaces a category with a statistic derived from the target. It can be effective for high-cardinality features, but the training rows must receive out-of-fold or otherwise leakage-controlled encodings. Scikit-learn's TargetEncoder uses internal cross-fitting for this reason, and its documentation warns that a direct fit-then-transform on the same training data can leak target information.^[24] CatBoost's ordered categorical statistics are another algorithm-specific response to the same problem.^[10]

Neural networks usually learn embeddings for categorical values. Native categorical support in tree libraries uses different algorithms and input contracts, so "native" does not mean identical behavior. Unknown categories, category type changes, missing categories, and serialization should be tested explicitly.

Missingness can itself be predictive. Tree libraries may learn a direction or category for missing values, while linear and neural models often use imputation plus a missingness indicator. In either case, the imputer and indicator policy belong inside the validation pipeline. A value that is missing because it was not collected can behave differently from one suppressed by a business rule.

Imbalance, thresholds, and calibration

Class imbalance should first change the evaluation design. Select metrics and thresholds from the costs of false positives and false negatives. Compare class weighting, threshold adjustment, and resampling rather than assuming one is required. If an oversampling method such as SMOTE is used, generate synthetic observations only within a training fold, never before the split.^[25]

Calibration matters when a score is interpreted as a probability or used in a cost calculation. Inspect reliability curves as well as log loss or Brier score. Post-hoc sigmoid, isotonic, or temperature calibration should be fit on data not used to train the base model. Scikit-learn's calibration guide uses cross-validation for this separation and notes that isotonic calibration can overfit on small calibration sets.^[26]

Feature engineering and interpretation

Domain features often matter more than a small algorithmic difference. Useful examples include elapsed time since an event, counts over a past-only window, ratios with a defined denominator, and aggregates over related records. Every feature must be reproducible at prediction time. "Days until closure" is leakage if closure is unknown when the prediction is made.

SHAP provides additive feature attributions for a fitted model and can support local inspection.^[27] Permutation importance can estimate how a feature affects held-out predictive performance. Neither method proves that a feature causes the outcome, and correlated or interchangeable features can divide importance in misleading ways. Interpretation should be checked against domain knowledge, data provenance, and alternative model specifications.

AutoML, benchmarks, and tooling

Auto-sklearn combines Bayesian optimization over scikit-learn pipelines with meta-learning and ensembles.^[28] AutoGluon, developed at Amazon Web Services, emphasizes multi-layer stacking of diverse tabular models.^[29] H2O AutoML trains several algorithm families and stacked ensembles.^[30] FLAML, developed at Microsoft Research, targets accurate configurations under low computational budgets.^[31]

AutoML is useful when the human tuning budget is small or when stacking is acceptable. It is not automatically fairer, simpler, or cheaper to serve. Compare the selected ensemble with its best compact member, and include preprocessing, fit failures, model size, and inference cost in the evaluation.

The AMLB study illustrates why a single AutoML ranking is fragile. It compared nine frameworks across 71 classification and 33 regression tasks and analyzed accuracy, inference time, failures, and task-dependent changes in relative rank.^[32] It does not support the baseline article's claim that one pair of systems is generally first or that AutoML usually adds one to three accuracy points over a tuned GBDT.

OpenML provides fixed tasks and splits for reproducible comparisons. The OpenML-CC18 suite contains 72 curated classification tasks selected under explicit size, imbalance, leakage, and difficulty criteria.^[33] Other commonly used collections include UCI datasets, Grinsztajn's 45-task benchmark, the 176 datasets analyzed by TabZilla, TALENT, and task-specific Kaggle competitions. Results should name the exact suite and version rather than treating "tabular benchmarks" as one dataset.

The mature open-source stack includes scikit-learn for pipelines and classical models, XGBoost, LightGBM, and CatBoost for boosting, PyTorch-based packages for neural models, and AutoGluon, H2O, auto-sklearn, and FLAML for automation. A reproducible report should record package versions, hardware, random seeds, preprocessing, search spaces, time limits, and failed trials.

Practical starting points

Scroll sideways for more →

This table is a test order, not a ranking. The correct final model is the simplest candidate that meets the predictive, operational, legal, and monitoring requirements on deployment-like data.

Limitations and deployment checks

Public benchmarks usually approximate independent and identically distributed prediction. Production tables often contain temporal drift, policy changes, delayed labels, correlated entities, and feedback loops. Re-evaluate by time and relevant subgroups, monitor both inputs and outcomes, and define retraining or rollback criteria before launch.

High predictive accuracy does not establish fairness, safety, or causal validity. In credit, hiring, health, and other consequential settings, evaluate error rates, calibration, and decision thresholds across legally and operationally relevant groups. A feature can act as a proxy even when a protected attribute is removed.

Foundation-model comparisons are especially time-sensitive. The supported table size can depend on row-feature tradeoffs, GPU memory, local versus hosted inference, and optional ensembling. Vendor or author benchmarks may use unreleased checkpoints or services that differ from downloadable weights. Confirm data residency, license terms, reproducibility, and exit options before making the model part of a critical pipeline.

Finally, all classifier families can fail silently under schema changes. Validate column types, category vocabularies, missing-value conventions, units, and feature availability at ingestion. Store the preprocessing graph with the model, test it on representative edge cases, and monitor the deployed probability distribution rather than only hard labels.

See also

• Tabular Models
• Tabular Regression Models
• Machine Learning
• Graph Machine Learning Models
• Decision Trees
• Gradient Boosting
• Random Forest
• Logistic Regression
• XGBoost
• LightGBM
• CatBoost
• TabNet
• TabPFN
• Auto-sklearn
• AutoGluon
• Scikit-learn
• Cross-validation
• Calibration
• Interpretability

References

1. ^Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). "Why do tree-based models still outperform deep learning on typical tabular data?" *NeurIPS 2022 Datasets and Benchmarks*. proceedings.neurips.cc/...-Datasets_and_Benchmarks
2. ^McElfresh, D., et al. (2023). "When Do Neural Nets Outperform Boosted Trees on Tabular Data?" *NeurIPS 2023 Datasets and Benchmarks*. proceedings.neurips.cc/...-Datasets_and_Benchmarks
3. ^Saito, T., and Rehmsmeier, M. (2015). "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets." *PLOS ONE*, 10(3), e0118432. doi.org/...journal.pone.0118432
4. ^Kaufman, S., Rosset, S., Perlich, C., and Stitelman, O. (2012). "Leakage in data mining: Formulation, detection, and avoidance." *ACM Transactions on Knowledge Discovery from Data*, 6(4), 15. doi.org/...2382577.2382579
5. ^Cox, D. R. (1958). "The Regression Analysis of Binary Sequences." *Journal of the Royal Statistical Society: Series B*, 20(2), 215-242. doi.org/...j.2517-6161.1958.tb00292.x
6. ^Breiman, L. (2001). "Random Forests." *Machine Learning*, 45, 5-32. doi.org/...A:1010933404324
7. ^Friedman, J. H. (2001). "Greedy Function Approximation: A Gradient Boosting Machine." *The Annals of Statistics*, 29(5), 1189-1232. doi.org/...1013203451
8. ^Chen, T., and Guestrin, C. (2016). "XGBoost: A Scalable Tree Boosting System." *KDD 2016*. doi.org/...2939672.2939785
9. ^Ke, G., et al. (2017). "LightGBM: A Highly Efficient Gradient Boosting Decision Tree." *NeurIPS 2017*. proceedings.neurips.cc/...669bdd9eb6b76fa-Abstract
10. ^Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. (2018). "CatBoost: unbiased boosting with categorical features." *NeurIPS 2018*. proceedings.neurips.cc/...c41c24863285549-Abstract
11. ^Scikit-learn developers. "HistGradientBoostingClassifier." Accessed July 28, 2026. scikit-learn.org/...HistGradientBoostingClassifier
12. ^XGBoost developers. "Categorical Data." Accessed July 28, 2026. xgboost.readthedocs.io/...categorical
13. ^Arik, S. O., and Pfister, T. (2021). "TabNet: Attentive Interpretable Tabular Learning." *AAAI 2021*. doi.org/...aaai.v35i8.16826
14. ^Somepalli, G., Schwarzschild, A., Goldblum, M., Bruss, C. B., and Goldstein, T. (2022). "SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training." openreview.net/forum
15. ^Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. (2021). "Revisiting Deep Learning Models for Tabular Data." *NeurIPS 2021*. proceedings.neurips.cc/...9edb0ac3b49229c-Abstract
16. ^Gorishniy, Y., Kotelnikov, A., and Babenko, A. (2025). "TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling." *ICLR 2025*. openreview.net/forum
17. ^Hollmann, N., Muller, S., Eggensperger, K., and Hutter, F. (2023). "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second." *ICLR 2023*. openreview.net/forum
18. ^Hollmann, N., et al. (2025). "Accurate predictions on small data with a tabular foundation model." *Nature*, 637, 319-326. nature.com/...s41586-024-08328-6
19. ^Qu, J., Holzmueller, D., Varoquaux, G., and Le Morvan, M. (2025). "TabICL: A Tabular Foundation Model for In-Context Learning on Large Data." *ICML 2025*. proceedings.mlr.press/...qu25d
20. ^Grinsztajn, L., et al. (2025, revised 2026). "TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models." Technical report. arxiv.org/...2511.08667
21. ^Qu, J., Holzmueller, D., Varoquaux, G., and Le Morvan, M. (2026). "TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model." *ICML 2026*, PMLR 306. openreview.net/...061658799e910197567fa39698b6.pdf
22. ^Grinsztajn, L., et al. (2026). "TabPFN-3: Technical Report." arxiv.org/...2605.13986
23. ^Prior Labs. "TabPFN-3." Accessed July 28, 2026. docs.priorlabs.ai/...tabpfn-3
24. ^Scikit-learn developers. "TargetEncoder." Accessed July 28, 2026. scikit-learn.org/...rn.preprocessing.TargetEncoder
25. ^Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002). "SMOTE: Synthetic Minority Over-sampling Technique." *Journal of Artificial Intelligence Research*, 16, 321-357. doi.org/...jair.953
26. ^Scikit-learn developers. "Probability calibration." Accessed July 28, 2026. scikit-learn.org/...calibration
27. ^Lundberg, S. M., and Lee, S.-I. (2017). "A Unified Approach to Interpreting Model Predictions." *NeurIPS 2017*. proceedings.neurips.cc/...6c43dfd28b67767-Abstract
28. ^Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. (2015). "Efficient and Robust Automated Machine Learning." *NeurIPS 2015*. proceedings.neurips.cc/...3f79975ec59a3a6-Abstract
29. ^Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. (2020). "AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data." arxiv.org/...2003.06505
30. ^H2O.ai. "H2O AutoML: Automatic machine learning." Accessed July 28, 2026. docs.h2o.ai/...automl
31. ^Wang, C., Wu, Q., Weimer, M., and Zhu, E. (2021). "FLAML: A Fast and Lightweight AutoML Library." *MLSys 2021*. proceedings.mlsys.org/...17068778f3c4523a-Abstract
32. ^Gijsbers, P., et al. (2024). "AMLB: an AutoML Benchmark." *Journal of Machine Learning Research*, 25(101), 1-65. jmlr.org/...22-0493
33. ^Bischl, B., et al. (2021). "OpenML Benchmarking Suites." *NeurIPS 2021 Datasets and Benchmarks*. arxiv.org/...1708.03731