Informace o publikaci
Comparative Evaluation of Large Language Models for Data Cleaning
| Autoři | |
|---|---|
| Rok publikování | 2026 |
| Druh | Stať ve sborníku |
| Konference | Northeast Decision Sciences Institute 2026 Annual Conference |
| Citace | |
| www | NEDSI 2026 Conference Proceedings |
| Popis | Large language models are increasingly used to help with data cleaning, but it is still unclear how well they handle structured, real-world datasets. This paper compares the performance of several popular language models on the same data cleaning task using a real dataset of job postings. Each model was asked to clean the dataset under identical conditions and to produce executable Python code. The models were evaluated based on whether their code ran successfully, how closely the cleaned data matched a human-prepared reference dataset, and how they handled common data quality issues such as missing values and duplicates. While most models were able to generate working code and preserve the overall structure of the dataset, they differed in how aggressively they applied cleaning rules, especially when dealing with missing values and duplicate records. These findings show that large language models can support data cleaning tasks, but their outputs vary in important ways. As a result, careful evaluation is needed before such models are used in automated data preprocessing pipelines. |