AI Assistants A Framework for Semi-Automated Data Wrangling

Tomas Petricek, Gerrit J.J. van den Burg, Alfredo Nazábal, Taha Ceritli, Ernesto Jiménez-Ruiz, Christopher K. I. Williams.

IEEE Transactions on Knowledge and Data Engineering (TKDE), To appear 2022

Data wrangling tasks such as obtaining and linking data from various sources, transforming data formats, and correcting erroneous records, can constitute up to 80% of typical data engineering work. Despite the rise of machine learning and artificial intelligence, data wrangling remains a tedious and manual task. We introduce AI assistants, a class of semi-automatic interactive tools to streamline data wrangling. An AI assistant guides the analyst through a specific data wrangling task by recommending a suitable data transformation that respects the constraints obtained through interaction with the analyst.

We formally define the structure of AI assistants and describe how existing tools that treat data cleaning as an optimization problem fit the definition. We implement AI assistants for four common data wrangling tasks and make AI assistants easily accessible to data analysts in an open-source notebook environment for data science, by leveraging the common structure they follow. We evaluate our AI assistants both quantitatively and qualitatively through three example scenarios. We show that the unified and interactive design makes it easy to perform tasks that would be difficult to do manually or with a fully automatic tool.

Paper and more information

• Download the paper pre-print (PDF)
• Download an archival copy from arXiv

Bibtex

If you want to cite the paper, you can use the following BibTeX information.

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
9: 

@article{aia-tkde2022,
  author={Petricek, Tomas and van den Burg, Gerrit J.J. and Nazábal, Alfredo
    and Ceritli, Taha and Jiménez-Ruiz, Ernesto and Williams, Christopher K. I.},
  title={AI Assistants: A Framework for Semi-Automated Data Wrangling},
  volume={To appear},
  journal={IEEE Transactions on Knowledge and Data Engineering},
  publisher={IEEE},
  year={2022},
}

If you have any comments, suggestions or related ideas, I'll be happy to hear from you! Send me an email at tomas@tomasp.net or get in touch via Twitter at @tomaspetricek.