Informace o publikaci
Deductive flexibility in humans and beyond: testing the tool with synthetic datasets
| Autoři | |
|---|---|
| Rok publikování | 2024 |
| Druh | Další prezentace na konferencích |
| Fakulta / Pracoviště MU | |
| Citace | |
| Popis | Our aim in this research is to compare the results of studies involving the Deductive Flexibility test (DFT) as carried out on human subjects vs synthetic datasets created with Large Linguistic Models (LLMs) in order to study the usefulness of the latter as a reliable means to test a psychometric tool and validate already gathered results. The name of the construct of deductive flexibility was coined by Urbański and Żyluk [2] in analogy to cognitive flexibility - an ability to switch between thinking about different concepts and thinking about multiple concepts simultaneously. Deductive flexibility can manifest in determining premises that imply a certain conclusion: this is the idea underlying the construction of the Deductive Flexibility Test (DFT). The phrase “can manifest” is used here because, although deductive flexibility could easily be characterised in logical terms (referring to the relation of logical entailment), its psychological operationalisation - in terms of a more expanded list of manifestations – still requires further analysis. DFT exhibits good psychometric properties. For example, in our two previous studies, we found its results to be normally distributed and the reliability of the tool varying between .72 (Cronbach’s alpha) to .82 (Guttman’s lambda2), on par with Raven’s Advanced Progressive Matrices (APM), used in parallel. Preliminary results of running DFT on LLMs (ChatGPT v. 3.5, 4, and 4o) suggest that, in general, they perform well in solving these types of tasks, achieving results ranging from 70% to 80% of correct solutions. We shall compare the results of the previous studies involving DFT with the ones involving synthetic datasets [3] designed to match the sociodemographic properties of the already existing sets of human subjects. Synthetic datasets, in our case created using ChatGPT 4o, promise to achieve more representative and diverse groups of participants than those recruited using traditional methods. These make them an interesting option for the aforementioned testing of psychometric tools and validating already gathered results. We shall create synthetic datasets to match two very different groups of human subjects. The first one consisted of 47 Polish students of different curricula at the Adam Mickiewicz University in Poznań, Poland (the study was conducted in Polish), aged 20 to 25. The second one consisted of 102 people, representatives of the educational sphere (students and teachers) of the Dnipro region in Ukraine, aged 21 to 72 (this study was conducted in Ukrainian). We shall carry out this study employing three different language versions of DFT: Polish, Ukrainian, and English. |