How are LLMs Used for Conceptual Modeling? An Exploratory Study on Interaction Behavior and User Perception

Large Language Models (LLMs) have opened new opportunities in modeling in general, and conceptual modeling in particular. With their advanced reasoning capabilities, accessible through natural language interfaces, LLMs enable humans to deepen their understanding of different application domains and enhance their modeling skills. However, the open-ended nature of these interfaces results in diverse interaction behaviors, which may also affect the perceived usefulness of LLM-assisted conceptual modeling. Existing works focus on various quality metrics of LLM outcomes, yet limited attention is given to how users interact with LLMs for such modeling tasks. To address this gap, we present the design and findings of an empirical study conducted with information systems students. After labeling the interactions according to their intentions (e.g., Create Model, Discuss, or Present), and representing them as an event log, we applied process mining techniques to discover process models. These models vividly capture the interaction behaviors and reveal recurrent patterns. We explored the differences in interacting with two LLMs (GPT 4.0 and Code Llama) for two modeling tasks (use case and domain modeling) across three application domains. Additionally, we analyzed user perceptions regarding the usefulness and ease of use of LLM-assisted conceptual modeling.