Understanding the Use and Impact of AI-Coding Assistants and Agents in System Development
Conference
Zhu, Weidong, Arunasalam, Arjun, Chen, Chen et al. Understanding the Use and Impact of AI-Coding Assistants and Agents in System Development
. 10.1145/3843967.3844685
Zhu, Weidong, Arunasalam, Arjun, Chen, Chen et al. Understanding the Use and Impact of AI-Coding Assistants and Agents in System Development
. 10.1145/3843967.3844685
AI coding assistants have become daily infrastructure in systems software development. Despite the wide adoption, we lack an understanding of how developers use coding AI in systems development, what challenges they encounter, how they mitigate them, and what capabilities they still need. This paper aims to close the gap with a large-scale empirical study of practitioner discourse. The study analyzes 192,279 developer discussions from Reddit and GitHub (2022 - 2026), using a large language model as a judge to distill each into a structured record and yielding 18,541 relevant observations. We find that AI-assisted coding tools are already applied to production systems work (73.6% of relevant records), whereas over three-quarters of the discussions describe a concrete failure. Failures cluster at the systems layer, and success is task-dependent, with debugging the only task reporting more negative than positive outcomes. Developers today cope by gating AI output through human review, tests, and continuous integration, a pattern that motivates systems-aware evaluation of coding AI.
