Comunque battute a parte, per chi non ha tempo/voglia di leggerselo tutto, la parte interessante imho e’ questa:
Discussion
Our main finding is that using AI to complete tasks that require a new skill (i.e., knowledge of a new Python library) reduces skill formation. In a randomized controlled trial, participants were assigned to the treatment condition (using an AI assistant, web search, and instructions) or the control condition (completing tasks with web search and instructions alone). The erosion of conceptual understanding, code reading, and debugging skills that we measured among participants using AI assistance suggests that workers acquiring new skills should be mindful of their reliance on AI during the learning process. Among participants who use AI, we find a stark divide in skill formation outcomes between high-scoring interaction patterns (65%-86% quiz score) vs low-scoring interaction patterns (24%-39% quiz score). The high scorers only asked AI conceptual questions instead of code generation or asked for explanations to accompany generated code; these usage patterns demonstrate a high level of cognitive engagement.
Contrary to our initial hypothesis, we did not observe a significant performance boost in task completion in our main study. While using AI improved the average completion time of the task, the improvement in efficiency was not significant in our study, despite the AI Assistant being able to generate the complete code solution when prompted. Our qualitative analysis reveals that our finding is largely due to the heterogeneity in how participants decide to use AI during the task. There is a group of participants who relied on AI to generate all the code and never asked conceptual questions or for explanations. This group finished much faster than the control group (19.5 minutes vs 23 minutes), but this group only accounted for around 20% of the participants in the treatment group. Other participants in the AI group who asked a large number of queries (e.g., 15 queries), spent a long time composing queries (e.g., 10 minutes), or asked for follow-up explanations, raised the average task completion time. These contrasting patterns of AI usage suggest that accomplishing a task with new knowledge or skills does not necessarily lead to the same productive gains as tasks that require only existing knowledge.
Together, our results suggest that the aggressive incorporation of AI into the workplace can have negative impacts on the professional development workers if they do not remain cognitatively engaged. Given time constraints and organizational pressures, junior developers or other professionals may rely on AI to complete tasks as fast as possible at the cost of real skill development. Furthermore, we found that the biggest difference in test scores is between the debugging questions. This suggests that as companies transition to more AI code writing with human supervision, humans may not possess the necessary skills to validate and debug AI-written code if their skill formation was inhibited by using AI in the first place.
Che imho dice una cosa interessante. In sintesi, gli unici che dimostrane un reale incremento di produttivita’ sono quelli che fanno pure “vibe coding”
There is a group of participants who relied on AI to generate all the code and never asked conceptual questions or for explanations. This group finished much faster than the control group (19.5 minutes vs 23 minutes), but this group only accounted for around 20% of the participants in the treatment group.
Che poi son quei casi in cui ti trovi API tokens, secrets, salcazzo, esposti in chiaro nel repo o peggio, direttamente esposti nel browser quando qualcuno accede al tuo servizio online.
Other participants in the AI group who asked a large number of queries (e.g., 15 queries), spent a long time composing queries (e.g., 10 minutes), or asked for follow-up explanations, raised the average task completion time.
Io rientro in questo tipo per dire. Ma va detto che effettivamente un guadagno ci sta comunque. Non ho granche’ voglia ormai, dopo 20+ anni, di scrivere il solito codice cinquanta volte. Sapendo dare direzioni e cosa controllare quando mi produce roba, mi risparmia il tedio e il tunnel carpale di farlo io.
Non se ne parla neanche quando son forzato a usare uno stack che non uso di solito, come in un progetto recente in cui ho dovuto usare per forza bicep e powershell (uso terraform e bash / python di solito). Di mettermi a imparare quello stack non solo non ho intenzione ma non c’era proprio tempo per deadlines assurde, in quel caso, magari il codice di per se’ non era il migliore, ma principi base KISS e DRY son stati rispettati e ugualmente una struttura modulare decente dell’intero framework che ho dovuto generare perche’ comunque l’esperienza ce l’ho.
Il rischio vero e’ per i “nuovi” che non hanno questa esperienza maturata e vengono buttati dentro con la minaccia “usa l’ai o resti indietro” e con restare indietro vuol dire “ti passano altri avanti e quella e’ la porta”.
Together, our results suggest that the aggressive incorporation of AI into the workplace can have negative impacts on the professional development workers if they do not remain cognitatively engaged. Given time constraints and organizational pressures, junior developers or other professionals may rely on AI to complete tasks as fast as possible at the cost of real skill development.
