Objective: It is increasingly understood that engagement with digital interventions entails physical, cognitive, and affective components. However, most empirical studies examining engagement with digital interventions focus on physical engagement measured via objective usage metrics. Although self-reported instruments have been developed to measure other components of engagement, they have either shown inadequate psychometric properties or conflated engagement with other related but distinct constructs, such as motivation. To address this gap, the present study developed and evaluated the psychometric properties of a new self-report instrument, the Engagement with Digital Interventions (EDI) scale, designed to more precisely measure physical, cognitive, and affective engagement.

Methods: The EDI scale was tested with a sample of 400 online U.S. participants. These participants completed two digital tasks (a self-monitoring task and a puzzle game) and used the scale to rate their engagement with each task. The factor structure of engagement was assessed using factor analysis. Internal consistency and construct validity were also assessed.

Results: Exploratory factor analysis (conducted on data from one task) revealed a three-factor structure that provided the best fit, accounting for 76.2% of the variance after removing the cross-loading item. Confirmatory factor analysis (CFA) of this factor structure showed a good fit to the data across both tasks (CFIs > .97, TLIs > .96, RMSEAs < .08, SRMR < .06). The instrument showed high internal consistency in both tasks (Cronbach’s alphas > .85). Convergent validity was supported by moderate to high correlations between the engagement components with theoretically related constructs (i.e., motivation and cognitive flow). Discriminant validity was supported by weaker correlations with the Big-5 personality traits. Convergent and discriminant validity were further validated by a CFA of multitrait-multimethod data that included engagement measured using a different method. Concurrent validity was confirmed as higher self-reported engagement with the puzzle game was significantly correlated with completing more levels of the game (beta = .219, SE = .108, p < .001).

Conclusion: The EDI scale demonstrated strong psychometric properties in measuring engagement with digital interventions. It is recommended for use in future digital intervention studies to further validate and refine its application.