When evidence-based programs are implemented into service organizations, they often prove to be less effective than when they are tested with efficacy trials in the controlled environments of research institutions. Little is currently known about which factors under which conditions predict successful program implementation outcomes (such as sustainability). The aim of these studies is to identify factors that promote or inhibit successful program implementation.
Participants for the four studies were Parent Management Training Oregon model (PMTO) therapists (n = 83) employed in Norwegian child mental health organizations (n = 67) and families receiving PMTO treatment (n = 162). Observational scores of therapist fidelity to the core components of the PMTO program were computed from ratings of video recorded PMTO therapy sessions using a previously validated fidelity of implementation code.
Data were analyzed using a multilevel structural equation model (SEM) with latent factors employing full-information maximum likelihood (FIML), multilevel confirmatory factor analysis, MANCOVA, contrast tests, and examination of factors in the models used to code interview data.
- More positive practitioner ratings of organizational social context predicted significantly higher observed therapist fidelity scores to the PMTO program, β = .53, p < .01, controlling for education. The effect size was large R2 = .29.
- Practitioners who worked in organizations with three PMTO therapists rated their workplace environments significantly higher than the aggregate ratings of those in agencies with one or two PMTO therapists, β = .29, p = .03. The effect size for this study was also large, Cohen’s d = .80.
- The factors identified from interviews with PMTO therapists with the most statements determined to be relevant to program implementation were: collegial support from PMTO therapists working in the same agencies, supportive organizational context, adaptability of the program to individual client circumstances, ongoing training, and program effectiveness.
- Eighty-four percent of PMTO practitioners’ interview statements fit into a model of implementation developed for use in this study, suggesting that the model is a good explanation of the implementation factors salient to participants in this sample.
- The number of PMTO therapists in an agency was significantly correlated with four implementation outcome variables (e.g., number of cases treated with PMTO in the past six months), controlling for agency size, length of time working with PMTO, and therapist perception of leadership. Contrast tests revealed that it was necessary to have at least three therapists in an agency to positively influence implementation outcomes.
The results from all four studies support each other providing evidence for the validity of the findings. During interviews, practitioners identified positive organizational context as a central implementation factor to their work quality; this finding was supported by a quantitative study that found therapists’ observed scores of program fidelity improved when they viewed their organizational social context more positively.
A second finding that was supported by multiple results is that therapists who had access to collegial support from other PMTO therapists in their agencies were more positive about their work environments, had more time to work with the PMTO program, and treated more clients with the program. There appears to be an advantage of having immediate, informal access to colleagues who share similar training and professional experiences. These results are notable in that they may be significant to program success in organizations and they appear to be the first studies to empirically test the effects of clustering EBP practitioners in mental health agencies and to use organizational context to predict observed fidelity scores. Finally, a new model of implementation factors developed to explain interview data from a sample of PMTO therapists shows promise for quantitative measure development.