By Brad Belsher, Ph.D
Sept. 30, 2019

Photo by Cpl. Tyler Viglione

There is growing interest for the Department of Defense (DOD) to implement computer-generated algorithms that use health care data to identify service members at heightened risk for suicide. The Psychological Health Center of Excellence conducted a systematic review of suicide prediction models (SPMs), and the findings were recently published in the journal "JAMA Psychiatry." The review synthesized all of the existing research to provide decision makers and the public with an overview of the current evidence on SPMs.

To put the results of this review in context, it's important to note the accuracy of any prediction, be it based on a clinical screening tool or complex computer model, depends on the frequency of the condition being predicted. In general, frequent events are easier to predict, relatively rare events are harder to predict, and extremely rare events are essentially unpredictable. Suicide mortality among active-duty service members falls in the relatively rare category at 21 deaths per 100,000 service members (2017 Department of Defense Suicide Event Report) and, therefore, is very difficult to predict. This means that any randomly identified service member will have 0.02 percent chance of dying from suicide. Suicide screeners attempt to increase this accuracy but do not perform very well.

SPMs attempt to increase prediction accuracy by using large-scale data sets — with hundreds to thousands of variables — and complex statistical procedures to identify a cohort of high-risk patients that have a greater likelihood of dying by suicide. For example, DOD investigators developed a prediction model that improved prediction accuracy to 1 percent. This means if this algorithm were implemented in standard practice, one out of every 100 soldiers flagged as high-risk would likely die by suicide, and conversely, 99 out of every 100 soldiers flagged would not die by suicide. Notably, more than half of service members who die by suicide would not be flagged by this model.

Our review found SPM prediction accuracy was similar across military, veteran, and civilian health systems. We also found no research that had rigorously evaluated the benefits, harms, feasibility, or costs of actually applying SPMs to patients (see Table 1 for examples of these and other considerations). Based on these findings and the lack of research beyond model development, we concluded that it would be premature to implement SPMs across large health systems.

Since publication of our systematic review, some who commented on the article deemed our conclusions to be pessimistic. We, however, consider them to be responsibly cautious. Some have argued treatment recommendations exist for other conditions with lower prediction rates. For example, cancer-reducing medications, such as Click to closeTamoxifenA synthetic drug used to treat breast cancer and infertility in women. It acts as an estrogen antagonist.tamoxifen, are recommended for women with at least a 3 percent risk for breast cancer. Research has also demonstrated these medications are relatively safe and can reduce cancer risk. Unfortunately, we do not have something comparable to tamoxifen for suicide prevention. The recently updated VA/DOD clinical practice guideline on suicide makes only one strong recommendation for treatment of suicide risk: cognitive behavioral therapy (CBT), which may be effective in reducing suicide attempts. The cost-benefit differences between dispensing medication and offering up to 12 sessions of CBT across large patient cohorts is considerable.

Another major concern is the potential for adverse consequences of falsely flagging service members as at high risk for suicide. Service members' employment is interlinked with their health care system, and there are different rules about health information than in the Department of Veterans Affairs or civilian health care systems. This enables health information to be shared between the medical side and line side to ensure service members are fit for duty. In the DOD environment, flagging large cohorts of service members as high risk for suicide based on a 1 percent predictive accuracy rate could have harmful effects on their careers (and by extension, their families) and their general well-being. Before any SPM goes live in the military health system, more research is needed to better evaluate these potential harms and benefits.

The history of health care is full of examples of initiatives rolled out prior to adequate assessments of benefits, harms, and costs. Decision makers must make use of the best evidence available at the time, along with other factors, to determine whether or not to fast-track an intervention into routine care. Based on the results of our review, we're cautious about advocating for the full-scale implementation of SPMs at this time. We're also cautiously optimistic science will continue to advance our ability to more effectively prevent suicide, especially in light of the unprecedented amount of ongoing suicide research.

Dr. Belsher is the chief of research translation and integration at the Psychological Health Center of Excellence. His primary areas of focus include deployment-related mental health, systematic review methodologies, health services research, dissemination of evidence-based mental health practice, and collaborative care.