One of the most common questions, in Salisbury’s view, is also the most problematic: “What’s the number?” Plan sponsors, providers and policymakers alike come to EBRI wondering what lump sum retirement plan participants must reach to be adequately prepared for retirement, he explains, or they ask for the average salary deferral percentage or income replacement ratio that’s going to guarantee plan success. Yet, as Salisbury stressed during the keynote address of the 2014 PLANSPONSOR National Conference, plan success is the result of myriad factors inside and outside a plan and can seldom be captured in a single metric.

“Just as individuals have very different reasons for investing in the 401(k) plan related to life stage, health, family situation and so many other factors, this comes through to the plan sponsors, who also have a wide variety of goals,” Salisbury says. “Helping plans define and monitor plan success is one of the great challenges currently facing retirement plan professionals.”

To ask for a single number is to assume a one-size-fits-all mentality, Salisbury says, and also overlooks the diversity of outcomes that can occur for different participants in the same plan. He points to a recent EBRI analysis that shows only about 17% of U.S. workers in the lowest income quartile who will require long-term, chronic illness care in retirement will be able to cover 100% of the cost of such care if current savings and cost trends hold. For workers in the highest income quartile, that figure jumps to 86%.

“So you have to ask yourself, is this success for the U.S. retirement system?” he explains. “Well, it’s successful for some.”

These numbers suggest that simply relying on income replacement ratios can be a misleading indicator of retirement readiness across a plan or even an entire retirement system, Salisbury says. A bottom-quartile worker with a relatively high income replacement ratio could struggle with expenses in retirement as much or more than a high-wage worker with a lower replacement ratio, he says. Additionally, the replacement ratio says little about the participant’s ability to meet ballooning health care costs and other moving variables.

“In that respect, generalized retirement income replacement ratios don’t tell anyone much of anything,” Salisbury continues. “We like to turn the question back on them and ask, 'Who are you thinking about? Which segment of the plan’s participant population are you considering, and why? What element of plan success do you want to measure?'d”

One of the first steps in achieving more meaningful plan outcome measures, Salisbury says, is to realize the same number can mean very different things to different individuals. He points to a data from the University of Michigan’s Health and Retirement Study and the Investment Company Institute, among others, which define plan success as a function of whether participants are “saving optimally.”

“They compared household net worth with the amount each household needed to save over a working life cycle to preserve its living standard and consumption in retirement,” Salisbury explains. “The authors found that about 80% of households in their samples had accumulated as much as or more wealth than their optimal targets.”

This seems like a great overall readiness number for the U.S. retirement system, he says, but one must be careful to put the number in perspective.

“Do you think Pennsylvania Avenue’s gutter is better to live in than K Street?” Salisbury asks. “Well sure, because it’s in front of the White House and in a better neighborhood. Well, that’s how to understand the success measures in these studies. They basically say, if you’ve spent your life living in one gutter, and then you’re able to move to a nicer gutter when you retire, still homeless in both cases, you have now so dramatically improved your lifestyle you have actually over saved and under consumed. The test is, are you as poor as you always were?”

Salisbury says he travels often to industry conferences and hears a lot of support for this type of analysis, “including from a large number of Senators and Representatives in Congress.” He is quick to add that this analysis is by no means wrong, but it shows the dangers of looking to a single metric to define plan or even system-wide success.

“It does tell us something about the workforce entering retirement, but not everything,” Salisbury explains. “Unfortunately, it’s become a main backdrop for our policy debate. There is very broad consensus that this is the right number, the right study and the right analysis.”