FMCSA tries to fix SMS flaws without admitting them
The Federal Motor Carrier Safety Administration has proposed future enhancements to the Safety Measurement System (SMS), the methodology that determines Compliance, Safety, Accountability (CSA) scores. FMCSA has actually proposed most of these changes before, but it is apparently is revisiting the changes in light of the ongoing review of CSA and SMS by the National Academy of Sciences.
FMCSA responded to comments but retained its originally proposed enhancements – tweaks to how the SMS methodology works – while ignoring objections to any SMS changes that did not address fundamental and systemic flaws.
The two newly proposed SMS changes don't really relate to how SMS works but rather which carriers SMS measures. In doing so, FMCSA once again – as it did in its proposed rule on carrier safety fitness determinations (SFDs) – implicitly acknowledges that volatility renders SMS unusable for many small carriers.
The two newly proposed enhancements are:
Increasing the data sufficiency threshold for the Crash Indicator Behavior Analysis Safety Improvement Category (BASIC) from 2 crashes in past 24 months to 3 crashes; and
Excluding from percentile rankings those carriers that do not have at least one inspection with a violation in the past year in the Hours of Service Compliance, Vehicle Maintenance, Hazardous Materials or Driver Fitness BASICs.
In the case of the Crash Indicator BASIC, FMCSA essentially acknowledges the difference between randomness and a pattern: "While this change does not substantively impact the effectiveness of the Crash Indicator BASIC, the greater data sufficiency standard of the BASIC would yield greater confidence that this BASIC is identifying carriers with established patterns of crashes thereby enabling the Agency to further focus its investigative resources on carriers with more crash involvement."
Going from two crashes to three crashes really isn't enough to establish a pattern and offset the law of small numbers, but it's significant that FMCSA acknowledges such an obvious concept.
The elimination of carriers that haven't had an inspection with a violation in the past year is subtler. FMCSA says that the change would mean that 1,243 carriers would no longer have a BASIC at or above the intervention threshold but that those carriers have a crash rate 4.8 times lower than the national average – i.e., 0.71 crashes per 100 power units (PUs) versus the average of 3.43.
"Therefore, removing these carriers from prioritization will allow the Agency to focus its intervention efforts on a set of carriers with a much higher crash rate of 5.20 crashes per 100 PUs," FMCSA said.
What FMCSA undoubtedly discovered is precisely what it refused to acknowledge in the SFD rulemaking but was exposed by our analysis: Very small carriers have very few crashes. Data provided under a Freedom of Information Act request revealed that 53% of the carriers that would have failed the data-only SFD method in 2011 had no crashes in the following year. So FMCSA is eliminating carriers that for the most part depress the average crash rate.
FMCSA likely chose the no-inspections-with-a-violation-in-the-past-year-qualifier because it was an easy change to make. But the agency could have chosen a different qualifier that would have targeted high crash rates even more efficiently. More on that another day.
