The setEncoderResolution(double ticksPerUnit, DistanceUnit distanceUnit) method calculates the resolution (in ticks/mm) as follows:

resolution = distanceUnit.toMm(ticksPerUnit);

The problem is resolution is not a distance, it is ticks Per distance, so using the distanceUnit.toMm method directly on ticksPerUnit does not work as desired. This could be corrected as follows:

resolution = 1.0 / distanceUnit.toMm(1.0 / ticksPerUnit);