Agilent Technologies 93000 SOC Series Training Manual page 570

Appendix B
The following formulas are used:
End point method: n = # of sampled points
s
Zero-point method:
Minimum RMS method:
Calculation of DAC DeviceLSB
The end point method assumes a straight line between the first
and last measured value.
The other two methods consider all the measured values and
derive the DeviceLSB from two sorts of linear regression
calculations.
DAC Integral Non-Linearity
Integral non-linearity (INL) measures the deviation of each output
value from a calculated straight line.
The straight line can be drawn as a best-fit line or between the end
points as shown below:
570