Internally Stored Double Precision Number Access - Agilent Technologies 8453 Operator's Manual

Internally Stored Double Precision Number Access

For validation purposes, it might become necessary to manually recalculate
the ChemStation results such as calibration curves, correlation coefficients,
theoretical plates, etc. When doing so the number format used in the
ChemStation has to be taken into account.
For all numbers stored internally within the ChemStation, the "C" data type
DOUBLE is used. This means that 14 significant digits are stored for each
number. The implementation of this data type adheres to the Microsoft
implementation of the IEEE standard for "C" data type and the associated
rounding rules (see Microsoft documents Q42980, Q145889 and Q125056).
Due to the non-limited number of parameters that might be used for the
calculation of the calibration table, it is not possible to calculate the exact
error possibly introduced by the propagation and accumulation of rounding
errors. Thorough testing with different calibration curve constructions
however has shown that the accuracy of up to 10 digits can be guaranteed.
Whereas the area, height and retention time repeatability of a
chromatographic analysis usually has 3 significant digits, 10 significant digits
within the calculations is sufficient. For this reason, the calibration, and other
tables, display a maximum of 10 significant digits.
If an external (manual) calculation for validation is required, it is
recommended that all digits used for the internal calculations are utilized.
Using the displayed and/or rounded data for the external calculations might
give results differing from the ChemStation due to rounding errors.
The following paragraph describes how to access all internally stored digits for
numbers typically required for manual calculations. In all cases, a data file
must be loaded and reported with the appropriate report style prior to
execution of the listed command. All commands are entered on the
ChemStation command line which may be enabled from the view menu. The
information in file "C:\CHEM32\TEMP.TXT" may be viewed using NOTEPAD
or a suitable TEXT editor.
Raw Peak Information:
• Retention Time
Understanding Your Agilent ChemStation
Evaluating System Suitability
Internally Stored Double Precision Number Access
11
255