Table of Contents
Advertisement
4 Number of Digit and Preci ion
Number of Digit
Standard Mode
The following applies when the check box next to the "Decimal Calculation" item on the Basic
Format dialog box is not selected.
• Up to 611 digits are stored in memory for integer values.
• Decimal values up to 15 digits are converted to fraction format and saved in memory. When
a mathematical expression cannot be converted to fraction format, the result is displayed in
decimal format.
• Values stored in memory are displayed as-is, regardless of how [Number Format] settings
(Normal 1, Normal 2, Fix 0 – 9, Sci 0 – 9) are configured (except when a decimal value is
displayed).
Decimal Mode
The following applies when the check box next to the "Decimal Calculation" item on the Basic
Format dialog box is selected.
• Values stored in Ans memory and values assigned to variables have the same number of
digits as defined for Standard mode values.
• Values are displayed in accordance with how [Number Format] settings (Normal 1, Normal 2,
Fix 0 – 9, Sci 0 – 9) are configured.
• Displayed values are rounded to the appropriate number of decimal places.
• Some applications store values using a mantissa up to 15 digits long and a 3-digit
exponent.
Preci ion
• Internal calculations are performed using 15 digits.
• The error for a single mathematical expression (Decimal mode calculation error) is 1 at the
10th digit. In the case of exponential display, calculation error is 1 at the least significant
digit. Note that performing consecutive calculations causes error to be cumulative. Error
is also cumulative for internal consecutive calculations performed for: ^(
n
r
C
, etc.
• Error is cumulative and tends to be larger in the vicinity of a function's singular point(s) and
inflection point(s), and the vicinity of zero. With sinh(
inflection point occurs when
-4-1
Number of Digits and Precision
x
= 0. In this vicinity, error is cumulative and precision is poor.
20060301
y
x
),
x
x
) and tanh(
), for example, the
x
x
n
r
,
!,
P
,
Advertisement
Table of Contents
