Tokens - mikroElektronika PIC Microcontrollers PIC12 User Manual

mikroC - C Compiler for Microchip PIC microcontrollers

TOKENS

Token is the smallest element of a C program that is meaningful to the compiler.
The parser separates tokens from the input stream by creating the longest token
possible using the input characters in a left–to–right scan.
mikroC recognizes following kinds of tokens:
- keywords,
- identifiers,
- constants,
- operators,
- punctuators (also known as separators).
Token Extraction Example
Here is an example of token extraction. Let's have the following code sequence:
inter =
First, note that
keyword
The programmer who wrote the code might have intended to write
inter = a + (++b)
but it won't work that way. The compiler would parse it as the following seven
tokens:
inter
=
a
++
+
b
;
Note that
page
38
a+++b;
would be parsed as a single identifier, rather than as the
inter
followed by the identifier
int
// identifier
// assignment operator
// identifier
// postincrement operator
// addition operator
// identifier
// semicolon separator
parses as (the longest token possible) followed by
+++ ++
MikroElektronika: Development tools - Books - Compilers
.
er
mikroC
making it simple...
.
+