[NEXT] [PREV]    HANDBOOK / REFLEX REFERENCE MANUAL /

How To Describe a Token

Gentle
Applications
Concepts
Examples
Handbook
Support
Download

For each token Token introduced in the Gentle specification, there must be a token description file Token.t . This file must specify a Lex rule that handles the token. Such a rule is given by a regular expression that matches the token, and an action that computes the attributes of the token.

   pattern { action }
If the rule does not fit on one line, the action may be written on several lines, but the opening brace must appear on the first line.

By way of an example, a token introduced in a Gentle specification as


   'token' Number (-> INT)
may be described in file Number.t:


   [0-9]+ {
      yylval.attr[1] = atoi(yytext);
      yysetpos();
      return Number;
   }
The pattern

   [0-9]+ 
matches a non-empty sequence of digits.

The line


   yylval.attr[1] = atoi(yytext);
converts the matched input token ( yytext) into an integer (using the C function atoi) and assigns it as the first attribute. The Lex variable yylval has a field attr which is an array of token attributes. attr[ i] stands for the i-th attribute.

The macro


   yysetpos();
defines the source position of the token (the source position is stored in yylval.attr[0]). In a Gentle specification, this value may be accessed with the @-predicate.

The line


   return Number;
returns the code of the token. The token code is a constant that has the same name as the token.





[NEXT] [PREV]