TABLE OF CONTENTS

plm/tokread [ Modules ]

[ Top ] [ Modules ]

NAME

   tokread -- Tokenizer and Reader module

DESCRIPTION

   The tokread module is responsible for reading from a source file and to
   convert source code to tokens.  Tokens are items that correspond to units
   of code in the source file.  The different kinds of tokens are identified
   by the token identifier and for some kinds of tokens it also has data,
   which is either a number of a string.  Many tokens do not have additional
   information.

   The Tokenizer and the Reader work together, and therefore have a common
   interface.  The tokenizer, being of a higher level than the reader,
   has the majority of the interface functions (read a token, go to next,
   and so on).  The only way the user should interact with the reader is when
   setting the source.  Duting the normal execution, the reader reads from
   source files which are given to the reader in the form of the filename.
   For debug purposes, it's also possible to read from a string.

   This is how the tokread module works:
   * the user initializes the reader by setting the filename of the source
     file,
   * then to read one token the user calls get$token, which returns the
     identifier of the token read.  On end of input it returns the tok$eof
     token, which is consistent with how parsers are usually described in
     literature.
   * For additional data from the token, the value is stored in token$value
     if it's a token with a numeric value. And when the token
     is of a type that contains a string (identifiers and string literals),
     the pointer to the string is stored in token$string external variable
     Both the number and the string values are temporary, so if the user
     intends to keep them for later use, he has to make a copy of them.
   * When the user has finished using the token and wants to move to the next
     one, he calls the tok$next function.  This function does not return
     anything as its only purpose is to advance to the next token.  To check
     the end of input condition and the token, get$token is used.

tokread/get$token [ Functions ]

[ Top ] [ tokread ] [ Functions ]

NAME

   get$token -- get the id of the current token

SYNOPSIS

   get$token

DESCRIPTION

   Returns the id of the current token.  Should be called after the reader
   is initialized.  In this case it returns the current value without
   advancing the cursor.  When the end of input is reached, a special token
   with the id tok$eof is returned.

tokread/print$token [ Functions ]

[ Top ] [ tokread ] [ Functions ]

NAME

   print$token -- print token

SYNOPSIS

   call print$token;

DESCRIPTION

   This procedure prints the current token in a text representation without
   any newline characters.

tokread/reader$set$ptr [ Functions ]

[ Top ] [ tokread ] [ Functions ]

NAME

   reader$set$ptr -- set reader string

SYNOPSIS

   call reader$set$ptr(ptr);

DESCRIPTION

   This is a procedure used to initialize a string that the reader will read.
   The ptr parameter is a pointer to a string that the reader should use.
   This procedure is only for debugging purposes and will perhaps be repaced
   or removed in later versions.

tokread/tok$next [ Functions ]

[ Top ] [ tokread ] [ Functions ]

NAME

   tok$next -- go to next token

SYNOPSIS

   call tok$next;

DESCRIPTION

   This procedure advances the current token to the next one if not at the
   end of input.  Otherwise it does nothing.