





            


                       TTSS ---- AA SSiimmppllee TTookkeenn SSccaannnniinngg LLiibbrraarryy
                                    RReelleeaassee 11..0066


                                    _P_a_u_l _D_u_B_o_i_s
                              _d_u_b_o_i_s_@_p_r_i_m_a_t_e_._w_i_s_c_._e_d_u
                     Wisconsin Regional Primate Research Center
                          Revision date:  18 October 1993


            Applications  often wish to pull strings apart into individ-
            ual tokens.  This document describes TS, a library  consist-
            ing  of  an unsophisticated set of routines providing simple
            token scanning operations.

            String tokenizing can often  be  done  satisfactorily  using
            strtok()  or  equivalent  function from the C library.  When
            such routines are insufficient, the routines described  here
            may  be  useful.   They offer, for example, quote and escape
            character parsing, and configurability of  underlying  scan-
            ning  properties,  within the confines of a fixed interface.
            TS provides a simple built-in scanner, which may be replaced
            by  alternate  routines as desired.  Applications may switch
            back and forth between scanners on the fly.

            11..  IInnssttaallllaattiioonn

            This release of TS is configured using imake and the  WRPRC2
            configuration  files,  so you also need to obtain the WRPRC2
            configuration distribution if you want to build it the usual
            way.   (If  you  want to avoid imake, the Makefile is simple
            enough that you should be able to tweak it by hand.)

            There is one library  to  be  built,  libtokenscan.a.   That
            library  should  be installed in a system library directory.
            The header file tokenscan.h should be installed in a  system
            header file directory.

            22..  EExxaammppllee

            The  canonical  method  of tokenizing a string with TS is as
            follows:












            Revision date:  18 October 1993     Printed:  1 August 19126





            Token Scanning Library      - 2 -


                 char buf[size], *p;

                 /* _._._._i_n_i_t_i_a_l_i_z_e _c_o_n_t_e_n_t_s _o_f _b_u_f _h_e_r_e_._._. */
                 TSScanInit (buf);   /* initialize scanner */
                 while ((p = TSScan ()) != (char *) NULL)
                 {
                      /* _._._._d_o _s_o_m_e_t_h_i_n_g _h_e_r_e _w_i_t_h _t_o_k_e_n _p_o_i_n_t_e_d _t_o _b_y _p _h_e_r_e_._._. */
                 }


            The scanner is initialized  by  passing  the  string  to  be
            scanned to TSScanInit() and TSScan() is called to get point-
            ers to successive tokens.  TSScan() returns NULL when  there
            are no more.

            33..  BBeehhaavviioorr ooff tthhee DDeeffaauulltt SSccaannnneerr

            The  default  scanner is destructive in that it modifies the
            string scanned (it writes nulls at the  end  of  each  token
            found),  so make a copy of the scanned string if you need to
            maintain an intact version.

            The scanner is controlled by delimiter, quote,  escape,  and
            end-of-string  (EOS)  characters.   The defaults for each of
            these are given below.


            center tab(:); l l .  delimiter:space tab quote:" ' escape:\
            EOS:null linefeed carriage-return


            In  the  simplest  case,  tokens are sequences of characters
            between delimiters.  Since the default  delimiters  are  the
            whitespace characters space and tab, tokens are sequences of
            non-whitespace characters.


                 This is a line ->   <This> <is> <a> <line>


            Quotes may be used to include  whitespace  within  a  token.
            Quotes  must match; hence one quote character may be used to
            quote another kind of quote character, if there is more than
            one.


                 "This is" a line    -><This is> <a> <line>
                 This" "is a line    -><This is> <a> <line>
                 "'" '"'        ->   <'> <">
                 "'"'"'         ->   <'">


            The  escape  character  turns off any special meaning of the
            next character, including another escape character.



            Revision date:  18 October 1993     Printed:  1 August 19126





                                        - 3 -     Token Scanning Library


                 What\'s up     ->   <What's> <up>
                 \\ is the escape    -><\> <is> <the> <escape>


            The EOS characters tell the scanner when to  quit  scanning.
            A null character always terminates the scan.  In the default
            case, linefeed and carriage return do as well.

            You can replace the delimiter, quote, escape, or EOS charac-
            ter sets.  This changes the particular characters that trig-
            ger the  above  behaviors,  without  changing  the  way  the
            default  scan  algorithm works.  Or you can replace the scan
            routine to make the scanner  behave  in  entirely  different
            ways.

            By  default,  multiple  consecutive delimiter characters are
            treated as a single delimiter.  A flag may  be  set  in  the
            scanner  structure  to  suppress delimiter concatenation, so
            that every delimiter character is significant.  This is use-
            ful  for  tokenizing  strings  in  which  empty  fields  are
            allowed: two consecutive delimiters are considered  to  have
            an empty token between them, and delimiters appearing at the
            beginning or end of a string signify an empty token  at  the
            beginning end of the string.

            The  difference  in treatment of strings when delimiters are
            concatenated versus when they are not is illustrated  below.
            Suppose the delimiter is colon (:) and the string to be tok-
            enized is:

                 :a:b::c:

            When delimiters are concatenated, the string contains  three
            tokens:


                 :a:b::c:       -> <a> <b> <c>


            When  all  delimiters are significant, string contains three
            empty tokens in addition:


                 :a:b::c:       -> <> <a> <b> <> <c> <>


            44..  PPrrooggrraammmmiinngg IInntteerrffaaccee

            Source files using TS routines  should  include  tokenscan.h
            and executables should be linked with -ltokenscan.

            A scanner is described by a data structure:





            Revision date:  18 October 1993     Printed:  1 August 19126





            Token Scanning Library      - 4 -


                 typedef struct TSScanner TSScanner;
                 struct TSScanner
                 {
                      void (*scanInit) ();
                      char *(*scanScan) ();
                      char *scanDelim;
                      char *scanQuote;
                      char *scanEscape;
                      char *scanEos;
                      int  scanFlags;
                 }


            Scanner   structures  may  be  obtained  or  installed  with
            TSGetScanner() and TSSetScanner().

            For each string to be  scanned,  the  application  passes  a
            pointer to it to TSScanInit(), which takes care of scan ini-
            tialization.  If the application requires initialization  to
            be  performed  in  addition to that done internally by TS, a
            pointer to a routine that does so should be installed in the
            scanInit  field of the scanner data structure.  It takes one
            argument, a pointer  to  the  string  to  be  scanned.   The
            default scanInit is NULL, which does nothing.

            scanDelim,  scanQuote,  scanEscape, and scanEos are pointers
            to null-terminated strings consisting of the set of  charac-
            ters  to  be  considered  delimiter,  quote, escape, and EOS
            characters, respectively.  The default values were described
            previously.

            scanScan  points  to  the routine that does the actual scan-
            ning.  It is called by TSScan() and should  be  declared  to
            take no arguments and return a character pointer to the next
            token in the current scan buffer.   Normally,  this  routine
            does  the  following: call TSGetScanPos() to get the current
            scan position, scan the token, call TSSetScanPos() to update
            the scan position, then return a pointer to the beginning of
            the token.  If there are no more tokens in the scan  buffer,
            the routine should return NULL, and should continue to do so
            until TSScanInit() is called again.

            scanFlags contains flags that modify the scanner's behavior.
            For  the default scanner, the default is zero.  If the tsNo-
            ConcatDelims flag is set, the scanner stops on every  delim-
            iter rather than treating sequences of contiguous delimiters
            as a single delimiter.

            The public routines in the TS library are described below.

            vvooiidd TTSSSSccaannIInniitt ((pp))
            cchhaarr    **pp;;

            Initializes the scanning  routines  to  make  the  character



            Revision date:  18 October 1993     Printed:  1 August 19126





                                        - 5 -     Token Scanning Library


            string pointed to by p the current scan buffer.

            cchhaarr **TTSSSSccaann (())

            Returns  a  pointer  to  the  next token in the current scan
            buffer, NULL if there are no more.  The token is  terminated
            by a null byte.  Scan behavior may be modified by substitut-
            ing alternate scan routines.

            Once TSScan() returns NULL, it continues to do so until  the
            scanner  is reinitialized with another call to TSScanInit().

            vvooiidd TTSSGGeettSSccaannnneerr ((pp))
            TTSSSSccaannnneerr**pp;;

            Gets the current  scanner  information  (initialization  and
            scan procedures; delimiter, quote, escape, and EOS character
            sets; and scanner flags) into the structure pointed to by p.

            vvooiidd TTSSSSeettSSccaannnneerr ((pp))
            TTSSSSccaannnneerr**pp;;

            Installs  a scanner.  If p itself if NULL, all default scan-
            ner values are reinstalled.  Otherwise, any pointer field in
            p  with a NULL value causes the corresponding value from the
            default scanner to be reinstalled, and  if  p->scanFlags  is
            zero,  the scanner flags are set to the default (also zero).

            vvooiidd TTSSGGeettSSccaannPPooss ((pp))
            cchhaarr    ****pp;;

            Puts the current position within  the  current  scan  buffer
            into  the argument, which should be passed as the address of
            a character pointer.  This is useful when you want  to  scan
            only enough of the buffer to partially classify it, then use
            the rest in some other way.

            vvooiidd TTSSSSeettSSccaannPPooss ((pp))
            cchhaarr    **pp;;

            Set the current scan position to p.

            iinntt TTSSIIssSSccaannDDeelliimm ((cc))
            cchhaarr    cc;;

            Returns non-zero if c is a member of the  current  delimiter
            character set, zero otherwise.

            iinntt TTSSIIssSSccaannQQuuoottee ((cc))
            cchhaarr    cc;;

            Returns non-zero if c is a member of the current quote char-
            acter set, zero otherwise.




            Revision date:  18 October 1993     Printed:  1 August 19126





            Token Scanning Library      - 6 -


            iinntt TTSSIIssSSccaannEEssccaappee ((cc))
            cchhaarr    cc;;

            Returns non-zero if c is a  member  of  the  current  escape
            character set, zero otherwise.

            iinntt TTSSIIssSSccaannEEooss ((cc))
            cchhaarr    cc;;

            Returns  non-zero  if  c is an end-of-string character, zero
            otherwise.

            iinntt TTSSTTeessttSSccaannFFllaaggss ((ffllaaggss))
            iinntt     ffllaaggss

            Returns non-zero if all bits in flags are set for  the  cur-
            rent scanner, zero otherwise.

            44..11..  OOvveerrrriiddiinngg SSccaannnniinngg RRoouuttiinneess

            It  is possible to switch back and forth between scan proce-
            dures on the fly, even in the middle of scanning  a  string.
            The  general  procedure  is to use TSGetScanner() to get the
            current scanner information, and TSSetScanner() to install a
            new  one  and  reinstall the old one when done with the new.
            If you switch between more than two scanners, another method
            may be necessary.

            It is possible to modify the default scanner without replac-
            ing it.  For instance, you could change the  default  delim-
            iter set but leave everything else the same, as follows:


                 TSScanner scanStruct;

                 TSGetScanner (&scanStruct);
                 scanStruct.scanDelim = " \t:;?,!";
                 TSSetScanner (&scanStruct);


            55..  MMiisscceellllaanneeoouuss

            A  scanner  can  be  nondestructive with respect to the line
            being scanned by using a scan routine that copies characters
            out of the scanned line into a second buffer and returning a
            pointer to the second buffer.  The  second  buffer  must  be
            large  enough to hold the largest possible token, of course.
            If the second buffer is a fixed area, the  host  application
            must  be careful not to call TSScan() again until it is done
            with the current token, or else make a copy of it first.  If
            the  second buffer is dynamically allocated, the application
            must be  ready  to  do  storage  management  of  the  tokens
            returned.




            Revision date:  18 October 1993     Printed:  1 August 19126





                                        - 7 -     Token Scanning Library


            Some  scanners  might  not need delimiter, quote, escape, or
            EOS characters at all, particularly if token boundaries  are
            context sensitive.

            66..  DDiissttrriibbuuttiioonn aanndd UUppddaattee AAvvaaiillaabbiilliittyy

            The  TS  distribution may be freely circulated and is avail-
            able for anonymous FTP access in the  /pub/TS  directory  on
            host  ftp.primate.wisc.edu.   Updates  appear  there as they
            become available.

            The WRPRC2 imake configuration file distribution  is  avail-
            able on ftp.primate.wisc.edu as well, in /pub/imake-stuff.

            If  you  do  not  have  FTP  access,  send requests to soft-
            ware@primate.wisc.edu.  Bug reports, questions,  suggestions
            and comments may be sent to this address as well.








































            Revision date:  18 October 1993     Printed:  1 August 19126


