
               SIAL : Simple Image Access Language
               ===================================

    Sial is a C interpreter that permits easy access to the symbol
and type information stored in a executable image like a coredump or live
memory interfaces (e.g. /dev/kmem, /dev/mem). It support a FULL C
syntax and the same variable and function scope and type. Symbols and
type information in the image become standard variables and types in
the sial scripts's context.

    This README focuses on the differences between sial and a C compiler
so, for C syntax information, please refer to a C reference manual. I
also explain the mechanisms of the API that allow sial to be inserted
into any debugging tool that deals with objects and can furnish symbol
and type information to sial through the API. The more specific lcrash
sial implementation is described and a howto on creating commands is
also given here.

Preprocessor commands
---------------------

	All preprocessor commands I knew of are supported.  #define,
	#undef, #ifdef, #if, #ifndef, #else, #elif, #endif and #include.

	This one is ignored:  #ident #pragma

	Sial has a builtin secondary parser for preprocessor expression
	evaluation.

Symbols
-------

	The symbols from the system image and their associated value
	are available from within sial. Since, most of the time, no
	type information is associated with the symbols, a reference to
	a symbol return the actual address of the symbol within the
	system image. So you might say that sial adds a level of
	reference for symbol information. Let's say there is a (int)
	symbol called "nproc" that contains the number of processes
	currently running on the system. To get the value of nproc from
	sial one would have to write something like this:

        void
        showprocs()
        {
        int i;
        int np;

                np=*(int*)nproc;

                for(i=0;i<np;i++) {

                        do something...
                }
        }

	Here you see that we threat nproc as a pointer to a void and
	cast it to a int*.  More precisely, since no type information
	is available each symbol should be treated as a void* pointer
	and cast to the proper type when referenced.

	A second example is the case of a pointer. Let's say that
	symbol procs is a pointer to a 'struct proc' type. Then from
	sial it should be treated as a 'struct proc **' not a 'struct
	proc *'.

        void
        showprocs()
        {
        struct proc* p;

                for(p=*(struct proc**)procs; p; p=p->p_next)

                        do something...
                }
        }

Variable Initialization
--------------------

	Variable assignment at the time of declaration is supported.
	Also, the function __init() will be executed, if it is defined,
	right after the macro file as been compiled.

	Using an uinitialized variable will generate a run time error.

Variable types
--------------

	All types made available from the API can be used. 
	These are the types already defined in the executable image.

	Floating point types are not supported at this time. I have no
	plan to support it.

	Declaration of arrays is not supported. To access a array from
	the image, use a pointer.

	Unions and structures type declarations are supported and can be
	used to create additional types that become available within
	the same macro file.

	Typedef are supported.

	Function pointers are not supported (use 'string' type instead,
	see "Operators" below)

	Sial defines a 'string' types to handle ansi C strings within
	the interpreter. This string type also support some of the
	operator (+, ==, =, !=, etc... see below)

Variable Scope
--------------

	All symbols available in the system image become global
	variable in the sial context.

	Variable declared within sial can be given one of 3 different
	scopes like in normal C.

	GLOBAL:  A variable that is declared outside a
	function that is not marked as static. this variable if
	available to all macros that have been load into the
	interpreter.

	Ex:  file1:

	    int global; int func() { }

        file2:

            func2()
            {
			int i;

				i=global;
			}

	NB: since sial currently validates variable existence only at
	run time there is no need to declare a 'global' as an 'extern'
	variable. At run time, if none of the currently loaded macros
	define a global variable called 'global' then 'i=global' will
	fail with a 'unknown variable' error.

	FILE:  A Variable that is declared outside any
	functions that is tagged with the static keyword. This
	variables is available to all functions defined in the
	same macro file.

        Ex:

        file1:

		static int maxproc=100;
		sraruc int array;

		__init() 
		{
		int i;
		
			for(i=0;i<10;i++) array[i]=-1;
			
		}

		void func1()
		{
		int i;

				for(i=0;i<maxproc;i++) ...
		}
		int getmaxproc() { return maxproc; }

		__init() is used to initialized the static 'array'.
		Array is a dynamic array. See below.
		Both func1() and getmaxproc() use 'maxproc'.
		getmaxproc() makes the local static variable 'maxproc'
		available to other, external functions.

	BLOCK: A variable that is declared at the start of a statement
	block. This variable is available only within this statement
	block.

        Ex:
		func()
		{
		int i;

				if(some condition) {

						int i;

				}
		}

    FUNCTION PARAMETER:

        Same scope as BLOCK, for this function.

Storage classes
---------------

	In the context of sial the register or volatile storage classes
	do not make any sense and are ignored by the interpreter.

	Like in true C, a variable declared within a function will be
	automatic (stack) by default. The 'static' keyword can be used
	to make a variable persistent. Outside of any function, all
	variables are considered persistent and the 'static' keyword
	can be used to change the variable scope from 'global' to
	'file' (see above).


Operators
---------

        All standard C operators are supported.
        
        Sial support the following operators for the 'string' type.

        +, !=, <, >, <=, >=.

        examples:

        s = "hello" + "World";

        if("hello" == "world" ) { ... }

        The 'in' operator is supported on dynamic arrays.

        if(i in array) { ... }
        str=i in array ? "yes" : "no";

	Function callbacks
	------------------

	Function calls through function pointers is not possible
	currently. Instead, use a 'string' type to achieve the same
	result. When sial is about to perform a call, it will look at
	the type of the variable used to name the function. If the type
	is 'string' it will use the value that string and call that
	function instead.

        func0(int i, int j)
        {
                printf("i=%d j=%d\n", i,j);
        }
        func1(string func)
        {
                func(1,2);
        }
        main()
        {
                func1("func0");
        }

	In the above example, func1() ends up calling func0() not
	func.  This can be used as a callback mechanism, specially
	useful for creating generating function that call s linked list
	of objects and calls a variable function for each object. Like
	a function that walks tasks or procs.

	The sizeof() operator is evaluated at execution time.  So you
	can supply a type, a variable, or any valid expression and the
	appropriate size will be returned.  The expression *will* be
	executed, so be careful.

Statements
----------

        All C statements except 'goto' are supported.

        The 'for(i in array)' is supported on dynamic arrays.

Dynamic arrays 
-------------------------------

	When indexing through a non pointer variable you end up
	creating a dynamic array.

    Example:

        int func()
        {
        char *cp, c;
        int array, i;

                cp=(char *)symbol;
                c=cp[10];

                array[0]="one string";
                array[12]="second string";

                for(i in array) { 

                        printf("array[%d]=%s\n", i, array[i]);
                }
                
        }

	In the 'c=cp[10]' statement, sial goes to the system image to
	get one 'char' at the address symbol+10.

	In the second case, 'array' is not a pointer, it's a 'int'. So
	sial threats all indexing through it as dynamic.

	Additionally, sial supports multi levels of dynamic index,
	which makes possible to create random trees of indexed values:

        int func()
        {
        int array, i, j;

                array[10]="array10";
                array[10][3]="array10,3";
                array[20]="array20";
                array[20][99]="array20,99";

                for(i in array) {

                        printf("array[%d]=%s\n", i, array[i]);

                        for(j in array[i]) {

                              printf("array[%d][%d]=%s\n", i, j, array[i][j]);

                        }

                }
        }

	I think it is a good thing to have, since system image access
	and analysis require frequent lists search. So for example,
	with dynamic arrays, one can walk the proc list taking note of
	the proc*, then walking a user thread list taking note of the
	thread* and gathering some metrics for each of these threads.
	In order to get to these metrics at some later point in the
	execution, something like this could be used:

        func()
        {
        proc *p;

                for(p in procs) {

                        thread *t;

                        for(t in procs[p]) {

                                int rss, size;

                                /* we use index 0 for rss and 1 for size */
                                printf("proc %p, thread %p, rss:size = %d:%d\n"
                                      , p, t, procs[p][t][0], procs[p][t][1]);
                        }
                }
        }

	Arrays are always passed by reference. On creation the
	reference count is set to one. So this array will exist
	untill the variable it's assigned to dies.

	Arrays can be created by dso's. See the DSo section below for more
	information and examples of this.
	

Sial API
--------

	Sial can be integrated into any tool that needs to access
	symbol and type information from some object. Currently it is
	integrated in lcrash and icrash (tools that access Linux and
	Irix kernel images respectively), but it should be possible to
	use it, for example, in dbx or gdb. The API gives a simple
	interface through which the host application sends symbol and
	type (including member) information and gives access to the
	image itself so that sial can read random blocks of data from
	the image.

    >> sial_builtin(bt *bt)

	Install some set of builtin function. See below
	(builtin api).


    >> sial_chkfname(char *fname, int silent);

	Will check for the exsistance of a function in sial.
	Typically used to check xtra entry points before the
	application registers a new command (see sial_setcallback).

    >> sial_open():

	The first function that should be called is sial_open().
	sial_open() will return a value of 1 if everything is ok or 0
	in case of some problem. This call initializes internal date
	for the sial package.

    >> sial_setapi(apiops* ops, int nbytes):

	This function will setup the callbacks that sial will use 
	to get information from the application.

	See 'callback interface' below.

    >> sial_load(char *name);

	To have sial load and compile a macro or a set of macro
	use sial_load().  Parameter name gives the name of the
	file to compile. If name points to a directory instead,
	then all the files in this directory will be load.  So
	an application would call sial_load() when it first
	starts up specifying some well known files or
	directories to load. For example $HOME/.xxx and
	/etc/xxx would be loaded, ~/.xxx containing user
	defined macros, and /etc/xxx containing system macros.

    >> sial_unload(char *funcname)

	To unload the a macro file use this function.
	"funcname" is the name of any global function in the
	file you want to unload.

    >> void sial_setcallback(void (*scb)(char *));

	To be called prior to any load calls.
	After each loads, sial will call this function
	back with the name of each functions compiled.
	Typicly, the application will then perform checks
	and potencially install a new command for this
	function.

	ex:
	void
	reg_callback(char *name)
	{
	char fname[MAX_SYMNAMELEN+sizeof("_usage")+1];
	_command_t cmds[2];

        	snprintf(fname, sizeof(fname), "%s_help", name);
        	if(!sial_chkfname(fname, 0)) return;
        	snprintf(fname, sizeof(fname), "%s_usage", name);
        	if(!sial_chkfname(fname, 0)) return;

        	cmds[0].cmd=strdup(name);
        	cmds[0].real_cmd=0;
        	cmds[0].cmdfunc=run_callback;
        	cmds[0].cmdparse=parse_callback;
        	cmds[0].cmdusage=usage_callback;
        	cmds[0].cmdhelp=help_callback;
        	cmds[1].cmd=0;
        	unregister_cmd(cmds[0].cmd);
        	(void)register_cmds(cmds);
	}

    >> sial_setipath(char *path)

	When sial processes a #include directive it will use
	the specified path as a search path.
	The usual PATH format is supported ex:
	"/etc/include:/usr/include".

    >> sial_setmpath(char *path)

	When sial_load() is called with a relative path name or
	just the name of a file, it will use a search PATH to
	locate it. The path parameter to sial_set,path() sets
	this path. The usual PATH format is supported ex:
	"/etc/xxx:/usr/lib/xxx".

    >> sial_setofile(FILE *ofile)

       	All output of sial commands will be send to file ofile.

    >> sial_cmd(char *cmd, char **argv, int nargs)

	This is the way to execute a sial command that as been
	loaded.  'cmd' is the name of function to call.  'argv'
	are the argument to this function.  'nargs' is the
	number of argument in array 'argv'.

	Sial_cmd() will process argv and make the corresponding
	values available to the function by creating global
	variables that the function can test and use.

    >> sial_showallhelp()

	This command will send a complete list of the commands
	long with the usage and help for each one of them. This
	function should be called when the user request
	something like 'help all'.

    >> sial_showhelp(char *func)

	This will display the help information for a particular
	function loaded in sial.

The callback interface
----------------------

	Everytime sial needs some piece of information, it will call
	the application back for it. THe sial_apiset() function is used
	to install this callback interface into sial. Here is the list
	of callback functions:

        typedef unsigned long long ull;

	Sial_apiset() passes this structure to sial:

	typedef struct {

        	int (*getmem)(ull, void *, int);
        	int (*putmem)(ull, void *, int);
        	int (*member)(char *, ull, type * , member *);
        	int (*getctype)(int ctype, char * , type*);
        	char* (*getrtype)(ull, type *);
        	int (*alignment)(ull);
        	int (*getval)(char *, ull *);
        	enum_t* (*getenum)(char *name);
        	def_t*  (*getdefs)();

	} apiops;


    The apiops* struct defines the following member and function pointers:

    -getmem(ull addr, void *buffer, int nbytes)

	Read nbytes from image at virtual address addr (32 or
	64 bit) to buffer.

    -putmem(ull addr, void *buffer, int nbytes)

	Write nbytes from buffer to image at virtual address
	addr (32 or 64 bit).

    -member(char *name, ull pidx, type *tm, member *m);

	Get information on a structure member called name.
	Pidx is a unique type index for the parent structure.
	The getctype() function should fill in this index in
	it's type*. The dwarf model uses unique indexes (die
	offsets) that can be used here.  'tm' will hold
	information on the type of the member.  'm' will hold
	information on the member specific stuff (bit sizes,
	bit offset etc.).

	Use the sial_member_...() functions to setup m.
	Use the sial_type_...() functions to setup t.

    -getctype(int ctype, char *name, type *tout)

	Get type information for a complex type.  Ctype
	specifies that name is a type of type struct/union or
	enum.  tout contain the returned type information.

    -getrtype(ull idx, type *t)

	Gets the type string linked to a typedef. For example,
	the gettdeftype(<idx of type ull>) would return
	"unsigned long long". This enables sial to drill down a
	typedef (a typedef can be build from a typedef
	itself) in order to perform proper type validation for
	assignment or function parameters or return values.

    -getval(char *sname, ull *value)

	Returns the value of symbol "sname" from the image. The
	value is returned in 'value'.  On any image this is
	address of the symbol within the image itself, not the
	value of the symbol itself. See explanation of this
	above.

    -getenum(char *name);

	Return a list of enum values.
	Sial will make these available as symbol for the duration
	of the compile.

    -getdefs()

	Return a list of #defines to be active througout the
	sial session.

The builtin API
---------------

	Sometime it is necessary to create a C function that will
	handle some piece of the work, that a macro cannot do. Sial's
	builtin function are implemented this way. Generic function
	like 'printf' or 'getstr' can get some parameter input from the
	macros and do something (printf) or they get some information,
	map it to a sial value and return it to a macro (getstr).


	Sial can load new functiosn from DSOs. If the extension of
	a file name is ".so" then sial opens it and gets a list
	of function specs from it. Unload of that file will
	uninstall these functions.

	The API between the dso and sial is quite simple at this time.
	It has not been exercised as must as it would need to, so it
	might get more flexible and thus complex in the future.

	Here are two examples of simple extensions. 

	This is an example of a simple extension. An equivalent
	os the "hello world" C program, but this one gets 2 parameters
	, one int and one string and returns the received int.

	#include "sial_api.h"

	value *
	helloworld(value *vi, value *vs)
	{
	int i=sial_getval(vi);
	char *s=(char*)sial_getval(vs);

		sial_msg("Hello to the world![%d] s=[%s]\n", i, s);
		return sial_makebtype(1);
	}

	BT_SPEC_TABLE = {
		{ "int hello(int i, string s)",	helloworld},
		{ 0, 0}
	};

	static char *buf;

	BT_INIDSO_FUNC()
	{
		sial_msg("Hello world being initialized\n");
		buf=sial_alloc(1000);
		return 1;
	}

	BT_ENDDSO_FUNC()
	{
		sial_msg("Hello world being shutdown\n");
		sial_free(buf);
	}

	The BT_SPEC_TABLE is scanned. It's a simple table
	with 2 entries per functions and terminated with
	a NULL prototype.

	The DSO initializer function is called.
	If it returns 0 then installtion is terminates.
	If it returns 1 we proceed forward.

	The prototype is compiled and a syntax error
	will send the error message to the application
	output file (stdout usually).

	When the prototype as compiled with no errors 
	the function is installed and ready to be used from
	sial macros.

	Type checking is performed by sial at
	execution time on both, the function parameters
	andthe function return.

	DSO's can also receive, create and manipulate dynamic arrays.
	Here is an example of this:

	#include "sial_api.h"

	#ifdef ARRAY_STATIC
	static value *v;
	#endif

	value *
	mkarray(value* vi)
	{
	int i=sial_getval(vi);
	#ifndef ARRAY_STATIC
	value *v=sial_makebtype(0);
	#endif

		sial_msg("Received value [%d]\n", i);
		/* build an array indexed w/ int w/ 2 string values */
		sial_addvalarray(v, sial_makebtype(0)
			, sial_makestr("Value of index 0"));
		sial_addvalarray(v, sial_makebtype(2)
			, sial_makestr("Value of index 2"));
	#if ARRAY_STATIC
		/* 
	   	For a static array use :
		Then the array will persist until you Free it.
		*/
	   	sial_refarray(v, 1);
	#endif
		return v;
	}

	value *
	showstrarray(value* va)
	{
	value *v1=sial_strindex(va, "foo");
	value *v2=sial_strindex(va, "goo");

		printf("array[1]=%d\n", sial_getval(v1));
		printf("array[2]=%d\n", sial_getval(v2));
		sial_addvalarray(va, sial_makestr("gaa"), sial_makebtype(3));
		sial_addvalarray(va, sial_makestr("doo"), sial_makebtype(4));
		sial_freeval(v1);
		sial_freeval(v2);
		return sial_makebtype(0);
	}

	value *
	showintarray(value* va)
	{
	value *v1=sial_intindex(va, 1);
	value *v2=sial_intindex(va, 2);

		printf("array[1]=%d\n", sial_getval(v1));
		printf("array[2]=%d\n", sial_getval(v2));
		sial_freeval(v1);
		sial_freeval(v2);
		return sial_makebtype(0);
	}

	BT_SPEC_TABLE = {
		{ "int mkarray(int i)",	mkarray},
		{ "void showintarray(int i)",showintarray},
		{ "void showstrarray(int i)",showstrarray},
		{ 0, 0}
	};

	static char *buf;

	BT_INIDSO_FUNC()
	{
		sial_msg("mkarray initialized\n");
	#ifdef ARRAY_STATIC
		/* we will need a static value to attach the
		   array too */
		v=sial_makebtype(0);
	#endif
		return 1;
	}

	BT_ENDDSO_FUNC()
	{
		sial_msg("mkarray being shutdown\n");
	#ifdef ARRAY_STATIC
		sial_freeval(v);
		/* freing the value decrements the reference
		   count by one. So, if none of the calling
		   macros copied the value to a static
		   sial variable, it will free the array */
	#endif
	}

Macro Construction
------------------

	When sial as been integrated into an application and a basic
	set of builtin command as been created, it is time to start
	creating the macro themselves. Some basic rules and conventions
	apply to macro construction that make the coding and
	documenting steps of macro definition easy.

	I will use the function foo as an example. Function foo is
	defined in file /usr/tmp/sial/foo. Function foo is a user
	callable function, meaning that it can be executed by the
	sial_cmd() function. The command input section of the
	application can thus call sial_cmd("foo", char *argv, int
	nargs) to execute the foo macro.

	------------ file foo -------------

	foo_opt(){ return "ab:c"; }

	foo_usage(){ return "[-a] [-b barg] [-c] addr [addr [addr...]]"; }

	foo_help(){ return "This is an example function"; }

	static int
	doproc(proc_t *p)
	{
			printf("p=0x%p\n", p);
	}

	int foo()
	{
	int all, i;
	string barg;

			if(exists(aflag)) all=1;
			else all=0;

			if(exists("bflag")) bval=barg;

			for(i in argv) {

					proc_t *p;

					p=(proc_t*)atoi(argv[i], 16);

					doproc(p);

			}
	}

	------------ end of file foo --------------

	The application calls sial_load() to load foo. Sial calls
	back the application with the names of all fucntions declared
	in that file. The aplication can then register commands for
	the user to type according to this list of functions.
	In this case 'foo'.

	The application then uses sial_cmd() to run a specific
	command 'foo'.

	Before executing the command, sial checks if a foo_opt()
	function exists and if so, calls it. This function returns the
	proper getopt() argument specification string. If this function
	does not exists then all arguments are passed down to the foo()
	function directly.

	If the arguments supplied by the user do not follow the proper
	syntax then the function foo_usage() will be called, if it
	exists. If the foo_usage() function does not exists, a generic
	error message is generated by sial.

	If the command 'help foo' is issued, the application should be
	calling sial_exefunc("help_foo", 0) whish will return a VALUE_S
	for the help for foo. Or whatever foo_help() returns.

	Each option, their associated value and addition arguments are
	made available to the foo funtion by creating the following
	global variables before the actual call.

	Each option, if specified, will trigger the existence of flag
	variable. In the foo() case, this means that variables aflag,
	bflag and cflag can possibly exist. The function
	exists("variable name") can then be used to test for this
	option presence.

	If an option has an associated value (getopt's ':' is specified
	on the string foo_opt() returns) this value is made available
	as a string type variable called Xarg, where X  is the option
	letter. In the case of foo() variable 'string barg' would exist
	if the -b option was supplied by the user.

	The rest of the arguments supplied by the user are made
	available in an array of 'string' called argv. argv[0] is
	set to the name of the function 'foo' and argc is a global
	that defines how many argv their are.

Builtin functions
=================

	Here is a description of the current set of builtin functions.

	unsigned long long 
	atoi(string value [, int base])
	
		Convert a string value to a long long. Base is the base
		that should be used to process the string e.g. 8, 10 or
		16. If not specified, then the standard numeric format
		will be scnanned for ex: 

			0x[0-9a-fA-F]+ : hexadecimal
			0[0-7]+        : octal
			[1-9]+[0-9]*   : decimal
		
		This function is used when converting command line
		arguments to pointers.
		
		Example:
		
		void
		mycommand()
		{
		int i;
		
			for(i=1;i<argc;i++) {
			
				struct proc *p;
				
				p=(struct proc*)atoi(argv[i],16);
				
				...
			}
		}
		
		User types this at the lcrash prompt:
		
		>> mycommand 0xa80000004ab14578
		
	int exists(string name)

		Checks for the existance of a variable. Returns 1 if
		the variables does exist and 0 otherwise. This function
		is mostly used to test if some options were specified
		on when the macro was executed from command line.

		It can also be used to test for image variable.

		example:

		void
		mycommand()
		{
			if(exists("aflag")) {

				// user has specified -a option
			}
		}

	void exit()
	
		Terminate macro excution now.

	int getchar()

		Get a single character from tty.

	string gets()

		Get a line of input from tty.
	
	string getstr(void *)
	
		Gets a null terminated string from the image at the
		address specified. Sial will read a series of 16 byte
		values from the image untill it find the \0 character.
		Up to 4000 bytes will be read this way.
		
	string getnstr(void *, int n)
	
		Gets n characters from the image at the specified
		address and returns the corresponding string.

	string itoa(unsigned long long)
	
		Convert a unsigned long long to a decimal string.

	void printf(char *fmt, ...);

		Send a formatted message to the screen or output file.
		For proper allignment of output on 32 and 64 bit systems
		one can use the %> sequence along with the %p format.

		On a 32 bit system %p will print a 8 character hexadecimal
		value and on a 64 bit system it will print a 16 character
		value. So, to get proper alignment on both type of systems
		use the %> format which will print nothing on a 64 bit system
		but will print 8 times the following character on a 32 bit
		system.

		example:

		struct proc *p;

		  printf("Proc	%>	uid 	pid\n");
		  printf("0x%p		%8d	%8d\n"
			, p, p->p_uid,p_p_pid);

	int sial_depend(string file)

		Loads a macro or directory of macros called
		'file'. Contrary to sial_load() it will not
		give any error messages. Returns 1 on success
		0 otherwise.

	int sial_load(string file)
	
		Loads and compiles a sial macro file.
		returns 1 if successful or 0 otherwise.
		
	void sial_unload(string file)
	
		Unload's a sial macro file
		
	string sprintf(string format, ...)
	
		Creates a string from the result of a sprintf.
		Example:
		
		void
		mycommand()
		{
		
		string msg;
		
			msg=sprintf("i=%d\n", i);
		}
		The result will be truncated to maxbytes if it would be
		longer.
		
	int strlen(string s)
	
		Return the length of string s.
		
	string substr(string s, int start, int len)
	
		Creates a string from the substring starting a charcater
		'start' of 's' for 'len' characters.
		example:
		
			s=substr("this is the original", 6, 2);
			
			So 's' will become "is".


	--------------------------------------------------------

Questions/Comments
Luc Chouinard, lucc@sgi.com
