User Defined Extensions in GrADS v2.0

From OpenGrads Wiki
Jump to: navigation, search

Introduction

This will be the home for the documentation of the OpenGrADS User Defined Extensions in GrADS v2.0. At this point this documentation is merely a design document for these features. It is intended for developers only. As such, the software implementing the features mentioned in this document does not yet exist.

Background: User Defined Extensions as Dynamic Linked Libraries

User Defined Extensions in GrADS are loaded the same way web browsers load their plug-ins. It makes use of the dlopen, dlsym() and dlclose() function calls, a device introduced by Sun Microsystems back in the 1980's that are available in most modern platforms. Let's look at a simple example that loads the cos function from the standard C math library, libm.so:

#include <stdio.h>
#include <dlfcn.h>
int main(int argc, char **argv) {
 void *handle;
 double (*cosine)(double);
 char *error;

 /* Open the existing libray libm.so */
 handle = dlopen ("libm.so", RTLD_LAZY); /* use libm.dylib on a Mac */
 if (!handle) {
   fprintf (stderr, "%s\n", dlerror());
   exit(1);
 }
  /* Retrieve a pointer to the function "cos" provided by libm */
 dlerror();    /* Clear any existing error */
 *(void **) (&cosine) = dlsym(handle, "cos");
 if ((error = dlerror()) != NULL) {
   fprintf (stderr, "%s\n", error);
   exit(1);
 }
 /* Print result and close the library */
 printf ("%f\n", (*cosine)(0.2));
 dlclose(handle);
 return 0;
}

The function dlopen() opens a shared library, while dlsym() returns a function pointer given a string with the name of a function in the library; you can guess what dlclose() does. Do a man dlopen for information on the other parameters. The code fragment above opens the math library libm.so, get a pointer to the cosine function cos and executes it. If you save this code fragment in file test.c, you usually compile and run it like this

% gcc -o test test.c -ldl
% ./test
0.980067

Now, you do not have to rely on system libraries, you can write your own loadable libraries. Consider this simple implementation of the cosine function keeping the leading terms of its Taylor series:

double cos ( double x ) {
   return (double) ( 1.0 - x*x*(0.5 - x*x/24.) );
}

If one saves this code fragment to a file called mycos.c, on many sytems, including Linux, a shared library (dynamically linked library) can be create with the command line

% gcc -shared -o mycos.so mycos.c

The particular command for creating a shared library varies from system to system. On Mac OS X, the syntax is

% gcc -c mycos.c
% libtool -dynamic -o mycos.dylib mycos.o

The OpenGrADS build mechanism can produce shared libraries for the most common systems. However, if you need to port extensions to a new unsupported system, getting this simple example to work is a good starting point.

Now that you have your own implementation of the cosine function, all you need to do is to change one line in the test.c sample code above (in a real application you would read these names from a file). Simply replace the line

 handle = dlopen ("libm.so", RTLD_LAZY);

with this one

 handle = dlopen ("./mycos.so", RTLD_LAZY);

Recompile and run:

% gcc -o test test.c -ldl
% ./test
0.980067

This simple example illustrates how a user defined function can be incorporated in an application at run time. The name of the library and function does not have to be known at compile time, it could be read from an external file. This is precisely how we implemented UDXTs in GrADS. A text file, the so-called UDXT table, contains the name of shared libraries and functions within. GrADS loads this table at start up, but loads an extension only upon the first usage. We ellaborate on these points next.

Developing User Defined Extensions

This section present some Use Cases and a proposed API for writing User Defined Extensions as dynamic linked libraries. Unlike the OpenGrADS implementation in GrADS v1.9.0-rc1, here we introduce a more robust API which isolates the user from the internals of the GrADS C application. For backward compatibility, a legacy Low-level API for Writing Extensions in GrADS v2.0 also exists, but users are strongly discouraged from using this API.

Writing User Defined Functions in C: User-level API

For each User Defined Function (UDF), the user provides a C function with a standard interface:

#include "gex.h"
int f_hello (void *gex, int argc, char **argv);

where gex.h is an include file supplied to the user, and gex is an opaque pointer that should not be directly manipulated by the user (see below). In addition, for each UDF, the user provides an entry in a User Defined Extension" (UDX) table which tells GrADS where to look for the UDF.

#  Type  API   Name      Function     Library               Description
# ------ --- ----------  ---------  -------------- --------------------------------- 
   udf    1    hello     f_hello    ^libhello.gex  "Short description of the function"

The Name (hello in this example) is the name of the function to be used in a GrADS expression; Function is the name of the C-function (f_hello in this case) inside the dynamically linked library (DLL) Library. The API version (1 in this example) is used to denote a particular version of the UDF API, allowing for future extensions.

The optional caret (^) in this example is useful to simplify installation: it tells GrADS that the DLL is found at the same directory where the User Defined Extension table is located, freeing the user from having to set environment variables such as LD_LIBRARY_PATH. The Description is used for producing a short summary in response to query udx.

The User Callable C-API for writing GrADS Extensions

The include file gex.h provides prototypes for some very basic callbacks functions that can be used to write the UDF. In particular, the user can evaluate GrADS expressions and store the final result of the function:

/* Evaluate GrADS expression */
int gex_eval (void *gex, char *expr, var_t *var);
/* Set result for a function */
int gex_store (void *gex, var_t *var);

where gex is an opaque pointer supplied to the user on input to the UDF; the user is not supposed to directly manipulate the contents of the pointer gex. The data structure var_t contains basic information about a variable, namely:

/* Variable data and metadata */
  typedef struct {
    int    type;   /* Variable type: grid of station */
    int    size;   /* size of data/mask */
    double *data;  /* data array */
    char   *mask;  /* undef mask */
    meta_t *meta;  /* variable metadata data */
  } var_t;

where meta_t is a data structure with information about the dimension environment and coordinate variables:

  /* Variable metadata */
  typedef struct {
    int idim, jdim; /* dimensions associated with each axis */
    int isiz, jsiz; /* axis sizes */
    int iflg, jflg; /* linear scaling flag or each axis */
    int x[2], y[2], z[2], t[2], e[2];   /* index ranges */
    double *lat, *lon, *lev, *ens;      /* world coordinates   */
    char   *time[6];                    /* [cc,yy,mm,dd,hh,nn] */
  } meta_t;

Notice that the data structure var_t contains basically the same type of information that was available through the transfer files in the classic UDFs of GrADS v1.9 and earlier.

In addition, the UDF developer can have the same control over GrADS as it woud be possible from the GrADS scripting language:

/* Execute a generic GrADS command */
int gex_cmd   (void *gex, char *cmd); 
int gex_rline (void *gex, int i, char *line); 
int gex_rword (void *gex, int i, int j, char *word);
Design Considerations
1. The input arguments argc/argv have been modeled after the familiar C main program, with an API being provided for the user to evaluate GrADS expressions as needed. The alternative is to evaluate the expressions inside the GrADS applicaton and pass the input var_t structure to the user code. It was felt that letting the user evaluate the expressions on demand could bring about huge memory savings in some cases.
2. It has been left to the user to verify the appropriateness of the input parameters given to the function. This simplifies the syntax of the UDX table, and also implementation of the interface to the main GrADS code.
3. One might consider providing an interface to the gaprnt() function.

Use Case 1: Hello World UDF in C

C code implementation:

#include "gex.h"
int f_hello (void *gex, int argc, char **argv) {
   int rc;
   var_t *var;

   printf("Hello, GrADS v2.0 World!\n");

   rc = gex_eval(gex,"0",var); /* return a zero grid  */
   if (rc) return rc;
   rc = gex_store(gex,var);
   return (rc);
}

UDX Table (file: hello.udxt)

#  Type  API     Name            Function                 Library
# ------ --- ---------------  ----------------  ---------------------------  
   udf    1    hello             f_hello         ^hello.gex

GrADS example:

% export GAUDXT=/path/to/hello.udxt
% grads 
ga-> open model.ctl
ga-> d hello()
Hello, GrADS v2.0 World!
Result value = 0

Use Case 2: Sqrt UDF in C

C code implementation (omitting error messages):

#include "gex.h"
int f_sqrt (void *gex, int argc, char **argv) {
   int i, rc;
   var_t *var;

   /* Evaluate the input expression */
   if ( argc !=1 ) return 1;
   rc = gex_eval(gex,argv[1],var); 
   if (rc) return rc;

   /* Evaluate the sqrt() */
   for (i=0; i<var.size; i++) {
     if (var.mask[i]) { /* defined? */
       if (var.data[i]<0.0) 
           var.mask[i] = 0; /* mark as undefined */
       else                 
           var.data[i] = sqrt(var.data[i]);
     }
   }

   /* Store the results */
   rc = gex_store(gex,var);
   return (rc);
}

UDX Table (file: sqrt.udxt)

#  Type  API     Name            Function                 Library
# ------ --- ---------------  ----------------  ---------------------------  
   udf    1    mysqrt             f_sqrt         ^sqrt.gex

GrADS example:

% export GAUDXT=/path/to/sqrt.udxt
% grads 
ga-> open model.ctl
ga-> d mysqrt(ts)
Contouring: 15.3 to 17.7 interval 0.3

Use Case 3: Sqrt UDF in Fortran

Consider this fortran subroutine which implements the core of the sqrt() calculation in Fortran:

subroutine ftnSqrt(sqrt_a,a,mask,n)
   integer,           intent(in)  :: n
   real(kind=8),      intent(in)  :: a(n)
   character(len=1),  intent(in)  :: mask(n)
   real(kin=8),       intent(out) :: sqrt_a(a)  
   where(mask==ichar(1) .and. a >= 0)
       sqrt_a = sqrt(a)
   elsewhere
       mask = ichar(0)
   endif
end subroutine ftn_sqrt

In order to interface this to GrADS you need this C code wrapper:

#include "gex.h"
void ftnsqrt_ ( float b[], float a[], char mask[], int n);
int f_sqrt (void *gex, int argc, char **argv) {
   int i, rc;
   var_t *var;
 
   /* Evaluate the input expression */
   if ( argc !=1 ) return 1;
   rc = gex_eval(gex,argv[1],var); 
   if (rc) return rc;
    
   /* Calculate the sqrt using fortran */ 
   ftnsqrt_(var.data,var.data,var.mask,var.size);
 
   /* Store the results */
   rc = gex_store(gex,var);
   return (rc);
}

UDX Table (file: sqrt.udxt)

#  Type  API     Name            Function                 Library
# ------ --- ---------------  ----------------  ---------------------------  
   udf    1    ftnsqrt             f_sqrt         ^sqrt.gex

GrADS example:

% export GAUDXT=/path/to/sqrt.udxt
% grads 
ga-> open model.ctl
ga-> d ftnsqrt(ts)
Contouring: 15.3 to 17.7 interval 0.3

Implementation details for developers

This section is intended for developers working with the main GrADS code base. The core of the UDX interface layer is contained in 2 files:

The interface to the main GrADS code occurs in 3 places.

grads.c
During initialization in the main program,, right before the command line loop, a simple call is made to the 'gaudx package:
 gaudxi(&gcmn); /* Initialize User Defined Extensions */
gafunc.c
After the built-in functions have been searched the following call is made to attempt to resolved the function with a UDF:
   ...
   if (cmpwrd("tregr",name)) fpntr = fftreg;
   if (cmpwrd("s2g1d",name)) fpntr = ffs2g1d;
   if (cmpwrd("lterp",name)) fpntr = fflterp;

   if (fpntr==NULL)
      *(void **) &fpntr = (void *) gaudf(name);
gauser.c
If User Defined Commands' are desired, the following call is introduced gacmd() right before the main dispatch block:
 ...
 retcod = (int) gaudc (com,pcm);
 if ( retcod >= 0 ) goto retrn; /* found a udc, all done */
 else retcod = 0;               /* not found, keep going */
 if (!(cmpwrd("clear",cmd) || cmpwrd("c",cmd))) gxfrme (9);
 if (*com=='\0' || *com=='\n') goto retrn;
 ...

The callbacks are implemented in files

Building User Defined Extensions from Sources

You do not need to build GrADS to build the extensions, it suffices to have the include file file gex.h. However, the most convenient way is to check out module Grads from the OpenGrADS repository (notice the capital "G"), which includes the standard GrADS sources as well as the extra extensions directory:

 % cd workspace
 % cvs -d ... co -P Grads  # Notice capital "G"

For compiling and creating a tarball with your extensions enter

 % make gex-dist

For simply building the sample extensions,

 % cd extensions
 % make
 % make install bindir=/path/to/bin/dir

You can also build from the individual directories,

 % cd hello
 % make

The GNUmakefiles for the individual packages include the fragment gex.mk and does not yet make use of automake. Future enhancements include compilers wrappers such as gexcc, gexf77 and gexf90.

For adding your own packages, create a directory under extensions/ and copy over the GNUmakefile from one of the other packages to your newly created directory. The hello package is an excellent place to start.

Installation

First download and install GrADS itself. The GrADS executables are typically placed in the directory /usr/local/bin. The libraries contained in this tar file are typically placed in the directory /usr/local/bin/gex. If you do not have write permission for your /usr/local/bin directory, you can put them in the ~/bin/gex subdirectory of your home directory or any other directory of your choice. Henceforth, we will refer to the GrADS installation directory as $GABIN, and assume that you install the dynamic extensions in subdirectory gex/.

Whatever location you choose to install the UDXTs you must set the environment variable GAUDXT to point to the location of your User Defined Extension Table. To use the one in this distribution set

  export GAUDXT=$GABIN/gex/udxt    (sh, ksh, bash)
  setenv GAUDXT $GABIN/gex/udxt    (csh, tcsh)

You may want to familiarize yourself with the contents of the file "$GABIN/gex/udx" and comment out any function that may cause any conflict to you.

Unlike the OpenGrADS UDX implementation in GrADS v1.9.0-rc1, if you keep your shared libraries in the same directory as the your UDX table there is no need to set the environment variables LD_LIBRARY_PATH/DYLD_LIBRARY_PATH. This is strongly recommended.

Verifying your installation

Start GrADS and enter

  ga-> query udx

To see a list of all your user defined commands and functions. You may want to try

  ga-> hello

which should print "Hello, World" to your screen.