Skip to content

Latest commit

 

History

History
1597 lines (1237 loc) · 59 KB

dfa.md

File metadata and controls

1597 lines (1237 loc) · 59 KB

The DFA package [ver. 0.5.7]

The DFA (a.k.a. Dynamic Function Array) package implements:

  • dynamic numeric and character arrays,
  • dynamic stacks (filo),
  • dynamic queues (fifo),
  • dynamic ordered stacks,
  • priority queues,
  • bitmap.

The package provides a set of macros, which allows to generate call routines simulating data structures mentioned above.

Few exemplary functions are also generated. See particular macro help for further details.

Package contains:

  1. macro createdfarray
  2. macro createdharray
  3. macro createdhfifo
  4. macro createdhordstack
  5. macro createdhprtqueue
  6. macro createdhstack
  7. proto bit64andprotodfa
  8. proto bit64orprotodfa
  9. functions bit64anddfa
  10. functions bit64ordfa
  11. macro createdfbitmap
  12. exec generatearrays
  13. clean generatearrays

SAS package generated by generatePackage, version 20231111

The SHA256 hash digest for package DFA: F*012375D87F66EB3A7BF5DDD0CC5AEE28851733EE33CC63231DF9045BEB958168

Content description

>>> %createDFArray() macro: <<<

The %createDFArray() macro allows to generate a dynamic function array which is a FCMP based approach to create dynamically allocated numeric array with possible values searching and WHICHN() function emulation.

Note: Arrays provided by the macro are one dimensional arrays.

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

%createDFArray( 
  arrayName           
 <,debug=0>
 <,simple=0>
 <,resizeFactor=0>
 <,outlib=work.DFAfcmp.package>
 <,hashexp=13>
 <,header=1>
)

Arguments description:

  1. arrayName - Required, creates a FCMP call subroutine which is also an array name. In the data step it is used in form of a call subroutine, e.g. call arrayName("Allocate", -3, 3). Has to satisfy FCMP function naming requirements, but with maximum of 24 characters.

  • debug= - Optional, the default value is 0. If set to 1 then it turns on a debugging mode.

  • simple= - Optional, the default value is 0. A simple dynamic function array is one which is not searchable and does not allows to use which() functionality. If set to 1 then it disables SEARCH and WHICH functionality. See examples below for details.

  • resizeFactor= - Optional, the default value is 0. If set to 0 then the dynamic array size is not changeable(mutable) after initial size allocation. If set not to 0 then arrays dimensions are mutable after allocation, i.e. even if an array is allocated for ten elements (like A[1:10]) you can do A[17] = 42 and it will resize itself dynamically. Hint! Set to, e.g. 4999, for faster allocation process. See examples below for details.

  • outlib= - Optional, the default value is work.DFAfcmp.package. It points the default location for new generated dynamic function arrays compiled by FCMP. Hint! Keep it as it is.

  • hashexp= - Optional, the default value is 13. It is the default hashexp= value for internal hash table used by the function.

  • header= - Optional, the default value is 1. Indicates if the proc fcmp outlib = &outlib.; header is added to the executed code. If not 1 then no header is added.

Created function arguments description:

A function generated by the macro is:

call &arrayName.(IO, position, value)

and accepts the following list of arguments and values:

  1. IO - is a character steering argument, possible values and behaviour they call are the following:
  • O, Output, R, Return - to get the data from an array,
  • I, Input - to insert the data into an array,
  • +, Add - to increment given position by a value,
  • C, Clear - to reduce an array to a single empty cell,
  • A, Allocate - to reserve space for array width and set starting values,
  • D, Dimension - to return minimal and maximal index of an array,
  • F, Find, Exist - to find out if a given value exist in an array,
  • W, Which - to search the first position of data in array, WHICHN() function emulator,
  • Sum - to return the sum of non-missing elements of an array,
  • Nonmiss - to return the number of non-missing elements of an array,
  • Avg, Mean, Average - to return the average of non-missing elements of an array,
  • Min, Minimum - to return the minimum of non-missing elements of an array,
  • Max, Maximum - to return the maximum of non-missing elements of an array.
  1. position - is a numeric argument and depends on the IO value. Behaves in the following way:
  • for O, Output, R, Return/ I, Input/ +, Add it takes an arrays index from (into) which data is get (put),
  • for C, Clear is ignored,
  • for A, Allocate sets the value of the minposition, i.e. the minimal position of the array index,
  • for D, Dimension it returns value of the minposition,
  • for Sum, Nonmiss, Avg, Mean, Average, Min, Minimum, Max, and Maximum is ignored,
  • for F, Find, Exist it returns number of occurrences of a given value in an array,
  • for W, Which it returns position the first occurrence of a given value in an array.

.3 value - is a numeric argument and depends on the IO value. Behaves in the following way:

  • for O, Output, R, Return it holds value retrieved from an array on a given position,
  • for I, Input it holds the value inserted into an array on a given position,
  • for +, Add it holds the value that is used to increment an array value at a given position,
  • for C, Clear is ignored,
  • for A, Allocate it sets the value of the maxposition, i.e. maximal position of the array index,
  • for D, Dimension it returns value of the maxposition
  • for Sum, Nonmiss, Avg, Mean, Average, Min, Minimum, Max, and Maximum it returns the calculated summary value,
  • for F, Find, Exist, W, Which is the value to be searched in an array.

The position and the value arguments are outargs, i.e. can be changed by the function.

EXAMPLES AND USECASES:

EXAMPLE 1. Dynamic, Searchable, and Immutable array:

  %createDFArray(ArrDSI);
  options APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example1;
    call ArrDSI("Allocate", 1, 10);
    L = 0; H = 0;
    call ArrDSI("Dim", L, H);
    put L= H=;

 * populate array with data ;
    do i = L to H;
      call ArrDSI("Input", i, i**2); 
    end;

 * searchability allows to find number of occurrences of value in the array ;
    F = .;
    call ArrDSI("Find", F, 16);
    put "Value 16 occurs " F "times";
    call ArrDSI("Find", F, 17);
    put "Value 17 occurs " F "times";

 * increase value of cell 4 by 1, and verify at WHICH position is 17 (by searchability);
    call ArrDSI("+", 4, 1);
    call ArrDSI("Which", F, 17);
    put "Value 17 occurred for the first time at position " F;
  
 * get values from the array ;
    Value = .;
    do i = L to H;
      call ArrDSI("Output", i, Value); 
      put i= Value=;
    end;

 * some basic statistics ;
    call ArrDSI("Sum", ., STAT); put "sum = " STAT;
    call ArrDSI("Avg", ., STAT); put "avg = " STAT;
    call ArrDSI("Min", ., STAT); put "min = " STAT;
    call ArrDSI("Max", ., STAT); put "max = " STAT;
    call ArrDSI("Cnt", ., STAT); put "cnt = " STAT;

 * immutability does _not_ allow to increase dimensions automatically;
 * this line returns an error ;
    call ArrDSI("Input", 42, -1); 
  run;

EXAMPLE 2. Dynamic, Searchable, and Mutable array:

  %createDFArray(ArrDSM, resizefactor=17);
  options APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example2;
    call ArrDSM("Allocate", -2, 2);
  
    do i = -2 to 2;
      call ArrDSM("Input", i, 2**i);
    end;
    
    L = .; H = .;
    call ArrDSM("Dim", L, H);
    put L= H=;

 * mutability allows to increase dimensions automatically
 * create index 3 and -3;
    call ArrDSM("+", 3, 8);
    call ArrDSM("+",-3, 0.125);
    call ArrDSM("Dim", L, H);
    put L= H=;
    
    Value = .;
    do i = L to H;
      call ArrDSM("O", i, Value);
      put i= Value=;
    end;

  run;

EXAMPLE 3. Dynamic, non-searchable (a.k.a. SiMPle), and Immutable array:

  %createDFArray(ArrDSMPLI, simple=1);
  options APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example3;
    call ArrDSMPLI("Allocate", -2, 2);
  
    do i = -2 to 2;
      call ArrDSMPLI("Input", i, 2**i);
    end;

 * non-searchable array (a.k.a. simple) does not allow ;
 * to find number of occurrences of value in the array ;
 * and verify what is the first position of a value ;
 * this lines return a warning ;
    call ArrDSMPLI("Exist", i, 1);
    call ArrDSMPLI("Which", i, 1);
  run;

EXAMPLE 4. Dynamic, non-searchable (a.k.a. SiMPle), and Mutable array:

  %createDFArray(ArrDSMPLM, simple=1, resizefactor=42);
  options APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example4;
    call ArrDSMPLM("Allocate", 1, 1);
  
 * mutability allows to increase dimensions automatically ;
    do i = -12 to 12;
      call ArrDSMPLM("Input", i, i*2);
    end;

 * non-searchable array (a.k.a. simple) does not allow ;
 * to find number of occurrences of value in the array ;
 * and verify what is the first position of a value ;
 * this lines return a warning ;
    i = .;
    call ArrDSMPLM("Exist", i, -24);
    put "Exist " i=;
    call ArrDSMPLM("Which", i,  24);
    put "Which " i=;
  run;

>>> %createDHArray() macro: <<<

The %createDHArray() macro allows to generate a dynamic hash array which is a FCMP based approach to create dynamically allocated numeric or character array

Note: Arrays provided by the macro are one dimensional arrays.

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

%createDHArray( 
  arrayName           
 <,debug=0>
 <,type=8>
 <,outlib=work.DFAfcmp.package>
 <,hashexp=13>
 <,header=1>
)

Arguments description:

  1. arrayName - Required, creates a FCMP call subroutine which is also an array name. In the data step it is used in form of a call subroutine, e.g. call arrayName("Allocate", -3, 3). Has to satisfy FCMP function naming requirements, but with maximum of 24 characters.

  • debug= - Optional, the default value is 0. If set to 1 then it turns on a debugging mode.

  • type= - Optional, the default value is 8. Indicates what type (numeric/character) and length are data portion of generated array. Should be in line with the LENGTH statement, e.g. 8, $ 30, etc. Determines if the value argument is numeric or character.

  • outlib= - Optional, the default value is work.DFAfcmp.package. It points the default location for new generated dynamic function arrays compiled by FCMP. Hint! Keep it as it is.

  • hashexp= - Optional, the default value is 13. It is the default hashexp= value for internal hash table used by the function.

  • header= - Optional, the default value is 1. Indicates if the proc fcmp outlib = &outlib.; header is added to the executed code. If not 1 then no header is added.

Created function arguments description:

A function generated by the macro is:

call &arrayName.(IO, position, value)

and accepts the following list of arguments and values:

  1. IO - is a character steering argument, possible values and behaviour they call are the following:
  • O, Output, R, Return - to get the data from an array,
  • I, Input - to insert the data into an array,
  • C, Clear - to reduce an array to a single empty cell,
  • L, Low, Lower, Lbound - to return minimal position of index,
  • H, High, Higher, Hbound - to return maximal position of index.
  1. position - is a numeric argument and depends on the IO value. Behaves in the following way:
  • for O, Output, R, Return/ I, and Input it is an array index from (into) which data is get (put),
  • for C it is ignored,
  • for L, Low, Lower, and Lbound it returns the first position of an index,
  • for H, High, Higher, and Hbound it returns the last position of an index,
  • otherwise is not modified.
  1. value - is a numeric or character argument (determined by the type=) and depends on the IO value. Behaves in the following way:
  • for O, Output, R, and Return it holds value retrieved from an array from a given position,
  • for I, Input it holds the value inserted into an array into a given position,
  • for C is ignored,
  • for L, Low, Lower, and Lbound returns first value of index,
  • for H, High, Higher, and Hbound returns last value of index,
  • otherwise is not modified.

The position and the value arguments are outargs, i.e. can be changed by the function.

EXAMPLES AND USECASES:

EXAMPLE 1. Dynamic, Hash-based, and Character array:

  %createDHArray(ArrDHC, type = $ 12); 
  options APPEND=(cmplib = WORK.DFAfcmp) ; 
 
  %let zeros = 6; *[to test bigger sizes];
  data Example1; 
   
    t = time(); 
    do _I_ = -1e&zeros. to 1e&zeros.; 
      _X_ = put(_I_*10, z12.); 
      call ArrDHC("Input", _I_, _X_); 
    end; 
    t = time() - t; 
    put t= / _X_= /; 
   
 * get the size info ; 
    LB = 0; HB = 0; 
    drop LB HB; 
      call ArrDHC('Lbound', LB, _X_); 
      call ArrDHC('Hbound', HB, _X_); 
    put LB= HB= /; 
   
    t = time(); 
    do _I_ = HB + 1 to LB - 1 by -1; 
      call ArrDHC('Output', _I_, _X_); 
      output;  
    end; 
    t = time() - t; 
    put t= / _X_= /; 
    
 * clear for further reuse ; 
    call ArrDHC('C', ., ''); 
    
  run;

EXAMPLE 2. Dynamic, Hash-based, and Numeric array:

  %createDHArray(ArrDHN);
  options APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example2;
  
    do i = -2 to 2;
      call ArrDHN("Input", i, 2**i);
    end;

    do i = -2 to 2;
      call ArrDHN("+", i, -10);
    end;
    
    v = .;
    do i = -2 to 2;
      call ArrDHN("Output", i, v);
      put i= v=;
    end;

  run;

>>> %createDHFifo() macro: <<<

The %createDHFifo() macro allows to generate a dynamic hash fifo which is a FCMP based approach to create dynamically allocated numeric or character "first in first out" queue

Interesting reading about implementing a fifo via hash table can be found in chapter 10.4 of the: "Data Management Solutions Using SAS Hash Table Operations: A Business Intelligence Case Study" book by Paul Dorfman and Don Henderson.

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

%createDHFifo( 
  fifoName           
 <,debug=0>
 <,type=8>
 <,outlib=work.DFAfcmp.package>
 <,hashexp=13>
 <,header=1>
)

Arguments description:

  1. fifoName - Required, creates a FCMP call subroutine which is also a fifo name. In the data step it is used in form of a call subroutine, e.g. call fifoName("Enqueue", 3). Has to satisfy FCMP function naming requirements, but with maximum of 24 characters.

  • debug= - Optional, the default value is 0. If set to 1 then it turns on a debugging mode.

  • type= - Optional, the default value is 8. Indicates what type (numeric/character) and length are data portion of generated array. Should be in line with the LENGTH statement, e.g. 8, $ 30, etc. Determines if the value argument is numeric or character.

  • outlib= - Optional, the default value is work.DFAfcmp.package. It points the default location for new generated dynamic function arrays compiled by FCMP. Hint! Keep it as it is.

  • hashexp= - Optional, the default value is 13. It is the default hashexp= value for internal hash table used by the function.

  • header= - Optional, the default value is 1. Indicates if the proc fcmp outlib = &outlib.; header is added to the executed code. If not 1 then no header is added.

Created function arguments description:

A function generated by the macro is:

call &fifoName.(IO, value)

and accepts the following list of arguments and values:

  1. IO - is a character steering argument, possible values and behaviour they call are the following:
  • O, Output, D, Dequeue, R, Return - to get the data from a fifo (and remove it from the fifo)
  • I, Input, E, Enqueue, and Insert - to insert the data into a fifo
  • C, Clear - to reduce a fifo to an empty one
  • P, Peek, T, and Tail - to peek the data from a fifo (and NOT remove it from the fifo)
  • H, Head - to peek the data from a fifo head (and NOT remove it from the fifo)
  • Sum - returns sum of nonmissing numeric elements of a stack
  • Avg, Mean, Average - returns average of nonmissing numeric elements of a stack
  • Nonmiss, Cnt - returns number of nonmissing elements of a stack
  • Height - returns height a stack
  1. value - is a numeric or character argument (determined by the type=) and depends on the IO value. Behaves in the following way:
  • for O, Output, D, Dequeue, R, Return it holds the value popped from the fifo,
  • for I, Input, E, Enqueue, Insert it holds the value to be pushed into the fifo,
  • for C, Clear it is ignored,
  • for P, Peek holds the value peeked from the fifo,
  • for Sum, Nonmiss, Cnt, Avg, Mean, Average, and Height it returns calculated summary value.

The value argument is outarg, i.e. can be changed by the function.

EXAMPLES AND USECASES:

EXAMPLE 1. Dynamic, Hash-based, and Character fifo:

  %createDHFifo(FifoDHC, type = $ 12); 
  options APPEND=(cmplib = WORK.DFAfcmp) ; 

 
  %let zeros = 6; *[to test bigger sizes];
  data Example1;
 
    t = time(); drop t; 
    do _I_ = 1 to 1e&zeros.; 
      _X_ = put(_I_*10, z12.); 
      call FifoDHC("Enqueue", _X_); 
    end; 
    t = time() - t;
 
    call FifoDHC("Height", _X_); 
    put t= / _X_=; 
   
    t = time(); 
    do _I_ = 1 to 1e&zeros. + 3; 
      call FifoDHC('Dequeue', _X_); 
      output;  
    end; 
    t = time() - t; 

    call FifoDHC("Height", _X_); 
    put t= / _X_=; 
 
    %* clear for further reuse *; 
    call FifoDHC('Clear', '');  

  run;

EXAMPLE 2. Dynamic, Hash-based, and Numeric fifo:

  %createDHFifo(FifoDHN); 
  options APPEND=(cmplib = WORK.DFAfcmp) ; 

  data Example2;
 
    do _I_ = 1,.,2,.,3,.,4,.,5,.,6;  
      call FifoDHN("E", _I_); 
    end; 

    call FifoDHN("Sum", _I_); 
    put "Sum    " _I_=;

    call FifoDHN("Avg", _I_); 
    put "Avg    " _I_=;

    call FifoDHN("Cnt", _I_); 
    put "Cnt    " _I_=;
   
    call FifoDHN("Height", _I_); 
    put "Height " _I_=; 

    call FifoDHN("Tail", _I_); 
    put "Tail of fifo is " _I_=; 

    call FifoDHN("Height", _I_); 
    put "Height after Tail " _I_=; 

    call FifoDHN("Head", _I_); 
    put "Head of fifo is " _I_=; 

    call FifoDHN("Height", _I_); 
    put "Height after Head" _I_=; 

    _X_ = 0;
    do _I_ = 1 to _I_; 
      call FifoDHN('D', _X_); 
      output;  
    end; 

    call FifoDHN("Height", _I_); 
    put "Height " _I_=;

  run;

>>> %createDHOrdStack() macro: <<<

The %createDHOrdStack() macro allows to generate a dynamic ORDERED hash stack which is a FCMP based approach to create dynamically allocated numeric or character ordered stack

Interesting reading about implementing a stack via hash table can be found in chapter 10.4 of the: "Data Management Solutions Using SAS Hash Table Operations: A Business Intelligence Case Study" book by Paul Dorfman and Don Henderson.

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

%createDHOrdStack( 
  fifoName           
 <,debug=0>
 <,type=8>
 <,order=A>
 <,outlib=work.DFAfcmp.package>
 <,hashexp=13>
 <,header=1>
)

Arguments description:

  1. stackName - Required, creates a FCMP call subroutine which is also a stack name. In the data step it is used in form of a call subroutine, e.g. call stackName("Push", 3). Has to satisfy FCMP function naming requirements, but with maximum of 24 characters.

  • debug= - Optional, the default value is 0. If set to 1 then it turns on a debugging mode.

  • type= - Optional, the default value is 8. Indicates what type (numeric/character) and length are data portion of generated array. Should be in line with the LENGTH statement, e.g. 8, $ 30, etc. Determines if the value argument is numeric or character.

  • order= - Optional, the default value is A. Indicates a method of ordering of the stack, allowed values are: A for ascending and D for descending.

  • outlib= - Optional, the default value is work.DFAfcmp.package. It points the default location for new generated dynamic function arrays compiled by FCMP. Hint! Keep it as it is.

  • hashexp= - Optional, the default value is 13. It is the default hashexp= value for internal hash table used by the function.

  • header= - Optional, the default value is 1. Indicates if the proc fcmp outlib = &outlib.; header is added to the executed code. If not 1 then no header is added.

Created function arguments description:

A function generated by the macro is:

call &stackName.(IO, value)

and accepts the following list of arguments and values:

  1. IO - is a character steering argument, possible values and behaviour they call are the following:
  • O, Output, Pop, G, Get, R, Return - to get the data from a stack (and remove it from the top),
  • I, Input, Push, Put, Insert - to insert the data into a stack,
  • C, Clear - to reduce a stack to an empty one,
  • P, Peek - to peek the data from a stack (and NOT remove it from the top),
  • Sum - returns sum of non-missing numeric elements of a stack,
  • Avg, Mean, Average - returns average of non-missing numeric elements of a stack,
  • Nonmiss, Cnt, Nnm - returns number of non-missing elements of a stack,
  • Height - returns height a stack,
  • Min, Minimum - returns minimum of non-missing elements of a stack,
  • Max, Maximum - returns maximum of non-missing elements of a stack.
  1. value - is a numeric or character argument (determined by the type=) and depends on the IO value. Behaves in the following way:
  • for O, Output, Pop, G, Get, R, Return it holds a value popped from a stack,
  • for I, Input, Push, Put, Insert it holds a value to be pushed into a stack,
  • for C, Clear it is ignored,
  • for P, Peek it holds a value peeked from a stack,
  • for Sum, Nonmiss, Cnt, Avg, Mean, Average, Height, Min, Minimum, Max, and Maximum it returns calculated summary value,

The value argument is outarg, i.e. can be changed by the function.

EXAMPLES AND USECASES:

EXAMPLE 1. Dynamic, Hash-based, and Character Descending Ordered stack:

  %createDHOrdStack(DescStackC, type = $ 12, order=D); 
  options APPEND=(cmplib = WORK.DFAfcmp) ;
 
  data Example1; 
    
    do _X_ = "A","B"," ","C","A"," ","B","C"; 
      call DescStackC("Push", _X_); 
    end; 
  
    length S $ 12;
    call DescStackC('Height', S); 
    put 'Height ' S;

    do until(strip(S) = "0"); 
      call DescStackC('Get', _X_); 
      call DescStackC('Height', S);
      put S= _X_=;
      output;
    end; 
    
    %* clear for further reuse *; 
    call DescStackC('Clear',''); 

  run;

EXAMPLE 2. Dynamic, Hash-based, and Numeric Ascending Ordered stack:

  %createDHOrdStack(DescStackN, order=A); 
  options APPEND=(cmplib = WORK.DFAfcmp) ;
 
  data Example2; 
   
    call missing(Sum, Avg, Min, Max, Cnt, Hgt, Peek);
    do _X_ = 1,6,2,.,5,3,4; 
      call DescStackN("Put", _X_); 
      call DescStackN('Sum', Sum);
      call DescStackN('Avg', Avg);
      call DescStackN('Min', Min);
      call DescStackN('Max', Max);
      call DescStackN('Cnt', Cnt);
      call DescStackN('Height', Hgt);
      put (_ALL_) (=);  
    end; 
   
    call DescStackN('Peek', Peek); 
    put Peek=;  

    do _I_ = 1 to Hgt; 
      call DescStackN('Output', _X_);
      keep _X_; 
      if _X_ > .z then output; 
    end; 
   
    call DescStackN('Peek', Peek); 
    put Peek=;
   
    %* clear for further reuse *; 
    call DescStackN('Clear',.);  

  run;

>>> %createDHPrtQueue() macro: <<<

The %createDHPrtQueue() macro allows to generate a dynamic PRIORITY hash queue which is a FCMP based approach to create dynamically allocated numeric or character priority queue

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

%createDHPrtQueue( 
  fifoName           
 <,debug=0>
 <,type=8>
 <,newOnTop=+>
 <,outlib=work.DFAfcmp.package>
 <,hashexp=13>
 <,header=1>
)

Arguments description:

  1. queueName - Required, creates a FCMP call subroutine which is also a queue name. In the data step it is used in form of a call subroutine, e.g. call queueName("Bottom", 1, 3). Has to satisfy FCMP function naming requirements, but with maximum of 24 characters.

  • debug= - Optional, the default value is 0. If set to 1 then it turns on a debugging mode.

  • type= - Optional, the default value is 8. Indicates what type (numeric/character) and length are data portion of generated array. Should be in line with the LENGTH statement, e.g. 8, $ 30, etc. Determines if the value argument is numeric or character.

  • newOnTop= - Optional, the default value is +. Indicates how to keep order in the same priority group, allowed values are + or -. Plus(+) sets new elements at the top of the group, minus(-) at the bottom.

  • outlib= - Optional, the default value is work.DFAfcmp.package. It points the default location for new generated dynamic function arrays compiled by FCMP. Hint! Keep it as it is.

  • hashexp= - Optional, the default value is 13. It is the default hashexp= value for internal hash table used by the function.

  • header= - Optional, the default value is 1. Indicates if the proc fcmp outlib = &outlib.; header is added to the executed code. If not 1 then no header is added.

Created function arguments description:

A function generated by the macro is:

call &queueName.(IO, position, value)

and accepts the following list of arguments and values:

  1. IO - is a character steering argument, possible values and behaviour they call are the following:
  • O, Output, D, Dequeue, R, Return - it pops/gets/outputs the data from the queue head (high priority),
  • B, Bottom - it pops/gets/outputs the data from the queue tail (low priority),
  • I, Input, E, Enqueue, Insert - it push/puts/inserts the data into the queue,
  • C, Clear - it reduces a queue to an empty one,
  • H, Head - it peeks the data from the queue head and NOT removes it,
  • T, Tail - it peeks the data from the queue tail and NOT removes it,
  • Sum - it returns sum of non-missing numeric elements of the queue,
  • Avg, Mean, Average - it returns average of non-missing numeric elements of the queue,
  • Nonmiss, Cnt - it returns number of non-missing elements of the queue,
  • Height - it returns height of the queue.
  1. position - is a numeric argument and depends on the IO value. Behaves in the following way:
  • for O, Output, D, Dequeue, R, Return and B, Bottom, or H, Head, T, Tail it holds a priority level of value popped from the queue,
  • for I, Input, E, Enqueue, Insert it holds a priority level of value to be pushed into the queue,
  • for C ignored,
  • for numeric queue and Sum, Nonmiss, Cnt, Avg, Mean, Average, Height returns calculated summary value.
  1. value - is a numeric or character argument (determined by the type=) and depends on the IO value. Behaves in the following way:
  • for O, Output, D, Dequeue, R, Return and B, Bottom or H, Head, T, Tail it holds a value popped from the queue,
  • for I, Input, E, Enqueue, Insert it holds a value to be pushed into the queue,
  • for C ignored,
  • for numeric queue and Sum, Nonmiss, Cnt, Avg, Mean, Average, Height returns calculated summary value,
  • otherwise does not modify value.

The position and the value arguments are outargs, i.e. can be changed by the function.

EXAMPLES AND USECASES:

EXAMPLE 1. Dynamic, Hash-based, and Character Priority queue:

  %createDHPrtQueue(PriorityQueuePC, type = $ 12, newOnTop=+);
  %createDHPrtQueue(PriorityQueueNC, type = $ 12, newOnTop=-); 
  options APPEND=(cmplib = WORK.DFAfcmp) ; 

  data Example1; 
   
    _I_ = .; 
    length _X_ _Y_ $ 3;
    do _X_ = "AAA","BBB","CCC","AA","BB","CC","A","B","C";
      _I_ + 1; 
      call PriorityQueuePC("I", mod(_I_, 3), _X_);
      call PriorityQueueNC("I", mod(_I_, 3), _X_); 
    end; 
   
    Height = .;
    call PriorityQueuePC('Height', Height, ''); 
    put Height=;

    do until(Height = 0); 
      call PriorityQueuePC('Dequeue', _I_, _X_);
      call PriorityQueueNC("Dequeue", _I_, _Y_);
      call PriorityQueueNC('Height', Height, '');
      put (_ALL_) (=); 
      output;  
    end; 
 
  run;

EXAMPLE 2. Dynamic, Hash-based, and Numeric Priority queue:

  %createDHPrtQueue(PriorityQueueN);
  options APPEND=(cmplib = WORK.DFAfcmp) ; 

  data Example2; 
   
    do _X_ = -5 to 5; 
      call PriorityQueueN("Enqueue", abs(_X_), _X_);
    end; 
   
    call missing(Sum, Avg, Cnt, Hgt); 
    call PriorityQueueN('Sum',    ., Sum);
    call PriorityQueueN('Avg',    ., Avg);
    call PriorityQueueN('Cnt',    ., Cnt);
    call PriorityQueueN('Height', ., Hgt);
    put (_ALL_) (=);

    do _N_ = 1 to Hgt; 
      call PriorityQueueN("Dequeue", _X_, _Y_);
      put _X_= _Y_=;
    end;  
  
  run;

>>> %createDHStack() macro: <<<

The %createDHStack() macro allows to generate a dynamic hash stack which is a FCMP based approach to create dynamically allocated numeric or character stack

Interesting reading about implementing a stack via hash table can be found in chapter 10.4 of the: "Data Management Solutions Using SAS Hash Table Operations: A Business Intelligence Case Study" book by Paul Dorfman and Don Henderson.

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

%createDHStack( 
  fifoName           
 <,debug=0>
 <,type=8>
 <,outlib=work.DFAfcmp.package>
 <,hashexp=13>
 <,header=1>
)

Arguments description:

  1. stackName - Required, creates a FCMP call subroutine which is also a stack name. In the data step it is used in form of a call subroutine, e.g. call stackName("Push", 3). Has to satisfy FCMP function naming requirements, but with maximum of 24 characters.

  • debug= - Optional, the default value is 0. If set to 1 then it turns on a debugging mode.

  • type= - Optional, the default value is 8. Indicates what type (numeric/character) and length are data portion of generated array. Should be in line with the LENGTH statement, e.g. 8, $ 30, etc. Determines if the value argument is numeric or character.

  • outlib= - Optional, the default value is work.DFAfcmp.package. It points the default location for new generated dynamic function arrays compiled by FCMP. Hint! Keep it as it is.

  • hashexp= - Optional, the default value is 13. It is the default hashexp= value for internal hash table used by the function.

  • header= - Optional, the default value is 1. Indicates if the proc fcmp outlib = &outlib.; header is added to the executed code. If not 1 then no header is added.

Created function arguments description:

A function generated by the macro is:

call &stackName.(IO, value)

and accepts the following list of arguments and values:

  1. IO - is a character steering argument, possible values and behaviour they call are the following:
  • O, Output, Pop, G, Get, R, Return - to get the data from a stack (and remove it from the top),
  • I, Input, Push, Put, Insert - to insert the data into a stack,
  • C, Clear - to reduce a stack to an empty one,
  • P, Peek - to peek the data from a stack (and NOT remove it from the top),
  • Sum - returns sum of non-missing numeric elements of a stack,
  • Avg, Mean, Average - returns average of non-missing numeric elements of a stack,
  • Nonmiss, Cnt, Nnm - returns number of non-missing elements of a stack,
  • Height - returns height a stack,
  1. value - is a numeric or character argument (determined by the type=) and depends on the IO value. Behaves in the following way:
  • for O, Output, Pop, G, Get, R, Return it holds a value popped from a stack,
  • for I, Input, Push, Put, Insert it holds a value to be pushed into a stack,
  • for C, Clear it is ignored,
  • for P, Peek it holds a value peeked from a stack,
  • for Sum, Nonmiss, Cnt, Avg, Mean, Average, Height it returns calculated summary value,

The value argument is outarg, i.e. can be changed by the function.

EXAMPLES AND USECASES:

EXAMPLE 1. Dynamic, Hash-based, and Character stack:

  %createDHStack(StackDHC, type = $ 12); 
  options APPEND=(cmplib = WORK.DFAfcmp) ; 

 
  %let zeros = 6; *[to test bigger sizes];
  data Example1;
 
    t = time(); drop t; 
    do _I_ = 1 to 1e&zeros.; 
      _X_ = put(_I_*10, z12.); 
      call StackDHC("Put", _X_); 
    end; 
    t = time() - t;
 
    call StackDHC("Height", _X_); 
    put t= / _X_=; 
   
    t = time(); 
    do _I_ = 1 to 1e&zeros. + 3; 
      call StackDHC('Pop', _X_); 
      output;  
    end; 
    t = time() - t; 

    call StackDHC("Height", _X_); 
    put t= / _X_=; 
 
    %* clear for further reuse *; 
    call StackDHC('Clear', '');  

  run;

EXAMPLE 2. Dynamic, Hash-based, and Numeric stack:

  %createDHStack(StackDHN); 
  options APPEND=(cmplib = WORK.DFAfcmp) ; 

  data Example2;
 
    do _I_ = 1,.,2,.,3,.,4,.,5,.,6;  
      call StackDHN("Put", _I_); 
    end; 

    call StackDHN("Sum", _I_); 
    put "Sum    " _I_=;

    call StackDHN("Avg", _I_); 
    put "Avg    " _I_=;

    call StackDHN("Cnt", _I_); 
    put "Cnt    " _I_=;
   
    call StackDHN("Height", _I_); 
    put "Height " _I_=; 

    _X_ = 0;
    do _I_ = 1 to _I_; 
      call StackDHN('Pop', _X_); 
      output;  
    end; 

    call StackDHN("Height", _I_); 
    put "Height " _I_=;

  run;

>>> bit64orPROTOdfa() proto function: <<<

The bit64orPROTOdfa() is external C function, this is the implementation of the bitwise OR operation on doubles. A double is returned.

Caution! For SAS numeric values only operations on first 53 bits are valid!

The function is used internally by functions in the DFA package.

SYNTAX:

The basic syntax is the following:

bit64orPROTOdfa(i, j)

Arguments description:

  1. i - A double numeric argument.

  2. j - A double numeric argument.

>>> bit64andPROTOdfa() proto function: <<<

The bit64andPROTOdfa() is external C function, this is the implementation of the bitwise AND operation on doubles. A double is returned.

Caution! For SAS numeric values only operations on first 53 bits are valid!

The function is used internally by functions in the DFA package.

SYNTAX:

The basic syntax is the following:

bit64andPROTOdfa(i, j)

Arguments description:

  1. i - A double numeric argument.

  2. j - A double numeric argument.

>>> bit64orDFA() subroutine: <<<

The bit64orDFA() function is an alternative to the 32 bit bitwise BOR() function working on SAS numerics. Allows to work on up to 53 bits of SAS numeric value.

The bit64orDFA() is an internal function of the DFA package.

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

bit64orDFA(a, b)

Arguments description:

  1. a - Argument is a SAS numeric values.

  2. B - Argument is a SAS numeric values.

EXAMPLES AND USECASES:

EXAMPLE 1. Basic test of bit64orDFA() and bit64andDFA()

  options ls = max ps = max;
  %let M = 53 ; %* 53 is maximum valid value;
  data _null_;
    array bitmask [ 0: &M] _temporary_ ;
    do P = 1 to &M ;
      bitmask[P] = 2**(P-1) ;
      put bitmask[P] = binary54. @;
      put bitmask[P] = best32.; 
    end ;
    bitmask[0] = bitmask[&M.] ;
      put bitmask[0] = best32. /;

      a=0;
      put a = binary54.;
      do P = 1 to &M ;
        a = BIT64ORDFA (a, bitmask[P]) ;
        put a = binary54.;
      end;
      put;
      
      b = 0;
      put b = binary54./;
      do P = 1 to &M ;
        b + (BIT64ANDDFA (a, bitmask[P]) ne .) ;
        put b = best32.;
      end;
  run;

>>> bit64andDFA() subroutine: <<<

The bit64andDFA() function is an alternative to the 32 bit bitwise BAND() function working on SAS numerics. Allows to work on up to 53 bits of SAS numeric value.

The bit64andDFA() is an internal function of the DFA package.

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

bit64andDFA(a, b)

Arguments description:

  1. a - Argument is a SAS numeric values.

  2. B - Argument is a SAS numeric values.

EXAMPLES AND USECASES:

EXAMPLE 1. Basic test of bit64orDFA() and bit64andDFA()

  options ls = max ps = max;
  %let M = 53 ; %* 53 is maximum valid value;
  data _null_;
    array bitmask [ 0: &M] _temporary_ ;
    do P = 1 to &M ;
      bitmask[P] = 2**(P-1) ;
      put bitmask[P] = binary54. @;
      put bitmask[P] = best32.; 
    end ;
    bitmask[0] = bitmask[&M.] ;
      put bitmask[0] = best32. /;

      a=0;
      put a = binary54.;
      do P = 1 to &M ;
        a = BIT64ORDFA (a, bitmask[P]) ;
        put a = binary54.;
      end;
      put;
      
      b = 0;
      put b = binary54./;
      do P = 1 to &M ;
        b + (BIT64ANDDFA (a, bitmask[P]) ne .) ;
        put b = best32.;
      end;
  run;

>>> %createDFBitmap() macro: <<<

The %createDFBitmap() macro allows to generate a dynamic function bitmap which is a FCMP based approach to create dynamically allocated numeric bitmnap.

Note: Arrays provided by the macro are one dimensional arrays.

The idea of a SAS bitmap is based on:

  1. SGF Paper 3101-2019 titeled Re-Mapping A Bitmap by Paul M. Dorfman and Lessia S. Shajenko https://www.sas.com/content/dam/SAS/support/en/sas-global-forum-proceedings/2019/3101-2019.pdf
  2. SUGI Paper 8-26 titeled Table Look-Up by Direct Addressing: Key-Indexing -- Bitmapping -- Hashing by Paul M. Dorfman https://support.sas.com/resources/papers/proceedings/proceedings/sugi26/p008-26.pdf

SYNTAX:

The basic syntax is the following, the <...> means optional parameters:

%createDFBitmap( 
  bitmapName           
 <,debug=0>
 <,outlib=work.DFAfcmp.package>
 <,type=32>
 <,header=1>
)

Arguments description:

  1. bitmapName - Required, creates a FCMP call subroutine which is also a bitmap name. In the data step it is used in form of a call subroutine, e.g. call bitmapName("Allocate", 3000). Has to satisfy FCMP function naming requirements, but with maximum of 24 characters.

  • debug= - Optional, the default value is 0. If set to 1 then it turns on a debugging mode.

  • outlib= - Optional, the default value is work.DFAfcmp.package. It points the default location for new generated dynamic function arrays compiled by FCMP. Hint! Keep it as it is.

  • type= - Optional, the default value is 32. Sets the type of bitwise operations executed internaly on the bitmap. The only valid values are 32 or 52, With 32 the BOR() and BAND() SAS functions are used and with 52 the bit64orDFA() and bit64and DFA() FCMP functions are used.

  • header= - Optional, the default value is 1. Indicates if the proc fcmp outlib = &outlib.; header is added to the executed code. If not 1 then no header is added.

Created function arguments description:

A function generated by the macro is:

call &bitmapName.(IO, position, value)

and accepts the following list of arguments and values:

  1. IO - is a character steering argument, possible values and behaviour they call are the following:
  • Check, Test, T - to get information if a bit is set to 1 (on) or not,
  • On, 1 - to set a selected bit to 1,
  • Off, 0 - to set a selected bit to 0,
  • C, Clear - to reduce a bitmat to a single empty cell,
  • A, Allocate - to reserve space for a bitmap and set all bits to 0,
  • A0, Allocate0 - to reserve space for a bitmap and set all bits to 0,
  • A1, Allocate1 - to reserve space for a bitmap and set all bits to 1,
  • D, Dim, Dimension - to returns minimal and maximal index of the bitmap.
  1. position - is a numeric argument and depends on the IO value. Behaves in the following way:
  • for On, Off, 1, 0/ Check, Test, T it is a bitmap index,
  • for C, Clear is ignored,
  • for A, Allocate sets the value of the minposition, i.e. the minimal position of the bitmap index,
  • for D, Dimension it returns value of the minposition,

.3 value - is a numeric argument and depends on the IO value. Behaves in the following way:

  • for Check, Test, T it holds value retrieved from a bitmap on a given position,
  • for On, Off, 1, 0, C, Clear is ignored,
  • for A, Allocate it sets the value of the maxposition, i.e. maximal position of the array index,
  • for D, Dimension it returns value of the maxposition

The position and the value arguments are outargs, i.e. can be changed by the function.

EXAMPLES AND USECASES:

EXAMPLE 1. Bitmap of type 32:

  %createDFBitmap(MyBitmap32,type=32,debug=1);
  options APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example1;
    call MyBitmap32("Allocate", -10, 100);
    L = 0; H = 0;
    call MyBitmap32("Dim", L, H);
    put L= H=;

 * populate array with data ;
    do i = L to H by 2;
      put i @;
      call MyBitmap32("1", i, .); 
    end;
    put;

 * get values from the array ;
    Value = .;
    do i = L to H;
      call MyBitmap32("T", i, Value); 
      put i= Value=;
    end;
  run;

EXAMPLE 2. Bitmap of type 52:

  %createDFBitmap(MyBitmap52,type=52,debug=1);
  options APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example2;
    call MyBitmap52("Allocate", -10, 100);
    L = 0; H = 0;
    call MyBitmap52("Dim", L, H);
    put L= H=;

 * populate array with data ;
    do i = L to H by 2;
      put i @;
      call MyBitmap52("1", i, .); 
    end;
    put;

 * get values from the array ;
    Value = .;
    do i = L to H;
      call MyBitmap52("T", i, Value); 
      put i= Value=;
    end;
  run;

EXAMPLE 3. Execution time test for type 52:

  %createDFBitmap(MyBigBitmap52,type=52,debug=0);
  options FULLSTIMER APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example3;
    call MyBigBitmap52("Allocate", -10, 2000000001);
    L = 0; H = 0;
    call MyBigBitmap52("Dim", L, H);
    put L= H=;

 * populate bitmap with data ;
    t = time();
    do i = L to H by 17;
      call MyBigBitmap52("1", i, .);
      x + 1; 
    end;
    t = time() - t;
    put "populate:" t= x=;

 * get values from the bitmap ;
    t = time();
    Value = .;
    do i = L to H;
      call MyBigBitmap52("T", i, Value); 
      x + (-Value);
    end;
    t = time() - t;
    put "search:" t= x=;
  run;

%*
L=-10 H=2000000001
populate:t=55.902999878 x=117647060
search:t=654.12900019 x=0
NOTE: The data set WORK.EXAMPLE3 has 1 observations and 6 variables.
NOTE: DATA statement used (Total process time):
      real time           11:50.42
      user cpu time       11:46.40
      system cpu time     0.45 seconds
      memory              301791.12k
      OS Memory           326332.00k
;

EXAMPLE 4. Execution time test for type 32:

  %createDFBitmap(MyBigBitmap32,type=32,debug=0);
  options FULLSTIMER APPEND=(cmplib = WORK.DFAfcmp) ;

  data Example4;
    call MyBigBitmap32("Allocate", -10, 2000000001);
    L = 0; H = 0;
    call MyBigBitmap32("Dim", L, H);
    put L= H=;

 * populate bitmap with data ;
    t = time();
    do i = L to H by 17;
      call MyBigBitmap32("1", i, .);
      x + 1; 
    end;
    t = time() - t;
    put "populate:" t= x=;

 * get values from the bitmap ;
    t = time();
    Value = .;
    do i = L to H;
      call MyBigBitmap32("T", i, Value); 
      x + (-Value);
    end;
    t = time() - t;
    put "populate:" t= x=;
  run;

%*
L=-10 H=2000000001
populate:t=50.417999983 x=117647060
populate:t=611.13600016 x=0
NOTE: The data set WORK.EXAMPLE4 has 1 observations and 6 variables.
NOTE: DATA statement used (Total process time):
      real time           11:02.07
      user cpu time       10:59.07
      system cpu time     1.46 seconds
      memory              489583.90k
      OS Memory           513876.00k
;

>>> generateArrays exec: <<<

The generateArrays exec file provides a list of automatically generated examples of functions emulating data structures.

The list of provided examples is the following:

  • SmpArray - Simple Immutable Dynamic Function Array
  • SmpMtbArray - Simple Mutable Dynamic Function Array
  • SrchArray - Searchable Immutable Dynamic Function Array
  • SrchMtbArray - Searchable Mutable Dynamic Function Array
  • DynArrayC - Dynamic Hash-based Character Function Array (length 256 bytes)
  • StackC - Dynamic Hash-based Character Stack (length 256 bytes)
  • StackN - Dynamic Hash-based Numeric Stack
  • FifoC - Dynamic Hash-based Character Fifo (length 256 bytes)
  • FifoN - Dynamic Hash-based Character Fifo
  • DescStackC - Dynamic Hash-based Character Descending Ordered Stack (length 256 bytes)
  • AscStackC - Dynamic Hash-based Character Ascending Ordered Stack (length 256 bytes)
  • DescStackN - Dynamic Hash-based Numeric Descending Ordered Stack
  • AscStackN - Dynamic Hash-based Numeric Ascending Ordered Stack
  • PrtQueueNTC - Dynamic Hash-based Character Priority Queue with New on Top (length 256 bytes)
  • PrtQueueNBC - Dynamic Hash-based Character Priority Queue with New on Bottom (length 256 bytes)
  • PrtQueueNTN - Dynamic Hash-based Numeric Priority Queue with New on Top
  • PrtQueueNBN - Dynamic Hash-based Numeric Priority Queue with New on Bottom
  • Bitmap32 - Dynamic Function Bitmap on 32 bit
  • Bitmap52 - Dynamic Function Bitmap on 52 bit

The outlib= option is set to work.DFAfcmp.package. The cmplib= option is updated automatically.

>>> generateArrays clean: <<<

The generateArrays clean file clears the list of automatically generated examples of functions emulating data structures provided in the generatearrays.sas exec file.

The cmplib= option is updated automatically.

License

Copyright (c) 2019 Bartosz Jablonski

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.