Articles Open array parameters and array of const by Rudy Velthuis

[emailx45]

Open array parameters and array of const
Rudy Velthuis

Спойлер: Content - 1st part

[SHOWTOGROUPS=4,20]
This article describes the syntax and use of open array parameters, and the use of the “array of const” parameter type. It also describes the internals of these two similar types of parameters, discusses lifetime issues, and gives a code solution for these issues. It has a short discusson on the confusion between open arrays and dynamic arrays, and between Variant arrays and variant open arrays.

Open array parameters
Open array parameters are special parameters which allow you to write procedures or functions (I will use the word routines, if I mean both) that can act on any array of the same base type, regardless of its size. To declare an open array parameter, you use a syntax like this:

procedure ListAllIntegers(const AnArray: array of Integer);
var
I: Integer;
begin
for I := Low(AnArray) to High(AnArray) do
WriteLn('Integer at index ', I, ' is ', AnArray[I]);
end;

You can call this procedure with any one-dimensional array of Integers, so you can call it with an array[0..1] of Integer as well as with an array[42..937] of Integer, or even a dynamic type array of Integer.

The code also demonstrates how you can determine the size of the array in the routine. Delphi knows the pseudo-functions Low and High. They are not real functions, they are just syntactic items in Delphi that take the form of a function, but actually rely on the compiler to substitute them for code. Low gives the lower bound of an array, and High the upper bound. You can also use Length, which returns the number of elements of the array.

But if you call the code with an array that is not zero-based, like for instance in the following (nonsense) example,

var
NonZero: array[7..9] of Integer;
begin
NonZero[7] := 17;
NonZero[8] := 325;
NonZero[9] := 11;
ListAllIntegers(NonZero);
end.

you will see that the output is like this:

Integer at index 0 is 17
Integer at index 1 is 325
Integer at index 2 is 11

That is because inside the procedure or function, the array is always seen as a zero based array. So for an open array parameter, Low is always 0, and High is adjusted accordingly (note that this is not necessarily true for other uses of High and Low, i.e. not on open array parameters). For open arrays, Length is always High + 1.

Slice
If you don’t want to use an entire array, but only a part of it, you can do that using the Slice pseudo-function. It is only allowed where an open array parameter is declared. It is used in this fashion:

const
Months: array[1..12] of Integer = (31, 28, 31, 30, 31, 30,
31, 31, 30, 31, 30, 31);
begin
ListAllIntegers(Slice(Months, 6));
end;

That will only display the first 6 values of the array, not all 12.

Note that with Slice, you can fool the compiler. If range checking is not on (in the Project options, or in the form of {$R-} or {$RANGECHECKS OFF}), and you give a value for the slice length that is higher than the actual length, this is not detected by the compiler. So you are telling the called function it can access elements it should actually not access.

This is not a problem with static arrays and a constant size. The compiler checks those at compile time and will issue an error. But if the size is a variable or the array is dynamic, the compiler can’t check this at compile time. Then it can happen that you do:

var
N: Integer;
begin
N := 18;
ListAllIntegers(Slice(Months, N));
end;

If ListAllIntegers really tries to access index 12 (remember, the index of an open array parameter is zero-based, so High should be 11) or higher, it is accessing beyond the bounds of the array. This is undefined behaviour. It can result in a crash, or other bad things, depending on what is actually accessed.
Without Slice, the compiler passes the right values for High, so this cannot happen. So be careful with this pseudo-function.

Internals
But how does that work; how can the function know the size of the array? It is quite simple. An open array parameter is actually a combination of two parameters, a pointer, which contains the address of the start of the array, and an integer, which contains the High value, adjusted for a zero base. So in fact the real parameter list of the procedure is something like this:

procedure ListAllIntegers(const AnArray: Pointer; High: Integer);

Each time you pass an array to an open array parameter, the compiler, which knows the size of the array, will pass its address and its adjusted High value to the procedure or function. For arrays of a static size, like array[7..9] of Integer, it uses the declared size to pass the High value; for dynamic arrays, it compiles code to get the High value of the array at runtime.

Usually, you can pass open arrays as const parameters. Open array parameters that are not passed as const will entirely be copied into local storage of the routine. The array is simply passed by reference, but if it is not declared const, the hidden start code of the routine will allocate room on the stack and copy the entire array to that local storage, using the reference as source address. For large arrays, this can be very inefficient. So if you don’t need to modify items in the array locally, make the open array parameter const.

Open array constructors
Sometimes you don’t want to declare and fill an array just so you can use it with an open array parameter. Luckily, Delphi allows you to declare open arrays on the spot, using the so called open array constructor syntax, which uses [ and ] to define the array. The above example with the NonZero array could also have been written like this:

ListAllIntegers([17, 325, 11]);

Here, the array is defined on the spot as [17, 325, 11].

Lifetime
An array created by an open array constructor is only valid as long as the function or procedure runs and is discarded right afterward. That is one reason why you can’t pass such an array to a var open array parameter.

Internally, the open array constructor simply makes room on the stack (the local variable frame) and puts copies of the values there. In other words, it creates an ad hoc array on the stack. Then it calls the function passing a pointer to the first value on the stack and the High value of that array. After the function call, the stack room is reclaimed, so the constructed array does not exist anymore.

Managed (reference counted) types like strings, interfaces, etc. are copied raw to the stack array and not finalized after the call, i.e. no reference counting is done. This is equivalent to passing a const parameter.

Confusion
Although the syntax is unfortunately very similar, an open array parameter should not be confused with a Delphi dynamic array. A dynamic array is an array that is maintained by Delphi, and of which you can change the size using SetLength. It is declared like:

type
TIntegerArray = array of Integer;

Unfortunately, this looks a lot like the syntax used for open array parameters. But they are not the same. An open array parameter will accept dynamic arrays like array of Month, but also static arrays like array[0..11] of Month. So in a function with an open array parameter, you can’t call SetLength on the parameter. If you really only want to pass dynamic arrays, you’ll have to declare them separately, and use the type name as parameter type.

type
TMonthArray = array of Month;

procedure AllKinds(const Arr: array of Month);
procedure OnlyDyn(Arr: TMonthArray);

Procedure AllKinds will accept static arrays as well as dynamic arrays, so SetLength can’t be used, since static arrays can’t be reallocated. Procedure OnlyDyn will only accept dynamic arrays, so you can use SetLength here (this will however use a copy, and not change the original array; if you want to change the length of the original array, use var Arr: TMonthArray in the declaration).

Note: You should not forget that in Delphi, parameters can ony be declared with type specifications, and not with type declarations. So the following formal parameters, which would be type declarations, are not possible:

function Sum(const Items: array[1..7] of Integer): Integer;
procedure MoveTo(Spot: record X, Y: Integer; end);

You’ll have to declare a type first, and use the specifications as parameter type:

type
TWeek = array[1..7] of Integer;
TSpot = record
X, Y: Integer;
end;

function Sum(const Items: TWeek): Integer;
procedure MoveTo(Spot: TSpot);

That is why array of Something in a parameter list can’t be a type declaration for a dynamic array either. It is always an open array declaration.
The latest Delphi compilers allow you to define dynamic array constants, so you can do:

var
MyDynArray: array of Integer;
begin
MyDynArray := [17, 325, 11];

You should not confuse these, despite the similar syntax, with open array constructors. If you code:

ListAllIntegers([17, 325, 11]);

then you are not passing a dynamic array constant, you are using an open array constructor.

TArray<T>

In versions of Delphi that know generics, you will see the type TArray<T> being used. It is declared as

type
...
TArray<T> = array of T;

This is a generic type, and only useful if T is actually replaced by a concrete type, for instance

procedure OnlyDyn(Arr: TArray<Month>);

The construct is being used more and more, because generic types can circumvent the general rule that type compatibility is not dependent on the form of the declaration, but on where the type is defined. So not all array of Month are assignment compatible with each other, although they have exactly the same form. But all TArray<Month> are assignment compatible, even if they are declared on the spot. This is an exception to the rule I explain in the text box above. TArray<T> can be used as parameter and even as return type, so you can have declarations like the following.

constructor Create(Limbs: TArray<TLimb>; Negative: Boolean);
function ToByteArray: TArray<Byte>;

Assembler
To use an open array from assembler, you must remember that an open array is in fact a combination of two parameters. The first parameter is a pointer to the start of the array, the second the adjusted High value. Generally, open arrays are passed as const or var. In all cases, the array is passed by reference.

Here is a simple example of a function that sums all integers in an array:

function Sum(const Data: array of Integer): Integer;
// EAX: address of the array
// EDX: adjusted High value
asm
MOV ECX,EAX // P := PInteger(Addr(Data));
XOR EAX,EAX // Result := 0;
OR EDX,EDX
JS @@Exit // if High(Data) < 0 then Exit;
@@Next: // repeat
ADD EAX,[ECX] // Result := Result + P^;
ADD ECX,TYPE Integer // Inc(PInteger(P));
DEC EDX // Dec(EDX);
JNS @@Next // until EDX < 0;
@@Exit:
end;

The example above uses the usual register calling convention. If your function or procedure has a different calling convention, you may have to address the open array differently, but still as a combination of an address and a High value. If the procedure or function is in fact an instance method, then the first parameter (here: EAX) will be its implicit Self parameter, and Data will be passed in the next two parameters, in this case EDX and ECX.


[/SHOWTOGROUPS]

 

[emailx45]

Спойлер: Content - 1st part

[SHOWTOGROUPS=4,20]
Array of const
Array of const is a special case of open arrays. Instead of passing only one type, you can pass a variety of types. Have a look at the declaration of the Format function in Delphi:

function Format(const Format: string; const Args: array of const): string;

(Actually, there is a second, overloaded function in some versions of Delphi, but I’ll simply ignore that here.)

The first parameter is a string which indicates how you want your values formatted, and the second is an array of const, so you can pass a range of values using a similar syntax as the one for open array constructors. So you can call it like:

var
Res: string;
Int: Integer;
Dub: Double;
Str: string;
begin
Int := Random(1000);
Dub := Random * 1000;
Str := 'Teletubbies';
Res := Format('%4d %8.3f %s', [Int, Dub, Str]);
end;

Note: The official name for array of const parameters is variant open array parameters. This should not be confused with the Variant type in Delphi, and the arrays it can contain. They are quite different, even though a TVarRec (see below) is a bit similar to how a Variant is internally stored. Even the name of the internal record of a Variant is confusingly similar: TVarData.

Internals
Internally, an array of const is an open array of TVarRec. The declaration of TVarRec is given in the online help for Delphi. It is a variant record (also not to be confused with the Variant type), that contains a field called VType, and an overlay of different types, some of which are only pointers. The compiler creates a TVarRec for each member in the open array constructor, fills the VType field with the type of the member, and places the value, or a pointer to it, in one of the other fields. Then it passes the array of TVarRec to the function.

Since each TVarRec contains type information, Format can use this to check if it matches with the type given in the format string. That is why you get a runtime error when passing a wrong type. You can tell the compiler that you want it to store a different type identifier, by casting to the desired type. If the type doesn’t match one of the allowed types in a TVarRec exactly, the compiler will try to convert it to a matching type, so if you pass a Double, it will convert it to an Extended and pass that instead. Of course there are limitations on what the compiler can do, so for instance passing an object isn’t going to work.

Inside the function or procedure, you can treat the members as TVarRec immediately. The help for Delphi gives an example how to do this.

Lifetime issues
What you should notice is, that values in the TVarRec which are passed as pointers only exist during the course of the function or procedure. As soon as the routine ends, these values don’t exist anymore. So you should not be tempted to return these pointers from the procedure or function, or to store the TVarRecs in an array outside the routine, unless you can make sure that you manage the values yourself.

If you must copy the TVarRecs to an array or variable outside the function (this can also be a var parameter), be sure to make a copy (i.e. on the heap) of the value, and replace the pointer in the TVarRec with one to the copy. You should also take care that the copy is disposed of when it is not needed anymore. An example follows:

Download

Спойлер: Code

type
TConstArray = array of TVarRec;

// Copies a TVarRec and its contents. If the content is referenced
// the value will be copied to a new location and the reference
// updated.
function CopyVarRec(const Item: TVarRec): TVarRec;
var
W: WideString;
begin
// Copy entire TVarRec first
Result := Item;
// Now handle special cases
case Item.VType of
vtExtended:
begin
New(Result.VExtended);
Result.VExtended^ := Item.VExtended^;
end;
vtString:
begin
// Improvement suggestion by Hallvard Vassbotn: only copy real length.
GetMem(Result.VString, Length(Item.VString^) + 1);
Result.VString^ := Item.VString^;
end;
vtPChar:
Result.VPChar := StrNew(Item.VPChar);
// There is no StrNew for PWideChar
vtPWideChar:
begin
W := Item.VPWideChar;
GetMem(Result.VPWideChar,
(Length(W) + 1) * SizeOf(WideChar));
Move(PWideChar(W)^, Result.VPWideChar^,
(Length(W) + 1) * SizeOf(WideChar));
end;
// A little trickier: casting to AnsiString will ensure
// reference counting is done properly.
vtAnsiString:
begin
// nil out first, so no attempt to decrement reference count.
Result.VAnsiString := nil;
AnsiString(Result.VAnsiString) := AnsiString(Item.VAnsiString);
end;
vtCurrency:
begin
New(Result.VCurrency);
Result.VCurrency^ := Item.VCurrency^;
end;
vtVariant:
begin
New(Result.VVariant);
Result.VVariant^ := Item.VVariant^;
end;
// Casting ensures proper reference counting.
vtInterface:
begin
Result.VInterface := nil;
IInterface(Result.VInterface) := IInterface(Item.VInterface);
end;
// Casting ensures a proper copy is created.
vtWideString:
begin
Result.VWideString := nil;
WideString(Result.VWideString) := WideString(Item.VWideString);
end;
vtInt64:
begin
New(Result.VInt64);
Result.VInt64^ := Item.VInt64^;
end;
vtUnicodeString:
begin
// Similar to AnsiString.
Result.VUnicodeString := nil;
UnicodeString(Result.VUnicodeString) := UnicodeString(Item.VUnicodeString);
end;
// VPointer and VObject don't have proper copy semantics so it
// is impossible to write generic code that copies the contents
end;
end;

function CreateConstArray(const Elements: array of const): TConstArray;
var
I: Integer;
begin
SetLength(Result, Length(Elements));
for I := Low(Elements) to High(Elements) do
Result[I] := CopyVarRec(Elements[I]);
end;

// use this function on copied TVarRecs only!
procedure FinalizeVarRec(var Item: TVarRec);
begin
case Item.VType of
vtExtended:
Dispose(Item.VExtended);
vtString:
Dispose(Item.VString);
vtPChar:
StrDispose(Item.VPChar);
vtPWideChar:
FreeMem(Item.VPWideChar);
vtAnsiString:
AnsiString(Item.VAnsiString) := '';
vtCurrency:
Dispose(Item.VCurrency);
vtVariant:
Dispose(Item.VVariant);
vtInterface:
IInterface(Item.VInterface) := nil;
vtWideString:
WideString(Item.VWideString) := '';
vtInt64:
Dispose(Item.VInt64);
vtUnicodeString:
UnicodeString(Item.VUnicodeString) := '';
end;
Item.VInteger := 0;
end;

// A TConstArray contains TVarRecs that must be finalized. This function
// does that for all items in the array.
procedure FinalizeVarRecArray(var Arr: TConstArray);
var
I: Integer;
begin
for I := Low(Arr) to High(Arr) do
FinalizeVarRec(Arr[I]);
Arr := nil;
end;
[code]
[/spoiler]

The functions above can help you manage TVarRecs outside the routine for which they were constructed. You can even use a TConstArray where an open array is declared. The following little program
[code]
program VarRecTest;

{$APPTYPE CONSOLE}

uses
SysUtils,
VarRecUtils in 'VarRecUtils.pas';

var
ConstArray: TConstArray;

begin
ConstArray := CreateConstArray([1, 'Hello', 7.9, IntToStr(1234)]);
try
WriteLn(Format('%d --- %s --- %0.2f --- %s', ConstArray));
Writeln(Format('%s --- %0.2f', Copy(ConstArray, 1, 2)));
finally
FinalizeConstArray(ConstArray);
end;
ReadLn;
end.

will produce the expected, but not very exciting, output

1 --- Hello --- 7.90 --- 1234
Hello --- 7.90

The little program above also demonstrates how you can use Copy to use only a part of the entire TConstArray. Copy will create a copy of the dynamic array, but not copy the contents, so you should not try to use Copy and then later on use FinalizeConstArray on that copy. In the program above, the copy will be removed automatically, since the lifetime of the copy is managed by the compiler.

Finally
Open arrays and arrays of const are powerful features of the language, but they come with a few caveats. I hope I succeeded in showing some of these, and how you can overcome them.

Questions regarding open arrays, array of const and many other language issues can be discussed on the Embarcadero Community forums (https://community.embarcadero.com/forum/programming), or e.g. on Stack Overflow.

[/SHOWTOGROUPS]