A simple defer feature for C

Jens Gustedt (INRIA, France) – Robert C. Seacord (NCC Group, USA)

org: ISO/IEC JCT1/SC22/WG14 document: N2895
target: IS 9899:2023 version: 2
date: 2021-12-31 license: CC BY

The final html for WG14 is produced with a little script, so please don’t worry too much about page layout or similar.

Also paper numbers like 2895 will be replaced automatically.


Revision history

Paper number Title Changes
N2542 Defer Mechanism for C Initial version WG14 poll:
N2895 A simple defer feature for C split of N2542 complicated parts → TR
based on lambdas

Introduction

Many implementations provide extensions that allow programmers to attach a block of cleanup code to another block or function such that this cleanup code is executed unconditionally when execution of the block is terminated. These existing extensions don’t have a unified API, some use pseudo-function calls, some compiler specific attributes and yet another a try/finally notion.

In a follow-up to N2542. we propose to encapsulate these extensions by a feature called defer that is syntactically and semantically inspired by the similar construct in the golang programming language. For an exhaustive overview on the possible extensions to the feature as proposed here, please refer to the link above.

For this paper here we make the assumption that simple lambdas are integrated into C23. Thereby the addition of the defer feature is much easier. Several properties of deferred statements that were contentious can be left to the programmer; they can be tuned according to each individual use case.

Rationale

Consistent resource management is key for the security of modern applications. Many modern programming languages (such as C++, Go, D, or Java), operating systems (such as POSIX) or libraries (such as boost) provide proper constructs to specify callbacks that are automatically launched on the exit of a block or function. For the complete motivation for this paper and the provided feature see N2542. For this paper here we concentrate on the functional intersection of several existing extensions to the C programming language that are implemented in the field; since none of these extensions is predominant and since they are syntactically quite different, we propose to map these on a similar feature as it is found in Go, coined defer.

The top-level view of this feature is that each defer declaration specifies a callback for which the execution is deferred until the current scope is left for whatever reason. For a given resource, this allows to specify a callback close to the resource definition, that “cleans up” at the end; it avoids the necessity to specify cleanup during further processing, each time that a exceptional condition is met.

Simplifying the feature by using lambdas

This paper builds on the assumption that at least simple lambdas are integrated into C23; otherwise it is obsolete. That assumption helps to simplify the proposal a lot and evacuates certain points that had be contentious, even between the original authors of N2542.

Status of the access to variables

In a defer statement that would be formulated with a compound statement as in

double* q = malloc(something);
defer { free(q); }

it would not be clear which value of q would be used for the call to free at the end of the surrounding block. Would it be the current value of q at the moment when the defer is met, or would it be the value that q has at the end of the execution of the block?

People were much divided here and it was not possible to reach a satisfying consensus, so probably the community is not (yet?) ready to establish a default behavior for that choice.

Basic models for the usage of captures

In this proposal we use lambdas to specify the defer callback and thus the decision about the point of evaluation of variables boils down all naturally to captures. So it is the user who will explicitly chose one model or the other according to their needs. There are several principal scenarios.

explicit shadow capture
double* q = malloc(something);
defer [q]{ free(q); };

Here, q is explicitly listed in the capture list and the value is frozen at the point of the defer and a new local object q shadows the use inside the defer callback.

explicit identifier capture
double* q = malloc(something);
defer [&q]{ free(q); };

q is explicitly listed as identifier capture. The identifier is evaluated when the defer callback is executed; in that case with the execution of the callback sees the last value of q when leaving the surrounding function or lambda.

explicit reference as value capture
double* q = malloc(something);
defer [qp = &q]{ free(*qp); };

An alternative to an identifier capture could be to take the address of the corresponding variable explicitly and memorize it in a value capture.

default shadow capture
double* q = malloc(something);
defer [=]{ free(q); };

All variables (and so q) are shadow captures and the value is frozen at the point of the defer.

default identifier capture
double* q = malloc(something);
defer [&]{ free(q); };

All variables (and so q) are identifier captures and the value is determined when the defer callback is executed.

Better expressivity

Using lambdas makes it even possible to use mixed captures as in the following.

enum { initial = 16, };
double buffer[initial] = { 0 };
...
size_t elements = 0;
double* q = buffer;
defer [orig = q, &q]{ if (orig != q) { free(q); }};
...
// increase elements somehow
...
// adjust the buffer
if (elements > initial) {
    double* pp = (q == buffer) ? malloc(sizeof(double[elements])) : realloc(q, sizeof(double[elements]));
    if (!pp) return EXIT_FAILURE;
    q = pp;
}
...

Here is possible to capture the initial value of q and use this inside the defer callback to test if the initial buffer (an automatic array) has been replaced by a large allocation.

Attachment of a defer feature to a function or a compound statement

We also had much debate if a deferred statement (as we called it) should be attached to a function or to a block (AKA compound statement). Using lambdas as callbacks instead of compound statements eases the argument and probably also the implementation.

The present proposal avoids to position itself with respect to that question.

Status of the deferred statement

In N2542 we had formulated the defer feature as a control statement, the present one changes this to make defer a declaration. This makes a description of the feature much easier, lambdas are first of all values, and it clearly anchors the lifetime and scope of applicability of the feature at the innermost enclosing block.

Existing practice for C

POSIX’ pthread_cleanup_push and pthread_cleanup_pop for thread cancelation

POSIX has these two functions (or macros) to ensure that a cleanup functionality can be attached to an implicit scope that is established by paired function calls to the following functions:

void pthread_cleanup_push(void (*routine)(void *), void *arg);
void pthread_cleanup_pop(int execute);

Here the argument arg is a pointer to a context that will be passed on to the routine callback when either the pthread_cleanup_pop call is met (execute has to be non-zero in that case) or if a thread exits prematurely, for example by pthread_exit or by being killed.

The calls must be paired, that is they must be statements that are attached to the same innermost compound statement. By that property they form some sort of implicit block, and implementations my even enforce that by hiding {}-pairs inside the macros.

If the cleanup task is a simple deallocation, the usage is relatively simple.

{
    double*const q = malloc(something);
    // implicitly starts a block
    pthread_cleanup_push(free, q);
    // use q as long as you wish
    ...
    // implicitly terminates the cleanup block
    // executes free unconditionally
    pthread_cleanup_pop(true);
    // now q falls out of scope
}

But such a simple usage does not extend easily when for example q may change during the execution of the inner code. For such a scenario a proper function that performs the cleanup has to be provided.

void destroy(void* p) {
    double** pp = p;
    free(*pp);
    *pp = 0;
}


{
    double* q = malloc(something);
    pthread_cleanup_push(destroy, &q);
    // use q as long as you wish, and change it if necessary
    ...
    // change q eventually
    ...
    pthread_cleanup_pop(true);
    // now q falls out of scope
}

Differences against the proposed feature

Compared to the defer feature as proposed here, this feature has several differences that we mostly see as disadvantages

The callbacks are called on preliminary exit from the thread.
The callbacks are necessarily associated to exactly one resource
The cleanup callbacks are not called on function return.
The pairing of the two calls is not properly embedded into the syntax and can be difficult to follow visually.
It is unspecified if such a pairing constitutes a scope. So local variables that are declared between the two calls may or may not survive after the second.
A callback has to be defined as a separate function, usually far from its use.
The function has to have a return type of void, a possible return value cannot be ignored.
The use of void* for the context undermines type safety.

Microsoft __try and __finally extentions

This feature is very similar in its functionality to what is proposed here. It allows to add a finally-block to a try-block such that the finally-block is executed independently from how the try-block is terminated.

{
    double* q;
    __try {
        q = malloc(something);
        // use q as long as you wish
        ...
    }
    __finally {
        free(q);
    }
    // now q falls out of scope
}

Differences against the proposed feature

Compared to the defer feature as proposed here, this feature has several differences that we mostly see as disadvantages

Only one finally-block can be added to a try-block
The finally-block has no access to local variables of the try-block
The resource that is cleaned up by the finally-block must live in a surrounding scope
An additional compound statement must be put around the try-finally combination to secure against accidental access of q.
The finally-block is usually declared far from the initialization of the feature to be cleaned up.
The syntax provides no obvious connection between try- and finally-block.

The cleanup attribute

The Gcc, Clang and Icc compilers implement a feature that has a functionality that has the same expressiveness to our proposal, namely the cleanup attribute. It allows to attach a callback to a variable in the top level scope of a function as follows:

    __attribute__((__cleanup__(callback))) toto x = initializer;

where callback has to be a function with the prototype

    void callback(toto*);

That is, callback is a function that receives a pointer to the variable and is supposed to do the necessary cleanup for that variable.

The main characteristics of this feature are:

Unfortunately this feature cannot be easily lifted into C23 as a standard attribute, because the cleanup feature clearly changes the semantics of a program and can thus not be ignored.

Removing a cleanup feature changes the semantics of the program.

Although this feature is attached to specific variables, it can easily be used and extended for arbitrary callbacks with a signature of void (*)(void). Therefore only a stub callback

#include <stdio.h>
#include <stdlib.h>

// A function that calls a particular callback
void callback_caller(void (**cbp)(void)) {
    (*cbp)();
}

is needed that then can be applied as follows

    __attribute__((__cleanup__(callback_caller))) void (*someUnusedId)(void) = local_callback;

That is, an auxiliary variable someUnusedId of pointer to function type is used to attach the meta-callback that in turn calls the callback that we are interested in.

Because Gcc has nested functions, code with functionality that is close to what we propose here could be something as the following.

#include <stdio.h>
#include <stdlib.h>

// A function that calls a particular callback
void callback_caller(void (**cbp)(void)) {
    (*cbp)();
}

void destroy_double(double** pp) {
    printf("freeing double %p\n", (void*)*pp);
    free(*pp);
}
void destroy_unsigned(unsigned** pp) {
    printf("freeing unsigned %p\n", (void*)*pp);
    free(*pp);
}

int main(void) {
    // ...
    __attribute__((__cleanup__(destroy_double))) double* A = malloc(sizeof(double[54]));
    // ...
    __attribute__((__cleanup__(destroy_unsigned))) unsigned* B = malloc(sizeof(unsigned[5]));
    // ...
    // A nested function for the purpose
    void local_callback(void) {
        // do the cleanup here
        printf("will be cleaning %p and %p\n", (void*)A, (void*)B);
    }
    __attribute__((__cleanup__(callback_caller))) void (*cb)(void) = local_callback;
}

This function has an output that is similar to

will be cleaning 0x56380b7222a0 and 0x56380b722460
freeing unsigned 0x56380b722460
freeing double 0x56380b7222a0

Clang has similar possibilities to express semantically the same features, but this would need either the use of a plain function (with less expressiveness) or the use of the block extension that they borrowed from Objective C. Similarly, the code can be adapted to Microsoft’s MVC to implement the same semantics with __try and __finally. We will not go into details how this can be achieved.

Our proposal puts the feature into a simple normative framework, makes it portable across implementations and, maybe most importantly, facilitates its use. Code with the same functionality as above is somewhat simpler expressed as follows.

#include <stdlib.h>

int main(void) {
    // ...
    double* A = malloc(sizeof(double[54]));
    defer [&]{
        printf("freeing A %p\n", (void*)A);
        free(A);
    }
    // ...
    unsigned* B = malloc(sizeof(unsigned[5]));
    defer [&]{
        printf("freeing B %p\n", (void*)B);
        free(B);
    }
    // ...
    defer [&]{
        // do the cleanup here
        printf("will be cleaning %p and %p\n", (void*)A, (void*)B);
    };
}

Differences against the proposed feature

Compared to the defer feature as proposed here, this feature has several differences that we mostly see as disadvantages

The callbacks are necessarily associated to exactly one resource
A callback has to be defined as a separate function, usually far from its use.
The function has to have a return type of void, a possible return value cannot be ignored.
The syntax cannot easily be adopted for C23 because it comes as a non-ignorable attribute.

Design choices

As explained above, using lambdas as a main tool to express defer callbacks implies that we don’t have to decide if variables inside these are accessed by their value when the defer is met or when it is executed. The different types of captures for lambdas here provide the possibility for users to chose the variant they need for their particular case.

Other design choices that we discussed for the original proposal are not directly impacted by the implementation as lambdas. We try to be mostly conservative here: the proposed feature should be functional, easy to implement and not inhibit future extensions that might be proposed with a TR.

Possible scopes of attachment

It was much discussed if defer should be a feature that is attached to functions or to blocks, and even the existing prior art follows different strategies, here. Whereas gcc’s cleanup attribute is attached to functions, POSIX’ cleanup functions and the try/finally are attached to possibly nested blocks.

This indicates that existing mechanism in compilers may have difficulties with a block model. So we only require it to be implemented for function bodies and make it implementation-defined if it is also offered for internal blocks.

Nevertheless, we think that it is important that the semantics for the feature are clearly specified for the case of blocks, such that all implementations that do offer such a feature follow the same semantics. Therefore it is also a constraint violation to ask for the feature for blocks on implementations that don’t have support for it.

Fixed order of encounter of defer declarations

One of the possible difficulties for implementing the feature is if the list of defer that has to be processed can have an order that depends on the execution. This could happen for example because some defer is conditionally omitted or when a local jump interchanges the order in which two defer declarations are seen for the first time.

We think that the feature should be implementable with very little effort and resources. So we propose that

These properties are enforced by an interdiction to jump over a defer declaration by means of switch, goto or longjmp. This requirement is a natural extension of the fact that jumping over any kind of declaration skips the initialization of a variable.

Behavior under abnormal termination of the block

The original proposal had several features that are designed to handle exceptional control flow, such as preliminary exists of the whole execution, of a thread or if signals are met. These features found a mixed reception and with this paper we do not want to impose any such feature for implementations that do not yet have mechanisms to which they could attach such features.

It is difficult to foresee which kind of requirements would be consensual for WG14, so we make one main proposal which leaves most of the stack unwind properties undefined (in the direct sense of the term) and only imposes that for any such scenarios none or all registered defer callbacks must be called. Two optional scenarios build on that, the first just forcing implementations to document their behavior by making the stack unwind features implementation-defined. The second additionally introduces feature tests that give the possibility to test dynamically at runtime if the present incarnation of the C library allows to unwind the stack or not.

thrd_exit and similar

The existence of the POSIX cleanup feature shows that there is a demand for tools that cleanup a whole stack of callbacks that are attached to a thread of execution. Also, implementations that are POSIX compliant and that would want to build upon their existing implementation of the cleanup feature should not be penalized.

exit and similar

For the terminating functions in <stdlib.h> we try to follow the directions that the standard already has for callbacks; that is in particular that abort should never call callbacks. On the other hand, a preliminary exit by one of these functions should always have defined behavior.

For other functions that implementations offer we cannot impose much, in particular if we want to allow future extensions (such as a panic function) or if we have to take well-established termination functions such a pthread_kill into account. Therefore we make the behavior of all such extensions explicitly undefined and leave room for implementations to be creative.

Signals

Signals are only scarcely specified in the C standard. In particular it only explicitly defines 3 software triggered signals for which we may specify behavior in case they lead to a termination of a thread or execution. We don’t think that it would make any sense for the standard to impose any type of behavior for the remaining 3 hardware interrupts that it describes, so we leave the handling of these signals undefined by omission.

Suggested changes

Syntax anchor

Add a new keyword defer to the list of keywords in 6.4.1.

Add a new term defer-declaration to the end of the declaration rule in 6.7 p1.

Change 6.7 p2:

2 A declaration other than a static_assert or, attribute or defer declaration shall declare at least a declarator (other than the parameters of a function or the members of a structure or union), a tag, or the members of an enumeration.

Specific clause

6.7.12 Defer declaration

Syntax

1 defer-declaration:

defer lambda-expression ;

Constraints

2 A defer declaration shall have block scope. It is implementation-defined if a defer declaration in a block other than the outermost block of a function definition or lambda expression is accepted.1)

3 The lambda expression shall have no parameter type list or such a list specified as () or (void).

Description

4 A defer declaration defines an unnamed object λ, called a defer callback, of lambda type and automatic storage duration that is initialized with the lambda expression. The object has a lifetime that corresponds to the current execution of the innermost block B in which the declaration is found. An abnormal termination of B is a termination of B that is caused by a function call that does not return, by a signal or by a goto statement. Sequenced immediately after the definition, λ is registered with the execution of B; when the execution of B terminates normally calls without arguments to the registered defer callbacks are sequenced as void expressions

Recursively, if during the execution of a defer callback λ a defer declaration is met during the execution of a block D, the corresponding defer callback κ is registered for that execution of D.2)

5 Jumps by means of switch, goto or longjmp shall not be used to jump over a defer declaration.3) If λ does not return, the behavior is undefined.4)

6 Unless specified otherwise, abnormal termination of the execution of B shall not call defer callbacks; the behavior is undefined unless the abnormal termination is caused


1) Thus an implementation may allow a defer declaration for example as the declaration expression of a for-loop or inside another compound statement, but programs using such a mechanism would not be portable. If a translation unit that uses such a defer declaration is not accepted, a diagnostic is required.

2) Thus the call to κ is terminated before λ returns.

3) This ensures that defer callbacks are properly initialized at the same time they are registered, that defer declarations are not revisited during the same execution of a block, and that, within their block, defer callbacks are registered in lexicographic order of their defer declarations.

4) So using calls to exit, thrd_exit, longjmp or any other library function that is specified with _Noreturn to terminate a defer callback has undefined behavior.

5) Implementations that provide other functionality to terminate execution are invited to document their behavior with respect to defer callbacks.

6) Implementations that provide other signal values that terminate execution per default are invited to document their behavior with respect to defer callbacks.

7) Implementations that provide other functionality to terminate execution of a thread, for example by killing it from another thread, are invited to document their behavior with respect to defer callbacks.


Optional addition for thrd_exit, exit, abort, SIGINT, SIGTERM and SIGABRT

Add to the new clause 6.7.12

7 It is implemementation-defined if an explicit call to thrd_exit calls any defer callbacks. If it does so, it calls the defer callbacks of all active execution of blocks of the thread that are registered before the call to thrd_exit, sequenced in the reverse order they had been registered. These calls happen before any other action defined for the thrd_exit library function are performed and take place within the scope of the block for which they have been registered.

8 Similarly, it is implementation-defined, if an explicit call to exit calls defer callbacks for the current thread. If the execution is terminated by a call to a different _Noreturn function of clause 7.22.4 than exit, no defer callbacks shall be called.

9 Similarly, it is implementation-defined, if a default handling of the signals SIGINT and SIGTERM that terminates execution calls defer callbacks for the current thread. If the execution is terminated because the signal SIGABRT occurred, no defer callbacks shall be called.

Alternative version for thrd_exit, exit, abort, SIGINT, SIGTERM and SIGABRT with feature tests

Add to the new clause 6.7.12

7 If the value of thrd_exit_defer is true, see 7.26, the defer callbacks of all active executions of blocks of the thread that are registered before an explicit call to thrd_exit are called, sequenced in the reverse order they had been registered. These calls happen before any other action defined for the thrd_exit library function are performed and take place within the scope of the block for which they have been registered. If the value is false, no defer callbacks are called.

8 Similarly, if the value of __exit_defer is true, see 7.22, an explicit call to exit calls the defer callbacks for the current thread. If the value is false, no defer callbacks are called. If the execution is terminated by a call to a different _Noreturn function of clause 7.22.4 than exit, no defer callbacks shall be called.

9 Similarly, if the value of sig_exit_defer is true, see 7.14, a default termination of the executions for the signals SIGINT or SIGTERM calls the defer callbacks for the current thread. If the value is false, no defer callbacks are called. If the execution is terminated because the signal SIGABRT occurred, no defer callbacks shall be called.

Add a new paragraph 7.14 p1’ (<signal.h>)

1’ The feature test macro STDC_VERSION_SIGNAL_H expands to the token yyyymmL .

Add a new paragraph 7.14 p5 (<signal.h>)

5 The macro

sig_exit_defer

expands to a value of type bool that is true if the implementation executes defer callbacks when the default handling of signals SIGINT and SIGTERM terminates the execution, see 6.7.12, and false otherwise; the expansion is not an lvalue and the value is the same for the whole execution.

Add a new paragraph 7.22 p6 (<stdlib.h>)

6 The macro

__exit_defer

expands to a value of type bool that is true if the implementation executes defer callbacks on explicit calls to exit, see 6.7.12, and false otherwise; the expansion is not an lvalue and the value is the same for the whole execution.

Add a new paragraph 7.26.1 p6 (<threads.h>, introduction)

6 The macro

thrd_exit_defer

expands to a value of type bool that is true if the implementation executes defer callbacks on explicit calls to thrd_exit, see 6.7.12, and false otherwise; the expansion is not an lvalue and the value is the same for the whole execution.

Optional addition of an example

Depending on the version of lambdas that are integrated into C23, this example might need small adjustments.

10 EXAMPLE In the following, the values of p, q and r are used as arguments to free at the end of the execution of main. Because the corresponding capture is a shadow capture, for p the initial value is used as argument to the call; for q it is an identifier capture and thus the value is used that was stored last before a return statement is met or the execution of the function body ends. Similarly, for r the value capture rp has the address of r and frees the last allocation for which the address was stored in r. The four return statements are all valid and according to the control flow that is taken the function executes 0, 1, 2, or 3 defer callbacks. If at least the first three allocations are successful, the storage is freed in the order r, q and p. If the call to realloc fails, the initial value of q is passed as argument to free.

#include <stdlib.h>
int main(void) {
   double*const p = malloc(sizeof(double[23]));
   if (!p) return EXIT_FAILURE;
   defer [p]{ free(p); };

   double* q = malloc(sizeof(double[23]));
   if (!q) return EXIT_FAILURE;
   defer [&q]{ free(q); };

   double* r = malloc(sizeof(double[23]));
   if (!r) return EXIT_FAILURE;
   defer [rp = &r]{ free(*rp); };
   {
       double* s = realloc(q, sizeof(double[32]));
       if (s) q = s;
       else return EXIT_FAILURE;
   }
}

Possible future extensions

Our orginal paper N2542. discusses a lot of features that could be added to the defer feature and that will give rise to a TR, such as

Other possible extensions would arise from the choices that are made in this proposal.

Default versions of captures

Using lambdas as the base feature as we propose here opens other possibilities, in particular for the status of captures. This example from the beginning is not valid with our proposal:

double* q = malloc(something);
defer { free(q); };

The use of a compound statement here could be seen as an indication that the access to q is meant to be an identifier capture, and we could then per default expand this to

double* q = malloc(something);
defer [&]{ free(q); };

On the other hand, in golang from where we borrowed this feature a version without {}

double* q = malloc(something);
defer free(q);

would use the value of q as it is evaluated when the defer declaration is met. So in this sense this would probably best be expanded with as value closure

double* q = malloc(something);
defer [=]{ free(q); };

Possible syntax extensions

Over all it seems that several extension of the syntax seem possible to the syntax

1 defer-declaration:

defer defer-callback ;

defer-callback:

no-argument-callable

function-call

compound-statement

no-argument-callable:

expression

With the following constraints

The expression of a no argument callable shall evaluate to a function pointer type or to a lambda value type that receive no arguments. A defer declaration with a function call or compound statement behaves, respectively, as if it were specified with lambda expressions as in the following

defer [=](void) { (void) function-call ; };

defer [&](void) compound-statement ;

The corresponding constraints for lambda expressions then shall apply.

Defer in file scope

If we would extend the possibility to have defer declarations in file scope this could have the similar semantics as calls to atexit.

FILE* logfile = 0;
defer []{
    if (logfile) {
        log2file(logfile, "execution is terminating\n</p>\n</html>\n");
        fclose(logfile);
    }
};

This could be equivalent to define a function

FILE* logfile = 0;
void logfile_callback(void) {
    if (logfile) {
        log2file(logfile, "execution is terminating\n</p>\n</html>\n");
        fclose(logfile);
    }
};

and to call atexit(logfile_callback) at program initialization before entering main.

Questions to WG14

Base

Does WG14 want to integrate a defer feature as proposed in 6.1 and 6.2 of N2895 into C23?

Options for abnormal termination

Since we don’t really have a possibility to vote for multiple choices, we propose to escalate the feature.

Does WG14 want to integrate previsions for abnormal termination as in 6.3 of N2895 into C23?

If the answer here is positive

Does WG14 want to replace the previsions for abnormal termination and use 6.4 instead of 6.3 of N2895 into C23?

Example

Does WG14 want to integrate the example as in 6.5 of N2895 into C23?