n3657 - Functions with Data - Closures in C (A Comprehensive Proposal Overviewing Blocks, Nested Functions, and Lambdas)

[ThePhD]

Functions with Data paper, without wording, is ready. Comments about the whole damn thing are welcome.

https://thephd.dev/_vendor/future_cxx/papers/C%20-%20Functions%20with%20Data%20-%20Closures%20in%20C.html

[ckwastra]

One minor question: § 3.3.5 (Explicit) Trampolines: Page-based Non-Executable Implementation states that Objective-C could run out of trampolines for Blocks because only a single page is used for this purpose. However, the post Implementing imp_implementationWithBlock() does not suggest such a limitation:

Once a full page of trampolines is allocated, we simply need to provide the individual trampolines on request, and allocate additional pages if the pool of trampolines is exhausted. [...]

So, I'm having trouble reconciling these two statements. What am I missing?

[lerno]

This is a comment taking issue with closures for C at all

I am strongly negative to closures in C. Once you have that hammer, everything looks like a nail function that should take a closure. And those APIs may look attractive, but for debugging and readability they are bad.

It's might not immediately be obvious why, but let's contrast it to "function + data" that we already have. The current state means you have two "moving parts": (1) the data to use (2) the function to invoke with that data. Only the first has any lifetime concerns and it is relatively straightforward to manage it. If needed to, the API can be so that the callee can introspect the data even.

Closures make the captured data opaque to the callee, assuming it does not need any such information. Superficially this seems like an ok trade-off, but it is surprisingly problematic in practice.

Another problem in practice is actually in the "benefit" closures tries to provide: not having to write an extra, separate, function.

// Example of non-closure code
void run_when_finished(Context *c, SomeData *result)
{
   int x = do_a_lot_of_things(c, result.data);
   schedule_something_else(c, x);
}
void run_something(Context *c)
{
  run_something(c.runner, c.abc, &run_when_finished, c);
}

Vs

void run_something(Context *c)
{
  // Let's use Apple Block syntax here as an example
  run_something(c.runner, c.abc), ^(SomeData *result) { 
    int x = do_a_lot_of_things(c, result.data);
    schedule_something_else(result.caller, x); 
  });
}

Now superficially, the latter looks better: it removes the need for a function and it just inline is halfway to async, right?

However, in practice in the long run, this is much worse to work with in a way that is non-obvious. But we can at least see that:

1. In the non-closure example we can directly at the top level see the "entry points" that will be invoked asynchronously, because it's at the top level.
2. We can also directly call this code if need to (we can of course extract this out of the callback and just make the closure invoke the function, but that is fiddly and removes any practical benefit of using closures)

For me personally I've worked with both in the MacOS/iOS context, with MacOS adding blocks and block-based APIs (GCD in particular), and it's just what I try to describe above: you're enticed into using the block API but later you always regret it, because it's very inflexible and then you're fairly stuck as it takes a lot of work to go from blocks -> invocations scheduled on the NSRunLoop.

The quality of callback APIs drastically dropped when MacOS started using callbacks pervasively, and as is probably well known, Swift then added async, which in ObjC / C is represented as Block callbacks, recreating JS "callback hell" outside of pure Swift usage.

So if C gets closures, we'll start seeing the same problem with nested callbacks, and ultimately the "solution" to this poor API design will then be to add async/await to the language.

Non-capturing lambdas

However, non-capturing lambdas doesn't suffer from these practical issues: they are not interesting to be used as async callbacks, so async callbacks functions are likely to still be moved to the top level, and because they don't hold lifetime state, that's no problem either. They're just functions defined inside functions, visibility handled like the static variable in a function.

---

TLDR; Closures shouldn't be in C. They don't actually give better API in the long run and makes for code that is worse to debug. Non-capturing lambdas are fine though, they add convenience to APIs like "qsort" and don't really change how C works.

[ThePhD]

One minor question: § 3.3.5 (Explicit) Trampolines: Page-based Non-Executable Implementation states that Objective-C could run out of trampolines for Blocks because only a single page is used for this purpose. However, the post Implementing imp_implementationWithBlock() does not suggest such a limitation:

Once a full page of trampolines is allocated, we simply need to provide the individual trampolines on request, and allocate additional pages if the pool of trampolines is exhausted. [...]

So, I'm having trouble reconciling these two statements. What am I missing?

This is an older limitation, I believe. I should make it clear: my understanding is that in the "olden days" they had a fixed number of trampolines mapped to a fixed number of data locations in the "w^x data, rx function trampoline" split. This is (reportedly, as in from users who were running up the trampoline count) hit the limitation, but that it's been an ancient history thing. I don't have the requisite hardware to give a good test of this; I'll just add a NOTE: to say it was an older bit.

[ThePhD]

... and then you're fairly stuck as it takes a lot of work to go from blocks -> invocations scheduled on the NSRunLoop.

Can you clarify what this looks like and why this takes a lot of work? Is it because NSRunLoop doesn't play nice with blocks at the API level (doesn't take a Block?), or because the opaqueness of the accessed data means you need to "refactor" such things to take a separate parameter and that's a pain-in-the-ass?

[fuhsnn]

I'm planning how to implement this. In my naive view, the required machinery for GNU capturing nested functions alone is sophisticated enough to also cover other modern PL features:

If formulating closures as functions with two elements:

• context: opaque pointer with compiler-maintained layout
• capture: locate stuff in context by name

then the following can be mapped onto this model:

• Nested functions / Lambda
• • context: parent function stack frame, reference or copy
• • capture: parent function variables
• Stackless functions
• • context: externally maintained heap memory
• • capture: their own local variables
• Coroutines:
• • stackless functions with progress counter
• Member functions/UFCS (possibly with _Capture(struct foo))
• • context: struct data from somewhere
• • capture: struct members by name
• Implicit context as in Odin (and Jai?)
• • context: pointer passed along calls
• • capture: the name "context"
• Other stuff done with GCC static chain, (GCC's own nested functions seem to be).

These are brought up because IMO it's a shame to miss the possibility to introduce them in unified fashion. Obviously it's impossible and controversial to try to add them all at once, I just hope the work with nested functions leaves room for future features to be done somewhat consistently. For C new features should be as neat and intuitive as possible to be "worth using", otherwise people would go "I can boilerplate it in portable ANSI C" and they are probably right.

In such direction I'd like to propose the syntax void fn({&}, int i), arguments are:

• it reflects the fact there's a context pointer being passed
• less intrusive to the grammar while clearly distinguished from regular functions
• • fits right at home in gibberish: int(*(*f({&}, int))[4])({=}, int({&}, int));
• • the parameter list is already crowded with magic stuff void fn(int i; int[static i][*], int i, ...), might as well continue the trend...
• ability to call with explicit contexts in the same format a = fn({ptr}, ...);
• • which maps directly to a = __builtin_call_static_chain(fn(...), ptr);
• • essential for stackless/coroutines
• can be extended to multiple context and explicit registers
• • void fn({gnu::static_chain}, {&} __asm("%r11"), int regular_params, ...)
• • unnecessary for C as a language, but I believe it's helpful for OS and runtimes

For working with wide pointers, I would prefer an interface similar to <stdarg.h> (because they are both standard fat pointers with user-provided types)(also leave room for unconventional targets):

 widefn_t(int(int, float)) wp;
 widefn_p(wp) = &closure_def;
 widefn_ctx(wp) = malloc(sizeof(_Context_of(closure_def)));
 *widefn_ctx(wp) = _Context_of(closure_def);
 int a = widefn_call(wp)(7, 3.14f);

Extremely naive fantasy from my part.