UDF Interruption

[VGSML]

Some times it is necessary to interrupt duckdb queries that call udf. Currently scalar and table udf don’t have context or something like this, that can indicate that query interrupt.
I know that this question should be addressed to the duckdb, but what do you think, can it be implemented with current c.api?
And second question. It would be great to have a UDFs interfaces that will accept and return arrow records instead go types, especially for the big row numbers scenarios.

[taniabogatsch]

Hi @VGSML, for your first question, have a look at the draft PR here and a possible alternative here. Maybe you could give a quick review / leave your thoughts?

[VGSML]

Or if functionInfoPtr unsafe.Pointer would contain not real pointer to the functionInfo, instead pointer to the calling context, by that we can get functionInfoPtr and dbContextPtr.

[taniabogatsch]

If we will call one udf though duckdb from several connections at the same time, we can't get real calling context, only context in which the func was registered.

bindInfo is a duckdb-side object, which has the client context for the specific invocation of the scalar udf. Similar to the bindInfo of table functions, we'd have to expose it. During bind, duckdb has its client context available, and we can set that in the bind info.

It's looks like the callback should be extended - add one more pointer, by which we could get real execution duckdb context.

DuckDB's execution context does not always have a client context available (check constraint and other edge cases), so I don't think we should rely on that - and we can get the client context when binding the scalar udf (on the duckdb side)

[taniabogatsch]

I'll sketch up a PR to duckdb with the C API side of this in the coming days, and ping you on it. :)

[taniabogatsch]

duckdb/duckdb#17449

[VGSML]

I have took a look in PR, it looks pretty gut! It can help us to solve this!

[VGSML]

Since we’ve already solved half of the problem (for scalar UDFs), @taniabogatsch, do you have any thoughts on table UDFs? We need to find a way to pass the context to the init, bind, and row-filling functions.

[taniabogatsch]

Hmm, I haven't started looking into that other half, yet. We have similar abstractions on the C++ side - so likely, we'll need to check if the C API already has functions in place to get the client context from the bind info (I don't think so). With that, I think we can then use the same approach for table UDFs?

[VGSML]

Hi @taniabogatsch, did you check the C API? Is there a function to get client context from bind info for table functions?

[VGSML]

It's looks like there are no related function. Can you add it to c api?

void duckdb_table_function_get_client_context(duckdb_bind_info info, duckdb_client_context *context) {
	if (!info || !context) {
		return;
	}
	auto &bind_info = GetCTableFunctionBindInfo(info);
	auto wrapper = new duckdb::CClientContextWrapper(bind_info.context);
	*context = reinterpret_cast<duckdb_client_context>(wrapper);
}

to the src/main/capi/table_function-c.api.

After it I could propose the solution for table functions

[taniabogatsch]

Hi @VGSML - I am a bit swamped at the moment, but I'll see if I can add it later next week. 👍 Otherwise, maybe you can open a PR against duckdb/duckdb yourself? Here's an example PR adding a new function: duckdb/duckdb#17870.

[VGSML]

I have tried, but I am not sure that I have done all right, especially in tests. Could you take a look at duckdb/duckdb#18232

[VGSML]

@taniabogatsch Hi, sometimes it is needed to cancel running queries. Are there any methods to do it, I didn't find((
The normal go behaviour is when the context is done, than a query should be canceled, but this is not implemented and I don't know APIs to do it.

[VGSML]

Super. Is there same APIs (Pending) for the arrow interface?
I currently use a lot arrow, for streaming results as arrow IPC. The QueryContextArrow uses duckdb_execute_prepared_arrow

[taniabogatsch]

I don't know much about the arrow interface, tbh. I know we recently did a major refactoring on the C API side, so I expect changes coming related to that.

[VGSML]

It looks like we will use the normal execution process and convert resulted chunks to the arrow.Records. What you think can it helps for streaming results, I mean to avoid their materialization?

[VGSML]

Do you know, when it should be released?

[taniabogatsch]

What you think can it helps for streaming results, I mean to avoid their materialization?

I don't know, I'd have to check myself. It'll be released as part of duckdb's next release, so 1.4