Redesign: Callbacks

For callbacks ideally we want to wrap native Rust Fn traits in a C callable function to support both closures and function pointers. We need to take special care to not allow panics (more specifically: stack unwinding) to cross the FFI border.

The libvirt callback APIs allow including an opaque void pointer. We can use that pointer to pass our closure to the C function which can then call it. From what I've seen, most callback APIs in libvirt include a free callback as well, so that the user data can be cleaned up. We can use that to free the stored closure. However, I know that at least the virSetErrorFunc function does not include a free callback. In that case, we need to store the closure ourselves to avoid leaking memory. To store a global callback closure, we need a runtime initiated global mutex. Unfortunately, this currently requires the once_cell crate as a dependency.

Fair warning, this is fairly advanced Rust, hopefully this will be the most complex part of the crate.

For a callback that includes a free callback:

extern "C" fn stream_event_cb_wrapper(
	stream: sys::virStreamPtr,
	events: libc::c_int,
	data: *mut libc::c_void,
) {
	// This is only a borrow of the inner box, so nothing will be freed.
	let cb: &Box<dyn FnMut(Error) + Send> = unsafe { &*(data as *mut Box<dyn FnMut(Error) + Send>) };
	let stream_events = StreamEvents::from_raw(events);

	// Stop stack unwinding at this point. Note that panics in the closures will be silently
	// dropped.
	std::panic::catch_unwind(|| {
		cb(stream_events);
	});
}

extern "C" fn stream_event_free_cb_wrapper(data: *mut libc::c_void) {
	// Reconstruct the box from the pointer, so that it can be freed.
	let cb = unsafe { Box::from_raw(data as *mut Box<dyn FnMut(Error) + Send>) };
	// This call is unnecessary but it makes it explicit what is happening.
	drop(cb);
}

impl Stream {
	fn add_callback(&self, events: StreamEvents, f: F)
	where
		F: FnMut(StreamEvents) + Send + 'static
	{
		// Box puts the contained value on the heap. The reason we need a double Box is because Box<dyn
		// Trait> is a fat pointer, it contains a pointer to both the struct and the vtable with the
		// trait functions. This fat pointer cannot be cast to a raw pointer for FFI because it does
		// not fit. Hence, we add a layer of indirection by double boxing, which results a normal
		// pointer.
		let cb: Box<Box<dyn FnMut(StreamEvents) + Send>> = Box::new(Box::new(f));

		// This leaks the memory in the Box, meaning it won't get freed at the end of the scope of
		// this function.
		let ptr = Box::into_raw(cb);
		unsafe {
			sys::virStreamEventAddCallback(
				...,
				Some(stream_event_cb_wrapper),
				ptr as *mut _,
				Some(stream_event_free_cb_wrapper),
			);
		}
	}
}

For a global callback:

// This is a very complex type, which is unfortunate. I'll explain every layer here:
//
// Lazy -> necessary because mutexes can only be initialized at runtime (i.e. Mutex::new is not
//         const (yet)). Lazy will just create the Mutex when ERROR_CB_CLOSURE is first accessed.
//
// Mutex -> Necessary to prevent a data race when multiple threads try to update the callback at
//          the same time. Rust enforces this, it's the only way to get a mutable borrow of the
//          contained value.
//
// Option -> Because no closure has been configured when starting, so we need to represent a None
//           value.
//
// Box<Box<...>> -> See above.
//
// dyn FnMut(Error) + Send -> This is a trait object, representing a closure that can be shared
//                            across threads (Send).
static ERROR_CB_CLOSURE: Lazy<Mutex<Option<Box<Box<dyn FnMut(Error) + Send>>>>> =
	Lazy::new(|| Mutex::new(None));

extern "C" fn error_cb_wrapper(data: *mut c_void, err_ptr: sys::virErrorPtr) {
	let cb: &Box<dyn FnMut(Error) + Send> = unsafe { &*(data as *mut Box<dyn FnMut(Error) + Send>) };
	let err = Error::from_ptr(err_ptr);
	std::panic::catch_unwind(|| {
		cb(err);
	});
}

fn set_error_func(f: F)
where
	F: FnMut(Error) + Send + 'static,
{
	let cb: Box<Box<dyn FnMut(Error) + Send>> = Box::new(Box::new(f));

	// This block simply locks the mutex. The Err case occurs when the mutex is poisoned, which
	// happens when the thread holding it panics. We don't really care about that so we always take
	// the lock.
	let mut guard = match ERROR_CB_CLOSURE.lock() {
		Ok(guard) => guard,
		Err(err) => err.into_inner(),
	};

	// Note that we do not leak the Box here as above, instead we only create a pointer.
	let ptr = cb.as_ref() as *const _ as *mut Box<dyn FnMut(Error) + Send>;
	unsafe {
		sys::virSetErrorFunc(ptr as *mut _, Some(error_cb_wrapper));
	}
	// Store the Box in the Mutex. If there was already a Box stored in the Mutex, it will be freed.
	*guard = Some(cb);
}