Guidance on how to embed sciter as a gstreamer plugin for an advanced overlay system in video pipelines
On non mobile OS, one can run webkit based sources and filters to load full blown javascript apps as overlays. I am trying to create a more lightweight approach and instead of using a full blow webkit, would like to use sciter.
gstreamer and its opengl infrastructure seem proper for this, but I don't know how to hook the engine.
What I have done so far was to embed thorvg graphics library https://github.com/thorvg/thorvg but it may be too bare bones for my case, one needs far more than just a drawing api.
I have started a plugin based on gstreamer's own gltestsrc, the existing thorvg bindings are inside <gst-production-overlay/gst-production-overlay-libs/ext/gl/gloverlaydraw.cpp> The embedding itself has 3 functions
void initGLview(GstGLBaseSrc *src, uint32_t w, uint32_t h) {
GstGLFuncs *gl = src->context->gl_vtable;
int32_t targetId;
gl->GetIntegerv(GL_FRAMEBUFFER_BINDING, &targetId);
// TODO: Bind sciter opengl backend to gstreamer's gl abstractions
}
void drawGLview(GstGLBaseSrc *src) {
GstGLFuncs *gl = src->context->gl_vtable;
gl->Enable(GL_BLEND);
gl->BlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
gl->ClearColor(0.0f, 0.0f, 0.0f, 1.0f);
gl->Clear(GL_COLOR_BUFFER_BIT);
// TODO: Invoke frame updated api of sciter
gl->Disable(GL_BLEND);
}
void setupGLview(GstGLBaseSrc *src, uint32_t w, uint32_t h) {
initGLview(src, w, h);
drawGLview(src);
}
void destroyGLview() { }
Some useful findings
- OpenGL apis in gstreamer https://github.com/GStreamer/gst-plugins-base/tree/master/gst-libs/gst/gl
- OpenGL plugins in gstreamer https://github.com/GStreamer/gst-plugins-base/tree/master/ext/gl
- Plugin development - how to test a plugin https://gstreamer.freedesktop.org/documentation/plugin-development/basics/testapp.html?gi-language=c
Gstreamer opengl backend can be configured programmatically or set by env vars, for example:
export GST_GL_PLATFORM=egl
export GST_GL_API=gles2
What is really awesome is that gstreamer has already bindings for all platforms imagined including android and ios, so this can run on mobile phones, where wpe engine is not possible to run. It is very very though to have custom graphics on top of live video grabbed from camera and then restreamed. Grabbing and streaming is solved by gstreamer extremely easy, but when it comes to drawing ... it seriously lacks, that's where i try to fit in sciter.
Find a working thorvg sample attached with build instructions
Starting up
- Fetch thorvg and build locally -
./build-thorvg.h
- Build gstreamer plugin library -
./build-overlay.sh
- Run the test pipeline
./develop.sh