Move 'compile' to 'sandbox.compileTrustedFunction' and 'sleep' to 'thread.sleep'

parent 2ae70dd2
......@@ -170,7 +170,7 @@ package cross-bred with Chrome DevTools with remote logging built in. Logs are
important, and could and should be invaluable tools for analyzing systems in
production, which is often not the case due to subpar native logging facilities.
### stdio.getchar : () => Number | Error EOF | undefined
### stdio.getchar() => Number | Error EOF | undefined
This is a sweet and simple little function, and ~~for now it's the only way to get at
the standard input stream (*stdin* in POSIX parlance)~~ you can use it instead of
getbuffer when you only want to get one byte at a time. Working with binary input
......@@ -216,7 +216,7 @@ process incoming data, and send it to...
### stdout : pull-stream Sink
### stdio.putchar : (Number) => Number | Error EOF | undefined
### stdio.putchar(Number) => Number | Error EOF | undefined
Writes a single byte to `stdout`. Be fancy and write your numbers in hex! See examples/stdin/stdout-unbuffered.js
Should return the same number you gave it, the EOF error if stdout got closed (untested),
undefined if non-blocking write is supported and the output was temporarily unavailable (untested),
......@@ -251,18 +251,24 @@ L1 cache misses and power consumption. This multiprocessing is achieved using th
OpenMP standard which is natively supported by most C compilers, rather than using
a non-portable threading library.
### threadpool.size() => Number
(Wraps `omp_get_num_threads()` if that means something to you.)
This is the number of physical threads that are active. Right now, that is fixed
and equal to the number of logical cores.
### threadpool.id() => Number
### thread.id() => Number
(Wraps `omp_get_thread_num()` if that means something to you.)
Each thread is assigned an `id`, which runs from 0 to threadpool.size() - 1.
Each thread is assigned an `id`, which runs from 0 to thread.pool().length - 1.
If for some reason you don't want to run multiple instances (e.g.
for a command line utility that reads from stdin and writes to stdout) add a
check at the beginning of your function and only run it if `threadpool.id() === 0`.
check at the beginning of your function and only run it if `thread.id() === 0`.
### thread.pool() => [ Number ]
This returns an array of the ids of all the threads, including this one.
Right now, that means it is an array of integers 0 through number of logical cores - 1.
At some point you'll be able to pass messages between threads and other cool stuff
using their ids.
### thread.sleep(Number milliseconds) => void
Suspends the current thread for the given number of milliseconds. Fractional
milliseconds are supported up to nanosecond resolution, but accuracy depends
on the underlying C library implementation of `nanosleep`.
## Event Loop
......@@ -298,54 +304,83 @@ Right now this is the only queue. I'm storing it in the heap-stash. The code in
using any C code. This may prove a totally awesome idea or a completely
terrible one; I don't know yet.
## Module System
## ~~Module System~~ Sandbox
A wise man once said,
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr">looks interesting!<br>to be honest, skip the module system. just run a single file and I&#39;ll use browserify.<br>but process/actor model: BIG YES</p>&mdash; Dominic Tarr (@dominictarr) <a href="https://twitter.com/dominictarr/status/868585203286949888">May 27, 2017</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
### compile : (filepath : String) => Function
CommonJS (as the Node.js `require` style of module loading is called) is overly
complicated. Large portions of its behavior is unspecified or implicit. It is
common for beginners to be confused as to why `require('foobar.js')` works but
`require('foo/bar.js')` does not, or to be surprised by Node's module caching
system or the weird directory structure dependencies need to be installed in.
So I've decided to leave modules as a "trivial exercise for the user". Just kidding!
There are two excellent ways to approach modules:
In dukboot, the base unit of code is not modules, but what
I call "function files". A function file is very simple: it's a file that contains
exactly one single JavaScript function. The argument to `compile` is always
a function filename. To keep you from writing spaghetti code, function files can
The first is to use a module bundler. Want to write ES6 modules? Look no further than [dukroll](https://gitlab.com/dukboot/dukroll).
However, bundling everything into one file though loses us some opportunity to sandbox
third party code. And what about dynamic module loading? We'll need that if
we ever want to do hot module reloading. This leads us to the second way, which
needs to enable us to load & compile code, without running `eval`, and delay executing it / execute it in a
controlled environment. I haven't nailed down the details yet, but that is what this
section is about.
### Function Files
In dukboot, the base unit of code is not modules, but what I call "function files".
A function file is very simple: it's a file that contains exactly one single JavaScript function.
### sandbox.compileTrustedFunction(String filepath) => Function
The argument to `compile` must be a path to a "function file" as defined above.
The path is always relative to the filename of the calling function. It expects
that file to contain a single JavaScript function, and it will be treated as a
"strict mode" function because that's the only kind anyone should be writing
these days. It is a "trusted" function because it is compiled in the same global scope & heap as the calling function.
I'm planning on adding ways to instantiate functions in a separate scope and heap.
To keep you from writing spaghetti code, function files can
only call compile on other function files in the same directory or a subdirectory.
In Node, it is common to see users attept to create deep folder hierarchies and
then break out of them with `require('./../../../foo/bar/baz')`. This is an
extremely poor practice and makes your code fragile and hard to refactor. If
you find yourself wanting to do this, dukboot won't let you. Instead, the
prefered convention is dependency injection - that is, require 'baz' somewhere
you find yourself wanting to do this, dukboot won't let you. Instead, use a sane
method like dependency injection. That is to say, require 'baz' somewhere
higher up and pass it as an argument to functions that need it. In fact, the
entire core module system works this way: your top-level process will simply receive
`modules` as one of its arguments.
entire core module system works this way: your top-level function receives
an `imports` object as it initial argument, which looks something like this:
It is always relative to the filename of the parent function. It expects
that file to contain a single JavaScript function, and it will be treated as a
"strict mode" function because that's the only kind anyone should be writing
these days. It is compiled in the same global scope & heap as the parent function.
Function files
```
{
"main": [object Object],
"stdio": [object Object],
"threadpool": [object Object],
"netLib": [object Object],
...
}
```
## Clocks / Timing
## Networking
### sleep : (milliseconds : Number) => void
Suspends the current thread for the given number of milliseconds. Fractional
milliseconds are supported up to nanosecond resolution, but accuracy depends
on the underlying C library implementation of `nanosleep`.
Currently I'm using netLib because... well the author states it best:
## Networking I/O
> The [netLib Network Communication Library](https://github.com/IanSeyler/netLib) is a free, open-source C library for network communication over TCP/IP. It currently compiles on Windows, Unix (Linux, \*BSD, Mac OS X) and BeOS. The netLib Network Communication Library was designed for simplicity, portability, and ease of use.
>
> -Ian Seyler
### supersocket - TBD
This is a pretty sweet library. It's a single C file, and does enough to make a basic HTTP server. The netLib library itself is well documented in `deps/netLib/docs/manual.html`. Right now I'm just thinly wrapping the original functions (which is why the module is `netLib` instead of something more general like `tcp`) so you should refer to the netLib documentation. But I will list the bindings here:
I would like to have some kind of suped up socket like zeromq or nanomsg built into dukboot, so that you can start building advanced network topologies right away without worrying about protocols. However, I can't get either of them to compile yet.
### netLib.netInit() => void
### netLib.netStop() => void
### netLib.netInitSocket() => SocketHandle
### netLib.netConnect(SocketHandle socket, String address, Number port) => void
### netLib.netDisconnect(SocketHandle socket) => void
### netLib.netListen(SocketHandle socket, Number port) => void
### netLib.netAccept(SocketHandle listenSock, SocketHandle netSock) => void
### netLib.netIsDataPending(SocketHandle socket) => void
### netLib.netSend(SocketHandle socket, Uint8Array buffer) => void
### netLib.netRecv(SocketHandle socket, Uint8Array buffer) => void
### netLib.netSocketStruct(SocketHandle socket) => { number: Integer, inuse: Boolean, retval: Integer, timeout: undefined, addr: undefined }
### netLib
> The [netLib Network Communication Library](https://github.com/IanSeyler/netLib) is a free, open-source C library for network communication over TCP/IP. It currently compiles on Windows, Unix (Linux, *BSD, Mac OS X) and BeOS. The netLib Network Communication Library was designed for simplicity, portability, and ease of use.
>
> -Ian Seyler
### supersocket - TBD
This is a pretty sweet library. It's a single C file, and does enough to make a basic HTTP server. I'm still working on the bindings, but the netLib library itself is well documented in `deps/netLib/docs/manual.html`. You can see which functions I've written bindings for in `examples/net/webserver.js`.
I would like to have some kind of suped up socket like zeromq or nanomsg built into dukboot, so that you can start building advanced network topologies right away without worrying about protocols. However, I can't get either of them to compile yet.
## Disk I/O
......
function (args) {
var env = args.environ;
var argv = args.argv;
var modules = args.modules;
var Queues = args.Queues;
function (imports) {
var env = imports.environ;
var argv = imports.argv;
var netLib = imports.netLib
var Queues = imports.Queues;
var timers = Queues.timers;
var compile = modules.compile;
var compile = imports.sandbox.compileTrustedFunction;
var print = imports.stdio.print
if (args.threadpool.id() !== 0) return
if (imports.thread.id() !== 0) return
var queue = []
......@@ -23,10 +24,6 @@ function (args) {
return true
}
// modules.webserver();
var print = stdio.print
var netLib = modules.netLib
var port = 8080;
var buffsize = 2000;
......
function (args) {
var env = args.environ;
var argv = args.argv;
var modules = args.modules;
var Queues = args.Queues;
function (imports) {
var env = imports.environ;
var argv = imports.argv;
var Queues = imports.Queues;
var timers = Queues.timers;
var compile = modules.compile;
// modules.webserver();
var print = args.stdio.print
var netLib = modules.netLib
var compile = imports.sandbox.compileTrustedFunction;
var print = imports.stdio.print
var netLib = imports.netLib
var port = 8080;
var buffsize = 2000;
......@@ -34,7 +31,7 @@ function (args) {
var buffer = Uint8Array.allocPlain(1000);
var tint = 0;
print("Web Server demo for netLib v0.4\nListening on port" + port + "\n");
print("Web Server demo for netLib v0.4\nListening on port " + port + "\n");
// Start the netLib library
netLib.netInit();
......
function (o) {
o.stdio.print(o.threadpool.id() + '/' + o.threadpool.size())
o.stdio.print(o.thread.id() + '/' + o.thread.pool().length)
}
......@@ -2,7 +2,7 @@
// dukroll (https://gitlab.com/dukboot/dukroll)
import pull from 'pull-stream'
if (imports.threadpool.id() === 0) {
if (imports.thread.id() === 0) {
var c = pull.count()
c(null, imports.stdio.print)
c(null, imports.stdio.print)
......
function (imports) {
if (imports.thread.id() % 2 === 0) {
imports.thread.sleep(5000)
imports.stdio.print(imports.thread.id() + ': ' + 'ha')
} else {
for (var i = 0; i < 5; i++) {
imports.stdio.print(imports.thread.id() + ': ' + i)
imports.thread.sleep(1000 * Math.random())
}
}
}
function (o) {
// For fun, let every thread race to read from stdin. Quite amusing.
// Results will (probably) be printed out of order.
var thread = o.threadpool.id()
var thread = o.thread.id()
var c
while (true) {
c = o.stdio.getchar()
......
......@@ -22,7 +22,7 @@ function (o) {
return true
}
if (o.threadpool.id() === 0) {
if (o.thread.id() === 0) {
var c
var chars = []
read(null, function next(end, data) {
......
function (o) {
function (imports) {
// If you want to read bytes in order, I would recommend having
// a single master thread read the bytes. You can divide the work later.
var thread = o.threadpool.id()
var thread = imports.thread.id()
if (thread === 0) {
var c;
while (true) {
c = o.stdio.getchar()
o.stdio.print(thread + ': ' + c)
c = imports.stdio.getchar()
imports.stdio.print(thread + ': ' + c)
if (c instanceof Error) break
o.modules.sleep(100)
imports.thread.sleep(100)
}
}
}
function (o) {
function (imports) {
// If you want to read bytes in order, I would recommend having
// a single master thread read the bytes. You can divide the work later.
var thread = o.threadpool.id()
var thread = imports.thread.id()
if (thread === 0) {
var UTF8decoder = new TextDecoder('utf-8')
var c
var buffer = new Uint8Array(8);
while (true) {
c = o.stdio.getbuffer(buffer)
c = imports.stdio.getbuffer(buffer)
if (c instanceof Error) break
if (c === undefined) o.modules.sleep(100)
if (c === undefined) imports.thread.sleep(100)
if (c > 0) {
o.stdio.print(thread + ': read ' + c + ' bytes')
o.stdio.print(UTF8decoder.decode(buffer.subarray(0, c)))
imports.stdio.print(thread + ': read ' + c + ' bytes')
imports.stdio.print(UTF8decoder.decode(buffer.subarray(0, c)))
}
}
}
......
function (o) {
if (o.threadpool.id() === 0) {
if (o.thread.id() === 0) {
// This runs in about 550ms on my machine.
// Interestingly, I get the same performance buffering "manually" like this,
// as I do by leaving stdout buffering enabled in the C layer and running
......
function (o) {
if (o.threadpool.id() === 0) {
if (o.thread.id() === 0) {
// On my system, this takes 18.0 seconds to run!
// This is because by default we are not buffering output.
// See `stdout-buffered` example next.
......
function (o) {
// If you want to read bytes in order, I would recommend having
// a single master thread read the bytes. You can divide the work later.
if (o.threadpool.id() === 0) {
if (o.thread.id() === 0) {
var c;
var chars = [];
while (true) {
......
function (o) {
function (imports) {
var queue = []
function schedule (task) {
......@@ -7,7 +7,7 @@ function (o) {
function read (end, cb) {
if (end) return cb(end)
var c = o.stdio.getchar()
var c = imports.stdio.getchar()
if (c === undefined) return schedule([read, end, cb])
if (c instanceof Error) return cb(c)
return schedule([cb, null, c])
......@@ -22,26 +22,26 @@ function (o) {
return true
}
if (o.threadpool.id() === 0) {
var unicodeStream = o.modules.compile('unicode-stream.js')
if (imports.thread.id() === 0) {
var unicodeStream = imports.sandbox.compileTrustedFunction('unicode-stream.js')
var readUnicode = unicodeStream(read)
var c
var chars = []
readUnicode(null, function next(end, char) {
if (end) throw end
o.stdio.print(char)
imports.stdio.print(char)
schedule([readUnicode, null, next])
// readUnicode(null, next)
})
var i = 0;
try {
while (true) {
o.stdio.print("Tick count: " + i)
imports.stdio.print("Tick count: " + i)
tick()
i++
}
} catch (e) {
o.stdio.print(e)
imports.stdio.print(e)
}
}
}
......@@ -13,7 +13,7 @@ function (imports) {
// A stupid way to make dependencies available, that will
// probably necessary to be compatible with ye olde node.
global.dir = imports.modules.dir
var testme = imports.modules.compile('subexample/grandchild.js');
var testme = imports.sandbox.compileTrustedFunction('subexample/grandchild.js');
imports.stdio.print(JSON.stringify(testme(3, 4)));
return
}
function (imports) {
var env = imports.main.environ();
var argv = imports.main.argv();
var modules = imports.modules;
var Queues = imports.Queues;
var timers = Queues.timers;
var testme = modules.compile('child.js');
var testme = imports.sandbox.compileTrustedFunction('child.js');
testme(imports);
return
}
function (imports) {
var env = imports.main.environ();
var argv = imports.main.argv();
var modules = imports.modules;
var thread = imports.thread;
var Queues = imports.Queues;
var timers = Queues.timers;
var compile = imports.modules.compile;
var compile = imports.sandbox.compileTrustedFunction;
var printTime = compile('printTime.js')(imports.stdio.print);
var orderedInsert = compile('orderedInsert.js')();
var byMstimestampAscending = function (a, b) {
......@@ -25,7 +25,7 @@ function (imports) {
// Dequeue
var event = timers.shift();
// Schedule next event (TODO: allow/handle waking up prematurely via an interupt)
modules.sleep(event.t - Date.now());
thread.sleep(event.t - Date.now());
// Run synchronously (for now)
event.cb();
}
......
......@@ -5,8 +5,7 @@
const int port = 8192;
const int buffsize = 2000;
void app_push_netLib(duk_context *ctx) {
duk_push_object(ctx);
void push_bindings_netLib(duk_context *ctx) {
duk_push_c_function(ctx, wrap_netInit, 0);
duk_put_prop_string(ctx, -2, "netInit");
duk_push_c_function(ctx, wrap_netStop, 0);
......
#include "duktape/duktape.h"
#include <omp.h>
#include "bindings_openmp.h"
void push_bindings_openmp(duk_context *ctx) {
duk_push_c_function(ctx, wrap_omp_get_thread_num, 0);
duk_put_prop_string(ctx, -2, "id");
duk_push_c_function(ctx, wrap_omp_get_num_threads, 0);
duk_put_prop_string(ctx, -2, "size");
}
duk_ret_t wrap_omp_get_thread_num(duk_context *ctx) {
duk_push_int(ctx, omp_get_thread_num());
return 1;
}
duk_ret_t wrap_omp_get_num_threads(duk_context *ctx) {
duk_push_int(ctx, omp_get_num_threads());
return 1;
}
#include "duktape/duktape.h"
#include "app_push_trusted_function.h"
int app_push_trusted_function (duk_context *ctx) {
#include "bindings_sandbox.h"
void push_bindings_sandbox(duk_context *ctx) {
duk_push_c_function(ctx, c_compile_trusted_function, 1);
duk_put_prop_string(ctx, -2, "compileTrustedFunction");
}
duk_ret_t c_compile_trusted_function (duk_context *ctx) {
// Safety check filename argument
app_assert_safe_path(ctx);
const char *filename = duk_get_string(ctx, -1);
......
#include <time.h>
#include <math.h>
#include <errno.h>
#include <omp.h>
#include "duktape/duktape.h"
#include "native_sleep.h"
#include "bindings_thread.h"
void push_bindings_thread(duk_context *ctx) {
duk_push_c_function(ctx, c_thread_id, 0);
duk_put_prop_string(ctx, -2, "id");
duk_push_c_function(ctx, c_thread_pool, 0);
duk_put_prop_string(ctx, -2, "pool");
duk_push_c_function(ctx, c_thread_sleep, 1);
duk_put_prop_string(ctx, -2, "sleep");
}
duk_ret_t c_thread_id(duk_context *ctx) {
duk_push_int(ctx, omp_get_thread_num());
return 1;
}
duk_ret_t c_thread_pool(duk_context *ctx) {
duk_push_array(ctx);
size_t numThreads = omp_get_num_threads();
for (size_t i = 0; i < numThreads; i++) {
duk_push_int(ctx, i);
duk_put_prop_index(ctx, -2, i);
}
return 1;
}
// Takes its parameter in milliseconds, but
// supports sub-millisecond accuracy by accepting a float.
duk_int_t native_sleep (duk_context *ctx) {
duk_int_t c_thread_sleep (duk_context *ctx) {
duk_double_t ms = duk_get_number(ctx, -1);
duk_double_t sec = 0;
struct timespec t;
......
......@@ -77,10 +77,6 @@ int main(int argc, char *argv[] /* char *environ[] */) {
// Construct the prime argument
duk_push_object(ctx);
// Provide access to some OpenMP thread info
duk_push_object(ctx);
push_bindings_openmp(ctx);
duk_put_prop_string(ctx, -2, "threadpool");
// Provide access to the environment
duk_push_object(ctx);
push_bindings_main(ctx);
......@@ -89,20 +85,26 @@ int main(int argc, char *argv[] /* char *environ[] */) {
duk_push_object(ctx);
push_bindings_stdio(ctx);
duk_put_prop_string(ctx, -2, "stdio");
// Provide access to threading primitives
duk_push_object(ctx);
push_bindings_thread(ctx);
duk_put_prop_string(ctx, -2, "thread");
// Provide access to TCP sockets
duk_push_object(ctx);
push_bindings_netLib(ctx);
duk_put_prop_string(ctx, -2, "netLib");
// Provide access to sandbox tools
duk_push_object(ctx);
push_bindings_sandbox(ctx);
duk_put_prop_string(ctx, -2, "sandbox");
// Provide access to native functions
app_unstash_global_string(ctx, "Duktape"); // summon from the magic stash
duk_put_prop_string(ctx, -2, "Duktape");
duk_push_object(ctx);
duk_push_c_function(ctx, app_push_dir, 1);
duk_put_prop_string(ctx, -2, "dir");
app_unstash_global_string(ctx, "Duktape"); // summon from the magic stash
duk_put_prop_string(ctx, -2, "Duktape");
duk_push_c_function(ctx, app_push_trusted_function, 1);
duk_put_prop_string(ctx, -2, "compile");
duk_push_c_function(ctx, app_assert_safe_path, 1);
duk_put_prop_string(ctx, -2, "assert_safe_path");
duk_push_c_function(ctx, native_sleep, 1);
duk_put_prop_string(ctx, -2, "sleep");
app_push_netLib(ctx);
duk_put_prop_string(ctx, -2, "netLib");
duk_put_prop_string(ctx, -2, "modules");
// Provide direct access to the event loop
duk_push_object(ctx);
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment