1. 18 Mar, 2019 1 commit
    • Yorick Peterse's avatar
      Rewrite the process scheduler from the ground up · 3e5882be
      Yorick Peterse authored
      The old scheduler was a little over two years old, and due for a
      rewrite. While it worked, it was not very efficient and many features
      were bolted on top; process pinning being an example.
      The new scheduler relies less heavy on locking, only using mutexes
      paired with condition variables to wake up sleeping threads. This will
      allow it to scale much better as the number of threads goes up.
      Another big benefit is clearer code. The old scheduler's code was a
      mess, largely because we focused more on getting a proof of concept out
      instead of building a scheduler for the next few years.
      == Suspending and rescheduling processes
      As part of this rewrite, the way timeouts and rescheduling of processes
      is handled is also rewritten. When a process is suspended and receives a
      message, the sender will try to reschedule it immediately. This makes
      sending messages a little bit more expensive, but allows for much faster
      rescheduling of processes. This also removes the need for a separate
      thread to perform a linear scan over a list of processes to determine
      which ones need to be rescheduled.
      Processes that suspend themselves with a timeout are stored in a binary
      heap, managed by a separate thread. Communication with this thread is
      done using a channel, offloading most of the work to the separate
      timeout thread. When a process with a timeout is rescheduled, its entry
      in the heap is marked as invalid instead of being removed. This makes
      the operation a constant time operation, at the cost of the binary heap
      getting fragmented. To combat fragmentation, the timeout thread will
      periodically remove invalid entries from the heap.
      Rescheduling processes is done entirely using atomic operations, instead
      of using mutexes. This requires some careful coding to take into account
      multiple threads trying to reschedule the same process, but should allow
      all of this to scale much better.
      The new approach of suspending and rescheduling processes requires one
      additional word of memory per process. This memory is used to mark the
      process as suspended, and to optionally store a pointer to its timeout
      (if one was used).
      == Message counts
      The number of messages in a mailbox is now stored explicitly using an
      atomic integer, instead of obtaining this from the synchronised
      data structures internal to a mailbox. This requires one word of extra
      memory per process, but makes it much cheaper to check if a process has
      messages. This is important, because when rescheduling a process such
      checks are performed several times.
      == Asynchronous IO and further improvements
      While this commit does not add support for asynchronous IO operations,
      the rewrite will make it easier to do so in future commits. The process
      lookup table also remains unchanged, but we're currently investigating
      if we can get rid of PIDs and the lookup table entirely; potentially
      speeding up process spawning by quite a bit.
  2. 23 Dec, 2018 1 commit
  3. 22 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Added method Float.to_bits · a4c2d681
      Yorick Peterse authored
      This method can be used to obtain the bitwise representation of a Float.
      This in turn can be used to perform an approximate equality comparison
      by checking bits of a Float.
  4. 19 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Move all "class" methods to module methods · 1a577430
      Yorick Peterse authored
      In the past I have gone back and forth a bit on the idea of using
      class/static methods or not. With this commit I'm making a final
      decision on this topic: Inko will not have class/static methods, at
      least not until 1.0. Instead, Inko will use module methods. This means
      that the following methods have been changed:
      * HashMap.from_array  -> std::hash_map.from_array
      * Trait.implement_for -> std::trait.implement
      * Integer.from_string -> std::integer.parse
      * Float.from_string   -> std::float.parse
      To make this happen some compiler changes had to be made to optimise
      `hash_map.from_array` and to use `trait.implement` instead of
  5. 16 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Parsing of Strings into Floats and Integers · 6558d6b5
      Yorick Peterse authored
      This adds support for parsing a String into a Float and an Integer.
      There are two ways of doing so:
      1. By sending `to_integer` or `to_float` to a `String`.
      2. Using `Integer.from_string` or `Float.from_string`.
      Using `to_integer` and `to_float` will perform a lossy conversion:
      returning 0 or 0.0 for invalid input. Using the `from_string` methods
      will result in a strict conversion, with an error being thrown for
      invalid input.
      Fixes #134
      Fixes #156
  6. 11 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Move std::reflection into std::mirror · b3c7e36f
      Yorick Peterse authored
      This moves all code from `std::reflection` into `std::mirror`, finally
      removing the need for the two separate modules. We also renamed
      `kind_of?` to `implements_trait?`, and implemented both it and
      `instance_of?` in pure Inko. This in turn allows us to remove some
      specialised VM instructions.
      Fixes #153
  7. 10 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Tests for std::process and remove receive_if · bf0d6d66
      Yorick Peterse authored
      This adds tests for `std::process`, and removes
      `std::process.receive_if`. The `receive_if` method has always been a bit
      of a weird method. It was originally meant to allow some form of pattern
      matching on messages, but this turned out to be less useful than hoped
      due to the return type still being `Dynamic`.
      The `timeout` argument was also not used properly, and solving this
      would not be easily doable.  This is because there was no single
      operation to apply the timeout to, instead the implementation of
      `receive_if` contained two `receive` calls, and some other logic.
      Handling all this would require us to keep track of the elapsed time
      ourselves, which is rather tricky should the process be stuck in a
      blocking receive.
      Instead of trying to come up with all kinds of crazy ideas, I have opted
      to simply remove `receive_if` for the time being. This makes some test
      runner code a bit more fragile (in theory), but I doubt this will pose
      any real problems any time soon.
  8. 08 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Removed storing of entire process statuses · 86cadf10
      Yorick Peterse authored
      Obtaining the process status has always been a bit questionable. For
      one, it's not particularly useful to see that a process is running or
      being garbage collected. Second, it requires a full 8 bytes of memory
      per process to store.
      In this commit, we drop the storing of full process statuses, and add a
      boolean flag "waiting for message" that we use instead where necessary.
      Currently this won't reduce the size of a process due to alignment
      requirements, but in the future we may be able to work around this by
      reducing the size of other fields.
  9. 06 Nov, 2018 1 commit
    • Yorick Peterse's avatar
      Move interpreter code out of the main loop · e2b5bff5
      Yorick Peterse authored
      A lot of code that used to reside directly in the interpreter loop now
      resides in separate modules and functions. This makes it a bit easier to
      maintain the code, as well as making it easier for a future JIT of sorts
      to reuse this code.
      Not all instructions have had their logic moved to separate functions.
      Typically, if an instruction modifies the instruction pointer its logic
      remains in the interpreter loop.
  10. 30 Oct, 2018 1 commit
    • Yorick Peterse's avatar
      Add a Foreign Function Interface for C code · 26f535fe
      Yorick Peterse authored
      This commit adds support for a basic Foreign Function Interface to C.
      This interface allows Inko code to dynamically load C libraries, obtain
      pointers to variables, and call functions. Data types are automatically
      converted whenever possible. Passing arbitrary Inko objects to C is not
      possible, as otherwise the garbage collector could release memory of
      objects still in use by C code.
  11. 11 Oct, 2018 1 commit
    • Yorick Peterse's avatar
      Rework handling of prototypes of built-in types · f39ce802
      Yorick Peterse authored
      This reworks how the prototypes of built-in types, such as ByteArray,
      are handled. Prior to this commit, various built-in types would use
      Object as their prototype, followed by the runtime correcting this. This
      required the use of `std::reflection` in various places, which would add
      unnecessary runtime overhead.
      Supporting FFI was also made more complicated in this setup. For
      example, in the FFI API a Pointer should be an instance of
      `std::pointer::Pointer`, but the VM has no built-in knowledge of this
      type, meaning it had to use Object as the prototype. This then required
      the FFI runtime code to fix the prototype every time a Pointer was
      created. This complicates the code, and in certain places requires
      different approaches to fix the prototype.
      In this commit, we make sure that all built-in types have a dedicated
      prototype in the VM. We also merge the various GetFooPrototype
      instructions into a single GetBuiltinPrototype instruction, reducing the
      number of instructions necessary. The compiler still exposes separate
      virtual instructions, though this is mostly to keep the prototype IDs
      out of the runtime.
      == Setting object names
      This new setup requires that for a few more types we get the prototype
      and set the object name manually. To make this easier, the compiler now
      supports the virtual instruction `set_object_name`. This allows modules
      to set the correct object name, without having to use the
      `@_object_name` instance attribute directly. This in turn means this
      attribute is now only used in two places:
      1. In the compiler, where it belongs.
      2. In `std::mirror`, in order to obtain the name of the object.
      == DefaultHasher is now in a separate module
      The DefaultHasher type used to reside in `std::hash_map`, but this
      didn't make much sense since it's not tied into the `HashMap` type. With
      the various prototype changes being made I decided that now was a good
      time to move `DefaultHasher` to its own module: `std::hasher`. In the
      future this module might provide hashers using other algorithms, but for
      now it only defines the `DefaultHasher` type.
  12. 08 Oct, 2018 2 commits
    • Yorick Peterse's avatar
      The Platform instruction now returns a string · c1c8f5db
      Yorick Peterse authored
      The "Platform" VM instruction now returns a string, instead of an
      integer. This string contains the name of the underlying platform, such
      as "windows", "linux", or "unix" for a generic Unix (like) system. This
      simplifies the runtime, while being able to detect more platforms. This
      in turn can be useful when binding to C using the upcoming FFI API.
    • Yorick Peterse's avatar
      Supported nested calls to process.blocking · 3d15c20e
      Yorick Peterse authored
      This fixes `std::process.blocking` so it supports nested calls, similar
      to the recently introduced method `std::process.pinned`.
      Fixes #128
  13. 07 Oct, 2018 2 commits
    • Yorick Peterse's avatar
      Expose thread pinning to the runtime · b79e9e8d
      Yorick Peterse authored
      The method `process.pinned` can be used to run a block while pinning the
      process to the current OS thread, unpinning automatically once the block
    • Yorick Peterse's avatar
      Pinning of processes to threads · faf02ea7
      Yorick Peterse authored
      This commits adds the ability to pin a process to a particular OS thread
      in a thread pool. This is useful for the FFI, as certain C functions or
      data structures require to be run on a specific thread. For example,
      libc's "errno" variable uses thread-local storage. This means that if we
      want to run a function that uses it and read "errno", we _have_ to
      ensure both operations are performed on the same thread.
      To support this, each Pool structure now has a Worker structure, which
      stores the thread ID of that worker. This ID can then be stored in a
      process, allowing the scheduler to determine which worker should run the
      == Process pinning
      Pinned processes can not be moved across pools, so the MoveToPool
      instruction becomes a noop for a pinned process. We can not panic in
      this case, as this would prevent pinned processes from using methods
      that try to move a process to the secondary pool.
  14. 20 Sep, 2018 1 commit
    • Yorick Peterse's avatar
      Basic support for late binding of Self · 72e16d4f
      Yorick Peterse authored
      This commit adds basic support for late binding of Self types. When a
      method returns _just_ Self, we now evaluate it into an actual type upon
      use, instead of upon definiing the return type. The use of Self inside a
      generic type, such as Array!(Self), still uses early binding.
      Supporting simple cases of late binding allows us to clean up
      std::range. For example, previously sending `successor` to a `Successor
      + Compare` would produce a `Successor` as the return type. This meant
      having to cast this back to `Successor + Compare` just to use methods
      such as `<=`. With this commit, the compiler will produce the proper
      `Successor + Compare` type, removing the need for manually casting
      With this commit also come some changes to the return types of bodies of
      object, trait, and impl blocks. Previously, the type of these blocks was
      set to the object (e.g. `Person` in `impl Person { ... }`), but this was
      not correct. Instead, the block return type should be inferred based on
      the last expression returned. These changes don't affect any existing
      code, as object/impl/trait themselves return the type of object they
      operate on, not what their _bodies_ return.
      See #107 for some more
      information on binding of Self types.
  15. 10 Sep, 2018 1 commit
  16. 07 Sep, 2018 1 commit
    • Yorick Peterse's avatar
      Fix sending to types that implement UnknownMessage · 3f038c6e
      Yorick Peterse authored
      In a previous refactoring of the compiler's type system, support for
      sending messages to objects that implement
      std::unknown_message::UnknownMessage was broken. This would prevent
      compiling code such as `Nil.foo`. This commit restores support for such
  17. 06 Sep, 2018 1 commit
    • Yorick Peterse's avatar
      Add support for deferred execution of blocks · 703ff73c
      Yorick Peterse authored
      The method `std::process.defer` can be used to schedule a block for
      execution when the calling scope returns, even when a panic is
      triggered. Such blocks are useful when cleaning up resources, such as
      files and sockets.
      Most languages use finalisers for this, but finalisers are difficult to
      implement in Inko. The garbage collector has to be able to deal with
      objects that are resurrected, and somehow be able to schedule finalisers
      for execution. Usage wise, finalisers are also more limited as they are
      defined when defining an object, not when using it. This means you
      (typically) can't create ad-hoc finalisers.
      The use of deferred blocks allows us to work around these restrictions
      and implementation difficulties, and the idea is inspired by Go. One
      downside is that directly using `std::process.defer` can lead to rather
      verbose code, but more high-level abstractions can be added on top
      The order in which blocks are executed is currently not officially
      specified, and thus should not be relied upon. Currently the order will
      be First In Last Out (FILO), but this may change at any given point in
      Using `std::process.defer` is very straightforward. For example, to
      close a file once we are done with it, you'd write something along the
      lines of the following:
          import std::process
          import std::fs::file
          let readme = try! file.read_only('README.md')
          process.defer {
          try! readme.read_string
  18. 29 Aug, 2018 1 commit
    • Yorick Peterse's avatar
      Fix setting the receiver type of Send nodes · 6fffc53a
      Yorick Peterse authored
      Not doing this could result in messages being sent to the wrong objects.
      For example, sending `foo` in an object would result in the receiver
      being `self`, even if `foo` was defined as a module method.
  19. 26 Aug, 2018 2 commits
    • Yorick Peterse's avatar
      Add support for registering panic handlers · 5e6920e3
      Yorick Peterse authored
      A panic handler is a block to execute when a process panics. Once the
      block finishes running, the process terminates. A panic handler can be
      registered using `std::process.panicking`:
          import std::process
          import std::stdio::stderr
          import std::vm
          process.panicking do (error) {
      Each process can only register a single panic handler, and newly
      registered handlers will overwrite any previous ones.
      The block passed to `std::process.panicking` is given the panic message.
      A stacktrace needs to be obtained manually using
    • Yorick Peterse's avatar
      Explicitly bind receivers to blocks and bindings · 79103c8e
      Yorick Peterse authored
      Prior to this commit, "self" was just syntax sugar for obtaining local
      variable 0. This variable in turn was populated by the (implicit)
      argument 0. In other words, this:
          def foo(bar) {
      Was more or less translated into the following:
          def foo(self_arg, bar) {
          foo(self, bar)
      While fairly simple to implement, this poses two problems:
      1. "self" is an implicit argument, which can be confusing for users. For
         example, when using mirrors to obtain the list of method arguments,
         "self" would be included in the list.
      2. The VM could not schedule a Block for execution on its own, because
         it doesn't know what object to pass as the first argument (= self).
      Problem 2 made it impossible to implement panic hooks in a nice way, and
      any future features that require the VM to schedule blocks (e.g. when
      trapping signals).
      To work around this, "self" is now explicitly bound to blocks, when they
      are defined. To execute methods, we use a new instruction:
      RunBlockWithReceiver. This instruction takes a receiver (= the object to
      use for "self") to use when executing the method. The receiver in this
      case will be the object the method was invoked on.
      When a block is created, we no longer create a new binding for it.
      Instead, we store the binding that the block captures, which we later
      set as the parent for the new binding when executing the block. This
      removes the need for allocating a Binding for every block that is
      defined, even when never executed. The block also stores the receiver to
      use when executing said block.
      This setup also means we can remove quite a bit of nasty bits from the
      compiler, as we no longer need to generate implicit arguments and local
      variables. It also allows us to (in the future) obtain the receiver of a
      block (for meta programming), without having to rely on the exact local
      variable index used to store this object.
  20. 21 Aug, 2018 1 commit
    • Yorick Peterse's avatar
      Added std::env for managing environment data · 69a7592e
      Yorick Peterse authored
      The module std::env can be used for obtaining and setting environment
      variables, the home directory, the temporary directory, and more. For
      example, one can obtain environment variable values as follows:
          import std::env
          env['HOME'] # => '/home/yorickpeterse'
      You can also set the value of a variable:
          import std::env
          env['HOME'] = '/home/foo'
      Removing variables is also possible:
          import std::env
          env.remove('HOME') # => Nil
      Or obtain the home directory:
          import std::env
          env.home_directory # => '/home/yorickpeterse'
      You can also obtain and set the working directory:
          import std::env
          try! env.working_directory          # => '/home/yorickpeterse'
          try! env.working_directory = '/tmp' # => '/tmp'
      Arguments can be retrieved using `std::env.arguments`:
          import std::env
          env.arguments # => ['foo', 'bar']
      == Executable changes
      The "inko" executable has been modified to pass additional commandline
      arguments to IVM. IVM in turn has been modified to expose these to the
      VM instructions. This requires us to explicitly store the passed
      arguments in a vm::state::State, as Rust's std::env::args() is immutable
      _and_ includes _all_ arguments (the bytecode file to execute, IVM
      options, etc).
      The arguments passed via the CLI are all interned, removing the need for
      allocating (potentially many) strings every time `std::env.arguments` is
      Fixes #136
  21. 09 Aug, 2018 1 commit
    • Yorick Peterse's avatar
      Expose module names and paths to the runtime · 7e462a2a
      Yorick Peterse authored
      Modules are now created using `Module.new`, instead of manually creating
      an object with its prototype set to `Module`. This in turn allows us to
      store the module name and path in every created module.
      Because methods on Module might conflict with user defined ones, the
      name and file path are not exposed as methods directly. Instead, a
      mirror can be used to retrieve this information. Since instance
      attributes can not be defined in a module, the use of these in Module
      will not conflict with anything.
  22. 08 Aug, 2018 1 commit
    • Yorick Peterse's avatar
      Always set "self" in a lambda · 64afd812
      Yorick Peterse authored
      Previously we would treat it as an argument, only setting it when left
      out. When we send "call" to a lambda, the first argument passed is the
      lambda itself. This would result in "self" being set to the lambda,
      because our override would only kick in if the argument was left out.
      To solve this, we _always_ set self to the module object in a lambda,
      even if an explicit argument is provided.
  23. 08 Jul, 2018 1 commit
  24. 04 Jul, 2018 1 commit
    • Yorick Peterse's avatar
      Turn Boolean back into a regular object · 6e073c2a
      Yorick Peterse authored
      Defining Boolean as a Trait was a nice idea, but it was too limiting.
      For example, the Inspect trait is defined after Boolean, meaning you
      couldn't pass True or False to an argument expecting an Inspect, since
      Boolean did not define Inspect as a required trait.
      To work around this we turn Boolean back into a regular object, and make
      True and False instances of this object. This allows us to refine these
      objects whenever necessary, at the cost of having to implement said code
      for Boolean, True, False.
  25. 02 Jul, 2018 2 commits
  26. 26 Jun, 2018 1 commit
  27. 24 Jun, 2018 2 commits
  28. 20 Jun, 2018 1 commit
    • Yorick Peterse's avatar
      Use byte arrays for IO related operations · c5897da0
      Yorick Peterse authored
      This completely reworks the IO system of both IVM and the Inko runtime.
      Instead of operating on strings or arrays of integers, the VM now
      operates on proper byte arrays. These byte arrays are stored as a
      Vec<u8> in the VM, instead of a Vec<ObjectPointer>. This ensures no
      space is wasted by storing bytes as 8 byte values.
      The standard library now operates on a ByteArray in various places,
      instead of using Array!(Integer). This ByteArray type is defined in the
      std::byte_array module, and acts similar to an Array. Unlike an Array,
      writing an out of bounds index will panic, as there is no reasonable
      default value to use for padding the byte array.
      Some related code has also been changed. For example,
      std::string.from_bytes has been moved to ByteArray.to_string and
      ByteArray.drain_to_string. String.to_bytes in turn has been moved to
      All of these changes combined means that reading a 64 MB file only
      requires about 70 MB in total, instead of requiring around 1 GB.
      Fixes #108
  29. 15 Jun, 2018 1 commit
    • Yorick Peterse's avatar
      Added a StringBuffer object · 852e0084
      Yorick Peterse authored
      This object can be used to efficiently concatenate multiple strings
      together, without allocating intermediate String objects on the heap.
  30. 14 Jun, 2018 3 commits
  31. 12 Jun, 2018 1 commit
  32. 10 Jun, 2018 2 commits