1. 18 Mar, 2019 1 commit
    • Yorick Peterse's avatar
      Rewrite the process scheduler from the ground up · 3e5882be
      Yorick Peterse authored
      The old scheduler was a little over two years old, and due for a
      rewrite. While it worked, it was not very efficient and many features
      were bolted on top; process pinning being an example.
      
      The new scheduler relies less heavy on locking, only using mutexes
      paired with condition variables to wake up sleeping threads. This will
      allow it to scale much better as the number of threads goes up.
      
      Another big benefit is clearer code. The old scheduler's code was a
      mess, largely because we focused more on getting a proof of concept out
      instead of building a scheduler for the next few years.
      
      == Suspending and rescheduling processes
      
      As part of this rewrite, the way timeouts and rescheduling of processes
      is handled is also rewritten. When a process is suspended and receives a
      message, the sender will try to reschedule it immediately. This makes
      sending messages a little bit more expensive, but allows for much faster
      rescheduling of processes. This also removes the need for a separate
      thread to perform a linear scan over a list of processes to determine
      which ones need to be rescheduled.
      
      Processes that suspend themselves with a timeout are stored in a binary
      heap, managed by a separate thread. Communication with this thread is
      done using a channel, offloading most of the work to the separate
      timeout thread. When a process with a timeout is rescheduled, its entry
      in the heap is marked as invalid instead of being removed. This makes
      the operation a constant time operation, at the cost of the binary heap
      getting fragmented. To combat fragmentation, the timeout thread will
      periodically remove invalid entries from the heap.
      
      Rescheduling processes is done entirely using atomic operations, instead
      of using mutexes. This requires some careful coding to take into account
      multiple threads trying to reschedule the same process, but should allow
      all of this to scale much better.
      
      The new approach of suspending and rescheduling processes requires one
      additional word of memory per process. This memory is used to mark the
      process as suspended, and to optionally store a pointer to its timeout
      (if one was used).
      
      == Message counts
      
      The number of messages in a mailbox is now stored explicitly using an
      atomic integer, instead of obtaining this from the synchronised
      data structures internal to a mailbox. This requires one word of extra
      memory per process, but makes it much cheaper to check if a process has
      messages. This is important, because when rescheduling a process such
      checks are performed several times.
      
      == Asynchronous IO and further improvements
      
      While this commit does not add support for asynchronous IO operations,
      the rewrite will make it easier to do so in future commits. The process
      lookup table also remains unchanged, but we're currently investigating
      if we can get rid of PIDs and the lookup table entirely; potentially
      speeding up process spawning by quite a bit.
      3e5882be
  2. 26 Dec, 2018 1 commit
  3. 23 Dec, 2018 1 commit
  4. 22 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Added method Float.to_bits · a4c2d681
      Yorick Peterse authored
      This method can be used to obtain the bitwise representation of a Float.
      This in turn can be used to perform an approximate equality comparison
      by checking bits of a Float.
      a4c2d681
  5. 19 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Move all "class" methods to module methods · 1a577430
      Yorick Peterse authored
      In the past I have gone back and forth a bit on the idea of using
      class/static methods or not. With this commit I'm making a final
      decision on this topic: Inko will not have class/static methods, at
      least not until 1.0. Instead, Inko will use module methods. This means
      that the following methods have been changed:
      
      * HashMap.from_array  -> std::hash_map.from_array
      * Trait.implement_for -> std::trait.implement
      * Integer.from_string -> std::integer.parse
      * Float.from_string   -> std::float.parse
      
      To make this happen some compiler changes had to be made to optimise
      `hash_map.from_array` and to use `trait.implement` instead of
      `Trait.implement_for`.
      1a577430
  6. 16 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Parsing of Strings into Floats and Integers · 6558d6b5
      Yorick Peterse authored
      This adds support for parsing a String into a Float and an Integer.
      There are two ways of doing so:
      
      1. By sending `to_integer` or `to_float` to a `String`.
      2. Using `Integer.from_string` or `Float.from_string`.
      
      Using `to_integer` and `to_float` will perform a lossy conversion:
      returning 0 or 0.0 for invalid input. Using the `from_string` methods
      will result in a strict conversion, with an error being thrown for
      invalid input.
      
      Fixes #134
      Fixes #156
      6558d6b5
  7. 11 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Move std::reflection into std::mirror · b3c7e36f
      Yorick Peterse authored
      This moves all code from `std::reflection` into `std::mirror`, finally
      removing the need for the two separate modules. We also renamed
      `kind_of?` to `implements_trait?`, and implemented both it and
      `instance_of?` in pure Inko. This in turn allows us to remove some
      specialised VM instructions.
      
      Fixes #153
      b3c7e36f
  8. 10 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Tests for std::process and remove receive_if · bf0d6d66
      Yorick Peterse authored
      This adds tests for `std::process`, and removes
      `std::process.receive_if`. The `receive_if` method has always been a bit
      of a weird method. It was originally meant to allow some form of pattern
      matching on messages, but this turned out to be less useful than hoped
      due to the return type still being `Dynamic`.
      
      The `timeout` argument was also not used properly, and solving this
      would not be easily doable.  This is because there was no single
      operation to apply the timeout to, instead the implementation of
      `receive_if` contained two `receive` calls, and some other logic.
      Handling all this would require us to keep track of the elapsed time
      ourselves, which is rather tricky should the process be stuck in a
      blocking receive.
      
      Instead of trying to come up with all kinds of crazy ideas, I have opted
      to simply remove `receive_if` for the time being. This makes some test
      runner code a bit more fragile (in theory), but I doubt this will pose
      any real problems any time soon.
      bf0d6d66
  9. 08 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Removed storing of entire process statuses · 86cadf10
      Yorick Peterse authored
      Obtaining the process status has always been a bit questionable. For
      one, it's not particularly useful to see that a process is running or
      being garbage collected. Second, it requires a full 8 bytes of memory
      per process to store.
      
      In this commit, we drop the storing of full process statuses, and add a
      boolean flag "waiting for message" that we use instead where necessary.
      Currently this won't reduce the size of a process due to alignment
      requirements, but in the future we may be able to work around this by
      reducing the size of other fields.
      86cadf10
  10. 27 Nov, 2018 1 commit
    • Yorick Peterse's avatar
      Improve remapping of initialised type parameters · 8450931d
      Yorick Peterse authored
      This improves the compiler's support for remapping type parameters to
      the appropriate type parameter instance. For example, consider type T
      that has type parameter A initialised to type parameter B, and B is
      initialised to C. Previously, when looking up the instance of A the
      compiler would produce B. As of this commit, the compiler would produce
      type C instead. This removes the need for type annotations in various
      cases, most notably when working with iterators.
      
      Fixes #117
      8450931d
  11. 25 Nov, 2018 2 commits
  12. 22 Nov, 2018 1 commit
  13. 21 Nov, 2018 2 commits
    • Yorick Peterse's avatar
      Remove uninitialised params from return types · d95a357c
      Yorick Peterse authored
      This changes the compiler so that type parameters that map to
      uninitialised type parameters have their instances removed. For type
      `Thing!(A -> B)` this means that the mapping of A to B is removed if B
      is an uninitialised type parameter.
      
      This change ensures that the compiler can better handle HashMap literals
      and other types that may not initialise all type parameters. This in
      turn ensures that code such as the following no longer errors:
      
          let x: HashMap!(String, String) = %[]
      
      Previously this would error because `%[]` was inferred has
      `HashMap!(Hash + Equal, V)`, and that type can not safely substitute
      `Hashmap!(String, String)`. With this change, `%[]` is basically
      inferred as `HashMap!(?, ?)`, allowing the compiler to then infer it
      into `HashMap!(String, String)`.
      d95a357c
    • Yorick Peterse's avatar
      8cd6b796
  14. 06 Nov, 2018 1 commit
    • Yorick Peterse's avatar
      Move interpreter code out of the main loop · e2b5bff5
      Yorick Peterse authored
      A lot of code that used to reside directly in the interpreter loop now
      resides in separate modules and functions. This makes it a bit easier to
      maintain the code, as well as making it easier for a future JIT of sorts
      to reuse this code.
      
      Not all instructions have had their logic moved to separate functions.
      Typically, if an instruction modifies the instruction pointer its logic
      remains in the interpreter loop.
      e2b5bff5
  15. 30 Oct, 2018 1 commit
    • Yorick Peterse's avatar
      Add a Foreign Function Interface for C code · 26f535fe
      Yorick Peterse authored
      This commit adds support for a basic Foreign Function Interface to C.
      This interface allows Inko code to dynamically load C libraries, obtain
      pointers to variables, and call functions. Data types are automatically
      converted whenever possible. Passing arbitrary Inko objects to C is not
      possible, as otherwise the garbage collector could release memory of
      objects still in use by C code.
      26f535fe
  16. 11 Oct, 2018 1 commit
    • Yorick Peterse's avatar
      Rework handling of prototypes of built-in types · f39ce802
      Yorick Peterse authored
      This reworks how the prototypes of built-in types, such as ByteArray,
      are handled. Prior to this commit, various built-in types would use
      Object as their prototype, followed by the runtime correcting this. This
      required the use of `std::reflection` in various places, which would add
      unnecessary runtime overhead.
      
      Supporting FFI was also made more complicated in this setup. For
      example, in the FFI API a Pointer should be an instance of
      `std::pointer::Pointer`, but the VM has no built-in knowledge of this
      type, meaning it had to use Object as the prototype. This then required
      the FFI runtime code to fix the prototype every time a Pointer was
      created. This complicates the code, and in certain places requires
      different approaches to fix the prototype.
      
      In this commit, we make sure that all built-in types have a dedicated
      prototype in the VM. We also merge the various GetFooPrototype
      instructions into a single GetBuiltinPrototype instruction, reducing the
      number of instructions necessary. The compiler still exposes separate
      virtual instructions, though this is mostly to keep the prototype IDs
      out of the runtime.
      
      == Setting object names
      
      This new setup requires that for a few more types we get the prototype
      and set the object name manually. To make this easier, the compiler now
      supports the virtual instruction `set_object_name`. This allows modules
      to set the correct object name, without having to use the
      `@_object_name` instance attribute directly. This in turn means this
      attribute is now only used in two places:
      
      1. In the compiler, where it belongs.
      2. In `std::mirror`, in order to obtain the name of the object.
      
      == DefaultHasher is now in a separate module
      
      The DefaultHasher type used to reside in `std::hash_map`, but this
      didn't make much sense since it's not tied into the `HashMap` type. With
      the various prototype changes being made I decided that now was a good
      time to move `DefaultHasher` to its own module: `std::hasher`. In the
      future this module might provide hashers using other algorithms, but for
      now it only defines the `DefaultHasher` type.
      f39ce802
  17. 08 Oct, 2018 2 commits
    • Yorick Peterse's avatar
      The Platform instruction now returns a string · c1c8f5db
      Yorick Peterse authored
      The "Platform" VM instruction now returns a string, instead of an
      integer. This string contains the name of the underlying platform, such
      as "windows", "linux", or "unix" for a generic Unix (like) system. This
      simplifies the runtime, while being able to detect more platforms. This
      in turn can be useful when binding to C using the upcoming FFI API.
      c1c8f5db
    • Yorick Peterse's avatar
      Supported nested calls to process.blocking · 3d15c20e
      Yorick Peterse authored
      This fixes `std::process.blocking` so it supports nested calls, similar
      to the recently introduced method `std::process.pinned`.
      
      Fixes #128
      3d15c20e
  18. 07 Oct, 2018 2 commits
    • Yorick Peterse's avatar
      Expose thread pinning to the runtime · b79e9e8d
      Yorick Peterse authored
      The method `process.pinned` can be used to run a block while pinning the
      process to the current OS thread, unpinning automatically once the block
      returns.
      b79e9e8d
    • Yorick Peterse's avatar
      Pinning of processes to threads · faf02ea7
      Yorick Peterse authored
      This commits adds the ability to pin a process to a particular OS thread
      in a thread pool. This is useful for the FFI, as certain C functions or
      data structures require to be run on a specific thread. For example,
      libc's "errno" variable uses thread-local storage. This means that if we
      want to run a function that uses it and read "errno", we _have_ to
      ensure both operations are performed on the same thread.
      
      To support this, each Pool structure now has a Worker structure, which
      stores the thread ID of that worker. This ID can then be stored in a
      process, allowing the scheduler to determine which worker should run the
      process.
      
      == Process pinning
      
      Pinned processes can not be moved across pools, so the MoveToPool
      instruction becomes a noop for a pinned process. We can not panic in
      this case, as this would prevent pinned processes from using methods
      that try to move a process to the secondary pool.
      faf02ea7
  19. 20 Sep, 2018 1 commit
    • Yorick Peterse's avatar
      Basic support for late binding of Self · 72e16d4f
      Yorick Peterse authored
      This commit adds basic support for late binding of Self types. When a
      method returns _just_ Self, we now evaluate it into an actual type upon
      use, instead of upon definiing the return type. The use of Self inside a
      generic type, such as Array!(Self), still uses early binding.
      
      Supporting simple cases of late binding allows us to clean up
      std::range. For example, previously sending `successor` to a `Successor
      + Compare` would produce a `Successor` as the return type. This meant
      having to cast this back to `Successor + Compare` just to use methods
      such as `<=`. With this commit, the compiler will produce the proper
      `Successor + Compare` type, removing the need for manually casting
      anything.
      
      With this commit also come some changes to the return types of bodies of
      object, trait, and impl blocks. Previously, the type of these blocks was
      set to the object (e.g. `Person` in `impl Person { ... }`), but this was
      not correct. Instead, the block return type should be inferred based on
      the last expression returned. These changes don't affect any existing
      code, as object/impl/trait themselves return the type of object they
      operate on, not what their _bodies_ return.
      
      See #107 for some more
      information on binding of Self types.
      72e16d4f
  20. 19 Sep, 2018 1 commit
    • Yorick Peterse's avatar
      Fixed arity of implements_trait? · c9835682
      Yorick Peterse authored
      The implements_trait? method in the compiler required different
      arguments in a few different places. This change ensures they all accept
      the same number of arguments.
      c9835682
  21. 17 Sep, 2018 1 commit
    • Yorick Peterse's avatar
      Support for type checking nested required traits · b1cb8a32
      Yorick Peterse authored
      This adds compiler support for handling nested required traits. For
      example:
      
          trait A {}
          trait B: A {}
          trait C: B {}
      
      Previously, when passing a C to an A, the compiler would error as it
      would only look for directly required traits in C. With this change this
      is no longer the case, as we recursively walk through all required
      traits to see if A (in the above example) is eventually required.
      b1cb8a32
  22. 10 Sep, 2018 3 commits
    • Yorick Peterse's avatar
      Release v0.2.5 · 4567d808
      Yorick Peterse authored
      4567d808
    • Yorick Peterse's avatar
      Fixed type checking of unknown_message · 38379fc1
      Yorick Peterse authored
      The compiler would never error if unknown_message was not implemented,
      as the return value would never evaluate to something falsy.
      38379fc1
    • Yorick Peterse's avatar
      Added std::test::assert.true and false · b29fdc09
      Yorick Peterse authored
      These methods can be used to assert that something is truthy or falsy.
      This means that instead of writing this:
      
          assert.equal(10 == 10, True)
      
      You can now write:
      
          assert.true(10 == 10)
      
      This commit also makes a change to the compiler to solve a type
      parameter error that could occur in certain cases.
      
      Fixes #144
      b29fdc09
  23. 09 Sep, 2018 1 commit
  24. 08 Sep, 2018 1 commit
  25. 07 Sep, 2018 2 commits
    • Yorick Peterse's avatar
      Parse trailing blocks as arguments · f827c138
      Yorick Peterse authored
      When sending a message using parentheses, if the expression is followed
      by a block (on the same line as the closing parenthesis), this block is
      treated as the final argument. This means that this:
      
          [10, 20, 30].each() do (number) {
            ...
          }
      
      Is parsed as this:
      
          [10, 20, 30].each(do (number) {
            ...
          })
      
      This is especially useful when passing regular arguments first. For
      example, instead of this:
      
          test.group 'This is the description of the group', {
            ...
          }
      
      You can now write this instead:
      
          test.group('This is the description of the group') {
            ...
          }
      
      The benefit of this is that it is more consistent when it comes to the
      use of parentheses. Prior to this commit, the recommended approach was
      to leave out parentheses if the last argument was a block. This however
      is a bit inconsistent. For example:
      
          foo(10, 20, 30)
      
          foo 10, 20, 30, do (thing) {
            ...
          }
      
      Supporting the trailing block syntax means there is now one case less
      where you leave out the parentheses.
      f827c138
    • Yorick Peterse's avatar
      Fix sending to types that implement UnknownMessage · 3f038c6e
      Yorick Peterse authored
      In a previous refactoring of the compiler's type system, support for
      sending messages to objects that implement
      std::unknown_message::UnknownMessage was broken. This would prevent
      compiling code such as `Nil.foo`. This commit restores support for such
      expressions.
      3f038c6e
  26. 06 Sep, 2018 2 commits
    • Yorick Peterse's avatar
      Fix parsing of arguments without parenthesis · 9c4be379
      Yorick Peterse authored
      This would break whenever passing an Array. For example, this would
      fail:
      
          foo ['bar'], 10
      
      Because "bracket_open" ("[") was not marked as a valid start token for a
      value, the parser would throw an error. Adding this token to the list of
      allowed tokens wasn't trivial though, as this would then confuse the
      parser when parsing code such as this:
      
          foo[10]
      
      This would be parsed as:
      
          foo([10])
      
      Instead of:
      
          foo.[](10)
      
      To solve this, when parenthesis are not provided, the first argument
      _must_ be preceded by a space. This ensures we can distinguish between
      `foo[10]` and `foo [10]`.
      9c4be379
    • Yorick Peterse's avatar
      Add support for deferred execution of blocks · 703ff73c
      Yorick Peterse authored
      The method `std::process.defer` can be used to schedule a block for
      execution when the calling scope returns, even when a panic is
      triggered. Such blocks are useful when cleaning up resources, such as
      files and sockets.
      
      Most languages use finalisers for this, but finalisers are difficult to
      implement in Inko. The garbage collector has to be able to deal with
      objects that are resurrected, and somehow be able to schedule finalisers
      for execution. Usage wise, finalisers are also more limited as they are
      defined when defining an object, not when using it. This means you
      (typically) can't create ad-hoc finalisers.
      
      The use of deferred blocks allows us to work around these restrictions
      and implementation difficulties, and the idea is inspired by Go. One
      downside is that directly using `std::process.defer` can lead to rather
      verbose code, but more high-level abstractions can be added on top
      easily.
      
      The order in which blocks are executed is currently not officially
      specified, and thus should not be relied upon. Currently the order will
      be First In Last Out (FILO), but this may change at any given point in
      time.
      
      Using `std::process.defer` is very straightforward. For example, to
      close a file once we are done with it, you'd write something along the
      lines of the following:
      
          import std::process
          import std::fs::file
      
          let readme = try! file.read_only('README.md')
      
          process.defer {
            readme.close
          }
      
          try! readme.read_string
      703ff73c
  27. 29 Aug, 2018 1 commit
    • Yorick Peterse's avatar
      Fix setting the receiver type of Send nodes · 6fffc53a
      Yorick Peterse authored
      Not doing this could result in messages being sent to the wrong objects.
      For example, sending `foo` in an object would result in the receiver
      being `self`, even if `foo` was defined as a module method.
      6fffc53a
  28. 28 Aug, 2018 1 commit
  29. 26 Aug, 2018 2 commits
    • Yorick Peterse's avatar
      Add support for registering panic handlers · 5e6920e3
      Yorick Peterse authored
      A panic handler is a block to execute when a process panics. Once the
      block finishes running, the process terminates. A panic handler can be
      registered using `std::process.panicking`:
      
          import std::process
          import std::stdio::stderr
          import std::vm
      
          process.panicking do (error) {
            stderr.print(error)
          }
      
          vm.panic('oops!')
      
      Each process can only register a single panic handler, and newly
      registered handlers will overwrite any previous ones.
      
      The block passed to `std::process.panicking` is given the panic message.
      A stacktrace needs to be obtained manually using
      `std::debug.stacktrace`.
      5e6920e3
    • Yorick Peterse's avatar
      Explicitly bind receivers to blocks and bindings · 79103c8e
      Yorick Peterse authored
      Prior to this commit, "self" was just syntax sugar for obtaining local
      variable 0. This variable in turn was populated by the (implicit)
      argument 0. In other words, this:
      
          def foo(bar) {
            self
          }
      
          foo
      
      Was more or less translated into the following:
      
          def foo(self_arg, bar) {
            self_arg
          }
      
          foo(self, bar)
      
      While fairly simple to implement, this poses two problems:
      
      1. "self" is an implicit argument, which can be confusing for users. For
         example, when using mirrors to obtain the list of method arguments,
         "self" would be included in the list.
      
      2. The VM could not schedule a Block for execution on its own, because
         it doesn't know what object to pass as the first argument (= self).
      
      Problem 2 made it impossible to implement panic hooks in a nice way, and
      any future features that require the VM to schedule blocks (e.g. when
      trapping signals).
      
      To work around this, "self" is now explicitly bound to blocks, when they
      are defined. To execute methods, we use a new instruction:
      RunBlockWithReceiver. This instruction takes a receiver (= the object to
      use for "self") to use when executing the method. The receiver in this
      case will be the object the method was invoked on.
      
      When a block is created, we no longer create a new binding for it.
      Instead, we store the binding that the block captures, which we later
      set as the parent for the new binding when executing the block. This
      removes the need for allocating a Binding for every block that is
      defined, even when never executed. The block also stores the receiver to
      use when executing said block.
      
      This setup also means we can remove quite a bit of nasty bits from the
      compiler, as we no longer need to generate implicit arguments and local
      variables. It also allows us to (in the future) obtain the receiver of a
      block (for meta programming), without having to rely on the exact local
      variable index used to store this object.
      79103c8e
  30. 21 Aug, 2018 1 commit
    • Yorick Peterse's avatar
      Added std::env for managing environment data · 69a7592e
      Yorick Peterse authored
      The module std::env can be used for obtaining and setting environment
      variables, the home directory, the temporary directory, and more. For
      example, one can obtain environment variable values as follows:
      
          import std::env
      
          env['HOME'] # => '/home/yorickpeterse'
      
      You can also set the value of a variable:
      
          import std::env
      
          env['HOME'] = '/home/foo'
      
      Removing variables is also possible:
      
          import std::env
      
          env.remove('HOME') # => Nil
      
      Or obtain the home directory:
      
          import std::env
      
          env.home_directory # => '/home/yorickpeterse'
      
      You can also obtain and set the working directory:
      
          import std::env
      
          try! env.working_directory          # => '/home/yorickpeterse'
          try! env.working_directory = '/tmp' # => '/tmp'
      
      Arguments can be retrieved using `std::env.arguments`:
      
          import std::env
      
          env.arguments # => ['foo', 'bar']
      
      == Executable changes
      
      The "inko" executable has been modified to pass additional commandline
      arguments to IVM. IVM in turn has been modified to expose these to the
      VM instructions. This requires us to explicitly store the passed
      arguments in a vm::state::State, as Rust's std::env::args() is immutable
      _and_ includes _all_ arguments (the bytecode file to execute, IVM
      options, etc).
      
      The arguments passed via the CLI are all interned, removing the need for
      allocating (potentially many) strings every time `std::env.arguments` is
      executed.
      
      Fixes #136
      69a7592e
  31. 18 Aug, 2018 1 commit