1. 11 May, 2019 4 commits
    • Yorick Peterse's avatar
      Release v0.4.0 · 0631b684
      Yorick Peterse authored
      0631b684
    • Yorick Peterse's avatar
      Fix non-blocking socket reads · 2fb8e93f
      Yorick Peterse authored
      Prior to this commit, if a read size was given we would use Rust's Take
      trait to read up to N bytes from a socket. Unfortunately, sockets don't
      produce an EOF until they are closed, which meant the read would
      continue indefinitely. To work around this we use the same buffering
      logic as is used by our recv_from implementation.
      2fb8e93f
    • Yorick Peterse's avatar
      Turn accepted sockets into non-blocking sockets · 0393df5b
      Yorick Peterse authored
      When a socket is produced using accept(), it does not inherit the
      non-blocking status from the listener. This would cause socket
      operations using such a socket to block.
      0393df5b
    • Yorick Peterse's avatar
      Added support for shutting down sockets · f9131174
      Yorick Peterse authored
      This is necessary to gracefully terminate STREAM sockets, as otherwise
      clients may produce "Connection reset by peer" errors.
      f9131174
  2. 10 May, 2019 2 commits
  3. 09 May, 2019 1 commit
    • Yorick Peterse's avatar
      Added support for non-blocking network IO · 3974f47c
      Yorick Peterse authored
      This adds support for performing non-blocking network operations, such
      as reading and writing to/from a socket. The runtime API exposed is
      similar to Erlang, allowing one to write code that uses non-blocking
      APIs without having to resort to using callbacks. For example, in a
      typicall callback based language you may write the following to read
      from a socket:
      
          socket.create do (socket) {
            socket.read do (data) {
      
            }
          }
      
      In Inko, you instead would (more or less) write the following:
      
          import std::net::socket::TcpStream
      
          let socket = try! TcpStream.new(ip: '192.0.2.0', port: 80)
          let message = try! socket.read_string(size: 4)
      
      The VM then takes care of using the appropriate non-blocking operations,
      and will reschedule processes whenever necessary.
      
      This functionality is exposed through the following runtime modules:
      
      * std::net::ip: used for parsing IPv4 and IPv6 addresses.
      * std::net::socket: used for TCP and UDP sockets.
      * std::net::unix: used for Unix domain sockets.
      
      The VM uses the system's native polling mechanism to determine when a
      file descriptor is available for a read or write. On Linux we use epoll,
      while using kqueue for the various BSDs and Mac OS. For Windows we use
      wepoll (https://github.com/piscisaureus/wepoll). Wepoll exposes an API
      that is compatible with the epoll API, but uses Windows IO completion
      ports under the hoods.
      
      When a process attempts to perform a non-blocking operation, the process
      is registered (combined with the file descriptor to poll) in a global
      poller and suspended. When the file descriptor becomes available for a
      read or write, the corresponding process is rescheduled. The polling
      mechanism is set up in such a way that a process can not be rescheduled
      multiple times at once.
      
      We do not use MIO (https://github.com/tokio-rs/mio), instead we use
      epoll, kqueue, and wepoll (using
      https://crates.io/crates/wepoll-binding) directly. At the time of
      writing, while MIO offers some form of support for Windows it comes with
      various issues:
      
      1. https://github.com/tokio-rs/mio/issues/921
      2. https://github.com/tokio-rs/mio/issues/919
      3. https://github.com/tokio-rs/mio/issues/776
      4. https://github.com/tokio-rs/mio/issues/913
      
      It's not clear when these issues would be addressed, as the maintainers
      of MIO appear to not have the experience and resources to resolve them
      themselves. MIO is part of the Google Summer of Code 2019, with the goal
      of improving Windows support. Unfortunately, this likely won't be done
      before the end of 2019, and we don't want to wait that long.
      
      Another issue with MIO is its implementation. Internally, MIO uses
      various forms of synchronisation which can make it expensive to use a
      single poller across multiple threads; it certainly is not a zero-cost
      library. It also offers more than we need, such as being able to poll
      arbitrary objects.
      
      We are not the first to run into these issues. For example, the Amethyst
      video game engine also ran into issues with MIO as detailed in
      https://community.amethyst.rs/t/sorting-through-the-mio-mess/561.
      
      With all of this in mind, I decided it was not worth the time to wait
      for MIO to get fixed, and to instead spend time directly using epoll,
      kqueue, and wepoll. This gives us total control over the code, and
      allows us to implement what we need in the way we need it. Most
      important of all: it works on Linux, BSD, Mac, and Windows.
      3974f47c
  4. 28 Mar, 2019 5 commits
    • Yorick Peterse's avatar
      Removed use of "extern crate" · 99c9f044
      Yorick Peterse authored
      This is no longer required in Rust 2018.
      99c9f044
    • Yorick Peterse's avatar
      Use Rust 2018 for the VM · 5ca5a292
      Yorick Peterse authored
      5ca5a292
    • Yorick Peterse's avatar
      Removed unused code from the compiler · e9331a77
      Yorick Peterse authored
      e9331a77
    • Yorick Peterse's avatar
      Reduce Immix block size to 8 KB · ff6297fd
      Yorick Peterse authored
      A block size of 8 KB reduces memory usage when running many processes,
      at the cost of a tiny increase in garbage collection timings. This
      increase is likely something we can optimise in the future. Reducing the
      block size also appears to speed up the test suite by 5-10 milliseconds.
      
      Fixes #159
      ff6297fd
    • Yorick Peterse's avatar
      Remove PIDs from processes · 11ee161e
      Yorick Peterse authored
      This removes the use of PIDs for processes in favour of using proper
      process objects. This means we no longer need to allocate PIDs. This
      allows us to spawn processes without having to obtain a lock on a global
      process table.
      
      The removal of PIDs means that sending messages feels a bit nicer now.
      Instead of writing this:
      
          import std::process
      
          let pid = process.spawn {}
      
          process.send(pid: pid, message: 'hello')
      
      You now write this instead:
      
          import std::process
      
          let proc = process.spawn {}
      
          proc.send('hello')
      
      We also remove the `std::process.channel` method and associated types.
      This API was not very pleasant to use, and was not very efficient
      either. We hope to (re-)introduce a better type-safe API for message
      passing in the future, if possible.
      
      Fixes #161
      11ee161e
  5. 26 Mar, 2019 1 commit
    • Yorick Peterse's avatar
      Refactor the global allocator and block sizes · c7c0befb
      Yorick Peterse authored
      This refactors the global allocator so that it doesn't require on
      explicit locking as much, and changes various parts of Immix blocks (and
      tests) to no longer rely directly on the block size. This makes it a bit
      easier to adjust the block size in the future, without having to fix
      various tests.
      c7c0befb
  6. 20 Mar, 2019 1 commit
    • Yorick Peterse's avatar
      Fix storing module methods in globals · bd3465a0
      Yorick Peterse authored
      Due to the use of an incorrect scope object, the compiler would use -1
      as the index for methods stored in module globals. Due to overflowing
      this would result in the VM allocating memory for 65 535 globals,
      resulting in at least 512 KB of memory being used for every module that
      defined a module method. Worse, if a module were to use these methods
      the VM might end up executing the wrong method due to all of them using
      the same slot.
      bd3465a0
  7. 19 Mar, 2019 5 commits
  8. 18 Mar, 2019 2 commits
    • Yorick Peterse's avatar
      Rewrite the process scheduler from the ground up · 3e5882be
      Yorick Peterse authored
      The old scheduler was a little over two years old, and due for a
      rewrite. While it worked, it was not very efficient and many features
      were bolted on top; process pinning being an example.
      
      The new scheduler relies less heavy on locking, only using mutexes
      paired with condition variables to wake up sleeping threads. This will
      allow it to scale much better as the number of threads goes up.
      
      Another big benefit is clearer code. The old scheduler's code was a
      mess, largely because we focused more on getting a proof of concept out
      instead of building a scheduler for the next few years.
      
      == Suspending and rescheduling processes
      
      As part of this rewrite, the way timeouts and rescheduling of processes
      is handled is also rewritten. When a process is suspended and receives a
      message, the sender will try to reschedule it immediately. This makes
      sending messages a little bit more expensive, but allows for much faster
      rescheduling of processes. This also removes the need for a separate
      thread to perform a linear scan over a list of processes to determine
      which ones need to be rescheduled.
      
      Processes that suspend themselves with a timeout are stored in a binary
      heap, managed by a separate thread. Communication with this thread is
      done using a channel, offloading most of the work to the separate
      timeout thread. When a process with a timeout is rescheduled, its entry
      in the heap is marked as invalid instead of being removed. This makes
      the operation a constant time operation, at the cost of the binary heap
      getting fragmented. To combat fragmentation, the timeout thread will
      periodically remove invalid entries from the heap.
      
      Rescheduling processes is done entirely using atomic operations, instead
      of using mutexes. This requires some careful coding to take into account
      multiple threads trying to reschedule the same process, but should allow
      all of this to scale much better.
      
      The new approach of suspending and rescheduling processes requires one
      additional word of memory per process. This memory is used to mark the
      process as suspended, and to optionally store a pointer to its timeout
      (if one was used).
      
      == Message counts
      
      The number of messages in a mailbox is now stored explicitly using an
      atomic integer, instead of obtaining this from the synchronised
      data structures internal to a mailbox. This requires one word of extra
      memory per process, but makes it much cheaper to check if a process has
      messages. This is important, because when rescheduling a process such
      checks are performed several times.
      
      == Asynchronous IO and further improvements
      
      While this commit does not add support for asynchronous IO operations,
      the rewrite will make it easier to do so in future commits. The process
      lookup table also remains unchanged, but we're currently investigating
      if we can get rid of PIDs and the lookup table entirely; potentially
      speeding up process spawning by quite a bit.
      3e5882be
    • Yorick Peterse's avatar
      Use Rust stable for rustfmt · cdaaa5f7
      Yorick Peterse authored
      cdaaa5f7
  9. 19 Feb, 2019 1 commit
  10. 28 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Use BufReader when reading bytecode from files · d80c4c30
      Yorick Peterse authored
      Using a 32KB buffer this allows us to drastically reduce the time spent
      parsing bytecode files. Consider the following program:
      
          import std::ansi
          import std::ffi
          import std::fs
          import std::mirror
          import std::stdio::stderr
          import std::stdio::stdin
          import std::stdio::stdout
          import std::time
      
          stdout.print('hello world')
      
      Prior to these changes, running this program (ignoring the compilation
      itme) takes around 240 milliseconds. With these changes, this is reduced
      to 20 milliseconds: a 12x improvement.
      d80c4c30
  11. 26 Dec, 2018 8 commits
  12. 25 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Reorganise and test std::time · 7f0cb46b
      Yorick Peterse authored
      std::time is now broken up into two modules:
      
      1. std::time, which provides time related types.
      2. std::time::duration, which provides the Duration type.
      
      This separation allows for the use of module methods to construct a
      Duration using different time units, instead of using methods defined on
      Duration itself. This means that instead of this:
      
          import std::time::Duration
      
          Duration.from_seconds(5)
      
      You now write this:
      
          import std::time::duration
      
          duration.from_seconds(5)
      
      == Renaming MonotonicTime to Instant
      
      MonotonicTime has been renamed to "Instant", which is a bit shorter and
      much easier to type. A new method has been added as well: "elapsed".
      This method can be used to measure the time that has elapsed since the
      Instant was created.
      
      == Adding and subtracting SystemTime and Instant
      
      The methods + and - for SystemTime and Instant no longer accept a
      ToFloat, instead they require a Duration. Accepting a ToFloat would
      allow you to add a SystemTime to another SystemTime, or add an Instant
      to an Instant. Neither make sense, so instead we now require the use of
      a Duration. This means that instead of this:
      
          import std::time
      
          time.now + 5
      
      You now have to write this:
      
          import std::time
          import std::time::duration
      
          time.now + duration.from_seconds(5)
      
      == Tests
      
      Finally, both std::time and std::time::duration have tests. These tests
      uncovered a few bugs in the implementation of SystemTime, which have
      been resolved.
      7f0cb46b
  13. 23 Dec, 2018 1 commit
  14. 22 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Added method Float.to_bits · a4c2d681
      Yorick Peterse authored
      This method can be used to obtain the bitwise representation of a Float.
      This in turn can be used to perform an approximate equality comparison
      by checking bits of a Float.
      a4c2d681
  15. 19 Dec, 2018 4 commits
    • Yorick Peterse's avatar
      Rename Immix bitmap structures to bytemaps · 57247e71
      Yorick Peterse authored
      These structures have been bytemaps for quite a while now, so let's
      start calling them that.
      57247e71
    • Yorick Peterse's avatar
      Fix lookups in ObjectMirror.implements_trait? · 9af64983
      Yorick Peterse authored
      This method would mistakingly keep looking up traits in the mirror
      subject, instead of using the objects in the prototype chain.
      9af64983
    • Yorick Peterse's avatar
      Added tests for std::trait · e33ed34f
      Yorick Peterse authored
      e33ed34f
    • Yorick Peterse's avatar
      Move all "class" methods to module methods · 1a577430
      Yorick Peterse authored
      In the past I have gone back and forth a bit on the idea of using
      class/static methods or not. With this commit I'm making a final
      decision on this topic: Inko will not have class/static methods, at
      least not until 1.0. Instead, Inko will use module methods. This means
      that the following methods have been changed:
      
      * HashMap.from_array  -> std::hash_map.from_array
      * Trait.implement_for -> std::trait.implement
      * Integer.from_string -> std::integer.parse
      * Float.from_string   -> std::float.parse
      
      To make this happen some compiler changes had to be made to optimise
      `hash_map.from_array` and to use `trait.implement` instead of
      `Trait.implement_for`.
      1a577430
  16. 17 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Enforce a limit on bytecode input sizes · 26f0023c
      Yorick Peterse authored
      This adds limits to various parts of the bytecode parser that are
      enforced when reading various literals, such as strings and Compiled
      Code objects. These limits prevent the VM from crashing with an OOM
      error when reading malformed (or _very_ large) bytecode files.
      26f0023c
  17. 16 Dec, 2018 1 commit
    • Yorick Peterse's avatar
      Parsing of Strings into Floats and Integers · 6558d6b5
      Yorick Peterse authored
      This adds support for parsing a String into a Float and an Integer.
      There are two ways of doing so:
      
      1. By sending `to_integer` or `to_float` to a `String`.
      2. Using `Integer.from_string` or `Float.from_string`.
      
      Using `to_integer` and `to_float` will perform a lossy conversion:
      returning 0 or 0.0 for invalid input. Using the `from_string` methods
      will result in a strict conversion, with an error being thrown for
      invalid input.
      
      Fixes #134
      Fixes #156
      6558d6b5