Commit bee67ef6 authored by bzt's avatar bzt

Refactoring in progress

parent bacd70eb
OS/Z Documentation
==================
Developer's Corner
------------------
Materials interesting for developers.
* [Project Structure](https://gitlab.com/bztsrc/osz/blob/master/docs/project.en.md) of OS/Z repository
* [Memory Management](https://gitlab.com/bztsrc/osz/blob/master/docs/memory.en.md) in OS/Z
* The [FS/Z](https://gitlab.com/bztsrc/osz/blob/master/docs/fs.en.md) file system, and it's [on disk format](https://gitlab.com/bztsrc/osz/blob/master/include/sys/fsZ.h)
* About [processes](https://gitlab.com/bztsrc/osz/blob/master/docs/process.en.md)
* [Scheduler](https://gitlab.com/bztsrc/osz/blob/master/docs/scheduler.en.md)
* Using [messages](https://gitlab.com/bztsrc/osz/blob/master/docs/messages.en.md) at different levels (aka. syscalls in OS/Z)
* How a [keypress](https://gitlab.com/bztsrc/osz/blob/master/docs/keypress.en.md) is processed
* [Security](https://gitlab.com/bztsrc/osz/blob/master/docs/security.en.md) checks
* Differences to [POSIX](https://gitlab.com/bztsrc/osz/blob/master/docs/posix.en.md)
* [Porting](https://gitlab.com/bztsrc/osz/blob/master/docs/porting.en.md) to new platforms
* [Compilation](https://gitlab.com/bztsrc/osz/blob/master/docs/compile.en.md)
* [Boot procedure](https://gitlab.com/bztsrc/osz/blob/master/docs/boot.en.md)
* [Testing](https://gitlab.com/bztsrc/osz/blob/master/docs/howto1-testing.en.md)
* [Debugging](https://gitlab.com/bztsrc/osz/blob/master/docs/howto2-debug.en.md)
* [Developing](https://gitlab.com/bztsrc/osz/blob/master/docs/howto3-develop.en.md)
SysAdmin's Playground
---------------------
Documentation on how to operate OS/Z.
* Available [device drivers](https://gitlab.com/bztsrc/osz/blob/master/docs/drivers.en.md)
* [Services](https://gitlab.com/bztsrc/osz/blob/master/docs/services.en.md) hierarchy and details
* [Virtual File System](https://gitlab.com/bztsrc/osz/blob/master/docs/vfs.en.md) and paths
* [Boot options](https://gitlab.com/bztsrc/osz/blob/master/docs/bootopts.en.md)
* [Multi language support](https://gitlab.com/bztsrc/osz/blob/master/docs/translate.en.md)
* Invoking [Rescue Shell](https://gitlab.com/bztsrc/osz/blob/master/docs/howto4-rescueshell.en.md)
* [Install](https://gitlab.com/bztsrc/osz/blob/master/docs/howto5-install.en.md) OS/Z on a USB stick and package management
* Manage and initialize [services](https://gitlab.com/bztsrc/osz/blob/master/docs/howto6-services.en.md).
User's Guide
------------
Information for end users.
* [User Interface](https://gitlab.com/bztsrc/osz/blob/master/docs/howto7-interface.en.md)
OS/Z Documentation
==================
Developer's Corner
------------------
Materials interesting for developers.
* [Project Structure](https://gitlab.com/bztsrc/osz/blob/master/docs/project.md) of OS/Z repository
* [Memory Management](https://gitlab.com/bztsrc/osz/blob/master/docs/memory.md) in OS/Z
* The [FS/Z](https://gitlab.com/bztsrc/osz/blob/master/docs/fs.md) file system, and it's [on disk format](https://gitlab.com/bztsrc/osz/blob/master/etc/include/sys/fsZ.h)
* About [processes](https://gitlab.com/bztsrc/osz/blob/master/docs/process.md)
* [Scheduler](https://gitlab.com/bztsrc/osz/blob/master/docs/scheduler.md)
* Using [messages](https://gitlab.com/bztsrc/osz/blob/master/docs/messages.md) at different levels (aka. syscalls in OS/Z)
* How a [keypress](https://gitlab.com/bztsrc/osz/blob/master/docs/keypress.md) is processed
* [Security](https://gitlab.com/bztsrc/osz/blob/master/docs/security.md) checks
* Differences to [POSIX](https://gitlab.com/bztsrc/osz/blob/master/docs/posix.md)
* [Porting](https://gitlab.com/bztsrc/osz/blob/master/docs/porting.md) to new platforms
* [Compilation](https://gitlab.com/bztsrc/osz/blob/master/docs/compile.md)
* [Boot procedure](https://gitlab.com/bztsrc/osz/blob/master/docs/boot.md)
* [Testing](https://gitlab.com/bztsrc/osz/blob/master/docs/howto1-testing.md)
* [Debugging](https://gitlab.com/bztsrc/osz/blob/master/docs/howto2-debug.md)
* [Developing](https://gitlab.com/bztsrc/osz/blob/master/docs/howto3-develop.md)
SysAdmin's Playground
---------------------
Documentation on how to operate OS/Z.
* Available [device drivers](https://gitlab.com/bztsrc/osz/blob/master/docs/drivers.md)
* [Services](https://gitlab.com/bztsrc/osz/blob/master/docs/services.md) hierarchy and details
* [Virtual File System](https://gitlab.com/bztsrc/osz/blob/master/docs/vfs.md) and paths
* [Boot options](https://gitlab.com/bztsrc/osz/blob/master/docs/bootopts.md)
* [Multi language support](https://gitlab.com/bztsrc/osz/blob/master/docs/translate.md)
* Invoking [Rescue Shell](https://gitlab.com/bztsrc/osz/blob/master/docs/howto4-rescueshell.md)
* [Install](https://gitlab.com/bztsrc/osz/blob/master/docs/howto5-install.md) OS/Z on a USB stick and package management
* Manage and initialize [services](https://gitlab.com/bztsrc/osz/blob/master/docs/howto6-services.md).
User's Guide
------------
Information for end users.
* [User Interface](https://gitlab.com/bztsrc/osz/blob/master/docs/howto7-interface.md)
OS/Z Dokumentáció
=================
Fejlesztői sarok
----------------
Fejlesztők számára érdekes anyagok
* Az OS/Z [projekt struktúrája](https://gitlab.com/bztsrc/osz/blob/master/docs/project.md)
* [Memória kezelés](https://gitlab.com/bztsrc/osz/blob/master/docs/memory.md) az OS/Z-ben
* Az [FS/Z](https://gitlab.com/bztsrc/osz/blob/master/docs/fs.md) fájl rendszer, és a [lemezformátuma](https://gitlab.com/bztsrc/osz/blob/master/include/sys/fsZ.h)
* [Processzekről](https://gitlab.com/bztsrc/osz/blob/master/docs/process.md)
* [Ütemezés](https://gitlab.com/bztsrc/osz/blob/master/docs/scheduler.md)
* [Üzenetküldés](https://gitlab.com/bztsrc/osz/blob/master/docs/messages.md) különböző szinteken (OS/Z rendszerhívások)
* Hogyan kezelődik le egy [billentyűleütés](https://gitlab.com/bztsrc/osz/blob/master/docs/keypress.md)
* [Biztonság](https://gitlab.com/bztsrc/osz/blob/master/docs/security.md)i ellenőrzések
* Eltérések a [POSIX](https://gitlab.com/bztsrc/osz/blob/master/docs/posix.md) szabványtól
* [Portolás](https://gitlab.com/bztsrc/osz/blob/master/docs/porting.md) új platformokra
* [Fordítás](https://gitlab.com/bztsrc/osz/blob/master/docs/compile.md)
* [Rendszerindulás](https://gitlab.com/bztsrc/osz/blob/master/docs/boot.md)
* [Tesztelés](https://gitlab.com/bztsrc/osz/blob/master/docs/howto1-testing.md)
* [Debuggolás](https://gitlab.com/bztsrc/osz/blob/master/docs/howto2-debug.md)
* [Fejlesztés](https://gitlab.com/bztsrc/osz/blob/master/docs/howto3-develop.md)
Rendszergazdák játszótere
-------------------------
Leírás arról, hogy hogyan üzemeltessük az OS/Z-t
* Rendelkezésre álló [eszköz meghajtók](https://gitlab.com/bztsrc/osz/blob/master/docs/drivers.md)
* [Szolgáltatások](https://gitlab.com/bztsrc/osz/blob/master/docs/services.md) hierarchy and details
* [Virtuális Fájl Rendszer](https://gitlab.com/bztsrc/osz/blob/master/docs/vfs.md) és elérési utak
* [Indítási opciók](https://gitlab.com/bztsrc/osz/blob/master/docs/bootopts.md)
* [Többnyelvűség](https://gitlab.com/bztsrc/osz/blob/master/docs/translate.md)
* A [vészhelyzeti parancsértelmező](https://gitlab.com/bztsrc/osz/blob/master/docs/howto4-rescueshell.md) hívája
* OS/Z [telepítés](https://gitlab.com/bztsrc/osz/blob/master/docs/howto5-install.md)e pendrájvra és a csomagkezelés
* [Szolgáltatások kezelése](https://gitlab.com/bztsrc/osz/blob/master/docs/howto6-services.md).
Felhasználói kézikönyv
----------------------
Végfelhasználóknak információk
* [Felhasználói felület](https://gitlab.com/bztsrc/osz/blob/master/docs/howto7-interface.md)
......@@ -6,7 +6,7 @@ Loader
OS/Z uses the [BOOTBOOT](https://gitlab.com/bztsrc/bootboot) protocol to get the system running.
The compatible [loader](https://gitlab.com/bztsrc/osz/tree/master/loader) is loaded by the firmware as the last step of POST.
On every platforms, the loader initializes the hardware (including the framebuffer), loads initrd and locates `sys/core` inside.
On every platform, the loader initializes the hardware (including the framebuffer), loads initrd and locates `sys/core` inside.
When found, it maps that at the top of the address space (-2M..0) and passes control to it.
Core
......@@ -16,16 +16,23 @@ First of all we have to clearify that `core` consist of two parts: one is platfo
architecture and platform specific (see [porting](https://gitlab.com/bztsrc/osz/blob/master/docs/porting.md)).
During boot, the execution is jumping forth and back from one part to the other.
The entry point that the loader calls is `_start`, which can be found in [src/core/(platform)/start.S](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/start.S).
It disables interrupts, sets up segments registers, stack frame and checks required CPU features. It also initializes [kprintf](https://gitlab.com/bztsrc/osz/blob/master/src/core/kprintf.c), so that it can call kpanic if needed.
Finally jumps to the C startup code `main()` in [src/core/main.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/main.c) which is a platform independent code.
The entry point that the loader calls is `_start`, which can be found in [src/core/(arch)/start.S](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/start.S).
It disables interrupts, sets up segments registers, distinguishes CPU cores, and puts APs in a spinloop.
It also initializes
Finally on the BSP jumps to the C startup code `main()` in [src/core/main.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/main.c) which is a platform independent code.
To boot up the computer, `core` does the following:
1. as OS/Z is polite, greets you with a "OS/Z starting..." message, then
2. `env_init` parses the [environment variables](https://gitlab.com/bztsrc/osz/blob/master/docs/bootopts.md) in [src/core/env.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/env.c) passed by the loader.
3. `pmm_init()` sets up Physical Memory Manager in [src/core/pmm.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/pmm.c).
4. `sys_init()` in [src/core/sys.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/sys.c), creates `IDLE` task (the CPU "driver") and enumerates system buses to locate and load [device drivers](https://gitlab.com/bztsrc/osz/blob/master/docs/drivers.md). It will also initialize the [IRQ Routing Table](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/isr.c#L117) (IRT).
1. `krpintf_init()` initializes the [boot console](https://gitlab.com/bztsrc/osz/blob/master/src/core/kprintf.c).
2. `platform_dbginit()` initializes the debug console on serial line in [src/core/(arch)/(platform)/platform.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/ibmpc/platform.c).
3. `platform_cpu()` initializes the CPU specific features in [src/core/(arch)/platform.S](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/platform.S).
4. `platform_srand()` initializes the random generator in [src/core/(arch)/platform.S](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/platform.S).
5. `env_init` parses the [environment variables](https://gitlab.com/bztsrc/osz/blob/master/docs/bootopts.md) in [src/core/env.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/env.c) passed by the loader.
6. `pmm_init()` sets up Physical Memory Manager in [src/core/pmm.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/pmm.c).
7. `vmm_init()` sets up Virtual Memory Manager and paging in [src/core/(arch)/vmm.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/vmm.c).
8. `drivers_init()` in [src/core/drivers.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/drivers.c), creates `IDLE` task (the CPU "driver") and enumerates system buses to locate and load [device drivers](https://gitlab.com/bztsrc/osz/blob/master/docs/drivers.md). It will also initialize the IRQ Routing Table (IRT).
7. `sys_init()` in [src/core/sys.c](https://gitlab.com/bztsrc/osz/blob/master/src/core/sys.c), creates `IDLE` task (the CPU "driver") and enumerates system buses to locate and load [device drivers](https://gitlab.com/bztsrc/osz/blob/master/docs/drivers.md). It will also initialize the [IRQ Routing Table](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/isr.c#L117) (IRT).
5. next it loads a [system service](https://gitlab.com/bztsrc/osz/blob/master/docs/services.md), `sys_service(SRV_FS)` which is a normal system service, except it has the initrd entirely mapped in in it's bss.
6. user interface is initialized with `sys_service(SRV_UI)`. These first three services (aka `IDLE`, `FS`, `UI`) are mandatory, unlike the rest (hence the upper-case).
7. loads additional, non-critical tasks by several `sys_service()` calls, like the `syslog`, `inet`, `sound` and `init` system services.
......
......@@ -26,39 +26,37 @@ Keys are ASCII names without spaces, values can be decimal and hexadecimal [numb
Boot Parameters
---------------
| Parameter | Default | Type | Subsystem | Description |
| --------- | :------: | ---- | --------- | ----------- |
| screen | 1024x768 | num<i>x</i>num | [loader](https://gitlab.com/bztsrc/osz/blob/master/loader) | required screen resolution, minimum 640x480 |
| kernel | sys/core | string | loader | the name of kernel executable on initrd |
| tz | default | string/num | core | time zone in minutes (-1440..1440) or "default" or "ask", see below |
| lang | en | string | core | language code, specifies the user interface's [language](https://gitlab.com/bztsrc/osz/blob/master/docs/translate.md) |
| debug | 0 | decimal | core | specifies which debug information to show (if [compiled with debugging](https://gitlab.com/bztsrc/osz/blob/master/Config), see below) |
| nrphymax | 2 | number | core | the number of pages to store physical RAM fragments (16 bytes each) |
| nrmqmax | 1 | number | core | the number of pages for Message Queue (64 bytes each) |
| nrlogmax | 2 | number | core | the number of pages for early syslog buffer |
| quantum | 100 | number | core | scheduler frequency, a task can allocate CPU continously for (quantum) timer interrupts. |
| display | mc | flag[,drv] | core | selects visual, and optionally the driver, see below |
| syslog | true | boolean | core | disable syslog [service](https://gitlab.com/bztsrc/osz/blob/master/docs/services.md) |
| networking | true | boolean | core | disable inet service |
| sound | true | boolean | core | disable sound service |
| rescueshell | false | boolean | core | if true, starts `/bin/sh` instead of user services |
| pathmax | 512 | number | fs | Maximum length of path, minimum 256 |
| cachelines | 16 | number | fs | Number of block cache lines, minimum 16 |
| cachelimit | 5 | percentage | fs | Flush and free up block cache if free RAM drops below this limit, 1%-50% |
| Parameter | Default | Type | Subsystem | Description |
| ---------- | :------: | ------- | --------- | ----------- |
| screen | 1024x768 | num<i>x</i>num | [loader](https://gitlab.com/bztsrc/osz/blob/master/loader) | required screen resolution, minimum 640x480 |
| kernel | sys/core | string | loader | the name of kernel executable on initrd |
| tz | default | string/num | core | time zone in minutes (-1440..1440) or "default" or "ask", see below |
| lang | en | string | core | language code, specifies the user interface's [language](https://gitlab.com/bztsrc/osz/blob/master/docs/translate.md) |
| debug | 0 | decimal | core | specifies which debug information to show (if [compiled with debugging](https://gitlab.com/bztsrc/osz/blob/master/Config), see below) |
| nrmqmax | 2 | number | core | the number of pages for Message Queue (64 bytes each) |
| dmabuf | 0 | number | core | DMA buffer size in pages (allocated in lowest 16M of RAM, defaults to 256 (1M) on ibmpc) |
| quantum | 1000 | number | core | scheduler frequency, a task can allocate CPU continously for (quantum) microseconds |
| display | mc | flag[,drv] | core | selects visual, and optionally the driver, see below |
| syslog | true | boolean | core | disable syslog [service](https://gitlab.com/bztsrc/osz/blob/master/docs/services.md) |
| networking | true | boolean | core | disable inet service |
| sound | true | boolean | core | disable sound service |
| rescueshell | false | boolean | core | if true, starts `/bin/sh` instead of user services |
| pathmax | 512 | number | fs | Maximum length of path, minimum 256 |
| cachelines | 16 | number | fs | Number of block cache lines, minimum 16 |
| cachelimit | 5 | percentage | fs | Flush and free up block cache if free RAM drops below this limit, 1%-50% |
| keyboard | pc105,en_us | str,str[,str] | ui | keyboard layout and mapping(s), see [etc/kbd](https://gitlab.com/bztsrc/osz/blob/master/etc/sys/etc/kbd) |
| focusnew | true | boolean | ui | automatically focus new windows |
| dblbuf | true | boolean | ui | use double buffering for window composition (memory consuming but fast) |
| lefthanded | false | boolean | ui | swap pointers |
| identity | false | boolean | init | run first time setup before servies to get machine's identity |
| clock | 0 | number | platform | override autodetected clock source |
| hpet | - | hexdec | platform | x86_64-acpi override autodetected HPET address |
| apic | - | hexdec | platform | x86_64-acpi override autodetected LAPIC address |
| ioapic | - | hexdec | platform | x86_64-acpi override autodetected IOAPIC address |
| focusnew | true | boolean | ui | automatically focus new windows |
| dblbuf | true | boolean | ui | use double buffering for window composition (memory consuming but fast) |
| lefthanded | false | boolean | ui | swap pointers |
| identity | false | boolean | init | run first time setup before servies to get machine's identity |
| hpet | - | hexdec | platform | x86_64-acpi override autodetected HPET address |
| apic | - | hexdec | platform | x86_64-acpi override autodetected LAPIC address |
| ioapic | - | hexdec | platform | x86_64-acpi override autodetected IOAPIC address |
Debugging
---------
This can be a numeric value, or a comma separated list of flags, see [debug.h](https://gitlab.com/bztsrc/osz/blob/master/etc/include/sys/debug.h).
This can be a numeric value, or a comma separated list of flags, see [debug.h](https://gitlab.com/bztsrc/osz/blob/master/include/sys/debug.h).
| Value | Flag | Define | Description |
| ----: | ---- | ------ | ----------- |
......@@ -68,13 +66,13 @@ This can be a numeric value, or a comma separated list of flags, see [debug.h](h
| 4 | el | DBG_ELF | debug [ELF loading](https://gitlab.com/bztsrc/osz/blob/master/src/core/elf.c#L66) |
| 8 | ri | DBG_RTIMPORT | debug [run-time linker](https://gitlab.com/bztsrc/osz/blob/master/src/core/elf.c#L486) imported values |
| 16 | re | DBG_RTEXPORT | debug run-time linker exported values |
| 32 | ir | DBG_IRQ | dump [IRQ Routing Table](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/isr.c#L121) |
| 32 | ir | DBG_IRQ | dump [IRQ Routing Table](https://gitlab.com/bztsrc/osz/blob/master/src/core/drivers.c#L121) |
| 64 | de | DBG_DEVICES | dump [System Tables](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/acpi/acpi.c) and [PCI devices](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/ibmpc/pci.c) |
| 128 | sc | DBG_SCHED | debug [scheduler](https://gitlab.com/bztsrc/osz/blob/master/src/core/sched.c) |
| 256 | ms | DBG_MSG | debug [message sending](https://gitlab.com/bztsrc/osz/blob/master/src/core/msg.c) |
| 512 | lo | DBG_LOG | dump [early syslog](https://gitlab.com/bztsrc/osz/blob/master/src/core/syslog.c) |
| 1024 | pm | DBG_PMM | debug [physical memory manager](https://gitlab.com/bztsrc/osz/blob/master/src/core/pmm.c) |
| 2048 | vm | DBG_VMM | debug [virtual memory manager](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/vmm.c) |
| 2048 | vm | DBG_VMM | debug [virtual memory manager](https://gitlab.com/bztsrc/osz/blob/master/src/core/vmm.c) |
| 4096 | ma | DBG_MALLOC | debug [libc memory allocation](https://gitlab.com/bztsrc/osz/blob/master/src/libc/bztalloc.c) |
| 8192 | bl | DBG_BLKIO | debug [block level I/O](https://gitlab.com/bztsrc/osz/blob/master/src/fs/vfs.c) |
| 16384 | fi | DBG_FILEIO | debug [file level I/O](https://gitlab.com/bztsrc/osz/blob/master/src/fs/taskctx.c) |
......@@ -114,21 +112,6 @@ the user for the current time if date could not be detected (no RTC). On the oth
RTC could be inaccurate). If you use "tz=0", then date and time will be in UTC, and boot will continue even if time is
unknown. In this case you'll see dates "0000-00-00T00:00:00+00:00" in the system log until ntpd sets the correct time.
Clock Source
------------
Either a numeric value or exactly one flag.
| Value | Flag | Platform | Description |
| ----: | ---- | -------- | ----------- |
| 0 | | - | auto detect |
| 1 | hp | x86_64-acpi | High Precision Event Timer (default) |
| 2 | pi | x86_64-acpi | Programmable Interval Timer (fallback) |
| 2 | pi | x86_64-ibmpc | Programmable Interval Timer (default) |
| 3 | rt | x86_64-ibmpc | Real Time Clock |
| 1 | ar | aarch64-rpi | Built-in ARM CPU timer (default) |
| 2 | sy | aarch64-rpi | BCM2837 System Timer |
Low Memory Footprint
--------------------
......@@ -145,5 +128,4 @@ Performance Tunning
If you aim for a fast system, and you have plenty of RAM and SSSE3 (x86_64) or Neon (AArch64) capable processor, do:
1. Use `OPTIMIZE = 1` in [Config](https://gitlab.com/bztsrc/osz/blob/master/Config). This will utilize SIMD whenever possible (requires recompilation).
2. Use double buffering with true color visual, on a card that natively uses ARGB packed pixel format (and not ABGR). In this case a much faster blit implementation is used (and optimized even further with point 1).
3. Use high precision timer. Delays smaller than the timer interrupt interval are implemented by a userspace side busy loop instead of blocking the task.
OS/Z Compilation
================
Requirements
------------
- GNU toolchain (make, gcc, binutils (gas, ld, objcopy, strip), see [tools/cross-gcc.sh](https://gitlab.com/bztsrc/osz/blob/master/tools/cross-gcc.sh))
- instead of gcc and ld, you can use LLVM Clang and LLVM lld
- bash or dash (shell scripts are used to [generate different files](https://gitlab.com/bztsrc/osz/blob/master/tools/drivers.sh) during compilation)
Configuration
-------------
The `core` is always compiled for a specific [machine](https://gitlab.com/bztsrc/osz/blob/master/docs/porting.md),
which can be controlled in [Config](https://gitlab.com/bztsrc/osz/blob/master/Config) with `ARCH` and `PLATFORM` variables.
Valid combinations are:
| ARCH | PLATFORM | OPTIMIZE | Description |
| ------- | -------- | -------- | ----------- |
| x86_64 | ibmpc | x | For old machines, uses PIC, PIT, RTC, enumerates PCI bus |
| x86_64 | acpi | 0 | For new machines, APIC, IOAPIC, HPET and parses ACPI tables |
| x86_64 | acpi | 1 | For new machines, SIMD, x2APIC, x2IOAPIC, HPET and parses ACPI tables |
| aarch64 | rpi | 0 | Raspberry Pi 3+ |
| aarch64 | rpi | 1 | Raspberry Pi 3+, NEON |
If you have the "dialog" tool installed on your system, you can run the following to get a nice ncurses based interface.
```shell
$ make config
```
<img height="64" src="https://gitlab.com/bztsrc/osz/raw/master/docs/oszcfg1.png" alt="make config">
Compilation
-----------
The make system is created in a way that you can say `make all` in almost every directory. If you do that in the top level
directory, it will compile everything and will also build disk image for you, but surpressing messages. In subdirectories make
will always display the full command executed during compilation.
For example if you enter `src/fs` and issue `make all` there, then only the filesystem subsystem will be recompiled, and
you'll be able to see the exact commands used.
So to compile all the things, simply type
```shell
$ make all
```
You should see something similar to this:
```
$ make all
TOOLS
src mkfs.c
src elftool.c
CORE
lnk sys/core (x86_64-ibmpc)
lnk sys/lib/libc.so (x86_64)
BASE
src sys/ui
src sys/bin/sh
src sys/bin/test
src sys/bin/sys
src sys/syslog
src sys/sound
src sys/fs
src sys/inet
src sys/init
DRIVERS
src sys/drv/fs/gpt.so
src sys/drv/fs/pax.so
src sys/drv/fs/vfat.so
src sys/drv/fs/fsz.so
src sys/drv/fs/tmpfs.so
src sys/drv/input/ps2.so
src sys/drv/display/fb.so
src sys/driver
USERSPACE
IMAGES
mkfs initrd
mkfs boot (ESP)
mkfs usr
mkfs var
mkfs home
mkfs bin/osZ-latest-x86_64-ibmpc.dd
```
Non-EFI loader
--------------
If you want to recompile `loader/boot.bin` and `loader/bootboot.bin`, you'll need [fasm](http://flatassembler.net).
Unfortunately GAS is not good enough at mixing 16, 32 and 64 bit instuctions, which is necessary for BIOS booting. To keep
my promise that you'll only need the GNU toolchain, I've added those pre-compiled BOOTBOOT binaries to the source.
See what you've done!
---------------------
The live disk image is generated to [bin/osZ-(ver)-(arch)-(platform).dd](https://gitlab.com/bztsrc/osz/blob/master/bin). Use the
`dd` command to write it on a USB stick or boot with qemu, bochs or VirtualBox.
```
$ dd if=bin/osZ-latest-x86_64-acpi.dd of=/dev/sdc
$ make testq
$ make testb
$ make testv
```
There's a detailed [tutorial on testing](https://gitlab.com/bztsrc/osz/blob/master/docs/howto1-testing.md).
OS/Z Compilation
================
OS/Z Fordítása
==============
Requirements
------------
Eszközök
--------
- GNU toolchain (make, gcc, binutils (gas, ld, objcopy, strip), see [tools/cross-gcc.sh](https://gitlab.com/bztsrc/osz/blob/master/tools/cross-gcc.sh))
- bash or dash (shell scripts are used to [generate different files](https://gitlab.com/bztsrc/osz/blob/master/tools/drivers.sh) during compilation)
- GNU eszköztár (make, gcc, binutils (gas, ld, objcopy, strip), lásd [tools/cross-gcc.sh](https://gitlab.com/bztsrc/osz/blob/master/tools/cross-gcc.sh))
- gcc és ld helyett LLVM Clang és LLVM lld is használható
- bash vagy dash (shell szkriptek [generálnak fájlokat](https://gitlab.com/bztsrc/osz/blob/master/tools/drivers.sh) fordítás közben)
Configuration
-------------
Konfigurálás
------------
The `core` is always compiled for a specific [machine](https://gitlab.com/bztsrc/osz/blob/master/docs/porting.md),
which can be controlled in [Config](https://gitlab.com/bztsrc/osz/blob/master/Config) with `ARCH` and `PLATFORM` variables.
Valid combinations are:
A `core` mindig egy adott [gépre](https://gitlab.com/bztsrc/osz/blob/master/docs/porting.md) fordul, amit a
[Config](https://gitlab.com/bztsrc/osz/blob/master/Config) fájlban az `ARCH` és `PLATFORM` változókkal állíthatunk be.
Az érvényes kombinációk a következők:
| ARCH | PLATFORM | Description |
| ---- | -------- | ----------- |
| x86_64 | ibmpc | For old machines, uses PIC and PIT (or RTC), enumerates PCI bus |
| x86_64 | acpi | For new machines, APIC, x2APIC, IOAPIC, HPET and parses ACPI tables |
| aarch64 | rpi | Raspberry Pi 3+ |
| ARCH | PLATFORM | OPTIMIZE | Leírás |
| ------- | -------- | -------- | ----------- |
| x86_64 | ibmpc | x | Régi gépekhez, PIC, PIT, RTC, PCI buszon detektál |
| x86_64 | acpi | 0 | Új gépekhez, APIC, IOAPIC, HPET és ACPI táblákat értelmez |
| x86_64 | acpi | 1 | Új gépekhez, SIMD, x2APIC, x2IOAPIC, HPET és ACPI táblákat értelmez |
| aarch64 | rpi | 0 | Raspberry Pi 3+ |
| aarch64 | rpi | 1 | Raspberry Pi 3+, NEON |
If you have the "dialog" tool installed on your system, you can run the following to get a nice ncurses based interface.
Ha van telepítve "dialog" segédprogram a gépegre, akkor a következő parancs egy kényelmes ncurses alapú felülettel segít.
```shell
$ make config
......@@ -29,34 +32,33 @@ $ make config
<img height="64" src="https://gitlab.com/bztsrc/osz/raw/master/docs/oszcfg1.png" alt="make config">
Compilation
-----------
Fordítás
--------
The make system is created in a way that you can say `make all` in almost every directory. If you do that in the top level
directory, it will compile everything and will also build disk image for you, but surpressing messages. In subdirectories make
will always display the full command executed during compilation.
A fordítási környezet úgy lett kialakítva, hogy majdnem minden könyvtárban kiadhatod a `make all` parancsot. Ha ezt a főkönyvtárban
teszed, akkor mindent lefordít és a lemezképeket is kigenerálja neked, de részletes üzenetek nélkül. Alkönyvtárakban kiadva a
make mindig megjeleníti a teljes parancsot, amit a fordításkoz és linkeléshez használ.
For example if you enter `src/fs` and issue `make all` there, then only the filesystem subsystem will be recompiled, and
you'll be able to see the exact commands used.
Például ha belépsz az `src/fs` mappába és ott adod ki a `make all` parancsot, akkor csak a fájl rendszer szolgáltatás fordítódik le,
és látni fogsz minden egyes felhasznált parancsot.
So to compile all the things, simply type
Szóval ha mindent szeretnél fordítani, akkor csak add ki
```shell
$ make all
```
You should see something similar to this:
Valami ilyesmit kell hogy láss:
```
$ make all
TOOLS
SEGÉDPROGRAMOK
src mkfs.c
src elftool.c
CORE
gen x86_64/ibmpc/isrs.S (PIC, numirq 32, idtsz 4096)
RENDSZERMAG
lnk sys/core (x86_64-ibmpc)
lnk sys/lib/libc.so (x86_64)
BASE
ALAPRENDSZER
src sys/ui
src sys/bin/sh
src sys/bin/test
......@@ -66,7 +68,7 @@ BASE
src sys/fs
src sys/inet
src sys/init
DRIVERS
ESZKÖZMEGHAJTÓK
src sys/drv/fs/gpt.so
src sys/drv/fs/pax.so
src sys/drv/fs/vfat.so
......@@ -75,8 +77,8 @@ DRIVERS
src sys/drv/input/ps2.so
src sys/drv/display/fb.so
src sys/driver
USERSPACE
IMAGES
ALKALMAZÁSOK
LEMEZKÉPEK
mkfs initrd
mkfs boot (ESP)
mkfs usr
......@@ -85,18 +87,18 @@ IMAGES
mkfs bin/osZ-latest-x86_64-ibmpc.dd
```
Non-EFI loader
--------------
Nem-EFI betöltő
---------------
If you want to recompile `loader/boot.bin` and `loader/bootboot.bin`, you'll need [fasm](http://flatassembler.net).
Unfortunately GAS is not good enough at mixing 16, 32 and 64 bit instuctions, which is necessary for BIOS booting. To keep
my promise that you'll only need the GNU toolchain, I've added those pre-compiled BOOTBOOT binaries to the source.
Ha újra akarod fordítani a `loader/boot.bin` és `loader/bootboot.bin` fájlokat, akkor szükséged lesz a [fasm](http://flatassembler.net)-ra.
Sajnos a GAS nem eléggé okos 16, 32 és 64 bites utasítások keverésében, ami elengedhetetlen a BIOS-ról induláskor. Hogy megtarthassam
az ígéretem, hogy csak GNU eszköztárra lesz szükség, hozzáadtam az előre lefordított BOOTBOOT bonárisokat a forrásokhoz.
See what you've done!
---------------------
Nézd mit csináltál!
-------------------
The live disk image is generated to [bin/osZ-(ver)-(arch)-(platform).dd](https://gitlab.com/bztsrc/osz/blob/master/bin). Use the
`dd` command to write it on a USB stick or boot with qemu, bochs or VirtualBox.
Az indítható lemezképet a [bin/osZ-(ver)-(arch)-(platform).dd](https://gitlab.com/bztsrc/osz/blob/master/bin) néven generálja. Ezt a
`dd` paranccsal kiírhatod pendrávjra vagy SD kártyára, illetve futtathatod qemu, bochs vagy VirtualBox alatt.
```
$ dd if=bin/osZ-latest-x86_64-acpi.dd of=/dev/sdc
......@@ -105,4 +107,4 @@ $ make testb
$ make testv
```
There's a detailed [tutorial on testing](https://gitlab.com/bztsrc/osz/blob/master/docs/howto1-testing.md).
Részletes [ismertető a tesztelésről](https://gitlab.com/bztsrc/osz/blob/master/docs/howto1-testing.md).
......@@ -6,9 +6,9 @@ Supported devices
* VESA 2.0 VBE, GOP, VideoCore (set up by [loader](https://gitlab.com/bztsrc/osz/blob/master/loader), 32 bit frame buffer)
* x86_64: syscall, NX protection
* x86_64-apic: IOAPIC + APIC, x2APIC, see [ISRs](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/isrs.sh)
* x86_64-ibmpc: PIC, [PIT](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/ibmpc/pit.S), [RTC](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/ibmpc/rtc.S)
* x86_64-ibmpc: [PIC](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/ibmpc/pic.S), [PIT](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/ibmpc/pit.S), [RTC](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/ibmpc/rtc.S)
* PS2 [keyboard](https://gitlab.com/bztsrc/osz/blob/master/src/drivers/input/ps2/keyboard.S) and [mouse](https://gitlab.com/bztsrc/osz/blob/master/src/drivers/input/ps2/mouse.S)
* AArch64: SVC, WNX protection
Planned drivers
---------------
......@@ -35,11 +35,16 @@ under these directories each driver has exactly one sub-directory. The compiled
driver will be placed in `sys/drv/(class)/(sub-directory).so`. For example, `src/drivers/input/ps2/main.c` will be compiled
to `sys/drv/input/ps2.so`.
Note that for performance, CPU, memory management and interrupt controllers (like PIC, IOAPIC) do not have separate drivers, they
are built into [core](https://gitlab.com/bztsrc/osz/blob/master/src/core/x86_64/isrs.sh). To
switch among them, you'll have to edit `PLATFORM` in [Config](https://gitlab.com/bztsrc/osz/blob/master/Config) and recompile.
Note that for performance, CPU, memory management, timers and interrupt controllers do not have separate drivers, they are built
into [core](https://gitlab.com/bztsrc/osz/tree/master/src/core). To switch among them, you'll have to edit `PLATFORM` and
`OPTIMIZE` in [Config](https://gitlab.com/bztsrc/osz/blob/master/Config) and recompile.
Timers are also built into the core, but you can switch among HPET, PIT and RTC with a boot time configurable [environment variable](https://gitlab.com/bztsrc/osz/blob/master/docs/bootopts.md).
| PLATFORM | OPTIMIZE | Interrupt controller | Scheduler timer | Wallclock timer |
| ------------ | -------- | -------------------- | --------------- | ---------------- |
| x86_64-ibmpc | x | PIC | PIT | RTC |
| x86_64-apic | 0 | IOAPIC | LAPIC Timer | HPET |
| x86_64-apic | 1 | x2IOAPIC | x2APIC Timer | HPET |
| aarch64-rpi | x | BCM Interrupt Ctrl | ARM CNTP_TVAL | BCM System Timer |
### Files
......
FS/Z File System
================
It was designed to store unimagineable amounts of data. It's a classic UNIX type file system with inodes and superblock
but without their limitations. There's no limit on number of files or directories in FS/Z (save storage capacity).
Logical sector numbers are stored in 2^128 bits to be future proof. Current implementation
only uses 2^64 bits (can handle drives up to 64 Zettabytes with 4096 bytes logical sector size, and 1 Yottabyte with 64k logical sector size).
Logical sector size can be 2048, 4096, 8192... etc. The suggested size matches the memory page size on the architecture. That is 4096
for x86_64 and aarch64.
The file system is designed to be recoverable by `fsck` as much as possible. A totally screwed up superblock can be reconstructed
by examining the sectors on a disk, looking for inodes of meta data. RAID mirroring is also possible with this file system.
For detailed bit level specification and on disk format see [fsZ.h](https://gitlab.com/bztsrc/osz/blob/master/include/sys/fsZ.h). For
the higher level abstraction layer, see [VFS](https://gitlab.com/bztsrc/osz/blob/master/docs/vfs.md).
Implementations
---------------
The file system driver is implemented 3 times:
1. in the [loader](https://gitlab.com/bztsrc/osz/blob/master/loader), which supports GPT and FAT to locate initrd, and FS/Z, cpio, tar, sfs to locate kernel inside the initrd. This is a read-only implementation, that requires defragmented files.
2. in the [core](https://gitlab.com/bztsrc/osz/blob/master/src/core/fs.c), which is used at early boot stage to load various files (like the FS/Z file system driver) from initrd. This is also a read-only implementation for defragmented files.
3. as a [file system driver](https://gitlab.com/bztsrc/osz/blob/master/src/drivers/fs/fsz/main.c) for the FS service, which is a fully featured implementation.
FS/Z Super Block
----------------
The super block is the first logical sector of the media and optionally repeated in the last sector. If the superblocks differ, they have
checksums to decide which sector is correct.
Logical Sector Number
---------------------
The size of a logical sector is recorded in the superblock. Sector number is a scalar value up to 2^128 and relative to
the superblock, which therefore resides on LSN 0, regardless whether it's on LBA 0 (whole disk) or any other (partition).
This means you can move the file system image around without the need of relocating sector addresses. Because LSN 0
stores the superblock which is never referenced from any file, so in mappings LSN 0 encodes a sector full of zeros,
allowing spare files and gaps inside files.
File ID
-------
The file id (fid in short) is a logical sector number, which points to a sector with i-node structure. That can be checked
by starting with the magic bytes "FSIN".
Inode
-----
The most important structure of FS/Z. It describes portion of the disk as files, directory entries etc. It's the first
1024 bytes of a logical sector. It can contain several versions thus allowing not only easy snapshot recovery, but per file
recovery as well.
FS/Z also stores meta information (like mime type and checksum) on the contents, and supports meta labels in inodes.
Contents have their own address space mapping (other than LSNs). To support that, FS/Z has two different LSN translation
mechanisms. The first one is similar to memory mapping, and the other stores variable sized areas, so called extents.
Sector List
-----------
Used to implement extents and marking bad and free sector areas on disks. A list item contains a starting LSN and the number
of logical sectors in that area, describing a continuous area on disk. Therefore every extent is 32 bytes long.
Sector Directory
----------------
A sector that has (logical sector size)/16 entries. The building block of memory paging like translations. As with sector lists,
sector directories can handle LSNs up to 2^128 (or 2^96 if data checksum enabled), but describe fix sized, non-continous areas on
disk. Not to confuse with common directories which are used to assign names to fids.
Directory
---------
Every directory has 128 bytes long entries which gives 2^121-1 as the maximum number of entries in one directory. The first
entry is reserved for the directory header, the others are normal fid and name assignments. Entries are alphabetically ordered
thus allowing fast 0(log n) look ups with `libc`'s [bsearch](https://gitlab.com/bztsrc/osz/blob/master/src/libc/stdlib.c).
16 bytes go for the fid, 1 byte reserved for the number of multibyte characters (not bytes) in the filename. That gives 111 bytes for
filename (maybe less in characters as UTF-8 is a variable length encoding). That's the hardest limitation in FS/Z, but let's
face it most likely you've never seen a filename longer than 42 characters in your whole life. Most filenames are below 16 characters.
This diff is collapsed.
......@@ -233,7 +233,7 @@ In addition to ELF symbols, you can also use these:
p switch to previous task
p 29 switch to task that's pid is 29
i switch bytes / mnemonics in code view
i isr_irq1+3 view instruction in function
i intr_1+3 view instruction in function
i +7F move disassembler window forward by 127 bytes
x 1234 dump memory at 0x1234
x /q dump in quad words
......
......@@ -181,7 +181,7 @@ int main(int argc, char **argv)
}
```
The full list of return codes can be found in [sysexits.h](https://gitlab.com/bztsrc/osz/blob/master/etc/include/sysexits.h),
The full list of return codes can be found in [sysexits.h](https://gitlab.com/bztsrc/osz/blob/master/include/sysexits.h),
but the most important ones are:
| Define | Meaning |
......
......@@ -2,9 +2,10 @@ OS/Z Memory Management
======================
Memory is allocated at several levels:
- [pmm_alloc()](https://gitlab.com/bztsrc/osz/tree/master/src/core/pmm.c) allocate physical RAM pages
- [kalloc()](https://gitlab.com/bztsrc/osz/tree/master/src/core/pmm.c) allocate core (kernel) bss memory
- [malloc()](https://gitlab.com/bztsrc/osz/tree/master/src/libc/bztalloc.c), realloc(), calloc(), free() allocate user space bss memory
- [pmm_alloc()](https://gitlab.com/bztsrc/osz/tree/master/src/core/pmm.c) allocate physical RAM pages (core only)
- [vmm_alloc()](https://gitlab.com/bztsrc/osz/tree/master/src/core/vmm.c) allocate virtual pages (core only)
- [kmalloc()](https://gitlab.com/bztsrc/osz/tree/master/src/libc/bztalloc.c), krealloc(), kfree() allocate core (kernel) bss memory
- [malloc()](https://gitlab.com/bztsrc/osz/tree/master/src/libc/bztalloc.c), realloc(), calloc(), free() allocate bss memory
- [smalloc()](https://gitlab.com/bztsrc/osz/tree/master/src/libc/bztalloc.c), srealloc(), scalloc(), sfree() allocate shared memory
Memory Mapping
......@@ -18,20 +19,25 @@ User Tasks
| Virtual Address | Scope | Description |
| ---------------- | ------- | ----------- |
| -512G .. -1G-1 | global | global [shared memory](https://gitlab.com/bztsrc/osz/tree/master/src/libc/bztalloc.c) space (user accessible, read/write) |
| -1G .. 0 | global | core memory (supervisor only, see below) |
| 0 .. 4096 | local | Task Control Block (read-only) |
| 4096 .. x | local | Message Queue (read-only, growing upwards) |
| -512G .. -4G-1 | global | global [shared memory](https://gitlab.com/bztsrc/osz/tree/master/src/libc/bztalloc.c) space (user accessible, read/write) |
This no cahceable address space is commonly available for all tasks on all cores. For passing big buffers and cross-CPU syncronization flags.
| Virtual Address | Scope | Description |
| ---------------- | ------- | ----------- |
| 0 .. 4096 | local | Task Control Block (supervisor only, NULL references will throw an exception) |
| 4096 .. nrmq | local | Message Queue (read-only)) |
| nrmq .. x | local | Message Queue circular buffer (read-only, growing upwards) |
| x .. 2M-1 | local | local stack (read/write, growing downwards) |
| 2M .. x | [process](https://gitlab.com/bztsrc/osz/tree/master/docs/process.md) | user program text segment (read only) |
| x .. 4G-1 | process | shared libraries (text read only / data read/write) |
| 4G .. 2^48-4G | local | dynamically [allocated tls memory](https://gitlab.com/bztsrc/osz/tree/master/src/libc/bztalloc.c) (growing upwards, read/write) |
| 2^48-4G .. 2^48 | local | pre-allocated mapped system buffers (screen, initrd, MMIO) |
| 2^48-4G .. 2^48 | local | pre-allocated / mapped system buffers (screen, initrd, MMIO etc.) |
If two mappings are identical save the TCB, message queue and stack, then they belong to the same process.
The maximum number of pending events in a queue is a boot time parameter and can be set in [environment](https://gitlab.com/bztsrc/osz/tree/master/etc/sys/config) with "nrmqmax". It's given
in pages, so multiply by page size and devide by sizeof(msg_t) which is 64 bytes. Defaults to 1 page, meaning 4096/64 = up to 64 pending events.
The maximum number of pending events in a queue is a boot time parameter and can be set in [environment](https://gitlab.com/bztsrc/osz/tree/master/etc/sys/config) with "nrmqmax".
It's given in pages, so multiply by page size and devide by sizeof(msg_t) which is 64 bytes. Defaults to 1 page, meaning 4096/64 = up to 64 pending events.
Please note that 2^56 (128T) is an architectural limit of x86_64, but OS/Z only implements 2^48 (512G) as of now.
......@@ -44,20 +50,33 @@ User interface has the linear frame buffer mapped at 2^48-4G. It's used when the
Filesystem service has the initrd mapped at 2^48-4G. Initrd maximum size is 16M on boot, but it can grow
dynamically in run-time up to 4G. The /dev/root device uses it, implemented in "devfs" as a in-memory disk.
### Drivers
Device drivers have the MMIO mapped at 2^48-4G, if that's supported on the platform.
Core Memory
-----------
The pages for the core are marked as supervisor only, meaning userspace programs cannot access it. They are also globally
mapped to every task's address space.
| Virtual Address | Scope | Description |
| ---------------- | ----- | ----------- |
| -1G .. -1G+16M | core | Initial ramdisk (used by fs_locate() during early stage of boot) |
| -96M .. -64M-1 | core | MMIO area (used by platform specific code) |