dedicated kthread interface and enablement of lrud auto-recycler

[lunaix-os.git] / README.md

diff --git a/README.md b/README.md
index bc0b8d4f5895dcc4d23a1a9d3905543fa253f4da..c4a4ea9579fd3f61f0c310e746abc6bfffa76814 100644 (file)
--- a/README.md
+++ b/README.md
@@ -1,73 +1,264 @@
 <p align="center">
 <p align="center">

-  <img width="256" src="docs/img/lunaix-os-logo.png">
+  <img width="auto" src="docs/img/lunaix-os-logo.png">

 </p>
 
 </p>
 

-# LunaixOS Project 
-LunaixOS - 一个简单的，详细的，POSIX兼容的（但愿！），带有浓重个人风格的操作系统。开发过程以视频教程形式在Bilibili呈现：[《从零开始自制操作系统系列》](https://space.bilibili.com/12995787/channel/collectiondetail?sid=196337)。
+<p align="center">
+  <a href="docs/README.cn.md">简体中文</a> | <a href="#the-lunaixos-project">English</a>
+</p>
+
+# The Lunaix Project
+
+The Lunaix kernel (or soon-to-be LunaixOS) is a hobby kernel, written entirely from scratch. Designed to be POSIX-compliance, performance critical and modern, with some independent thoughts being applied in architectural design.
+
+This project is based solely on first principles. Meaning that it does neither copy-pasting nor recycling other os projects/tutorials. However, this doesn't necessarily imply superiority; In fact, Lunaix starts out as a mean to challenge my understanding in OS theory and also a platform for experimenting some advanced (and cool) kernel features. If you are a kernel hobbyist who want some new perspectives or just simply get fatigued on those recycled content, then you have came to the right place!
+
+
+| ![showcase_lunaix-over-serial.png](docs/img/showcase_lunaix-over-serial.png) |
+|:--:|
+| *Lunaix over serial* (`console=/dev/ttyS0`) |
+
+| ![showcase_lunaix-over-serial.png](docs/img/showcase_lunaix-over-vga-text.png) |
+|:--:|
+| *Lunaix over VGA text mode* (`console=/dev/ttyVCON0`) |

 
 

-## 当前进度以及支持的功能
+> By the way, do you know there is an online video course  by the author on the design of lunaix? [Check it out](https://space.bilibili.com/12995787/channel/collectiondetail?sid=196337) (although it is in Chinese!)

 
 

-该操作系统支持x86架构，运行在保护模式中，采用宏内核架构，目前仅支持单核心。内存结构采用经典的3:1划分，即低3GiB为用户地址空间（0x400000 ~ 0xBFFFFFFF），内核地址空间重映射至高1GiB（0xC0000000 ~ 0xFFFFFFFF）。内存的详细布局可参考[LunaixOS内存地图](docs/img/lunaix-os-mem.png)
+## Features

 
 

-在下述列表中，则列出目前所支持的所用功能和特性。列表项按照项目时间戳进行升序排列。
+In a nutshell, Lunaix is a multi-architectural general purpose kernel, and she is:

 
 

-+ 使用Multiboot进行引导启动
-+ APIC/IOAPIC作为中断管理器和计时器
-+ ACPI
-+ 虚拟内存
-+ 内存管理与按需分页（Demand Paging）
-+ 键盘输入
-+ 多进程
-+ 14个常见的Linux/POSIX系统调用（[附录1](#appendix1)）
-+ 用户模式
-+ 信号机制（进行中）
++ fully-preemptive
++ modular design with configurable components at compile-time and extendable subsystems
++ high-performance by utilising advanced caching techniques and infrastructure.
++ fault-tolerance with sophisticated builtin error handling and tracing techniques.
++ robust in nature with techniques such as proactive deadlock detection and driver isolation mechanism.

 
 

-## 目录结构
+The author has put a significant amount of time on devising better abstractions, advance kernel features and various optimisation techniques. To give a better understanding (and appreciation) of the works being done, the following non-exhaust list has been compiled with features that are currently supported in lunaix:

 
 

-| | |
++ Multi-ISA
+  + x86_32
+  + x86_64
+  + Aarch64 (W.I.P)
++ Boot protocol
+  + abstraction for different protocol
+  + configurable kernel boot-time parameters
++ Platform resource management and definition
+  + ACPI
+  + Devicetree
++ Memory management
+  + architecture-neutral abstraction
+  + on-demand paging
+  + copy-on-write and page sharing
+  + compound page support
+  + explicit huge page support (sorry, not THP!)
+  + reverse mapping indexing (rmap)
+  + memory compaction (W.I.P)
+  + slab-alike object allocator
+  + highmem support
+  + remote address space accessing
++ Multi-tasking
+  + Protection level and process image isolation
+  + Native threading support (no more lightweight process nonsense)
+  + Signal mechanism
+  + Kernel level multi-tasking (i.e. kernel threads)
+  + Round-robin scheduling (for now)
+  + Preemptive kernel design
+  + taskfs: file system interface to process and threads
++ File system
+  + virtual file system framework
+  + ...with POSIX compliant interfaces
+  + file system mounting mechanism
+  + page cache for file IO
+  + node cache for vfs structure representation.
+  + ext2 (rev.0, rev.1)
+  + iso9660 (rock-ridge)
+  + twifs: file system interface to kernel states.
++ Device management and interrupt handling
+  + architecture-neutral design
+  + generalised driver framework
+  + generalised irq framework
+  + driver modularisation design
+  + support asynchronous device model
+  + devfs: file system interface to device subsystem
++ Block I/O (blkio)
+  + generalised block IO interface and encapsulation
+  + blkio packets caching
+  + asynchronous blkio operation in nature
++ Serial I/O
+  + POSIX-compliant serial port model
+  + serial device driver framework (part of driver framework)
++ Caching Infrastructure
+  + primitive: generic sparse associative array (spatial data)
+  + LRU replacement policy and pooling
+  + kernel daemon for scheduled cache eviction
++ Error handling and detection
+  + stack back-tracing with symbol resolution
+  + nested exception unfolding
+  + CPU state dumping
+  + Deadlock/hung-up detection
+
+For the device drivers that are currently support see below:
+
++ Arhcitecture Neutral
+  + UART 16650-compatible driver
+  + Serial ATA AHCI
+  + PCI 3.0
+  + PCIe 1.1
+  + Standard VGA
++ Intel x86
+  + RTC (Intel PCH)
+  + IOAPIC irq controller
+  + APIC Timer
+  + Legacy i8042 keyboard controller
++ ARM
+  + GICv3
+  + PL011 (W.I.P)
+
+## Project Structure
+
+| Path | Description |

 |-----|------|
 |-----|------|

-| [lunaix-os](lunaix-os/) | LunaixOS源代码 |
-| [slides](slides/) | 视频中所用的幻灯片和补充材料 |
-| [reference-material](reference-material/)| 标准，技术文档和参考文献 |
+| [lunaix-os](../lunaix-os/) | LunaixOS source code |
+| [slides](../slides/) | Slides used in my videos |
+| [reference-material](../reference-material/)| References |
+
+## Compile and Build
+
+Building lunaix is simple, no more bloated dependencies to install, basic `build-essentials` installation plus a python are sufficient.
+
++ gcc (recommend v12+)
++ binutils
++ make
++ python (recommend v3.11+)
+
+And also one should have environment variable `ARCH=<arch>` exported, where `<arch>` is one of the supported arhcitecture (`x86_32`, `x86_64`, `aarch64`).
+
+For cross compilation, also export `CX_PREFIX` to the gcc prefix for the corresponding `<arch>`.
+
+The following `make` actions are then available to use.
+
+| Make command | Usage |
+| ---- | ---- |
+| `make all`               | Build the kernel bin |
+| `make rootfs`            | Build the stock rootfs |
+| `make clean`             | clean |
+| `make config`            | menuconfig |
+
+A successful build will give `build/bin/kernel.bin`.
+
+**Please note: this is the kernel, not a bootable image, it require a bootloader to boot and specify the rootfs.**
+
+## Get Started
+
+If you are impatient, or just want something to run and don't want to went through tedious process of configuring rootfs and tweak kernel parameters. You can use the `live_debug.sh` provided in the lunaix root directory to quickly bring up the system with default parameters (also used by the author for debugging).
+
+Following the steps:
+
+1. Select an architecture `<arch>`
+2. Check the compilation prerequisites and presence of `qemu-system-<arch>`
+3. Optionally export `CX_PREFIX` if you are building for another architecture.
+4. Run `make ARCH=<arch> rootfs` to build stock rootfs image, require support of `dd`，`mkfs.ext2`, `mount -o loop`, `mktemp`.
+5. Run `ARCH=<arch> live_debug.sh` to boot in QEMU with gdb hooked (one should see a gdb session)
+6. telnet to `localhost:12345`, this is QEMU emulated serial port
+7. type `c` in the active gdb session and commence emulation.
+8. Congrats, enjoy your lunaix!
+(or submit an issue)
+
+
+## Booting the kernel
+
+Since lunaix is a kernel, much like linux. It requires additional setup to do the magic. And, as in "much like linux", methods to make linux kernel boot can also apply to lunaix without or with little translation, as we will discuss below.
+
+The bootloader part is generic, any bootloader, for example GRUB will work (not tested for UEFI, but I expect this would be an exception), or booting up in QEMU using `-kernel` option

 
 

-## 参考教程
+The kernel command line, is however, a bit differentiated.
+The syntax is similar, both takes form of space-separated array of `<key>=<val>` pairs or boolean `<flag>`.

 
 

-**没有！！** 本教程以及该操作系统均为原创，没有基于任何市面上现行的操作系统开发教程，且并非是基于任何的开源内核的二次开发。
+Currently, lunaix support the following options

 
 

-为了制作LunaixOS，作者耗费大量时间和精力钻研技术文档，手册，理论书籍以及现行工业标准，从而尽量保证了知识的一手性。（这样一来，读者和听众们也算是拿到了二手的知识，而不是三手，四手，甚至n手的知识）。
+| Option | Default Value | Optional | Usage |
+| ------ | ----          |  -----   | ----  |
+| console | `/dev/ttyS0`   | No | Specify the system console device, path within lunaix's devfs |
+| rootfs | `/dev/block/sda` | No | Specify the device contain rootfs image, path within lunaix's devfs |
+| init | `/init` | Yes | Path within rootfs of the `init` |

 
 

-大部分的文档和标准可以在上述的[reference-material](reference-material/)中找到。

 
 

-当然，您也可以参考以下列表来了解现阶段的LunaixOS都使用了哪些资料（本列表会随着开发进度更新）：
+## Submit an Issue

 
 

-#### 手册，标准，技术文档
-+ [Intel 64 and IA-32 Architecture Software Developer's Manual (Full Volume Bundle)](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html)
-+ [ACPI Specification (version 6.4)](https://uefi.org/sites/default/files/resources/ACPI_Spec_6_4_Jan22.pdf)
+If one ran into bug, one can submit an issue by filling up the following template
+
+```
+1. Describe the problem
+    "How does it look like, anything descriptive: visual, sonic, emotional experience"
+
+2. Steps to reproduce
+    "How you ran into this mess?"
+
+3. Expected behaviour
+    "What do you intended/expected to achieve/to be"
+
+4. Lunaix's panic trace (if applicable)
+
+5. Other clues that you think might be helpful
+```
+
+
+## Limitations
+
+The development process is still in motion, any limitation can be categorised as a feature yet to be. However, some features that the author considered to be the most urgent and wish the matters to be discussed.
+
+Lunaix is under impression of uniprocessor and not capable of running in SMP environment. This is major held back of being a modern operating system. It has the highest priority among all other tasks
+
+Lunaix do not have a mature (or even, an infant) user space ecosystem, mainly because the lack of a proper and sophisticated libc. Efforts need to be done for porting one to the target. However, given the author's tight schedule, this task is unfortunately still beyond the horizon.
+
+## Acknowledgement
+
+Albeit one must realise that the author has mentioned it in the very beginning, the author would like to emphaise **again** on the nature of this project.
+
+As a personal challenge, this project is independently developed by the author single-handly, which means:
+
++ No reference to existing tutorials, books, online courses or any open source project that might provide any example, hint or working prototype on the design and implementation of kernel, subsystems or anythings that can be contributed towards a working prototype.
++ The author has no prior knowledge on Linux kernel through out 90% of the project time.
++ All knowledge on the kernel design is coming from the basic textbook on operating system theory, that is, *Modern Operating System* by Tanenbaum.
++ All knowledge on the system programming is coming from the basic textbook, that is, *Computer System - A Programmer's Perspective Third Edition*
++ All knowledge on the generic framework design and driver development are ingested from various technical specifications gathered across the Internet.
+
+## References
+
++ Intel 64 and IA-32 Architecture Software Developer's Manual (Full Volume Bundle)
++ ACPI Specification (version 6.4)
++ Devicetree Specification
++ ARM® Generic Interrupt Controller (v3)
++ Arm® Architecture Reference Manual (Profile-A)
++ Procedure Call Standard for the Arm® 64-bit Architecture (AArch64)

 + IBM PC/AT Technical Reference
 + IBM VGA/XGA Technical Reference
 + 82093AA I/O Advanced Programmable Controller (IOAPIC) (Datasheet)
 + MC146818A (Datasheet)
 + Intel 500 Series Chipset Family Platform Controller Hub (Datasheet - Volume 2)
 + IBM PC/AT Technical Reference
 + IBM VGA/XGA Technical Reference
 + 82093AA I/O Advanced Programmable Controller (IOAPIC) (Datasheet)
 + MC146818A (Datasheet)
 + Intel 500 Series Chipset Family Platform Controller Hub (Datasheet - Volume 2)

-
-#### 理论书籍
-+ *Computer System - A Programmer's Perspective Third Edition (CS:APP)* (Bryant, R & O'Hallaron, D)
++ PCI Local Bus Specification, Revision 3.0
++ PCI Express Base Specification, Revision 1.1
++ PCI Firmware Specification, Revision 3.0
++ Serial ATA - Advanced Host Controller Interface (AHCI), Revision 1.3.1
++ Serial ATA: High Speed Serialized AT Attachment, Revision 3.2
++ SCSI Command Reference Manual
++ ATA/ATAPI Command Set - 3 (ACS-3)
++ ECMA-119 (ISO9660)
++ Rock Ridge Interchange Protocol (RRIP: IEEE P1282)
++ System Use Sharing Protocol (SUSP: IEEE P1281)
++ Tool Interface Standard (TIS) Portable Formats Specification (Version 1.1)
++ *Computer System - A Programmer's Perspective Third Edition* (Bryant, R & O'Hallaron, D), a.k.a. CS:APP

 + *Modern Operating System* (Tanenbaum, A)
 + *Modern Operating System* (Tanenbaum, A)

-+ 《汇编语言》（王爽） - 用于入门Intel语法的x86汇编（对于AT&T语法，推荐阅读CS:APP）
-+ ~~《微机原理与接口技术》 - 用于大致了解x86架构的微机体系（更加细致的了解可以阅读Intel Manual）~~ （已过时，推荐阅读CS:APP）
-
-#### 网站
-+ [OSDev](https://wiki.osdev.org/Main_Page) - 杂七杂八的参考，很多过来人的经验。作者主要用于上古资料查询以及收集；技术文献，手册，标准的粗略总结；以及开发环境/工具链的搭建。
-+ [FreeVGA](http://www.osdever.net/FreeVGA/home.htm) - 98年的资源！关于VGA编程技术的宝藏网站。
-+ GNU CC 和 GNU LD 的官方文档。
-
-#### 其他
-+ Linux Manual - 用于查询*nix API的一些具体行为。
++ Free VGA, http://www.osdever.net/FreeVGA/home.htm 
++ GNU CC & LD online documentation.
++ PCI Lookup, https://www.pcilookup.com/
++ Linux man pages

 
 

+## Appendix 1: Supported System Call<a id="appendix1"></a>

 
 

-## 附录1：支持的系统调用<a id="appendix1"></a>
+**Unix/Linux/POSIX**

 
 

-### Unix/Linux/POSIX
-1. `sleep(2)`
+1. `sleep(3)`

 1. `wait(2)`
 1. `waitpid(2)`
 1. `fork(2)`
 1. `wait(2)`
 1. `waitpid(2)`
 1. `fork(2)`


@@ -80,8 +271,91 @@ LunaixOS - 一个简单的，详细的，POSIX兼容的（但愿！），带有
 1. `_exit(2)`
 1. `sigreturn(2)`
 1. `sigprocmask(2)`
 1. `_exit(2)`
 1. `sigreturn(2)`
 1. `sigprocmask(2)`

-1. `signal(2)   `
+1. `signal(2)`
+1. `kill(2)`
+1. `sigpending(2)`
+1. `sigsuspend(2)`
+2. `read(2)`
+2. `write(2)`
+2. `open(2)`
+2. `close(2)`
+2. `mkdir(2)`
+2. `lseek(2)`
+2. `readdir(2)`
+2. `readlink(2)`
+2. `readlinkat(2)`
+2. `rmdir(2)`
+2. `unlink(2)`
+2. `unlinkat(2)`
+2. `link(2)`
+2. `fsync(2)`
+2. `dup(2)`
+2. `dup2(2)`
+2. `symlink(2)`
+2. `chdir(2)`
+2. `fchdir(2)`
+2. `getcwd(2)`
+2. `rename(2)`※
+2. `mount(2)`
+2. `unmount` (a.k.a `umount(2)`)
+2. `getxattr(2)`※
+2. `setxattr(2)`※
+2. `fgetxattr(2)`※
+2. `fsetxattr(2)`※
+2. `ioctl(2)`
+2. `getpgid(2)`
+2. `setpgid(2)`
+2. `mmap(2)`
+2. `munmap(2)`
+2. `execve(2)`
+3. `poll(2)` (via `pollctl`)
+3. `epoll_create(2)` (via `pollctl`)
+3. `epoll_ctl(2)` (via `pollctl`)
+3. `epoll_wait(2)` (via `pollctl`)
+4. `pthread_create`
+4. `pthread_self`
+4. `pthread_exit`
+4. `pthread_join`
+4. `pthread_kill`
+4. `pthread_detach`
+4. `pthread_sigmask`
+4. `getuid`
+4. `getgid`
+4. `geteuid`
+4. `getegid`
+4. `getgroups`
+4. `setuid`
+4. `setgid`
+4. `seteuid`
+4. `setegid`
+4. `setgroups`
+4. `chroot`
+4. `fchmodat`
+4. `fchownat`
+4. `faccessat`
+
+**LunaixOS**
+
+1. `yield`
+2. `geterrno`
+3. `realpathat`
+
+( **※**：Indicate syscall is not tested )
+
+## Appendix 2: Debugging with GDB remotely via UART
+
+**(((( Broken after a refactoring years ago, need rework ))))**
+
+The LunaixOS kernel comes with a built-in GDB debugging server, which runs on COM1@9600Bd. However, LunaixOS must be in debug mode before involving GDB.
+
+One could trigger the debug mode by writing a byte sequence `0x40` `0x63` `0x6D` `0x63`, to the same serial port. A text "DEBUG MODE" with magenta-coloured background shall be present at the bottom of the screen.
+
+Note that, whenever the text appears, the LunaixOS always halt all activities other than the debugging server, which means no scheduling and no external interrupt servicing. Users are now recommended to attach their GDB and drive the kernel with the debugging workflow.
+
+Currently, LunaixOS implements the required minimal server-side command subset required by GDB Remote Protocol, namely, `g`, `G`, `p`, `P`, `Q`, `S`, `k`, `?`, `m`, `M`, `X`. Which should be enough to cover most debugging activities.
+
+When debugging is finished, one shall disconnect with `kill` command. This command will not force LunaixOS to power down the computer, instead it just resume the execution (identical behavior as `c` command). However, disconnecting does not means exiting of debug mode. The debug mode is still actived and any subsequent GDB attaching request shall remain the highest priority amongst all other activity. One shall deactivate the debug mode by writing byte sequence `0x40` `0x79` `0x61` `0x79` to the port, after GDB detached.

 
 

-### LunaixOS自有
+### Limitations

 
 

-1. `yield`
\ No newline at end of file
+Currently, one should avoid the use of `info stack`, `bt` or any other command that involves stack unwinding or stack backtracing. As it will somehow corrupt the stack layout and result in undefined behaviour. This issue should be addressed in future releases.