X-Git-Url: https://scm.lunaixsky.com/lunaix-os.git/blobdiff_plain/f8e5ab50164a93a90ffa3d6d59a140547f810e90..1171754ec28161647e916c3339f8da3d1804e1be:/README.md diff --git a/README.md b/README.md index d2a6c09..8135165 100644 --- a/README.md +++ b/README.md @@ -2,86 +2,439 @@

-# LunaixOS Project -LunaixOS - 一个简单的，详细的，POSIX兼容的（但愿！），带有浓重个人风格的操作系统。开发过程以视频教程形式在Bilibili呈现：[《从零开始自制操作系统系列》](https://space.bilibili.com/12995787/channel/collectiondetail?sid=196337)。 +

+ 简体中文 | English +

+ +# The Lunaix Project + +The Lunaix kernel (or soon-to-be LunaixOS) is a hobby kernel, started in 2022, +written entirely from scratch. With a big ambition of being POSIX-compliance, +and designed with performance and modernity in mind, with some independent +thoughts being applied in architectural design. + + +This project is built entirely from first principles - meaning no code was +copy-pasted or recycled from other OS projects or tutorials. Lunaix didn't begin +as an attempt to outperform anything; it started as a personal challenge to +learn OS theory and explore advanced kernel features through original design +and implementation. Everything you see here was developed solo, part-time, +with only hardware specs and my beloved **Modern Operating Systems** as guidance. + +If you're a kernel hobbyist looking for a fresh take or tired of mass-produced, tutorial-style +projects, then welcome, you're in the right place! + + +| ![showcase_lunaix-over-serial.png](docs/img/showcase_lunaix-over-serial.png) | +|:--:| +| *Lunaix over serial* (`console=/dev/ttyS0`) | + +| ![showcase_lunaix-over-serial.png](docs/img/showcase_lunaix-over-vga-text.png) | +|:--:| +| *Lunaix over VGA text mode* (`console=/dev/ttyVCON0`) | + +> By the way, do you know there is an online video course by the author on the design of lunaix? [Check it out](https://space.bilibili.com/12995787/channel/collectiondetail?sid=196337) (although it is in Chinese!) + +## Features + +Lunaix is a multi-architecture, general-purpose kernel built with performance, +modularity, and robustness in mind. Its design emphasizes advanced abstractions, +proactive error detection, and subsystem isolation. Lunaix is + ++ **Fully-preemptive** for responsive multi-tasking ++ **Modular** with compile-time configurable components and extensible + subsystems ++ **High-performance**, leveraging modern caching strategies and efficient + infrastructures. ++ **Fault-tolerant**, with built-in error handling and stack backtracing ++ **Robust by design**, using mechanisms like proactive deadlock detection + and driver isolation + +A significant amount of effort has gone into crafting clean abstractions, +implementing advanced kernel features, and applying performance optimizations +throughout the system. + +To better illustrate the scope of work already done, the following non-exhaustive +list outlines currently supported features in Lunaix: + +### List of All Features +
+ +Click to Expand + ++ Multi-ISA + + x86_32 + + x86_64 + + Aarch64 (WIP) ++ Boot protocol + + abstraction for different protocol + + configurable kernel boot-time parameters ++ Platform resource management and definition + + read-only ACPI table interpretation + + full devicetree implementation ++ Memory management + + architecture-neutral abstraction + + highmem + + copy-on-write + + page sharing + + explicit huge page + + on-demand paging + + compound page + + reverse mapping (rmap) + + memory compaction (WIP) + + slab-style object allocator + + inter-process address space access ++ Multi-tasking + + fully preemptive + + protection levels + + process isolation + + native threading + + signal mechanism + + round-robin scheduler (for now) + + kernel level multi-tasking (i.e. kernel threads) + + taskfs: file system interface to process and threads ++ File system + + POSIX-compliant interface + + virtual file system framework + + file system mounting mechanism + + page cache for file IO + + inode/dnode caching + + ext2 (rev.0, rev.1) + + iso9660 (rock-ridge) + + twifs: kernel state fs interface. ++ Device management and interrupt handling + + unified IRQ framework + + unified driver framework for heterogenous devices + + modular driver model allow compiled-time toggling + + asynchronous operation model supported + + devfs: device fs interface. ++ Block I/O (blkio) + + unified block IO interface + + IO request packets caching + + asynchronous IO operation ++ Serial I/O + + POSIX-compliant serial IO model ++ Caching Infrastructure + + LRU replacement policy and pooling + + kernel daemon for dynamic and transparent cache managements ++ Error handling and detection + + stack backtracing with symbol resolution + + stack unwinding for nested exception + + CPU state dumping + + deadlock/hung-up detection + +
+ +### List of Built-in Drivers + +
+ +Click to Expand + ++ Architecture Neutral + + UART 16650-compatible driver + + Serial ATA AHCI + + PCI 3.0 + + PCIe 1.1 + + Standard VGA ++ Intel x86 + + RTC (Intel PCH) + + IOAPIC IRQ controller + + APIC Timer + + Legacy i8042 keyboard controller ++ ARM + + GICv3 + + PL011 (WIP) +
+ +## Documentations + +There are documentations avaliable if you want to dig deeper. Keep in mind they are still work in progress and missing localisation. + ++ [Luna's Tour - Into Lunaix's Inner Workings](docs/lunaix-internal.md) ++ [Syscall Number Assignments](docs/lunaix-syscall-table.md) + +## Compile and Build + +Building lunaix is simple, no more bloated dependencies to install, basic +`build-essentials` installation plus a python are sufficient. + ++ gcc (recommend v12+) ++ binutils ++ make ++ python (recommend v3.11+) + +And also one should have environment variable `ARCH=` exported, where +`` is one of the supported arhcitecture (`x86_32`, `x86_64`, `aarch64`). + +For cross compilation, also export `CX_PREFIX` to the gcc prefix for the +corresponding ``. + +The following `make` actions are then available to use. + +| Make command | Usage | +| ---- | ---- | +| `make all` | Build the kernel bin | +| `make rootfs` | Build the stock rootfs | +| `make clean` | clean | +| `make config` | run configuration tool | +| `make reconfig` | revert to default and restart configuration | + +A successful build will give `build/bin/kernel.bin`. + +> Please note: this is the kernel, not a bootable image, it require a bootloader +> to boot and specify the rootfs. + +## Get Started + +If you are impatient, or just want something to run and don't want to went +through tedious process of configuring rootfs and tweak kernel parameters. You +can use the `live_debug.sh` provided in the lunaix root directory to quickly +bring up the system with default parameters (also used by the author for +debugging). + +### Quick Start + +This will get you up and running real quick. We will use `x86_64` as example. + +Assuming a Linux or other Unix-like shell environment. We also expect you to be able to sort out trivial issues like missing packages. + +**Select your target** +```sh +$ export ARCH=x86_64 +``` + +**Check Python** +```sh +$ python --version +``` + +Ensure at least `3.11` + +**Check compiler** +```sh +$ gcc -dumpmachine +``` + +Ensure `x86_64-linux-gnu` or anything resemble `x86_64` + +**Check QEMU** +```sh +$ which qemu-system-x86_64 +``` -## 当前进度以及支持的功能 +Should display a valid installation path -该操作系统支持x86架构，运行在保护模式中，采用宏内核架构，目前仅支持单核心。内存结构采用经典的3:1划分，即低3GiB为用户地址空间（0x400000 ~ 0xBFFFFFFF），内核地址空间重映射至高1GiB（0xC0000000 ~ 0xFFFFFFFF）。内存的详细布局可参考[LunaixOS内存地图](docs/img/lunaix-os-mem.png) +**Optional: Setting up Cross-Compiler** -在下述列表中，则列出目前所支持的所用功能和特性。列表项按照项目时间戳进行升序排列。 +```sh +$ export CX_PREFIX=x86_64-linux-gnu- +``` -+ 使用Multiboot进行引导启动 -+ APIC/IOAPIC作为中断管理器和计时器 -+ ACPI -+ 虚拟内存 -+ 内存管理与按需分页（Demand Paging） -+ 键盘输入 -+ 多进程 -+ 14个常见的Linux/POSIX系统调用（[附录1](#appendix1)） -+ 用户模式 -+ 信号机制（进行中） +**Run Configuration** +```sh +make config +``` -## 目录结构 +Then hitting `q` in the interactive shell to accept default setting. -| | | -|-----|------| -| [lunaix-os](lunaix-os/) | LunaixOS源代码 | -| [slides](slides/) | 视频中所用的幻灯片和补充材料 | -| [reference-material](reference-material/)| 标准，技术文档和参考文献 | +**Build stock rootfs** -## 参考教程 +```sh +make rootfs +``` -**没有！！** 本教程以及该操作系统均为原创，没有基于任何市面上现行的操作系统开发教程，且并非是基于任何的开源内核的二次开发。 +Note, this requires root for creating fs image. You can check script `lunaix-os/scripts/mkrootfs` if you feel uncertain. -为了制作LunaixOS，作者耗费大量时间和精力钻研技术文档，手册，理论书籍以及现行工业标准，从而尽量保证了知识的一手性。（这样一来，读者和听众们也算是拿到了二手的知识，而不是三手，四手，甚至n手的知识）。 +**Build & Run** +```sh +./live_debug.sh +``` -大部分的文档和标准可以在上述的[reference-material](reference-material/)中找到。 +you should see gdb now take control of your shell -当然，您也可以参考以下列表来了解现阶段的LunaixOS都使用了哪些资料（本列表会随着开发进度更新）： +**Connect to serial via telnet** -#### 手册，标准，技术文档 -+ [Intel 64 and IA-32 Architecture Software Developer's Manual (Full Volume Bundle)](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html) -+ [ACPI Specification (version 6.4)](https://uefi.org/sites/default/files/resources/ACPI_Spec_6_4_Jan22.pdf) +Open up another window or session +```sh +telnet localhost 12345 +``` + +**Commence simulation** + +Back to the gdb session and type `c` to countine + + +**Watch Lunaix booting!** + +Congrats, enjoy your lunaix! (or submit an issue) + + +### Not so Quick Start + +Here is a slower and yet more verbose steps: + +1. Select an architecture `` +2. Check the compilation prerequisites and presence of `qemu-system-` +3. Optionally export `CX_PREFIX` if you are building for another architecture. +4. Run `make ARCH= rootfs` to build stock rootfs image, require support + of `dd`，`mkfs.ext2`, `mount -o loop`, `mktemp`. +5. Run `ARCH= live_debug.sh` to boot in QEMU with gdb hooked (one should + see a gdb session) +6. telnet to `localhost:12345`, this is QEMU emulated serial port +7. type `c` in the active gdb session and commence emulation. +8. Congrats, enjoy your lunaix! +(or submit an issue) + + +## Booting the kernel + +Since lunaix is a kernel, much like linux. It requires additional setup to do +the magic. And, as in "much like linux", methods to make linux kernel boot can +also apply to lunaix without or with little translation, as we will discuss +below. + +The bootloader part is generic, any bootloader, for example GRUB will work (not +tested for UEFI, but I expect this would be an exception), or booting up in QEMU +using `-kernel` option + +The kernel command line, is however, a bit differentiated. +The syntax is similar, both takes form of space-separated array of `=` +pairs or boolean ``. + +Currently, lunaix support the following options + +| Option | Default Value | Optional | Usage | +| ------ | ---- | ----- | ---- | +| console | `/dev/ttyS0` | No | Specify the system console device, path within lunaix's devfs | +| rootfs | `/dev/block/sda` | No | Specify the device contain rootfs image, path within lunaix's devfs | +| init | `/init` | Yes | Path within rootfs of the `init` | + + +## Submit an Issue + +If one ran into bug, one can submit an issue by filling up the following template + +``` +1. Describe the problem + "How does it look like, anything descriptive: visual, sonic, emotional experience" + +2. Steps to reproduce + "How you ran into this mess?" + +3. Expected behaviour + "What do you intended/expected to achieve/to be" + +4. Lunaix's panic trace (if applicable) + +5. Other clues that you think might be helpful +``` + + +## Limitations + +The development process is still in motion, any limitation can be categorised as +a feature yet to be. However, some features that the author considered to be the +most urgent and wish the matters to be discussed. + +Lunaix is under impression of uniprocessor and not capable of running in SMP +environment. This is major held back of being a modern operating system. It has +the highest priority among all other tasks + +Lunaix do not have a mature (or even, an infant) user space ecosystem, mainly +because the lack of a proper and sophisticated libc. Efforts need to be done for +porting one to the target. However, given the author's tight schedule, this task +is unfortunately still beyond the horizon. + +## Acknowledgement + +Albeit one must realise that the author has mentioned it in the very beginning, +the author would like to emphaise **again** on the nature of this project. + +As a personal challenge, this project is independently developed by the author +single-handly, which means: + ++ No reference to existing tutorials, books, online courses or any open source + project that might provide any example, hint or working prototype on the + design and implementation of kernel, subsystems or anythings that can be + contributed towards a working prototype. ++ The author has no prior knowledge on Linux kernel through out 90% of the + project time. ++ All knowledge on the kernel design is coming from the basic textbook on + operating system theory, that is, *Modern Operating System* by Tanenbaum. ++ All knowledge on the system programming is coming from the basic textbook, + that is, *Computer System - A Programmer's Perspective Third Edition* ++ All knowledge on the generic framework design and driver development are + ingested from various technical specifications gathered across the Internet. + +## References + ++ Intel 64 and IA-32 Architecture Software Developer's Manual (Full Volume Bundle) ++ ACPI Specification (version 6.4) ++ Devicetree Specification ++ ARM® Generic Interrupt Controller (v3) ++ Arm® Architecture Reference Manual (Profile-A) ++ Procedure Call Standard for the Arm® 64-bit Architecture (AArch64) + IBM PC/AT Technical Reference + IBM VGA/XGA Technical Reference + 82093AA I/O Advanced Programmable Controller (IOAPIC) (Datasheet) + MC146818A (Datasheet) + Intel 500 Series Chipset Family Platform Controller Hub (Datasheet - Volume 2) - -#### 理论书籍 -+ *Computer System - A Programmer's Perspective Third Edition (CS:APP)* (Bryant, R & O'Hallaron, D) ++ PCI Local Bus Specification, Revision 3.0 ++ PCI Express Base Specification, Revision 1.1 ++ PCI Firmware Specification, Revision 3.0 ++ Serial ATA - Advanced Host Controller Interface (AHCI), Revision 1.3.1 ++ Serial ATA: High Speed Serialized AT Attachment, Revision 3.2 ++ SCSI Command Reference Manual ++ ATA/ATAPI Command Set - 3 (ACS-3) ++ ECMA-119 (ISO9660) ++ Rock Ridge Interchange Protocol (RRIP: IEEE P1282) ++ System Use Sharing Protocol (SUSP: IEEE P1281) ++ Tool Interface Standard (TIS) Portable Formats Specification (Version 1.1) ++ *Computer System - A Programmer's Perspective Third Edition* (Bryant, R & O'Hallaron, D), a.k.a. CS:APP + *Modern Operating System* (Tanenbaum, A) -+ 《汇编语言》（王爽） - 用于入门Intel语法的x86汇编（对于AT&T语法，推荐阅读CS:APP） -+ ~~《微机原理与接口技术》 - 用于大致了解x86架构的微机体系（更加细致的了解可以阅读Intel Manual）~~ （已过时，推荐阅读CS:APP） - -#### 网站 -+ [OSDev](https://wiki.osdev.org/Main_Page) - 杂七杂八的参考，很多过来人的经验。作者主要用于上古资料查询以及收集；技术文献，手册，标准的粗略总结；以及开发环境/工具链的搭建。 -+ [FreeVGA](http://www.osdever.net/FreeVGA/home.htm) - 98年的资源！关于VGA编程技术的宝藏网站。 -+ GNU CC 和 GNU LD 的官方文档。 - -#### 其他 -+ Linux Manual - 用于查询*nix API的一些具体行为。 - - -## 附录1：支持的系统调用 - -### Unix/Linux/POSIX -1. `sleep(2)` -1. `wait(2)` -1. `waitpid(2)` -1. `fork(2)` -1. `getpid(2)` -1. `getppid(2)` -1. `getpgid(2)` -1. `setpgid(2)` -1. `brk(2)` -1. `sbrk(2)` -1. `_exit(2)` -1. `sigreturn(2)` -1. `sigprocmask(2)` -1. `signal(2) ` - -### LunaixOS自有 - -1. `yield` ++ Free VGA, http://www.osdever.net/FreeVGA/home.htm ++ GNU CC & LD online documentation. ++ PCI Lookup, https://www.pcilookup.com/ ++ Linux man pages + +## Appendix 1: Supported System Call + +Refer to [Lunaix Syscall Table](docs/lunaix-syscall-table.md) + +## Appendix 2: Debugging with GDB remotely via UART + +**(((( Broken after a refactoring years ago, need rework ))))** + +
+ Click to expand +The LunaixOS kernel comes with a built-in GDB debugging server, which runs on +COM1@9600Bd. However, LunaixOS must be in debug mode before involving GDB. + +One could trigger the debug mode by writing a byte sequence `0x40` `0x63` `0x6D` +`0x63`, to the same serial port. A text "DEBUG MODE" with magenta-coloured background shall be present at the bottom of the screen. + +Note that, whenever the text appears, the LunaixOS always halt all activities +other than the debugging server, which means no scheduling and no external +interrupt servicing. Users are now recommended to attach their GDB and drive +the kernel with the debugging workflow. + +Currently, LunaixOS implements the required minimal server-side command subset +required by GDB Remote Protocol, namely, `g`, `G`, `p`, `P`, `Q`, `S`, `k`, `?`, +`m`, `M`, `X`. Which should be enough to cover most debugging activities. + +When debugging is finished, one shall disconnect with `kill` command. This +command will not force LunaixOS to power down the computer, instead it just +resume the execution (identical behavior as `c` command). However, disconnecting +does not means exiting of debug mode. The debug mode is still actived and any +subsequent GDB attaching request shall remain the highest priority amongst all +other activity. One shall deactivate the debug mode by writing byte sequence +`0x40` `0x79` `0x61` `0x79` to the port, after GDB detached. + +### Limitations + +Currently, one should avoid the use of `info stack`, `bt` or any other command +that involves stack unwinding or stack backtracing. As it will somehow corrupt +the stack layout and result in undefined behaviour. This issue should be +addressed in future releases. +
\ No newline at end of file