X-Git-Url: https://scm.lunaixsky.com/lunaix-os.git/blobdiff_plain/8c6f505faaa66e18cdca108dca549d4ad806a077..7909fc6dec8d1499c5b9fa331a810a77b08c84a2:/README.md diff --git a/README.md b/README.md index b8111ec..310fd75 100644 --- a/README.md +++ b/README.md @@ -3,140 +3,210 @@

- 简体中文 | English + 简体中文 | English

-# LunaixOS Project - -LunaixOS - 一个简单的，详细的，POSIX兼容的（但愿！），带有浓重个人风格的操作系统。开发过程以视频教程形式在Bilibili呈现：[《从零开始自制操作系统系列》](https://space.bilibili.com/12995787/channel/collectiondetail?sid=196337)。 - -## 当前进度以及支持的功能 - -该操作系统支持x86架构，运行在保护模式中，采用宏内核架构，目前仅支持单核心。内存结构采用经典的3:1划分，即低3GiB为用户地址空间（0x400000 ~ 0xBFFFFFFF），内核地址空间重映射至高1GiB（0xC0000000 ~ 0xFFFFFFFF）。内存的详细布局可参考[LunaixOS内存地图](docs/img/lunaix-os-mem.png) - -在下述列表中，则列出目前所支持的所用功能和特性。列表项按照项目时间戳进行升序排列。 - -+ 使用Multiboot进行引导启动 -+ APIC/IOAPIC作为中断管理器和计时器 -+ ACPI -+ 虚拟内存 -+ 内存管理与按需分页（Demand Paging） -+ 键盘输入 -+ 多进程 -+ 50个常见的Linux/POSIX系统调用（[附录1](#appendix1)） -+ 用户模式 -+ 信号机制 -+ PCI 3.0 -+ PCIe 1.1 (WIP) -+ Serial ATA AHCI -+ 文件系统 - + 虚拟文件系统 - + ISO9660 - + 原生 - + Rock Ridge拓展 (WIP) -+ 远程GDB串口调试 (COM1@9600Bd) - -已经测试过的环境： - -+ QEMU (>=7.0.0) -+ Bochs（SATA功能不支持） -+ Virtualbox -+ Dell G3 3779 - -## 目录结构 - -| | | -| ----------------------------------------- | ---------------------------- | -| [lunaix-os](lunaix-os/) | LunaixOS源代码 | -| [slides](slides/) | 视频中所用的幻灯片和补充材料 | -| [reference-material](reference-material/) | 标准，技术文档和参考文献 | - -## 编译与构建 - -### 环境搭建 - -构建该项目需要满足以下条件： - -+ gcc (目标平台: i686-elf) +# The Lunaix Project + +The Lunaix, LunaixOS, or Lunaix kernel to be exact, is a multi-architectural general purpose kernel written from scratch by the author *independently*. And it is the author's years-long personal endeavor and also a challenge to oneself for writing a functioning kernel **without** any external reference such as existing implementation, tutorial and books (with code) on kernel design. + +The overall design is aimed to be modern and POSIX-compliance. In a nutshell, Lunaix is: + ++ fully-preemptive ++ modular design with configurable components at compile-time and extendable subsystems ++ high-performance by utilising advanced caching techniques and infrastructure. ++ fault-tolerance with sophisticated builtin error handling and tracing techniques. ++ robust in nature with techniques such as proactive deadlock detection and driver isolation mechanism. + +To give a better understanding (and appreciation) of the works, the following list has been compiled with all feature that lunaix currently supported: + ++ Multi-ISA + + x86_32 + + x86_64 + + Aarch64 (W.I.P) ++ Boot protocol abstraction ++ Platform resource management and definition + + ACPI + + Devicetree ++ Memory management + + multi-architecture abstraction + + on-demand paging + + copy-on-write and page sharing + + compound page support + + explicit huge page support (sorry, not THP!) + + reverse mapping indexing (rmap) + + memory compaction (W.I.P) + + slab-alike object allocator ++ Multi-threadingg and multi-tasking + + Protection level and memory space isolation + + Native threading support (no more lightweight process nonsense) + + Signal handling + + Kernel-level multi-tasking (i.e. kernel threads) + + Round-robin scheduling (for now) + + Fully-preemptive kernel design + + taskfs: file system interface to process and threads ++ File system + + virtual file system framework + + ...with POSIX compliant file system interface + + mountable file system + + page cache for file IO + + node cache for vfs structure representation. + + ext2 (rev.0, rev.1) + + iso9660 (rock-ridge) + + twifs: file system interface to kernel states. ++ Device management and interrupt handling + + architectural decoupled design + + generalised driver framework + + generalised irq framework + + driver modularisation design + + support asynchronous device model + + devfs: file system interface to device subsystem ++ Block I/O (blkio) + + generalised block IO interface and encapsulation + + blkio request caching + + asynchronous blkio operation in nature ++ Serial I/O + + POSIX-compliant serial port model + + serial device driver framework (part of driver framework) ++ Caching Infrastructure + + primtive: generic sparse associative array + + LRU replacement policy and pooling + + kernel daemon for periodical cache eviction ++ Error handling and detection + + stack back-tracing with symbol resolution + + nested exception unfolding + + CPU state dumping + + Deadlock/hung-up detection + +For the device driver that is currently support see below: + ++ Arhcitecture Neutral + + UART 16650-compatible driver + + Serial ATA AHCI + + PCI 3.0 + + PCIe 1.1 + + Standard VGA ++ Intel x86 + + RTC (Intel PCH) + + IOAPIC irq controller + + APIC Timer + + Legacy i8042 keyboard controller ++ ARM + + GICv3 + + PL011 (W.I.P) + +By the way, do you know there is an online video course by the author on the design of lunaix? [Check it out](https://space.bilibili.com/12995787/channel/collectiondetail?sid=196337) (although it is in Chinese!) + +## Project Structure + +| | | +|-----|------| +| [lunaix-os](../lunaix-os/) | LunaixOS source code | +| [slides](../slides/) | Slides used in my videos | +| [reference-material](../reference-material/)| References | + +## Compile and Build + +You will need following dependencies in order to build Lunaix + ++ gcc (recommend v12+) + binutils + make -+ xorriso -+ grub-mkrescue ++ python (recommend v3.11+) -**注意：gcc不能是本机自带的，必须要从源码编译，并配置目标平台为：`i686-elf`，以进行交叉编译。配置过程可参考[附录二：编译gcc作为交叉编译器](#appendix2)。** +And also one should have environment variable `ARCH=` exported, where `` is one of the supported arhcitecture (`x86_32`, `x86_64`, `arm64`) -### Docker镜像 +The following `make` actions are available to use. -对于开发环境，本项目也提供了Docker镜像封装。开箱即用，无需配置，非常适合懒人或惜时者。详细使用方法请转到：[Lunaix OSDK项目](https://github.com/Minep/os-devkit)。 +| Make command | Usage | +| ---- | ---- | +| `make all` | Build the kernel bin | +| `make rootfs` | Build the stock rootfs | +| `make clean` | clean | +| `make config` | menuconfig | -### 构建选项 +A successful build will give `build/bin/kernel.bin`. -假若条件满足，那么可以直接执行`make all`进行构建，完成后可在生成的`build`目录下找到可引导的iso。 +**Please note: this is the kernel, not a bootable image, it require a bootloader to boot and specify the rootfs.** -本项目支持的make命令： -| 命令 | 用途 | -| ------------------------ | ----------------------------------------------- | -| `make all` | 构建镜像（`-O2`，但禁用CSE相关的优化项 **※** ） | -| `make instable` | 构建镜像（`-O2`，开启CSE相关优化） | -| `make all-debug` | 构建适合调试用的镜像（`-Og`） | -| `make run` | 使用QEMU运行build目录下的镜像 | -| `make debug-qemu` | 构建并使用QEMU进行调试 | -| `make debug-bochs` | 构建并使用Bochs进行调试 | -| `make debug-qemu-vscode` | 用于vscode整合 | -| `make clean` | 删除build目录 | +## Booting the kernel -**※：由于在`-O2`模式下，GCC会进行CSE优化，这导致LunaixOS会出现一些非常奇怪、离谱的bug，从而影响到基本运行。具体原因有待调查。** +Since lunaix is a kernel, much like linux. It require additional setup to make the magic. And, as it is "much like linux", methods to make linux kernel boot can also apply to lunaix without or with little translation, as we will discuss below. -## 运行以及Issue +The bootloader part is generic, any bootloader, for example GRUB will work (not tested for UEFI, but I expect this would be an exception), or booting up in QEMU using `-kernel` option -运行该操作系统需要一个虚拟磁盘镜像，可以使用如下命令快速创建一个： +The kernel command line, is however, a bit differentiated. +The syntax is similar, both takes form of space-separated array of `=` pairs or boolean ``. -```bash -qemu-img create -f vdi machine/disk0.vdi 128M -``` +Currently, lunaix support the following options -如果你想要使用别的磁盘镜像，需要修改`configs/make-debug-tool` +| Option | Default Value | Optional | Usage | +| ------ | ---- | ----- | ---- | +| console | `/dev/ttyS0` | No | Specify the system console device, path within lunaix's devfs | +| rootfs | `/dev/block/sda` | No | Specify the device contain rootfs image, path within lunaix's devfs | +| init | `/init` | Yes | Path within rootfs of the `init` | -找到这一行： +### A quick 'Get Started' -``` --drive id=disk,file="machine/disk0.vdi",if=none \ -``` +One can use the `live_debug.sh` provided in the lunaix root directory to quickly bring up the system with default parameter (which is used by the author for debugging). -然后把`machine/disk0.vdi`替换成你的磁盘路径。 +Following the steps: -有很多办法去创建一个虚拟磁盘，比如[qemu-img](https://qemu-project.gitlab.io/qemu/system/images.html)。 +1. Select a architecture `` +2. Check the compilation prerequisites and presence of `qemu-system-` +3. Run `make ARCH= user` to build the stock user program +4. Run `make ARCH= rootfs` to build stock rootfs image, require support of `dd`，`mkfs.ext2`, `mount -o loop`, `mktemp`. +5. Run `ARCH= live_debug.sh` to boot in QEMU with gdb hooked (one should see a gdb session) +6. telenet to `localhost:12345`, this is QEMU emulated serial port +7. type `c` in the active gdb session and commence emualtion. +8. Congrats, enjoy your lunaix! +(or submit an issue) -在大多数情况下，我都会尽量保证本机运行无误后，push到仓库中。同时，该系统是经过虚拟机和真机测试。如果发现在使用`make all`之后，虚拟机中运行报错，则一般是编译器优化问题。这个问题笔者一般很快就会修复，如果你使用别的版本的gcc（笔者版本11.2），出现了此问题，欢迎提issue。请参考[附录3：Issue的提交](#appendix3) -下面列出一些可能会出现的问题。 +## Submit an Issue -#### 问题#1： QEMU下8042控制器提示找不到 +If one ran into bug, one can submit an issue by filling up the following template -这是QEMU配置ACPI时的一个bug，在7.0.0版中修复了。 +``` +1. Describe the problem + "How does it look like, anything descriptive: visual, sonic, emotional experience" +2. Steps to reproduce + "How you ran into this mess?" +3. Expected behaviour + "What do you intended/expected to achieve/to be" +4. Lunaix's panic trace (if applicable) +5. Other clues that you think might be helpful +``` -#### 问题#2：多进程运行时，偶尔会出现General Protection错误 -这很大概率是出现了竞态条件。虽然是相当不可能的。但如果出现了，还是请提issue。 +## Limitations -#### 问题#3：Bochs无法运行，提示找不到AHCI控制器 +The development process is still in motion, any limitation can be categorised as yet to be finished feature. However, some feature that the author considered to be the most urgent and wish the matters to be discussed. -正常，**因为Bochs不支持SATA**。请使用QEMU或VirtualBox。 +Lunaix is under assumption of uniprocessor and not capable of running in SMP environment. This is major held back of being a modern operating system. It has the highest priority among all other tasks -## 参考教程 +Lunaix is do not have a mature user space ecosystem, mainly because the lack of a proper and sophisticated libc. Efforts need to be done for porting one to the target. However, given the author's tight schedule, this task is unfortunately still beyond the horizon. -**没有！！** 本教程以及该操作系统均为原创，没有基于任何市面上现行的操作系统开发教程，且并非是基于任何的开源内核的二次开发。 +## Acknowledgement -为了制作LunaixOS，作者耗费大量时间和精力钻研技术文档，手册，理论书籍以及现行工业标准，从而尽量保证了知识的一手性。（这样一来，读者和听众们也算是拿到了二手的知识，而不是三手，四手，甚至n手的知识）。 +Albeit one must realise that the author has mentioned it in the very beginning, the author would like to emphaise **again** on the nature of this project. -大部分的文档和标准可以在上述的[reference-material](reference-material/)中找到。 +As a personal challenge, this project is independently developed by the author single-handly, which means: -当然，您也可以参考以下列表来了解现阶段的LunaixOS都使用了哪些资料（本列表会随着开发进度更新）： ++ No reference to existing tutorial, books, online course or any open source project that might provide example, hint or working prototype on the design and implementation of kernel, it's various subsystems or anythings that can be contributed towards a working prototype. ++ The author has no prior knowledge on Linux kernel through out 90% of the project time. ++ All knowledge on the kernel design is comming from the basic textbook on operating system theory, that is, *Modern Operating System* by Tanenbaum. ++ All knowledge on the system programming is commingg from the basic textbook, that is, *Computer System - A Programmer's Perspective Third Edition* ++ All knowledge on the generic framework design and driver development are ingested from various technical specifications gathered across internet. -#### 手册，标准，技术文档 +## References -+ [Intel 64 and IA-32 Architecture Software Developer's Manual (Full Volume Bundle)](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html) -+ [ACPI Specification (version 6.4)](https://uefi.org/sites/default/files/resources/ACPI_Spec_6_4_Jan22.pdf) ++ Intel 64 and IA-32 Architecture Software Developer's Manual (Full Volume Bundle) ++ ACPI Specification (version 6.4) ++ Devicetree Specification ++ ARM® Generic Interrupt Controller (v3) ++ Arm® Architecture Reference Manual (Profile-A) ++ Procedure Call Standard for the Arm® 64-bit Architecture (AArch64) + IBM PC/AT Technical Reference + IBM VGA/XGA Technical Reference + 82093AA I/O Advanced Programmable Controller (IOAPIC) (Datasheet) @@ -149,32 +219,18 @@ qemu-img create -f vdi machine/disk0.vdi 128M + Serial ATA: High Speed Serialized AT Attachment, Revision 3.2 + SCSI Command Reference Manual + ATA/ATAPI Command Set - 3 (ACS-3) -+ [ECMA-119 (ISO9660)](https://www.ecma-international.org/publications-and-standards/standards/ecma-119/) ++ ECMA-119 (ISO9660) + Rock Ridge Interchange Protocol (RRIP: IEEE P1282) + System Use Sharing Protocol (SUSP: IEEE P1281) + Tool Interface Standard (TIS) Portable Formats Specification (Version 1.1) - -**免责声明：PCI相关的标准最终解释权归PCI-SIG所有。此处提供的副本仅供个人学习使用。任何商用目的须向PCI-SIG购买。** - -#### 理论书籍 - -+ *Computer System - A Programmer's Perspective Third Edition (CS:APP)* (Bryant, R & O'Hallaron, D) ++ *Computer System - A Programmer's Perspective Third Edition* (Bryant, R & O'Hallaron, D), a.k.a. CS:APP + *Modern Operating System* (Tanenbaum, A) -+ 《汇编语言》（王爽） - 用于入门Intel语法的x86汇编（对于AT&T语法，推荐阅读CS:APP） -+ ~~《微机原理与接口技术》 - 用于大致了解x86架构的微机体系（更加细致的了解可以阅读Intel Manual）~~ （已过时，推荐阅读CS:APP） - -#### 网站 ++ Free VGA, http://www.osdever.net/FreeVGA/home.htm ++ GNU CC & LD online documentation. ++ PCI Lookup, https://www.pcilookup.com/ ++ Linux man pages -+ [OSDev](https://wiki.osdev.org/Main_Page) - 杂七杂八的参考，很多过来人的经验。作者主要用于上古资料查询以及收集；技术文献，手册，标准的粗略总结；以及开发环境/工具链的搭建。 -+ [FreeVGA](http://www.osdever.net/FreeVGA/home.htm) - 98年的资源！关于VGA编程技术的宝藏网站。 -+ GNU CC 和 GNU LD 的官方文档。 -+ [PCI Lookup](https://www.pcilookup.com/) - PCI设备编号查询 - -#### 其他 - -+ Linux Manual - 用于查询*nix API的一些具体行为。 - -## 附录1：支持的系统调用 +## Appendix 1: Supported System Call **Unix/Linux/POSIX** @@ -228,117 +284,40 @@ qemu-img create -f vdi machine/disk0.vdi 128M 2. `mmap(2)` 2. `munmap(2)` 2. `execve(2)` - -**LunaixOS自有** +3. `poll(2)` (via `pollctl`) +3. `epoll_create(2)` (via `pollctl`) +3. `epoll_ctl(2)` (via `pollctl`) +3. `epoll_wait(2)` (via `pollctl`) +4. `pthread_create` +4. `pthread_self` +4. `pthread_exit` +4. `pthread_join` +4. `pthread_kill` +4. `pthread_detach` +4. `pthread_sigmask` + +**LunaixOS** 1. `yield` 2. `geterrno` 3. `realpathat` -4. `syslog` - -( **※**：该系统调用暂未经过测试 ) - -## 附录2：编译gcc作为交叉编译器 - -注意，gcc需要从源码构建，并配置为交叉编译器，即目标平台为`i686-elf`。你可以使用本项目提供的[自动化脚本](slides/c0-workspace/gcc-build.sh)，这将会涵盖gcc和binutils源码的下载，配置和编译（没什么时间去打磨脚本，目前只知道在笔者的Ubuntu系统上可以运行）。 - -**推荐**手动编译。以下编译步骤搬运自： - -**首先安装构建依赖项：** - -```bash -sudo apt update &&\ - apt install -y \ - build-essential \ - bison\ - flex\ - libgmp3-dev\ - libmpc-dev\ - libmpfr-dev\ - texinfo -``` - -**开始编译：** - -1. 获取[gcc](https://ftp.gnu.org/gnu/gcc/)和[binutils](https://ftp.gnu.org/gnu/binutils)源码 -2. 解压，并在同级目录为gcc和binutil新建专门的build文件夹 - -现在假设你的目录结构如下： - -``` -+ folder - + gcc-src - + binutils-src - + gcc-build - + binutils-build -``` - -3. 确定gcc和binutil安装的位置，并设置环境变量：`export PREFIX=<安装路径>` 然后设置PATH： `export PATH="$PREFIX/bin:$PATH"` -4. 设置目标平台：`export TARGET=i686-elf` -5. 进入`binutils-build`进行配置 - -```bash -../binutils-src/configure --target="$TARGET" --prefix="$PREFIX" \ - --with-sysroot --disable-nls --disable-werror -``` - -然后 `make && make install` - -6. 确保上述的`binutils`已经正常安装：执行：`which i686-elf-as`，应该会给出一个位于你安装目录下的路径。 -6. 进入`gcc-build`进行配置 - -```bash -../gcc-src/configure --target="$TARGET" --prefix="$PREFIX" \ - --disable-nls --enable-languages=c,c++ --without-headers -``` - -然后编译安装（取决性能，大约10~20分钟）： - -```bash -make all-gcc &&\ - make all-target-libgcc &&\ - make install-gcc &&\ - make install-target-libgcc -``` - -8. 验证安装：执行`i686-elf-gcc -dumpmachine`，输出应该为：`i686-elf` - -**将新编译好的GCC永久添加到`PATH`环境变量** - -虽然这是一个常识性的操作，但考虑到许多人都会忽略这一个额外的步骤，在这里特此做出提示。 - -要想实现这一点，只需要在shell的配置文件的末尾添加：`export PATH="<上述的安装路径>/bin:$PATH"`。 - -这个配置文件是取决于你使用的shell，如zsh就是`${HOME}/.zshrc`，bash则是`${HOME}/.bashrc`；或者你嫌麻烦的，懒得区分，你也可以直接修改全局的`/etc/profile`文件，一劳永逸（但不推荐这样做）。 - -至于其他的情况，由于这个步骤其实在网上是随处可查的，所以就不在这里赘述了。 - -## 附录3：Issue的提交 - -由于目前LunaixOS没有一个完善强大的内核追踪功能。假若Lunaix的运行出现任何问题，还请按照以下的描述，在Issue里面提供详细的信息。 -最好提供： +( **※**：Indicate syscall is not tested ) -+ 可用于复现问题的描述和指引（如Lunaix运行平台的软硬件配置） -+ 错误症状描述 -+ （如可能）运行截图 -+ 错误消息（如果给出） -+ 寄存器状态的dump -+ （如可能）提供错误发生时，EIP附近的指令（精确到函数）。如果使用`make all-debug`，会提供`build/kdump.txt`，你可以在这里面定位。或者也可以直接`objdump` -+ （如可能）虚拟内存映射信息（QEMU下可使用`info mem`查看）。 +## Appendix 2: Debugging with GDB remotely via UART -## 附录4：串口GDB远程调试 +**(((( Not working yet, need rework ))))** -LunaixOS内核集成了最基本的GDB远程调试服务器。可通过串口COM1在9600波特率上与之建立链接。但是，在将GDB与内核链接起来之前，还需要让内核处在调试模式下。 +The LunaixOS kernel comes with a built-in GDB debugging server, which runs on COM1@9600Bd. However, LunaixOS must be in debug mode before involving GDB. -要进入调试模式，需要往串口（波特率如上）写入字节串 `0x40` `0x63` `0x6D` `0x63`。此时，如果屏幕底部出现一条品红色背景的`DEBUG` 字样，那么就说明LunaixOS已处在调试模式下。 +One could trigger the debug mode by writing a byte sequence `0x40` `0x63` `0x6D` `0x63`, to the same serial port. A text "DEBUG MODE" with magenta-coloured background shall be present at the bottom of the screen. -注意，在这个时候，LunaixOS会开始在`COM1`上监听GDB协议信息，并且暂停一切的活动（如调度，以及对外部中断的一切响应）。用户此时需要将GDB与其挂载，并使用GDB的工作流来指示内核下一步的动作。 +Note that, whenever the text appears, the LunaixOS always halt all activities other than the debugging server, which means no scheduling and no external interrupt servicing. Users are now recommended to attach their GDB and drive the kernel with the debugging workflow. -在目前，为了防止代码过于臃肿，LunaixOS实现的是GDB远程协议要求的最小服务端命令子集：`g`, `G`, `p`, `P`, `Q`, `S`, `k`, `?`, `m`, `M`, `X`。足以满足大部分的调试需求。 +Currently, LunaixOS implements the required minimal server-side command subset required by GDB Remote Protocol, namely, `g`, `G`, `p`, `P`, `Q`, `S`, `k`, `?`, `m`, `M`, `X`. Which should be enough to cover most debugging activities. -当结束调试的时候，请使用GDB的`kill`指令进行连接的断开。注意，这个指令会使得LunaixOS恢复所有暂停的活动，进入正常的运行序列，但并不会退出调试模式。GDB的挂载请求依然在LunaixOS中享有最高优先权。如果需要退出调试模式，需要往串口写入字节串：`0x40` `0x79` `0x61` `0x79`。 +When debugging is finished, one shall disconnect with `kill` command. This command will not force LunaixOS to power down the computer, instead it just resume the execution (identical behavior as `c` command). However, disconnecting does not means exiting of debug mode. The debug mode is still actived and any subsequent GDB attaching request shall remain the highest priority amongst all other activity. One shall deactivate the debug mode by writing byte sequence `0x40` `0x79` `0x61` `0x79` to the port, after GDB detached. -### GDB调试注意事项 +### Limitations -在调试中，请避免使用`info stack`，`bt`或者任何涉及 **栈展开（Stack Unwinding）** 或者 **栈回溯（Stack Backtracing）** 的指令。否则，LunaixOS很有可能会出现 **不可预料的行为** 。 +Currently, one should avoid the use of `info stack`, `bt` or any other command that involves stack unwinding or stack backtracing. As it will somehow corrupt the stack layout and result in undefined behaviour. This issue should be addressed in future releases.