updated readme, fix configuration shell invoked twice.

[lunaix-os.git] / README.md

diff --git a/README.md b/README.md
index a941d853b4e0dc0ed589c1b4a5eb4eabaa0ab20c..fe73edc437d3c2498af76fa60ec5cb438e0640f3 100644 (file)
--- a/README.md
+++ b/README.md
@@ -8,85 +8,110 @@
 
 # The Lunaix Project
 
 
 # The Lunaix Project
 

-The Lunaix, LunaixOS, or Lunaix kernel to be exact, is a multi-architectural general purpose kernel written from scratch by the author *independently*. It is the author's years-long personal endeavor and also a challenge to oneself for writing a functioning kernel **without** any external reference such as existing implementations, tutorials and books (with code) on kernel design. As a result, this project is a manifestation of the author's own research and understanding on the board topic of operating system design and implementation.
+The Lunaix kernel (or soon-to-be LunaixOS) is a hobby kernel, written entirely from scratch. Designed to be POSIX-compliance, performance critical and modern, with some independent thoughts being applied in architectural design.

 
 

-## Introduction

 
 

-The overall design of lunaix is aimed to be modern and POSIX-compliance. In a nutshell, Lunaix is:
+This project is built entirely from first principles - meaning no code was copy-pasted or recycled from other OS projects or tutorials. Lunaix didn't begin as an attempt to outperform anything; it started as a personal challenge to learn OS theory and explore advanced kernel features through original design and implementation. Everything you see here was developed solo, part-time, with only hardware specs and my beloved **Modern Operating Systems** as guidance.

 
 

-+ fully-preemptive
-+ modular design with configurable components at compile-time and extendable subsystems
-+ high-performance by utilising advanced caching techniques and infrastructure.
-+ fault-tolerance with sophisticated builtin error handling and tracing techniques.
-+ robust in nature with techniques such as proactive deadlock detection and driver isolation mechanism.
+If you're a kernel hobbyist looking for a fresh take or tired of mass-produced projects, then welcome, you're in the right place!

 
 

-The author has put a significant amount of time on devising better abstractions, advance kernel features and various optimisation techniques. To give a better understanding (and appreciation) of the works being done, the following non-exhaust list has been compiled with features that are currently supported in lunaix:
+
+| ![showcase_lunaix-over-serial.png](docs/img/showcase_lunaix-over-serial.png) |
+|:--:|
+| *Lunaix over serial* (`console=/dev/ttyS0`) |
+
+| ![showcase_lunaix-over-serial.png](docs/img/showcase_lunaix-over-vga-text.png) |
+|:--:|
+| *Lunaix over VGA text mode* (`console=/dev/ttyVCON0`) |
+
+> By the way, do you know there is an online video course  by the author on the design of lunaix? [Check it out](https://space.bilibili.com/12995787/channel/collectiondetail?sid=196337) (although it is in Chinese!)
+
+## Features
+
+Lunaix is a multi-architecture, general-purpose kernel built with performance, modularity, and robustness in mind. Its design emphasizes advanced abstractions, proactive error detection, and subsystem isolation. Lunaix is
+
++ **Fully-preemptive** for responsive multi-tasking
++ **Modular** with compile-time configurable components and extensible subsystems
++ **High-performance**, leveraging modern caching strategies and efficient infrastructures.
++ **Fault-tolerant**, with built-in error handling and stack backtracing
++ **Robust by design**, using mechanisms like proactive deadlock detection and driver isolation
+
+A significant amount of effort has gone into crafting clean abstractions, implementing advanced kernel features, and applying performance optimizations throughout the system.
+
+To better illustrate the scope of work already done, the following non-exhaustive list outlines currently supported features in Lunaix:
+
+<details>
+
+<summary>List of all features implemented</summary>

 
 + Multi-ISA
   + x86_32
   + x86_64
 
 + Multi-ISA
   + x86_32
   + x86_64

-  + Aarch64 (W.I.P)
+  + Aarch64 (WIP)

 + Boot protocol
   + abstraction for different protocol
   + configurable kernel boot-time parameters
 + Platform resource management and definition
 + Boot protocol
   + abstraction for different protocol
   + configurable kernel boot-time parameters
 + Platform resource management and definition

-  + ACPI
-  + Devicetree
+  + read-only ACPI table interpretation
+  + full devicetree implementation

 + Memory management
   + architecture-neutral abstraction
 + Memory management
   + architecture-neutral abstraction

+  + highmem
+  + copy-on-write
+  + page sharing
+  + explicit huge page

   + on-demand paging
   + on-demand paging

-  + copy-on-write and page sharing
-  + compound page support
-  + explicit huge page support (sorry, not THP!)
-  + reverse mapping indexing (rmap)
-  + memory compaction (W.I.P)
-  + slab-alike object allocator
-  + highmem support
-  + remote address space accessing
+  + compound page
+  + reverse mapping (rmap)
+  + memory compaction (WIP)
+  + slab-style object allocator
+  + inter-process address space access

 + Multi-tasking
 + Multi-tasking

-  + Protection level and process image isolation
-  + Native threading support (no more lightweight process nonsense)
-  + Signal mechanism
-  + Kernel level multi-tasking (i.e. kernel threads)
-  + Round-robin scheduling (for now)
-  + Preemptive kernel design
+  + fully preemptive
+  + protection levels
+  + process isolation
+  + native threading
+  + signal mechanism
+  + round-robin scheduler (for now)
+  + kernel level multi-tasking (i.e. kernel threads)

   + taskfs: file system interface to process and threads
 + File system
   + taskfs: file system interface to process and threads
 + File system

+  + POSIX-compliant interface

   + virtual file system framework
   + virtual file system framework

-  + ...with POSIX compliant interfaces

   + file system mounting mechanism
   + page cache for file IO
   + file system mounting mechanism
   + page cache for file IO

-  + node cache for vfs structure representation.
+  + inode/dnode caching

   + ext2 (rev.0, rev.1)
   + iso9660 (rock-ridge)
   + ext2 (rev.0, rev.1)
   + iso9660 (rock-ridge)

-  + twifs: file system interface to kernel states.
+  + twifs: kernel state fs interface.

 + Device management and interrupt handling
 + Device management and interrupt handling

-  + architecture-neutral design
-  + generalised driver framework
-  + generalised irq framework
-  + driver modularisation design
-  + support asynchronous device model
-  + devfs: file system interface to device subsystem
+  + unified IRQ framework
+  + unified driver framework for heterogenous devices
+  + modular driver model allow compiled-time toggling
+  + asynchronous operation model supported
+  + devfs: device fs interface.

 + Block I/O (blkio)
 + Block I/O (blkio)

-  + generalised block IO interface and encapsulation
-  + blkio packets caching
-  + asynchronous blkio operation in nature
+  + unified block IO interface
+  + IO request packets caching
+  + asynchronous IO operation

 + Serial I/O
 + Serial I/O

-  + POSIX-compliant serial port model
-  + serial device driver framework (part of driver framework)
+  + POSIX-compliant serial IO model

 + Caching Infrastructure
 + Caching Infrastructure

-  + primitive: generic sparse associative array (spatial data)

   + LRU replacement policy and pooling
   + LRU replacement policy and pooling

-  + kernel daemon for scheduled cache eviction
+  + kernel daemon for dynamic and transparent cache managements

 + Error handling and detection
 + Error handling and detection

-  + stack back-tracing with symbol resolution
-  + nested exception unfolding
+  + stack backtracing with symbol resolution
+  + stack unwinding for nested exception

   + CPU state dumping
   + CPU state dumping

-  + Deadlock/hung-up detection
+  + deadlock/hung-up detection

 
 

-For the device drivers that are currently support see below:
+</details>

 
 

-+ Arhcitecture Neutral
+<details>
+
+<summary>List of currently supported device</summary>
+
++ Architecture Neutral

   + UART 16650-compatible driver
   + Serial ATA AHCI
   + PCI 3.0
   + UART 16650-compatible driver
   + Serial ATA AHCI
   + PCI 3.0


@@ -94,14 +119,13 @@ For the device drivers that are currently support see below:
   + Standard VGA
 + Intel x86
   + RTC (Intel PCH)
   + Standard VGA
 + Intel x86
   + RTC (Intel PCH)

-  + IOAPIC irq controller
+  + IOAPIC IRQ controller

   + APIC Timer
   + Legacy i8042 keyboard controller
 + ARM
   + GICv3
   + APIC Timer
   + Legacy i8042 keyboard controller
 + ARM
   + GICv3

-  + PL011 (W.I.P)
-
-By the way, do you know there is an online video course  by the author on the design of lunaix? [Check it out](https://space.bilibili.com/12995787/channel/collectiondetail?sid=196337) (although it is in Chinese!) 
+  + PL011 (WIP)
+</details>

 
 ## Project Structure
 
 
 ## Project Structure
 


@@ -137,40 +161,117 @@ A successful build will give `build/bin/kernel.bin`.
 
 **Please note: this is the kernel, not a bootable image, it require a bootloader to boot and specify the rootfs.**
 
 
 **Please note: this is the kernel, not a bootable image, it require a bootloader to boot and specify the rootfs.**
 

-## Booting the kernel
+## Get Started

 
 

-Since lunaix is a kernel, much like linux. It requires additional setup to do the magic. And, as in "much like linux", methods to make linux kernel boot can also apply to lunaix without or with little translation, as we will discuss below.
+If you are impatient, or just want something to run and don't want to went through tedious process of configuring rootfs and tweak kernel parameters. You can use the `live_debug.sh` provided in the lunaix root directory to quickly bring up the system with default parameters (also used by the author for debugging).

 
 

-The bootloader part is generic, any bootloader, for example GRUB will work (not tested for UEFI, but I expect this would be an exception), or booting up in QEMU using `-kernel` option
+### Quick Start

 
 

-The kernel command line, is however, a bit differentiated.
-The syntax is similar, both takes form of space-separated array of `<key>=<val>` pairs or boolean `<flag>`.
+This will get you up and running real quick. We will use `x86_64` as example.

 
 

-Currently, lunaix support the following options
+Assuming a Linux or other Unix-like shell environment.

 
 

-| Option | Default Value | Optional | Usage |
-| ------ | ----          |  -----   | ----  |
-| console | `/dev/ttyS0`   | No | Specify the system console device, path within lunaix's devfs |
-| rootfs | `/dev/block/sda` | No | Specify the device contain rootfs image, path within lunaix's devfs |
-| init | `/init` | Yes | Path within rootfs of the `init` |
+#### Select your target
+```sh
+$ export ARCH=x86_64
+```

 
 

-### A quick 'Get Started'
+#### Check Python
+```sh
+$ python --version
+```

 
 

-One can use the `live_debug.sh` provided in the lunaix root directory to quickly bring up the system with default parameters (also used by the author for debugging).
+Ensure at least `3.11`

 
 

-Following the steps:
+#### Check compiler
+```sh
+$ gcc -dumpmachine
+```
+
+Ensure `x86_64-linux-gnu` or anything resemble `x86_64`
+
+#### Check QEMU
+```sh
+$ which qemu-system-x86_64
+```
+
+Should display a valid installation path
+
+#### Optional: Setting up Cross-Compiler
+```sh
+$ export CX_PREFIX=x86_64-linux-gnu-
+```
+
+#### Run Configuration
+```sh
+make config
+```
+
+Then hitting `q` in the interactive shell to save and quit
+
+#### Build stock rootfs
+
+```sh
+make rootfs
+```
+
+#### Build & Run
+```sh
+./live_debug.sh
+```
+
+you should see gdb now take control of your shell
+
+#### Connect to serial via telnet
+
+Open up another window or session
+```sh
+telnet localhost 12345
+```
+
+#### Commence simulation
+
+Back to the gdb session and type `c` to countine
+
+
+#### Watch Lunaix booting!
+
+Congrats, enjoy your lunaix! (or submit an issue)
+
+
+### Not so Quick Start
+
+Here is a slower and yet more verbose steps:

 
 1. Select an architecture `<arch>`
 2. Check the compilation prerequisites and presence of `qemu-system-<arch>`
 
 1. Select an architecture `<arch>`
 2. Check the compilation prerequisites and presence of `qemu-system-<arch>`

-3. Run `make ARCH=<arch> user` to build the stock user program
+3. Optionally export `CX_PREFIX` if you are building for another architecture.

 4. Run `make ARCH=<arch> rootfs` to build stock rootfs image, require support of `dd`，`mkfs.ext2`, `mount -o loop`, `mktemp`.
 5. Run `ARCH=<arch> live_debug.sh` to boot in QEMU with gdb hooked (one should see a gdb session)
 4. Run `make ARCH=<arch> rootfs` to build stock rootfs image, require support of `dd`，`mkfs.ext2`, `mount -o loop`, `mktemp`.
 5. Run `ARCH=<arch> live_debug.sh` to boot in QEMU with gdb hooked (one should see a gdb session)

-6. telenet to `localhost:12345`, this is QEMU emulated serial port
+6. telnet to `localhost:12345`, this is QEMU emulated serial port

 7. type `c` in the active gdb session and commence emulation.
 8. Congrats, enjoy your lunaix!
 (or submit an issue)
 
 
 7. type `c` in the active gdb session and commence emulation.
 8. Congrats, enjoy your lunaix!
 (or submit an issue)
 
 

+## Booting the kernel
+
+Since lunaix is a kernel, much like linux. It requires additional setup to do the magic. And, as in "much like linux", methods to make linux kernel boot can also apply to lunaix without or with little translation, as we will discuss below.
+
+The bootloader part is generic, any bootloader, for example GRUB will work (not tested for UEFI, but I expect this would be an exception), or booting up in QEMU using `-kernel` option
+
+The kernel command line, is however, a bit differentiated.
+The syntax is similar, both takes form of space-separated array of `<key>=<val>` pairs or boolean `<flag>`.
+
+Currently, lunaix support the following options
+
+| Option | Default Value | Optional | Usage |
+| ------ | ----          |  -----   | ----  |
+| console | `/dev/ttyS0`   | No | Specify the system console device, path within lunaix's devfs |
+| rootfs | `/dev/block/sda` | No | Specify the device contain rootfs image, path within lunaix's devfs |
+| init | `/init` | Yes | Path within rootfs of the `init` |
+
+

 ## Submit an Issue
 
 If one ran into bug, one can submit an issue by filling up the following template
 ## Submit an Issue
 
 If one ran into bug, one can submit an issue by filling up the following template


@@ -178,11 +279,15 @@ If one ran into bug, one can submit an issue by filling up the following templat
 ```
 1. Describe the problem
     "How does it look like, anything descriptive: visual, sonic, emotional experience"
 ```
 1. Describe the problem
     "How does it look like, anything descriptive: visual, sonic, emotional experience"

+

 2. Steps to reproduce
     "How you ran into this mess?"
 2. Steps to reproduce
     "How you ran into this mess?"

+

 3. Expected behaviour
     "What do you intended/expected to achieve/to be"
 3. Expected behaviour
     "What do you intended/expected to achieve/to be"

+

 4. Lunaix's panic trace (if applicable)
 4. Lunaix's panic trace (if applicable)

+

 5. Other clues that you think might be helpful
 ```
 
 5. Other clues that you think might be helpful
 ```
 


@@ -266,22 +371,22 @@ As a personal challenge, this project is independently developed by the author s
 2. `mkdir(2)`
 2. `lseek(2)`
 2. `readdir(2)`
 2. `mkdir(2)`
 2. `lseek(2)`
 2. `readdir(2)`

-2. `readlink(2)`※
-2. `readlinkat(2)`※
-2. `rmdir(2)`※
-2. `unlink(2)`※
-2. `unlinkat(2)`※
-2. `link(2)`※
-2. `fsync(2)`※
+2. `readlink(2)`
+2. `readlinkat(2)`
+2. `rmdir(2)`
+2. `unlink(2)`
+2. `unlinkat(2)`
+2. `link(2)`
+2. `fsync(2)`

 2. `dup(2)`
 2. `dup2(2)`
 2. `dup(2)`
 2. `dup2(2)`

-2. `symlink(2)`※
+2. `symlink(2)`

 2. `chdir(2)`
 2. `fchdir(2)`
 2. `getcwd(2)`
 2. `rename(2)`※
 2. `mount(2)`
 2. `chdir(2)`
 2. `fchdir(2)`
 2. `getcwd(2)`
 2. `rename(2)`※
 2. `mount(2)`

-2. `unmount` (a.k.a `umount(2)`)※
+2. `unmount` (a.k.a `umount(2)`)

 2. `getxattr(2)`※
 2. `setxattr(2)`※
 2. `fgetxattr(2)`※
 2. `getxattr(2)`※
 2. `setxattr(2)`※
 2. `fgetxattr(2)`※


@@ -303,6 +408,20 @@ As a personal challenge, this project is independently developed by the author s
 4. `pthread_kill`
 4. `pthread_detach`
 4. `pthread_sigmask`
 4. `pthread_kill`
 4. `pthread_detach`
 4. `pthread_sigmask`

+4. `getuid`
+4. `getgid`
+4. `geteuid`
+4. `getegid`
+4. `getgroups`
+4. `setuid`
+4. `setgid`
+4. `seteuid`
+4. `setegid`
+4. `setgroups`
+4. `chroot`
+4. `fchmodat`
+4. `fchownat`
+4. `faccessat`

 
 **LunaixOS**
 
 
 **LunaixOS**
 


@@ -314,7 +433,7 @@ As a personal challenge, this project is independently developed by the author s
 
 ## Appendix 2: Debugging with GDB remotely via UART
 
 
 ## Appendix 2: Debugging with GDB remotely via UART
 

-**(((( Not working yet, need rework ))))**
+**(((( Broken after a refactoring years ago, need rework ))))**

 
 The LunaixOS kernel comes with a built-in GDB debugging server, which runs on COM1@9600Bd. However, LunaixOS must be in debug mode before involving GDB.
 
 
 The LunaixOS kernel comes with a built-in GDB debugging server, which runs on COM1@9600Bd. However, LunaixOS must be in debug mode before involving GDB.