From: Lunaixsky Date: Sat, 10 Aug 2024 13:52:28 +0000 (+0100) Subject: Second Extended Filesystem (ext2) and other improvements (#33) X-Git-Url: https://scm.lunaixsky.com/lunaix-os.git/commitdiff_plain/270869139db617e29a35bb9ded41087bb702f9ac?ds=inline;hp=270869139db617e29a35bb9ded41087bb702f9ac Second Extended Filesystem (ext2) and other improvements (#33) * Add a wrapper layer to vfs interface with intension to serve an unified API for file-system driver implementation and thus makes the flow clear. * Localise iso9660's header file * Make qemu launch cmd out of makefile to enable better flexibility * remove an out-dated FIXME annotation * Add ext2 protocol definitions and v_superblock construction * Sparse Caching: cache framework for caching spatial data unit (#34) * Add framework for sparse caching facility Added a spinlock as a future-proof placeholder * * Add bcache_zone to allow more fine gran control over lru zone used when creating a bcache * Add lock on entire bcache instance to avoid concurrent modification on internal data structure * Add missing free function for lru zone, introduce async freeing when a zone is too busy to be freed onsite * Add interface for destroying spinlock when a guarded object is about to be freed, thus avoid locking-after-free. * Refactor inode pcache to use bcache for better centralisation * lru: add proper synchronisation ensurance to lru_zone * lru: add twimap integration to all lru_zones * lru: add extra statistical metrics * change naming to spinlock relatives for better clarity * twimap: go_next now called before read, to capture corner case of empty list * clean-up and compliation fixes * * blk_buf: buffering mechanism for blkio requests and their payloads * Replace locking on each bcache node with ref counter, thus avoid creating weired locking pattern and potentially dead lock. * add documentation on the stages that owloysius will recognise * housekeeping: remove not-needed file * * ext2: parsing superblock and block descriptor table * ext2: locating and reading inode * fsapi: add new helper method for initializing inode * fsapi: add support to block buffering, allowing read/write of file system block to be efficient. * * ext2: add support to read the inode data block and indirect link following. * ext2: introduce btlb to accelerate indirect link following. * blkbuf: add `refonce` to increament the internal ref counter for better cache management * housekeeping: make the inline function at hash.h static * housekeeping: rename fsapi_{get|put}blk to fsblock_{take|put} * * Change the directory iteration pattern in vfs This allow us to add support of lseek functionality to directory. i.e. seek to i-th directory entry, similar to seekdir(2) * * ext2: directory iteration and seeking. * ext2: add a generic iterator for ext2 objects * minor refactoring on some code snippet to reduce verbosity * * ext2: read and read_page support on data blocks * * proper error handling and reporting on blkbuf and ext2 * * vfs: fix an issue that async freeing of inode and dnode cachings could potentially result undefined access to an already freed v_superblock object. By tracking the no. of references of vsb and free it when and only when nobody is reference it. * vfs: remove the auto-handling of . and .. dir entry from the readdir interface. Thus makes it a file system driver's responsibility * ext2: fix the incorrect index usage when addressing an ext2 directory entry. * ext2: add strneq for bounded string comparision. * minor refactoring and tidying. * * make things compile * iso9660: fix the issue when selecting name for the 00/01 dir ident * inode::itype bit-field redefinition & bug fixes * rework the itype bit-fields for better clarity. * fix couple of bugs in ls that does not check errno of readdir syscall * add a new directive '?' to debug_sh (sh) to check exit status of previous invocation * ext2: properly functioned the directory listing * fix strncpy behavior to be smarter * corret the inode number offset * provide v_mount struct before mounting stuff, which makes more sense * fix issue of btrie tree retrive incorrect leaf when order is a odd number * fix the itype conversion from ext2 to lunaix * ext2: basic file read functioning * fix incorrect return size calculation in ext2::read and pcache::read * make ext2db_iter terminate when datablock reaching the size of file. * fix that cat does not check on negative return value of read upon read(2) error. * ext2: inode and data block allocation * ext2: add support to allocate inode and data block * ext2: add support to de-allocate inode and data block * ext2: managing the ext2_gdesc with bcache * ext2: rename some functions for conciseness * usr/sh: QoL improvement by faking the PATH env var * ext2: inode data blocks allocation * ext2: unifying the indirect block walking * ext2: data block allocation for inode * ext2: mkdir, rmdir, generic dir_ent insert/delete * ext2: add mkdir and rmdir * ext2: add inode creation * ext2: add ability to do file write * vfs: add the missing destruct callback for dnode * fix: add a null check on bbuf_t related operations * ext2: symlink, mknod, link/unlink, sync and fixes * ext2: add support to symlink get/set * ext2: add support to inode creation * ext2: add link and unlink * ext2: directory renaming * ext2: fsync and full fs syncing * ext2: add proper un-mounting * ext2: inode metadata update & sync * vfs : some refactoring * vfs : add O_TRUNC and fix O_APPEND behaviour * vfs : update the fsize after write * twimap: fix issue that it failed to read mapped content * update LBuild with new files, fix some issue with LConfig * fix issue where LConfigProvider::has_config does not capture the case of boolean typed config * fix issue where CHeaderConfigProvider:: try to appened non-str typed value to the header file generation flow * various fixes for merging the master branch * fix: the incorrect use of clz on unsigned long data type * ref: add 64bits variances of clzl, llog2. provide handy definition of most significant bit index of unsigned long and int. * ref: uncaptialize the ILOG2 * fix: the kernel stack creation and copy algorithm, make it less hacky * fix: assertion on thread state in pwake is too strong. if a thread is already awakened by other event or killed, just remove it from wait list * fix: change the tag calculation in ext2::bTLB, so it can contain validity bit * fix: a block buffer should ref again when inserting into bTLB * fix: mixed use of size_t and unsigned int, this is not working on 64bits platform * fix: boot disk prober automatically use last device it scaned when no blkdev can be identified as boot disk * feat: qemu.py (autoqemu) will now skip the block device if the image is not found, so it provides a less disastrous fallback behavior * fix: always vzalloc for the bltb buffer. * fix logic error in ext2 file write & inode creation flow * rename the fsblock_take to fsblock_get * add the fsblock_take for doing refonce function * add a test program that write random string into file * fix issues in dirent creation and file write * fix: incorrect calaculation of rec_len * fix: confusion on inode number (1 based) and inode index (0 based) * fix: racing when one process creating file while other try to read that directory structure. * fix: pcache write must not enforce the fpos to be page aligned all the time, otherwise, page cache paddings will be written. * fix: autoqemu: smp and memory parameter not set. * fix: rng generator failed to populate whole buffer other than the beginning few bytes * fix: failsafe unwrapper should follow x86_64 convention. * fix: blkbuf: define null bbuf as not initialized. * modified ext2 now passed the e2fsck validation * fix: sync the changes to superblock. * fix: correct the rec_len when the dirent is last of this data block * fix: (ahci) add sanity check on DMA buffer size * fix: i_blocks calculation should include the intermediate blocks allocated for building indrections. * ref: some minor changes to the fsapi_vsb_* * feat: add a printk emulation for quick log printing without the need of explicit specify the LOG_MODULE * fix: skip the sync if a blkio_req is pending. previous version only skip it after the access flag is changed, which may lead to undesired behaviour on some cases. * fix issue in ext2_mkdir * fix: break the pwait into two parts: add to wait list and the actual wait start. Thus to resolve the racing problem in the blkio by ensure that the active thread will always in the blkio_req's waiting list even when the completion interrupt arrive before we invoke the actual waiting. * fix: dirty page cache should be flushed before closing the file. * fix: make most system calls preemptive in kernel space, to increase the responsive time and avoid interrupt squashing for i/o intensive device. * feat: add mkdir program for testing * protect the blkio and blkbuf with mutex lock. * ref: separate the non-reentrant and reentrant (nested) mutex locking/unlocking. * Improvements patch: Kernel preemptions (#42) * added mechanism for checking any stalling in kernel space * ref: wrap the syscall dispatching into a proper C function. * ref: separate the sched_pass into two category: yield and preempt * ref: add a redirection checking stage prior to the context restore, which enable us to extend in the future (if needed) * feat: add the to more pwait variant, for indicate that the event waiting should be opted in stall checking * feat: add thread statistics on kernel/user transition * fix: outdated kernel stack reclaiming corrupt the other threads * feat: add an extra config term to detect dead loop in kernel space. * minor adjustment on formating * add comments on explaining switch_signposting * file deletion and add file resizing * fix: tested and fixed the file deletion. * feat: inode resizing for dynamically deallocate data block that is not used after shrinking. * fix: incorrect thread reference when checking stalled task * feat: test program for file deletion/unlinking. * update the inode attrs when setting symlink * ref: change some comments * fix: add FSTYPE_PSEUDO to distinguish filesystem that is capable of device-less mounting * feat: a wrapper for filesystem registration * ref: refactor currently avaliable file system registration code with aforementioned wrapper * update readme * * rewrite the ext2_rename, to make it right * feat: add support to FILE_TYPE and LARGE_FILE feature. * fix: feature check on mounting. * fix: memory leakage on ext2_dnode. * fix: double free on ext2_dnode when ext2_mkdir fail. * fix: properly release the ext2_dnode after ext2dr_remove. * ref: code clean up on ext2. ---