1The XFS File System 2=================== 3 4This document describes how to test the XFS file system, XFS file system API for haiku 5and Its current status on haiku. 6 7 8Testing XFS File System 9----------------------- 10 11There are three ways we can test XFS : 12 13- Using xfs_shell. 14- Using userlandfs. 15- Building a version of haiku with XFS support and then mounting a file system. 16 17But before that we will need to create XFS images for all testing purposes. 18 19Creating File System Images 20^^^^^^^^^^^^^^^^^^^^^^^^^^^ 21 22Currently only linux has full XFS support so we will use linux for generating file system images. 23 24First we need to create an empty sparse image using command:: 25 26 $ dd if=/dev/zero of=fs.img count=0 bs=1 seek=5G 27 28The output will be:: 29 30 0+0 records in 31 0+0 records out 32 0 bytes (0 B) copied, 0.000133533 s, 0.0 kB/s 33 34Do note that we can create images of whatever size or name we want, for example the above command 35creates fs.img of size 5 GB, if we alter seek = 10G it will create fs.img with size 10 GB. 36 37The XFS file system on linux supports two versions, V4 and V5. 38 39To put XFS V5 file system on our sparse image run command:: 40 41 $ /sbin/mkfs.xfs fs.img 42 43The output will be:: 44 45 meta-data =fs.img isize=512 agcount=4, agsize=65536 blks 46 = sectsz=512 attr=2, projid32bit=1 47 = crc=1 finobt=1, sparse=1, rmapbt=0 48 = reflink=1 49 data = bsize=4096 blocks=262144, imaxpct=25 50 = sunit=0 swidth=0 blks 51 naming =version 2 bsize=4096 ascii-ci=0, ftype=1 52 log =internal log bsize=4096 blocks=2560, version=2 53 = sectsz=512 sunit=0 blks, lazy-count=1 54 realtime =none extsz=4096 blocks=0, rtextents=0 55 56To put XFS V4 file system on our sparse image run command:: 57 58 $ /sbin/mkfs.xfs -m crc=0 file.img 59 60The output will be:: 61 62 meta-data=fs.img isize=256 agcount=4, agsize=327680 blks 63 = sectsz=512 attr=2, projid32bit=0 64 data = bsize=4096 blocks=1310720, imaxpct=25 65 = sunit=0 swidth=0 blks 66 naming =version 2 bsize=4096 ascii-ci=0 67 log =internal log bsize=4096 blocks=2560, version=2 68 = sectsz=512 sunit=0 blks, lazy-count=1 69 realtime =none extsz=4096 blocks=0, rtextents=0 70 71**The linux kernel will support older XFS v4 filesystems by default until 2025 and 72Support for the V4 format will be removed entirely in September 2030** 73 74Now we can mount our file system image and create entries for testing XFS haiku driver. 75 76Test using xfs_shell 77^^^^^^^^^^^^^^^^^^^^^^^ 78 79The idea of fs_shell is to run the file system code outside of haiku. We can run it 80as an application, it provides a simple command line interface to perform various 81operations on the file system (list directories, read and display files, etc). 82 83First we have to compile it:: 84 85 jam "<build>xfs_shell" 86 87Then run it:: 88 89 jam run ":<build>xfs_shell" fs.img 90 91Where fs.img is the file system image we created from linux kernel. 92 93Test directly inside Haiku 94^^^^^^^^^^^^^^^^^^^^^^^^^^ 95 96First build a version of haiku with XFS support, to do this we need to add "xfs" to the image 97definition `here <https://git.haiku-os.org/haiku/tree/build/jam/images/definitions/minimum#n239>`__. 98 99Then compile haiku as usual and run the resulting system in a virtual machine or on real hardware. 100 101We can then try to mount an XFS file system using command on Haiku:: 102 103 mount -t xfs <path to image> <path to mount folder> 104 105for example:: 106 107 mount -t xfs /boot/home/Desktop/fs.img /boot/home/Desktop/Testing 108 109Here fs.img is file system image and Testing is mount point. 110 111Test using userlandfs 112^^^^^^^^^^^^^^^^^^^^^ 113 114To be updated 115 116 117Haiku XFS API 118------------- 119 120* | All the necessary hooks for file system like xfs_mount(), open_dir(), read_dir() etc.. 121 are implemented inside **kernel_interface.cpp** file. 122 | It acts as an interface between Haiku kernel and XFS file system. 123 | Documentation for all necessary file system hooks can be found 124 `here <https://www.haiku-os.org/docs/api/fs_modules.html>`_ 125 | 126 127* | Whenever we run a file system under fs_shell we can't use system headers, 128 fs_shell compatible headers are there which needs to be used whenever we try 129 to mount XFS file system using xfs_shell. 130 | To resolve this problem we use **system_dependencies.h** header file which takes 131 care to use correct headers whenever we mount XFS file system either using xfs_shell 132 or directly inside Haiku. 133 134* XFS stores data on disk in Big Endian byte order, to convert data into host order 135 all classes and data headers has **SwapEndian()** function, Its better to have all data 136 conversions at one place to avoid future problems related to data byte order. 137 138* XFS SuperBlock starts at ondisk offset 0, the definition of SuperBlock is in **xfs.h** file. 139 140* | A Volume is an instance of file system and defined in **Volume.h** file. 141 XFS Volume contains SuperBlock, file system device and essential functions 142 like Identify(), mount() etc... 143 | *Identify()* function reads SuperBlock from disk and verifies it. 144 | *Mount()* function mounts file system device and publishes root inode of file system 145 (Typically root inode number for XFS is 128). 146 | 147 148* | XFS uses TRACE Macro to debug file system, definitions for TRACE, ERROR and ASSERT 149 are defined at **Debug.h** in the form of Macro. 150 | To enable TRACE calls just add ``#define TRACE_XFS`` in Debug.h file and 151 vice versa to disable it. 152 | 153 154* | XFS V5 introduced metadata checksums to ensure the integrity of metadata in file system, 155 It uses CRC32C checksum algorithm. For XFS all checksums related functions are defined in 156 **Checksum.h** header file. 157 | It contains following functions : 158 * *xfs_verify_cksum()* to verify checksum for buffer. 159 * *xfs_update_cksum()* to update checksum for buffer. 160 | **XFS stores checksum in little endian byte order unlike other ondisk data which is stored 161 in big endian byte order** 162 163* XFS V5 introduced many other fields for metadata verification like *BlockNo* *UUID* *Owner* 164 etc.. All this fields are common in every data header and so are their checks. So to not 165 repeat same checks again and again for all headers we created a *VerifyHeader* template 166 function which is defined in **VerifyHeader.h** file. This function is commonly used in 167 all forms of headers for verification purposes. 168 169Inodes 170^^^^^^ 171 172XFS inodes comes in three versions: 173 174* Inode V1 & V2. (Version 4 XFS) 175* Inode V3. (Version 5 XFS) 176 177Version 1 inode support is already deprecated on linux kernel, Haiku XFS supports it only 178in read format. When we will have write support for XFS we will only support V2 and V3 inodes. 179 180V1 & V2 inodes are 256 bytes while V3 inodes are 512 bytes in size allowing more data to be 181stored directly inside inode. 182 183**CoreInodeSize()** is a helper funtion which returns size of inode based on version of XFS and 184is used throughout our XFS code. 185 186**DIR_DFORK_PTR** is a Macro which expands to void pointer to the data offset in inode, which 187could be either shortform entries, extents or B+Tree root node depending on the data format 188of inode (di_format). 189 190Similarly **DIR_AFORK_PTR** Macro expands to void pointer to the attribute offset in inode, 191which could be either shortform attributes, attributes extents or B+Tree node depending on 192the attribute format of Inode (di_aformat). 193 194Since size of inodes could differ based on different versions of XFS we pass CoreInodeSize() 195function as a parameter to DIR_DFORK_PTR and DIR_AFORK_PTR macros to return correct pointer offset. 196 197**di_forkoff** specifies the offset into the inode's literal area where the extended attribute 198fork starts. This value is initially zero until an extended attribute is created. 199It is fixed for V1 & V2 inode's while for V3 Inodes it is dynamic in size, 200allowing complete use of inode's literal area. 201 202Directories 203^^^^^^^^^^^ 204 205Depending on the number of entries inside directory, XFS divides directories into five formats : 206 207* Shortform directory. 208* Block directory. 209* Leaf directory. 210* Node directory. 211* B+Tree directorcy. 212 213Class DirectoryIterator in **Directory.h** file provides an interface between kernel request 214to open, read directory and all forms of directories. It first identifies correct format of 215entries inside inode and then returns request as per format found. 216 217**Shortform directory** 218 219* When the number of entries inside directory are small enough such that we can store all 220 metadata inside inode itself, this form of directory is known as shortform directory. 221* We can check if a directory is shortform if the format of inode is *XFS_DINODE_FMT_LOCAL*. 222* The header for ShortForm entries is located at data fork pointer inside inode, which we cast 223 directly to *ShortFormHeader*. 224* Since number of entries are short we can simply iterate over all entries for *Lookup()* and 225 *GetNext()* functions. 226 227**Block directory** 228 229* When number of entries expand such that we can no longer store all directory metadata 230 inside inode we use extents. 231* We can check if a directory is extent based if the format of inode is *XFS_DINODE_FMT_EXTENTS*. 232* In Block directory we have a single directory block for Data header, leaf header 233 and free data header. This simple fact helps us to determine if given extent format 234 in inode is block directory. 235* Since XFS V4 & V5 data headers differs we use a virtual class *ExtentDataHeader* which 236 acts as an interface between V4 & V5 data header, this class only stores pure virtual 237 functions and no data. 238* *CreateDataHeader* returns a class instance based on the version of XFS mounted. 239* Since now we have a virtual class with V_PTRS we need to be very careful with data stored 240 ondisk and data inside class, for example we now can't use sizeof() operator on class to 241 return its size which is consistent with its size inside disk. To handle this issue helper 242 function like *SizeOfDataHeader* are created which needs to be used instead of sizeof() operator. 243* In *GetNext()* function we simply iterate over all entries inside buffer, though a found 244 entry could be unused entry so we need to have checks if a entry found is proper entry. 245* In *Lookup()* function first we generate a hash value of entry for lookup, then we find 246 lowerbound of this hash value inside leaf entries to get address of entry inside data. 247 At last if entry matches we return B_OK else we return B_ENTRY_NOT_FOUND. 248 249**Leaf directory** 250 251* When number of entries expand such that we can no longer store all directory metadata inside 252 directory block we use leaf format. 253* In leaf directory we have a multiple directory block for Data header and free data header, 254 while single directory block for leaf header. 255* To check if given extent based inode is leaf type, we simply check for offset inside last 256 extent map, if its equal to *LEAF_STARTOFFSET* then the given inode is leaf type else it is 257 node type. 258* Since XFS V4 & V5 leaf headers differs we use a virtual class *ExtentLeafHeader* which acts 259 as an interface between V4 & V5 leaf header, this class only stores pure virtual functions 260 and no data. 261* *CreateLeafHeader* returns a class instance based on the version of XFS mounted. 262* Instead of sizeof() operator on ExtentLeafHeader we should always use *SizeOfLeafHeader()* function 263 to return correct size of class inside disk. 264* *Lookup()* and *GetNext()* functions are similar to block directories except now we don't use single 265 directory block buffer. 266 267TODO : Document Node and B+Tree based directories. 268 269Files 270^^^^^ 271 272XFS stores files in two formats : 273 274* Extent based file. 275* B+Tree based file. 276 277All implementation of read support for files is inside *Inode()* class in **Inode.h** file. 278 279When the format inside inode of file is *XFS_DINODE_FMT_EXTENTS* it is an extent based file, 280to read all data of file we simply iterate over all extents which is very similar to how we 281do it in Extent based directories. 282 283When the file becomes too large such that we cannot store more extent maps inside inode the 284format of file is changed to B+Tree. When the format inside inode of file is 285*XFS_DINODE_FMT_BTREE* it is an B+Tree based file, to read all data of file 286first we read blocks of B+Tree to extract extent maps and then read extents 287to get file's data. 288 289 290Current Status of XFS 291--------------------- 292 293Currently we only have read support for XFS, below briefly summarises read support for all formats. 294 295 296Directories 297^^^^^^^^^^^ 298 299**Short-Directory** 300 Stable read support for both V4 and V5 inside Haiku. 301 302**Block-Directory** 303 Stable read support for both V4 and V5 inside Haiku. 304 305**Leaf-Directory** 306 Stable read support for both V4 and V5 inside Haiku. 307 308**Node-Directory** 309 Stable read support for both V4 and V5 inside Haiku. 310 311**B+Tree-Directory** 312 Unstable read support for both V4 and V5, due to so many read from disk entire 313 process inside Haiku is too slow. 314 315Files 316^^^^^ 317 318**Extent based Files** 319 | *xfs_shell* - stable read support for both V4 and V5. 320 | *Haiku* - Unstable, Cat command doesn't print entire file and never terminates process. 321 322**B+Tree based Files** 323 | *xfs_shell* - stable read support for both V4 and V5. 324 | *Haiku* - Unstable, Cat command doesn't print entire file and never terminates process. 325 326Attributes 327^^^^^^^^^^ 328 329Currently we have no extended attributes support for xfs. 330 331Symlinks 332^^^^^^^^ 333 334Currently we have no symlinks support for xfs. 335 336XFS V5 exclusive features 337^^^^^^^^^^^^^^^^^^^^^^^^^ 338 339**MetaData Checksumming** 340 Metadata checksums for superblock, Inodes, and data headers are implemented. 341 342**Big Timestamps** 343 Currently we have no support. 344 345**Reverse mapping btree** 346 Currently we have no support, this data structure is still under construction 347 and testing inside linux kernel. 348 349**Refrence count btree** 350 Currently we have no support, this data structure is still under construction 351 and testing inside linux kernel. 352 353Write Support 354^^^^^^^^^^^^^ 355 356Currently we have no write support for xfs. 357 358 359References 360---------- 361 362The best and only reference for xfs is latest version of "xfs_filesystem_structure" 363written by Linux-XFS developers. 364 365The pdf version of above Doc can be found 366`here <http://ftp.ntu.edu.tw/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf>`_ 367