UNIVERSAL DATA STORAGE - FILE ADDRESS SYSTEM ================================================ In order to be able to extend the data without limitations we need to define: 1. machine settings a) define one root folder in the /data folder under each disk partition /partition1 /aisystem /data 5b174ca605b042b285cd310abf5289a7 /partition2 /aisystem /data aab048a5ae57435f88d0409ccf85e59d .............. /partition100 /aisystem /data fa3ec39fe0334231937ebb70643b6582 b) define the number of directories per root folder /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 2 ... 100 c) define the number of level directories /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 1 1 1 1 1 ............ 2 ... 100 2 ... 100 2 ... 100 1 1 2 ... 100 d) define the number of files per directory /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 5b174ca605b042b285cd310abf5289a7.db aa77cceab82c42449253e556cf217433.db ................................... 7059086ca44840f3b5c2ef482ec6a250.db 2 6d2626ba02774a72a21303d5fdb98190.db ................................... fa3ec39fe0334231937ebb70643b6582.db 37b14a996b1c495b952de81294beed47.db e) define the number of records per file /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 5b174ca605b042b285cd310abf5289a7.db 1 2 ... 1000000 2. one UUID for each machine 3. one UUID's for each root folder under a disk partition 4. one ID for each sub-directory under a root folder or directory 5. one UUID for each file 6. one ID for each record under a file 7. expansion rules: a) no files will be stored under /data or root folder b) we start counting from 1 and not 0, because the first database record has always the ID = 1 c) once the settings limits (i.e. maximum number of root folders, directories, files and subdirectory levels) has been reached for a machine, we expand on another machine Rationale ========================== 1. we work with heterogenous systems that span different machine configurations, operating systems and partitions 2. file systems differences; some file systems can store larger numbers of files per directory and larger numbers of directories than others 3. we use uuid for each machine because a machine name or IP address can change 4. we use uuid for each partition's root folder because for some operating systems the drive letters can be changed 5. we use uuid for each file name because the records have only id's 6. we use id's for each record because we need fast access to it 7. we use root folders to achieve data localization Expansion Algorithm ========================== We use width-first expansion starting from each root folder. Breadth-first can only increase data dispersion. Expansion Steps: 1. create root folder /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 2. create directory 1 /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 3. create database files under directory 1 /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 6d2626ba02774a72a21303d5fdb98190.db ................................... fa3ec39fe0334231937ebb70643b6582.db 37b14a996b1c495b952de81294beed47.db 4. once the maximum number of files per directory has been reached create a new directory 2 /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 6d2626ba02774a72a21303d5fdb98190.db ................................... fa3ec39fe0334231937ebb70643b6582.db 37b14a996b1c495b952de81294beed47.db 2 5. create database files under directory 2 /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 6d2626ba02774a72a21303d5fdb98190.db ................................... fa3ec39fe0334231937ebb70643b6582.db 37b14a996b1c495b952de81294beed47.db 2 5b174ca605b042b285cd310abf5289a7.db aa77cceab82c42449253e556cf217433.db ................................... 7059086ca44840f3b5c2ef482ec6a250.db 6. once the maximum number of directories has been reached at the same level create a subdirectory under directory 1 /partition1/aisystem/data/ 5b174ca605b042b285cd310abf5289a7 1 1 2 ... 100 7. create database files under directory 1 (i.e. 1/1) 8. once the maximum number of files per directory has been reached at the same level create a new directory 2 (i.e. 1/2) 9. once the maximum number of directories has been reached at the same level create a new directory 1 (i.e. 1/1/1) 10. ....................... Future Optimization Directions ================================ 1. explore expansion based on database file type (i.e. Property_Archives, Objects, Properties...).