file.q

File and Folder Discovery

This library provides functionality to find files and folders within directories on disk.

.file.listFolder / .file.listFolderPaths

These functions list the contents of the specified folder. .file.listFolderPaths returns files and folders with the supplied path prepended (so they can be used with get for example). .file.listFolder returns just file and folder names.

.file.find / .file.findFilePaths

These functions return files that match a specified regular expression. Again, .file.findFilePaths will return any match files or folders with the supplied path prepended, where as .file.find will just return their names.

The regex to find files can be specified either as a symbol, which is then interpreted as match anywhere, or a kdb regex string.

Examples

Using symbol as regex:

q) srcFolder:` sv .file.getCwd[],`src;
Running system command: "echo %cd%"

q) .file.findFilePaths[`log; srcFolder]
,`:C:\Users\jasra_000\git\kdb-common/src/log.q

Using kdb string regex:

q) .file.findFilePaths["*.q"; srcFolder]
`:C:\Users\jasra_000\git\kdb-common/src/convert.q`:C:\Users\jasra_000\git\kdb-common/src/cron.q`:C:\Users\j..

Using string for exact match:

q) .file.findFilePaths["util.q"; srcFolder]
,`:C:\Users\jasra_000\git\kdb-common/src/util.q

.file.tree

This function recursively descends from the specified root folder down all child folders until no more folders are found (it behaves similarly to the Linux command tree). All discovered files are returned with the specified path prefixed.

Please note that folders that are symbolic links will be followed so you must ensure that there are no circular references within the folder structure you wish to tree.

.file.ensureDir

This function checks if the specified folder exists on disk already. If it doesn't it creates the folder before returning. It uses the OS library to create the new folder

.file.loadDir

This is a simple wrapper around system "l" with the conversion of a folder path symbol into a string.

.file.getCwd

This function returns the current working directory of the kdb process. It uses the OS library to get the raw result from the OS and then ensures it's in a file path symbol format.

Example

OS library working directory function:

q) .os.run[`pwd;::]
Running system command: "echo %cd%"
"C:\\Users\\jasra_000\\git\\kdb-common "

File library .file.getCwd

q) .file.getCwd[]
Running system command: "echo %cd%"
`:C:\Users\jasra_000\git\kdb-common

.file.isCompressed

This function returns true if the specified file has been compressed by kdb+ native compression.

.file.replace[source; target]

Replaces the target specified file or folder with the specified source. For files this is equivalent to a move, but for folders, it will delete the target folder before moving.

Example

q) .file.ls `:/tmp/hdb
`s#`2021.01.23`2021.01.24`2021.10.28`2021.10.29`2021.10.31`sym

q) .file.replace[`:/tmp/hdb/2021.10.31; `:/tmp/hdb/2021.10.29]
2021.11.02 13:12:35.234 DEBUG pid-516 jas 0 [./src/util.q:.util.system(20):1] Running system command: "rm -rvf /tmp/hdb/2021.10.29"
2021.11.02 13:12:35.249 DEBUG pid-516 jas 0 [./src/util.q:.util.system(20):1] Running system command: "mv /tmp/hdb/2021.10.31 /tmp/hdb/2021.10.29"

q) .file.ls `:/tmp/hdb
`s#`2021.01.23`2021.01.24`2021.10.28`2021.10.29`sym

.file.kdb.getLength

An optiimised on-disk element length function. Instead of having to read the full file (e.g. count get x), only the first 16 bytes (or compression map and first chunk if a compressed file) need to be read. This results in ~99% performance improvement with large files.

This method currently only works for single-typed lists (excluding new format GUID lists). For any other types, the function will fallback to the count get x.

See the original pull request for full performance comparisons.

.file.hdb.qPar[hdbRoot; partVal]

Wrapper for .Q.par that converts any relative paths in par.txt to absolute paths and doesn't require specifying a table argument.

q) read0 `$":/tmp/hdb-par/par.txt"
"./P1"
"./P2"

q) .Q.par[`$":/tmp/hdb-par"; .z.d; `]
`:./P1/2021.11.02/

q) .file.hdb.qPar[`$":/tmp/hdb-par"; .z.d]
`:/tmp/hdb-par/./P1/2021.11.02/