--- name: cloud-object-storage-s3fs title: Mounting a bucket using `s3fs` description: Applications that expect to read and write to a NFS-style filesystem can use `s3fs`, which can mount a bucket as directory while preserving the native object format for files. last-updated: 2024-04-18 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/cloud-object-storage?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Mounting a bucket using `s3fs` {: #s3fs} Applications that expect to read and write to a NFS-style filesystem can use `s3fs`, which can mount a bucket as directory while preserving the native object format for files. {: shortdesc} This allows you to interact with your cloud storage using familiar shell commands, like `ls` for listing or `cp` to copy files, as well as providing access to legacy applications that rely on reading and writing from local files. For a more detailed overview, [visit the project's official README](https://github.com/s3fs-fuse/s3fs-fuse). Looking for instructions for how to use IBM Cloud® Object Storage in an IBM Cloud Kubernetes Service cluster? Go to the [IBM Cloud Kubernetes Service documentation](https://cloud.ibm.com/docs/containers?topic=containers-storage-cos-understand&format=markdown) instead. {: tip} ## Prerequisites {: #s3fs-prereqs} * IBM Cloud account and an instance of IBM Cloud® Object Storage * A Linux or macOS environment * Credentials (either an [IAM API key](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam-overview&format=markdown) or [HMAC credentials](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main&format=markdown)) ## Installation {: #s3fs-install} On Debian or Ubuntu: ```sh sudo apt-get install automake autotools-dev fuse g++ git libcurl4-openssl-dev libfuse-dev libssl-dev libxml2-dev make pkg-config ``` {: codeblock} On RHEL and CentOS 7 or newer by means of EPEL: ```sh sudo yum install epel-release sudo yum install s3fs-fuse ``` {: codeblock} Your device must have public connection to pull this EPEL repo, as it is not available in IBM's private repo. See [How to install EPEL on RHEL and CentOS Stream](https://www.redhat.com/sysadmin/install-epel-linux) for more information. {: important} The official `s3fs` documentation suggests using `libcurl4-gnutls-dev` instead of `libcurl4-openssl-dev`. Either work, but the OpenSSL version may result in better performance. {: tip} For macOS, you will need to build `s3fs` from source: Ensure you have the following packages installed (all are available via [Homebrew](https://brew.sh)): - `macfuse` - `automake` - `gcc` - `curl` - `libxml2` - `pkg-config` - `openssl` And as noted in the output of the `openssl` install, you'll need to set these environment variables: ```sh export LDFLAGS="-L/usr/local/opt/openssl@3/lib" export CPPFLAGS="-I/usr/local/opt/openssl@3/include" export PKG_CONFIG_PATH="/usr/local/opt/openssl@3/lib/pkgconfig" ``` {: screen} Be aware that [macFUSE is closed-source software](https://osxfuse.github.io) containing a kernel extension, and may require a license for commercial use. {: important} First clone the Github repository: ```sh git clone https://github.com/s3fs-fuse/s3fs-fuse.git  ``` {: screen} Then build `s3fs`: ```sh cd s3fs-fuse ./autogen.sh ./configure make ``` {: screen} And install the binary: ```sh sudo make install ``` {: screen} ## Configuration {: #s3fs-config} Store your credentials in a file containing either `:` or `:`. This file needs to have limited access so run: ```sh chmod 0600 ``` {: codeblock} Now you can mount a bucket using: ```sh s3fs -o url=http{s}:// -o passwd_file= ``` {: codeblock} If the credentials file only has an API key (no HMAC credentials), you'll need to add the `ibm_iam_auth` flag as well: ```sh s3fs -o url=http{s}:// -o passwd_file= -o ibm_iam_auth ``` {: codeblock} The `` in the example refers to an existing bucket and the `` is the local path where you want to mount the bucket. The `` must correspond to the [bucket's location](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints&format=markdown). The `credentials_file` is the file created with the API key or HMAC credentials. Now, `ls ` will list the objects in that bucket as if they were local files (or in the case of object prefixes, as if they were nested directories). ## Performance optimization {: #s3fs-performance} While performance will never be equal to a true local filesystem, it is possible to use some advanced options to increase throughput. ```sh s3fs -o url=http{s}:// –o passwd_file= \ -o cipher_suites=AESGCM \ -o kernel_cache \ -o max_background=1000 \ -o max_stat_cache_size=100000 \ -o multipart_size=52 \ -o parallel_count=30 \ -o multireq_max=30 \ -o dbglevel=warn ``` {: codeblock} 1. `cipher_suites=AESGCM` is only relevant when using an HTTPS endpoint. By default, secure connections to IBM COS use the `AES256-SHA` cipher suite. Using an `AESGCM` suite instead greatly reduces the CPU load on your client machine, caused by the TLS crypto functions, while offering the same level of cryptographic security. 2. `kernel_cache` enables the kernel buffer cache on your `s3fs mountpoint`. This means that objects will only be read once by `s3fs`, as repetitive reading of the same file can be served from the kernel’s buffer cache. The kernel buffer cache will only use free memory which is not in use by other processes. This option is not recommend if you expect the bucket objects to be overwritten from another process/machine while the bucket is mounted, and your use-case requires live accessing the most up-to-date content. 3. `max_background=1000` improves `s3fs` concurrent file reading performance. By default, FUSE supports file read requests of up to 128 KB. When asking to read more than that, the kernel split the large request to smaller sub-requests and lets s3fs process them asynchronously. The `max_background` option sets the global maximum number of such concurrent asynchronous requests. By default, it is set to 12, but setting it to an arbitrary high value (1000) prevents read requests from being blocked, even when reading many files simultaneously. 4. `max_stat_cache_size=100000` reduces the number of redundant HTTP `HEAD` requests sent by `s3fs` and reduces the time it takes to list a directory or retrieve file attributes. Typical file system usage makes frequent access to a file’s metadata via a `stat()` call which maps to `HEAD` request on the object storage system. By default, `s3fs` caches the attributes (metadata) of up to 1000 objects. Each cached entry takes up to 0.5 KB of memory. Ideally, you would want the cache to be able to hold the metadata for all the objects in your bucket. However, you may want to consider the memory usage implications of this caching. Setting it to `100000` will take no more than 0.5 KB * 100000 = 50 MB. 5. `multipart_size=52` will set the maximum size of requests and responses sent and received from the COS server, in MB scale. `s3fs` sets this to 10 MB by default. Increasing this value also increases the throughput (MB/s) per HTTP connection. On the other hand, the latency for the first byte served from the file will also increase. Therefore, if your use-case only reads a small amount of data from each file, you probably do not want to increase this value. Furthermore, for large objects (say, over 50 MB) throughput increases if this value is small enough to allow the file to be fetched concurrently using multiple requests. I find that the optimal value for this option is around 50 MB. COS best practices suggest using requests that are multiples of 4 MB, and thus the recommendation is to set this option to 52 (MB). 6. `parallel_count=30` sets the maximum number of requests sent concurrently to COS, per single file read/write operation. By default, this is set to 5. For very large objects, you can get more throughput by increasing this value. As with the previous option, keep this value low if you only read a small amount of data of each file. 7. `multireq_max=30` When listing a directory, an object metadata request (`HEAD`) is sent per each object in the listing (unless the metadata is found in cache). This option limits the number of concurrent such requests sent to COS, for a single directory listing operation. By default it is set to 20. Note that this value must be greater or equal to the `parallel_count` option above. 8. `dbglevel=warn` sets the debug level to `warn` instead of the default (`crit`) for logging messages to /var/log/syslog. ## Limitations {: #s3fs-limitations} It is important to remember that s3fs may not be suitable for all applications, as object storage services have high-latency for time to first byte and lack random write access. Workloads that only read big files, like deep learning workloads, can achieve good throughput using `s3fs`.