Version: 5.10.1

NFS client

LeilFS implements a File System Abstraction Layer (FSAL) for NFS-Ganesha to allow connecting NFS clients to the cluster.

NFS-Ganesha is an NFS v3, v4.0, v4.1, and v4.2 fileserver that runs in user mode on most UNIX/Linux systems. NFS-Ganesha is used by LeilFS for offering NFS v3 and v4 services.

The LeilFS NFS integration uses the SaunaFS FSAL module, so the FSAL name in the Ganesha configuration is still SaunaFS. Starting with SaunaFS FSAL 5.5.0, the supported NFS-Ganesha version is 9.2. The following sections cover the recommended installation flow and the basic Ganesha settings required to connect NFS clients to a LeilFS cluster.

More information and documentation to set up other specific options and advanced setups for NFS-Ganesha are available at the following link: https://github.com/nfs-ganesha/nfs-ganesha/wiki/Configurationfile

Installing NFS-Ganesha

NFS-Ganesha installation is independent from the SaunaFS FSAL installation. However, the installed Ganesha version must be compatible with the SaunaFS FSAL version that will be used.

The following compatibility matrix summarizes the supported version mapping currently documented for the SaunaFS FSAL:

SaunaFS FSAL version range	Supported NFS-Ganesha version
>= 4.0.0 and < 5.3.0	4.3
>= 5.3.0 and < 5.5.0	7.1
>= 5.5.0	9.2

Some Debian repositories do provide NFS-Ganesha packages. However, the version available in a distribution repository may differ from the version supported by the SaunaFS FSAL. For example, Debian may publish versions newer or older than the version shown as supported in the compatibility matrix above.

There are no official Debian packages released directly from the NFS-Ganesha GitHub repository. Therefore, unless your target distribution already provides the exact supported version, the recommended approach is to build the Ganesha binaries from source code using the upstream tag that matches the supported NFS-Ganesha version from the compatibility matrix above.

For example, if your SaunaFS FSAL version requires NFS-Ganesha 9.2, use the upstream tag V9.2:

# clone nfs-ganesha v9.2
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git checkout V9.2
git submodule update --recursive --init

# build nfs-ganesha v9.2
mkdir build
cmake -B build src \
          -DCMAKE_BUILD_TYPE=RelWithDebInfo \
          -DCMAKE_INSTALL_PREFIX=/usr \
          -DUSE_FSAL_SAUNAFS=ON \
          -DUSE_FSAL_CEPH=OFF \
          -DUSE_FSAL_GLUSTER=OFF \
          -DUSE_FSAL_GPFS=OFF \
          -DUSE_FSAL_KVSFS=OFF \
          -DUSE_FSAL_LIZARDFS=OFF \
          -DUSE_FSAL_LUSTRE=OFF \
          -DUSE_FSAL_PROXY_V3=OFF \
          -DUSE_FSAL_PROXY_V4=OFF \
          -DUSE_FSAL_RGW=OFF \
          -DUSE_FSAL_XFS=OFF \
          -DUSE_GSS=ON \
          -DUSE_MONITORING=OFF \
          -DUSE_RADOS_RECOV=OFF \
          -DRADOS_URLS=OFF \
          -D_MSPAC_SUPPORT=OFF

# install the nfs-ganesha binaries and libraries
sudo cmake --build build --target install

Please note that some dependences are required to build NFS-Ganesha. Below is the list of packages we use:

sudo apt update
sudo apt install -y --no-install-recommends \
         bison \
         build-essential \
         cmake \
         doxygen \
         flex \
         git \
         libacl1-dev \
         libblkid-dev \
         libcap-dev \
         libdbus-1-dev \
         libjemalloc-dev \
         libkrb5-dev \
         libnfsidmap-dev \
         libnsl-dev \
         libunwind-dev \
         liburcu-dev \
         lsb-release \
         pkg-config

Note

Avoid mixing Ubuntu packages from unrelated releases just to obtain a newer Ganesha version. That was an older workaround and no longer reflects the supported LeilFS workflow.

Installing and setting up SaunaFS FSAL

After Ganesha is installed, the next step is to install the SaunaFS FSAL package that matches the SaunaFS version you want to deploy.

If you have not configured the SaunaFS package repository yet, follow the instructions in Installation.

Then install the FSAL package:

apt install saunafs-nfs-ganesha

Compatibility requirement

The package saunafs-nfs-ganesha installs only the SaunaFS FSAL module. It does not install NFS-Ganesha itself.

Before installing or starting NFS-Ganesha, make sure the installed Ganesha version matches the supported version for your SaunaFS FSAL release. Using an incompatible Ganesha build or distribution package may cause the FSAL module to fail to load at startup.

Hint

The package saunafs-nfs-ganesha installs the SaunaFS FSAL library (libfsalsaunafs.so).

When starting NFS-Ganesha, make sure the daemon can locate libfsalsaunafs.so in its configured FSAL library path. Otherwise, the SaunaFS FSAL will not be loaded and NFS clients will not be able to connect to the cluster.

On Ubuntu systems, the saunafs-nfs-ganesha package typically installs libfsalsaunafs.so under:

/usr/lib/x86_64-linux-gnu/ganesha/

If you built NFS-Ganesha from source and it uses a different FSAL library directory, you should create a symbolic link from the installed library to the directory expected by your Ganesha installation.

For example, if Ganesha expects FSAL libraries under /usr/lib/ganesha/, you can run:

sudo ln -s /usr/lib/x86_64-linux-gnu/ganesha/libfsalsaunafs.so /usr/lib/ganesha/libfsalsaunafs.so

Below there is a basic /etc/ganesha/ganesha.conf example file to use SaunaFS FSAL:

###################################################
# The ganesha node connects to the saunafs master server
# with the ip address 192.168.99.100:
#
# To work correctly, all that is required is an EXPORT
###################################################

EXPORT
{
    # Export Id (mandatory, each EXPORT must have a unique Export_Id)
    Export_Id = 77;

    # Exported path (mandatory)
    Path = "/";

    # Pseudo Path (required for NFS v4)
    Pseudo = "/";

    # Required for access (default is None)
    # Could use CLIENT blocks instead
    Access_Type = RW;
    Squash = None;
    Attr_Expiration_Time = 0;

    # Exporting FSAL
    FSAL {
        Name = SaunaFS;
        # The address of the LeilFS Master Server or Floating IP if using uRaft
        hostname = "192.168.99.100";
        # The port to connect to on the Master Server
        port = "9421";
        # How often to retry to connect
        io_retries = 5;
        cache_expiration_time_ms = 2500;
    }

    # Which NFS protocols to provide
    Protocols = 3, 4;
}

One important aspect to consider is the 'Name' value must be set to SaunaFS. Otherwise NFS-Ganesha will not use SaunaFS FSAL.

After finishing the setup of SaunaFS FSAL, start NFS-Ganesha. If your installation provides a systemd service, enable and start it with:

systemctl enable nfs-ganesha
systemctl start nfs-ganesha

If you installed Ganesha directly from upstream sources and no service file was installed, you can start it manually instead:

sudo ganesha.nfsd -f /etc/ganesha/ganesha.conf -L /var/log/ganesha.log

Connecting NFS clients to LeilFS clusters

Before connecting NFS clients to LeilFS clusters, make sure package nfs-common is already installed. This package contains programs like statd, showmount and mount.nfs that are needed for NFS clients to connect successfully.

For connecting NFS clients to one LeilFS cluster, we can use the following command:

mount -vvvv ganesha_server_ip:/ganesha_export /local_mountpoint

The option vvvv is optional and enables verbose mode to see whether the mount command was successful and other useful debug information.
The second parameter is the IP address assigned to the Ganesha server, followed by the export (defined at ganesha.conf) to which we want to connect.
The third parameter is the local mount point defined to access (locally) to the export defined at the ganesha.conf file.

Below, there is an example to connect a NFS client (locally) to NFS-Ganesha server installed at the same workstation:

mount -vvvv localhost:/ /mnt/export1/

The folder /mnt/export1/ is the directory under which the NFS share will be mounted on the client machine. This directory can be changed to any directory name.

Once NFS client is connected to the cluster, we can perform the following operations:

stat
readdir
create folders and files
remove dir and files
cat
symbolic links
change permissions
copy files and folders
rename files and folders

NFS protocol version, client's authorization, and multiple exports

NFS-Ganesha supports NFS v3, 4.0, 4.1, and 4.2. When connecting NFS clients to the cluster, it’s possible to define a specific version of NFS to be used by the client. Below there are some examples to connect NFS clients with different NFS versions:

NFS v3: mount -o nfsvers=3 localhost:/ /mnt/nfs3
NFS v4.1: mount -o v4.1 localhost:/ /mnt/nfs41
NFS v4.2 (default in Ubuntu 22.04): mount localhost:/ /mnt/nfs42

Another important step during configuration of an export is the list of clients authorized to access the export. The following example shows different ways to allow clients to connect to a given export:

EXPORT
{
    …
    CLIENT
    {
        #Clients = 192.168.208.236;
        #Clients = *;
        #Clients = 192.168.208.0/24;
        #Clients = mfsmaster;
    }
     …
}

The list of clients is usually defined inside a CLIENT section at the ganesha.conf file. The most frequent ways to authorize NFS clients are:

Specific IP address for a single client: Clients = 192.168.208.236;
All clients: Clients = *;
Specific subnet: Clients = 192.168.208.0/24.
Hostname of clients: Clients = nfsclient01;

NFS-Ganesha also allows to define multiple exports for the same namespace. The following example shows the definition of multiple exports in the same ganesha.conf file.

EXPORT
{
     Attr_Expiration_Time = 0;
     Export_Id = 1;
     Path = /export1;
     Pseudo = /e1;
     Access_Type = RW;

     FSAL
     {
          Name = SaunaFS;
          hostname = localhost;
          port = ${saunafs_info_[matocl]};
     }

     Protocols = 3, 4;
}

EXPORT
{
     Attr_Expiration_Time = 0;
     Export_Id = 2;
     Path = /export2;
     Pseudo = /e2;
     Access_Type = RW;

     FSAL
     {
          Name = SaunaFS;
          hostname = localhost;
          port = ${saunafs_info_[matocl]};
     }

     Protocols = 3, 4;
}

EXPORT
{
     Attr_Expiration_Time = 0;
     Export_Id = 97;
     Path = /;
     Pseudo = /e97;
     Access_Type = MDONLY;

     FSAL
     {
          Name = SaunaFS;
          hostname = localhost;
          port = ${saunafs_info_[matocl]};
     }

     Protocols = 4;
}

EXPORT
{
     Attr_Expiration_Time = 0;
     Export_Id = 99;
     Path = /;
     Pseudo = /e99;
     Access_Type = RO;

     FSAL
     {
          Name = SaunaFS;
          hostname = localhost;
          port = ${saunafs_info_[matocl]};
     }

     Protocols = 4;
}

The definition of multiple exports allows us to define different levels of access (RW, MDONLY, RO), which guarantee a fine-grained level of permissions for different folders in the namespace.

The way to connect to multiple exports is the same as before, we only need to define a NFS directory for each export.

NFS client

Installing NFS-Ganesha​

Installing and setting up SaunaFS FSAL​

Connecting NFS clients to LeilFS clusters​

NFS protocol version, client's authorization, and multiple exports​

Installing NFS-Ganesha

Installing and setting up SaunaFS FSAL

Connecting NFS clients to LeilFS clusters

NFS protocol version, client's authorization, and multiple exports