Containerized CloudFTP Configuration

Last updated on: 9 February 2022
Applies to File Fabric versions: 2106.00 and above.

CloudFTP provides FTP, FTPS and SFTP file transfer services to any of the storage attached to the File Fabric regardless of the storage type. For example, with CloudFTP files can be uploaded to or downloaded from an Amazon S3 File Fabric provider using SFTP.

Until version 2106 of the File Fabric CloudFTP was delivered as a native service running directly under the File Fabric VM's operations system. Starting with version 2106, CloudFTP is delivered as a containerized service that replaces the older, native CloudFTP service. This document covers migration to and configuration of the containerized version of CloudFTP.

Upgrading the File Fabric When CloudFTP is Already in Use

If you are running a version older than 2106 your native CloudFTP service will be replaced by the new containerized version of CloudFTP. To use the new CloudFTP you will have to start it once. From then on it will run and restart automatically unless you choose to stop it manually (more on this below).

When you start the new CloudFTP for the first time, a new configuration will be generated automatically. Depending on how you were making CloudFTP available over the network and other details of your configuration, the newly generated configuration may be usable without change or you may have to adjust the configuration manually.

After the new configuration has been generated automatically, you should compare the new configuration with the configuration(s) - FTP (if you are using FTP, FTP with TLS or FTPS), SFTP or both - that were in use prior to the upgrade. If any variables in your old configuration has a non-blank value that is different from the value it has in the new configuration, replace the value in the new configuration with the value from the old configuration.

Your pre-upgrade FTP configuration is here:

/var/www/smestorage/ftpserver/ftpserver.conf

Your pre-upgrade SFTP configuration is here:

/var/www/smestorage/config/ftpserver/sftpserver/sftpserver.conf

Automatic Configuration of CloudFTP Services

Each of the CloudFTP file transfer services requires a configuration. For many customers the configuration created by automatic configuration, which happens when CloudFTP is started the first time, will be sufficient and no additional (manual) configuration will be required. In some cases however, manual configuration is required.

Automatically generated configurations should be inspected to confirm their suitability even when no manual configuration appears to be needed.

Automatic configuration with little or no need for additional (manual) configuration will be sufficient in most cases when the following conditions are met:

  • FTP and SFTP (but not FTPS) are to be enabled.
  • The File Fabric node is attached directly to the network over which it will be accessed or, if a network device such as a firewall connects the File Fabric host to the network over which it will be accessed, then the File Fabric's address is NATed or PATed directly to an external address using a one-to-one mapping.
  • The File Fabric node has egress access to the public internet.
  • The default ports and ranges for FTP and SFTP are acceptable.

To trigger automatic configuration, start CloudFTP for the first time by running the following as the root user:

cd /var/www/smestorage/containers/cloudftp/
docker-compose up -d 

Automatic configuration only occurs the first time that CloudFTP is started. Should you ever wish to return CloudFTP to its not-yet-configured state, contact SME Suport.

Since appliance version 2106.00 the CloudFTP file transfer services (FTP, FTPS, SFTP) are off by default for new installations and for upgrades. Upon starting CloudFTP, FTP and SFTP will become enabled. Additional steps are required to enable FTPS.

The CloudFTP file transfer services, their default ports and their default states when CloudFTP has been started without additional configuration are:

Protocol Port Default state Notes
FTP 21 Enabled Passive mode enabled, TCP Ports 20000 - 20010
FTP w/TLS 21 Enabled Same as FTP, with Self Signed Certificates (FTP Explicit)
FTPS 990 Disabled Same as FTP w/TLS
SFTP 2200 Enabled SSH File Transfer Protocol uses default RSA key

The File Fabric's firewall is pre-configured to support use of these services over these ports.

Stopping and Starting CloudFTP's File Transfer Services

CloudFTP is managed by the root user. All commands must be run as root from the following directory:

/var/www/smestorage/containers/cloudftp

To start the file transfer services, run:

docker-compose up -d

Once CloudFTP has been started for the first time it will continue to run unless it it stopped manually and it will restart on reboots. Starting CloudFTP manually is only required when the service is being switched on for the first time and after it has been brought down manually.

To stop the CloudFTP file transfer services, run:

docker-compose down

To restart the services, run:

docker-compose restart

Enabling and Disabling Individual File Transfer Services

CloudFTP is a wrapper service that turns the FTP, FTPS and SFTP file transfer services on and off. You can explicitly control which of the three will be turned on by CloudFTP by setting the value of the:

disable_<service_name>

variable for each of the three services in the appropriate configuration file (configuration files are discussed in later sections). A value of '0' disables the file transfer service; a value of '1' enables it. Restart CloudFTP after changing the value of any of those variables to ensure that the service change takes effect.

Adding or Changing Domain Names

You can use the File Fabric's domain name to access any or all of the file transfer services. You can also add different domain names or FQDNs for any or all of the file transfer services. For example, if your File Fabric i's domain name is:

  • myfilefabric.com

you might choose to add:

  • ftp.myfilefabric.com

for FTP and/or:

  • filefabricftps.com

for FTPS.

To use a new domain name just add type A or CNAME record to DNS associating the new domain name with your File Fabric. Any valid domain name can be used; the file transfer services do not validate domain names.

Manual Configuration

If any of the criteria from the automatic configuration section above is not met, it will be necessary to manually update the configuration.

FTP, FTP with TLS and FTPS Configuration


FTP Configuration File


To access the FTP configuration file, which is used to configure FTP, FTP with TLS and FTPS, log into the File Fabric and elevate to root.

The file ftpserver.conf contains many FTP, FTP w/TLS, and FTPS settings. It can be found at:

/var/www/smestorage/containers/cloudftp/configs/ftpserver.conf

Defaults:

Settings Details
ftp_server_ip=xxx.xxx.xxx.xxx The listening IP address interface of the FTP services. Generally this is 0.0.0.0
port=21 Port for FTP
FTPISport=990 Port for FTPS
smeserver=IP of File Fabric Web API (Defaults to containers host)
countprocesses=20 Limits the total number of connections
ftp_timeout=180 Timeout in seconds for connections where there are no activity
pathToSSLkey=/etc/pki/tls/private/localhost.keyPrivate key certificate. These are self-configured
pathToSSLcert=/etc/pki/tls/certs/localhost.crt Public key certificate. These are self-configured
limitConnectionsForOneUser=5 Limits the amount of connections one user can open
min_port=20000Minimum port # for Passive Mode
max_port=20010Maximum port # for Passive Mode
max_speed_write_to_disk=0 This reduces the writing to disk speed on upload. Specified in bytes per second
max_download_speed=0 Limits the download speed on read access. Specified in bytes per second.
max_upload_speed=0 Limits the upload speed. Specified in bytes per second.
maximumlimitsizeupload=10737418240 Limits the maximum file size for upload allowed. Specified in bytes.
tmpfolder=/scratch/cloudftp Upload location on disk.
timeoutForAcceptingConnections=180 Timeout of service when awaiting a data connection
disable_ftp=0 Disables or enables the FTP ability
disable_ftps=1 Disables or enables the FTPS ability
ssl_version=TLSv1.1+TLSv1.2 Controls the supports SSL Versions
ssl_cipher_list=ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2:!RC4:!3DES Controls the available Ciphers
external_ip_for_passive_mode=1.2.3.4 The External IP address for clients to connect to for Passive mode
debug=100 Debug level

Log File

Since the CloudFTP services log to stdout (standard-out), log files can be observed with:

docker-compose logs cloudftp

To obtain some most recent number of lines, for example the last 200 log lines, run

docker-compose logs --tail=200 cloudftp

Setting the IP Address for FTP and FTPS

The main element of automatic configuration is detection of the server's user facing (that is, public) IP address. This IP address is needed by CloudFTP to allow FTP clients to open Passive connections with the FTP or FTPS file transfer services.

The automatic configuration process uses a public service to detect the File Fabric's public IP address. It will run the following command.

dig +short myip.opendns.com @resolver1.opendns.com

This command will timeout or fail if it cannot reach the File Fabric. Also, in certain configurations it can return an incorrect IP address.

If it is necessary to set the Passive IP address manually, working as root edit the FTP server configuration file as follows:

vim /var/www/smestorage/containers/cloudftp/configs/ftpserver.conf

Find the line “external_ip_for_passive_mode” like follows and adjust the IP address

external_ip_for_passive_mode=1.2.3.4

Save, and exit the file, and relaunch the CloudFTP services as follows:

cd /var/www/smestorage/containers/cloudftp
docker-compose restart

Advanced FTP and FTPS Setup

Systems publicly exposing FTP based protocols might need additional setup to meet security requirements. Sme common types of changes are detailed below; for help with other advanced changes contact SME Support.

Custom Certificates

FTP with TLS and FTPS clients are not as strict as web browsers when using self signed certificates. Regardless, if a client requires a properly signed certificate the process is as follows:

Move the certificate and private key to the following locations:

/var/www/smestorage/containers/cloudftp/certs/localhost.crt  (certificate)
/var/www/smestorage/containers/cloudftp/certs/localhost.key  (private key)


Then restart CloudFTP.

FTP Passive Mode through NAT/PAT

FTP Passive Mode requires that the FTP server sends the client the port and IP address of File Fabric. When the File Fabric is secured behind a public firewall the internal IP address will most likely not match the public IP address. Set up passive mode as follows:

  1. Update the following entry in /var/www/smestorage/containers/cloudftp/configs/ftpserver.conf

    a. external_ip_for_passive_mode=xxx.xxx.xxx.xxx

  2. Restart the CloudFTP service
FTP/FTPS Port Number Changes

FTP and FTPS are configured to run on the standard default ports listed above. These can be changed if required.

To change the port numbers, edit the docker-compose yaml file:

vim /var/www/smestorage/containers/cloudftp/docker-compose.yml

Change the public ports to those that you require, for example to change the FTP port from 21 to 8821 find the ports section and edit both values corresponding to the new port number.

    ports:
      - "8821:8821"
      - "990:990"
      - "20000-20010:20000-20010"

After saving the docker-compose.yml, edit the FTP server configuration file:

/var/www/smestorage/containers/cloudftp/configs/ftpserver.conf

to reflect the port as well, for example:

# Port where CloudFTP Should Be Listening
port=8821


Finally, restart the CloudFTP services and verify the configuration.

Note: External firewalls, load balancers and/or NAT/PAT configurations may also need to be updated to pass traffic on the new port(s) to the Enterprise File Fabric.

FTP and FTPS Rate Limiting

The rates at which FTP and FTPS write to disk, downloads data and upload data can be limited through configuration. Update the following entries in ftpserver.conf and restart CloudFTP service to enable rate limiting. Value are in bytes per second:

max_speed_write_to_disk=2097152    
max_download_speed=3145728  
max_upload_speed=3145728
FTP and FTPS User Connection Limiting

The maximum number of concurrent FTP and FTPS connections allowed for each user can be limited through configuration. Update the following entry in ftpserver.conf and restart CloudFTP service to enable rate limiting.

limitConnectionsForOneUser=5


See list further up in this page above for additional configuration settings.

SFTP Configuration


SFTP Configuration Files

To access the sftp configuration files log into the File Fabric and elevate to root.

The file sftpserver.conf contains many SFTP settings. It can be found in: /var/www/smestorage/containers/cloudftp/configs/sftpserver.conf

Defaults:

Settings Details
ftp_server_ip=0.0.0.0Interface addresses listening for sftp
port=2200 Default port for SFTP
smeserver=IP of File Fabric Web API (Defaults to containers host)
pathToKey=./keys/sshhostrsakey|RSA Private Key mapped to container| |pathToCert=./keys/sshhostrsakey.pubRSA Public Cert mapped to container
tmpFolder=/scratch/cloudsftpScratch Folder
countprocesses=30Limit of total connections
timeout=360SFTP No Activity Timeout
connectiontimelimit=60Max time for SFTP client to establish a connection
maximumlimitsizeupload=10737418240Max SFTP upload file size in bytes
max_speed_write_to_disk=2097152
max_upload_speed=3145728Limit upload speed (0 - no limit)
max_download_speed=3145728Limit download speed (0 - no limit)
limitConnectionsForOneUser=5Connections per user
SMALL_FILE_SIZE=1048576Files below this size do not use scratch
debugmode=0Enable/disable debugging
disable_sftp=0Enable/disable SFTP
supported_ciphers =Limit supported ciphers (blank = all ciphers supported)
supported_macs = Limit supported ciphers (blank = all ciphers supported)


Log File

Since the CloudFTP services log to stdout (standard-out), log files can be observed with:

docker-compose logs cloudsftp

To obtain the last 200 log lines, run

docker-compose logs --tail=200 cloudsftp


SSH Keys

The SFTP service self-generates SSH keys in order to serve the SFTP connection.

In most cases there is no need to be concerned by this. If you would like to supply your own SSH keys, these can be dropped in at the following location:

/var/www/smestorage/containers/cloudftp/keys/ssh_host_rsa_key   (private key)
/var/www/smestorage/containers/cloudftp/keys/ssh_host_rsa_key.pub   (public key)

Unique SSH Keys are generated on first startup, so you do not necessarily have to regenerate the keys.

If you want to generate the keys, delete the existing keys from /var/www/smestorage/containers/cloudftp/keys/ and restart the CloudFTP service. New keys will be created in this location automatically.


SFTP Default Port Number Change

By default SFTP is set to use port 2200. This can be changed to another port, but to change SFTP to the its well known value of port 22, SSH must be moved to another port.

Changing the SSH Port

As root user edit the following file: /etc/ssh/sshd_config Uncomment the line highlighted in yellow and change port number to a new number such as 2222

# If you want to change the port on a SELinux system, you have to tell
# SELinux about this change.
# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
#
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

After saving the file run the following command to let SELinux know of the change. Substitute the selected port number for the text in yellow ( #PORTNUMBER)

semanage port -a -t ssh_port_t -p tcp #PORTNUMBER 

Changing port numbers also requires firewall changes to IP tables. Edit /etc/sysconfig/iptables

Change the entry highlighted in yellow to the new SSH port.TODO Yellow
Change the entry highlighted in red to the new SFTP port TODO Red

-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 990 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --sport 1024: --dport 1024: -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --sport 1024: --dport 20001:20100 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 990 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 2200 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

Restart iptables and SSH with the following command:

systemctl restart iptables
systemctl restart sshd

Before moving on, open a new ssh connection to the file fabric using the new port number. Ensure connectivity is functioning on the new port before closing the existing session or moving on.

Change SFTP Port Number

Edit docker-compose.yml in /var/www/smestorage/containers/cloudftp/
Change line with SFTP port 2200 to the new value and save.

    ports:
      - "2200:2200"

Next edit the SFTP config file in /var/www/smestorage/containers/cloudftp/configs/sftpserver.conf and update the port number there as well.

Finally, restart CloudFTP services as described above and test service.

External Firewalls, Load Balancers and/or NAT/PAT records may also need updating to direct traffic from the Internet to the new internal port number.


SFTP Rate Limiting

The rates at which SFTP writes to disk, downloads data and uploads data can be limited through configuration. Update the following entries in sftpserver.conf and restart CloudFTP service to enable rate limiting. Value are in bytes per second:

max_speed_write_to_disk=2097152    
max_download_speed=3145728  
max_upload_speed=3145728


SFTP User Connection Limiting

The maximum number of concurrent SFTP connections allowed for each user can be limited through configuration. Update the following entry in sftpserver.conf and restart CloudFTP service to enable rate limiting.

limitConnectionsForOneUser=5

See list further up in this page above for additional configuration settings.

Scratch Space Configuration

FTP, FTPS and SFTP uploads all may sometimes require scratch space. Scratch space is needed if the uploaded client cannot or does not tell CloudFTP the size of the file that will be uploaded, for example when the client is a camera that is streaming a live recording. Many popular clients don't inform the server (CloudFTP in this case) of the size of the file that is being uploaded even when that information is available to them.

Available scratch space should be large enough to accommodate the largest permitted upload file size multiplied by the number of permitted concurrent uploads (countprocesses * maximumlimitsizeupload).

Scratch space is configured in the service configuration files like this: for SFTP (note the uppercase 'F'):

tmpFolder=<path-to_temp_folder>

and like this for FTP and FTPS:

tmpfolder=<path-to_temp_folder>

By default these are set to /scratch/cloudftp. To increase available disk space for scratch you can set the path to a different location or you can optionally mount a large volume at the default location.