# Appliance Troubleshooting

#### Last updated on March 15, 2024

This document covers troubleshooting of the Nasuni Access Anywhere Server. The Server runs as a virtual machine that may be scaled out horizontally (run as multiple instances).

The virtual appliance is a hardened CentOS image with:
   * Access Anywhere components
   * MariaDB database server
   * Apache Solr search engine

The most common deployment scenarios are:
  * Single server with database - with Access Anywhere Engine and database server on a single instance
  * Single server separate database - The Access Anywhere Engine runs on one instance. The database runs separately either as an additional instance or through a database-as-as-service.
  * High availability - The Access Anywhere Engine runs on multiple instances behind a load balancer. A high availability database cluster or database-as-as-service is used.
  * The Apache Solr service is optional, but required to support content search. It should run on a separate machine instance.

## Prerequisites

Before you begin troubleshooting and checking Access Anywhere Server you should have the following information:

   * Server domain name
   * Are multiple servers being used?
     * Load balancer domain name
     * All server instance hostnames or IP addresses
  * Is Cloud FTP being used?
  * Is ClamAV being used?
  * Passwords:
    * smeconfiguser password(s)
    * root password(s)
    * database credentials (optional)

## Accessible from Network

The web server should start up automatically and be accessible when the appliance server starts. From a remote machine use a browser or the following command to test connectivity to the web server. Curl is available on Linux and Mac.

    curl -k https://hostname | head -n 20

If port 443 is not responding, and to validate network performance, run ping from a remote machine. Ping is available on Linux, Windows and Mac.

    ping hostname

## Command Line Access

If the server is reachable remote shell into the system as smeconfiguser.

Run this command from the command line of a machine that has the ssh utility installed (Linux or Mac), or run the equivalent using a Windows tool like putty.  You may be prompted about the authenticity of the host. If so, answer 'yes'. You will be prompted for the password.

    ssh smeconfiguser@hostname

On success you will see a Linux command prompt. Unless otherwise noted commands in this document can be run as smeconfiguser. 

To open a shell as root or change to the user smestorage use the command su. You cannot log into the machine remotely directly as root:

    su root
    su smestorage

### Connection Refused

If the password fails several times you will be locked out.

    ssh: connect to host 10.0.10.194 port 22: Connection refused

To verify, log in via the console as root and execute:

    iptables -L f2b-SSH -n

If your IP address is locked you can unlock via fail2ban (as root):

    fail2ban-client set ssh-iptables unbanip 192.168.1.1

## Appliance Logs

### Log

This is a general log file for the appliance.

    tail -f /var/www/smestorage/sitelogs/logits.txt

You should see the last few lines of the log file, and new lines should appear from time to time as the appliance is used. Lines containing the word "Error" indicate a possible problem with the way the appliance has been set up or is being used.

The tail -f command will run until you terminate it (Ctrl-c).

### Error Log

These files are created the first time an error is received:

    tail -f /var/www/smestorage/sitelogs/errorlogs.txt
    tail -f /var/www/smestorage/sitelogs/errorlogs_trace.txt

You may see the last few lines of the log file, and new lines may appear from time to time as the appliance is used. Lines containing the word "Error" indicate a possible problem with the appliance. The file <code>errorlogs_trace.txt</code> contains a full trace of errors in <code>errorlogs.txt</code>.

### Upload Log

This log provides information on upload threads including M-Stream.

    tail -f /var/www/smestorage/tmp/tempstorage/log.txt
    


### Email Log

    tail -f /var/www/smestorage/sitelogs/allemails.txt

Sent emails “To” addresses and “Subject”s are logged here.

### Cron Job Log

Cron jobs kick off housekeeping services and background tasks

    tail -f /var/www/smestorage/cron/log.txt



### Log Archive

An archive of logs can be found at:

    /var/www/smestorage/tmp/logsarchive

### Log Rotation

A log rotation and archive script runs under cron.  Logs are removed after 30 days.

    /var/www/smestorage/cron/logroller.pl

The configuration file is at:

    /var/www/smestorage/cron/config_logroller.conf

## Ports In Use

To resolve a port conflict or to determine what ports are in use by what service use:

    netstat -plnt

## CPU Check

Check CPU usage or check for a runaway process using top. Investigate processes maxing out CPU over three refreshes.

    top

In the third line, which is labelled "%Cpu(s):", the fourth number (labelled "id") shows the percentage of CPU cycles that are idle. If this number is less than 10% then your CPU is very busy. If t remains at less than 10% for more than a few seconds then your CPU may be overloaded. Sometimes this indicates that a program is in an error state.

## Memory Check

Look for memory issues with top.

    top

In the fourth line, which is labelled "KiB Mem :" or "Mem:" depending on the version of top, the fourth number (labelled "free") shows the amount of free memory in kilobytes. If this number is less than 150,000 then your server is probably low on memory.

## Disk Space Check

    df -h

In the "use" column, a value close to or at 100% indicates a severe lack of space. Generally a value above 89% indicates that space should be cleared.

## Disk Space procedure 

You can check the disk space by running the command

    df -kh
    
If a table has run out of memory you will see errors in the NAAError Logs

    /var/www/smestorage/sitelogs/logits.txt

If you have configured a notification email, then you will receive notification email with the errors.

If you ran out of diskspace please see: [[:server/determine_layout]]
 
    DB! Table './smestorage/TABLE' is marked as crashed and should be repaired
    Symptom - You open the configured NAA appliance url in a browser and see an empty page

## Action to Bring up the Appliance

### Increase Disk Size

To increase the diskspace on NAA appliance see the recipe to increase disk space: https://storagemadeeasy.com/wiki/server/appladmin

### Repair the Database

ssh into the appliance as smeconfiguser

Backup the database, this is the easiest way to find the crashed tables.

    mysqldump -u smestore -p --opt smestorage > smestorage.sql

IF YOU GET AN ERROR MESSAGE INDICATING A CRASHED TABLE:

ssh to appliance and run the following command

    mysql -u smestore -p smestorage

Enter the password

Make sure the database is smestorage 

    use smestorage

And then repair the table that has crases

    repair table <TABLE_NAME>

Go back to the database backup step until the backup completes without errors. 

### Delete compiled templates

NAAuses compiled templates. If disk space is low the templates can become corrupted. To fix this:

ssh in to appliance as smeconfiguser

Then sudo as root and then smestorage linux user. 

    su - root
    su - smestorage

Go to the templates directory

    cd /var/www/smestorage/public_html/smarty/site/templates_c

Delete all the compiled templates by executing the following command

    rm *.tpl.php

This should help you get your Appliance back online.

## Find Problem Files

This command finds the top 50 files above 10M:

    find / -xdev -type f -size +10M -exec du -sh {} ';' | sort -rh | head -n50
## Process Check

Check that the following services are healthy (more detail in sections below)

    systemctl status php-fpm
    systemctl status httpd
    systemctl status jetty
    systemctl status crond
    systemctl status mariadb   # if running locally
    docker ps | grep memcached_memcached

## PHP-FPM Service

If you see this error check status of PHP-FPM:

    The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

You can do this by issuing this command:

    systemctl status php-fpm

You'll see the real status of the service, it may have hung and the status will be active - either way it's a good idea to restart the PHP-FPM service. You will need root access (or use sudo).

    sudo systemctl restart php-fpm

### Logs

    /var/log/php-fpm/error.log

### Configuration

The configuration file is located here:

    /etc/php-fpm.d/www.conf
    
Please do not change settings without working with support. 

## Apache HTTP Server (Web Server)

Check that the Apache HTTP Server is running. su to root.

    systemctl status httpd

To start the HTTP Server

    systemctl status httpd

To stop the HTTP Server

    systemctl status httpd

### Configuration

HTTPD server configuration files are located in the following two directories:

    /etc/httpd/conf

    /etc/httpd/conf.d

### Logs

Apache Httpd server logs are located at:

    /etc/httpd/logs

    tail /etc/httpd/logs/access_log
    tail /etc/httpd/logs/filefabric-error_log
    tail /etc/httpd/logs/filefabric-access_log
    tail /etc/httpd/logs/ssl_accees_log
    tail /etc/httpd/logs/ssl_error_log
    tail /etc/httpd/logs/webdav.filefabric-error_log
    tail /etc/httpd/logs/webdav.filefabric-access_log
    tail /etc/httpd/logs/ssl_webdavfilefabric_log

## Memcache

If your memcached stops working or it hangs then this manifests itself as users being unable to upload files.

When you’ll try to upload something you will get a message:

**Can not find uploading process meta data.**

This means a record could not be added to memcached and because of that the upload failed.

Another indication of a memcached issue is that you may receive an email notification that a session write failed:

{{:appliance:session_not_initialized_email.png?600|}}

or see a message in the browser that the storage session could not be initialized:

{{:appliance:session_not_initialized_browser.png?600|}}

and Access Anywhere may not be usable. 

To solve any of these memcached issues, as root issue the following command:

<code>
yum install sme-containers-memcached
cd /var/www/smestorage/containers/memcached
docker-compose restart
</code>

## Jetty / Apache Solr / PDF Annotation

The Jetty service is used for Apache Solr and PDF Annotation. It runs as a Java process, by default listening on localhost port 7070.

Check the health of jetty using the command:

    systemctl status jetty

You should see a few lines of output including one that says, "Active: active (exited) ". If you do not then the service is not running. This will prevent content search from working.

To check that Apache Solr is running and responsive on the appliance run:

    curl -u solr:drom6etsh9Onk "http://127.0.0.1:7070/sme/select?q=the&start=0&rows=100&wt=json&indent=true"

### Configuration Files

    /home/sme/sme_jetty/start.ini

    /smedata/sme_solr/solr.xml

See https://docs.storagemadeeasy.com/server/solr for more information.
### Logs

    /home/sme/sme_jetty/logs/solr.log
    
    /var/solr/logs/solr.log

### Production

Note: For production Apache Solr should be running on a separate instance to Access Anywhere Server (Web Tier).
### Access Solr Admin GUI Remotely

SSH can be used to create an HTTP tunnel to access the Solr admin from another machine. For example,

   ssh -L 8983:files.example.com:8983 smeconfiguser@files.example.com

From a browser:

    http://localhost:8983/solr 
    
The default username is "solr" with password drom6etsh9Onk.

## Cloud FTP Service (CloudFTP/CloudFTPS)

See [[server/cloudftp-configuration]] for more information.

### Client Testing

    sftp -v  user@hostname

## Local FTP Service

For production use this service should be removed along with the demo accounts.

An appliance FTP server listens by default on IP Address 127.0.0.1 and port 2001. It is used for default storage for the clouduser.

Status of local ftp service

    systemctl status vsftpd

Start FTP Server

    systemctl start vsftpd

To stop

    systemctl stop vsftpd

### FTP Server Configuration

    /etc/vsftpd/vsftpd.conf

## Cron Service

The cron service executes cron jobs that roll logs and kick off periodic tasks for the system such as daily maintenance tasks. These scripts should be run only once in a multi-server environment.

For version 1705.00 and above cron runs on all instances using cronmutex.php to make sure only one is executed:

    php /usr/bin/cronmutex.php default 716f3 900 && /var/www/smestorage/cron/scheduler_daily.pl

Check the health of cron using the service command:

    systemctl status crond

You should see a few lines of output including one that says, "Active: active (running) ". If you do not then the service is not running. This will prevent some functions from working.

If it is not then you should enable it:

    systemctl start crond

### Logs

A log of cron jobs being started can be found at:

    /var/log/cron

A detailed log of the task manager can be found at:

    /var/www/smestorage/cron/log.txt

### Configuration

To see cron jobs for a server:

    crontab -u smestorage -l      

### Note

These cron jobs use scripts in:

    /var/www/smestorage/cron
    /var/www/smestorage/config/cron/config.conf

To see crontab jobs run as root (currently only freshclam):

    cat /etc/crontab            

## CloudDAV

CloudDAV is our implementation of WebDAV on top of Access Anywhere. It runs as a CGI script from /var/www/smestorage/webdav_html/cgi-bin.

### Log

    /var/www/smestorage/webdav_html/cgi-bin/log.txt

### Configuration

    /var/www/smestorage/config/webdav_html/configuration

## Cloud S3

Cloud S3 is our implementation of an Amazon S3 compatible API on top of Access Anywhere.

### Log

    /var/www/smestorage/s3_html/cgi-bin/log.txt

### Configuration

    # See <VirtualHost *:80>
    #  DocumentRoot /var/www/smestorage/s3_html
    /etc/httpd/conf/httpd.conf 


## ClamAV Virus Scanner

The Antivirus scanner ClamAV is included with the appliance and can be used to check all uploaded files.

In order to be used virus scanning must be enabled on a per-organization (tenant) basis through Organization Policies. Scanning of individual uploaded files can be verified through the audit log if logging of File add/updates is turned on.

The ClamAV process is called <code>/usr/sbin/clamd</code> and runs as a system daemon. When the daemon is running it creates a filesystem socket that is used to communicate to and from the file upload process.

Check the health of the ClamAV scanner using the service command:

    systemctl status clamd@scan

In High Availability configurations each appliance leverages its local copy of ClamAV.

### Error Messages

If a file is uploaded, antivirus scanning is enabled, and the daemon is not running, the user will see the following message:

> Uploading of 1 files failed
>
> [Restart] [Cancel]
>
> Seems file is not uploaded. Uploading in progress?
>
> [Close]

Verify scans are being successful through the audit trail. The policy “Audit File add/update” must be enabled:

File sme-solution-brief.pdf uploaded to My Cloud files/mybucket. Scanned with antivirus ClamAV 0.99.2/24143/

### Log

    /var/log/sme-clamd.log

### Configuration

    /etc/clamd.d/scan.conf

    https://linux.die.net/man/5/clamd.conf 

### ClamAV Antivirus Database Updater

Virus definitions are updated once an hour (see /etc/crontab) with freshclam. To check connection to the online database (and update definitions) run as root:

    freshclam

### Log

    /var/log/freshclam.log

### Configuration

    /var/www/smestorage/config/clamd/freshclam.conf

## Email

The appliance includes an email server

    /var/log/maillog

## License (Access Anywhere)

This error on attempted login indicates problems with the license key including not present or expired.

> Sorry, Organization accounts are not supported. No valid key. Contact with your administrator.

The license is configured for each appliance through the Appliance Administration interface under Settings > License Key. This is reached by logging in as the appladmin user at https://hostname which you can do without a valid license.

The appliance license can also be viewed and changed from within the appliance at:

    /var/www/smestorage/config/public_html/license.txt

High Availability: The license must be configured on every instance.

## Version

There are several ways to check the version of the appliance. 

### Appliance Administration

Log in as the appliance admin. From the hamburger menu under the menu “Admin” see the appliance version and build number.

    System version: 1803.02

    Version build: 2018022700008
    
(The hotfix number (.xx) is only shown for versions 1803.00 and above) 

### Command line

1) From the shell use the System Package Manager

   yum info sme-ff-filefabric.x86_64

2) From the database

   SELECT * FROM smestorage.se_version;

3) From the shell as smeconfiguser run the alias:

   smeversion

## Upgrade Backups

    /var/www/smestorage/patches

Keeps copy of public_html after upgrades.

## Database Service

If the database is running locally check the service is running:

    systemctl status mariadb

You should see a few lines of output including one that says, "Active: active (running) ". If you do not then the service is not running. This will prevent the Cloud File Manager from working.

You can log into the local database through:

    mysql

You should be successful or see an error message like"ERROR 1045 (28000): Access denied for user 'smeconfiguser'@'localhost' (using password: NO)".

If you do not then the database is not accessible from this machine. For remote databases this may be a network issue.

### Local Database Service

If there is something wrong with the MariaDB Server you will most likely see this page when attempting to access the cloud file manager:

> It seems we encountered a problem. Please contact support and provide as much details as possible as to how this occurred.
>
> Thanks, and apologies for any inconvenience.
> 
> Please first check the mysql status:
>
>     systemctl status mariadb

If the service is up and running it is likely there are some corrupted tables due to a power outage. You can see next step how to fix them.

Or you can try to restart the service – after restart mysql will check the state of tables and will try to repair them

    systemctl restart mariadb

### Backup

You can backup the database using the following command:

    mysqldump smestorage >smestorage.sql

### Configuration

    /etc/my.cnf

## Fail2Ban

The NAA Appliance ships with a customized version of Fail2Ban (http://www.fail2ban.org/). 

Fail2Ban scans logs file for malicious patterns ie. DoS attacks, too many password failures, SSH logins, seeking exploits, trying to scan for download links etc. 

If a malicious pattern is detected it automatically updates the firewall rules to reject IP addresses for a specified amount of time (10 minutes). Fail2Ban is constantly working and scanning providing extra protection for the appliance.

### Log

    /var/log/fail2ban.log

### Unbanning an IP Address

Once you are the root user we'll need to find the IP address that was banned, and then un-ban it. To do this run :

    iptables -L f2b-SSH -n

In that list you may see your IP address. With that IP address we then want to run (swap the IP address with your one):

    fail2ban-client set ssh-iptables unbanip 192.168.1.1

Your IP address should now be unbanned.

https://www.storagemadeeasy.com/wiki/server/bestpractices/ 

## Appliance Backups

We recommend customers use tools from the hypervisor vendor or third-parties to backup the appliance and database.