Standalone Drive Troubleshooting

Documentation | Release Notes

This page describes in detail how the components of the drive work for education and support. These instructions are for release 4.10.14 and above of the Windows standalone drive.

The drive runs as a “Filesystem in Userspace” (FUSE). As such, it does not need elevated permissions, and runs for a single user. The drive handles file system requests from applications, and maps them to the storage provider.

The drive runs as two processes. The Drive and the Drive Uploader. The Drive process handles requests from the FUSE drive such as open, write and rename. When an updated or created file is closed it's handed to the Uploader which sends the file to object storage.

Make sure both the Drive and the Drive Uploader are not running before reinstalling.

The drive runs as user application for a single user, as a mapped drive. Different users will run their own instance of the drive, typically with different credentials.

A Drive can also be licensed for Server installations, and installed as a service. Applications typically then access the drive via a UNC Path. Please contact sales@storagemadeeasy.com for more information.

When the Drive sees an open file request it checks object storage to see if the file exists and when it was last modified. If that latest version is not already in local cache the drive downloads it.

Read/write requests (block I/O) are mapped by the Drive into operations against cached files. When a file is closed through the drive it is passed to the Uploader.

Some requests, including directory listings, file deletions and file and folder renames and moves are synchronous. That is, the drive does not return control to the application until the request is completed. There may though be multiple requests in play at one time, even from the same application.

The Drive Uploader manages the upload of files to storage asynchronously. It can upload many files at the same time in parallel. It also handles retries, and can recover from machine restarts or crashes.

The drive uses a single cache for both reads and writes. This allows the same file to be written and then read immediately.

The cache location can be configured under Settings. It must be on a local drive. Since the cache directory is a folder you can also use standard tools to verify what's there, and if needed, to remove files.

Files that have been created but not uploaded are flagged with the OFFLINE attribute. These will appear in a directory list with parenthesis around their size.

C:\Users\steven\AppData\Local\IBM COS\Cache\marketing101> dir/s
08/07/2019  03:27 PM    <DIR>          .
08/07/2019  03:27 PM    <DIR>          ..
09/28/2017  08:40 PM   (2,625,098,990) Two_Gentlemen_of_Verona_2019.mp4
08/07/2019  03:25 PM        27,351,864 OfficeC2RClient.exe
           2 File(s)  2,652,450,854 bytes

If cache expiration is enabled an expiration purge task runs every 30 minutes. It will check modified and created time to determine if a file in cache should be deleted.

Files with the Online attribute will never be expired.

The Drive Uploader goes through a Recovery mode on startup. It scans the cache to identify files with an Online attribute that signifies that they have not been uploaded.

As an extra precaution recovery mode is also run once a day.

There are a number of logs that can help troubleshoot issues relating to the operation of the drive. Turn on the logs via the menu from the toolbar icon.

Log files are created only when the first applicable log entry is written. Log files are rotated after a certain size.

The Drive and Uploader Logs are focused on the interaction with the external object storage (over the network) and provide a great way to see how the system is performing in general as well as a way to trouble-shoot specific issues. Requests from the FUSE layer, such as block I/O and metadata are not logged. Cache hits for files and metadata (directory information) are not logged.

To troubleshoot specific requests Fiddler can also be useful.

Logs when the Uploader starts (or is restarted by the Drive).

2019-01-31 16:36:51:408  [TID 9292] Started C:\Program Files (x86)\IBM\Drive\IBMCOS.Drive.Uploader.exe

Logs start of download attempt, when an object is brought into cache. Shows thread identifier (TID).

2019-01-31 18:20:02:393  [TID 9596] \bucket1\folder1\filename.ext

Logs unsuccessful download attempts. Shows time in milliseconds [2567] and error message. Correlated with download.log.

2019-01-31 18:25:02:393  [TID 9596] [2567] \bucket1\folder1\filename.ext - The operation timed out [server.com:443]

Logs successful downloads. Shows time in milliseconds [245]. Correlated with download.log.

2019-01-31 18:25:05:393  [245] \bucket1\folder1\filename.ext

This log includes requests initiated to retrieve folder listings and the status of files. Windows resolves child subfolders by iterating through parent folders so you will see multiple entries even for one folder request on the drive. Directories listings are cached based on the Setting “Folder Refresh Timeout”.

2019-02-01 10:48:27:850  [TID 8884] \bucket1
2019-02-01 10:48:27:881  [TID 3936] \bucket1\images
2019-02-01 10:48:27:928  [TID 7352] \bucket1\images\2019
2019-02-01 10:48:28:209  [TID 5452] \bucket1\images\2019\01
2019-02-01 10:48:28:458  [TID 5452] \bucket1\images\2019\01\sunset.png

This log lists directory requests that failed. Time [18972] is shown in milliseconds.

2019-01-31 16:39:32:419  [TID 8544] [18972] \bucket1\images\2018 - The operation timed out [server.com:443]

This log lists directory requests that succeeded. Time [245] is shown in milliseconds.

2019-02-01 10:48:27:850  [TID 8884] [245] \bucket1
2019-02-01 10:48:27:881  [TID 3936] [145] \bucket1\images
2019-02-01 10:48:27:928  [TID 7352] [36] \bucket1\images\2019
2019-02-01 10:48:28:209  [TID 5452] [93] \bucket1\images\2019\01
2019-02-01 10:48:28:458  [TID 5452] [109] \bucket1\images\2019\01\sunset.png

This log is the same as the prior listed.log but logs after the internal directory cache has been updated.

Logs when the Drive sends a message to the Uploader.

Logs when the Uploader receives a message from the Drive and adds to it's in-memory upload queue.

Logs files the Uploader picks up during recovery to add to it's in-memory upload queue.

Logs start of upload. In the first iteration an upload will be attempted 5 times before being logged in failed.log.

2019-01-23 09:01:08:599  \\?\C:\Cache\bucket1\sunset1.jpg [12952 B] [attempt #1]
2019-01-23 09:01:46:777  \\?\C:\Cache\bucket1\sunset2.jpg [41735 B] [attempt #1]
2019-01-23 09:02:01:879  \\?\C:\Cache\bucket1\sunset1.jpg [38246 B] [attempt #2]

2019-01-11 14:58:54:804  \\?\C:\Cache\bucket1\sunset1.jpg - The operation timed out
2019-01-12 12:05:33:631  \\?\C:\Cache\bucket1\sunset2.jpg - The operation timed out
2019-01-15 09:20:27:761  \\?\C:\Cache\bucket1\sunset3.jpg - 
  The request has not succeeded.
  Response code: 408.
  Request Timeout
2019-01-15 10:13:42:515  \\?\C:\Cache\bucket1\sunset4.jpg - Uploaded, but cloud size is different than local size

Successful upload.

Successful upload after retrying.

Misc errors with Drive.

Misc errors with Uploader.