Adding an S3 Compatible Cloud Provider

Storage Made Easy is able to connect to many on-premise and cloud object storage providers through our Open S3 connector. The connector supports all advanced functionality including real-time synchronization, trash, locking, and mstream.

You can then easily manage and access object storage through the File Fabric desktop and mobile clients.

The first step is to choose to add the Open S3 provider to your SME Account either as a first step on activation or later from the Dashboard which is accessible from the top menu.

You need to grant the File Fabric access your S3 Compatible Cloud. To do this you need to enter an endpoint, an Access Key and a Secret Key. This information is used to request access to your S3 compatible Cloud. All authentication data is stored encrypted, with the encryption key being stored in a separate key server for security.

After you enter your authentication details and these are accepted the File Fabric will list any buckets that are available. You then choose which buckets you wish to manage through the File Fabric platform, and which will be the default bucket**. As part of this process you can choose to create a new default bucket if you wish.

Any buckets you choose not to index / sync will not be visible within the File Fabric. You would need to go back to the Open S3 settings from the Dashboard to add them to your account. This is also the case with any new buckets you add directly from the S3 compatible provider.

Once buckets have been selected the File Fabric creates a background task to index all the metadata, or if you have selected it, the content for indexing. This may take several minutes, or longer if you have millions of files.

** The default bucket is used for interactions with Smart folders.

• Cloud Refresh Mode - Determines whether folders are refreshed on demand using real-time refresh. Use this option when objects may be changed outside of the File Fabric.
• Use Server Side Encryption for Upload - Enables SSE if supported by the provider.
• Use Basic Encoding - Enable if requested by support
• Use multipart upload - Uses Multipart Upload APIs for large files for reliability and performance.
• Multipart upload size (MB) - Files larger than this will be uploaded using Multipart Upload APIS. Default minimum is 4096 MB.
• API Signature Version - Choose Version 2 or Version 4.
• Allow direct upload in client apps - Allows clients to upload to the storage provider using pre-signed URLs. The provider endpoint must be accessible by the client device.
• Support Trash - Older versions of files that are deleted (or updated depending on policy) are moved to trash rather than being deleted.

Choose Synchronize files to refresh File Fabric metadata after changes have been made to the storage outside of File Fabric.

S3 Compatible Clouds that are known to work with the Open S3 provider include:

• Eucalyptus Walrus
• Google Storage
• OpenStack Swift
• Nifty Cloud Storage
• Connectria
• Constant Cloud Storage
• Tiscali Storage Cloud
• GreenQloud
• SeeWeb Storage Cloud
• Riakcs
• Cloudian
• LunaCloud
• HostEurope
• Dunkel Storage

Once Clouds are added they can be used with the web file manager and also the Linux, Mac, Windows, iOS, Android, Windows Phone, and BlackBerry Apps.

Storage Made Easy also provides an S3 compatible interface that provides S3 access to any mapped clouds even if they do not support the S3 API directly. If you arrived at this page looking for this, please see this Wiki page.

The Open S3 provider doesn't impose limits such as bucket naming, number of buckets, object size, number of parts (for multi-part upload) and length of object keys except where the S3 API is also limited.

If the provider restricts an operation, and an error is returned to the File Fabric, an error will be returned to the client application.

The connector uses the following operations:

• ListBuckets
• CreateBucket
• HeadBucket
• DeleteBucket
• ListObjects
• HeadObject
• PutObject
• GetObject (including range reads)
• CopyObject (including x-amz-copy-source)
• DeleteObject
• CreateMultipartUpload
• CompleteMultipartUpload
• AbortMultipartUpload
• UploadPart
• UploadPartCopy

If MPU is not supported by the provider the connector will failover to non-MPU APIs.

The File Fabric will connect to the S3 compatible storage using the AWS Signature Version 4 Signing Process, or if not successful, Version 2. This can be changed in the provider settings.

Some S3 compatible storage providers may limit the rate at which it processes requests. This page: File Fabric Handling of Rate-Limiting Storage Providers explains how the File Fabric responds to rate limiting.

For most recoverable errors the File Fabric will retry transparently three times before returning an error to the client.