Adding an S3 Compatible Cloud Provider
Last updated on Oct 24, 2023
The File Fabric is able to connect to many on-premises and cloud object storage providers through the Open S3 connector. The connector supports all advanced functionality including real-time synchronization, trash, locking, and M-Stream™ File Transfer Acceleration.
You can then easily manage and access object storage through the File Fabric desktop and mobile clients.
Choosing the Open S3 Provider
The first step is to choose to add the Open S3 provider to the File Fabric either as a first step on activation or later from the Dashboard which is accessible from the top menu.
You need to grant the File Fabric access your S3 Compatible Cloud. To do this you need to enter an endpoint, an Access Key and a Secret Key. This information is used to request access to your S3 compatible Cloud. All authentication data is stored encrypted, with the encryption key being stored in a separate key server for security.
After you enter your authentication details and these are accepted the File Fabric will list any buckets that are available. You then choose which buckets you wish to manage through the File Fabric platform, and which will be the default bucket**. As part of this process you can choose to create a new default bucket if you wish.
Any buckets you choose not to index / sync will not be visible within the File Fabric. You would need to go back to the Open S3 settings from the Dashboard to add them to your account. This is also the case with any new buckets you add directly from the S3 compatible provider.
Once buckets have been selected the File Fabric creates a background task to index all the metadata, or if you have selected it, the content for indexing. This may take several minutes, or longer if you have millions of files.
** The default bucket is used for interactions with Smart folders.
- Cloud Refresh Mode - Determines whether folders are refreshed on demand using real-time refresh. Use this option when objects may be changed outside of the File Fabric (ie. the storage is being used in a bi-modal fashion).
- Use Server Side Encryption for Upload - Enables SSE if supported by the provider.
- Use Basic Encoding - Enable if requested by support
- Use multipart upload - Uses Multipart Upload APIs for large files for reliability and performance.
- Multipart upload size (MB) - Files larger than this will be uploaded using Multipart Upload APIS. Default minimum is 4096 MB.
- API Signature Version - Choose Version 2 or Version 4.
- Allow direct upload in client apps - Allows clients to upload to the storage provider using pre-signed URLs. The provider endpoint must be accessible by the client device.
- Support Trash - Older versions of files that are deleted (or updated depending on policy) are moved to trash rather than being deleted.
Choose Synchronize files to refresh File Fabric metadata after changes have been made to the storage outside of the File Fabric.
S3 Compatible Clouds
S3 Compatible Clouds that are known to work with the Open S3 provider include:
- Backblaze B2 Cloud Storage - Backblaze Provider also available
- HyperStore Object Storage - Cloudian Provider also available
- Google Storage - Google Storage Provider also available
- NetApp StorageGRID - requires File Fabric 2006.03
- OpenStack Swift
The Open S3 provider doesn't impose restrictions of limits on bucket naming, number of buckets, object size, number of parts (for multi-part upload) and length of object keys except where the S3 API is also restricted.
If the provider restricts an operation, and an error is returned to the File Fabric, an error will be returned to the client application.
S3 Compatible APIs
The connector uses the following S3 operations:
- GetObject (including range reads)
- CopyObject (including x-amz-copy-source)
- CreateMultipartUpload *
- CompleteMultipartUpload *
- AbortMultipartUpload *
- UploadPart *
- UploadPartCopy *
Note: * If MPU is not supported by the provider the connector will failover to non-MPU APIs.
Signing API Requests
The File Fabric will connect to the S3 compatible storage using the AWS Signature Version 4 Signing Process, or if not successful, Version 2. This can be changed in the provider settings.
Some S3 compatible storage providers may limit the rate at which it processes requests. This page: File Fabric Handling of Rate-Limiting Storage Providers explains how the File Fabric responds to rate limiting.
For most recoverable errors the File Fabric will retry transparently three times before returning an error to the client.