Amazon S3 Storage

Last updated on Aug 6, 2021

Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of websites. The service aims to maximize the benefits of scale and to pass those benefits on to developers.

The File Fabric enables easy access, management, use of Amazon S3 storage to anyone, not just developers.

The AWS GovCloud (US) is also supported. When adding the provider choose Amazon S3 GovCloud US Non-FIPS or FIPS.

See also Using Glacier and Glacier Deep Archive Storage.

See also Adding an S3 Compatible Cloud Provider

You can choose to add the the Amazon S3 Service to the File Fabric by first navigating to your Cloud Dashboard Menu>My Dashboard tab and then choosing the Add new Provider.

 Login to your Amazon Web Services Account. From the Dashboard view click on the account link and then click on the security credentials link. It is from here that you will be able to obtain the relevant keys needed to connect your Amazon S3 Account with the File Fabric.

To restrict the account's access to only required S3 operations and resources create an custom IAM policy and add to the Amazon user for the account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListAllMyBuckets",
                "s3:AbortMultipartUpload",
                "s3:RestoreObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:DeleteBucket"
            ],
            "Resource": "*"
        }
    ]
}

To restrict the account's access to a specific bucket, you could create a policy like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource":"arn:aws:s3:::thisbucketonly"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:RestoreObject"
            ],
            "Resource": "arn:aws:s3:::thisbucketonly/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        }
    ]
}

From the Wizard that is launched when clicking the “add new provider” from the DashBoard enter the Amazon S3 keys that you retrieved from the prior step and click 'continue'.

**If you have a problem authenticating consider re-generating your secret key from your Amazon Web Services Account.

After you enter your authentication details and these are accepted the File Fabric will discover any S3 buckets that are available. You can choose which buckets you wish to add to your Account, and which will be the default bucket**. As part of this process you can choose to create a new default bucket if you wish, and also choose the reason.

Any buckets you choose not to index / sync will not be available to be worked with an you would need to go back to the S3 settings from the DashBoard to add them to your account. This is also the case with any new buckets you add directly from S3.

** The default bucket is what is used for interactions with Smart folders.

After your authentication has been verified and you have chosen the buckets to work with you will be required to sync your meta data which enables the File Fabric to interact with your files. You can choose to do this directly in the browser in which case you need to keep the browser open until it completes, or you can choose to have the service do this server-side. You will be still notified in the browser but if you close the browser the meta-sync will continue.

7 Post Sync report

When the sync / index has completed you can choose to access the report of what was indexed.

At this point your files can be accessed and managed directly from the Cloud File Manager and also the different desktop and mobile access clients.

The S3 settings can be accessed from navigating to the 'Dashboard→Amazon S3' Settings link.

From here you can:

You can also choose to resync the meta-data of the provider

Changing Access Keys

You can also enter both the new Access Key and Secret Key here if you are rotating or Access Keys. The keys should be in the same AWS account with access to the same buckets.

Direct Download

Enabling Direct Download allows client applications and share links to download objects directly from the AWS servers (via signed URLs) rather than through the File Fabric.

Direct download is not supported when Customer-Provided Encryption Keys (SSE-C) are enabled, nor for files stored in Glacier or Glacier Deep Archive.

When you work with Amazon S3 files you can choose to do so from the web file manager or any of the access clients Apps.

When you first import the meta-data from Amazon S3 you will be asked to set a default bucket. A default bucket is what is used if you add data to any smart folders such as 'My Syncs' etc. When you add data to these folders then the data actually resides on your default S3 bucket that you chose at setup. For example if you added data to the smart folder 'My Syncs' then in your default S3 bucket there would be a folder created called 'My Syncs' where this information would reside.

Similarly if you create something in the root of “My Amazon S3 files” or whatever else you choose to call this we will automatically try to create a bucket, but if the bucket name is taken the default behaviour is to create a folder, and this folder would reside in your default bucket. If you want precise control over only adding a new bucket then navigate to your S3 Settings from your DashBoard and add a new bucket from here.

Any folders/files that you create in normal buckets are stored directly within the buckets on S3. The rules above only apply when using smart folders, which you can choose to use or not to use.

Also we don't do anything to your files, such as other S3 provider can do ie. we don't rename them or apply an encoding to the file name etc. Your files are stored with the same name and format as you upload them

We also keep additional meta-data as compared to S3. An example of this is the local timestamp. If you upload files to S3 using our desktop sync tools then we are able to keep the local timestamp which direct uploads to S3 do not.

For objects uploaded to S3 the metadata property Content-Type is added based on the file extension.

Restrictions

The Amazon S3 provider doesn't impose limits such as number of buckets, object size, number of parts (for multi-part upload) and length of object keys except where the S3 API is also limited.

Bucket names are also not restricted with the exception that bucket names with dots (periods) are not supported due to security issues with virtual-host-style addressing over HTTPS.

If S3 restricts an operation, and an error is returned to the File Fabric, an error will be returned to the client application.

Rate Limiting

Amazon S3 may limit the rate at which it processes requests. This page: File Fabric Handling of Rate-Limiting Storage Providers explains how the File Fabric responds to rate limiting.

This website uses cookies. By using the website, you agree with storing cookies on your computer. Also you acknowledge that you have read and understand our Privacy Policy. If you do not agree leave the website.More information about cookies