Jibe and Cloudian

Jibe is able to receive events from Cloudian to identify S3 objects that have been modified outside the Access Anywhere Server and sync them with Access Anywhere metadata.

For general information see Jibe Documentation.

The Jibe SQS Source connects to a Cloudian SQS queue to retrieve S3 notification events created by Cloudian on change.

Sequence:

1. S3 API - Applications call Cloudian S3 APIs to create and delete objects.
2. S3 Events - Cloudian S3 buckets send S3 Event notifications to a Cloudian SQS queue.
3. Message - Jibe consumes messages from the queue, each of which contains one or more S3 Event notifications.
4. Request Sync - Jibe asks the Access Anywhere server to resync objects.
5. Object Sync - Access Anywhere verifies the object status with S3 and updates its metadata.

To configure Jibe for Cloudian you will need:

• Cloudian configured for SQS.
  • You will need an access key and secret, and endpoints for SQS and S3.
• Access to the AWS Command Line Interface. You can use from a desktop. Upgrade to the latest version.

Jibe needs an Access Anywhere account with Administrator permission in order to synchronize objects. (For SQS this does not need to be the Org Administrator.)

A bucket can send events to one queue. One queue may receive events from multiple buckets. Jibe can consume events from multiple queues. Jibe polls queues for new messages once every 20 seconds (by default).

You can create queues and bucket notifications through the aws cli.

Create a new access key and secret that is used

Create an AWS profile called “Cloudian” and define your access key and secret.

aws --profile Cloudian configure

You will also need to add a region for the AWS CLI to work. You can use “region1”.

To create a queue with the name “jibe-source-activity” use the endpoint of your Cloudian SQS service:

aws --profile Cloudian --endpoint-url "http://sqs.company.com:18090" sqs create-queue --queue-name jibe-source-activity

This returns the QueueUrl which will be used in other functions.

{
    "QueueUrl": "http://sqs.region1.cloudian.com/021bafbcfb00b80c2014e8b004594af6/jibe-source-activity"
}

Note: To find the Queue URLs of existing queues use the command

aws --profile Cloudian --endpoint-url "http://sqs.company.com:18090" sqs list-queues

To remove a queue you'll use the Queue URL For example,

aws --profile Cloudian --endpoint-url "http://sqs.company.com:18090" sqs delete_queue --queue_url "http://sqs.region1.cloudian.com/021bafbcfb00b80c2014e8b004594af6/jibe-source-activity"

You'll need the QueueArn to create an S3 notification.

aws --profile Cloudian --endpoint-url "http://sqs.company.com:18090" sqs get-queue-attributes --queue-url "http://sqs.region1.cloudian.com/021bafbcfb00b80c2014e8b004594af6/jibe-source-activity" --attribute-names QueueArn

will return the QueueArn

{
    "Attributes": {
        "QueueArn": "arn:aws:sqs:fmt:021bafbcfb00b80c2014e8b004594af6:jibe-source-activity"
    }
}

Use the QueueArn to create a queue_config.json file like this:

{
    "QueueConfigurations": [
        {
            "Id": "jibeevents",
            "QueueArn": "arn:aws:sqs:fmt:021bafbcfb00b80c2014e8b004594af6:jibe-source-activity",
            "Events": [
                "s3:ObjectCreated:*",
                "s3:ObjectRemoved:*"
            ]
        }
    ]
}

Then using the Cloudian S3 endpoint (different to the SQS endpoint) you can set a subscription for mybucket.

aws --profile=Cloudian --endpoint="http://s3.company.com" s3api put-bucket-notification-configuration --notification-configuration file://queue_config.json --bucket mybucket

Check the current configuration for a bucket with:

aws --profile=Cloudian --endpoint="http://s3.company.com" s3api get-bucket-notification-configuration --bucket mybucket

Continue for additional buckets that will be monitored.

See Installation.

Change the endpoint and credentials in /root/config/jibe-config.json to those of your Access Anywhere Server. The minimum settings are:

{
    "endpoint":"https://example.com",
    "login":"admin@mcompany",
    "password":"*****"
    "upload_folder": "/Cloudian Files/jibe-reports",
    "flavor": "SQS",
    "sources": [
        {
            "name": "Cloudian files",
            "sqs_profile": "Cloudian"
        }
    ]
}

An SQS source provides the location and credentials for the queue.

The source must have a “name”. The name is used to look up the Cloudian provider.

"name": "Cloudian files"

Since the name is unique if you have multiple sources for a provider you can specify the provider by name explicitly:

"name": "queue3",
"provider": "Cloudian files"

A Cloudian source must have a flavor or “SQS”. If set at the root a flavor is inherited by all sources unless set explicitly.

"flavor": "SQS"

An endpoint must be provided for Cloudian SQS:

"sqs_endpoint_url": "http://s3-sqs.smestorage.com:18090"

The default queue name is 'jibe-source-activity'. It can be also be set within the source.

"queue_name": "my-jibe-queue"

Jibe will look up a queue by name. You can provide the full URL instead:

"queue_url": "https://sqs.us-east-1.amazonaws.com/410679986536/jibe-source-activity"

Credentials can be provided through the configuration file. These should be different to the credentials used for the Access Anywhere provider so that those events can be ignored.

"aws_access_key_id" : "AIDALDXPRHQHQE5VYHTNW",
"aws_secret_access_key" : "1z2X4GgRV+2vQz6chTqyhQjtYwbCYUm1shX6s5"
"aws_region" : "region1"

If these are not given the “default” profile is used. It can also be specified:

"aws_profile": "Cloudian"

The profile looks for a file called: /root/jibe/config/aws_credentials

[Cloudian]
aws_access_key_id = 00f635084aad231334067
aws_secret_access_key = +RrYCDsuqhXOAHkm9zF2bTVl2e2wyXmU5r8ypTW0/t

And one called /root/jibe/config/aws_config:

[profile Cloudian]
region = region1

The source can be configured to skip the processing of S3 events from actions generated by Access Anywhere. The principal can be identified from the log file. Upload a temporary file through Access Anywhere. In the info.log you will see a Sync event“

Sync: s3:ObjectCreated:Put->create '/Cloudian files/sqstest1/file.pdf' ('Cloudian SQS' event 14 seconds ago at 2022-02-21 18:16:40) file_id=152939201 {'principal_id': '021bafbcfb00b80c2014e8b004594af8'}

Use the principal in the source in the configuration file:

"skip_principal_id" : "021bafbcfb00b80c2014e8b004594af8"

For more information see Jibe Configuration and Jibe Logging.

Jibe includes a self-test which uses the Cloudian S3 API to create test events. The S3 endpoint must be provided:

"aws_endpoint_url": "http://s3.smestorage.com:18090"

To use different credentials use “sqs” for SQS access and “aws” for S3 testing. For example,

"sources": [
    {
        "name" : "Amazon S3 files",
        "sqs_profile": "default",
        "aws_profile": "testing",
        "sqs_endpoint_url": "http://s3-sqs.smestorage.com:18090",
        "aws_endpoint_url": "http://s3.smestorage.com:18090",
        "self_test_folder" : "sme-oakland/jibe_test",
        "self_test_enabled" : true
    }

For more information see Self Test.

This error can occur if an SQS operation is attempted against the S3 endpoint or vice versa.

An error occurred (MissingParameter) when calling the GetBucketNotificationConfiguration operation: A required parameter for the specified action is not supplied.