# Jibe and Cloudian

Jibe is able to receive events from Cloudian to identify S3 objects that have been
modified outside the Access Anywhere Server and sync them with Access Anywhere metadata.

For general information see [[/jibe]].

## Overview

The Jibe SQS Source connects to a Cloudian SQS queue to retrieve S3 notification
events created by Cloudian on change.

Sequence:

1. **S3 API** - Applications call Cloudian S3 APIs to create and delete objects.
1. **S3 Events** - Cloudian S3 buckets send S3 Event notifications to a Cloudian SQS queue.
1. **Message** - Jibe consumes messages from the queue, each of which contains one or more S3 Event notifications.
1. **Request Sync** - Jibe asks the Access Anywhere server to resync objects.
1. **Object Sync** - Access Anywhere verifies the object status with S3 and updates its metadata.


#### Jibe Sync with Cloudian
{{ /jibe:aws-sqs:aws-sqs-arch.png?500 |}}


## Getting Started

### Prerequisites 

To configure Jibe for Cloudian you will need:

 * Cloudian configured for SQS.
   * You will need an access key and secret, and endpoints for SQS and S3.
 * Access to the [AWS Command Line Interface](https://aws.amazon.com/cli/). You can use from a desktop. Upgrade to the latest version.

### Access Anywhere Setup

Jibe needs an Access Anywhere account with Administrator permission in order to synchronize objects.
(For SQS this does not need to be the Org Administrator.)

### Cloudian Setup

A bucket can send events to one queue. One queue may receive events from multiple buckets.
Jibe can consume events from multiple queues. Jibe polls queues for new messages
once every 20 seconds (by default).

You can create queues and bucket notifications through the aws cli.

#### Create Access Key and Secret

Create a new access key and secret that is used 

#### Create a profile

Create an AWS profile called "Cloudian" and define your access key and secret.

    aws --profile Cloudian configure

You will also need to add a region for the AWS CLI to work. You can use "region1".


#### Create a Queue

To create a queue with the name "jibe-source-activity" use the endpoint of your Cloudian SQS service:

    aws --profile Cloudian --endpoint-url "http://sqs.company.com:18090" sqs create-queue --queue-name jibe-source-activity

This returns the QueueUrl which will be used in other functions.

    {
        "QueueUrl": "http://sqs.region1.cloudian.com/021bafbcfb00b80c2014e8b004594af6/jibe-source-activity"
    }

Note: To find the Queue URLs of existing queues use the command

    aws --profile Cloudian --endpoint-url "http://sqs.company.com:18090" sqs list-queues

#### Delete a Queue

To remove a queue you'll use the Queue URL For example,

    aws --profile Cloudian --endpoint-url "http://sqs.company.com:18090" sqs delete_queue --queue_url "http://sqs.region1.cloudian.com/021bafbcfb00b80c2014e8b004594af6/jibe-source-activity"


#### Look up QueueArn

You'll need the QueueArn to create an S3 notification. 

    aws --profile Cloudian --endpoint-url "http://sqs.company.com:18090" sqs get-queue-attributes --queue-url "http://sqs.region1.cloudian.com/021bafbcfb00b80c2014e8b004594af6/jibe-source-activity" --attribute-names QueueArn

will return the QueueArn

    {
        "Attributes": {
            "QueueArn": "arn:aws:sqs:fmt:021bafbcfb00b80c2014e8b004594af6:jibe-source-activity"
        }
    }

#### Set Queue Configuration for Bucket

Use the QueueArn to create a `queue_config.json` file like this:

    {
        "QueueConfigurations": [
            {
                "Id": "jibeevents",
                "QueueArn": "arn:aws:sqs:fmt:021bafbcfb00b80c2014e8b004594af6:jibe-source-activity",
                "Events": [
                    "s3:ObjectCreated:*",
                    "s3:ObjectRemoved:*"
                ]
            }
        ]
    }


Then using the Cloudian S3 endpoint (different to the SQS endpoint) you can set a subscription for `mybucket`.

    aws --profile=Cloudian --endpoint="http://s3.company.com" s3api put-bucket-notification-configuration --notification-configuration file://queue_config.json --bucket mybucket

Check the current configuration for a bucket with:

    aws --profile=Cloudian --endpoint="http://s3.company.com" s3api get-bucket-notification-configuration --bucket mybucket

Continue for additional buckets that will be monitored. 

## Jibe Installation
See [[/jibe/docker|Installation]].

#### Configuration File 

Change the endpoint and credentials in `/root/config/jibe-config.json` to those of your
Access Anywhere Server. The minimum settings are:

    {
        "endpoint":"https://example.com",
        "login":"admin@mcompany",
        "password":"*****"
        "upload_folder": "/Cloudian Files/jibe-reports",
        "flavor": "SQS",
        "sources": [
            {
                "name": "Cloudian files",
                "sqs_profile": "Cloudian"
            }
        ]
    }

#### Source

An SQS source provides the location and credentials for the queue.

The source must have a "name". The name is used to look up the
Cloudian provider.

    "name": "Cloudian files"

Since the name is unique if you have multiple sources for a provider
you can specify the provider by name explicitly:

    "name": "queue3",
    "provider": "Cloudian files"

A Cloudian source must have a flavor or "SQS". If set at the root
a flavor is inherited by all sources unless set explicitly.

    "flavor": "SQS"

An endpoint must be provided for Cloudian SQS:

    "sqs_endpoint_url": "http://s3-sqs.smestorage.com:18090"

The default queue name is 'jibe-source-activity'. It can be also be set
within the source.

    "queue_name": "my-jibe-queue"

Jibe will look up a queue by name. You can provide the full URL instead:

    "queue_url": "https://sqs.us-east-1.amazonaws.com/410679986536/jibe-source-activity"

Credentials can be provided through the configuration file. These
should be different to the credentials used for the Access Anywhere 
provider so that those events can be ignored.

    "aws_access_key_id" : "AIDALDXPRHQHQE5VYHTNW",
    "aws_secret_access_key" : "1z2X4GgRV+2vQz6chTqyhQjtYwbCYUm1shX6s5"
    "aws_region" : "region1"

If these are not given the "default" profile is used. It can also be specified:

    "aws_profile": "Cloudian"

The profile looks for a file called: `/root/jibe/config/aws_credentials`

    [Cloudian]
    aws_access_key_id = 00f635084aad231334067
    aws_secret_access_key = +RrYCDsuqhXOAHkm9zF2bTVl2e2wyXmU5r8ypTW0/t

And one called `/root/jibe/config/aws_config`:

    [profile Cloudian]
    region = region1

#### Skip Principal

The source can be configured to skip the processing of S3 events from actions generated by Access Anywhere.
The principal can be identified from the log file. Upload a temporary file through Access
Anywhere. In the info.log you will see a Sync event"

    Sync: s3:ObjectCreated:Put->create '/Cloudian files/sqstest1/file.pdf' ('Cloudian SQS' event 14 seconds ago at 2022-02-21 18:16:40) file_id=152939201 {'principal_id': '021bafbcfb00b80c2014e8b004594af8'}

Use the principal in the source in the configuration file:

    "skip_principal_id" : "021bafbcfb00b80c2014e8b004594af8"

For more information see [[/jibe/configuration|Jibe Configuration]] and [[/jibe/logging|Jibe Logging]].

## Self Test

Jibe includes a self-test which uses the Cloudian S3 API to create
test events. The S3 endpoint must be provided:

    "aws_endpoint_url": "http://s3.smestorage.com:18090"

To use different credentials use "sqs" for SQS access and "aws" for
S3 testing. For example,

    "sources": [
        {
            "name" : "Amazon S3 files",
            "sqs_profile": "default",
            "aws_profile": "testing",
            "sqs_endpoint_url": "http://s3-sqs.smestorage.com:18090",
            "aws_endpoint_url": "http://s3.smestorage.com:18090",
            "self_test_folder" : "sme-oakland/jibe_test",
            "self_test_enabled" : true
        }

For more information see [[/jibe/self-test|Self Test]].

## Troubleshooting

### AWS CLI Errors

 This error can occur if an SQS operation is attempted against the S3 endpoint or vice versa.

    An error occurred (MissingParameter) when calling the GetBucketNotificationConfiguration operation: A required parameter for the specified action is not supplied.