Jibe and AWS Lambda

For general information on Jibe see Jibe Documentation.

Jibe is able to run as a AWS Lambda function processing S3 event notifications. The steps are:

1. S3 API - External applications call Amazon S3 APIs to create and delete objects.
2. S3 Events - Amazon S3 buckets send S3 Event notifications to an Amazon SQS queue.
3. Message - Our AWS Lambda function receives and unpacks S3 notifications.
4. Request Sync - The function asks the Access Anywhere Server to refresh that object via th the REST API syncOnProviderEvent
5. Object Sync - The Access Anywhere Server verifies the object status on S3 and updates metadata.

S3 Events can only be sent to queues in the same region. For buckets in the other region send events through to an Amazon Simple Notification Service topic in that region which in turn forwards to the queue in the target region.

The AWS Lambda function, SQS queues, buckets and if needed, SNS topics can be configured through the AWS Management Console.

These are the steps to configure the Lambda function manually. They can also be useful in troubleshooting any issues with the automated setup.

Create a “Standard” SQS Queue in the region where your S3 buckets are located.

1. Navigate to the SQS Management Console.
2. Change regions to where (most of) your S3 buckets are located.
3. Create a Standard Queue with the Name jibe-source-activity.

Change the Queue Access Policy to allow the S3 service to send events:

{
    "Version": "2012-10-17",
    "Id": "example-ID",
    "Statement": [
        {
            "Sid": "example-statement-ID",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "SQS:SendMessage"
            ],
            "Resource": "arn:aws:sqs:us-east-1:410679986536:jibe-source-activity",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:s3:*:*:*"
                },
                "StringEquals": {
                    "aws:SourceAccount": "410679986536"
                }
            }
        }
    ]
}

Select a bucket to configure from the S3 Management Console.

1. Go to the Properties tab
2. Scroll to Event notifications
3. Select Add notification
4. Change Send to to SQS Queue
5. Select your queue from the drop-down

If you get this error you'll need to add permissions to your SQS Queue.

Unable to validate the following destination configurations. Permissions on the destination queue do not allow S3 to publish notifications from this bucket. (arn:aws:sqs:us-east-1:1234567890:bucket-activity)

You can't send S3 events to a queue in another region. Instead, create an SNS topic.

Select a bucket to configure from the S3 Management Console.

1. Go to the Properties tab
2. Scroll to Advanced settings and select Events
3. Select Add notification
4. Change Send to to SNS Topic
5. Select your topic from the drop-down

If you get this error you'll need to add permissions to your SNS Topic.

Unable to validate the following destination configurations. Permissions on the destination topic do not allow S3 to publish notifications from this bucket. (arn:aws:sns:us-east-1:1234567890:bucket-activity)

Create a config.json file with an Access Anywhere Server endpoint and credentials. For example,

{
    "apiendpoint" : "https://storagemadeeasy.com",
    "login" : "adminuser@example.com",
    "password" : "Passsword123"
}

Create a zip of the script and config file:

zip sme_jibe_sync.zip jibe_core.py config.json

Navigate to the AWS Lambda Management Console.

1. Choose Create function.
2. Choose Author from scratch.
3. Basic information: * Function name: smejibesync * Runtime: python3.8
4. Permissions:

   * Create a new role and attach these permission policies * AWSLambdaRole * AWSLambdaSQSQueueExecutionRole

5. Submit (which creates function)
6. Verify you are in the Designer view with the Lambda function selected.

   * Scroll down to Function code. * Change Handler to sme_jibe_sync.lambda_handler * Change Code entry type to Upload a .zip file. * Select Upload and locate smejibesync.zip. * Change timeout from 3 to 60 seconds * Set throttle to 10 * Select Save, you'll stay on the same page.

7. Select Layers, still within the Designer tab.
8. Select Create layer

   Add third-party libraries. These packages courtesy of to Keith's Layers (Klayers).

   arn:aws:lambda:us-east-1:770693421928:layer:Klayers-python38-requests:5

Create test using “Amazon S3 Put” template.

From the Lambda Console select the function and view the Configuration.

Add a trigger from the Queue:

1. Select Add trigger
2. Select SQS Queue
3. Choose an SQS queue to read messages from.
4. Enable trigger
5. Then Add

The SQS queue will now be visible as a trigger.