# Jibe Logging

Jibe uses Python's [logging module](https://docs.python.org/3/library/logging.html#module-logging) for application logging and notifications. 

A config file specifies what messages should be logged and where. The config file is a JSON [dictConfig](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig) object allowing logging to be highly customized. 

The logging config file can be changed at runtime. Jibe checks every minute for changes in configuration files.

## Jibe Application Logging

From within a Python application a [Logger Object](https://docs.python.org/3/library/logging.html#logger-objects) is a named object in which messages (log records) are sent for processing. Following Python convention, Jibe creates and uses logger objects that correspond to the project module concatenated with the names of Python files such as:

 * jibe.bucket
 * jibe.provider
 * jibe.reports
 * jibe.syncevent

 There are a couple of loggers that provide more granularity (you can see why later):

 * jibe.provider.sync - for provider-level sync warnings and errors
 * jibe.reports.ProviderReport - for report generation

The messages sent to the loggers also have a log level. Jibe uses the following Python log levels:
 
 * DEBUG: Low level system information for debugging purposes
 * INFO: General system information
 * WARNING: Information describing a minor problem that has occurred.
 * ERROR: Information describing a major problem that has occurred.
 * CRITICAL: Information describing a critical problem that has occurred.

## Log Config File Format

The log config file specifies how log messages are output. From what loggers are messages captured, what log levels, to what destination files and systems, and in what format.

The `dictConfig` format includes a number of JSON sections including:
 
 * loggers
 * handlers
 * formatters

## Loggers

A logger in the config file listens for all messages sent to that logger and below. For example, `jibe.provider`
will listen for messages sent to `jibe.provider` and `jibe.provider.sync`.

    "loggers": {
      "jibe.provider": {
        "level": "WARNING",
        "handlers": [ "alert_slack_handler"],
        "propagate": true
      }

Setting `level` to "WARNING" will listen for messages with that level and above including "ERROR" and "CRITICAL".

Handlers determine what happens next to the message, how they are output. A logger can define multiple named handlers. 

## Handlers

A handler describes a particular logging behavior, such as writing a message to the screen, to a file, or to a Slack channel.

Like loggers, handlers also have a log level. If the log level of a log record doesn’t meet or exceed the level of the
handler, the handler will ignore the message.

This is an example of a handler that writes to a file that rotates once a day.

    "handlers": {
        "info_file_handler": {
            "class": "logging.handlers.TimedRotatingFileHandler",
            "level": "INFO",
            "when": "midnight",
            "utc": false,
            "formatter": "detailed",
            "filename": "logs/info.log",
            "encoding": "utf8"
        }
    }

Handlers also specify a formatter that determines how the log message is formatted.

### Slack Handler

A Slack handler can be created. The handler definition will look like something like this:

    "slack_handler": {
        "class": "src.slack_log_handler.SlackLogHandler",
        "api_key": "xoxb-91872398123-098123987123-KHSQRKQHWEQWEEWQ",
        "level": "WARNING",
        "formatter": "slack",
        "channel":"#general",
        "fail_silent" : true
    }

The `api_key` is a Slack OAuth token. First, create a Slack application, called 'Jibe' for example,
at (https://api.slack.com/apps). Under the application 'Features' click 'OAuth & Permissions'. You'll
then see 'Bot User OAuth Access Token'. It needs an OAuth Scope of **chat:write** only.

#### api_key (required)
Generate a key at https://api.slack.com/

#### channel (required)
Set which channel you want to post to, e.g. "#general".

You need to give permission for the app to send messages to that channel.

    /invite @Jibe

#### username
The username that will post to Slack. Defaults to "Python logger".

#### icon_url
URL to an image to use as the icon for the logger user

#### icon_emoji
emoji to use as the icon. Overrides icon_url. If neither icon_url nor icon_emoji is set, ❗ will be used.

#### fail_silent
Defaults to False. If the API key is invalid or for some other reason the API call returns an error, this option will silently ignore the API error. If you enable this setting, make sure you have another log handler that will also handle the same log events, or they may be lost entirely.

## Formatters

Formatters are named string templates for outputting log records. These are examples included with Jibe but can
include other [LogRecord attributes](https://docs.python.org/3/library/logging.html#logrecord-attributes) too.

    "formatters": {
      "simpler": {
        "format": "%(levelname)s - %(message)s"
      },
      "simple": {
        "format": "%(asctime)s - %(levelname)s - %(message)s"
      },
      "detailed": {
        "format": "%(asctime)s - %(name)s - %(threadName)s - %(levelname)s - %(message)s"
      }
    }

The "detailed" format looks like this:

    2020-07-29 00:00:11,491 - jibe.core - DoTask-02-Sync_1 - INFO - syncOnProviderEvent: PutObject:not_found - my-big-bucket/2020-07-28-19-53-04.zip. 2 attempts left. 

The "simpler" format looks like this:

    INFO - syncOnProviderEvent: PutObject:not_found - my-big-bucket/2020-07-28-19-53-04.zip. 2 attempts left.

## Logger Examples

The `jibe.provider` logger provides important notifications when providers are being synchronized.
To ignore messages on app startup use "WARNING". To only see sync delay messages use `jibe.provider.sync`.

      "jibe.provider": {
        "level": "INFO",
        "handlers": [ "alert_slack_handler"],
        "propagate": true
      }

Example messages:

    jibe.provider.sync - MainThread - INFO - Provider ReSync Found (Started Completed '2020-07-01 18:22:39 UTC') - Sales GovCloud
    jibe.provider.sync - MainThread - INFO - Provider ReSync Found (Started Completed '2020-06-23 16:00:31 UTC') - Eng Cloud
    jibe.provider - DoTask-09 - INFO - Start Folder Monitor 'Shared Media' - [AWSLogs]
    jibe.provider.sync - DoTask-02 - WARNING - Providers Sync Delayed: ['Eng Cloud - 4 days, ReSync Running',
    jibe.provider.sync - DoTask-02 - WARNING - Providers Sync Delayed: ['Sales GovCloud - 4 hours, Real-Time Refresh On']
    jibe.provider.sync - DoTask-27 - WARNING - Providers No Longer Sync Delayed: ['Eng Cloud']
    jibe.provider.sync - DoTask-27 - WARNING - Providers Still Sync Delayed: ['Sales GovCloud - 1 days, Real-Time Refresh On']
    jibe.provider - DoTask-01 - WARNING - Bucket removed S3 Cloud/sc-bucketing (Folder Id: 323257338, Jibe Sync: False)
    jibe.provider - DoTask-35 - WARNING - Found New Provider - 'Shared Media'
    jibe.provider - DoTask-35 - WARNING - New bucket - Shared Media/sharedmedia123_logs
    jibe.provider - DoTask-35 - WARNING - New bucket - Shared Media/media_assets_123
    jibe.provider - DoTask-09 - WARNING - Start Folder Monitor 'Shared Media' - [AWSLogs]
    jibe.provider - DoTask-09 - WARNING - Stop Folder Monitor 'Eng Cloud' - [AWSLogs]