Jibe Logging

Jibe uses Python's logging module for application logging and notifications.

A config file specifies what messages should be logged and where. The config file is a JSON dictConfig object allowing logging to be highly customized.

The logging config file can be changed at runtime. Jibe checks every minute for changes in configuration files.

From within a Python application a Logger Object is a named object in which messages (log records) are sent for processing. Following Python convention, Jibe creates and uses logger objects that correspond to the project module concatenated with the names of Python files such as:

• jibe.bucket
• jibe.provider
• jibe.reports
• jibe.syncevent

  There are a couple of loggers that provide more granularity (you can see why later):

• jibe.provider.sync - for provider-level sync warnings and errors
• jibe.reports.ProviderReport - for report generation

The messages sent to the loggers also have a log level. Jibe uses the following Python log levels:

* DEBUG: Low level system information for debugging purposes * INFO: General system information * WARNING: Information describing a minor problem that has occurred. * ERROR: Information describing a major problem that has occurred. * CRITICAL: Information describing a critical problem that has occurred.

The log config file specifies how log messages are output. From what loggers are messages captured, what log levels, to what destination files and systems, and in what format.

The dictConfig format includes a number of JSON sections including:

* loggers * handlers * formatters

A logger in the config file listens for all messages sent to that logger and below. For example, jibe.provider will listen for messages sent to jibe.provider and jibe.provider.sync.

"loggers": {
  "jibe.provider": {
    "level": "WARNING",
    "handlers": [ "alert_slack_handler"],
    "propagate": true
  }

Setting level to “WARNING” will listen for messages with that level and above including “ERROR” and “CRITICAL”.

Handlers determine what happens next to the message, how they are output. A logger can define multiple named handlers.

A handler describes a particular logging behavior, such as writing a message to the screen, to a file, or to a Slack channel.

Like loggers, handlers also have a log level. If the log level of a log record doesn’t meet or exceed the level of the handler, the handler will ignore the message.

This is an example of a handler that writes to a file that rotates once a day.

"handlers": {
    "info_file_handler": {
        "class": "logging.handlers.TimedRotatingFileHandler",
        "level": "INFO",
        "when": "midnight",
        "utc": false,
        "formatter": "detailed",
        "filename": "logs/info.log",
        "encoding": "utf8"
    }
}

Handlers also specify a formatter that determines how the log message is formatted.

A Slack handler can be created. The handler definition will look like something like this:

"slack_handler": {
    "class": "src.slack_log_handler.SlackLogHandler",
    "api_key": "xoxb-91872398123-098123987123-KHSQRKQHWEQWEEWQ",
    "level": "WARNING",
    "formatter": "slack",
    "channel":"#general",
    "fail_silent" : true
}

The api_key is a Slack OAuth token. First, create a Slack application, called 'Jibe' for example, at (https://api.slack.com/apps). Under the application 'Features' click 'OAuth & Permissions'. You'll then see 'Bot User OAuth Access Token'. It needs an OAuth Scope of chat:write only.

Generate a key at https://api.slack.com/

Set which channel you want to post to, e.g. “#general”.

You need to give permission for the app to send messages to that channel.

/invite @Jibe

The username that will post to Slack. Defaults to “Python logger”.

URL to an image to use as the icon for the logger user

emoji to use as the icon. Overrides iconurl. If neither iconurl nor icon_emoji is set, ❗ will be used.

Defaults to False. If the API key is invalid or for some other reason the API call returns an error, this option will silently ignore the API error. If you enable this setting, make sure you have another log handler that will also handle the same log events, or they may be lost entirely.

Formatters are named string templates for outputting log records. These are examples included with Jibe but can include other LogRecord attributes too.

"formatters": {
  "simpler": {
    "format": "%(levelname)s - %(message)s"
  },
  "simple": {
    "format": "%(asctime)s - %(levelname)s - %(message)s"
  },
  "detailed": {
    "format": "%(asctime)s - %(name)s - %(threadName)s - %(levelname)s - %(message)s"
  }
}

The “detailed” format looks like this:

2020-07-29 00:00:11,491 - jibe.core - DoTask-02-Sync_1 - INFO - syncOnProviderEvent: PutObject:not_found - my-big-bucket/2020-07-28-19-53-04.zip. 2 attempts left. 

The “simpler” format looks like this:

INFO - syncOnProviderEvent: PutObject:not_found - my-big-bucket/2020-07-28-19-53-04.zip. 2 attempts left.

The jibe.provider logger provides important notifications when providers are being synchronized. To ignore messages on app startup use “WARNING”. To only see sync delay messages use jibe.provider.sync.

  "jibe.provider": {
    "level": "INFO",
    "handlers": [ "alert_slack_handler"],
    "propagate": true
  }

Example messages:

jibe.provider.sync - MainThread - INFO - Provider ReSync Found (Started Completed '2020-07-01 18:22:39 UTC') - Sales GovCloud
jibe.provider.sync - MainThread - INFO - Provider ReSync Found (Started Completed '2020-06-23 16:00:31 UTC') - Eng Cloud
jibe.provider - DoTask-09 - INFO - Start Folder Monitor 'Shared Media' - [AWSLogs]
jibe.provider.sync - DoTask-02 - WARNING - Providers Sync Delayed: ['Eng Cloud - 4 days, ReSync Running',
jibe.provider.sync - DoTask-02 - WARNING - Providers Sync Delayed: ['Sales GovCloud - 4 hours, Real-Time Refresh On']
jibe.provider.sync - DoTask-27 - WARNING - Providers No Longer Sync Delayed: ['Eng Cloud']
jibe.provider.sync - DoTask-27 - WARNING - Providers Still Sync Delayed: ['Sales GovCloud - 1 days, Real-Time Refresh On']
jibe.provider - DoTask-01 - WARNING - Bucket removed S3 Cloud/sc-bucketing (Folder Id: 323257338, Jibe Sync: False)
jibe.provider - DoTask-35 - WARNING - Found New Provider - 'Shared Media'
jibe.provider - DoTask-35 - WARNING - New bucket - Shared Media/sharedmedia123_logs
jibe.provider - DoTask-35 - WARNING - New bucket - Shared Media/media_assets_123
jibe.provider - DoTask-09 - WARNING - Start Folder Monitor 'Shared Media' - [AWSLogs]
jibe.provider - DoTask-09 - WARNING - Stop Folder Monitor 'Eng Cloud' - [AWSLogs]