Send logs to the Cloud (Part 1.)

by Posted on
Photo by Tirza van Dijk on Unsplash

We have explored methods to aggregate logs into a central log server in prior posts, with minimal changes to the various log sources. (This can be quite important when dealing with a fleet of clients number in the thousands.)

In this missive, we will explore methods of sending those logs into AWS for further processing. It is assumed that an AWS account exists and that one has the necessary permissions to modify cloudwatch log groups, s3 buckets, and the IAM profiles that deal with those.

Our first step is to install the AWS cloudwatch agent on the log aggregator. Our log host is running the latest Ubuntu (20.04) and has all updates. (We will assume you have installed and configured the AWS CLI, if not, see our prior post.)

Step 1. Create an IAM role for log host with access to CloudWatch logs.

  1. Open the IAM console at https://console.aws.amazon.com/iam/ and Authenticate as necessary.
  2. In the navigation pane, choose Roles.
  3. Chose the role attached to your ec2 loghost instance.
  4. Choose Attach Policies, Create Policy.
  5. In the browser window/tab that opens; select the JSON tab
  6. Enter the following JSON policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
    ],
      "Resource": [
        "*"
    ]
  }
 ]
}
  1. Whence complete, select Review Policy and allow the Policy Validator to perform a JSON lint.
  2. Add a policy name and description; thence select Create Policy
  3. Return to the Add Permissions page for your role and attach the policy.

Step 2: Install and Configure the Logs agent on your loghost. (Ubuntu 20.04).

  1. Connect to your loghost instance
  2. Perform an “sudo apt-get update; apt-get upgrade -y”
    1. Download the agent and install as follows: (Take note of the architecture; AMD64 vs. ARM64)
      1. wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
      2. sudo dpkg -i -E ./amazon-cloudwatch-agent.deb
  3. Run the configuration wizard as follows:
    1. sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
      1. Select Linux
      2. Select EC2
      3. Select root
      4. We do not wish to run StatsD
      5. We do not wish to monitor CollectD
      6. We do not wish to monitor host stats.
      7. We do not have an existing config file.
      8. We do wish to monitor log files.
      9. Our path (for this test environment) is /var/log/remote
      10. We will accept the defaults for log group and log stream
      11. We will not store in SSM.
    2. The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.
  4. Start the cloudwatch agent via the command:
    1. /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

This will create a BASIC skeletal configuration file; if one wishes to monitor the auth.log entries collected from the remote hosts one may create a configuration file similar to:

{
        "agent": {
                "run_as_user": "root"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/remote/auth.json",
                                                "log_group_name": "remote",
                                                "log_stream_name": "test-vpc"
                                        }
                                ]
                        }
                }
        }
}

Note: the log group name and log stream name must conform to aws naming standards for cloudwatch.

In this case I have retained the default log_group_name of remote; and have set my log_stream_name to the name of the vpc my log aggregator resides in.

To aggregate more logs into the stream one may develop a hierarchy similar to :

{
        "agent": {
                "run_as_user": "root"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/remote/auth.json",
                                                "log_group_name": "test-vpc",
                                                "log_stream_name": "auth.json"
                                        },
                                        {
                                                "file_path": "/var/log/remote/syslog.json",
                                                "log_group_name": "test-vpc",
                                                "log_stream_name": "syslog.json"
                                        }
                                ]
                        }
                }
        }
}

The contents of /var/log/remote/auth.json will be captured in the log group named after the vpc that is the source of said logs, and the log stream is the actual aggregated logs for all instances in that vpc reporting.

After manually modifying the configuration file, one must reload the configuration via the command:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

One should also review the cloud watch agents log files located in /opt/aws/amazon-cloudwatch-agent/logs for proper operations.

As a test sshing from another test instance to my log-source, results in the following being recorded in the cloudwatch log group test-vpc, under the log stream auth.json, in properly formatted JSON ready for ingestion into any number of analytics systems.

Published by crawls