AWS Event Monitoring, “In the Beginning…”

I recently had dinner with an old friend and partner in chaos; we are security practitioners and face some of the same challenges. During our discussions, my associate took me to task for my security monitoring methods and suggested a different approach. (Of course, he has a better way. That is what friends are for, ONEUPMANSHIP!)

The story goes back to re:Invent ~2014, whence he and I attended a chalkboard session on AWS Cloudtrails, Cloudwatch, SNS, and SES.

To put some definition to all of this “Amazon Alphabet Soup,” let’s see what these things are:

From the Amazon AWS Website:

Cloudtrail: AWS CloudTrail monitors and records account activity across your AWS infrastructure, giving you control over storage, analysis, and remediation actions.

Cloudwatch: Amazon CloudWatch collects and visualizes real-time logs, metrics, and event data in automated dashboards to streamline your infrastructure and application maintenance.

Note: Cloudwatch has been split with a service called EventBridge.

EventBridge: Amazon EventBridge is a serverless event bus that lets you receive, filter, transform, route, and deliver events.

SNS: Amazon Simple Notification Service (SNS) sends notifications in two ways, A2A and A2P. A2A provides high-throughput, push-based, many-to-many messaging between distributed systems, microservices, and event-driven serverless applications.

SES: Amazon Simple Email Service (SES) lets you reach customers confidently without an on-premises Simple Mail Transfer Protocol (SMTP) system.

The concept of the chalkboard session was that Cloudtrails was configured to record ALL API-level events and was integrated with Cloudwatch/EventBridge to trigger notifications via SNS or SES when specific events occurred. (Consider this a roll your own SIEM.)

The main issue I and S.A.F.H. saw was that the messages/notifications were very generic, like placing a speedbump sign after the speedbump. We could, of course, do better. Thus starting a multi-year adventure in creating a framework. I’ll not bore the reader to tears with the ugly details, but the system did, on several occasions, detect and derail attacks on applications and, at one point, stop a data breach in progress. Despite the pithy code review comments, “Can I have some sauce with this spaghetti.”

The actual issue with a roll-your-own application is similar to casual sex, “One mistake and you wind up supporting it for life.” The alternative is one of the many commercial frameworks. While they are easy on the implementation and support side, they can be rigid and may not provide the intelligence desired while flooding the Security team with useless events.

While lamenting the above, my associate commented, “Have you seen Wazuh? It’s an offshoot of the Ossec project and integrates with several AWS Services.”

A quick review of the Wazuh website shows a lot of promise; it is open source, so if one needs to get under the hood, one can; it integrates with a long list of AWS services; there is a paid support option, so one is not supporting this alone; it is sufficiently mature that it is included on the AWS Marketplace, and finally there is a cloud service option, for those who do not want to manage hardware or virtual systems.

To quote from the AWS Marketplace:

Wazuh is a free, open source and enterprise-ready security monitoring solution for threat detection, integrity monitoring, incident response, and regulatory compliance.

The solution includes the Wazuh server, which is in charge of analyzing the data received from the agents, processing events through decoders and rules, and using threat intelligence to look for well-known IOCs (Indicators Of Compromise). A single Wazuh server can analyze data from hundreds or thousands of agents. Alerts generated by Wazuh are sent to Wazuh indexer, where they are indexed and stored. The unique integration between Wazuh and Wazuh dashboard provides a powerful user interface for data visualization and analysis. The server is also used to manage the agents, configuring and upgrading them remotely when necessary. Additionally, the server is capable of sending orders to the agents, for example, to trigger a response when a threat is detected.

Wazuh provides a security solution capable of monitoring your infrastructure, detecting threats, intrusion attempts, system anomalies, poorly configured applications, and unauthorized user actions. It also provides a framework for incident response and compliance, all in one platform.

As with a prior product investigation, I’ll install this on my space-heater server in my office. I’ll not bore the reader to tears with the installation of Ubuntu 22.10, AWS CLI v2, creation of access keys, etc., etc. Still, I will dive directly into installing the wazuh software and base configuration.

The Wazuh installation guide is quite in-depth for a distributed installation, but I will use the Quickstart option as we will install this on a single machine.

Checking the minimum hardware requirements, I suspect my space heater exceeds these by several light years. However, the base OS is Ubuntu 22.10, so we are good there, and the required browsers are not an issue for a modern client system. (Sorry, I.E., is NOT supported.)

Logging into my space heater as my primary user, I issue the following commands to download and run the Wazuh installation assistant:

curl -sO https://packages.wazuh.com/4.3/wazuh-install.sh && sudo bash ./wazuh-install.sh -a

Many messages will be displayed concerning the installation, but one must watch for the following summary line(s), precisely the Admin password.

INFO: --- Summary ---
INFO: You can access the web interface https://<wazuh-dashboard-ip>
    User: admin
    Password: <ADMIN_PASSWORD>
INFO: Installation finished.

Access the Wazuh dashboard by browsing HTTPS://<wazuh-server-ip> using the above credentials. You will note a browser warning as the certificate is self-signed.

An examination of the /var/ossec directory reveals several directories and sub-directories:

/var/ossec
├── active-response
├── agentless
├── api
├── backup
├── bin
├── etc
├── framework
├── integrations
├── lib
├── logs
├── queue
├── ruleset
├── stats
├── tmp
├── var
└── wodles

Of these, we will be dealing primarily with the, etc directory:

/var/ossec/etc
├── 20221115-1430-ossec.conf
├── 20221124-0700-ossec.conf
├── 20221124-0945-osssec.conf
├── 20221124-1030-osssec.conf
├── 20221125-0930-ossec.conf
├── client.keys
├── decoders
├── internal_options.conf
├── lists
├── local_internal_options.conf
├── localtime
├── ossec.conf
├── rootcheck
├── rules
├── shared
├── sslmanager.cert
└── sslmanager.key

The primary configuration file is ossec.conf, and you will note that several backups have been taken. As a matter of good systems administration, one should take a backup of critical files before modifying them.

The /var/ossec/etc/rules will hold our local customized alerting rules (we will cover that in another post).

As with all things, it is not good to perform actions in AWS as the root user or act as root on a UNIX box; similarly, one should create a Wazuh Internal user for normal operations.

As we have already logged in as the admin user, we will navigate the security panel

Within the Security panel, we will select Internal Users and create an internal user:

We will then provide the required username and password and assign a backend role of admin.

Thence Select Create at the bottom right of the screen.

Using an Incognito window, access the wazuh dashboard and log in with the newly created credentials.

In our next post, we will begin the process of adding the AWS Integration. Before that, a basic cloudtrail setup will be performed, do see C1 – BASIC CLOUDTRAIL SETUP. Whilst the post is possibly outdated as AWS changes faster than the weather in Texas, it will provide basic guidance.