official release of v3.3.0 (#1287)

* bumping version to 3.3.0 * Demisto playbook (#1239) * Supports dynamic parameters (#1244) * Add dynamic param support * I am caveman unga bunga smash * Oogey boogey beh * [apps][aliyun] Set EndTime in the request (#1247) * [apps][aliyun] Set EndTime in the request * github action sucks, changing comment to retrigger action Co-authored-by: Chunyong Lin <[email protected]> * updating metavar/description for cli flag (#1252) * support for packaging user specified conf directory (#1253) * adding packaging support for user specified config path * adding test for copying directory to alternate destination * pr feedback * adding fix for omitting coverage for forks (#1256) * another attempt at coveralls BS (#1257) * Update getting-started.rst (#1254) The current Getting Started instructions don't mention that you need to add the `set` command. As it stands, this is the error I received when setting up Streamalert: ``` (.env) jordan@mac:~/src/aws/streamalert/streamalert$ python manage.py output aws-sns usage: manage.py output [-h] {set,set-from-file,generate-skeleton,get,list} ... manage.py output: error: invalid choice: 'aws-sns' (choose from 'set', 'set-from-file', 'generate-skeleton', 'get', 'list') ``` Co-authored-by: Ryxias <[email protected]> Co-authored-by: ryandeivert <[email protected]> * Feature artifact extractor (#1250) * bumping version to 3.2.0 * migrating Athena function to use tf_lambda module (#1217) * rename of athena function * updating terraform generation code to use tf_lambda module * updating tf_athena module to remove lambda code * updates for packaging, rollback, and deploy * misc updates related to config path renaming, etc * removing no-longer-used method (athena is default) * addressing PR feedback * adding more granular time prefix to athena client * fixing duplicate resource issues (#1218) * fixing duplicate resource issues * fixing some other bugs in #1217 * fixing tf targets for athena deploy (#1220) * adding "--config-dir" flag to CLI to support specifying path for config files (#1224) * adding support for supplying path to config via CLI flag * misc touchups * updating publishers to accept configurable paths (#1223) * moving matchers outside of rules directory * updating rules for new matcher path * updating unit test for consistency * making publisher locations configurable * fixing typo * updating tf_lambda module to remove extra resources (#1225) * fixing rollback for all functions, removing 'all' flag for function deploys (#1222) * updating rollback functionality to include all funcs * updating tests to check for rollback of all funcs * updating docs * fixing tf cycle and index issue (#1226) * [core] Artifact Extractor lambda code * [core] load firehose client for artifact extractor * [core] Move FirehoseClient to shared folder * [test] Here we go pylint * [docs] Add high level Normalization doc * Ooops, leftover print * Address coment about docc * bumping version to 3.3.0 * Remove a FIXME comment * Add terraform resources * Fix some issues discovered during terraform build * [test] Add unit test cases and tune some code during testing * [cli] update artifact extractor module resource for lambda deploy * [doc] Update docstring * pylint * Address comments * Address more comments * [bugs] Fixed couple bugs before normalization code change * [core] Refactor normalization code, unit test cases and add new ones * [core] Re-implement normalization code \O/ * [docs] Update docs * [docs] More docs * Rework normalization logic to use key path from conf/schemas/*.json to find original key * [tests] update unit test cases * [rule][conf] Update conf right_to_left_character rule to use new normalization * [docs] Update docs and address comments * Fix a bug and update the unit test helper * Remove unnecessary comments * buggy, remove None values from normalization field * Add record id to artifacts and record * [tf] Upgrade terraform aws provider to 2.48.0 * Add condition to normalizer * [docs] Update docs * Address comment * Add three custom metrics * [cli] fix undeclared module issue related to artifact_extractor * [doc] Update artifact extractor deploy instruction Co-authored-by: Ryan Deivert <[email protected]> Co-authored-by: Chunyong Lin <[email protected]> * [config] Add Okta log schema (#1263) * [config] Add Okta log schema * Add test record * Fix tests * Fix tests Co-authored-by: Matt Muller <[email protected]> * Add additional G-Suite Admin Audit types. (#1260) Co-authored-by: darkjokelady <[email protected]> * Update getting-started.rst (#1255) * Update getting-started.rst Fix path to `cloudtrail_root_account_usage.py` rule being modified in the Getting Started documentation. * test ci change in fork * second update for ci tests in forks Co-authored-by: Ryxias <[email protected]> Co-authored-by: ryandeivert <[email protected]> Co-authored-by: ryandeivert <[email protected]> Co-authored-by: darkjokelady <[email protected]> * [core] fix bug when normalization config empty (#1262) * [core] fix bug when normalization config empty * [test] Update unit test case * [docs] Update how to search artifacts table Co-authored-by: Chunyong Lin <[email protected]> * CLI support for extra user supplied terraform files (#1267) * adding cli arg to supply additional terraform config files * removing old tf cleanup code since temp path will be used * cliconfig support for temp tf directory * updates to tf_runner and run_command for temp tf path * removing tf clean command since runs are now idempotent * packaging change for tf temp path * logic for copying files to tf temp path * removing init backend option * cleanup * fix unit tests * config support for extra tf files * doc update for `terraform_files` setting * unit test for cliconfig terraform files * fix for init backend outside of generate logic * update to support supplying static dir for builds * fixing issue with streamalert.zip not existing at build times (#1269) * Move artifact extractor logic to classifier (#1268) * [core] Move artifact extractor logic to classifier * [core] Add send_to_artifacts flag to normalizer * [cli] Remove leftover variables, permissions * [core] Fix bugs, update custom metrics for artifacts * [tests] Update test cases * [docs] Update docs * [cli] Update artifact_extract.tf.json path after PR #1267 merged Co-authored-by: Chunyong Lin <[email protected]> * rebuilding pkg on every tf run (#1270) * ensuring prefix is a lowercase string (#1272) * updating dependencies (#1277) * updating deps * updating precompiled deps * misc cleanup * [core][apps] Increase aliyun timeout (#1274) Co-authored-by: Chunyong Lin <[email protected]> * proper cloudwatch events permissions for cross account access (#1276) * updating cloudwatch events module to support advanced event brige rule * adding proper support for cloudwatch event permission for cross account cwe * terraform gen code for new cross account cwe perms * doc updates for x-acct cwe perms * fix readme * reverting usage of cloudformation stack * allowing optional scopes * proper provider support for different regions * fixing pylint * adding role arn to target * installing venv in vagrant (#1278) * fixing copying of zips, since lambda layers are zips (#1279) * cloudtrail module config tweak (#1280) * updating cloudtrail module config slightly * updating unit tests and docs for cloudtrail module change * fixing default for enable_events * update to docs * raising exceptions when error occurs while downloading from s3 (#1281) * raising exceptions with s3 download errors * fixing unit test * addressing issue with 0 byte files in s3 (#1284) * adding support for other accounts to publish to sns topic (#1283) * fixing a bug I think but who really knows (#1285) * adding fix for #1282 (#1286) Co-authored-by: Ryxias <[email protected]> Co-authored-by: darkjokelady <[email protected]> Co-authored-by: Chunyong Lin <[email protected]> Co-authored-by: Jordan Wright <[email protected]> Co-authored-by: themullinator <[email protected]> Co-authored-by: Matt Muller <[email protected]> Co-authored-by: Gavin <[email protected]>
airbnb · Aug 5, 2020 · bfde778 · bfde778
1 parent a0a284c
commit bfde778
Show file tree

Hide file tree

Showing 111 changed files with 3,868 additions and 625 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -41,7 +41,7 @@ jobs:
       - name: ${{ matrix.task.name }}
         run: ${{ matrix.task.command }}
       - name: Submit Coverage
-        run: coveralls
+        run: ([ -z "$COVERALLS_REPO_TOKEN" ] && echo "coveralls is skipped in forked repo tests" || coveralls)
         if: matrix.task.name == 'Test'
         env:
           COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
diff --git a/conf/clusters/prod.json b/conf/clusters/prod.json
@@ -63,6 +63,9 @@
       "prefix_cluster_gsuite_admin_sm-app-name_app": [
         "gsuite"
       ],
+      "prefix_cluster_okta_logevents-app-name_app": [
+        "okta"
+      ],
       "prefix_cluster_onelogin-events-app-name_app": [
         "onelogin"
       ],

diff --git a/conf/global.json b/conf/global.json
@@ -5,6 +5,7 @@
     "region": "us-east-1"
   },
   "general": {
+    "terraform_files": [],
     "matcher_locations": [
       "matchers"
     ],
@@ -27,6 +28,11 @@
       "read_capacity": 5,
       "write_capacity": 5
     },
+    "artifact_extractor": {
+      "enabled": false,
+      "firehose_buffer_size": 128,
+      "firehose_buffer_interval": 900
+    },
     "firehose": {
       "use_prefix": true,
       "buffer_interval": 900,

diff --git a/conf/schemas/carbonblack.json b/conf/schemas/carbonblack.json
@@ -1043,7 +1043,31 @@
         "uid",
         "username",
         "sha256"
-      ]
+      ],
+      "normalization": {
+        "command": [
+          {
+            "path": ["command_line"],
+            "function": "Command line"
+          }
+        ],
+        "path": [
+          {
+            "path": ["path"],
+            "function": "Process path"
+          },
+          {
+            "path": ["parent_path"],
+            "function": "Process parent path",
+            "send_to_artifacts": false
+          },
+          {
+            "path": ["process_path"],
+            "function": "Process parent path",
+            "send_to_artifacts": false
+          }
+        ]
+      }
     }
   },
   "carbonblack:ingress.event.regmod": {
@@ -1445,4 +1469,4 @@
       }
     }
   }
-}
+}
diff --git a/conf/schemas/cloudwatch.json b/conf/schemas/cloudwatch.json
@@ -88,7 +88,55 @@
       "time": "string",
       "version": "string"
     },
-    "parser": "json"
+    "parser": "json",
+    "configuration": {
+      "normalization": {
+        "event_name": ["detail", "eventName"],
+        "account": [
+          {
+            "path": [
+              "account"
+            ],
+            "function": "Destination account ID"
+          },
+          {
+            "path": [
+              "detail",
+              "userIdentity",
+              "principalId"
+            ],
+            "function": "Source account ID"
+          }
+        ],
+        "ip_address": [
+          {
+            "path": [
+              "detail",
+              "sourceIPAddress"
+            ],
+            "function": "Source IP addresses"
+          }
+        ],
+        "user_agent": [
+          "detail",
+          "userAgent"
+        ],
+        "user_identity": [
+          {
+            "path": ["detail", "userIdentity", "type"],
+            "function": "User identity type"
+          },
+          {
+            "path": ["detail", "userIdentity", "arn"],
+            "function": "User identity arn"
+          },
+          {
+            "path": ["detail", "userIdentity", "userName"],
+            "function": "User identity username"
+          }
+        ]
+      }
+    }
   },
   "cloudwatch:flow_logs": {
     "schema": {
@@ -144,4 +192,4 @@
       "quotechar": "'"
     }
   }
-}
+}
diff --git a/conf/schemas/okta.json b/conf/schemas/okta.json
@@ -0,0 +1,35 @@
+{
+  "okta:logevents": {
+    "schema": {
+      "uuid": "string",
+      "published": "string",
+      "eventType": "string",
+      "version": "string",
+      "severity": "string",
+      "legacyEventType": "string",
+      "displayMessage": "string",
+      "actor": {},
+      "client": {},
+      "outcome": {},
+      "target": [],
+      "transaction": {},
+      "debugContext": {},
+      "authenticationContext": {},
+      "securityContext": {},
+      "request": {}
+    },
+    "parser": "json",
+    "configuration": {
+      "optional_top_level_keys": [
+        "legacyEventType",
+        "displayMessage",
+        "client",
+        "outcome",
+        "transaction",
+        "debugContext",
+        "authenticationContext",
+        "request"
+      ]
+    }
+  }
+}
diff --git a/conf/schemas/osquery.json b/conf/schemas/osquery.json
@@ -48,7 +48,21 @@
         "log_type",
         "logNumericsAsNumbers",
         "numerics"
-      ]
+      ],
+      "normalization": {
+        "command": [
+          {
+            "path": ["columns", "command"],
+            "function": "Command line from shell history"
+          }
+        ],
+        "file_path": [
+          {
+            "path": ["columns", "history_file"],
+            "function": "Shell history file path"
+          }
+        ]
+      }
     }
   },
   "osquery:snapshot": {

diff --git a/constraints.txt b/constraints.txt
@@ -0,0 +1,2 @@
+# botocore requires a version of docutils < 0.16, but sphinx-rtd-theme's requirement of >=0.12 breaks this
+docutils<0.16
diff --git a/docs/images/artifacts.png b/docs/images/artifacts.png
diff --git a/docs/images/cloudwatch_events.png b/docs/images/cloudwatch_events.png
diff --git a/docs/images/join_search.png b/docs/images/join_search.png
diff --git a/docs/images/normalization-arch.png b/docs/images/normalization-arch.png
diff --git a/docs/source/config-clusters.rst b/docs/source/config-clusters.rst
@@ -210,7 +210,9 @@ Example: CloudTrail via S3 Events
     },
     "modules": {
       "cloudtrail": {
-        "enable_s3_events": true
+        "s3_settings": {
+          "enable_events": true
+        }
       }
     }
   }
@@ -242,8 +244,10 @@ Example: CloudTrail via CloudWatch Logs
     },
     "modules": {
       "cloudtrail": {
-        "send_to_cloudwatch": true,
-        "enable_s3_events": false,
+        "s3_settings": {
+          "enable_events": true
+        },
+        "send_to_cloudwatch": true
       },
       "kinesis": {
         "streams": {
@@ -269,18 +273,30 @@ Options
 ==============================  ===================================================  ===============
 **Key**                         **Default**                                          **Description**
 ------------------------------  ---------------------------------------------------  ---------------
-``s3_cross_account_ids``        ``[]``                                               Grant write access to the CloudTrail S3 bucket for these account IDs. The primary, aka deployment account ID, will be added to this list.
+``enabled``                     ``true``                                             Toggle the ``cloudtrail`` module
 ``enable_logging``              ``true``                                             Toggle to ``false`` to pause logging to the CloudTrail
 ``exclude_home_region_events``  ``false``                                            Ignore events from the StreamAlert deployment region. This only has an effect if ``send_to_cloudwatch`` is set to ``true``
 ``is_global_trail``             ``true``                                             If ``true``, the CloudTrail is applied to all regions
 ``send_to_cloudwatch``          ``false``                                            Enable CloudTrail delivery to CloudWatch Logs. Logs sent to CloudWatch Logs are forwarded to this cluster's Kinesis stream for processing. If this is enabled, the ``enable_s3_events`` option should be disabled to avoid duplicative processing.
-``cloudwatch_destination_arn``  (Computed from CloudWatch Logs Destination module)   CloudWatch Destination ARN used for forwarding data to this cluster's Kinesis stream. This has a default value but can be overriden here with a different CloudWatch Logs Destination ARN
+``cloudwatch_destination_arn``  (Computed from CloudWatch Logs Destination module)   CloudWatch Destination ARN used for forwarding data to this cluster's Kinesis stream. This has a default value but can be overridden here with a different CloudWatch Logs Destination ARN
 ``send_to_sns``                 ``false``                                            Create an SNS topic to which notifications should be sent when CloudTrail puts a new object in the S3 bucket. The topic name will be the same as the S3 bucket name
-``enable_s3_events``            ``false``                                            Enable S3 events for the logs sent to the S3 bucket. These will invoke this cluster's classifier for every new object in the CloudTrail S3 bucket
-``s3_bucket_name``              ``prefix-cluster-streamalert-cloudtrail``            Name of the S3 bucket to be used for the CloudTrail logs. This can be overriden, but defaults to ``prefix-cluster-streamalert-cloudtrail``
-``s3_event_selector_type``      ``""``                                               An S3 event selector to enable object level logging for the account's S3 buckets. Choices are: "ReadOnly", "WriteOnly", "All", or "", where "" disables object level logging for S3
+``allow_cross_account_sns``     ``false``                                            Allow account IDs specified in the ``cross_account_ids`` array within the ``s3_settings`` (see below) to also send SNS notifications to the created SNS Topic
+``s3_settings``                 ``None``                                             Configuration options for CloudTrail related to S3. See the `S3 Options`_ section below for details.
 ==============================  ===================================================  ===============
 
+S3 Options
+----------
+The ``cloudtrail`` module has a subsection of ``s3_settings``, which contains options related to S3.
+
+========================  ===================================================  ===============
+**Key**                   **Default**                                          **Description**
+------------------------  ---------------------------------------------------  ---------------
+``cross_account_ids``     ``[]``                                               Grant write access to the CloudTrail S3 bucket for these account IDs. The primary, aka deployment account ID, will be added to this list.
+``enable_events``         ``false``                                            Enable S3 events for the logs sent to the S3 bucket. These will invoke this cluster's classifier for every new object in the CloudTrail S3 bucket
+``ignore_digest``         ``true``                                             If ``enable_events`` is enabled, setting ``ignore_digest`` to ``false`` will also process S3 files that are created within the ``AWSLogs/<account-id>/CloudTrail-Digest``. Defaults to ``true``.
+``bucket_name``           ``prefix-cluster-streamalert-cloudtrail``            Name of the S3 bucket to be used for the CloudTrail logs. This can be overridden, but defaults to ``prefix-cluster-streamalert-cloudtrail``
+``event_selector_type``   ``""``                                               An S3 event selector to enable object level logging for the account's S3 buckets. Choices are: "ReadOnly", "WriteOnly", "All", or "", where "" disables object level logging for S3
+========================  ===================================================  ===============
 
 .. _cloudwatch_events:
 
@@ -325,6 +341,18 @@ Example
             "EC2 Instance Terminate Successful",
             "EC2 Instance Terminate Unsuccessful"
           ]
+        },
+        "cross_account": {
+          "accounts": {
+            "123456789012": [
+              "us-east-1"
+            ]
+          },
+          "organizations": {
+            "o-aabbccddee": [
+              "us-east-1"
+            ]
+          }
         }
       },
       "kinesis": {
@@ -341,7 +369,7 @@ Example
   }
 
 This creates a CloudWatch Events Rule that will publish all events that match the provided
-``event_pattern`` to the Kinesis stream for this cluster. Note in the example above that a custom
+``event_pattern`` to the Kinesis Stream for this cluster. Note in the example above that a custom
 ``event_pattern`` is supplied, but may be omitted entirely. To override the default ``event_patten``
 (shown below), a value of ``None`` or ``{}`` may also be supplied to capture all events,
 regardless of which account the logs came from. In this case, rules should be written against
@@ -353,9 +381,21 @@ Options
 =====================  ===================================  ===============
 **Key**                **Default**                          **Description**
 ---------------------  -----------------------------------  ---------------
-``event_pattern``      ``{"account": ["<accound_id>"]}``    The `CloudWatch Events pattern <http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html>`_ to control what is sent to Kinesis
+``event_pattern``      ``{"account": ["<account-id>"]}``    The `CloudWatch Events pattern <http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html>`_ to control what is sent to Kinesis
+``cross_account``      ``None``                             Configuration options to enable cross account access for specific AWS Accounts and Organizations. See the `Cross Account Options`_ section below for details.
 =====================  ===================================  ===============
 
+Cross Account Options
+---------------------
+The ``cross_account`` section of the ``cloudwatch_events`` module has two subsections, outlined here. Usage of these is also shown in the example above.
+
+=====================  ===========  ===============
+**Key**                **Default**  **Description**
+---------------------  -----------  ---------------
+``accounts``           ``None``     A mapping of *account IDs* and regions for which cross account access should be enabled. Example: ``{"123456789012": ["us-east-1"], "234567890123": ["us-west-2"]}``
+``organizations``      ``None``     A mapping of *organization IDs* and regions for which cross account access should be enabled. Example: ``{"o-aabbccddee": ["us-west-2"]}``
+=====================  ===========  ===============
+
 
 .. _cloudwatch_logs:
 
@@ -432,8 +472,8 @@ Options
 =====================  ===========  ===============
 **Key**                **Default**  **Description**
 ---------------------  -----------  ---------------
+``enabled``            ``true``     Toggle the ``cloudwatch_logs_destination`` module
 ``cross_account_ids``  ``[]``       Authorize StreamAlert to gather logs from these accounts
-``enabled``            ``true``     Toggle the CloudWatch Logs module
 ``excluded_regions``   ``[]``       Do not create CloudWatch Log destinations in these regions
 =====================  ===========  ===============
 
@@ -497,7 +537,7 @@ Options
 ==========================  ===========  ===============
 **Key**                     **Default**  **Description**
 --------------------------  -----------  ---------------
-``enabled``                 ``false``    Toggle the CloudWatch Monitoring module
+``enabled``                 ``false``    Toggle the ``cloudwatch_monitoring`` module
 ``kinesis_alarms_enabled``  ``true``     Toggle the Kinesis-specific metric alarms
 ``lambda_alarms_enabled``   ``true``     Toggle the Lambda-specific metric alarms
 ``settings``                ``{}``       Alarm-specific settings (see below)
@@ -748,7 +788,7 @@ Options
 =====================  =============================================================================================================================================  ===============
 **Key**                **Default**                                                                                                                                    **Description**
 ---------------------  ---------------------------------------------------------------------------------------------------------------------------------------------  ---------------
-``enabled``            ---                                                                                                                                            Toggle flow log creation
+``enabled``            ``true``                                                                                                                                       Toggle the ``flow_logs`` module
 ``flow_log_filter``    ``[version, account, eni, source, destination, srcport, destport, protocol, packets, bytes, windowstart, windowend, action, flowlogstatus]``   Toggle flow log creation
 ``log_retention``      ``7``                                                                                                                                          Day for which logs should be retained in the log group
 ``enis``               ``[]``                                                                                                                                         Add flow logs for these ENIs

diff --git a/docs/source/config-global.rst b/docs/source/config-global.rst
@@ -61,6 +61,9 @@ Configuration
 
   {
     "general": {
+      "terraform_files": [
+        "/absolute/path/to/extra/terraform/file.tf"
+      ],
       "matcher_locations": [
         "matchers"
       ],
@@ -90,6 +93,7 @@ Options
 ``scheduled_query_locations``  Yes            ``["scheduled_queries"]``  List of local paths where ``scheduled_queries`` are defined
 ``publisher_locations``        Yes            ``["publishers"]``         List of local paths where ``publishers`` are defined
 ``third_party_libraries``      No             ``["pathlib2==2.3.5"]``    List of third party dependencies that should be installed via ``pip`` at deployment time. These are libraries needed in rules, custom code, etc that are defined in one of the above settings.
+``terraform_files``            No             ``[]``                     List of local paths to Terraform files that should be included as part of this StreamAlert deployment
 =============================  =============  =========================  ===============
 
 

diff --git a/docs/source/getting-started.rst b/docs/source/getting-started.rst
@@ -208,7 +208,7 @@ Open ``conf/clusters/prod.json`` and change the ``data_sources`` section to look
 
 .. code-block:: bash
 
-  python manage.py output aws-sns
+  python manage.py output set aws-sns
 
   Please supply a short and unique descriptor for this SNS topic: test-email
 
@@ -217,7 +217,7 @@ Open ``conf/clusters/prod.json`` and change the ``data_sources`` section to look
 If you look at ``conf/outputs.json``, you'll notice that the SNS topic was automatically added.
 
 7. Configure a rule to send to the alerts topic.
-We will use ``rules/community/cloudtrail/cloudtrail_root_account_usage.py`` as an example, which
+We will use ``rules/community/cloudwatch_events/cloudtrail_root_account_usage.py`` as an example, which
 alerts on any usage of the root AWS account. Change the rule decorator to:
 
 .. code-block:: python

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -82,6 +82,7 @@ Table of Contents
    rule-promotion
    historical-search
    scheduled-queries
+   normalization
    conf-schemas-examples
    troubleshooting
    faq