Release eventgen 7.1.0. Merge to master branch. (#373)

* Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Fix dateutil package bug * cleanup memory (#326) * cleanup memory * addressing comment * Fixed standalone healthcheck (#328) * Add python2 libs required by perf test case (#331) * make splunk_eventgen as a module (#332) * Fix oom caused by ujson (#336) * Fix oom caused by ujson * Fix test env * add release automation script (#335) * add release automation script * Fix process leak and start/stop 500 issue (#344) * Fix process leak and start/stop 500 issue * Fix test case fail * Fix high system load issue (#345) * Adding an option (#340) * Adding an option * Adding ujson2 for python3 as well * minor improvements * Scp plugin (#343) * intermediate code for scp_plugin * code clean up * Added doc * revert windbag * Cap * added test * revert (#347) * changing name (#349) * Change (#350) * change token type (#351) * Add multithread support (#353) * Add multithread support * Fix typo * Revert coverage version to 4.5.4 (#355) * Fix CI failure cased by jinja new version and update the log config to reduce log in app (#358) * Add missing documentation for negative floats, as introduced in PR#127 (#360) * Bump nokogiri from 1.10.4 to 1.10.8 in /docs (#359) Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.4 to 1.10.8. - [Release notes](https://github.com/sparklemotion/nokogiri/releases) - [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md) - [Commits](sparklemotion/nokogiri@v1.10.4...v1.10.8) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Li Wu <[email protected]> * Fix jinja plugin bug and test addon sample (#361) * regex stanza bugfix (#365) * prelim changes * fix regex length matching * fix wildcard sample + csv cases * small csv fix, add test case * revert accidental change * Upgraded test instance * add httpevent collector * forgot to update stanza name * add escape for csv matching + test case * add sample, remove stanza * push not working Co-authored-by: tonyl <[email protected]> * Improvement/release script (#368) * add PR to master * update title in sent PR Co-authored-by: Li Wu <[email protected]> * Correct python2/3 issue (#363) * msg must be a bytes like object * msg must be a bytes like object Co-authored-by: Li Wu <[email protected]> * update eventgen version to 7.1.0 Co-authored-by: Li Wu <[email protected]> Co-authored-by: Tony Lee <[email protected]> Co-authored-by: Patrik Nordlén <[email protected]> Co-authored-by: Guodong Wang <[email protected]> Co-authored-by: David Wang <[email protected]> Co-authored-by: Guodong Wang <[email protected]> Co-authored-by: Erwin Vrolijk <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ryan Faircloth <[email protected]>
splunk · Apr 6, 2020 · 873c615 · 873c615
1 parent 50fbd9c
commit 873c615
Show file tree

Hide file tree

Showing 55 changed files with 758 additions and 223 deletions.
diff --git a/Makefile b/Makefile
@@ -39,10 +39,10 @@ test_helper:
 	docker cp . ${EVENTGEN_TEST_IMAGE}:$(shell pwd)
 
 	@echo 'Verifying contents of pip.conf'
-	docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "cd $(shell pwd); pip install dist/splunk_eventgen*.tar.gz"
+	docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "cd $(shell pwd); pip3 install dist/splunk_eventgen*.tar.gz"
 
 	@echo 'Installing test requirements'
-	docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "pip install --upgrade pip;pip install -r $(shell pwd)/requirements.txt"
+	docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "pip3 install --upgrade pip;pip3 install -r $(shell pwd)/requirements.txt;pip3 install git+https://github.com/esnme/ultrajson.git"
 
 	@echo 'Make simulated app dir and sample for modular input test'
 	docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "cd $(shell pwd); cd ../..; mkdir -p modinput_test_app/samples/"
@@ -51,12 +51,15 @@ test_helper:
 	@echo 'Installing docker-compose'
 	bash install_docker_compose.sh
 
+	@echo 'Build a docker image'
+	docker build -t provision_splunk:latest -f tests/large/provision/Dockerfile tests/large/provision
+
 	@echo 'Start container with splunk'
 	docker-compose -f tests/large/provision/docker-compose.yml up &
 
 	sleep 120
 	@echo 'Provision splunk container'
-	docker-compose -f tests/large/provision/docker-compose.yml exec -T splunk sh -c 'cd /opt/splunk;./provision.sh;/opt/splunk/bin/splunk enable listen 9997 -auth admin:changeme;/opt/splunk/bin/splunk add index test_0;/opt/splunk/bin/splunk add index test_1;/opt/splunk/bin/splunk restart'
+	docker exec --user splunk provision_splunk_1 sh -c 'cd /opt/splunk;./provision.sh;./add_httpevent_collector.sh;/opt/splunk/bin/splunk enable listen 9997 -auth admin:changeme;/opt/splunk/bin/splunk add index test_0;/opt/splunk/bin/splunk add index test_1;/opt/splunk/bin/splunk restart'
 
 run_tests:
 	@echo 'Running the super awesome tests'

diff --git a/dockerfiles/Dockerfile b/dockerfiles/Dockerfile
@@ -4,6 +4,8 @@ RUN apk --no-cache upgrade && \
 	apk add --no-cache --update \
 	python3 \
 	python3-dev \
+	python2-dev \
+	py2-pip \
 	gcc \
 	libc-dev \
 	libffi-dev \
@@ -26,7 +28,9 @@ RUN apk --no-cache upgrade && \
 	mkdir -p /root/.ssh && \
 	chmod 0700 /root/.ssh && \
 	passwd -u root && \
-	pip3 install git+git://github.com/esnme/ultrajson.git
+	# install dependencies of conduct2 used by perf
+	pip2 install filelock twisted requests queuelib ujson psutil crochet msgpack-python unidecode attrdict service_identity && \
+	pip2 install git+https://github.com/esnme/ultrajson.git
 
 COPY dockerfiles/sshd_config /etc/ssh/sshd_config
 COPY dockerfiles/entrypoint.sh /sbin/entrypoint.sh

diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
@@ -1,89 +1,58 @@
-6.5.0
-- Added metrics output mode
-- Fixed regex token replacement issue
-- Added test coverage information
-- Increased functional test coverage
-- Eventgen server complete revamp and standalone mode support
-- Added contributor license
-- Updated Dockerfile
-- Added documentation
-- Fixed bugs / stability / optimized speed
-
-6.4.0
-- Fixed exception log error
-- Fixed CircleCI status badage error
-- Fixed navigation error for app if installed with Splunk Stream
-- Fixed generatorWorkers not working error
-- Fixed interval error when end = 1
-- Fixed fileName in global stanza error
-- Added 3rd party libs in SA-Eventgen App
-- Added httpeventAllowFailureCount for httpevent
-- Added 3rd party libs in license credit
-- Disabled logging queue in multiprocess mode
-- Changed implementation of extendIndex for better performance
-
-6.3.6
-- Added functional tests for jinja template and modular input feature
-- Fixed default jinja template directory is not correctly resolved when sampleDir is set issue
-- Fixed verbose flag not working in splunk_eventgen command line issue
-- Fixed index, source, sourcetype are not correct when using splunkstream mode issue
-- Fixed ssh to container issue
-- Fixed perdayvolume without end setting error
-- Updated documentation for better reading and remove unrelated part
-
-6.3.5
-- Added extendIndexes feature to support a list of indexes
-- Fixed timer and token logic
-- Changed end=-1 to continuously iterate without stopping
-- Changed end=0 to not execute
-- Added a linter for code quality
-- Updated docs / docs format
-- Added a suite of functional tests
-
-6.3.4:
-- Documentation cleanup
-- Jinja template bugfix in app
-- Implementation of 'timeMultiple’ option
-- Templates for bugs/feature requests
-- Fixed Jinja test configuration stanzas
-- Default behavior for 'count' edge cases
-
-6.3.3:
-- Added performance metrics compared to Eventgen 5.x
-- New config option for generation-time metrics: outputCounter
-- Jinja template fixes
-- Timestamp parsing fix
-- Output queueing fix for outputMode splunkstream
-- Count rater fixes, now supports indefinite generation
-
-6.3.2:
-- Fixed verbosity bug
-- Added documentation
-
-6.3.1:
-- Fixed Eventgen Volume APIs
-- Improved Eventgen Server Logging
-- Corrected Eventgen Server and Controller conf syncing issue
-- Adding verbosity options (ERROR, INFO, DEBUG) to Eventgen modinput
-- Implemented future event generation support in replay mode
-- Fixed Jinja template's missing default values
-- Adjusted logging message levels for less verbosity
-- Fixed event count off by 1 issue
-- Fixed unnecessary empty data generators being created
-- Updated dependency list
-
-6.3.0:
-- Bug fixes for the customer issues
-- Documentation upgrade
-- Code refactoring for version unification
-- Logging improvements
-
-6.2.1:
-- Fixing SA-Eventgen Dashboard and log searching
-- Improving internal logging and fixing splunkd logging issue
-- Fixing timestamping in default generator
-- Fixing custom plugin integration
-- Fixing SA-Eventgen app settings
-- Supporting Eventgen 5 backward compatibility with additional features
-- Better modinput process management
-- Minor Bugfixes with various customer cases
+**7.1.0**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/7.1.0)
+
+**7.0.0**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/7.0.0)
+
+**6.5.2**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.5.2)
+
+
+**6.5.1**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.5.1)
+
+
+**6.5.0**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.5.0)
+
+**6.4.0**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.4.0)
+
+**6.3.6**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.3.6)
+
+**6.3.5**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.3.5)
+
+**6.3.4**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.3.4)
+
+**6.3.3**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.3.3)
+
+**6.3.2**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.3.2)
+
+**6.3.1**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.3.1)
+
+**6.3.0**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.3.0)
+
+**6.2.1**:
+
+- Check the release note and download the package/source from [Here](https://github.com/splunk/eventgen/releases/tag/6.2.1)
+
diff --git a/docs/CONFIGURE.md b/docs/CONFIGURE.md
@@ -323,8 +323,7 @@ Tokens in the default generator can override the sample to allow dynamic content
       and <end> is a number greater than 0 and greater than or equal to <start>. If rated,
       will be multiplied times hourOfDayRate and dayOfWeekRate.
     * For float[<start>:<end>], the token will be replaced with a random float between
-      start and end values where <start> is a number greater than 0
-      and <end> is a number greater than 0 and greater than or equal to <start>.
+      start and end values where <end> is a number greater than or equal to <start>.
       For floating point numbers, precision will be based off the precision specified
       in <start>. For example, if we specify 1.0, precision will be one digit, if we specify
       1.0000, precision will be four digits. If rated, will be multiplied times hourOfDayRate and dayOfWeekRate.

diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock
@@ -206,7 +206,7 @@ GEM
       jekyll-seo-tag (~> 2.1)
     minitest (5.12.2)
     multipart-post (2.1.1)
-    nokogiri (1.10.4)
+    nokogiri (1.10.8)
       mini_portile2 (~> 2.4.0)
     octokit (4.14.0)
       sawyer (~> 0.8.0, >= 0.5.3)

diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md
@@ -121,17 +121,38 @@ outputWorkers = <number of worker threads>
     * Generally if using TCP based outputs like splunkstream, more could be required
     * Defaults to 1
 
-outputMode = modinput | s2s | file | splunkstream | stdout | devnull | spool | httpevent | syslogout | tcpout | udpout | metric_httpevent
+outputMode = scsout | modinput | s2s | file | splunkstream | stdout | devnull | spool | httpevent | syslogout | tcpout | udpout | metric_httpevent
     * Specifies how to output log data. Modinput is default.
+    * If setting scsout, should set scsEndPoint and scsAccessToken. scsClientId, scsClientSecret, and scsRetryNum are optional.
     * If setting spool, should set spoolDir
     * If setting file, should set fileName
     * If setting splunkstream, should set splunkHost, splunkPort, splunkMethod,
       splunkUser and splunkPassword if not Splunk embedded
     * If setting s2s, should set splunkHost and splunkPort
-    * If setting syslogout, should set syslogDestinationHost and syslogDestinationPort
+    * If setting syslogout, should set syslogDestinationHost and syslogDestinationPort. A UDP port listening on Splunk needs to be configured. https://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkEnterprisehandlessyslogdata
     * If setting httpevent, should set httpeventServers
     * If setting metric_httpevent, should set httpeventServers and make sure your index is a splunk metric index
 
+scsEndPoint = <host>
+    * Should be a full url to the scs endpoint
+
+scsAccessToken = <token>
+    * Should be a scs access token. Do not include "Bearer". 
+
+scsClientId = <id>
+    * Optional
+    * SCS client id that is used to renew the access token if it expires during the data generation
+    * If not supplied, will not renew the access token and data transmission might fail
+
+scsClientSecret = <secret>
+    * Optional
+    * SCS client secret that is used to renew the access token if it expires during the data generation
+    * If not supplied, will not renew the access token and data transmission might fail
+
+scsRetryNum = <int>
+    * Optional and defaults to 0
+    * Retry a failing data transmission batch
+
 syslogDestinationHost = <host>
     * Defaults to 127.0.0.1
 
@@ -543,8 +564,7 @@ token.<n>.replacement = <string> | <strptime> | ["list","of","strptime"] | guid
       and <end> is a number greater than 0 and greater than or equal to <start>.
       If rated, will be multiplied times hourOfDayRate and dayOfWeekRate.
     * For float[<start>:<end>], the token will be replaced with a random float between
-      start and end values where <start> is a number greater than 0
-      and <end> is a number greater than 0 and greater than or equal to <start>.
+      start and end values where <end> is a number greater than or equal to <start>.
       For floating point numbers, precision will be based off the precision specified
       in <start>. For example, if we specify 1.0, precision will be one digit,
       if we specify 1.0000, precision will be four digits. If rated,

diff --git a/release_tool/README.md b/release_tool/README.md
@@ -0,0 +1,15 @@
+# Release tool
+
+Use script to bump the release verison and create the release PR to merge to develop branch.
+
+**Note: this script only works with python3.**
+
+- If you have generated your github access token, you can use the following command to bump versions and send PR automatically.
+    ```bash
+    python prepare_release_branch.py -v -n <release_version> -a <your_access_token>
+    ```
+
+- If the access token is not given, this script only is only used to bump the release version and push the commit to remote repo. You need to go to github web page to create your PR manually.
+    ```
+    python prepare_release_branch.py -v -n <release_version>
+    ```