- module: report all not used property keys
- kafka: support to specify consumerGroupId, for more flexible consumer structure setup
- search: update es to 6.5.4
- http: fallback to TLSv1.2 for HTTPS, TLSv1.3 causes various of issues (will check whether openjdk 11.0.2 or 12 improves)
- httpclient: improved retry handling, mark current connection to be ineligible if 503
- httpclient: change HTTPResponse status from enum to int, as external server may uses arbitrary or extended values
- search: update es to 6.5.3
- module: report error if one property is not used, and encourage static config style (avoid conditional property key reading)
- httpclient: make HTTPRequest.uri modifiable, allow ws client interceptor change uri dynamically
- httpclient: close response if got 503 in retry
- app: support to use APP_NAME or -Dcore.appName to override appName, to support deploy same image/app with different name (multi tenancy)
- elasticsearch: added ElasticSearchMigration to support migration
- kafka: include kafka lib into framework, no need to add runtimeOnly in application (for simplifying, and our app is mostly service on cloud)
- redis: supports to connect to multiple redis dbs from one app
- executor: make task rejection as warning during shutdown
- session: support to use Header to pass sessionId, for mobile-api
- search: update es to 6.5.1
- kafka: update kafka to 2.1.0
- redis: multGet values return in same order of keys
- kafka: set consumer client.id for monitoring
- api: change typescript api nullable definition to "type | null", to make it stricter
- httpclient: update okHTTP to 3.12.0
- httpclient: use okHTTP as implementation to support http/2.0 added connectTimeout, as for internal api or external page fetching, it requires different settings
- executor: shutdown executor in 2 steps, as there may multiple executors support submit task with delay, to support application retry task, to avoid sleep during task execution
- http: shutdown http server at last, to accept incoming request during shutdown enabled HTTP2 support disabled "always write keep alive response header", since it's default for HTTP/1.1 HTTP/1.0 client is less popular, for apache benchmark tool (ab), use alternative one (e.g. h2load from nghttp2, which will be installed with curl+http2 support)
- db: close connection if query timed out, refer to core.framework.impl.db.Connections for reason
- mongo: replaced fongo with mongo-java-server, update driver to 3.9.0
- kafka: update to 2.0.1
- search: update es to 6.5.0
- log-processor: changed timefield from date to @timestamp, as default value for timelion or visualization builder
- log: collect cpu usage stat, in container env, system load != cpu container/java process usage
- http: changed Response.empty() return 200 status code due to jdk httpclient bug, it hangs if server return 204 no content, without content-length, see https://bugs.openjdk.java.net/browse/JDK-8211437
- http: disable http2 for both server and client, as jdk http client may cause busy loop issue if transaction is not proper closed
- util: removed InputStreams, with JDK 11, inputStream added readAllBytes() method to read all bytes
- httpclient: support gzip content-type
- log: update slfj4 to 1.8, which switched to service provider model to bind logger factory and added java module support
- httpClient: fix keep alive timeout setting
- httpClient: tweak request/response to make it more low level style,
- api: refined query param bean behavior, empty value will be treated as null, refer to core.framework.impl.web.bean.QueryParamMapper
- management: added /_sys/vm, /_sys/thread, /_sys/heap to provide more detailed diagnostic info (similar result of jcmd)
- httpClient: fix bug on shouldRetry
- jdk: update to java 11
- log: internal tweaking, make logParam be aware of max param length, consume less memory with large trace log
- db: removed test initDB().runScript(), since in real life we always prefer to use repository(), as refactor/entity typing/validation reason, script can still be achieved by calling database directly
- util: !!! renamed Strings.isEmpty to Strings.isBlank to match jdk 11
- validation: !!! renamed @NotEmpty to @NotBlank to match jdk 11 naming convention
- httpClient: replaced apache httpClient with JDK builtin impl
- db: log params for batch operations, disable ssl for mysql
- log: adjust logger name abbr, only leave last 2 tokens for long package
- log: reduce trace log, only log thread/date once at beginning, replace timestamp with duration in nanos
- log-processor: removed id in action/trace, redundant with "_id"
6.8.4 (9/18/2018 - 9/20/2018) !!! action log format changed, please use latest log-processor, docker: neowu/log-processor:6.8.4 !!!
- http: not setting Date response header as it's not necessary
- util: delete some methods of Files, not too useful in cloud env
- api: for api ts definition change LocalDateTime/LocalDate to string type
- log: renamed old refId to correlationId, and now refId is for directly reference action, bulk message handler now logs all clients/correlationIds/refIds
- http: removed request.pathParam/request.queryParam, for type safety, we use api interface/impl, so only keep low level interface to keep flexibility
- http: path param now only supports string/integer/long/enum
- kafka: bulk message handler logs all message in trace for troubleshooting
- api: webservice client generates more efficient code for dynamic path pattern
- http: change framework bad request error code to "INVALID_HTTP_REQUEST"
- db: changed db().defaultIsolationLevel() to db().isolationLevel(), and if specified, isolation level will be set on connection creation, rather than before every transaction started (to avoid unnecessary db operation)
- api: webservice client logs the calling method, tweak http client logging
- http: make ip access check as built in logic, to deny before routing check (which may return not found or method not allowed)
- api: added api().bean(beanClass) for raw request/response/ws bean
- util: removed Exceptions.error(), use new Error(Strings.format()) instead, which is more straightforward and code analyzer friendly (easier to check whether hide root cause)
- session: removed session.remove(key), just use session.set(key, null)
- redis: log returned value
- validator: log bean if failed to validate, to help troubleshooting
- http: removed module config, route(), replaced with http().route(method,path,controller)
- http: mask json response, beanBody/httpClient
- http: set tcp keep alive timeout to 620s, to adapt to both AWS ELB and gcloud LB
- http: move graceful shutdown logic into IO thread, and not count as action, before reading request
- http: process health-check in IO thread
- log-processor: fix index template should use strict_date_optional_time as date format
- ws: draft websocket impl
- api: tweak api client retry, shorten keep alive timeout, and retry on socketException (for connection reset)
- db: skip query.fetch() if limit is 0
- search: update to 6.4.0, !!! removed @Index(index, type), replaced with @Index(name) as multiple types per index is deprecated and will be removed in ES 7.0 switched to java high level client, which uses HTTP:9200 removed DeleteByQuery, as no actual usage for now, can be substituted by ForEach if needed
- cache: support evict and put multiple keys, change keys to Collection, since in real case, keys usually are constructed from List or Set
- kafka: removed /_sys kafka admin support, for actual cases, it's still better use kafka provided scripts
- log-processor: change index pattern to name-yyyy.MM.dd, to make clean index job works with metricbeats
- utils: deleted core.framework.util.Charsets, in favor of java.nio.charset.StandardCharsets (as IDE prompts)
- kafka: tweak producer, added request rate/size metrics, enable compression
- log: make log-processor leverage built-in kafak support, to index action as well
- redis,mongo,elasticserch: fix forEach elapsed time tracking
- redis: replace setnx with set command
- kafka: update to 2.0.0
- redis: check values must not be empty in encoding
- log: remove track debug log to make trace log concise
- redis: support List
- db: support batchExecute
- search: support update by script
- http: update undertow to 2.0.11
- db: !!! replaced repository.update with repository.partialUpdate(), repository.update will update all fields and field can be NULL
- search: update to 6.3.2
- util: removed Lists.newArrayList(...values)/Sets.newHashSet(...values)/Maps.newHashMap(k,v) in favor of JDK 10 builtin List/Set/Map.of/copyOf
- html: !!! changed Message.get(key) to return String rather than Optional, and throw error if key not existed, to make it more strict
- kafka: update to 1.1.1
- db: fix SQLParams may throw exception which cause action log failed to send
- template: support data url in html template
- validation: if bean has default value, it requires to put @NotNull (this may break, review all request/response, and run unit test to verify)
- web: query param mapper ignore not existed keys, in order to support default value in query bean
due to removed support List as request/response type
- web: request.bean must pass class type
- validation: remove support to validate top level List
- validation: converted core.framework.impl.validate.ValidationException as internal exception, to treat as internal error, in application level, use BadRequestException instead
- webservice: validate response bean on both controller and webserviceClient
- json: removed Optional support and jackson dependency, as only use case of Optional is responseType
- test: renamed methods in core.framework.test.Assertions, to avoid conflicting with assertj
- site: !!! updated csp design, allow to specify entire csp to keep maximum flexibility, this is due to difficulty of external sdk integration, e.g. facebook, ga
use site().security().csp(csp) or sys.webSecurity.csp to configure, site().security() will enable rest headers without csp - sys: validate keys in sys.properties
- api: removed support List as request/response due to security concerns
- api: added api().httpClient() to configure http client for api client, and potentially support local experiment code to call website/interface directly
- api: retry on NoHttpResponseException with non-idempotent methods, trade off for availability/complexity, best result we can get with keep-alive + graceful shutdown + retry
- session: !!! redis session key changed from "sessionId:{id}" to "sessionId:{sha256(id)}" for security reason, so the redis log won't show clear text session id value, update lib will lose all existing user session, please deploy on scheduled time
- httpClient: added retry support, refer to core.framework.internal.http.RetryHandler for details make API client to retry (in kube env, persistent connection can be stale during deployment)
- api: make RemoteServiceException exposes https status
- site: make default error handler fits most of cases to reduce the need of creating custom error handler
- make http server / scheduler / kafka listener / executor shutdown gracefully
- httpClient: added DefaultServiceUnavailableRetryStrategy, enable evictIdleConnections
- api: assert same service interface must not have duplicate method name
- search: update to 6.3.0, add analysis-common plugin for stemmer support during test
- executor: support to submit void task as syntax sugar
- api: tweak enum generation to use JAVA_ENUM_NAME: "PROPERTY_VALUE" pattern
- db: added db().batchSize() to configure batch size on batchInsert and batchDelete
- db: enable rewriteBatchedStatements=true for MySQL
- api: retire old /_sys/api, promote v2
- test: replace EnumConversionValidator/EnvResourceValidator with assertJ extension, added validator assertions, refer to core.framework.test.Assertions
- api: make error message more friendly when service response returns value type
- api: add validation to prevent from using same simple class name for service interface and request/response bean, in order to improve maintainability and simplify API typescript definition generation
- api: added /_sys/api/v2 for new version of api definition exposing (client js will read json and generate client code directly)
- db: fix sql params log should log @DBEnumValue value instead of enum.name()
- site: finalize csp design, make img-src supports data:, use sys.webSecurity.trustedSources to configure
- http: update undertow to 2.0.9
- search: support auto complete
- site: update site.enableWebSecurity(String... trustedSources) to use CSP to replace x-frame-options since it's deprecated
- executor: tweak use case when task submit another task to executor, to support async long polling or retry use cases
- http: update undertow to 2.0.7
- api: make error message more friendly when service method param misses @PathParam
- config: make Config class stateful, use override style to configure during test
- search: !!! moved search to core-ng-search/core-ng-search-test modules, use config(SearchConfig.class)/config(InitSearchConfig.class) to configure, refer to log-processor gradle config for dependency config
- mongo: !!! moved mongo to core-ng-mongo/core-ng-mong-test modules, use config(MongoConfig.class) to configure
- db: Query added fetchOne()
- scheduler: log error when trigger returned invalid next execution time
- thread: removed core.framework.util.Threads.availableProcessors, since java 10 supports cpu limits well in docker/kube
- db: update connection max idle timeout to 1 hour, to fit most scenario (e.g. IDC with firewall)
- scheduler: removed secondly trigger, replaced with custom trigger to make it more flexible
- search: update es to 6.2.4
- executor: !!! removed built-in Executor binding, please use executor().add(); in config to keep same behavior, allow executor().add(name, poolSize) to create multiple pools
- bug: fix site().publishAPI(cidrs) not setting cidr correctly
- kafka: added POST /_sys/kafka/topic/:topic/message/:messageId, to allow publish message thru internal management API
- bug: fix array param format
- bug: fix multiple kafka management controller conflict
- site: added publishAPI() / sys.site.publishAPI.allowCIDR to allow access /sys/_api from trusted network
- kafka: update to 1.1.0, add management controller method to increase partition/delete records
- search: update es to 6.2.3
- db: add query.project()
- action: simplify actionId naming scheme, since actionId doesn't need to be used in path anymore examples: action=api:patch:/ajax/product/:id, action=http:get:/, action=topic:some-topic, action=job:some-job
- api: typescript definition generates string enum
- kafka: update to 1.0.1, add config to register publisher without topic (instead of passing null)
- http: update undertow to 2.0.3
- log: log masking redesign, only customer data from form param/json body require masking mask sessionId value in cookies log !!! removed CustomMessageFilter, replace with masked field to simplify usage
- jdk: drop java 8 support
- log: removed write action/trace to file, updated sys.properties log key to sys.log.appender sys.log.appender=console => write action/trace to console sys.log.appender=kafkaURI => forward log to kafka (in cloud env, console logging is preferred no matter it's docker or systemd/journald, or use log forwarding)
- http: update undertow to 2.0.1
- http: limit max requestURL length to 1000
- search: update es to 6.2.2
- http: not handling /health-check as action anymore, means /health-check will not be part of action log, to reduce noisy in kube env
- httpClient: removed HTTPRequest static method shortcut, use new HTTPRequest(method, uri) instead, to enforce consistent api style
- api: tweak api generation
- redis: support increaseBy
- api: tweak api generation to fit client impl, refer to https://github.com/neowu/frontend-demo-project/blob/master/website-frontend-ts/src/service/user.ts as example
- inject: bind(object) will inject object, to make it easier to register bean with both manual wired and autowired dependencies
- properties: removed support of loading properties from file path, for kube we will using env/jvm argument overriding
- httpClient: add basic auth support
- search: update es to 6.2.1
- http: add ContentType.IMAGE_PNG constant, (e.g. used by captcha controller)
- api: change returned content type to javascript, make it easier to view by browser
- test: removed core.framework.test.EnvWebValidator, since all static file/content will be handled in frontend project, copy the impl to your own project if you still need
- http: set max entity size to 10M, to prevent from large post body
- http: support text/xml as body
- http: change ip whitelist to cidr to support subnet
- api: support configure api client timeout and slow operation threshold (default is 30s and 15s)
- log: add redis read/write entries tracking, index read/write entries as null if not set
- search: update es to 6.1.3
- api: add "_sys/api" to return typescript definition, used by frontend
- html: added "autofocus", "allowfullscreen", "hidden" & "async" to boolean attributes
- api: remove openapi impl, not useful in actual development life cycle, we plan to impl tool to generate typescript ajax client from interface directly
- api: temporary removed: update API client with 30s timeout and 15s slow operation threshold (will support configure in next version)
- search: update es to 6.1.2
- html: added "required", "sortable" to boolean attributes
- http: added gzip support, added cache param for static content
- api: update API client with 30s timeout and 15s slow operation threshold
- http: support ip whitelist(due to gcloud public LB does not support ingress IP restriction), update undertow to 1.4.22
- db: change mysql jdbc driver to com.mysql.cj.jdbc.Driver (works with mysql:mysql-connector-java:6.0.6), old one is deprecated, https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-api-changes.html
- test: update junit to 5.0.3 and tweak dependency
- search: update es to 6.1.1
- http: support patch method, for partial update webservice, e.g. update status for one entity