Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qps/rate limit & reduce expire seek range & expire-key hash #204

Open
wants to merge 61 commits into
base: master
Choose a base branch
from

Commits on Sep 3, 2019

  1. Configuration menu
    Copy the full SHA
    db84a39 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3e4b89e View commit details
    Browse the repository at this point in the history
  3. close the connection when meet empty command

    xufangping committed Sep 3, 2019
    Configuration menu
    Copy the full SHA
    c89b27f View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2019

  1. some illegal clients send unreadable command which may cause titan cr…

    …ash. just after read the command we check if it is in command list we support, else will skip it
    xufangping committed Sep 6, 2019
    Configuration menu
    Copy the full SHA
    43bd6c3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    95bd04b View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2019

  1. add expire left seconds metrics

    xufangping committed Sep 18, 2019
    Configuration menu
    Copy the full SHA
    63b131d View commit details
    Browse the repository at this point in the history
  2. fix expire left time calculating

    xufangping committed Sep 18, 2019
    Configuration menu
    Copy the full SHA
    a9c28fe View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2019

  1. Configuration menu
    Copy the full SHA
    e5ff501 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2019

  1. delete zadd args output

    xufangping committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    fb11c67 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2019

  1. first version of rate limit

    xufangping committed Nov 7, 2019
    Configuration menu
    Copy the full SHA
    586b30d View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2019

  1. 1 use rate.limiter to implement limit and read tikv key/value to get …

    …new limit
    
    2 just balance limit base in active titan server num
    3 unit test of rate limit passed
    xufangping committed Nov 21, 2019
    Configuration menu
    Copy the full SHA
    27f54e7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e5018e4 View commit details
    Browse the repository at this point in the history
  3. 1 change log level--limit not set, to debug

    2 add setlimit.sh
    xufangping committed Nov 21, 2019
    Configuration menu
    Copy the full SHA
    ecd78a2 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0648ab8 View commit details
    Browse the repository at this point in the history
  5. in startSyncNewLimit, just read all match limit once for every comman…

    …d's qps/rate
    xufangping committed Nov 21, 2019
    Configuration menu
    Copy the full SHA
    f86245a View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2019

  1. change got limit log trace

    xufangping committed Nov 22, 2019
    Configuration menu
    Copy the full SHA
    157d70a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fd0d8cc View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    756a0e6 View commit details
    Browse the repository at this point in the history
  4. fix titan active time decoding bug

    xufangping committed Nov 22, 2019
    Configuration menu
    Copy the full SHA
    46bb6da View commit details
    Browse the repository at this point in the history
  5. fix limit local percent bug

    xufangping committed Nov 22, 2019
    Configuration menu
    Copy the full SHA
    a839995 View commit details
    Browse the repository at this point in the history
  6. fix titan active time parsing bug

    xufangping committed Nov 22, 2019
    Configuration menu
    Copy the full SHA
    815efa4 View commit details
    Browse the repository at this point in the history
  7. add updateLimitPercent log

    xufangping committed Nov 22, 2019
    Configuration menu
    Copy the full SHA
    9612571 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    cc4255a View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2019

  1. Configuration menu
    Copy the full SHA
    f00d8c5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    da49bfa View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2019

  1. limit can also work on auth-disabled titan server

    xufangping committed Nov 26, 2019
    Configuration menu
    Copy the full SHA
    30f7d2c View commit details
    Browse the repository at this point in the history
  2. decrease the lock range in reportStat

    xufangping committed Nov 26, 2019
    Configuration menu
    Copy the full SHA
    bcbd5a7 View commit details
    Browse the repository at this point in the history
  3. reportLocalStat run in itself go routine

    xufangping committed Nov 26, 2019
    Configuration menu
    Copy the full SHA
    1a50695 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2019

  1. 1 change how to balance

    2 change qps/rate metrics 3 labels: namespace, command, localip
    xufangping committed Nov 29, 2019
    Configuration menu
    Copy the full SHA
    a7753c4 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2019

  1. update some config item's description and log

    xufangping committed Dec 2, 2019
    Configuration menu
    Copy the full SHA
    fda1e66 View commit details
    Browse the repository at this point in the history
  2. add limit changed log

    xufangping committed Dec 2, 2019
    Configuration menu
    Copy the full SHA
    a474656 View commit details
    Browse the repository at this point in the history
  3. commandLimiter store globalQpsLimit/globalRateLimit in int64, use it …

    …to calculate the local limit in updatePercent() and judge if limit is changed, this resolve float precision lost problem
    xufangping committed Dec 2, 2019
    Configuration menu
    Copy the full SHA
    8ed0698 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2019

  1. when all titan'a limiter qps is < devideUsage of its weight percent, …

    …add current titan weight again
    xufangping committed Dec 3, 2019
    Configuration menu
    Copy the full SHA
    6ae93be View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2019

  1. Configuration menu
    Copy the full SHA
    70b3f15 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2019

  1. 1 resolve the problem that titan still process command for remote clo…

    …sed pipelined connection
    
    2 add namespace log for readCommand failure and client processing go routine's exit
    xufangping committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    a22ca0f View commit details
    Browse the repository at this point in the history
  2. 1 BytesArray add judge for write eror

    2 use remoteClosed flag to interrupt processing when pipeline connection is closed. if close ctx.Done(channel), and write() again, titan will crush for re-close channel
    xufangping committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    8403a21 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2019

  1. refactor commandLimiter's qpsLimiter/ratelimiter and their globalLimi…

    …t, lock, localPercent to a LimiterWrapper, and decrease the lock range
    xufangping committed Dec 7, 2019
    Configuration menu
    Copy the full SHA
    2d59108 View commit details
    Browse the repository at this point in the history
  2. fix balance limit's bug

    xufangping committed Dec 7, 2019
    Configuration menu
    Copy the full SHA
    b034581 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2019

  1. 1 in Exec(), add transaction begin, cmd args num, cmd func, reply fun…

    …c metrics
    
    2 cmd args num metrics include the key, which doesn't before
    3 in AutoCommit, do adding transaction begin, cmd func, reply func metrics before log.debug()
    xufangping committed Dec 11, 2019
    Configuration menu
    Copy the full SHA
    3b71e69 View commit details
    Browse the repository at this point in the history
  2. simplify the msg recv/send log in debug level

    xufangping committed Dec 11, 2019
    Configuration menu
    Copy the full SHA
    0fb99b7 View commit details
    Browse the repository at this point in the history
  3. fix bug of setting cmd arg num metric in Exec()

    xufangping committed Dec 11, 2019
    Configuration menu
    Copy the full SHA
    37c1c68 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2019

  1. add expire/gc round/seek/commit cost metrics

    xufangping committed Dec 12, 2019
    Configuration menu
    Copy the full SHA
    9de2ae9 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2019

  1. prevent connection more and more: when tikv command cost high, client…

    …s reconnect before it finish
    xufangping committed Dec 22, 2019
    Configuration menu
    Copy the full SHA
    6c2328d View commit details
    Browse the repository at this point in the history

Commits on Dec 25, 2019

  1. fix bug of connection limit

    lugang committed Dec 25, 2019
    Configuration menu
    Copy the full SHA
    bbfce19 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2020

  1. 1 add metrics range, fix cost exceed max value problem

    2 add config item to enable/disable max-connection limit
    3 add txn failure metrics in multi/exec scenario
    xufangping committed Feb 18, 2020
    Configuration menu
    Copy the full SHA
    93cd7df View commit details
    Browse the repository at this point in the history
  2. Merge branch 'fpxu_dev' into 'master'

    Fpxu dev
    
    See merge request zz-arch/zhuanzhuan_zzkv!13
    xufangping committed Feb 18, 2020
    Configuration menu
    Copy the full SHA
    7e2521b View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2020

  1. Configuration menu
    Copy the full SHA
    ad7cff1 View commit details
    Browse the repository at this point in the history
  2. 1 fix unlock bug in limit-connection

    xufangping committed Feb 19, 2020
    Configuration menu
    Copy the full SHA
    aeca4b3 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bb617b0 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2020

  1. Configuration menu
    Copy the full SHA
    003e6b6 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2020

  1. fix a failed case in tests/redis/unit/expire.tcl

    xufangping committed Feb 26, 2020
    Configuration menu
    Copy the full SHA
    4423eaf View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2020

  1. expire seek start in last processed expireKeys to reduce rocksdb toms…

    …tone problem
    xufangping committed Feb 27, 2020
    Configuration menu
    Copy the full SHA
    cc15799 View commit details
    Browse the repository at this point in the history
  2. fix expire log

    xufangping committed Feb 27, 2020
    Configuration menu
    Copy the full SHA
    d9a1baf View commit details
    Browse the repository at this point in the history
  3. also reduce expire seek range when no expire keys or all expire keys …

    …> now
    xufangping committed Feb 27, 2020
    Configuration menu
    Copy the full SHA
    83a370d View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2020

  1. also reduce expire seek range when no expire keys or all expire keys …

    …> now
    xufangping committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    d75dd34 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2020

  1. comment unsupported string functions in command list, and delete its …

    …redis test case
    xufangping committed Mar 2, 2020
    Configuration menu
    Copy the full SHA
    3519ff9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c4eea83 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2020

  1. hash expire-key, it will improve the keys num handled every seconds, …

    …and prevent that current expire-keys region writing node have higher load than other nodes
    xufangping committed Apr 21, 2020
    Configuration menu
    Copy the full SHA
    bd43ec2 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2020

  1. 1 fix runExpire's problem that unhashed goroutine scan hashed expire-key

    2 fix expire_test.go: gc key can get before runExpire/doExpire if expireat < now; add 2 testCases for hash/string expire: check key is deleted and gc key exists after runExpire
    xufangping committed Apr 24, 2020
    Configuration menu
    Copy the full SHA
    fe6e84b View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2020

  1. unhash expire goroutine use individual limit configuration item、expir…

    …ed&round&seek&commit metrics label
    xufangping committed Apr 26, 2020
    Configuration menu
    Copy the full SHA
    977b457 View commit details
    Browse the repository at this point in the history
  2. buckets of comand/limit/commandFunc/txn commit cost is too many to lo…

    …ad for dashboard, replace 1.4 with 2, buckets num decrease 50%
    xufangping committed Apr 26, 2020
    Configuration menu
    Copy the full SHA
    9161d7e View commit details
    Browse the repository at this point in the history