You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
During the cut-over operation gh-ost issues a lock tables on the tables before they're renamed. After the rename an unlock tables is issued to unlock the tables
Today, if gh-ost pauses/freezes (process remains running but is unresponsive due to a host problem) between the lock tables and unlock tables, the locks are not released. We haven't explained what could cause the host running gh-ost to essentially freeze execution, but we had this occur in production and locks were never released until the MySQL wait_timeout (for killing idle connections)
This theoretically can be reproduced by:
Adding a pause after the lock tables step in the cut-over (hand-wavy)
Freeze the gh-ost process with kill -TSTP [pid] or kill -STOP [pid]
Observe the table locks never getting released until wait_timeout (default 30 minutes)
To address this, I plan to shorten the wait_timeout of the applier MySQL session during cut-over only, as this is the only time where a short idle timeout is advantageous. After the cut-over the wait_timeout for the session will be restored to the server default
This work began in #1401 and is completed with #1406
The text was updated successfully, but these errors were encountered:
timvaillancourt
changed the title
cut-over locks never released when gh-ost terminates
cut-over locks never released when gh-ost pauses
Apr 11, 2024
timvaillancourt
changed the title
cut-over locks never released when gh-ost pauses
cut-over locks never released when gh-ost pauses mid-cut-over
Apr 11, 2024
timvaillancourt
changed the title
cut-over locks never released when gh-ost pauses mid-cut-over
cut-over locks not released when gh-ost pauses mid-cut-over
Apr 11, 2024
During the cut-over operation
gh-ost
issues alock tables
on the tables before they're renamed. After the rename anunlock tables
is issued to unlock the tablesToday, if
gh-ost
pauses/freezes (process remains running but is unresponsive due to a host problem) between thelock tables
andunlock tables
, the locks are not released. We haven't explained what could cause the host runninggh-ost
to essentially freeze execution, but we had this occur in production and locks were never released until the MySQLwait_timeout
(for killing idle connections)This theoretically can be reproduced by:
lock tables
step in the cut-over (hand-wavy)gh-ost
process withkill -TSTP [pid]
orkill -STOP [pid]
wait_timeout
(default 30 minutes)To address this, I plan to shorten the
wait_timeout
of the applier MySQL session during cut-over only, as this is the only time where a short idle timeout is advantageous. After the cut-over thewait_timeout
for the session will be restored to the server defaultThis work began in #1401 and is completed with #1406
The text was updated successfully, but these errors were encountered: