Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cut-over locks not released when gh-ost pauses mid-cut-over #1407

Closed
timvaillancourt opened this issue Apr 11, 2024 · 0 comments · Fixed by #1406
Closed

cut-over locks not released when gh-ost pauses mid-cut-over #1407

timvaillancourt opened this issue Apr 11, 2024 · 0 comments · Fixed by #1406
Assignees

Comments

@timvaillancourt
Copy link
Collaborator

timvaillancourt commented Apr 11, 2024

During the cut-over operation gh-ost issues a lock tables on the tables before they're renamed. After the rename an unlock tables is issued to unlock the tables

Today, if gh-ost pauses/freezes (process remains running but is unresponsive due to a host problem) between the lock tables and unlock tables, the locks are not released. We haven't explained what could cause the host running gh-ost to essentially freeze execution, but we had this occur in production and locks were never released until the MySQL wait_timeout (for killing idle connections)

This theoretically can be reproduced by:

  1. Adding a pause after the lock tables step in the cut-over (hand-wavy)
  2. Freeze the gh-ost process with kill -TSTP [pid] or kill -STOP [pid]
  3. Observe the table locks never getting released until wait_timeout (default 30 minutes)

To address this, I plan to shorten the wait_timeout of the applier MySQL session during cut-over only, as this is the only time where a short idle timeout is advantageous. After the cut-over the wait_timeout for the session will be restored to the server default

This work began in #1401 and is completed with #1406

@timvaillancourt timvaillancourt self-assigned this Apr 11, 2024
@timvaillancourt timvaillancourt changed the title cut-over locks never released when gh-ost terminates cut-over locks never released when gh-ost pauses Apr 11, 2024
@timvaillancourt timvaillancourt changed the title cut-over locks never released when gh-ost pauses cut-over locks never released when gh-ost pauses mid-cut-over Apr 11, 2024
@timvaillancourt timvaillancourt linked a pull request Apr 11, 2024 that will close this issue
2 tasks
@timvaillancourt timvaillancourt changed the title cut-over locks never released when gh-ost pauses mid-cut-over cut-over locks not released when gh-ost pauses mid-cut-over Apr 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant