Skip to content

RDS MySQL — Can’t drop schema due to Hangfire connections (DDL stuck on metadata locks)

0

Service/Resource: Amazon RDS for MySQL

Urgency: Maintenance blocked

Issue: Unable to drop schema ---_SCHEMA. DROP TABLE / DROP DATABASE stays “Waiting for table metadata lock” even after stopping Hangfire services; the services/jobs keep restarting and re‑connecting, so DDL never completes.

What we’ve tried:

  • Stopped/disabled Hangfire services (they restart/reconnect unexpectedly).
  • Killed sessions with CALL mysql.rds_kill(---) / CALL mysql.rds_kill_query(---).
  • Dropped Hangfire tables individually with FOREIGN_KEY_CHECKS=0 (still hits MDL waits).
  • Checked performance_schema.metadata_locks (often empty; MDL instrumentation uncertain).
  • Temporarily changed Security Group inbound rules
  • Tried short timeouts (metadata_lock_timeout, wait_timeout, interactive_timeout) to fail fast.

Blockers observed:

  • Persistent metadata locks from idle/sleep sessions on ---_SCHEMA.
  • Hangfire jobs immediately re‑acquire locks after kills.
  • DDL stuck; maintenance window slipping.

Kindly Help Me To

  1. Any RDS‑specific constraints or steps to safely force schema/table drop when application processes keep re‑attaching.
  2. Recommended timeout values (metadata_lock_timeout, wait_timeout, interactive_timeout) to avoid long DDL stalls.
  3. Guidance to prevent re‑connects during maintenance (SG/NACL patterns), and whether Event Scheduler can safely call mysql.rds_kill on RDS.
  4. Best way on RDS MySQL to identify and kill GRANTED MDL holders for a specific schema/table (and confirm if/how to enable MDL instrumentation in Performance Schema).

Goal: Safely drop the Hangfire tables and/or the entire ---_SCHEMA without prolonged metadata lock waits or service impact.

asked 9 months ago201 views

1 Answer
0

This is a common issue with metadata locks in RDS MySQL when applications maintain persistent connections. Here's how to address your situation:

  1. RDS-specific approaches to force schema/table drop:
  • Use CALL mysql.rds_kill_query() to terminate specific blocking queries rather than entire sessions
  • For persistent issues, consider a brief maintenance window where you:
  • Modify security groups to block all application traffic temporarily
  • Kill all existing connections with mysql.rds_kill()
  • Execute your DDL operations
  • Restore security group settings
  • Note that dropping databases created via the RDS CreateDBInstance API is not allowed; only databases created using rdsadmin.create_database can be dropped
  1. Recommended timeout values:
  • Set lock_wait_timeout (not metadata_lock_timeout) to a reasonable value like 60-300 seconds
  • Configure wait_timeout and interactive_timeout to shorter values (e.g., 300-600 seconds) during maintenance to prevent long-lived idle connections
  1. Preventing reconnects during maintenance:
  • Temporarily modify security groups to block application traffic completely (not just restricting to 0.0.0.0/32 which is invalid)
  • Use Network ACLs as a secondary defense if security groups aren't sufficient
  • The Event Scheduler can be used to call mysql.rds_kill on RDS, but ensure it doesn't conflict with other maintenance operations
  1. Identifying and killing MDL holders:
  • In Aurora MySQL version 2, you can identify blocked sessions by querying performance_schema tables or sys schema views
  • Example query to find blocking sessions:
SELECT blocking_session_id FROM performance_schema.metadata_locks ml
JOIN performance_schema.threads t ON ml.owner_thread_id = t.thread_id
WHERE ml.object_schema = 'your_schema_name'
  • To enable MDL instrumentation, ensure the Performance Schema is enabled and the metadata_locks consumer is turned on
  • Use CALL mysql.rds_kill() with the identified session IDs

For your Hangfire issue specifically, I recommend temporarily blocking all application traffic at the security group level, killing all existing connections, performing your DDL operations, and then restoring access. This approach ensures the applications can't reconnect during the critical maintenance window.
Sources
Troubleshooting for Amazon RDS for Db2 - Amazon Relational Database Service
MySQL RDS instance connection error | AWS re:Post
synch/cond/sql/MDL_context::COND_wait_status - Amazon Aurora

answered 9 months ago

  • What I already tried:

    • Stopped Hangfire services, but workers restart/reconnect.
    • Killed sessions via CALL mysql.rds_kill(---) / CALL mysql.rds_kill_query(---) based on SHOW FULL PROCESSLIST.
    • Dropped Hangfire tables individually with FOREIGN_KEY_CHECKS=0; DDL still hits MDL waits.
    • Checked performance_schema.metadata_locks (sometimes empty; MDL instrumentation status unclear).
    • Temporarily modified Security Group rules; plan to revert to least privilege. SGs are allow‑lists.

    Clarification (MySQL on RDS): I understand DROP DATABASE is allowed for normal MySQL schemas (not system schemas). The earlier note about only dropping DBs created via rdsadmin.create_database applies to RDS Oracle, not RDS MySQL.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.