AWS RDS PostgreSQL Upgrade 15.4 -> 16.2 Fails

Hi, I have attempted to upgrade an RDS PostgreSQL cluster with only one writer and no AZ or read replica in any other regions. So pretty simple setup.

While attempting upgrade based on the table here https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html#USER_UpgradeDBInstance.PostgreSQL.MajorVersion

But when I select Modify DB and Apply Upgrades, I am getting Exception and Upgrade Fails.

Following are log snippets:

pg_upgrade_internal.log

target databases: Database: template1 relname: pg_catalog.pg_largeobject: reloid: 2613 reltblspace: relname: pg_catalog.pg_largeobject_loid_pn_index: reloid: 2683 reltblspace: Database: postgres relname: pg_catalog.pg_largeobject: reloid: 2613 reltblspace: relname: pg_catalog.pg_largeobject_loid_pn_index: reloid: 2683 reltblspace: Checking for contrib/isn with bigint-passing mismatch ok executing: SELECT pg_catalog.set_config('search_path', '', false); Checking for presence of required libraries fatal Your installation references loadable libraries that are missing from the new installation. You can add these libraries to the new installation, or remove the functions using them from the old installation. A list of problem libraries is in the file: /rdsdbdata/db/pg_upgrade_output.d/20240722T140203.221/loadable_libraries.txt "/rdsdbbin/aurora-16.2.16.2.2.6167.0/bin/pg_ctl" -w -D "/rdsdbdata/db" -o "--config_file=/rdsdbdata/config_new/postgresql.conf --survivable_cache_mode=off" -m fast stop >> "/rdsdbdata/db/pg_upgrade_output.d/20240722T140203.221/log/pg_upgrade_server.log" 2>&1

pg_upgrade_server.log

command: "/rdsdbbin/aurora-16.2.16.2.2.6167.0/bin/pg_ctl" -w -l "/rdsdbdata/db/pg_upgrade_output.d/20240722T140203.221/log/pg_upgrade_server.log" -D "/rdsdbdata/db" -o "-p 5433 -b -c synchronous_commit=off -c fsync=off -c full_page_writes=off --config_file=/rdsdbdata/config_new/postgresql.conf --survivable_cache_mode=off -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/rdsdbdata/log/upgrade' --rds.metadata_quick_recovery_mode=off --rds.buffer_cache_quick_recovery_mode=off" start >> "/rdsdbdata/db/pg_upgrade_output.d/20240722T140203.221/log/pg_upgrade_server.log" 2>&1 waiting for server to start....2024-07-22 14:03:49.881 GMT [1018] LOG: pgaudit extension initialized .2024-07-22 14:03:51.805 GMT [1018] LOG: starting PostgreSQL 16.2 on x86_64-pc-linux-gnu, compiled by x86_64-pc-linux-gnu-gcc (GCC) 9.5.0, 64-bit .2024-07-22 14:03:51.812 GMT [1018] LOG: listening on Unix socket "/rdsdbdata/log/upgrade/.s.PGSQL.5433" 2024-07-22 14:03:51.816 GMT [1018] LOG: Waiting for runtime initialization complete... ....2024-07-22 14:03:55.842 GMT [1038] LOG: database system was shut down at 2023-09-06 19:24:58 GMT 2024-07-22 14:03:56.772 GMT [1038] LOG: corrupted statistics file "pg_stat/pgstat.stat" 2024-07-22 14:03:56.793 GMT [1018] LOG: database system is ready to accept connections 2024-07-22 14:03:56.794 GMT [1040] LOG: update plans task started done server started 2024-07-22 14:03:57.047 GMT [1018] LOG: background worker "plans update worker" (PID 1040) exited with exit code 1 2024-07-22 14:03:57.826 GMT [1047] ERROR: could not access file "$libdir/rtpostgis-2.5": No such file or directory 2024-07-22 14:03:57.826 GMT [1047] STATEMENT: LOAD '$libdir/rtpostgis-2.5' command: "/rdsdbbin/aurora-16.2.16.2.2.6167.0/bin/pg_ctl" -w -D "/rdsdbdata/db" -o "--config_file=/rdsdbdata/config_new/postgresql.conf --survivable_cache_mode=off" -m fast stop >> "/rdsdbdata/db/pg_upgrade_output.d/20240722T140203.221/log/pg_upgrade_server.log" 2>&1 waiting for server to shut down....2024-07-22 14:03:57.914 GMT [1018] LOG: received fast shutdown request 2024-07-22 14:03:57.914 GMT [1018] LOG: aborting any active transactions 2024-07-22 14:03:57.914 GMT [1041] LOG: resource monitoring process shutting down 2024-07-22 14:03:57.919 GMT [1036] LOG: shutting down .............2024-07-22 14:04:11.834 GMT [1018] LOG: database system is shut down done server stopped

List of installed extensions:

                                 List of installed extensions

Name | Version | Schema | Description ---------+---------+------------+--------------------------------------------------------------------- plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language postgis | 3.3.3 | public | PostGIS geometry, geography, and raster spatial types and functions (2 rows)

No extensions were removed or added pre-upgrade or post-upgrade

Appreciate any feedback/pointers.