{"id":808,"date":"2021-01-25T19:52:12","date_gmt":"2021-01-25T18:52:12","guid":{"rendered":"https:\/\/spoton.cz\/?p=808"},"modified":"2021-01-25T19:54:10","modified_gmt":"2021-01-25T18:54:10","slug":"starting-mysqld-daemon-with-databases-from-config-databases-loop","status":"publish","type":"post","link":"https:\/\/spoton.cz\/index.php\/2021\/01\/25\/starting-mysqld-daemon-with-databases-from-config-databases-loop\/","title":{"rendered":"“Starting mysqld daemon with databases from \/config\/databases” loop"},"content":{"rendered":"\n

MariaDB, or MySQL is stuck in the startup loop with a vague error: “Starting mysqld daemon with databases from \/config\/databases”. In fact, it is an error, where the InnoDB portion encountered corrupt data \/ tables. <\/p>\n\n\n\n

What was the reason for it and how to resolve this issue? Read on.<\/p>\n\n\n\n\n\n\n\n

The problem<\/h2>\n\n\n\n

I’m lazy. I’m impatient. I don’t document stuff, that I just play with. And so, I did not document the correct naming scheme for my DB docker containers. And so it happened, that I scaled up a production, non-cluster enabled, single instance MariaDB to a total number of 3. Guess what? I corrupted my DB. The problem did not appear right away, it took some time, but I got this:<\/p>\n\n\n\n

p_db.1.9j4hoc3ck4e1@swarm3    | [s6-init] making user provided files available at \/var\/run\/s6\/etc...exited 0.\np_db.1.9j4hoc3ck4e1@swarm3    | [s6-init] ensuring user provided files have correct perms...exited 0.\np_db.1.9j4hoc3ck4e1@swarm3    | [fix-attrs.d] applying ownership & permissions fixes...\np_db.1.9j4hoc3ck4e1@swarm3    | [fix-attrs.d] done.\np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] executing container initialization scripts...\np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 01-envfile: executing... \np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 01-envfile: exited 0.\np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 10-adduser: executing... \np_db.1.9j4hoc3ck4e1@swarm3    | usermod: no changes\np_db.1.9j4hoc3ck4e1@swarm3    | \np_db.1.9j4hoc3ck4e1@swarm3    | -------------------------------------\np_db.1.9j4hoc3ck4e1@swarm3    |           _         ()\np_db.1.9j4hoc3ck4e1@swarm3    |          | |  ___   _    __\np_db.1.9j4hoc3ck4e1@swarm3    |          | | \/ __| | |  \/  \\ \np_db.1.9j4hoc3ck4e1@swarm3    |          | | \\__ \\ | | | () |\np_db.1.9j4hoc3ck4e1@swarm3    |          |_| |___\/ |_|  \\__\/\np_db.1.9j4hoc3ck4e1@swarm3    | \np_db.1.9j4hoc3ck4e1@swarm3    | \np_db.1.9j4hoc3ck4e1@swarm3    | Brought to you by linuxserver.io\np_db.1.9j4hoc3ck4e1@swarm3    | -------------------------------------\np_db.1.9j4hoc3ck4e1@swarm3    | \np_db.1.9j4hoc3ck4e1@swarm3    | To support LSIO projects visit:\np_db.1.9j4hoc3ck4e1@swarm3    | https:\/\/www.linuxserver.io\/donate\/\np_db.1.9j4hoc3ck4e1@swarm3    | -------------------------------------\np_db.1.9j4hoc3ck4e1@swarm3    | GID\/UID\np_db.1.9j4hoc3ck4e1@swarm3    | -------------------------------------\np_db.1.9j4hoc3ck4e1@swarm3    | \np_db.1.9j4hoc3ck4e1@swarm3    | User uid:    911\np_db.1.9j4hoc3ck4e1@swarm3    | User gid:    911\np_db.1.9j4hoc3ck4e1@swarm3    | -------------------------------------\np_db.1.9j4hoc3ck4e1@swarm3    | \np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 10-adduser: exited 0.\np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 30-config: executing... \np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 30-config: exited 0.\np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 40-initialise-db: executing... \np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 40-initialise-db: exited 0.\np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 99-custom-scripts: executing... \np_db.1.9j4hoc3ck4e1@swarm3    | [custom-init] no custom files found exiting...\np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] 99-custom-scripts: exited 0.\np_db.1.9j4hoc3ck4e1@swarm3    | [cont-init.d] done.\np_db.1.9j4hoc3ck4e1@swarm3    | [services.d] starting services\np_db.1.9j4hoc3ck4e1@swarm3    | [services.d] done.\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:22:27 mysqld_safe Logging to syslog.\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:22:27 mysqld_safe Starting mysqld daemon with databases from \/config\/databases\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:23:00 mysqld_safe Logging to syslog.\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:23:00 mysqld_safe Starting mysqld daemon with databases from \/config\/databases\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:23:33 mysqld_safe Logging to syslog.\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:23:33 mysqld_safe Starting mysqld daemon with databases from \/config\/databases\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:24:06 mysqld_safe Logging to syslog.\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:24:06 mysqld_safe Starting mysqld daemon with databases from \/config\/databases\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:24:39 mysqld_safe Logging to syslog.\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:24:39 mysqld_safe Starting mysqld daemon with databases from \/config\/databases\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:25:12 mysqld_safe Logging to syslog.\np_db.1.9j4hoc3ck4e1@swarm3    | 210125 12:25:12 mysqld_safe Starting mysqld daemon with databases from \/config\/databases\n<\/pre>\n\n\n\n
This keeps on looping over and over again, as the daemon tries to start an SQL instance, but something is not quite right and it fails over and over again. <\/p>\n\n\n\n
After some time, I kindof figured out, that it might be corruption of the tablespaces, or something similar, beats me. I did not have proper logging set up, as I figured out later. Yeah, docker…<\/p>\n\n\n\n
The proposed solution<\/h2>\n\n\n\n
Once almost figured out what theproblem might be, I resorted to looking through https:\/\/dev.mysql.com\/doc\/refman\/5.6\/en\/forcing-innodb-recovery.html <\/a> where I found a hint of a way to make a backup of the data. The fact, that I will have to do that manually did not hurt that much, but the process is not that simple. First of all, putting the Innodb to recovery. There are different levels for that. Nothing lower than 6 worked for me, so I set <\/p>\n\n\n\n
[mysqld] <\/code>\ninnodb_force_recovery = <\/code>6<\/pre>\n\n\n\n
in the \/config\/custom.cnf<\/p>\n\n\n\n
My MySQL runs on NFS, so I have the complete directory of \/config mounted from outside. This includes data, configuration and logs.<\/p>\n\n\n\n
After the recovery is set, the proposed solution is to make a DB dump and then recover the data in a new fresh install. Well, not quite for me. All I got were errors – possibly connected to the corruption itself<\/p>\n\n\n\n
Lost connection to MySQL server during query when dumping table `wp_comments`...<\/code><\/pre>\n\n\n\n
and<\/p>\n\n\n\n
mysqldump: Couldn't execute 'SELECT \/*!40001 SQL_NO_CACHE *\/<\/code>...<\/pre>\n\n\n\n
The real solution that worked for me<\/h2>\n\n\n\n
As a dump of the whole database was not possible for me, but I could clearly list the contents of the whole database, all I did was to find all the tables in my database:<\/p>\n\n\n\n
# to connect to the mysql docker container\ndocker exec -it 2c6effa01d73 \/bin\/bash \n\ncd \/config\n\n# connect to mysql and list the tables\nmysql -u <user> -p\nMariaDB [(none)]> use mydatabase\nMariaDB [mydatabase]> show tables;\n+-----------------------+\n| Tables_in_mydatabase  |\n+-----------------------+\n| wp_commentmeta        |\n| wp_comments           |\n| wp_links              |\n| wp_options            |\n| wp_postmeta           |\n| wp_posts              |\n| wp_term_relationships |\n| wp_term_taxonomy      |\n| wp_termmeta           |\n| wp_terms              |\n| wp_usermeta           |\n| wp_users              |\n+-----------------------+\n12 rows in set (0.001 sec)\n\n<\/pre>\n\n\n\n
OK, got the list of all tables, cool. Now let’s try to save each single one.<\/p>\n\n\n\n
tables='wp_commentmeta       \nwp_comments          \nwp_links             \nwp_options           \nwp_postmeta          \nwp_posts             \nwp_term_relationships\nwp_term_taxonomy     \nwp_termmeta          \nwp_terms             \nwp_usermeta          \nwp_users'\n\nfor table in $tables; do mysqldump -u user -p mydatabase $table > mydatabase.$table.dmp\n# depending on your setup, you might need to type in a password several times, or specify it in the command itself\n\nls\n-rw-r--r-- 1 abc  abc     3802 Jan 25 13:10 custom.cnf\ndrwxr-xr-x 1 abc  abc      260 Jan 25 13:11 databases\ndrwxr-xr-x 1 abc  abc       10 Jan 25 13:10 log\n-rw-r--r-- 1 abc  abc     3397 Jan 25 13:07 mydatabase.wp_commentmeta.dmp\n-rw-r--r-- 1 abc  abc    17412 Jan 25 13:07 mydatabase.wp_comments.dmp\n-rw-r--r-- 1 abc  abc     2797 Jan 25 13:07 mydatabase.wp_links.dmp\n-rw-r--r-- 1 abc  abc   571141 Jan 25 13:07 mydatabase.wp_options.dmp\n-rw-r--r-- 1 abc  abc   910387 Jan 25 13:07 mydatabase.wp_postmeta.dmp\n-rw-r--r-- 1 abc  abc   294184 Jan 25 13:07 mydatabase.wp_posts.dmp\n-rw-r--r-- 1 abc  abc     2191 Jan 25 13:07 mydatabase.wp_termmeta.dmp\n-rw-r--r-- 1 abc  abc     4193 Jan 25 13:07 mydatabase.wp_term_relationships.dmp\n-rw-r--r-- 1 abc  abc     4948 Jan 25 13:07 mydatabase.wp_terms.dmp\n-rw-r--r-- 1 abc  abc     5126 Jan 25 13:07 mydatabase.wp_term_taxonomy.dmp\n-rw-r--r-- 1 abc  abc     3989 Jan 25 13:07 mydatabase.wp_usermeta.dmp\n-rw-r--r-- 1 abc  abc     2852 Jan 25 13:07 mydatabase.wp_users.dmp<\/pre>\n\n\n\n
I should now have all the files with table dumps in my \/config directory. Let’s do the ugly part. Stop the docker container, move the DB files away from \/config\/databases. This path should now be empty.<\/p>\n\n\n\n
Remove the line about innodb recovery mode from \/config\/custom.cnf or comment it out.<\/p>\n\n\n\n
# innodb_force_recovery = 6<\/pre>\n\n\n\n
Start docker container, I should see a clean startup. This time, my DB container is named spoton_cz_DB \ud83d\ude42 As you can see, there is some config t be desired, specifically for the root user, but more on that later.<\/p>\n\n\n\n
spoton_cz_db.1.m0z67958dx89@swarm2    | [s6-init] making user provided files available at \/var\/run\/s6\/etc...exited 0.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [s6-init] ensuring user provided files have correct perms...exited 0.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [fix-attrs.d] applying ownership & permissions fixes...\nspoton_cz_db.1.m0z67958dx89@swarm2    | [fix-attrs.d] done.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] executing container initialization scripts...\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 01-envfile: executing... \nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 01-envfile: exited 0.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 10-adduser: executing... \nspoton_cz_db.1.m0z67958dx89@swarm2    | usermod: no changes\nspoton_cz_db.1.m0z67958dx89@swarm2    | \nspoton_cz_db.1.m0z67958dx89@swarm2    | -------------------------------------\nspoton_cz_db.1.m0z67958dx89@swarm2    |           _         ()\nspoton_cz_db.1.m0z67958dx89@swarm2    |          | |  ___   _    __\nspoton_cz_db.1.m0z67958dx89@swarm2    |          | | \/ __| | |  \/  \\ \nspoton_cz_db.1.m0z67958dx89@swarm2    |          | | \\__ \\ | | | () |\nspoton_cz_db.1.m0z67958dx89@swarm2    |          |_| |___\/ |_|  \\__\/\nspoton_cz_db.1.m0z67958dx89@swarm2    | \nspoton_cz_db.1.m0z67958dx89@swarm2    | \nspoton_cz_db.1.m0z67958dx89@swarm2    | Brought to you by linuxserver.io\nspoton_cz_db.1.m0z67958dx89@swarm2    | -------------------------------------\nspoton_cz_db.1.m0z67958dx89@swarm2    | \nspoton_cz_db.1.m0z67958dx89@swarm2    | To support LSIO projects visit:\nspoton_cz_db.1.m0z67958dx89@swarm2    | https:\/\/www.linuxserver.io\/donate\/\nspoton_cz_db.1.m0z67958dx89@swarm2    | -------------------------------------\nspoton_cz_db.1.m0z67958dx89@swarm2    | GID\/UID\nspoton_cz_db.1.m0z67958dx89@swarm2    | -------------------------------------\nspoton_cz_db.1.m0z67958dx89@swarm2    | \nspoton_cz_db.1.m0z67958dx89@swarm2    | User uid:    911\nspoton_cz_db.1.m0z67958dx89@swarm2    | User gid:    911\nspoton_cz_db.1.m0z67958dx89@swarm2    | -------------------------------------\nspoton_cz_db.1.m0z67958dx89@swarm2    | \nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 10-adduser: exited 0.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 30-config: executing... \nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 30-config: exited 0.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 40-initialise-db: executing... \nspoton_cz_db.1.m0z67958dx89@swarm2    | Setting Up Initial Databases\nspoton_cz_db.1.m0z67958dx89@swarm2    | Installing MariaDB\/MySQL system tables in '\/config\/databases' ...\n....\n....\n....\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 40-initialise-db: exited 0.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 99-custom-scripts: executing... \nspoton_cz_db.1.m0z67958dx89@swarm2    | [custom-init] no custom files found exiting...\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] 99-custom-scripts: exited 0.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [cont-init.d] done.\nspoton_cz_db.1.m0z67958dx89@swarm2    | [services.d] starting services\nspoton_cz_db.1.m0z67958dx89@swarm2    | [services.d] done.\nspoton_cz_db.1.m0z67958dx89@swarm2    | 210125 13:11:04 mysqld_safe Logging to syslog.\nspoton_cz_db.1.m0z67958dx89@swarm2    | 210125 13:11:04 mysqld_safe Starting mysqld daemon with databases from \/config\/databases\n<\/pre>\n\n\n\n
Then once again, connect to the instance and restore the tables with<\/p>\n\n\n\n
root@2c6effa01d73:\/config\/databases# mysql -u ventil -p spotoncz < spotoncz.wp_commentmeta.dmp<\/pre>\n\n\n\n
For each table, mind you. And. DONE. You have a fresh DB, no corruption and the data have been saved. All data for a wordpress site, including users, passwords, comments, pages and what not are not lost forever. <\/p>\n\n\n\n
Cheers.<\/p>\n","protected":false},"excerpt":{"rendered":"
MariaDB, or MySQL is stuck in the startup loop with a vague error: “Starting mysqld daemon with databases from \/config\/databases”. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[5,9],"tags":[111,109,106,110,114,61,113,112,107,108],"_links":{"self":[{"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/posts\/808"}],"collection":[{"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/comments?post=808"}],"version-history":[{"count":2,"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/posts\/808\/revisions"}],"predecessor-version":[{"id":810,"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/posts\/808\/revisions\/810"}],"wp:attachment":[{"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/media?parent=808"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/categories?post=808"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/spoton.cz\/index.php\/wp-json\/wp\/v2\/tags?post=808"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}