Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(entrypoint): Delay on auto-install to avoid error before retry #2222

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

joshtrichards
Copy link
Member

The start-up of new MariaDB/MySQL container + creation of the initial user specified via the MYSQL_* variables can take about ~4-6s even in a relatively capable/fast environment:

MariaDB container startup log with timestamps
test-nc29-dbpass-db-1  | 2024-05-22 21:54:33+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.17+maria~ubu2004 started.
[...]
test-nc29-dbpass-db-1  | 2024-05-22 21:54:39 0 [Note] mariadbd: ready for connections.

When attempting a fully automated Nextcloud installation (i.e. populating a NEXTCLOUD_* variables in Compose), this can trigger a failure of the install attempt the first time around. This generates a scary looking error before we retry 10 seconds later:

Scary stack trace if database container happens to not be fully initialized yet
test-nc29-dbpass-app-1  | Starting nextcloud installation
test-nc29-dbpass-app-1  | Error while trying to create admin account: An exception occurred in the driver: SQLSTATE[HY000] [1045] Access denied for user 'ncuser'@'localhost' (using password: YES)
test-nc29-dbpass-app-1  | Trace: #0 /var/www/html/3rdparty/doctrine/dbal/src/Connection.php(1943): Doctrine\DBAL\Driver\API\MySQL\ExceptionConverter->convert(Object(Doctrine\DBAL\Driver\PDO\Exception), NULL)
test-nc29-dbpass-app-1  | #1 /var/www/html/3rdparty/doctrine/dbal/src/Connection.php(1891): Doctrine\DBAL\Connection->handleDriverException(Object(Doctrine\DBAL\Driver\PDO\Exception), NULL)
test-nc29-dbpass-app-1  | #2 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(257): Doctrine\DBAL\Connection->convertException(Object(Doctrine\DBAL\Driver\PDO\Exception))
test-nc29-dbpass-app-1  | #3 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(192): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->connectTo('primary')
test-nc29-dbpass-app-1  | #4 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(224): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->performConnect('primary')
test-nc29-dbpass-app-1  | #5 /var/www/html/lib/private/Setup/AbstractDatabase.php(147): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->ensureConnectedToPrimary()
test-nc29-dbpass-app-1  | #6 /var/www/html/lib/private/Setup/MySQL.php(43): OC\Setup\AbstractDatabase->connect(Array)
test-nc29-dbpass-app-1  | #7 /var/www/html/lib/private/Setup.php(339): OC\Setup\MySQL->setupDatabase('ncadmin')
test-nc29-dbpass-app-1  | #8 /var/www/html/core/Command/Maintenance/Install.php(104): OC\Setup->install(Array, NULL)
test-nc29-dbpass-app-1  | #9 /var/www/html/3rdparty/symfony/console/Command/Command.php(298): OC\Core\Command\Maintenance\Install->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #10 /var/www/html/3rdparty/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #11 /var/www/html/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OC\Core\Command\Maintenance\Install), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #12 /var/www/html/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #13 /var/www/html/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #14 /var/www/html/console.php(113): OC\Console\Application->run()
test-nc29-dbpass-app-1  | #15 /var/www/html/occ(11): require_once('/var/www/html/c...')
test-nc29-dbpass-app-1  | #16 {main}
test-nc29-dbpass-app-1  | 
test-nc29-dbpass-app-1  | Previous: Doctrine\DBAL\Driver\PDO\Exception: SQLSTATE[HY000] [1045] Access denied for user 'ncuser'@'localhost' (using password: YES)
test-nc29-dbpass-app-1  | Trace: #0 /var/www/html/3rdparty/doctrine/dbal/src/Driver/PDO/MySQL/Driver.php(40): Doctrine\DBAL\Driver\PDO\Exception::new(Object(PDOException))
test-nc29-dbpass-app-1  | #1 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(255): Doctrine\DBAL\Driver\PDO\MySQL\Driver->connect(Object(SensitiveParameterValue))
test-nc29-dbpass-app-1  | #2 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(192): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->connectTo('primary')
test-nc29-dbpass-app-1  | #3 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(224): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->performConnect('primary')
test-nc29-dbpass-app-1  | #4 /var/www/html/lib/private/Setup/AbstractDatabase.php(147): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->ensureConnectedToPrimary()
test-nc29-dbpass-app-1  | #5 /var/www/html/lib/private/Setup/MySQL.php(43): OC\Setup\AbstractDatabase->connect(Array)
test-nc29-dbpass-app-1  | #6 /var/www/html/lib/private/Setup.php(339): OC\Setup\MySQL->setupDatabase('ncadmin')
test-nc29-dbpass-app-1  | #7 /var/www/html/core/Command/Maintenance/Install.php(104): OC\Setup->install(Array, NULL)
test-nc29-dbpass-app-1  | #8 /var/www/html/3rdparty/symfony/console/Command/Command.php(298): OC\Core\Command\Maintenance\Install->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #9 /var/www/html/3rdparty/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #10 /var/www/html/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OC\Core\Command\Maintenance\Install), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #11 /var/www/html/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #12 /var/www/html/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #13 /var/www/html/console.php(113): OC\Console\Application->run()
test-nc29-dbpass-app-1  | #14 /var/www/html/occ(11): require_once('/var/www/html/c...')
test-nc29-dbpass-app-1  | #15 {main}
test-nc29-dbpass-app-1  | 
test-nc29-dbpass-app-1  | Previous: PDOException: SQLSTATE[HY000] [1045] Access denied for user 'ncuser'@'localhost' (using password: YES)
test-nc29-dbpass-app-1  | Trace: #0 /var/www/html/3rdparty/doctrine/dbal/src/Driver/PDO/MySQL/Driver.php(33): PDO->__construct('mysql:host=loca...', 'ncuser', Object(SensitiveParameterValue), Array)
test-nc29-dbpass-app-1  | #1 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(255): Doctrine\DBAL\Driver\PDO\MySQL\Driver->connect(Object(SensitiveParameterValue))
test-nc29-dbpass-app-1  | #2 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(192): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->connectTo('primary')
test-nc29-dbpass-app-1  | #3 /var/www/html/3rdparty/doctrine/dbal/src/Connections/PrimaryReadReplicaConnection.php(224): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->performConnect('primary')
test-nc29-dbpass-app-1  | #4 /var/www/html/lib/private/Setup/AbstractDatabase.php(147): Doctrine\DBAL\Connections\PrimaryReadReplicaConnection->ensureConnectedToPrimary()
test-nc29-dbpass-app-1  | #5 /var/www/html/lib/private/Setup/MySQL.php(43): OC\Setup\AbstractDatabase->connect(Array)
test-nc29-dbpass-app-1  | #6 /var/www/html/lib/private/Setup.php(339): OC\Setup\MySQL->setupDatabase('ncadmin')
test-nc29-dbpass-app-1  | #7 /var/www/html/core/Command/Maintenance/Install.php(104): OC\Setup->install(Array, NULL)
test-nc29-dbpass-app-1  | #8 /var/www/html/3rdparty/symfony/console/Command/Command.php(298): OC\Core\Command\Maintenance\Install->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #9 /var/www/html/3rdparty/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #10 /var/www/html/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OC\Core\Command\Maintenance\Install), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #11 /var/www/html/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #12 /var/www/html/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
test-nc29-dbpass-app-1  | #13 /var/www/html/console.php(113): OC\Console\Application->run()
test-nc29-dbpass-app-1  | #14 /var/www/html/occ(11): require_once('/var/www/html/c...')
test-nc29-dbpass-app-1  | #15 {main}
test-nc29-dbpass-app-1  | Retrying install...
test-nc29-dbpass-app-1  | Nextcloud was successfully installed

Yes, we retry it 10s later and it all works out, but we can avoid this entirely by doing the delay before we even hit the retry cycle.

This PR:

  • introduces an initial 10s only during new installations if MySQL/MariaDB/PostgreSQL are specified which should eliminate the failure in a "typical" or even somewhat slower environment
  • Does not introduce a new delay (well, okay 1s) for sqlite, where this is presumed to be a non-issue
  • adds a note to the initial delay message that we'll retry if needed so the operator is less likely to stress if it fails the first time around

@joshtrichards joshtrichards added enhancement 3. to review feature: auto config (environment variables) Auto configuring via environment variables feature: installing First run / fresh installation labels May 22, 2024
max_retries=10
try=0
echo "Waiting ($db_wait) for initialization of database to complete (NOTE: if this fails the first time we'll retry again in 10s)..."
sleep $db_wait
until run_as "php /var/www/html/occ maintenance:install $install_options" || [ "$try" -gt "$max_retries" ]
Copy link
Member

@pierreozoux pierreozoux May 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
until run_as "php /var/www/html/occ maintenance:install $install_options" || [ "$try" -gt "$max_retries" ]
until run_as "php /var/www/html/occ maintenance:install $install_options" 2> /tmp/install_error || [ "$try" -gt "$max_retries" ]

try=$((try+1))
sleep 10s
done
if [ "$try" -gt "$max_retries" ]; then
echo "Installing of nextcloud failed!"
echo "Installation of Nextcloud failed!"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
echo "Installation of Nextcloud failed!"
echo "Installation of Nextcloud failed!"
cat /tmp/install_error

@pierreozoux
Copy link
Member

I get the idea ( I contribute rarely here, so maybe don't take into account my comment), but usually adding a sleep somewhere is not the best idea.

So I think, what you try to avoid is the scary error.

Then, why not hiding it? I did a suggestion, that might work to hide it ( I didn't tested ).

Hope it helps :)

@J0WI
Copy link
Contributor

J0WI commented Jun 4, 2024

I'm not fully convinced to just add a random sleep before every step. This might be useful for some cases, but slows it down for those cases where you connect it to a database service that is already running.

You can speed-up MariaDB initialisation with MARIADB_INITDB_SKIP_TZINFO.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3. to review enhancement feature: auto config (environment variables) Auto configuring via environment variables feature: installing First run / fresh installation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants