clk: at91: avoid sleeping early

clk: at91: avoid sleeping early

commit 658fd65cf0b0d511de1718e48d9a28844c385ae0 upstream.

It is not allowed to sleep to early in the boot process and this may lead
to kernel issues if the bootloader didn't prepare the slow clock and main
clock.

This results in the following error and dump stack on the AriettaG25:
   bad: scheduling from the idle thread!

Ensure it is possible to sleep, else simply have a delay.

Reported-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Link: https://lkml.kernel.org/r/20190920153906.20887-1-alexandre.belloni@bootlin.com
Fixes: 80eded6ce8bb ("clk: at91: add slow clks driver")
Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
[bwh: Backported to 3.16:
 - Drop changes in clk_sama5d4_slow_osc_prepare()
 - Adjust filename, context]
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>