[FIXED] Cron jobs run ok as script but not @reboot

Issue

I'm running a Django (v3.1.7) application on a headless remote Ubuntu (v20.04.3) that I'm looking to have start at reboot using crontab. The script below runs all ok from the command line. I've been through here crontabs-reboot-only-works-for-root and the linked suggestions as well but still haven't identified the issue. I've added logging (see *.log below) but do not get any helpful output.

EDIT: The test screen works both from the command line and at reboot. The django and streaming screens do not work at reboot. Checking python executable with which python3 returns /usr/bin/python3. So it looks to me that I have the correct permissions for reboot.sh and cron should be able to find python3 as it is in PATH= in the crontab file.

/home/myapp/reboot.sh

#!/bin/bash
echo "$(date) Starting reboot.sh"

start_django () {
   echo "$(date) Entering start_django"
   screen -S django -dm bash -c 'cd /home/myapp && /usr/local/bin/gunicorn myapp.wsgi:application --bind 0.0.0.0:8000' > /home/myapp/django_cron.log 2>&1 
   sleep 1
   echo "$(date) Exiting start_django"
}

start_stream () {
   echo "$(date) Entering start_stream"
   screen -S streaming -dm bash -c 'cd /home/myapp/data_api && python3 api_stream_client.py' > /home/myapp/stream_cron.log 2>&1
   sleep 1
   echo "$(date) Exiting start_stream"
}

start_test () {
  echo "$(date) Entering start_test"
  screen -S testa -dm bash -c 'cd /home/myapp && exec bash' > /home/myapp/test_cron.log 2>&1
  sleep 1
  echo "$(date) Exiting start_test"
}


if screen -list | grep -q "No Sockets found"; then
    echo "$(date) No screens available"
    echo "$(date) Starting test"
    start_test
    echo "$(date) Starting django"
    start_django
    echo "$(date) Starting stream"
    start_stream
else
    if screen -list | grep -q "django"; then
        echo "$(date) django exists"
    else
        echo "$(date) django does not exist - starting now"
        start_django
    fi

    if screen -list | grep -q "streaming"; then
        echo "$(date) stream exists"
    else
        echo "$(date) stream does not exist - starting now"
        start_stream
    fi

    if screen -list | grep -q "test"; then
        echo "$(date) test exists"
    else
        echo "$(date) test does not exist - starting now"
        start_test
    fi
fi

contab -l

# m h  dom mon dow   command
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
@reboot sleep 60 && /home/myapp/reboot.sh > /home/myapp/cron.log 2>&1

/var/log/cron.log

Jan 28 15:21:44 localhost systemd-timesyncd[637]: Network configuration changed, trying to establish connection.
Jan 28 15:21:44 localhost cron[814]: (CRON) INFO (pidfile fd = 3)
Jan 28 15:21:44 localhost cron[814]: (CRON) INFO (Running @reboot jobs)
Jan 28 15:21:44 localhost CRON[824]: (root) CMD (sleep 60 && /home/financialdashboard/reboot.sh > /home/financialdashboard/cron.log 2>&1)
Jan 28 15:21:46 localhost systemd-timesyncd[637]: Network configuration changed, trying to establish connection.
Jan 28 15:21:50 localhost systemd-timesyncd[637]: Network configuration changed, trying to establish connection.
Jan 28 15:21:55 localhost systemd-timesyncd[637]: Network configuration changed, trying to establish connection.
Jan 28 15:21:55 localhost systemd-timesyncd[637]: Initial synchronization to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).

/home/myapp/cron.log

Fri Jan 28 15:22:44 UTC 2022 Starting reboot.sh
Fri Jan 28 15:22:44 UTC 2022 No screens available
Fri Jan 28 15:22:44 UTC 2022 Starting test
Fri Jan 28 15:22:44 UTC 2022 Entering start_test
Fri Jan 28 15:22:45 UTC 2022 Exiting start_test
Fri Jan 28 15:22:45 UTC 2022 Starting django
Fri Jan 28 15:22:45 UTC 2022 Entering start_django
Fri Jan 28 15:22:46 UTC 2022 Exiting start_django
Fri Jan 28 15:22:46 UTC 2022 Starting stream
Fri Jan 28 15:22:46 UTC 2022 Entering start_stream
Fri Jan 28 15:22:47 UTC 2022 Exiting start_stream

django_cron.log, stream_cron.log and test_cron.log are all empty

Solution

Fixed: The django and streaming screens where relying on some environment variables which are not loaded with cron. See here where-can-i-set-environment-variables-that-crontab-will-use for more details.

I was able to get logging working for the screens by following this gnu-screen-logtstamp-string and this save-screen-program-output-to-a-file .

My updated /home/myapp/reboot.sh:

#!/bin/bash
SHELL=/bin/bash

echo "$(date) Starting reboot.sh"

# Load environment variables
source /etc/profile.d/env.sh

start_django () {
    echo "$(date) Entering start_django"
    screen -c /home/myapp/django_cron_log.conf -L -S django -dm bash -c 'cd /home/myapp && /usr/local/bin/gunicorn myapp.wsgi:application --bind 0.0.0.0:8000'
    echo "$(date) Exiting start_django"
}

start_stream () {
    echo "$(date) Entering start_stream"
    screen -c /home/myapp/stream_cron_log.conf -L -S streaming -dm bash -c 'cd /home/myapp/data_api && python3 api_stream_client.py && exec bash'
    echo "$(date) Exiting start_stream"
}

if screen -list | grep -q "No Sockets found"; then
    echo "$(date) No screens available"

    echo "$(date) Starting django"
    start_django

    echo "$(date) Starting stream"
    start_stream
else
    if screen -list | grep -q "django"; then
        echo "$(date) django exists; not starting django"
    else
        echo "$(date) django does not exist - starting now"
        start_django
    fi

    if screen -list | grep -q "streaming"; then
        echo "$(date) stream exists; not starting stream"
    else
        echo "$(date) stream does not exist - starting now"
        start_stream
    fi
fi

The -c /home/myapp/*_cron_log.conf -L in screen will be removed for production.

Answered By - qwertytam