How to Schedule Python Scripts With Cron - The Only Guide You'll Ever Need

Photo by Joel & Jasmin Førestbird on Unsplash

When it comes to repetitive tasks, you’re better of automating them. This article will teach you how.

After reading, you’ll know how to automate the execution of two Python scripts for fetching, parsing, and saving data from the web. Let’s get started!

The article is structured as follows:

What is Cron?
Writing the Scripts
Editing Crontab File
Testing
MacOS Gotchas
Conclusion

What is Cron?

Think of Cron as one of the easiest ways to schedule tasks in Linux and macOS environments. The word "Cron" comes from the Greek word "Chronos" (time), and the word "Crontab" stands for "Cron table" or time table. You’ll learn in a bit what this table refers to.

You should use Cron any time you want to automate something, like an OS job or a Python script. Needless to say, but an automated Python script can do basically anything.

On Linux and macOS, the Crontab consists of six fields. The first five are reserved for the date and time of scheduled execution (minute, day of month, month of year, day of week), and the last field is reserved for a command to be executed.

You’re free to use asterisks, values, and ranges for the date-time fields, but more on that in a bit.

You now know what Cron and Crontab are, but we’ll still need some Python scripts before working on scheduling. Let’s cover them next.

Writing the Scripts

Today we’ll schedule two simple scripts. Their job is to fetch the data from the following dummy websites:

These sites contain dummy data on various subjects, but we’ll focus only on the users and posts. The only reason we’re handling two sites is to show how to schedule cron jobs at different time intervals.

Both scripts will fetch the data via requests.get() API call, parse the JSON data, process it (keep only certain attributes), convert it to a Pandas DataFrame, and save it to a CSV file. Sounds like a lot, but it’s only a few lines of code.

Let’s start with the get_users.py file:

The get_posts.py file will be more or less identical – except for the processing part:

Finally, make sure to create the output just where your scripts are. That’s all we need. If you were to run the scripts, the CSVs would save in the output directory with the <name>_<timeastamp>.csv name structure. The timestamps are here to ensure files don’t get overridden.

Let’s see how to schedule the tasks next.

Editing the Crontab File

It doesn’t matter if you’re on Linux or macOS – this section will be identical. There are some permission gotchas for macOS, but we’ll get to that later.

As discussed before, you must follow a specific syntax to schedule cron jobs. The good news is, you can use Crontab.guru website to work on your scheduling.

We want get_users.py to run on every even minute (e.g., 0, 2, 4) and get_posts.py to run on every odd minute (e.g., 1, 3, 5). Here’s the correct pattern to run a job on every even minute:

Image 1 - Cron job pattern for every even minute execution (image by author) — Image 1 – Cron job pattern for every even minute execution (image by author)

And here’s for the odd minutes:

Image 2 - Cron job pattern for every odd minute execution (image by author) — Image 2 – Cron job pattern for every odd minute execution (image by author)

Neat – let’s use these patterns to schedule executions. Open up a Terminal window and execute pwd and which python3 commands to get the absolute paths to your script folder and Python:

Image 3 - Obtaining absolute paths (image by author) — Image 3 – Obtaining absolute paths (image by author)

Once you have these, enter the crontab -e command to edit a cron file, or to make one if it doesn’t exist:

Image 4 - Accessing crontab file (image by author) — Image 4 – Accessing crontab file (image by author)

It will open a VIM editor – from there, click on the I key on your keyboard to enter insertion mode. You’ll have to specify the scheduling pattern, full path to the Python executable, and full path to the script to make scheduling work. Use the following image as a reference:

Image 5 - Editing a crontab file (image by author) — Image 5 – Editing a crontab file (image by author)

Once done, press the ESC key to exit the insert mode, immediately followed by :wq and ENTER keys. This will save the crontab files, and you’ll be back in the Terminal window instantly:

Image 6 - Successful crontab file creation (image by author) — Image 6 – Successful crontab file creation (image by author)

To verify the file was successfully saved, you can use the crontab -l command – it will list all scheduled jobs:

Image 7 - Listing all scheduled jobs (image by author) — Image 7 – Listing all scheduled jobs (image by author)

And that’s all you have to do. Let’s check if scheduling works in the next section.

Testing

There’s nothing left for you to do but to sit and watch your scripts executed every minute. Here’s how my output folder looks like after a couple of minutes:

Image 8 - Output folder (image by author) — Image 8 – Output folder (image by author)

As you can see, everything works as expected. Don’t forget to delete these two jobs, or you’ll end up with 1440 new CSV files per day.

MacOS Gotchas

Linux users shouldn’t have any issues, but the story is different on macOS. By default, macOS doesn’t give full disc access to Terminal and Cron, so you’ll have to do that manually.

Just to be clear – you won’t get any errors, but the output folder will remain empty.

If that’s the case, please follow this article – it will show you how to grant full disc access to both Terminal and Cron.

Conclusion

And there you have it – how to easily schedule Python scripts with Cron on Linux and macOS.

The possibilities are endless – from scheduled web scraping to automated execution of ETL pipelines. The code can change, but the principle remains identical.

Have fun!

Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.