© The Chancellor, Masters and Scholars of The University of Oxford. All rights reserved.

Scheduling a Python program to run daily in AWS

FEATURES

• The program will fetch a list of HTTP URLs
• The data is stored in a S3 bucket
• The program creates the bucket if it doesn't exist
• A new output directory is created for each trigger date

Instructions

Step 1. Go to the AWS Console and type "Lambda" in the search bar

---

Step 2. Press "Create a function"

---

Step 3. Configure the function

Choose "Author from scratch", Python, and a name for your function (for example: daily_python_task)

---

Step 4. Paste the code in the editor

---

Step 5. Add dependencies

When we import a Python module, for example 'import requests', we need to provide a layer that includes this module.

Press "Add layer"

Choose "AWSSDKPandas-Python313"

If you change the code and add custom modules, you might need to create a .zip file and a custom layer

---

Step 6. Adjust timeout

Change the default timeout (3 seconds) to a reasonable value (1 minute)

---

Step 7. Add permissions

Access the role associated to the lambda function

Press "Add permissions"

Add S3 full access

---

Testing

Step 1. Press "Test"

Create an event if needed, wait for the output

---

Step 2. Go to the AWS Console and type "S3" in the search bar

---

Step 3. Check the content of the bucket

---

Scheduling

Step 1. Press "Add trigger"

---

Step 2. Set the source of the trigger

It should be EventBridge

---

Step 3. Create the trigger rule

Give it a name (for example: daily_python_trigger), setup the Schedule Expression, for example cron(0 0 * * ? *) (runs daily at midnight UTC)

---