Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updating job export to work with Helm #89

Merged
merged 3 commits into from
Dec 5, 2024
Merged

updating job export to work with Helm #89

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cb28298
updating job export to work with Helm
EliseCastle23 Dec 4, 2024
9427b7e
Merge branch 'master' into feat/helm-fixes
MichaelLukowski Dec 5, 2024
6bfb73c
change from print statement to runtime error
EliseCastle23 Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 31 additions & 11 deletions job_export.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,25 @@
filters = json.dumps({"filter": input_data.get("filter", {})})
case_ids = gql.execute(filters=filters)

with open("/peregrine-creds.json") as pelican_creds_file:
peregrine_creds = json.load(pelican_creds_file)

DB_URL = "jdbc:postgresql://{}/{}".format(
peregrine_creds["db_host"], peregrine_creds["db_database"]
)
DB_USER = peregrine_creds["db_username"]
DB_PASS = peregrine_creds["db_password"]
try:
with open("/peregrine-creds.json") as pelican_creds_file:
peregrine_creds = json.load(pelican_creds_file)
except (FileNotFoundError, json.JSONDecodeError) as e:
print(f"Failed to load credentials file: {e}")
peregrine_creds = {}

# Set variables, prioritizing environment variables
DB_HOST = os.getenv("DB_HOST", peregrine_creds.get("db_host"))
DB_DATABASE = os.getenv("DB_DATABASE", peregrine_creds.get("db_database"))
DB_USER = os.getenv("DB_USER", peregrine_creds.get("db_username"))
DB_PASS = os.getenv("DB_PASS", peregrine_creds.get("db_password"))

# Construct the database URL if possible
if DB_HOST and DB_DATABASE:
DB_URL = f"jdbc:postgresql://{DB_HOST}/{DB_DATABASE}"
else:
DB_URL = None
print("DB_HOST or DB_DATABASE is missing. DB_URL cannot be constructed.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we raise an error in this scenario? 🤔 So the application doesn't continue thinking everything is ok, but it has no database. Not sure what we usually do for that in other services.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah if no db exists then the job should throw runtime error


dictionary_url = os.environ["DICTIONARY_URL"]
dictionary, model = init_dictionary(url=dictionary_url)
Expand Down Expand Up @@ -174,8 +185,17 @@
COMMONS = "https://" + hostname + "/"

# try sending to indexd
with open("/indexd-creds.json") as indexd_creds_file:
indexd_creds = json.load(indexd_creds_file)
try:
with open("/indexd-creds.json") as indexd_creds_file:
indexd_creds = json.load(indexd_creds_file)
gdcapi_credential = indexd_creds["user_db"]["gdcapi"]
except (FileNotFoundError, json.JSONDecodeError, KeyError) as e:
print(f"Failed to load indexd credentials file or missing keys: {e}")
indexd_creds = {}
gdcapi_credential = None

# Load the indexd credential (fallback to SHEEPDOG env variable)
indexd_creds = os.getenv("SHEEPDOG", gdcapi_credential)

s3_url = "s3://" + pelican_creds["manifest_bucket_name"] + "/" + avro_filename

Expand All @@ -195,7 +215,7 @@

indexd_record = indexd_submit(
COMMONS,
indexd_creds["user_db"]["gdcapi"],
indexd_creds,
avro_filename,
os.stat(fname).st_size,
[s3_url],
Expand Down
Loading