Skip to content

Use Alembic for SQL Migrations #639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 16 commits into
base: master
Choose a base branch
from
Draft

Use Alembic for SQL Migrations #639

wants to merge 16 commits into from

Conversation

cmyui
Copy link
Member

@cmyui cmyui commented Feb 26, 2024

Describe your changes

Related Issues / Projects

Checklist

  • I've manually tested my code

cmyui
cmyui commented Feb 26, 2024
View reviewed changes
app/settings.py Outdated
@@ -18,7 +18,7 @@
DB_USER = os.environ["DB_USER"]
DB_PASS = os.environ["DB_PASS"]
DB_NAME = os.environ["DB_NAME"]
DB_DSN = f"mysql://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}"
DB_DSN = f"mysql+pymysql://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to undo this/move it to be alembic-specific; we want aiomysql at runtime

@cmyui
Copy link
Member Author

cmyui commented Feb 26, 2024

Diffs observed from generating the base schema

INFO  [alembic.autogenerate.compare] Detected removed table 'relationships'
INFO  [alembic.autogenerate.compare] Detected removed table 'startups'
INFO  [alembic.autogenerate.compare] Detected removed index 'nmapsets_id_uindex' on 'mapsets'
INFO  [alembic.autogenerate.compare] Detected removed table 'mapsets'
INFO  [alembic.autogenerate.compare] Detected removed table 'performance_reports'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=128) to String(length=128, collation='utf8') on 'achievements.name'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=256) to String(length=256, collation='utf8') on 'achievements.desc'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=16) to String(length=16, collation='utf8') on 'clans.name'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=6) to String(length=6, collation='utf8') on 'clans.tag'
INFO  [alembic.autogenerate.compare] Detected changed index 'clans_name_uindex' on 'clans': unique=True to unique=False
INFO  [alembic.autogenerate.compare] Column 'comments.target_id' has no type within the model; can't compare
INFO  [alembic.autogenerate.compare] Detected column comment 'comments.target_id'
INFO  [alembic.autogenerate.compare] Detected type change from ENUM('replay', 'map', 'song') to Enum('REPLAY', 'BEATMAP', 'SONG', name='target_type') on 'comments.target_type'
INFO  [alembic.autogenerate.compare] Detected type change from INTEGER() to FLOAT(precision=6, scale=3) on 'comments.time'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=80) to String(length=80, collation='utf8') on 'comments.comment'
INFO  [alembic.autogenerate.compare] Detected column comment 'comments.colour'
INFO  [alembic.autogenerate.compare] Detected column comment 'ingame_logins.ip'
INFO  [alembic.autogenerate.compare] Detected column comment 'logs.from'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=2048) to String(length=2048, collation='utf8') on 'logs.msg'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=2048) to String(length=2048, collation='utf8') on 'mail.msg'
INFO  [alembic.autogenerate.compare] Detected type change from ENUM('osu!', 'private') to Enum('OSU', 'PRIVATE', name='server') on 'maps.server'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=32) to String(length=32) on 'maps.md5'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=128) to String(length=128, collation='utf8') on 'maps.artist'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=128) to String(length=128, collation='utf8') on 'maps.title'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=128) to String(length=128, collation='utf8') on 'maps.version'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=19) to String(length=19, collation='utf8') on 'maps.creator'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=256) to String(length=256, collation='utf8') on 'maps.filename'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=32) to String(length=32) on 'ratings.map_md5'
INFO  [alembic.autogenerate.compare] Detected type change from BIGINT(unsigned=True) to Integer() on 'scores.id'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=32) to String(length=32) on 'scores.map_md5'
INFO  [alembic.autogenerate.compare] Detected type change from FLOAT(precision=7, scale=3) to FLOAT(precision=6, scale=3) on 'scores.pp'
INFO  [alembic.autogenerate.compare] Detected type change from TINYINT() to Integer() on 'scores.status'
INFO  [alembic.autogenerate.compare] Detected type change from TINYINT() to Integer() on 'scores.mode'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=32) to String(length=32) on 'scores.online_checksum'
INFO  [alembic.autogenerate.compare] Detected type change from BIGINT(unsigned=True) to Integer() on 'stats.tscore'
INFO  [alembic.autogenerate.compare] Detected type change from BIGINT(unsigned=True) to Integer() on 'stats.rscore'
INFO  [alembic.autogenerate.compare] Detected type change from BIGINT() to Integer() on 'tourney_pool_maps.mods'
INFO  [alembic.autogenerate.compare] Detected type change from TINYINT() to Integer() on 'tourney_pool_maps.slot'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(length=64) to String(length=16) on 'tourney_pools.name'
INFO  [alembic.autogenerate.compare] Detected type change from MEDIUMTEXT() to String(length=32, collation='utf8') on 'users.name'
INFO  [alembic.autogenerate.compare] Detected type change from MEDIUMTEXT() to String(length=32, collation='utf8') on 'users.safe_name'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=60) to String(length=60) on 'users.pw_bcrypt'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=2) to String(length=2) on 'users.country'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=16) to String(length=16, collation='utf8') on 'users.custom_badge_name'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=2048) to String(length=2048, collation='utf8') on 'users.userpage_content'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=36) to String(length=36) on 'users.api_key'
INFO  [alembic.autogenerate.compare] Detected removed index 'users_irc_key_uindex' on 'users'
INFO  [alembic.autogenerate.compare] Detected added index ''users_name_uindex'' on '('name',)'
INFO  [alembic.autogenerate.compare] Detected added index ''users_safe_name_uindex'' on '('safe_name',)'
INFO  [alembic.autogenerate.compare] Detected removed column 'users.irc_key'

(This is basically the todo list for this pr)

@cmyui cmyui changed the title Base alembic working Replace legacy custom migration system with alembic Feb 26, 2024
@cmyui cmyui changed the title Replace legacy custom migration system with alembic Replace legacy custom db migration system with alembic Feb 26, 2024
@cmyui cmyui force-pushed the alembic-migrations branch from 170f206 to bf91e78 Compare February 26, 2024 14:44
cmyui
cmyui commented Feb 26, 2024
View reviewed changes
cmyui
cmyui commented Feb 26, 2024
View reviewed changes
alembic.ini

[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can set this up to just defer to logging.yaml

cmyui
cmyui commented Feb 26, 2024
View reviewed changes
alembic.ini
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could use pre-commit hooks here? We could probably call into quite a few of them directly if we really had to. Maybe we'll find it's not worth using this & pre-commit after the fact is ok

cmyui
cmyui commented Feb 26, 2024
View reviewed changes
alembic/versions/158030827560_base_schema.py Outdated
Copy link
Member Author

@cmyui cmyui Feb 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: this is still missing the existing seed data, as well as a couple of the tables that already exist (we should really have a 1:1 representation of the existing db in this PR, so that it gives us the ability to use alembic to remove those existing parts of the schema in future PRs -- otherwise we'll have a desync between user's db states & where alembic thinks they are).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also now missing index from #642

@cmyui cmyui force-pushed the alembic-migrations branch from a044f62 to 1bca9ce Compare March 4, 2024 00:34
@NiceAesth
Copy link
Member

this would be epic to get in

@cmyui cmyui force-pushed the alembic-migrations branch from 1bca9ce to 54299f8 Compare May 21, 2024 15:00
@cmyui cmyui changed the title Replace legacy custom db migration system with alembic Use Alembic for SQL Migrations Apr 30, 2025
@cmyui
Copy link
Member Author

cmyui commented May 1, 2025

@7mochi I think this is the main thing still remaining to align: #639 (comment)

@cmyui
Copy link
Member Author

cmyui commented May 1, 2025

resolved the merge conflicts

@cmyui cmyui assigned cmyui and 7mochi May 1, 2025
@cmyui cmyui added the code quality Something is implemented poorly label May 1, 2025
cmyui and others added 3 commits May 2, 2025 00:11
@cmyui
add index from #642 to alembic state
e17ed78
lock
3a72c82
@7mochi @pre-commit-ci
chore: fix alembic migrations (#708)
c7203e1 
* chore: fix alembic migrations

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@cmyui cmyui force-pushed the alembic-migrations branch from f1bb7f8 to c7203e1 Compare May 2, 2025 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NiceAesth NiceAesth Awaiting requested review from NiceAesth NiceAesth will be requested when the pull request is marked ready for review NiceAesth is a code owner

@7mochi 7mochi Awaiting requested review from 7mochi 7mochi will be requested when the pull request is marked ready for review 7mochi is a code owner

@infernalfire72 infernalfire72 Awaiting requested review from infernalfire72 infernalfire72 will be requested when the pull request is marked ready for review infernalfire72 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@cmyui cmyui

@7mochi 7mochi

Labels
code quality Something is implemented poorly
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cmyui @NiceAesth @7mochi