[indexer]: Create script to autogenerate indexer schema

[tomxey]

Description of change

Create script to autogenerate indexer schema.

Based on upstream patch MystenLabs/sui#19337

Links to any relevant issues

fixes #5008

Type of change

• Enhancement (a non-breaking change which adds functionality)

How the change has been tested

./scripts/generate_indexer_schema.sh
cargo ci-clippy
cargo +nightly ci-fmt
cargo test --profile simulator --package=iota-indexer --features shared_test_runtime

Change checklist

Tick the boxes that are relevant to your changes, and delete any items that are not.

• I have followed the contribution guidelines for this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have checked that new and existing unit tests pass locally with my changes

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

4 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
apps-backend	⬜️ Ignored (Inspect)	Visit Preview		Jan 28, 2025 4:40pm
apps-ui-kit	⬜️ Ignored (Inspect)	Visit Preview	💬 Add feedback	Jan 28, 2025 4:40pm
rebased-explorer	⬜️ Ignored (Inspect)	Visit Preview		Jan 28, 2025 4:40pm
wallet-dashboard	⬜️ Ignored (Inspect)	Visit Preview	💬 Add feedback	Jan 28, 2025 4:40pm

[kodemartin]

lgtm ⛱️

[muXxer]

I would like to propose a change here. It would be nice to put the requirements like postgres and diesel into a docker container, so the developer don't need to install those dependencies on their machine just to execute that script, and we have a defined environment for generation of the schema.

I would move the script to a subdirectory indexer-generate-schema.
Content of the Dockerfile:

# Use the official PostgreSQL image as a base
FROM postgres:15

# Install dependencies
RUN apt-get update && apt-get install -y \
    curl \
    build-essential \
    libpq-dev && \
    rm -rf /var/lib/apt/lists/*

ARG RUST_TOOLCHAIN_VERSION

# Install Rust
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain ${RUST_TOOLCHAIN_VERSION}

# Install Diesel CLI
RUN /root/.cargo/bin/cargo install diesel_cli --no-default-features --features postgres

# Add the Rust and Diesel CLI binaries to PATH
ENV PATH="/root/.cargo/bin:${PATH}"

Little build script build.sh

#!/bin/bash
set -x
set -e

if ! command -v git &> /dev/null; then
    echo "git not installed" >&2
    exit 1
fi

REPO_ROOT=$(git rev-parse --show-toplevel)

echo "Parse the rust toolchain version from 'rust-toolchain.toml'..."
RUST_TOOLCHAIN_VERSION=$(grep -oE 'channel = "[^"]+' ${REPO_ROOT}/rust-toolchain.toml | sed 's/channel = "//')
if [ -z "$RUST_TOOLCHAIN_VERSION" ]; then
    echo "Failed to parse the rust toolchain version"
    exit 1
fi

docker build  --build-arg RUST_TOOLCHAIN_VERSION=${RUST_TOOLCHAIN_VERSION} -t postgres-rust-diesel .

generate.sh

#!/bin/bash
# Copyright (c) Mysten Labs, Inc.
# Modifications Copyright (c) 2024 IOTA Stiftung
# SPDX-License-Identifier: Apache-2.0
#
# Update iota-indexer's generated src/schema.rs based on the schema after
# running all its migrations on a clean database.
set -x
set -e

if ! command -v git &> /dev/null; then
    echo "git not installed" >&2
    exit 1
fi

REPO=$(git rev-parse --show-toplevel)

# Set up the Docker container with PostgreSQL and Diesel CLI
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgrespw
CONTAINER_NAME=postgres-rust-diesel
docker run --rm -d \
    --name ${CONTAINER_NAME} \
    -e POSTGRES_USER=${POSTGRES_USER} \
    -e POSTGRES_PASSWORD=${POSTGRES_PASSWORD} \
    -e RUST_TOOLCHAIN_VERSION=${RUST_TOOLCHAIN_VERSION} \
    -v "${REPO}:/workspace" \
    -w /workspace \
    ${CONTAINER_NAME}

# Wait for Postgres to be ready
RETRIES=0
while ! docker exec ${CONTAINER_NAME} pg_isready -p 5432 --username ${POSTGRES_USER}; do
  if [ $RETRIES -gt 30 ]; then
    echo "Postgres failed to start" >&2
    docker stop ${CONTAINER_NAME}
    exit 1
  fi
  sleep 1
  RETRIES=$((RETRIES + 1))
done

# Run migrations and generate the schema.rs file
docker exec ${CONTAINER_NAME} diesel migration run \
  --database-url "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@localhost:5432" \
  --migration-dir "/workspace/crates/iota-indexer/migrations/pg"

docker exec ${CONTAINER_NAME} diesel print-schema \
  --database-url "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@localhost:5432" \
  --patch-file "/workspace/crates/iota-indexer/src/schema.patch" \
  --except-tables "^objects_version_|_partition_" \
  > "${REPO}/crates/iota-indexer/src/schema.rs"

# Cleanup: Stop and remove the container
docker stop ${CONTAINER_NAME}

# Applying the patch may destroy the formatting, fix it
rustfmt +nightly "${REPO}/crates/iota-indexer/src/schema.rs"

[tomxey]

I would like to propose a change here. It would be nice to put the requirements like postgres and diesel into a docker container, so the developer don't need to install those dependencies on their machine just to execute that script, and we have a defined environment for generation of the schema.
...

@muXxer I like the idea. Pushed changes with dockerized script version.

[kodemartin]

Good one! I added a few minor comments that would add some cherry on top of the improvement proposed by @muXxer