Enable pydoclint

[eapolinario]

Tracking issue

N/A

Why are the changes needed?

pydoclint is a well known linter that will help bring flytekit coding standards up a notch.

What changes were proposed in this pull request?

A few changes:

1. Enable pydoclint as a pre-commit hook.
2. Create a baseline file (here called pydoclint-errors-baseline.txt) to help catch issues going forward while having a path to resolving existing issues
   i. this baseline file is regenerated in case a rule is fixed in a PR due to --auto-regenerate-baseline defaulting to true.

Notice that the majority of the existing failures are a DOC301 according to the count of violations by type:

❯ cat pydoclint-errors-baseline.txt | grep "DOC" | cut -d':' -f1 | sort | uniq -c
   7     DOC001
  18     DOC101
  18     DOC103
  12     DOC105
   7     DOC106
  10     DOC107
   6     DOC109
   5     DOC110
  12     DOC201
  14     DOC203
 200     DOC301
   2     DOC402
   2     DOC404
   6     DOC501
   8     DOC503
  33     DOC601
  36     DOC603
   5     DOC605

Here's a list of violation codes in the docs: https://jsh9.github.io/pydoclint/violation_codes.html

Keep in mind that ruff is in the process of implementing the pydoclint rules, so sometime in the future we will be able to just switch to that.

How was this patch tested?

Run pre-commit run --all-files pydoclint locally (after generating the baseline file)

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

Summary by Bito

Integration of pydoclint as a pre-commit hook for Google-style docstring standards, alongside implementation of a new Kubernetes StatefulSet Data Service plugin. The changes include configuration setup, baseline tracking for docstring violations (focusing on DOC301, DOC601/603, DOC101/103), environment variable support for secrets, and improved coroutine handling in type transformers. Features optimized cloud storage operations with configurable chunk sizes.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 5

[flyte-bot]

Code Review Agent Run #9e84bd

Actionable Suggestions - 0

Additional Suggestions - 1

• .pre-commit-config.yaml - 1

  • Consider simplifying exclude pattern in pydoclint · Line 37-37

Review Details

• Files reviewed - 2 · Commit Range: e3f13cc..e3f13cc

  • .pre-commit-config.yaml
  • pydoclint-errors-baseline.txt
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.58%. Comparing base (88b651c) to head (eb4f9ee).
Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3077      +/-   ##
==========================================
- Coverage   92.59%   83.58%   -9.01%     
==========================================
  Files           4        3       -1     
  Lines          54      195     +141     
==========================================
+ Hits           50      163     +113     
- Misses          4       32      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[flyte-bot]

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Other Improvements - Code Quality Enhancement - Docstring Linting Setup	- .pre-commit-config.yaml - Added pydoclint configuration with Google style enforcement - pydoclint-errors-baseline.txt - Created baseline file to track existing docstring violations
Other Improvements - Code Quality Enhancement - Docstring Linting Setup	- pydoclint-errors-baseline.txt - Created baseline file documenting existing docstring violations across multiple plugins - pyproject.toml - Updated ignore-words-list to include 'assertIn' for docstring linting

[thomasjpfan]

@eapolinario There is a merge conflict. Otherwise, this looks good for us to get started.

[flyte-bot]

Code Review Agent Run #7693fe

Actionable Suggestions - 0

Additional Suggestions - 10

• plugins/flytekit-k8sdataservice/utils/resources.py - 1

  • Consider more robust zero value validation · Line 13-20
• flytekit/image_spec/default_builder.py - 1

  • Consider more descriptive environment variable name · Line 110-111
• plugins/flytekit-k8sdataservice/setup.py - 1

  • Consider more specific kubernetes version range · Line 7-7
• plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_agent.py - 2

  • Consider consolidating duplicate test cleanup code · Line 158-161
  • Consider consolidating duplicate test code · Line 164-303
• tests/flytekit/unit/core/test_data_persistence.py - 1

  • Consider verifying chunksize value in test · Line 237-238
• plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/task.py - 1

  • Consider using snake_case for attributes · Line 18-26
• plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/k8s/manager.py - 1

  • Consider simplifying StatefulSet status check condition · Line 174-178
• flytekit/image_spec/image_spec.py - 1

  • Consider adding python_exec path validation · Line 91-91
• flytekit/core/data_persistence.py - 1

  • Consider more descriptive env var name · Line 57-57

Review Details

• Files reviewed - 39 · Commit Range: e3f13cc..eb4f9ee

  • .pre-commit-config.yaml
  • Dockerfile.agent
  • docs/source/plugins/k8sstatefuldataservice.rst
  • flytekit/core/data_persistence.py
  • flytekit/core/type_engine.py
  • flytekit/image_spec/default_builder.py
  • flytekit/image_spec/image_spec.py
  • flytekit/models/security.py
  • plugins/flytekit-envd/flytekitplugins/envd/image_builder.py
  • plugins/flytekit-envd/tests/test_image_spec.py
  • plugins/flytekit-k8sdataservice/dev-requirements.txt
  • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/__init__.py
  • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/agent.py
  • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/k8s/kube_config.py
  • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/k8s/manager.py
  • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/sensor.py
  • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/task.py
  • plugins/flytekit-k8sdataservice/setup.py
  • plugins/flytekit-k8sdataservice/tests/k8sdataservice/k8s/test_kube_config.py
  • plugins/flytekit-k8sdataservice/tests/k8sdataservice/k8s/test_manager.py
  • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_agent.py
  • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_sensor.py
  • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_task.py
  • plugins/flytekit-k8sdataservice/tests/k8sdataservice/utils/test_resources.py
  • plugins/flytekit-k8sdataservice/utils/infra.py
  • plugins/flytekit-k8sdataservice/utils/resources.py
  • plugins/flytekit-omegaconf/flytekitplugins/omegaconf/dictconfig_transformer.py
  • plugins/flytekit-omegaconf/tests/test_dictconfig_transformer.py
  • plugins/setup.py
  • pydoclint-errors-baseline.txt
  • pyproject.toml
  • tests/flytekit/integration/remote/test_remote.py
  • tests/flytekit/integration/remote/workflows/basic/get_secret.py
  • tests/flytekit/unit/core/image_spec/test_default_builder.py
  • tests/flytekit/unit/core/test_data_persistence.py
  • tests/flytekit/unit/core/test_dataclass.py
  • tests/flytekit/unit/core/test_generice_idl_type_engine.py
  • tests/flytekit/unit/core/test_list.py
  • tests/flytekit/unit/core/test_type_engine.py
• Files skipped - 2

  • .github/workflows/pythonbuild.yml - Reason: Filter setting
  • plugins/flytekit-k8sdataservice/README.md - Reason: Filter setting
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by