dmctl 8.5.1 can not config export from 7.5.1 cluster

[bolt-leandro]

What did you do?

We upgraded tiup dmctl from v.7.5.4 to v.8.5.1.

• dmctl:v7.5.4 can config export from DM cluster running on 7.5.1 without problems
• dmctl:v8.5.1 gives an error on config export from DM cluster running on 7.5.1

shell> tiup dm display dm-cluster
Starting component dm: /root/.tiup/components/dm/v1.16.1/tiup-dm display dm-cluster
Cluster type:       dm
Cluster name:       dm-cluster
Cluster version:    v7.5.1

shell> tiup dmctl:v7.5.4 --master-addr dm-cluster-dm_master-1.domain:8261 config export
Starting component dmctl: /root/.tiup/components/dmctl/v7.5.4/dmctl/dmctl --master-addr dm-cluster-dm_master-1.domain:8261 config export
export configs to directory `configs` succeed

shell> tiup dmctl:v8.5.1 --master-addr dm-cluster-dm_master-1.domain:8261 config export
Starting component dmctl: /root/.tiup/components/dmctl/v8.5.1/dmctl/dmctl --master-addr dm-cluster-dm_master-1.domain:8261 config export
Error: rpc error: code = Unimplemented desc = unknown method ListSourceConfigs for service pb.Master

What did you expect to see?

A successful backup of configs created:
export configs to directory `configs` succeed

• dmctl 8.5.1 should be backwards compatible with previous 7.5.x major version

What did you see instead?

An error and no task and source backup files were created.
Error: rpc error: code = Unimplemented desc = unknown method ListSourceConfigs for service pb.Master

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

Starting component dmctl: /root/.tiup/components/dmctl/v8.5.1/dmctl/dmctl -V
Release Version: v8.5.1
Git Commit Hash: 0a10d1fae9245c480e31df30f8b13ca4810c3b04
Git Branch: HEAD
UTC Build Time: 2025-01-10 06:50:25
Go Version: go1.23.4
Failpoint Build: false

How did you deploy DM: tiup or manually?

TiUP

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)

[dveeden]

I think this should either work, or there should be a version check when connecting to DM master. And maybe a version matrix in the docs?

[lance6716]

/cc @OliverS929

[OliverS929]

I will take a look.

[OliverS929]

/assign OliverS929

[OliverS929]

This backward incompatibility was introduced in this pull request, which decoupled encryption from dmctl and introduced new RPC calls: Encrypt, ListTaskConfigs, and ListSourceConfigs. As a result, you are encountering the error:

Error: rpc error: code = Unimplemented desc = unknown method ListSourceConfigs for service pb.Master.

However, to my knowledge, dmctl does not explicitly support backward compatibility. dmctl is tightly coupled with DMMaster and DMWorker in each version. As new features are introduced and refactoring takes place, changes to the RPC calls and protobuf structures used between dmctl and DM instances are often required. Therefore, it seems reasonable to assume that dmctl and DM should be aligned to the same version.

Additionally, the functionality provided by dmctl depends on support from DM instances. As such, I don’t foresee many use cases that would justify upgrading dmctl independently of the DM version. I would appreciate it if you could clarify the specific scenarios you have in mind.

However, I agree with @dveeden that we should consider implementing a version check to prevent this type of error, as it could be confusing for users.

/cc @alastori Please take a look and assess whether a version check is necessary for dmctl.

[dveeden]

I think for people managing multiple DM clusters it would be reasonable to expect these to be the current LTS version and one LTS version below that.

So if you have say 10 DM clusters you upgrade one at a time.
Then either you upgrade dmctl before the upgrade (which requires it to be compatible with n and n-1) or after the upgrade which requires DM master to be compatible with n and n-1.
The benefit of this is that one version of dmctl can be used with all clusters.

And if that doesn't work a 1-to-1 mapping between dmctl and dm master would be ok.