From f550fc06b3ae3e55cf8fce415a04874a177a19d0 Mon Sep 17 00:00:00 2001 From: Hongbin Lu Date: Sat, 27 Aug 2022 09:12:58 +0000 Subject: [PATCH] Document a known issue about the v2 to v3 migration * Update the 3.4 doc about the known issue of v2 to v3 migration * Remove v2 migration doc from 3.5 and 3.6 Fixes #8804 Signed-off-by: Hongbin Lu --- content/en/docs/v3.4/op-guide/v2-migration.md | 6 ++ content/en/docs/v3.5/op-guide/v2-migration.md | 61 ------------------- .../en/docs/v3.5/upgrades/upgrading-etcd.md | 4 -- content/en/docs/v3.6/op-guide/v2-migration.md | 61 ------------------- .../en/docs/v3.6/upgrades/upgrading-etcd.md | 4 -- 5 files changed, 6 insertions(+), 130 deletions(-) delete mode 100644 content/en/docs/v3.5/op-guide/v2-migration.md delete mode 100644 content/en/docs/v3.6/op-guide/v2-migration.md diff --git a/content/en/docs/v3.4/op-guide/v2-migration.md b/content/en/docs/v3.4/op-guide/v2-migration.md index fa32f5dc91cdb..c21d2c35ffae6 100644 --- a/content/en/docs/v3.4/op-guide/v2-migration.md +++ b/content/en/docs/v3.4/op-guide/v2-migration.md @@ -40,6 +40,12 @@ First, all members in the etcd cluster must converge to the same state. This can Second, migrate the v2 keys into v3 with the [migrate][migrate_command] (`ETCDCTL_API=3 etcdctl migrate`) command. The migrate command writes keys in the v2 store to a user-provided transformer program and reads back transformed keys. It then writes transformed keys into the mvcc store. This usually takes at most tens of seconds. +{{% alert title="known issue" color="warning" %}} + The migrate command has a known issue that new members added after the migration do not receive existing data if no .snap file exists. + The workaround is to perform a snapshot restore after the migration. + For more information see [GitHub issue](https://github.com/etcd-io/etcd/issues/8804). +{{% /alert %}} + Restart the etcd members and everything should just work. For etcd v3.3+, run `ETCDCTL_API=3 etcdctl endpoint hashkv --cluster` to ensure key-value stores are consistent post migration. diff --git a/content/en/docs/v3.5/op-guide/v2-migration.md b/content/en/docs/v3.5/op-guide/v2-migration.md deleted file mode 100644 index fa32f5dc91cdb..0000000000000 --- a/content/en/docs/v3.5/op-guide/v2-migration.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -title: Migrate applications from using API v2 to API v3 -weight: 4850 -description: A guide for migrating from API v2 to API v3 ---- - -The data store v2 is still accessible from the API v2 after upgrading to etcd3. Thus, it will work as before and require no application changes. With etcd 3, applications use the new grpc API v3 to access the mvcc store, which provides more features and improved performance. The mvcc store and the old store v2 are separate and isolated; writes to the store v2 will not affect the mvcc store and, similarly, writes to the mvcc store will not affect the store v2. - -Migrating an application from the API v2 to the API v3 involves two steps: 1) migrate the client library and, 2) migrate the data. If the application can rebuild the data, then migrating the data is unnecessary. - -## Migrate client library - -API v3 is different from API v2, thus application developers need to use a new client library to send requests to etcd API v3. The documentation of the client v3 is available at https://pkg.go.dev/github.com/etcd-io/etcd/clientv3. - -There are some notable differences between API v2 and API v3: - -- Transaction: In v3, etcd provides multi-key conditional transactions. Applications should use transactions in place of `Compare-And-Swap` operations. - -- Flat key space: There are no directories in API v3, only keys. For example, "/a/b/c/" is a key. Range queries support getting all keys matching a given prefix. - -- Compacted responses: Operations like `Delete` no longer return previous values. To get the deleted value, a transaction can be used to atomically get the key and then delete its value. - -- Leases: A replacement for v2 TTLs; the TTL is bound to a lease and keys attach to the lease. When the TTL expires, the lease is revoked and all attached keys are removed. - -## Migrate data - -Application data can be migrated either offline or online. Offline migration is much simpler than online migration and is recommended. - -Sometimes an etcd cluster will possibly have v3 data which should not be overwritten. In this case, the migration process may want to confirm no v3 data is committed before proceeding. One way to check the cluster has no v3 keys is to issue the following `etcdctl` command, which scans the entire v3 keyspace for any key, expecting `0` as output: - -```sh -ETCDCTL_API=3 etcdctl get "" --from-key --keys-only --limit 1 | wc -l -``` - -### Offline migration - -Offline migration is very simple but requires etcd downtime. If an etcd downtime window spanning from seconds to minutes is acceptable, offline migration is a good choice and is easy to automate. - -First, all members in the etcd cluster must converge to the same state. This can be achieved by stopping all applications that write keys to etcd. Alternatively, if the applications must remain running, configure etcd to listen on a different client URL and restart all etcd members. To check if the states converged, within a few seconds, use the `ETCDCTL_API=3 etcdctl endpoint status` command to confirm that the `raft index` of all members match (or differ by at most 1 due to an internal sync raft command). - -Second, migrate the v2 keys into v3 with the [migrate][migrate_command] (`ETCDCTL_API=3 etcdctl migrate`) command. The migrate command writes keys in the v2 store to a user-provided transformer program and reads back transformed keys. It then writes transformed keys into the mvcc store. This usually takes at most tens of seconds. - -Restart the etcd members and everything should just work. - -For etcd v3.3+, run `ETCDCTL_API=3 etcdctl endpoint hashkv --cluster` to ensure key-value stores are consistent post migration. - -**Warn**: When v2 store has expiring TTL keys and migrate command intends to preserve TTLs, migration may be inconsistent with the last committed v2 state when run on any member with a raft index less than the last leader's raft index. - -### Online migration - -If the application cannot tolerate any downtime, then it must migrate online. The implementation of online migration will vary from application to application but the overall idea is the same. - -First, write application code using the v3 API. The application must support two modes: a migration mode and a normal mode. The application starts in migration mode. When running in migration mode, the application reads keys using the v3 API first, and, if it cannot find the key, it retries with the API v2. In normal mode, the application only reads keys using the v3 API. The application writes keys over the API v3 in both modes. To acknowledge a switch from migration mode to normal mode, the application watches on a switch mode key. When switch key’s value turns to `true`, the application switches over from migration mode to normal mode. - -Second, start a background job to migrate data from the store v2 to the mvcc store by reading keys from the API v2 and writing keys to the API v3. - -After finishing data migration, the background job writes `true` into the switch mode key to notify the application that it may switch modes. - -Online migration can be difficult when the application logic depends on store v2 indexes. Applications will need additional logic to convert mvcc store revisions to store v2 indexes. - -[migrate_command]: https://github.com/etcd-io/etcd/tree/master/etcdctl#migrate-options diff --git a/content/en/docs/v3.5/upgrades/upgrading-etcd.md b/content/en/docs/v3.5/upgrades/upgrading-etcd.md index b9df045b0b91b..d102843e020e4 100644 --- a/content/en/docs/v3.5/upgrades/upgrading-etcd.md +++ b/content/en/docs/v3.5/upgrades/upgrading-etcd.md @@ -6,10 +6,6 @@ description: Documentation list for upgrading etcd clusters and applications This section contains documents specific to upgrading etcd clusters and applications. -## Moving from etcd API v2 to API v3 - -* [Migrate applications from using API v2 to API v3][migrate-apps] - ## Upgrading an etcd v3.x cluster * [Upgrade etcd from 3.0 to 3.1](../upgrade_3_1/) * [Upgrade etcd from 3.1 to 3.2](../upgrade_3_2/) diff --git a/content/en/docs/v3.6/op-guide/v2-migration.md b/content/en/docs/v3.6/op-guide/v2-migration.md deleted file mode 100644 index fa32f5dc91cdb..0000000000000 --- a/content/en/docs/v3.6/op-guide/v2-migration.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -title: Migrate applications from using API v2 to API v3 -weight: 4850 -description: A guide for migrating from API v2 to API v3 ---- - -The data store v2 is still accessible from the API v2 after upgrading to etcd3. Thus, it will work as before and require no application changes. With etcd 3, applications use the new grpc API v3 to access the mvcc store, which provides more features and improved performance. The mvcc store and the old store v2 are separate and isolated; writes to the store v2 will not affect the mvcc store and, similarly, writes to the mvcc store will not affect the store v2. - -Migrating an application from the API v2 to the API v3 involves two steps: 1) migrate the client library and, 2) migrate the data. If the application can rebuild the data, then migrating the data is unnecessary. - -## Migrate client library - -API v3 is different from API v2, thus application developers need to use a new client library to send requests to etcd API v3. The documentation of the client v3 is available at https://pkg.go.dev/github.com/etcd-io/etcd/clientv3. - -There are some notable differences between API v2 and API v3: - -- Transaction: In v3, etcd provides multi-key conditional transactions. Applications should use transactions in place of `Compare-And-Swap` operations. - -- Flat key space: There are no directories in API v3, only keys. For example, "/a/b/c/" is a key. Range queries support getting all keys matching a given prefix. - -- Compacted responses: Operations like `Delete` no longer return previous values. To get the deleted value, a transaction can be used to atomically get the key and then delete its value. - -- Leases: A replacement for v2 TTLs; the TTL is bound to a lease and keys attach to the lease. When the TTL expires, the lease is revoked and all attached keys are removed. - -## Migrate data - -Application data can be migrated either offline or online. Offline migration is much simpler than online migration and is recommended. - -Sometimes an etcd cluster will possibly have v3 data which should not be overwritten. In this case, the migration process may want to confirm no v3 data is committed before proceeding. One way to check the cluster has no v3 keys is to issue the following `etcdctl` command, which scans the entire v3 keyspace for any key, expecting `0` as output: - -```sh -ETCDCTL_API=3 etcdctl get "" --from-key --keys-only --limit 1 | wc -l -``` - -### Offline migration - -Offline migration is very simple but requires etcd downtime. If an etcd downtime window spanning from seconds to minutes is acceptable, offline migration is a good choice and is easy to automate. - -First, all members in the etcd cluster must converge to the same state. This can be achieved by stopping all applications that write keys to etcd. Alternatively, if the applications must remain running, configure etcd to listen on a different client URL and restart all etcd members. To check if the states converged, within a few seconds, use the `ETCDCTL_API=3 etcdctl endpoint status` command to confirm that the `raft index` of all members match (or differ by at most 1 due to an internal sync raft command). - -Second, migrate the v2 keys into v3 with the [migrate][migrate_command] (`ETCDCTL_API=3 etcdctl migrate`) command. The migrate command writes keys in the v2 store to a user-provided transformer program and reads back transformed keys. It then writes transformed keys into the mvcc store. This usually takes at most tens of seconds. - -Restart the etcd members and everything should just work. - -For etcd v3.3+, run `ETCDCTL_API=3 etcdctl endpoint hashkv --cluster` to ensure key-value stores are consistent post migration. - -**Warn**: When v2 store has expiring TTL keys and migrate command intends to preserve TTLs, migration may be inconsistent with the last committed v2 state when run on any member with a raft index less than the last leader's raft index. - -### Online migration - -If the application cannot tolerate any downtime, then it must migrate online. The implementation of online migration will vary from application to application but the overall idea is the same. - -First, write application code using the v3 API. The application must support two modes: a migration mode and a normal mode. The application starts in migration mode. When running in migration mode, the application reads keys using the v3 API first, and, if it cannot find the key, it retries with the API v2. In normal mode, the application only reads keys using the v3 API. The application writes keys over the API v3 in both modes. To acknowledge a switch from migration mode to normal mode, the application watches on a switch mode key. When switch key’s value turns to `true`, the application switches over from migration mode to normal mode. - -Second, start a background job to migrate data from the store v2 to the mvcc store by reading keys from the API v2 and writing keys to the API v3. - -After finishing data migration, the background job writes `true` into the switch mode key to notify the application that it may switch modes. - -Online migration can be difficult when the application logic depends on store v2 indexes. Applications will need additional logic to convert mvcc store revisions to store v2 indexes. - -[migrate_command]: https://github.com/etcd-io/etcd/tree/master/etcdctl#migrate-options diff --git a/content/en/docs/v3.6/upgrades/upgrading-etcd.md b/content/en/docs/v3.6/upgrades/upgrading-etcd.md index b9df045b0b91b..d102843e020e4 100644 --- a/content/en/docs/v3.6/upgrades/upgrading-etcd.md +++ b/content/en/docs/v3.6/upgrades/upgrading-etcd.md @@ -6,10 +6,6 @@ description: Documentation list for upgrading etcd clusters and applications This section contains documents specific to upgrading etcd clusters and applications. -## Moving from etcd API v2 to API v3 - -* [Migrate applications from using API v2 to API v3][migrate-apps] - ## Upgrading an etcd v3.x cluster * [Upgrade etcd from 3.0 to 3.1](../upgrade_3_1/) * [Upgrade etcd from 3.1 to 3.2](../upgrade_3_2/)