feat: add shard-group duration settings to CLI and API #20620

hinst · 2021-01-27T11:08:59Z

Closes #19764

Changes:

Shard duration can be set using command-line client
Shard duration can be set using Rest API

CHANGELOG.md updated with a link to the PR (not the Issue)
Well-formatted commit messages
Rebased/mergeable
Tests pass
http/swagger.yml updated (if modified Go structs or API)
Feature flagged (if modified API)
Documentation updated or issue created (provide link to issue/pr)
Signed CLA (if not already signed)

…ient

…d-line client

… CLI and API

danxmoran · 2021-01-28T18:33:19Z

Hi @hinst, thank you for this contribution! Since this affects the server's public API, I'll need to check with our Cloud team to be sure it's something they're also able to implement and support.

danxmoran · 2021-03-01T20:04:09Z

@hinst apologies for the long delay. The good news is that we intent to accept this change! Ideally as part of our upcoming 2.0.5 release.

I need to double-check the proposed API for any edge cases and open a parallel PR against our Cloud implementation (returning a "not implemented" error if the new feature is used there)

danxmoran · 2021-03-01T20:07:44Z

CHANGELOG.md

@@ -60,6 +60,7 @@ Replacement `tsi1` indexes will be automatically generated on startup for shards
 1. [20565](https://github.com/influxdata/influxdb/pull/20565): Update V1 API spec to document all valid Accept headers and matching Content-Types.
 1. [20578](https://github.com/influxdata/influxdb/pull/20578): Respect the --skip-verify flag when running `influx query`.
 1. [20495](https://github.com/influxdata/influxdb/pull/20495): Update Flux functions list in UI to reflect that `v1` package was renamed to `schema`.
+1. [19764](https://github.com/influxdata/influxdb/pull/20620): Add shard duration settings to CLI and API


After rebasing onto / merging with latest master, this will need to move up to the "Features" section under "unreleased". Would propose:

Suggested change

1. [19764](https://github.com/influxdata/influxdb/pull/20620): Add shard duration settings to CLI and API

1. [20620](https://github.com/influxdata/influxdb/pull/20620): Add support for explicitly setting shard-group durations. Thanks @hinst!

danxmoran · 2021-03-01T20:13:53Z

cmd/influx/bucket.go

@@ -73,6 +74,7 @@ func (b *cmdBucketBuilder) cmdCreate() *cobra.Command {

 	cmd.Flags().StringVarP(&b.description, "description", "d", "", "Description of bucket that will be created")
 	cmd.Flags().StringVarP(&b.retention, "retention", "r", "", "Duration bucket will retain data. 0 is infinite. Default is 0.")
+	cmd.Flags().StringVarP(&b.shardGroupDuration, "shardDuration", "", "", "Shard group duration used internally in storage. Default is 7 days")


Suggested change

cmd.Flags().StringVarP(&b.shardGroupDuration, "shardDuration", "", "", "Shard group duration used internally in storage. Default is 7 days")

cmd.Flags().StringVarP(&b.shardGroupDuration, "shardDuration", "", "", "Shard group duration used internally by the storage engine. Can only be specified when using InfluxDB OSS.")

danxmoran · 2021-03-01T20:15:12Z

cmd/influx/bucket.go

@@ -335,6 +343,7 @@ func (b *cmdBucketBuilder) printBuckets(printOpt bucketPrintOpt) error {
 			"ID":              bkt.ID.String(),
 			"Name":            bkt.Name,
 			"Retention":       bkt.RetentionPeriod,
+			"Shard duration":  bkt.ShardGroupDuration,


Need to think through what this will look like when listing objects returned by InfluxDB Cloud.

danxmoran · 2021-03-01T20:19:17Z

cmd/influx/bucket.go

@@ -320,7 +328,7 @@ func (b *cmdBucketBuilder) printBuckets(printOpt bucketPrintOpt) error {



It's too far from the diff for me to comment on directly, but I think we'll also need support in the update command.

The server-side code for the update path will also need changed. At the moment, I believe it will reset the shard-group duration every time the retention policy is updated. We probably instead want the update API to accept both an RP and a shard-group duration (each one optional), and use:

Duration is unset = don't change the current shard-group duration

Duration set to 0 = have the server recompute a "good" duration based on the bucket's RP

danxmoran · 2021-03-01T20:19:51Z

http/swagger.yml

+        shardGroupDuration:
+          type: integer
+          format: int64
+          description: shard duration measured in nanoseconds


Instead of nanos, could we use seconds to match the units used by retentionRules?

danxmoran · 2021-03-01T20:20:54Z

bucket.go

@@ -36,6 +36,7 @@ type Bucket struct {
 	Description         string        `json:"description"`
 	RetentionPolicyName string        `json:"rp,omitempty"` // This to support v1 sources
 	RetentionPeriod     time.Duration `json:"retentionPeriod"`
+	ShardGroupDuration  time.Duration `json:"shardGroupDuration"`


We may need to make this:

Suggested change

ShardGroupDuration time.Duration `json:"shardGroupDuration"`

ShardGroupDuration *time.Duration `json:"shardGroupDuration,omitempty"`

So Cloud can distinguish not-set from explicitly-0. Will get back to you after I work on the Cloud side of things...

danxmoran · 2021-03-09T14:27:57Z

For anyone watching this PR: I've pulled @hinst's commits into a branch of my own, where I'm iterating on the API design

rajaimrankhan · 2021-03-09T14:41:54Z

Hi

danxmoran · 2021-03-09T22:12:43Z

Superseded by #20911.

Thanks for getting this started @hinst! I'll make sure it gets into 2.0.5

Alexander Savinykh added 5 commits January 25, 2021 11:31

fix(cmd/influx) make shard duration configurable

eef8ce4

fix(cmd/influx) make shard duration configurable from command-line cl…

244b7d7

…ient

fix(cmd/influx) make shard duration configurable from command-line cl…

f6acdeb

…ient

fix(cmd/influx) added API documentation for shardGroupDuration field

fea66bc

fix(cmd/influx) shortened shard group duration column title in comman…

92644b6

…d-line client

hinst changed the title ~~Shard duration config 19764~~ Add shard duration settings to the CLI Jan 27, 2021

hinst changed the title ~~Add shard duration settings to the CLI~~ fix(influx/cmd): Add shard duration settings to the CLI Jan 27, 2021

hinst changed the title ~~fix(influx/cmd): Add shard duration settings to the CLI~~ fix(influx/cmd): Add shard duration settings to CLI and API Jan 27, 2021

Alexander Savinykh added 3 commits January 27, 2021 12:13

fix(influx/cmd): Updated CHANGELOG.md: Add shard duration settings to…

f8dcd97

… CLI and API

Merge branch 'master' into shard-duration-config-19764

df0d11d

fix(influx/cmd): reformatted bucket.go

b16764d

danxmoran self-assigned this Jan 28, 2021

hinst marked this pull request as ready for review January 28, 2021 23:23

russorat requested a review from danxmoran February 3, 2021 17:19

timhallinflux added area/2.x OSS 2.0 related issues and PRs area/api area/cli labels Feb 23, 2021

wagerfield mentioned this pull request Feb 24, 2021

Add shard duration settings to the CLI #19764

Closed

danxmoran changed the title ~~fix(influx/cmd): Add shard duration settings to CLI and API~~ feat: add shard-group duration settings to CLI and API Feb 24, 2021

danxmoran reviewed Mar 1, 2021

View reviewed changes

danxmoran mentioned this pull request Mar 9, 2021

feat(cmd/influx): allow setting shard-group durations for buckets via API and CLI #20911

Merged

danxmoran closed this Mar 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add shard-group duration settings to CLI and API #20620

feat: add shard-group duration settings to CLI and API #20620

hinst commented Jan 27, 2021 •

edited

Loading

danxmoran commented Jan 28, 2021

danxmoran commented Mar 1, 2021

danxmoran Mar 1, 2021

danxmoran Mar 1, 2021

danxmoran Mar 1, 2021

danxmoran Mar 1, 2021

danxmoran Mar 1, 2021

danxmoran Mar 1, 2021

danxmoran commented Mar 9, 2021

rajaimrankhan commented Mar 9, 2021

danxmoran commented Mar 9, 2021

	1. [19764](https://github.com/influxdata/influxdb/pull/20620): Add shard duration settings to CLI and API
	1. [20620](https://github.com/influxdata/influxdb/pull/20620): Add support for explicitly setting shard-group durations. Thanks @hinst!

	cmd.Flags().StringVarP(&b.shardGroupDuration, "shardDuration", "", "", "Shard group duration used internally in storage. Default is 7 days")
	cmd.Flags().StringVarP(&b.shardGroupDuration, "shardDuration", "", "", "Shard group duration used internally by the storage engine. Can only be specified when using InfluxDB OSS.")

		@@ -320,7 +328,7 @@ func (b *cmdBucketBuilder) printBuckets(printOpt bucketPrintOpt) error {

	ShardGroupDuration time.Duration `json:"shardGroupDuration"`
	ShardGroupDuration *time.Duration `json:"shardGroupDuration,omitempty"`

feat: add shard-group duration settings to CLI and API #20620

feat: add shard-group duration settings to CLI and API #20620

Conversation

hinst commented Jan 27, 2021 • edited Loading

danxmoran commented Jan 28, 2021

danxmoran commented Mar 1, 2021

danxmoran Mar 1, 2021

Choose a reason for hiding this comment

danxmoran Mar 1, 2021

Choose a reason for hiding this comment

danxmoran Mar 1, 2021

Choose a reason for hiding this comment

danxmoran Mar 1, 2021

Choose a reason for hiding this comment

danxmoran Mar 1, 2021

Choose a reason for hiding this comment

danxmoran Mar 1, 2021

Choose a reason for hiding this comment

danxmoran commented Mar 9, 2021

rajaimrankhan commented Mar 9, 2021

danxmoran commented Mar 9, 2021

hinst commented Jan 27, 2021 •

edited

Loading