Merge branch 'master' into feat/upgrade

.circleci/config.yml

@@ -450,7 +450,7 @@ jobs:
     environment:
       GOCACHE: /tmp/go-cache
       GOFLAGS: "-mod=readonly -p=4" # Go on Circle thinks 32 CPUs are available, but there aren't.
-      DOCKER_VERSION: 2.0.0-beta
+      DOCKER_VERSION: 2.0.0-rc
     working_directory: /go/src/github.com/influxdata/influxdb
     steps:
       - checkout

CHANGELOG.md

@@ -1,4 +1,4 @@
-## v2.0.0-beta.17 [unreleased]
+## v2.0.0-rc.0 [2020-09-29]
 
 ### Breaking Changes
 
@@ -37,6 +37,8 @@ need to update any InfluxDB CLI config profiles with the new port number.
 1. [19508](https://github.com/influxdata/influxdb/pull/19508): Add subset of InfluxQL coordinator options as flags
 1. [19457](https://github.com/influxdata/influxdb/pull/19457): Add ability to export resources by name via the CLI
 1. [19640](https://github.com/influxdata/influxdb/pull/19640): Turn on Community Templates
+1. [19663](https://github.com/influxdata/influxdb/pull/19663): Added InfluxDB v2 Listener, NSD, OPC-UA, and Windows Event Log to the sources page
+1. [19662](https://github.com/influxdata/influxdb/pull/19662): Add `max-line-length` switch to `influx write` command to address `token too long` errors for large inputs
 
 ### Bug Fixes
 
@@ -53,6 +55,7 @@ need to update any InfluxDB CLI config profiles with the new port number.
 1. [19066](https://github.com/influxdata/influxdb/pull/19066): Drop deprecated /packages route tree
 1. [19116](https://github.com/influxdata/influxdb/pull/19116): Support more types for template envRef default value and require explicit default values
 1. [19104](https://github.com/influxdata/influxdb/pull/19104): Remove orgs/labels nested routes from the API.
+1. [19653](https://github.com/influxdata/influxdb/pull/19653): Remove PointBatcher from tsdb package API
 
 ### Features
 

README.md

@@ -1,48 +1,15 @@
 # InfluxDB [![CircleCI](https://circleci.com/gh/influxdata/influxdb.svg?style=svg)](https://circleci.com/gh/influxdata/influxdb)
 [![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://www.influxdata.com/slack)
 
-
 InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or monitoring and alerting purposes, user dashboards, and visualizing and exploring the data and more. The master branch on this repo now represents the latest InfluxDB, which now includes functionality for Kapacitor (background processing) and Chronograf (the UI) all in a single binary.
 
 The list of InfluxDB Client Libraries that are compatible with the latest version can be found in [our documentation](https://v2.docs.influxdata.com/v2.0/reference/api/client-libraries/).
 
 If you are looking for the 1.x line of releases, there are branches for each minor version as well as a `master-1.x` branch that will contain the code for the next 1.x release. The master-1.x [working branch is here](https://github.com/influxdata/influxdb/tree/master-1.x). The [InfluxDB 1.x Go Client can be found here](https://github.com/influxdata/influxdb1-client).
 
-## State of the Project
-
-The latest InfluxDB 1.x is the stable release and recommended for production use. The InfluxDB that is on the master branch is currently in the beta stage. This means that it is still **NOT** recommended for production usage. There may be breaking API changes, breaking changes in the [Flux language](https://github.com/influxdata/flux), changes in the underlying storage format that will require you to delete all your data, and significant changes to the UI. The beta is intended for feature exploration and gathering feedback on the available feature set. It **SHOULD NOT** be used for performance testing, benchmarks, or other stress tests.
-
-Additional features will arrive during the beta period until we reach general availability (GA). We will be cutting versioned releases at least every two weeks starting in the first release. There will also be nightly builds based off the latest code in master.
-
-Once we close on the final feature set of what will be in the first GA release of InfluxDB in the 2.x line, we will move into the release candidate (RC) phase. At that point, we do not expect there to be breaking changes to the API or Flux language. We may still need to make a breaking change prior to GA due to some unforseen circumstance, but it would need to be extremely important and will be clearly communicated via the changelog and all available channels.
-
-Our current plans are to release RCs suitable for production usage, but we will re-evaluate in consultation with the community as the cycle progresses. During the RC period, we will focus on feedback from users, bug fixes, performance, and additive features (where time permits).
-
-### What you can expect in the Beta and RC Phases
-
-#### Beta
-**Releases every two weeks or as needed**
-
-Planned additions include:
-- Compatibility layer with 1.x including: 1.x HTTP Write API  and HTTP Read API support for InfluxQL
-- Import Bulk Data from 1.x - convert TSM from 1.x to 2.x
-- Performance tuning, stability improvements, and fine tuning based on community feedback.
-- Finalization of supported client libraries starting with JavaScript and Go.
-
-#### RC
-**As needed**
-
-Planned activities include:
-- Performance tuning, stability improvements, and fine-tuning based on community feedback.
-
-### What is **NOT** planned?
-- Migration of users/security permissions from InfluxDB v1.x to 2.x.  ACTION REQUIRED: Re-establish users and permissions within the new unified security model which now spans the underlying database and user interface.
-- Migration of Continuous Queries.  ACTION REQUIRED: These will need to be re-implemented as Flux tasks.
-- Direct support by InfluxDB for CollectD, StatsD, Graphite, or UDP.  ACTION REQUIRED: Leverage Telegraf 1.9+ along with the InfluxDB v2.0 output plugin to translate these protocols/formats.
-
 ## Installing from Source
 
-We have nightly and weekly versioned Docker images, Debian packages, RPM packages, and tarballs of InfluxDB available at the [InfluxData downloads page](https://portal.influxdata.com/downloads/). We also provide the `influx` command line interface (CLI) client as a separate binary available at the same location.
+We have nightly and versioned Docker images, Debian packages, RPM packages, and tarballs of InfluxDB available at the [InfluxData downloads page](https://portal.influxdata.com/downloads/). We also provide the `influx` command line interface (CLI) client as a separate binary available at the same location.
 
 ## Building From Source
 
@@ -126,7 +93,7 @@ $ go test ./...
 
 ## Getting Started
 
-For a complete getting started guide, please see our full [online documentation site](https://v2.docs.influxdata.com/v2.0/). 
+For a complete getting started guide, please see our full [online documentation site](https://docs.influxdata.com/influxdb/v2.0/). 
 
 To write and query data or use the API in any way, you'll need to first create a user, credentials, organization and bucket.
 Everything in InfluxDB is organized under a concept of an organization. The API is designed to be multi-tenant.
@@ -164,7 +131,7 @@ Confirm? (y/n): y
 
 UserID                  Username        Organization    Bucket
 033a3f2c5ccaa000        marty           InfluxData      Telegraf
-Your token has been stored in /Users/marty/.influxdbv2/configs
+Your token has been stored in /Users/marty/.influxdbv2/credentials
 ```
 
 You can run this command non-interactively using the `-f, --force` flag if you are automating the setup.
@@ -184,7 +151,7 @@ Once setup is complete, a configuration profile is created to allow you to inter
 ```bash
 $ bin/$(uname -s | tr '[:upper:]' '[:lower:]')/influx config
 Active	Name	URL			            Org
-*	    default	http://localhost:8086	InfluxData
+*	    default	http://localhost:9999	InfluxData
 ```
 
 ## Writing Data

SECURITY.md

@@ -7,10 +7,10 @@ These versions of InfluxDB are currently being supported with security updates.
 | Version | Supported          |
 | ------- | ------------------ |
 | 2.0.x   | :white_check_mark: |
+| 1.8.x   | :white_check_mark: |
 | 1.7.x   | :white_check_mark: |
-| 1.6.x   | :white_check_mark: |
-| < 1.6.x   | :x:                |
+| < 1.7.x | :x:                |
 
 ## Reporting a Vulnerability
 
-InfluxData takes security and our users' trust very seriously. If you believe you have found a security issue in any of our open source projects, please responsibly disclose it by contacting security@influxdata.com. More details about security vulnerability reporting, including our GPG key, can be found here. https://www.influxdata.com/how-to-report-security-vulnerabilities/
+InfluxData takes security and our users' trust very seriously. If you believe you have found a security issue in any of our open source projects, please responsibly disclose it by contacting security@influxdata.com. More details about security vulnerability reporting, including our GPG key, can be found at https://www.influxdata.com/how-to-report-security-vulnerabilities/.

cmd/influx/setup.go

@@ -222,6 +222,10 @@ func onboardingRequest() (*influxdb.OnboardingRequest, error) {
 }
 
 func nonInteractive() (*influxdb.OnboardingRequest, error) {
+	if len(setupFlags.password) < internal2.MinPasswordLen {
+		return nil, internal2.ErrPasswordIsTooShort
+	}
+
 	req := &influxdb.OnboardingRequest{
 		User:            setupFlags.username,
 		Password:        setupFlags.password,
@@ -253,7 +257,7 @@ func interactive() (req *influxdb.OnboardingRequest, err error) {
 	} else {
 		req.User = internal2.GetInput(ui, "Please type your primary username", "")
 	}
-	if setupFlags.password != "" {
+	if setupFlags.password != "" && len(setupFlags.password) >= internal2.MinPasswordLen {
 		req.Password = setupFlags.password
 	} else {
 		req.Password = internal2.GetPassword(ui, false)
@@ -285,7 +289,7 @@ func interactive() (req *influxdb.OnboardingRequest, err error) {
 			rpStr := internal2.GetInput(ui, "Please type your retention period in hours.\r\nOr press ENTER for infinite.", strconv.Itoa(influxdb.InfiniteRetention))
 			rp, err := strconv.Atoi(rpStr)
 			if rp >= 0 && err == nil {
-				req.RetentionPeriod = uint(rp)
+				req.RetentionPeriod = uint(rp) * uint(time.Hour)
 				break
 			}
 		}

cmd/influx/write.go

@@ -37,6 +37,7 @@ type writeFlagsType struct {
 	Debug                      bool
 	SkipRowOnError             bool
 	SkipHeader                 int
+	MaxLineLength              int
 	IgnoreDataTypeInColumnName bool
 	Encoding                   string
 	ErrorsFile                 string
@@ -84,6 +85,7 @@ func cmdWrite(f *globalFlags, opt genericCLIOpts) *cobra.Command {
 	cmd.PersistentFlags().BoolVar(&writeFlags.Debug, "debug", false, "Log CSV columns to stderr before reading data rows")
 	cmd.PersistentFlags().BoolVar(&writeFlags.SkipRowOnError, "skipRowOnError", false, "Log CSV data errors to stderr and continue with CSV processing")
 	cmd.PersistentFlags().IntVar(&writeFlags.SkipHeader, "skipHeader", 0, "Skip the first <n> rows from input data")
+	cmd.PersistentFlags().IntVar(&writeFlags.MaxLineLength, "max-line-length", 16_000_000, "Specifies the maximum number of bytes that can be read for a single line")
 	cmd.Flag("skipHeader").NoOptDefVal = "1" // skipHeader flag value is optional, skip the first header when unspecified
 	cmd.PersistentFlags().BoolVar(&writeFlags.IgnoreDataTypeInColumnName, "xIgnoreDataTypeInColumnName", false, "Ignores dataType which could be specified after ':' in column name")
 	cmd.PersistentFlags().MarkHidden("xIgnoreDataTypeInColumnName") // should be used only upon explicit advice
@@ -319,6 +321,7 @@ func fluxWriteF(cmd *cobra.Command, args []string) error {
 			Precision:          writeFlags.Precision,
 			InsecureSkipVerify: flags.skipVerify,
 		},
+		MaxLineLength: writeFlags.MaxLineLength,
 	}
 	if err := s.Write(ctx, orgID, bucketID, r); err != nil && err != context.Canceled {
 		return fmt.Errorf("failed to write data: %v", err)

cmd/influxd/upgrade/setup.go

@@ -37,6 +37,9 @@ func onboardingRequest() (*influxdb.OnboardingRequest, error) {
 }
 
 func nonInteractive() (*influxdb.OnboardingRequest, error) {
+	if len(options.target.password) < internal.MinPasswordLen {
+		return nil, internal.ErrPasswordIsTooShort
+	}
 	req := &influxdb.OnboardingRequest{
 		User:            options.target.userName,
 		Password:        options.target.password,
@@ -68,7 +71,7 @@ func interactive() (req *influxdb.OnboardingRequest, err error) {
 	} else {
 		req.User = internal.GetInput(ui, "Please type your primary username", "")
 	}
-	if options.target.password != "" {
+	if options.target.password != "" && len(options.target.password) >= internal.MinPasswordLen {
 		req.Password = options.target.password
 	} else {
 		req.Password = internal.GetPassword(ui, false)

cmd/internal/input.go

@@ -11,6 +11,13 @@ import (
 	"github.com/tcnksm/go-input"
 )
 
+const MinPasswordLen int = 8
+
+var (
+	ErrPasswordNotMatch   = fmt.Errorf("passwords do not match")
+	ErrPasswordIsTooShort = fmt.Errorf("password is too short")
+)
+
 // vt100EscapeCodes
 var (
 	KeyEscape   = byte(27)
@@ -46,10 +53,6 @@ func GetConfirm(ui *input.UI, promptFunc func() string) bool {
 	}
 }
 
-var errPasswordNotMatch = fmt.Errorf("passwords do not match")
-
-var errPasswordIsTooShort error = fmt.Errorf("password is too short")
-
 func GetSecret(ui *input.UI) (secret string) {
 	var err error
 	query := string(PromptWithColor("Please type your secret", ColorCyan))
@@ -89,15 +92,15 @@ enterPassword:
 			Mask:      false,
 			ValidateFunc: func(s string) error {
 				if len(s) < 8 {
-					return errPasswordIsTooShort
+					return ErrPasswordIsTooShort
 				}
 				return nil
 			},
 		})
 		switch err {
 		case input.ErrInterrupted:
 			os.Exit(1)
-		case errPasswordIsTooShort:
+		case ErrPasswordIsTooShort:
 			ui.Writer.Write(PromptWithColor("Password too short - minimum length is 8 characters.\n\r", ColorRed))
 			continue
 		default:
@@ -115,7 +118,7 @@ enterPassword:
 			Hide:      true,
 			ValidateFunc: func(s string) error {
 				if s != password {
-					return errPasswordNotMatch
+					return ErrPasswordNotMatch
 				}
 				return nil
 			},

kit/io/limited_read_closer.go

@@ -11,38 +11,49 @@ var ErrReadLimitExceeded = errors.New("read limit exceeded")
 // io.LimitedReader. It allows us to obtain the limit error at the time of close
 // instead of just when writing.
 type LimitedReadCloser struct {
-	*io.LimitedReader
-	err   error
-	close func() error
+	R             io.ReadCloser // underlying reader
+	N             int64         // max bytes remaining
+	err           error
+	closed        bool
+	limitExceeded bool
 }
 
 // NewLimitedReadCloser returns a new LimitedReadCloser.
 func NewLimitedReadCloser(r io.ReadCloser, n int64) *LimitedReadCloser {
-	// read up to max + 1 as limited reader just returns EOF when the limit is reached
-	// or when there is nothing left to read. If we exceed the max batch size by one
-	// then we know the limit has been passed.
 	return &LimitedReadCloser{
-		LimitedReader: &io.LimitedReader{R: r, N: n + 1},
-		close:         r.Close,
+		R: r,
+		N: n,
 	}
 }
 
+func (l *LimitedReadCloser) Read(p []byte) (n int, err error) {
+	if l.N <= 0 {
+		l.limitExceeded = true
+		return 0, io.EOF
+	}
+	if int64(len(p)) > l.N {
+		p = p[0:l.N]
+	}
+	n, err = l.R.Read(p)
+	l.N -= int64(n)
+	return
+}
+
 // Close returns an ErrReadLimitExceeded when the wrapped reader exceeds the set
 // limit for number of bytes.  This is safe to call more than once but not
 // concurrently.
 func (l *LimitedReadCloser) Close() (err error) {
-	defer func() {
-		if cerr := l.close(); cerr != nil && err == nil {
-			err = cerr
-		}
-
-		// only call close once
-		l.close = func() error { return nil }
-	}()
-
-	if l.N < 1 {
+	if l.limitExceeded {
 		l.err = ErrReadLimitExceeded
 	}
-
+	if l.closed {
+		// Close has already been called.
+		return l.err
+	}
+	if err := l.R.Close(); err != nil && l.err == nil {
+		l.err = err
+	}
+	// Prevent l.closer.Close from being called again.
+	l.closed = true
 	return l.err
 }