Skip to content

Latest commit

 

History

History
89 lines (69 loc) · 3.69 KB

README.md

File metadata and controls

89 lines (69 loc) · 3.69 KB

1. Usage in NiFi

You can use NiFi processor (nifi-cql-nar-*.nar).

  • ✅ PutCQL as NiFi v2 processor with controller, where inputs are FlowFiles

2. PutCQL (NiFi processor)

2.1 Preconditions for usage

You have to do these steps (only one-time action):

  1. You need the NAR file 'nifi-cql-nar-*.nar'
    • the last version is here
  2. Import the *.nar file to the NiFi lib directory
    • expected location in Linux e.g. '/opt/nifi/current-nifi/lib'
    • expected location in Windows e.g. 'C:\Program Files\Apache\Nifi\lib'
  3. STOP NiFi
  4. START NiFi

2.2 Add controller and setup properties

CQLControllerService, select processor

CQLControllerService

CQLControllerService, properties

CQLControllerService setting (key items):

  • IP Addresses:
    • IP addresses of CQL engine with comma delimiter e.g. '10.129.53.10, 10.129.53.11, 10.129.53.12' or 'localhost', etc.
  • Port:
    • Port for communication with CQL engine (default is 9042).
  • Secure Connection Bundle:
    • Secure Connection Bundle for access to AstraDB (it is the link to '*.zip' file, downloaded from AstraDB web).
    • NOTE: the 'username' is 'clientId' and 'password' id 'secret', these values are from the file '*-token.json', downloaded from AstraDB web.
  • Username:
    • Username for login to CQL.
  • Password:
    • Password for login to CQL.
  • Local Data Center:
    • Name of local data center in CQL typically e.g. 'dc1' or 'datacenter1', etc.
  • Connection Timeout:
    • Connection timeout in seconds (default is 900 seconds).
  • Request Timeout:
    • Request timeout in seconds (default is 60 seconds).
  • Consistency Level:
    • Default consistency level (e.g. LOCAL_ONE, LOCAL_QUORUM, etc., default is LOCAL_ONE).
  • SSL Context Service:
    • SSL context service for CQL connection (default is without setting).

2.3 Add processor and setup properties

PutCQL, select processor

PutCQL, processor

PutCQL, properties

Input

  • FlowFile with CSV content for import (content see), where the CSV content (with header) is based on 'keyspace.table' definition in CQL (from data types point of view).

Output

  • cql.count (FlowFile attribute)
    • The amount of write rows to CQL.

PutCQL setting (key items):

  • Service Controller:
  • Write Consistency Level:
    • Consistency level for write operation (e.g. LOCAL_ONE, LOCAL_QUORUM, etc., default is LOCAL_ONE).
  • Table:
    • Schema and table name in CQL for write/put a data (expected format 'keyspace.table', e.g. 'cqlschema.cqltable').
    • The data types defined in the CQL table will be used for value conversions from the CSV file.
  • Batch Size:
    • Size of batch for write to CQL (default is 200).
  • Batch Type:
    • Batch type with relation to an atomicity of batch operation (default UNLOGGED, it is without atomicity).
  • Dry Run:
    • The simulation of write to CQL (default is false).