[Roadmap] Apache InLong Roadmap 2022 H1

[dockerzhang]

Description

Hello, everyone, we made a roadmap for 2022 recently, welcome to discuss and contribute.

Overview

Goal

1, Create a pluggable architecture for all modules, especially for agent(source) and sort(sink).
2, Add offline data streams management for InLong.
3, Cover common data sources, data types, and data destinations.

Tasks

1. System

• [Feature] Add audit module to handle all metrics info which reported by client sdk in separate processing #1883
• [INLONG-718] Expose All Metrics by Prometheus Freamwork #1307
• [Feature] Refactoring to plugin architecture #1780
• [INLONG-719] Support Deploying All Modules on Kubernetes #1308
• [Feature] Support Standalone deployment for InLong #2301

2. Manager

• [Feature] Add Data Stream Status Management #2302
• [Feature] Sort Flink Job Management #2303
• [Feature] Metadata Management without ZooKeeper #2304
• [Feature][Manager] Add Cluster Management for MQ/DataProxy/Sort Cluster #2305
• [Feature][Manag] Batch Data Stream Management #2306
• [Feature][Manager] Offline Data Stream Management #2308

3. Agent

• [Feature] Add Database Collect Source For Agent #2309
• [Feature][Aagent] Support Mobile Source #2310
• [Feature] support collect data from Kafka #2311
• [Feature] support collect data with different data types #2312

4. DataProxy

• [Feature] DataProxy Support produce messages for multi MQ clusters at the same time #2313
• [INLONG-2229] Support pulsar anthentication in Manager and Sort #2249
• [INLONG-720] Dataproxy Support Kafka #1309

5. Sort

• [Feature] Support ClickHouse #1715
• [Feature] Support Iceberg #1714
• [Feature][InLong-Sort] Support Kafka sink in InLong-Sort #2250
• [Feature] Support Sort Data to Elasticsearch #2314
• [Feature] Sort Data Stream with Different Data Types #2315
• [Feature][InLong-Sort] Bump flink version to 1.13.5 #2183

6. Community

• plan two developer meetup
• [Improve]Replacing components that do not comply with ASF 3RD PARTY LICENSE POLICY #2281
• [Improve] update the architecture for office-website #1810

Use case

No response

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[shink]

I did some work to support Kubernetes for InLong in #1308, and fixed some reference and configuration bugs in #1562, but there is still some work to do, such as adding some configuration items, completing related documents, etc. I will complete these tasks in 2022.

[dockerzhang]

for ##4990 created, this issue will close. all unfinished tasks will be added to the new roadmap issue.