Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/flowise multi database support #575

Merged
merged 6 commits into from
Jul 19, 2023
Merged

Feature/flowise multi database support #575

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
10563bd
Merge pull request #1 from FlowiseAI/main
atishhamte Jul 15, 2023
c5eff32
Multiple Database Support
atishhamte Jul 18, 2023
fe7e0d1
Merge branch 'main' into feature/FlowiseMultiDatabaseSupport
atishhamte Jul 18, 2023
318bd51
Update .env.example
atishhamte Jul 18, 2023
4c4fd6f
Resolved linting error
atishhamte Jul 18, 2023
7f5acf3
Readme updated
atishhamte Jul 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,18 +131,25 @@ FLOWISE_PASSWORD=1234
Flowise support different environment variables to configure your instance. You can specify the following variables in the `.env` file inside `packages/server` folder. Read [more](https://docs.flowiseai.com/environment-variables)

| Variable | Description | Type | Default |
| -------------------------- | ---------------------------------------------------------------- | ------------------------------------------------ | ----------------------------------- |
| ---------------- | ---------------------------------------------------------------- | ------------------------------------------------ | ----------------------------------- |
| PORT | The HTTP port Flowise runs on | Number | 3000 |
| FLOWISE_USERNAME | Username to login | String |
| FLOWISE_PASSWORD | Password to login | String |
| DEBUG | Print logs onto terminal/console | Boolean |
| LOG_PATH | Location where log files are stored | String | `your-path/Flowise/packages/server` |
| LOG_LEVEL | Different log levels for loggers to be saved | Enum String: `error`, `info`, `verbose`, `debug` | `info` |
| DATABASE_PATH | Location where database is saved | String | `your-home-dir/.flowise` |
| FLOWISE_USERNAME | Username to login | String | |
| FLOWISE_PASSWORD | Password to login | String | |
| DEBUG | Print logs from components | Boolean | |
| LOG_PATH | Location where log files are stored | String | `your-path/Flowise/logs` |
| LOG_LEVEL | Different levels of logs | Enum String: `error`, `info`, `verbose`, `debug` | `info` |
| APIKEY_PATH | Location where api keys are saved | String | `your-path/Flowise/packages/server` |
| EXECUTION_MODE | Whether predictions run in their own process or the main process | Enum String: `child`, `main` | `main` |
| TOOL_FUNCTION_BUILTIN_DEP | NodeJS built-in modules to be used for Tool Function | String | |
| TOOL_FUNCTION_EXTERNAL_DEP | External modules to be used for Tool Function | String | |
| OVERRIDE_DATABASE | Override current database with default | Enum String: `true`, `false` | `true` |
| DATABASE_TYPE | Type of database to store the flowise data | Enum String: `sqlite`, `mysql`, `postgres` | `sqlite` |
| DATABASE_PATH | Location where database is saved (When DATABASE_TYPE is sqlite) | String | `your-home-dir/.flowise` |
| DATABASE_HOST | Host URL or IP address (When DATABASE_TYPE is not sqlite) | String | |
| DATABASE_PORT | Database port (When DATABASE_TYPE is not sqlite) | String | |
| DATABASE_USERNAME | Database username (When DATABASE_TYPE is not sqlite) | String | |
| DATABASE_PASSWORD | Database password (When DATABASE_TYPE is not sqlite) | String | |
| DATABASE_NAME | Database name (When DATABASE_TYPE is not sqlite) | String | |

You can also specify the env variables when using `npx`. For example:

Expand Down
15 changes: 14 additions & 1 deletion packages/server/.env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,23 @@ PORT=3000
# FLOWISE_USERNAME=user
# FLOWISE_PASSWORD=1234
# DEBUG=true
# DATABASE_PATH=/your_database_path/.flowise
# APIKEY_PATH=/your_api_key_path/.flowise
# LOG_PATH=/your_log_path/.flowise/logs
# LOG_LEVEL=debug (error | warn | info | verbose | debug)
# EXECUTION_MODE=main (child | main)

OVERRIDE_DATABASE="true"
DATABASE_TYPE="sqlite" # sqlite, mysql, postgres

# When database is sqlite
# DATABASE_PATH=/your_database_path/.flowise

# When database is not sqlite
# DATABASE_PORT=""
# DATABASE_HOST=""
# DATABASE_NAME="flowise"
# DATABASE_USER=""
# DATABASE_PASSWORD=""

# TOOL_FUNCTION_BUILTIN_DEP=crypto,fs
# TOOL_FUNCTION_EXTERNAL_DEP=moment,lodash
21 changes: 14 additions & 7 deletions packages/server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,18 +34,25 @@ FLOWISE_PASSWORD=1234
Flowise support different environment variables to configure your instance. You can specify the following variables in the `.env` file inside `packages/server` folder. Read [more](https://docs.flowiseai.com/environment-variables)

| Variable | Description | Type | Default |
| -------------------------- | ---------------------------------------------------------------- | ------------------------------------------------ | ----------------------------------- |
| ---------------- | ---------------------------------------------------------------- | ------------------------------------------------ | ----------------------------------- |
| PORT | The HTTP port Flowise runs on | Number | 3000 |
| FLOWISE_USERNAME | Username to login | String |
| FLOWISE_PASSWORD | Password to login | String |
| DEBUG | Print logs onto terminal/console | Boolean |
| LOG_PATH | Location where log files are stored | String | `your-path/Flowise/packages/server` |
| LOG_LEVEL | Different log levels for loggers to be saved | Enum String: `error`, `info`, `verbose`, `debug` | `info` |
| DATABASE_PATH | Location where database is saved | String | `your-home-dir/.flowise` |
| FLOWISE_USERNAME | Username to login | String | |
| FLOWISE_PASSWORD | Password to login | String | |
| DEBUG | Print logs from components | Boolean | |
| LOG_PATH | Location where log files are stored | String | `your-path/Flowise/logs` |
| LOG_LEVEL | Different levels of logs | Enum String: `error`, `info`, `verbose`, `debug` | `info` |
| APIKEY_PATH | Location where api keys are saved | String | `your-path/Flowise/packages/server` |
| EXECUTION_MODE | Whether predictions run in their own process or the main process | Enum String: `child`, `main` | `main` |
| TOOL_FUNCTION_BUILTIN_DEP | NodeJS built-in modules to be used for Tool Function | String | |
| TOOL_FUNCTION_EXTERNAL_DEP | External modules to be used for Tool Function | String | |
| OVERRIDE_DATABASE | Override current database with default | Enum String: `true`, `false` | `true` |
| DATABASE_TYPE | Type of database to store the flowise data | Enum String: `sqlite`, `mysql`, `postgres` | `sqlite` |
| DATABASE_PATH | Location where database is saved (When DATABASE_TYPE is sqlite) | String | `your-home-dir/.flowise` |
| DATABASE_HOST | Host URL or IP address (When DATABASE_TYPE is not sqlite) | String | |
| DATABASE_PORT | Database port (When DATABASE_TYPE is not sqlite) | String | |
| DATABASE_USERNAME | Database username (When DATABASE_TYPE is not sqlite) | String | |
| DATABASE_PASSWORD | Database password (When DATABASE_TYPE is not sqlite) | String | |
| DATABASE_NAME | Database name (When DATABASE_TYPE is not sqlite) | String | |

You can also specify the env variables when using `npx`. For example:

Expand Down
2 changes: 2 additions & 0 deletions packages/server/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@
"flowise-ui": "*",
"moment-timezone": "^0.5.34",
"multer": "^1.4.5-lts.1",
"mysql": "^2.18.1",
"pg": "^8.11.1",
"reflect-metadata": "^0.1.13",
"socket.io": "^4.6.1",
"sqlite3": "^5.1.6",
Expand Down
60 changes: 52 additions & 8 deletions packages/server/src/ChildProcess.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,14 +142,58 @@ export class ChildProcess {
* @returns {DataSource}
*/
async function initDB() {
const homePath = process.env.DATABASE_PATH ?? path.join(getUserHome(), '.flowise')
const childAppDataSource = new DataSource({
type: 'sqlite',
database: path.resolve(homePath, 'database.sqlite'),
synchronize: true,
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
let childAppDataSource
let homePath
switch (process.env.DATABASE_TYPE) {
case 'sqlite':
homePath = process.env.DATABASE_PATH ?? path.join(getUserHome(), '.flowise')
childAppDataSource = new DataSource({
type: 'sqlite',
database: path.resolve(homePath, 'database.sqlite'),
synchronize: process.env.OVERRIDE_DATABASE == 'true',
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
break
case 'mysql':
childAppDataSource = new DataSource({
type: 'mysql',
host: process.env.DATABASE_HOST,
port: parseInt(process.env.DATABASE_PORT || '3306'),
username: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
database: process.env.DATABASE_NAME,
charset: 'utf8mb4',
synchronize: process.env.OVERRIDE_DATABASE == 'true',
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
break
case 'postgres':
childAppDataSource = new DataSource({
type: 'postgres',
host: process.env.DATABASE_HOST,
port: parseInt(process.env.DATABASE_PORT || '5432'),
username: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
database: process.env.DATABASE_NAME,
synchronize: process.env.OVERRIDE_DATABASE == 'true',
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
break
default:
homePath = process.env.DATABASE_PATH ?? path.join(getUserHome(), '.flowise')
childAppDataSource = new DataSource({
type: 'sqlite',
database: path.resolve(homePath, 'database.sqlite'),
synchronize: process.env.OVERRIDE_DATABASE == 'true',
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
break
}

return await childAppDataSource.initialize()
}

Expand Down
59 changes: 50 additions & 9 deletions packages/server/src/DataSource.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,56 @@ import { getUserHome } from './utils'
let appDataSource: DataSource

export const init = async (): Promise<void> => {
const homePath = process.env.DATABASE_PATH ?? path.join(getUserHome(), '.flowise')

appDataSource = new DataSource({
type: 'sqlite',
database: path.resolve(homePath, 'database.sqlite'),
synchronize: true,
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
let homePath
switch (process.env.DATABASE_TYPE) {
case 'sqlite':
homePath = process.env.DATABASE_PATH ?? path.join(getUserHome(), '.flowise')
appDataSource = new DataSource({
type: 'sqlite',
database: path.resolve(homePath, 'database.sqlite'),
synchronize: process.env.OVERRIDE_DATABASE == 'true',
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
break
case 'mysql':
appDataSource = new DataSource({
type: 'mysql',
host: process.env.DATABASE_HOST,
port: parseInt(process.env.DATABASE_PORT || '3306'),
username: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
database: process.env.DATABASE_NAME,
charset: 'utf8mb4',
synchronize: process.env.OVERRIDE_DATABASE == 'true',
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
break
case 'postgres':
appDataSource = new DataSource({
type: 'postgres',
host: process.env.DATABASE_HOST,
port: parseInt(process.env.DATABASE_PORT || '5432'),
username: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
database: process.env.DATABASE_NAME,
synchronize: process.env.OVERRIDE_DATABASE == 'true',
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
break
default:
homePath = process.env.DATABASE_PATH ?? path.join(getUserHome(), '.flowise')
appDataSource = new DataSource({
type: 'sqlite',
database: path.resolve(homePath, 'database.sqlite'),
synchronize: process.env.OVERRIDE_DATABASE == 'true',
entities: [ChatFlow, ChatMessage, Tool],
migrations: []
})
break
}
}

export function getDataSource(): DataSource {
Expand Down
2 changes: 1 addition & 1 deletion packages/server/src/entity/ChatFlow.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ export class ChatFlow implements IChatFlow {
@Column()
name: string

@Column()
@Column({ type: "text" })
flowData: string

@Column({ nullable: true })
Expand Down
2 changes: 1 addition & 1 deletion packages/server/src/entity/ChatMessage.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ export class ChatMessage implements IChatMessage {
@Column()
chatflowid: string

@Column()
@Column({ type: "text" })
content: string

@Column({ nullable: true })
Expand Down
2 changes: 1 addition & 1 deletion packages/server/src/entity/Tool.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ export class Tool implements ITool {
@Column()
name: string

@Column()
@Column({ type: "text" })
description: string

@Column()
Expand Down