Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Add Multi Modal Capabilities #94

Merged
merged 22 commits into from
Feb 27, 2024
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
2926344
Render image upload and record audio buttons based on chatbot config
0xi4o Feb 12, 2024
d9d0256
Generate build
0xi4o Feb 12, 2024
bb21191
Make chat window components structure simpler and render image previews
0xi4o Feb 12, 2024
369800f
Fix styling issues
0xi4o Feb 12, 2024
6d12d79
Make audio recording work
0xi4o Feb 12, 2024
6166d5a
Fix issue with sending audio inputs
0xi4o Feb 13, 2024
266c9eb
Show uploads in user chat bubble
0xi4o Feb 13, 2024
a022aaa
Support drag and drop for image uploads
0xi4o Feb 13, 2024
552c352
Fix drag overlay styles
0xi4o Feb 13, 2024
e2a3751
Fix merge conflicts
0xi4o Feb 13, 2024
45097b3
Fix drag overlay default state
0xi4o Feb 13, 2024
15b7f0c
Update recording indicator's color to red
0xi4o Feb 13, 2024
b864516
Fix styles for user chat message bubble when uploads are present
0xi4o Feb 13, 2024
f4a8337
Show transcribed audio inputs as text along with audio clip in user m…
0xi4o Feb 19, 2024
ed75cfb
Generate new build
0xi4o Feb 19, 2024
f044b13
Use api endpoint to stream file uploads in messages instead of using …
0xi4o Feb 20, 2024
af26ff7
Use base64 as src while uploading and switch to api after getting a r…
0xi4o Feb 20, 2024
1f00668
Remove base64 data from being saved to localstorage
0xi4o Feb 20, 2024
d414bc6
Fix image upload not accepting the same images after the first time
0xi4o Feb 21, 2024
f6ca591
Fix previous images breaking in chat
0xi4o Feb 21, 2024
f11e84b
Fix base64 being saved for agents
0xi4o Feb 23, 2024
bbbc344
Merge branch 'main' into feature/multimodal
HenryHengZJ Feb 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dist/components/Badge.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 24 additions & 2 deletions dist/components/Bot.d.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,35 @@
import { BotMessageTheme, TextInputTheme, UserMessageTheme } from '@/features/bubble/types';
export type FileEvent<T = EventTarget> = {
target: T;
};
type ImageUploadConstraits = {
fileTypes: string[];
maxUploadSize: number;
};
export type UploadsConfig = {
imgUploadSizeAndTypes: ImageUploadConstraits[];
isImageUploadAllowed: boolean;
isSpeechToTextEnabled: boolean;
};
type FilePreviewData = string | ArrayBuffer;
type FilePreview = {
data: FilePreviewData;
mime: string;
name: string;
preview: string;
type: string;
};
type messageType = 'apiMessage' | 'userMessage' | 'usermessagewaiting';
type observerConfigType = (accessor: string | boolean | object | MessageType[]) => void;
export type observersConfigType = Record<'observeUserInput' | 'observeLoading' | 'observeMessages', observerConfigType>;
export type FileUpload = Omit<FilePreview, 'preview'>;
export type MessageType = {
message: string;
type: messageType;
sourceDocuments?: any;
fileAnnotations?: any;
fileUploads?: Partial<FileUpload>[];
};
type observerConfigType = (accessor: string | boolean | object | MessageType[]) => void;
export type observersConfigType = Record<'observeUserInput' | 'observeLoading' | 'observeMessages', observerConfigType>;
export type BotProps = {
chatflowid: string;
apiHost?: string;
Expand Down
2 changes: 1 addition & 1 deletion dist/components/Bot.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions dist/components/ImageUploadButton.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import { JSX } from 'solid-js/jsx-runtime';
type ImageUploadButtonProps = {
buttonColor?: string;
isDisabled?: boolean;
isLoading?: boolean;
disableIcon?: boolean;
} & JSX.ButtonHTMLAttributes<HTMLButtonElement>;
export declare const ImageUploadButton: (props: ImageUploadButtonProps) => JSX.Element;
export declare const Spinner: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
export {};
//# sourceMappingURL=ImageUploadButton.d.ts.map
1 change: 1 addition & 0 deletions dist/components/ImageUploadButton.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions dist/components/RecordAudioButton.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import { JSX } from 'solid-js/jsx-runtime';
type RecordAudioButtonProps = {
buttonColor?: string;
isDisabled?: boolean;
isLoading?: boolean;
disableIcon?: boolean;
} & JSX.ButtonHTMLAttributes<HTMLButtonElement>;
export declare const RecordAudioButton: (props: RecordAudioButtonProps) => JSX.Element;
export declare const Spinner: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
export {};
//# sourceMappingURL=RecordAudioButton.d.ts.map
1 change: 1 addition & 0 deletions dist/components/RecordAudioButton.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion dist/components/bubbles/GuestBubble.d.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
import { MessageType } from '../Bot';
type Props = {
message: string;
message: MessageType;
apiHost?: string;
chatflowid: string;
chatId: string;
showAvatar?: boolean;
avatarSrc?: string;
backgroundColor?: string;
Expand Down
2 changes: 1 addition & 1 deletion dist/components/bubbles/GuestBubble.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions dist/components/buttons/CancelButton.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import { JSX } from 'solid-js/jsx-runtime';
type CancelButtonProps = {
buttonColor?: string;
isDisabled?: boolean;
isLoading?: boolean;
disableIcon?: boolean;
} & JSX.ButtonHTMLAttributes<HTMLButtonElement>;
export declare const CancelButton: (props: CancelButtonProps) => JSX.Element;
export {};
//# sourceMappingURL=CancelButton.d.ts.map
1 change: 1 addition & 0 deletions dist/components/buttons/CancelButton.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions dist/components/buttons/ImageUploadButton.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import { JSX } from 'solid-js/jsx-runtime';
type ImageUploadButtonProps = {
buttonColor?: string;
isDisabled?: boolean;
isLoading?: boolean;
disableIcon?: boolean;
} & JSX.ButtonHTMLAttributes<HTMLButtonElement>;
export declare const ImageUploadButton: (props: ImageUploadButtonProps) => JSX.Element;
export declare const Spinner: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
export {};
//# sourceMappingURL=ImageUploadButton.d.ts.map
1 change: 1 addition & 0 deletions dist/components/buttons/ImageUploadButton.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions dist/components/buttons/RecordAudioButton.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import { JSX } from 'solid-js/jsx-runtime';
type RecordAudioButtonProps = {
buttonColor?: string;
isDisabled?: boolean;
isLoading?: boolean;
disableIcon?: boolean;
} & JSX.ButtonHTMLAttributes<HTMLButtonElement>;
export declare const RecordAudioButton: (props: RecordAudioButtonProps) => JSX.Element;
export declare const Spinner: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
export {};
//# sourceMappingURL=RecordAudioButton.d.ts.map
1 change: 1 addition & 0 deletions dist/components/buttons/RecordAudioButton.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions dist/components/buttons/SendButton.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import { JSX } from 'solid-js/jsx-runtime';
type SendButtonProps = {
sendButtonColor?: string;
isDisabled?: boolean;
isLoading?: boolean;
disableIcon?: boolean;
} & JSX.ButtonHTMLAttributes<HTMLButtonElement>;
export declare const SendButton: (props: SendButtonProps) => JSX.Element;
export declare const DeleteButton: (props: SendButtonProps) => JSX.Element;
export declare const Spinner: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
export {};
//# sourceMappingURL=SendButton.d.ts.map
1 change: 1 addition & 0 deletions dist/components/buttons/SendButton.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions dist/components/icons/AddImageIcon.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import { JSX } from 'solid-js/jsx-runtime';
export declare const AddImageIcon: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
//# sourceMappingURL=AddImageIcon.d.ts.map
1 change: 1 addition & 0 deletions dist/components/icons/AddImageIcon.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions dist/components/icons/CircleDotIcon.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import { JSX } from 'solid-js/jsx-runtime';
export declare const CircleDotIcon: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
//# sourceMappingURL=CircleDotIcon.d.ts.map
1 change: 1 addition & 0 deletions dist/components/icons/CircleDotIcon.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions dist/components/icons/RecordIcon.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import { JSX } from 'solid-js/jsx-runtime';
export declare const RecordIcon: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
//# sourceMappingURL=RecordIcon.d.ts.map
1 change: 1 addition & 0 deletions dist/components/icons/RecordIcon.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions dist/components/icons/TrashIcon.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import { JSX } from 'solid-js/jsx-runtime';
export declare const TrashIcon: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
//# sourceMappingURL=TrashIcon.d.ts.map
1 change: 1 addition & 0 deletions dist/components/icons/TrashIcon.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions dist/components/icons/XIcon.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import { JSX } from 'solid-js/jsx-runtime';
export declare const XIcon: (props: JSX.SvgSVGAttributes<SVGSVGElement>) => JSX.Element;
//# sourceMappingURL=XIcon.d.ts.map
1 change: 1 addition & 0 deletions dist/components/icons/XIcon.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions dist/components/icons/index.d.ts
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@
export * from './AddImageIcon';
export * from './CircleDotIcon';
export * from './DeleteIcon';
export * from './RecordIcon';
export * from './SendIcon';
export * from './TrashIcon';
export * from './XIcon';
//# sourceMappingURL=index.d.ts.map
2 changes: 1 addition & 1 deletion dist/components/icons/index.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion dist/components/index.d.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
export * from './SendButton';
export * from './buttons/SendButton';
export * from './TypingBubble';
//# sourceMappingURL=index.d.ts.map
2 changes: 1 addition & 1 deletion dist/components/index.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions dist/components/inputs/textInput/components/TextInput.d.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { Setter } from 'solid-js';
import { FileEvent, UploadsConfig } from '@/components/Bot';
type Props = {
placeholder?: string;
backgroundColor?: string;
Expand All @@ -7,6 +9,10 @@ type Props = {
fontSize?: number;
disabled?: boolean;
onSubmit: (value: string) => void;
uploadsConfig?: Partial<UploadsConfig>;
setPreviews: Setter<unknown[]>;
onMicrophoneClicked: () => void;
handleFileChange: (event: FileEvent<HTMLInputElement>) => void;
};
export declare const TextInput: (props: Props) => import("solid-js").JSX.Element;
export {};
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion dist/queries/sendMessageQuery.d.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import { MessageType } from '@/components/Bot';
import { FileUpload, MessageType } from '@/components/Bot';
export type IncomingInput = {
question: string;
history: MessageType[];
uploads?: FileUpload[];
overrideConfig?: Record<string, unknown>;
socketIOClientId?: string;
chatId?: string;
Expand Down
Loading