Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]Graph save/load, updating oneflow version got error: user_op_conf.cpp:87] Check failed: attr.get() != nullptr attr_name: query_head_size #153

Closed
strint opened this issue Mar 13, 2023 · 1 comment

Comments

@strint
Copy link
Collaborator

strint commented Mar 13, 2023

Using oneflow SD with graph save/load, updating oneflow version got error: user_op_conf.cpp:87] Check failed: attr.get() != nullptr attr_name: query_head_size

oneflow-0.9.1.dev20230216+cu117 works, updating to oneflow-0.9.1.dev20230312+cu117 and oneflow-0.9.1.dev20230309+cu117 will raise Error

F20230313 10:33:57.451692 1326001 user_op_conf.cpp:87] Check failed: attr.get() != nullptr attr_name: query_head_size
*** Check failure stack trace: ***
@ 0x7fe57e7a7c9a google::LogMessage::Fail()
@ 0x7fe57e7aabd1 google::LogMessage::SendToLog()
@ 0x7fe57e7a77c9 google::LogMessage::Flush()
@ 0x7fe57e7ab4b9 google::LogMessageFatal::~LogMessageFatal()
@ 0x7fe576a967a9 oneflow::user_op::UserOpConfWrapper::Attr4Name()
@ 0x7fe57866c50e oneflow::user_op::(anonymous namespace)::FusedMultiHeadAttentionInferenceKernel::Compute()
@ 0x7fe5776572f7 oneflow::UserKernel::ForwardUserKernel()
@ 0x7fe5776574bb oneflow::UserKernel::ForwardDataContent()
@ 0x7fe577623f63 oneflow::Kernel::Forward()
@ 0x7fe577624069 oneflow::Kernel::Launch()
@ 0x7fe577772e58 oneflow::(anonymous namespace)::LightActor<>::ProcessMsg()
@ 0x7fe577d4d62f oneflow::Thread::PollMsgChannel()
@ 0x7fe577d4dcc1 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN7oneflow6ThreadC4ERKNS3_8StreamIdEEUlvE_EEEEE6_M_runEv
@ 0x7fe6b3ff9de4 (unknown)
@ 0x7fe72701e609 start_thread
@ 0x7fe726f43133 clone

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Reference: https://github.com/Oneflow-Inc/diffusers/issues/93#issuecomment-1465422388

@strint strint changed the title GraphUpdate oneflow got [Bug]Graph save/load, updating oneflow version got error: user_op_conf.cpp:87] Check failed: attr.get() != nullptr attr_name: query_head_size Mar 13, 2023
@strint
Copy link
Collaborator Author

strint commented Mar 13, 2023

It turns out that the graphs are saved with oneflow-0.9.1.dev20230216+cu117 , but load with newer versions of oneflow.

The newer versions of oneflow has updated some Operator Config, and Operater Config is part of the saved Graph.

So the newer versions oneflow cannot load the old saved graph. Please compile and save graphs with the new version oneflow, then you can load the new graph.

Reference: https://github.com/Oneflow-Inc/diffusers/issues/93#issuecomment-1465471787

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant