LLM input and output is not capture by LangfuseConnector

[himadri-bhattacharjee]

Describe the bug
LLM input and output is not capture by LangfuseConnector.
I run the sample example given in https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/langfuse. langfuse trace is created but input and output of trace is set as null.

i have capture the REST API call.
`
HTTP Method: POST
Headers:
Host: ***************
Accept: /
Accept-Encoding: gzip, deflate, zstd
Connection: keep-alive
User-Agent: python-httpx/0.28.1
Authorization: Basic ***************==
Content-Type: application/json
x_langfuse_sdk_name: python
x_langfuse_sdk_version: 2.57.13a0
x_langfuse_public_key: ***************
Content-Length: 5355

Cookies: No Cookies
Body:
{"batch": [{"id": "7c6c7b22-189a-48cb-99f6-61e00409a754", "type": "trace-create", "body": {"id": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "timestamp": "2025-02-21T11:55:50.666134Z", "name": "my-llm-langfuse", "public": false}, "timestamp": "2025-02-21T11:55:50.666134Z"}, {"id": "cd1c1f25-06a3-488c-91e4-65e96458c1f9", "type": "trace-create", "body": {"id": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.pipeline.input_data": "{"prompt_builder": {"template_variables": {"location": "Berlin"}, "template": [{"_role": "system", "_meta": {}, "_name": null, "_content": [{"text": "Always respond in German even if some input data is in other languages."}]}, {"_role": "user", "_meta": {}, "_name": null, "_content": [{"text": "Tell me about {{location}}"}]}]}}"}}, "timestamp": "2025-02-21T11:55:50.667128Z"}, {"id": "9c3443df-5776-4cd7-acf1-2896a02e4144", "type": "trace-create", "body": {"id": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.pipeline.output_data": "{}"}}, "timestamp": "2025-02-21T11:55:50.667128Z"}, {"id": "1346ca92-f360-453e-af92-4b70e585c835", "type": "trace-create", "body": {"id": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.pipeline.metadata": "{}"}}, "timestamp": "2025-02-21T11:55:50.667128Z"}, {"id": "2bf8ea3c-7bdd-4eca-bccb-f58e1ef4d5c6", "type": "trace-create", "body": {"id": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.pipeline.max_runs_per_component": 100}}, "timestamp": "2025-02-21T11:55:50.667128Z"}, {"id": "c36e7cbb-0830-48fd-a94e-2af01a7fdaf3", "type": "span-create", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "name": "prompt_builder", "startTime": "2025-02-21T11:55:50.667323Z", "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.667323Z"}, {"id": "577a6d23-129a-4fd3-b6c6-d25de9c90cb8", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.name": "prompt_builder"}, "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.667323Z"}, {"id": "97deefa0-b2e9-47de-90b9-1da27dc8ba25", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.type": "ChatPromptBuilder"}, "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.667323Z"}, {"id": "ec7e4bea-7934-4de9-8542-bc22dbff8775", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.input_types": "{"template": "list", "template_variables": "dict"}"}, "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.667323Z"}, {"id": "90717703-599d-4c30-9e7e-5294c7dd2eda", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.input_spec": "{"template": {"type": "typing.Optional[typing.List[haystack.dataclasses.chat_message.ChatMessage]]", "senders": []}, "template_variables": {"type": "typing.Optional[typing.Dict[str, typing.Any]]", "senders": []}}"}, "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.667323Z"}, {"id": "ba578ec9-c8b7-4747-bd71-65f865118fb2", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.output_spec": "{"prompt": {"type": "typing.List[haystack.dataclasses.chat_message.ChatMessage]", "receivers": ["llm"]}}"}, "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.667323Z"}, {"id": "ffb65e41-3434-4116-b512-593a6258f9af", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "input": {"template": [{"_role": "system", "_content": [{"text": "Always respond in German even if some input data is in other languages."}], "_name": null, "_meta": {}}, {"_role": "user", "_content": [{"text": "Tell me about {{location}}"}], "_name": null, "_meta": {}}], "template_variables": {"location": "Berlin"}}, "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.667323Z"}, {"id": "b3decd86-577b-41c9-8e81-b08d86ae49c6", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.visits": 1}, "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.669461Z"}, {"id": "908c2313-f45e-4cfe-90d9-b73740d957d0", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "output": {"prompt": [{"_role": "system", "_content": [{"text": "Always respond in German even if some input data is in other languages."}], "_name": null, "_meta": {}}, {"_role": "user", "_content": [{"text": "Tell me about Berlin"}], "_name": null, "_meta": {}}]}, "id": "3f158d7f-0738-4900-a932-7acecd39ee96"}, "timestamp": "2025-02-21T11:55:50.669461Z"}, {"id": "abff525a-d5a9-4ffd-851a-04cc1f56a74d", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "id": "3f158d7f-0738-4900-a932-7acecd39ee96", "endTime": "2025-02-21T11:55:50.669461Z"}, "timestamp": "2025-02-21T11:55:50.669461Z"}], "metadata": {"batch_size": 15, "sdk_integration": "default", "sdk_name": "python", "sdk_version": "2.57.13a0", "public_key": "***************"}}
127.0.0.1 - - [21/Feb/2025 17:25:52] "POST /api/public/ingestion HTTP/1.1" 200 -
HTTP Method: GET
Headers:
Host: ***************
Accept: /
Accept-Encoding: gzip, deflate, zstd
Connection: keep-alive
User-Agent: python-httpx/0.28.1
X-Fern-Language: Python
Authorization: Basic ***************==
X-Langfuse-Sdk-Name: python
X-Langfuse-Sdk-Version: 2.57.13a0
X-Langfuse-Public-Key: ***************

Cookies: No Cookies
Body:

127.0.0.1 - - [21/Feb/2025 17:25:54] "GET /api/public/projects HTTP/1.1" 200 -
HTTP Method: POST
Headers:
Host: ***************
Accept: /
Accept-Encoding: gzip, deflate, zstd
Connection: keep-alive
User-Agent: python-httpx/0.28.1
Authorization: Basic ***************==
Content-Type: application/json
x_langfuse_sdk_name: python
x_langfuse_sdk_version: 2.57.13a0
x_langfuse_public_key: ***************
Content-Length: 2352

Cookies: No Cookies
Body:
{"batch": [{"id": "33151550-fbb6-4c4c-a075-38ab6a95c14d", "type": "span-create", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "name": "tracer", "startTime": "2025-02-21T11:55:52.737410Z", "id": "c97a810c-47ab-4447-b984-623bf55ee0f3"}, "timestamp": "2025-02-21T11:55:52.737410Z"}, {"id": "46a43e0c-878b-418c-a168-a9eb6ce67979", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.name": "tracer"}, "id": "c97a810c-47ab-4447-b984-623bf55ee0f3"}, "timestamp": "2025-02-21T11:55:52.737410Z"}, {"id": "a32d1418-65bb-4e13-815f-36d48d3143a8", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.type": "LangfuseConnector"}, "id": "c97a810c-47ab-4447-b984-623bf55ee0f3"}, "timestamp": "2025-02-21T11:55:52.737410Z"}, {"id": "3079e0d6-e080-41c8-9344-3686eaf6eadc", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.input_types": "{"invocation_context": "NoneType"}"}, "id": "c97a810c-47ab-4447-b984-623bf55ee0f3"}, "timestamp": "2025-02-21T11:55:52.737410Z"}, {"id": "a0f2f58e-9a1a-4375-b035-600632935dff", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.input_spec": "{"invocation_context": {"type": "typing.Optional[typing.Dict[str, typing.Any]]", "senders": []}}"}, "id": "c97a810c-47ab-4447-b984-623bf55ee0f3"}, "timestamp": "2025-02-21T11:55:52.737410Z"}, {"id": "445bdec9-dd53-4b98-8abf-274a5b066736", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "metadata": {"haystack.component.output_spec": "{"name": {"type": "str", "receivers": []}, "trace_url": {"type": "str", "receivers": []}}"}, "id": "c97a810c-47ab-4447-b984-623bf55ee0f3"}, "timestamp": "2025-02-21T11:55:52.737410Z"}, {"id": "9c181e1e-6b5c-47a8-8d2a-b85d161d7f92", "type": "span-update", "body": {"traceId": "e6d85bd3-1182-4d2d-b210-1ff0373fe27f", "input": {"invocation_context": null}, "id": "c97a810c-47ab-4447-b984-623bf55ee0f3"}, "timestamp": "2025-02-21T11:55:52.737410Z"}], "metadata": {"batch_size": 7, "sdk_integration": "default", "sdk_name": "python", "sdk_version": "2.57.13a0", "public_key": "***************"}}
127.0.0.1 - - [21/Feb/2025 17:25:55] "POST /api/public/ingestion HTTP/1.1" 200 -
`

Describe your environment (please complete the following information):

• Haystack version: 2.10.1
• langfuse-haystack: 0.8.0

[himadri-bhattacharjee]

[henryclw]

@himadri-bhattacharjee @indranilr @julian-risch
I experienced the same issue today. But later I found out my tracing content is not correctly set.
Traces will not be logged to Langfuse because Haystack tracing is disabled. To enable, set the HAYSTACK_CONTENT_TRACING_ENABLED environment variable to true before importing Haystack.
After following the guide on https://docs.haystack.deepset.ai/v2.4/docs/tracing#content-tracing , I was able to capture the input and output of the llm.

Would you mind confirming that the content tracing is set to true when you experience this problem? Thank you, hope this would solve the issue.

[himadri-bhattacharjee]

sample_code.txt

here is my sample code , haystack content tracing is enabled ,
@henryclw can you please share the python package details ?
i am using Haystack version: 2.10.1, langfuse-haystack: 0.8.0

[henryclw]

I'm using:

haystack-ai==2.10.3
haystack-experimental==0.7.0
langfuse==2.59.7
langfuse-haystack==0.8.0

I tried you sample code, things just worked. The input/output of the trace is null, while the span of llm, the span of prompt_builder have valid input/output.

Would you mind providing your langfuse trace url to confirm if every input/output is null or just the input/output of the trace is null?

[himadri-bhattacharjee]

@henryclw is this the expected behavior " input/output of the trace is null and while the span of llm, the span of prompt_builder have valid input/output" ?

i can find input and output under llm , but another problem it's not able to compute the cost since i am using a model called "se-gpt-4o" , is there any way to provide/register cost of custom model ?

[henryclw]

@himadri-bhattacharjee Thank you for your reply.

is this the expected behavior " input/output of the trace is null and while the span of llm, the span of prompt_builder have valid input/output" ?

Great question. I'm not sure. Maybe it better to ask the developers of haystack @julian-risch . As far as I know as a haystack user, the pipeline do have the input, like the data field we inputted when calling pipe.run. As for the output, the pipeline output is not very clear. However there are outputs from each components.

i can find input and output under llm , but another problem it's not able to compute the cost since i am using a model called "se-gpt-4o" , is there any way to provide/register cost of custom model ?

The haystack integration logs everything as a span, even with LLM generation, which should use generation instead of span. I think for the time being if you really want to keep track of the cost of API, you either need to
A. create a temporary ugly wrapper to log as generation manually
or
b. use other method like a sperate LLM API key for each project only. Then check the API cost from the API provider side.

And I just opened issue #1475 and #1476 to address these issues.