Skip to content

Conversation

@enyst
Copy link
Collaborator

@enyst enyst commented Nov 8, 2025

Summary

  • Fix telemetry JSON logging to avoid circular reference warnings.
  • Add helper example to compare assistant content between telemetry logs and persisted events and produce a Markdown report of cases where either side has empty content.

Why

  • During investigation of Gemini tool-call behavior, we saw frequent telemetry logging warnings due to circular references in payloads. This reduces reliability of collecting raw responses.
  • The helper script streamlines reproducing and inspecting cases where assistant content is empty on either side, which is valuable for debugging providers returning reasoning/tool_calls with empty content.

Changes

  • openhands-sdk/openhands/sdk/llm/utils/telemetry.py
    • Improve _safe_json: use BaseModel.model_dump(exclude_none=True) for any Pydantic models, handle sets and bytes, avoid relying on dict fallback that may carry circular references.
  • examples/01_standalone_sdk/27_compare_content_vs_telemetry.py
    • New example: runs a short conversation with persistence + telemetry enabled, scans recent telemetry and assistant MessageEvents, and writes a Markdown report listing pairs where content is empty in telemetry or events (includes embedded JSON for the telemetry response and the event).

Testing

  • pre-commit (ruff/pyright/pycodestyle) passes for both files.
  • Manual run with gemini-2.5-pro (10_persistence) confirms telemetry files now write cleanly without circular reference errors.

Backward compatibility

  • No breaking API changes; improvements are internal and an additional example.

Closes

  • Part of the investigation for Issue Bug: Wrong ConversationExecutionStatus? #1094; this PR adds the helper and addresses telemetry robustness. The empty-content behavior for tool-call turns is expected; the helper helps surface any counterexamples.

Co-authored-by: openhands openhands@all-hands.dev

@enyst can click here to continue refining the PR


Agent Server images for this PR

GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python amd64, arm64 nikolaik/python-nodejs:python3.12-nodejs22 Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:6b57876-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-6b57876-python \
  ghcr.io/openhands/agent-server:6b57876-python

All tags pushed for this build

ghcr.io/openhands/agent-server:6b57876-golang-amd64
ghcr.io/openhands/agent-server:6b57876-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:6b57876-golang-arm64
ghcr.io/openhands/agent-server:6b57876-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:6b57876-java-amd64
ghcr.io/openhands/agent-server:6b57876-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:6b57876-java-arm64
ghcr.io/openhands/agent-server:6b57876-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:6b57876-python-amd64
ghcr.io/openhands/agent-server:6b57876-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:6b57876-python-arm64
ghcr.io/openhands/agent-server:6b57876-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:6b57876-golang
ghcr.io/openhands/agent-server:6b57876-java
ghcr.io/openhands/agent-server:6b57876-python

About Multi-Architecture Support

  • Each variant tag (e.g., 6b57876-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 6b57876-python-amd64) are also available if needed

enyst and others added 4 commits November 8, 2025 03:55
- Use BaseModel.model_dump(exclude_none=True) for any Pydantic models
- Handle sets and bytes explicitly
- Avoid relying on __dict__ which can include circular refs or runtime-only fields

This removes frequent 'Telemetry logging failed: Circular reference detected' warnings during logging LLM requests/responses.

Co-authored-by: openhands <openhands@all-hands.dev>
… events

- Runs a short conversation with persistence + telemetry enabled
- Scans recent telemetry logs and assistant MessageEvents
- Writes a Markdown report listing pairs where content is empty on either side,
  with embedded JSON for the telemetry response and the event

Co-authored-by: openhands <openhands@all-hands.dev>
- Group MessageEvent and ActionEvent by llm_response_id
- Consider ActionEvent.thought as assistant-side content for tool-call turns
- Report per-response, embedding telemetry and event JSONs

Co-authored-by: openhands <openhands@all-hands.dev>
- Remove tool usage to force text-only turns
- Group by llm_response_id, consider MessageEvent only
- Include reasoning_content presence/preview; drop Observation/ActionEvent reporting
- Report only cases where content is empty but reasoning present on either side

Co-authored-by: openhands <openhands@all-hands.dev>
@openhands-ai
Copy link

openhands-ai bot commented Nov 8, 2025

Looks like there are a few issues preventing this PR from being merged!

  • GitHub Actions are failing:
    • Check Documented Examples

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #1110 at branch `openhands/telemetry-safe-json`

Feel free to include any additional details that might help me get this PR into a better state.

You can manage your notification settings

@enyst enyst closed this Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants