Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka topic names are wrong, likely memory re-use issue #1928

Open
skandragon opened this issue Mar 4, 2025 · 2 comments · May be fixed by #1936
Open

Kafka topic names are wrong, likely memory re-use issue #1928

skandragon opened this issue Mar 4, 2025 · 2 comments · May be fixed by #1936
Assignees
Labels
bug Something isn't working

Comments

@skandragon
Copy link

Describe the bug

Kafka topic names look mangled, possible buffer / memory re-use issue

We publish on these four topics:

telemetry-stats-metrics
telemetry-stats-logs
telemetry-stats-spans
graph-db-updates

We also consume from the graph-db-updates topic in the same container.

However, we are seeing these topic names in the traces exported via auto-instrumentation:

graph-db-updateslogsics
graph-db-updatesmetrics
graph-db-updateslogs
graph-db-updateslogsics
graph-db-updatesmetrics
telemetry-stats-logsics

From my C coding days, I suspect this is copying into a buffer that already has content, or some other sort of improper re-use of memory. It looks like overall the buffer may start with something and then the end of that name sticks around forever.

Environment

  • OS: Kubernetes (linux)
  • Go Version: 1.23.x and 1.24.x
  • Version: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.21.0
  • Using Kafka mod: github.com/segmentio/kafka-go v0.4.47

To Reproduce

I'm not sure how to reproduce this behavior, as I have not nailed down the smallest example that will cause this problem to occur.

Expected behavior

Topic names should match what I am using.

Additional context

None.

@skandragon skandragon added the bug Something isn't working label Mar 4, 2025
@RonFed RonFed self-assigned this Mar 4, 2025
@skandragon
Copy link
Author

I took a look at the probe code and can't find anything obviously incorrect, but I have never touched this sort of thing before. It looks like this is accessing a Go string and that should have a defined length, but I am not sure if the problem is in the copying of the topic name, or the re-use of the buffer it is copied into. I suspect the latter.

If I had to guess, I think get_go_string_from_user_ptr() copies a Go string into a C character array that will start out null terminated, so the copy will work properly initially, but subsequent copies will just make the string overwrite.

I think this may expose two issues, all of which are so common in C... Not adding a null byte to terminate a C string, and two, not leaving room for it in the buffer, so it may be possible to extract more than you should when reading this C string.

@skandragon
Copy link
Author

diff --git a/internal/include/go_types.h b/internal/include/go_types.h
index 1bf386b9..f659790c 100644
--- a/internal/include/go_types.h
+++ b/internal/include/go_types.h
@@ -181,6 +181,10 @@ static __always_inline bool get_go_string_from_user_ptr(void *user_str_ptr, char
         return false;
     }

+    if size_to_read < max_len {
+        dst[size_to_read] = '\0';
+    }
+
     return true;
 }
 #endif

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants