What are the conditions for a service graph virtual node to exist?

[ogxd]

Hello!

In the Tempo Service Graph documentation, it is said that

Virtual nodes can be detected [...] A client span does not have its matching server span, but has a peer attribute present [...] The default peer attributes are peer.service, db.name and db.system.

I do have many client spans with peer.service set, however I can't see any corresponding node in the service graph.

I do however have nodes for SQL server for which I have enabled automatic instrumentation, which sets peer.service, db.system and db.name. I notice that the node name corresponds to the db.name here.

Do you know why I have an "AdDiffusionRun" node but no "TensorFlowServing" node in my service graph? It's not a database so I don't want to set db.name.

[ogxd]

Ah I think I found the answer myself: I am using the opentelemetry-contrib servicegraphconnector and the attributes are different than the ones on the tempo documentation.

virtual_node_peer_attributes: the list of attributes need to match for building virtual server node, the higher the front, the higher the priority.
Default: [db.name, net.sock.peer.addr, net.peer.name, rpc.service, net.sock.peer.name, net.peer.name, http.url, http.target]

[ogxd]

According to latest semantics version (1.24) these attributes are now obsolete:

• net.peer.name (replaced by server.address on client spans and client.address on server spans)
• net.sock.peer.name (removed)
• net.sock.peer.addr (replaced by network.peer.address)
• http.url (replaced by url.full)
• http.target (split to url.path and url.query)

Only db.name and rpc.service are still "valid".

On the tempo side, peer.service, db.name and db.system are still compliant with conventions, so I think it would make sense to change the defaults of the connector to this.

[ogxd]

Regarding the virtual nodes, I have experimented with virtual_node_peer_attributes and I still couldn't get virtual nodes to appear. For instance if I set it to virtual_node_peer_attributes: [peer.service], I can't see any additional client or server labels for this generated metric:
sum by (server) (rate(traces_service_graph_request_total[5m]))

Maybe I misunderstood how this is supposed to work?

[ogxd]

Ok I think I found out, virtual nodes are under a alpha feature gate (disabled by default).

[ogxd]

I created an issue: open-telemetry/opentelemetry-collector-contrib#31734

[joe-elliott]

Thanks for following up on this!