Pretty Waterfalls in Honeycomb
When first setting up the Absinthe app I work on, someone added OpenCensus Honeycomb package. The way it was set up was very basic: each top-level query had tracing attached.
:object queries do field :getStuff, :stuff do meta :trace, true arg :input, :string resolve &Resolver.stuff_resolver/3 end end
This got us information on how long each query took, so we could see which query needed to be optimized and do some debugging when performance issues hit. But it didn’t drill down like I wanted.
Making it better
The first time I tried to solve this, I looked at how telemetry works in the BEAM, and I didn’t want to do a bunch of manual stuff at the start and end of every function. Then I found Telemetry Decorator and thought I had a handy little solution. Nope. Turns out, BEAM telemetry and OpenCensus are different systems that maybe don’t talk to each other. Or at least, they don’t do automagic.
More recently, I dug into it again. This time, I looked at the OpenCensus documentation on tracing and found out about
with_child_span do ... end
(Sidebar: I love that they give both Erlang and Elixir examples!)
So, I tried adding that wrapper to a bunch of functions. I got traces for them! …but they weren’t attached to their queries. So then I tried adding
with_child_span to resolver functions, and that was a big mistake. That doesn’t compile; don’t do that.
What was I missing?
Diving into the Absinthe-specific documentation, I spotted this:
Until Absinthe merge and publish their telemetry support (see below) and you upgrade, you’ll also need to set :trace in the metadata for any field for which you want tracing to happen
“Any field.” Not “any query.” Field. Let’s try it.
object :stuff do field :id, non_null(:id) field :thing, :string field :percentage, :integer, resolve: &Resolver.stuff_percentage/3 end
object :stuff do field :id, non_null(:id) field :thing, :string field :percentage, :integer, meta: [trace: true] resolve: &Resolver.stuff_percentage/3 end
And we got our pretty waterfall graphs showing up in Honeycomb. They still break anywhere you use
Async functions. I suspect getting around that would involve a closure, grabbing the span from the parent function then passing it as an argument to the
Async-wrapped function, which would then manually set up its context.
Three things to remember when using OpenCensus in an Absinthe app:
meta: [trace: true]to anything that has a resolver
- wrap methods in
- don’t wrap the resolver functions