Contents

Pretty Waterfalls in Honeycomb

Contents

When first setting up the Absinthe app I work on, someone added OpenCensus Honeycomb package. The way it was set up was very basic: each top-level query had tracing attached.

:object queries do
  field :getStuff, :stuff do
    meta :trace, true

    arg :input, :string
    resolve &Resolver.stuff_resolver/3
  end
end

This got us information on how long each query took, so we could see which query needed to be optimized and do some debugging when performance issues hit. But it didn’t drill down like I wanted.

Making it better

The first time I tried to solve this, I looked at how telemetry works in the BEAM, and I didn’t want to do a bunch of manual stuff at the start and end of every function. Then I found Telemetry Decorator and thought I had a handy little solution. Nope. Turns out, BEAM telemetry and OpenCensus are different systems that maybe don’t talk to each other. Or at least, they don’t do automagic.

More recently, I dug into it again. This time, I looked at the OpenCensus documentation on tracing and found out about

with_child_span do
  ...
end

(Sidebar: I love that they give both Erlang and Elixir examples!)

So, I tried adding that wrapper to a bunch of functions. I got traces for them! …but they weren’t attached to their queries. So then I tried adding with_child_span to resolver functions, and that was a big mistake. That doesn’t compile; don’t do that.

What was I missing?

Diving into the Absinthe-specific documentation, I spotted this:

Until Absinthe merge and publish their telemetry support (see below) and you upgrade, you’ll also need to set :trace in the metadata for any field for which you want tracing to happen

“Any field.” Not “any query.” Field. Let’s try it.

object :stuff do
  field :id, non_null(:id)
  field :thing, :string
  field :percentage, :integer,
    resolve: &Resolver.stuff_percentage/3
end

became

object :stuff do
  field :id, non_null(:id)
  field :thing, :string
  field :percentage, :integer,
    meta: [trace: true]
    resolve: &Resolver.stuff_percentage/3
end

And we got our pretty waterfall graphs showing up in Honeycomb. They still break anywhere you use Async functions. I suspect getting around that would involve a closure, grabbing the span from the parent function then passing it as an argument to the Async-wrapped function, which would then manually set up its context.

TL;DR

Three things to remember when using OpenCensus in an Absinthe app:

  1. add meta: [trace: true] to anything that has a resolver
  2. wrap methods in with_child_span
  3. don’t wrap the resolver functions