Verily Logo

View all solutions

Data Capabilities

Data Catalog

Workbench

Platform

For Health Plans

For Employers

Lightpath

Digital Biomarkers

Retinal Service

Numetric

Digital Protocols

Evidence

Site CTMS

Viewpoint

Wastewater

Sightline

Solutions

A documentary series that highlights the promise of precision health

N of 1 series image

N of 1 series

View all perspectives

Precision health blog

Tech blog

Publications

Press releases

Webinars

Perspectives

Bringing the promise of precision health to everyone, every day.

About us card image

Our story

Careers

News

Leadership

Contact us

About us

Verily

Welcome to Verily’s tech blog — a look under the hood at product engineering. Read about our key learnings, thought processes and capabilities that enable Verily’s platform. We’re making healthcare data AI-ready, along with powering other innovations to realize the promise of precision health.

observing in microscope

Our platform

person in lab coat transferring samples

default

Welcome to Verily’s tech blog — a look under the hood at product engineering for Verily’s platform.

Blog | Tech Blog

Implementing FHIRPath in Go: How we tackled challenges 

In our previous post of Verily’s Tech Blog, we walked through the motivation and design principles behind building our FHIRPath expression engine in Go. In this followup, we’re delving into some of the trickier technical details of the implementation — and the thoughtful (and sometimes, clever) optimizations we made to improve performance and developer ergonomics.


 library is a native Go implementation of the 

 R4 Go bindings. This enables us to run high-volume evaluations on FHIR data with minimal latency, directly within Verily’s 

At Verily, we deal with A LOT of FHIR data.

 Our engines routinely see millions of events per minute, and we use FHIRPath to determine which events should be routed to specific functionalities. Given this volume, we need evaluations to be as fast as possible to minimize latency.


Based on experience when performing these queries and filters, we know we’ve reused the same FHIRPath expressions many times against large quantities of different resources. Naturally, this made the case that we needed to optimize runtime over the compile-time. Why? Because even if the compilation takes a long time, it’s generally only needed once, whereas the evaluation can occur multiple times. 


Therefore, we designed our interface with this in mind, following the pattern already laid out in the built-in 

func Compile(expr string, options ...CompileOption) (*Expression, error) 

func (e *Expression) Evaluate(input []fhir.Resource, options ...EvaluateOption) (system.Collection, error)

This divide in the API enables the heavier parsing logic to be done once in the Compile(...) step, leaving only executable Expression objects that can be Evaluated as needed.

that defines the FHIRPath language is written using a recursive definition, where expression is defined in terms of additional expression objects. Simplified, it looks something like:

        : identifier                                            

When parsing expressions such as ‘Patient.name.given’, this produces a parse-tree similar to this:

With a depth-first traversal of this tree producing an Expression at each node, it will produce an expression-tree that looks something like this:

Notice that the deepest expression is evaluated first, but requires the result from the parent expression before it is able to execute. For longer expressions, this depth scales linearly for each invocation. This becomes costly in memory overhead due to each recursive call increasing the stack-usage, and can also increase the overall execution runtime due to the increased amount of indirection.

We improved this runtime cost by flattening the indirection to be a contiguous collection of Expression objects, which changes expressions that would otherwise have been:

func (e *FieldExpression) Evaluate(ctx *Context, input system.Collection) (system.Collection, error) {

    input, err = e.Parent.Evaluate(ctx, input)

Into expressions that are self-contained, and don’t need to be aware of their ancestors:

We were able to do this by producing an ExpressionSequence, which is an Expression in itself. The ExpressionSequence is just a simple wrapper around []Expression which evaluates the sub-expression, and passes the result from the previous evaluation directly into the result from the next expression:

func (s *ExpressionSequence) Evaluate(ctx *Context, input system.Collection) (system.Collection, error) {

    result, err := expr.Evaluate(ctx, output)

As we visit each node in the parse-tree, we produce an Expression, and each Expression gets joined into the ExpressionSequence to produce a contiguous series of expressions.

Doing this, we were able to avoid the deep call stack, and unwrap much of the indirection into a single sequence!

The FHIRPath specification requires that the root-node of expressions may optionally be the name of a 

. If it matches a valid type, it should filter the expression if the input is also of the named resource. Otherwise, it treats the input as a field to be filtered. This optional prefix makes paths more readable to humans that may not be as well-versed in the computer syntax. This process may appear weird to a developer, but it’s intuitive and readable for project managers. 


To achieve this efficiently, processes need to quickly determine if a given string is a valid FHIR resource type. To avoid repeated lookups and improve performance, we employ a simple caching mechanism. Specifically, we store all valid FHIR resource names in a hash-map keyed off the string names.

Upon initialization of the fhirpath-go library, the map is populated with all known FHIR resource type names. This initialization only occurs once on startup. When a FHIRPath expression is evaluated and a resource name check is required, we simply perform a lookup in the map.

If we didn’t produce this map, we would have to use some other mechanism, such as protoreflect, to inspect the resource descriptors defined in the google/fhir repository, being far more computationally intensive to walk through. A map, on the other hand, effectively provides constant-time lookups to verify if a string identifier is a known/valid resource name!

Codes were an interesting challenge when working with the Google FHIR library, as they are represented as enum values within the protobuf definitions, and wrapped in a message which defines value, extension, and id fields.

In the FHIRPath specification, codes are represented only by string entries; there is no high-level type, and extension and id fields are handled as underscore-prefixed sibling fields. A consequence of this is that parsing expressions like Patient.telecom.where(system = ‘fax’) becomes interesting: the Patient.Telecom[i].System is a strong type of datatypes_go_proto.ContactPoint_SystemCode, which is a struct containing an enum, but the ’fax’ value is a string.

To handle this, we need to normalize the fields into a common type. So, we need to normalize values like this into a proper string type first, and we can do this through the protoreflect API. Additionally, the enum values are all defined in SHOUT_CASE notation, necessitating the massage of this data to fit FHIR’s convention, which is in kebab-case.

Simplified, the logic that handles this looks something like: 

field := rv.Descriptor().Fields().ByName(protoreflect.Name("value"))

code := string(field.Enum().Values().ByNumber(enum).Name())

By properly normalizing and transforming code representations, we ensured seamless interoperability between FHIRPath and the Google FHIR library — bridging the gap between the different data models.

Additionally, we encountered a challenge with the google/fhir repository, and how the protobuf descriptors model hierarchical data.

In FHIR, Resources are hierarchically defined. For R4, a Questionnaire

a DomainResource which is, itself, a Resource. However, the protocol-buffer format has no real notion of "inheritance". It works around this by doing something a little hacky, which is defining this concept of a

The ContainedResource

the Resource, but effectively defined as a massive oneof that enumerates

. In terms of data-modeling, this makes great sense, and is easy to work with. However, for working with FHIRPath, it’s a chore.


For example, consider the FHIRPath expression: Bundle.entry.resource.name. As we extract each independent field from the previous collection, you would expect .resource to return a Resource, which you can then just grab the next name field from. However, this isn’t the case.

Instead, we get the fated ContainedResource. This generated go-type wrapper isn’t just one type, but rather multiple types. Cleaned up, it looks like:

    OneofResource isContainedResource_OneofResource

Where isContainedResource_OneofResource is an interface that is satisfied by

, in a wrapped struct object:

type isContainedResource_OneofResource interface {

type ContainedResource_ActivityDefinition struct {

  ActivityDefinition *activity_definition_go_proto.ActivityDefinition

The real Resource we want is the wrapped type inside the struct. So how are we supposed to get it out so that we can get the name field from the resource field?

You might want to reach for the naive approach of manually doing a switch statement, like:

    return ContainedResource_Account.Account

  case (*ContainedResource_ActivityDefinition):

    return ContainedResource_ActivityDefinition.ActivityDefinition

However, it turns out that the protoreflect API here saves the day through some reflection witchcraft. Since protobufs know that only one of the fields can be active for a oneof, we can query the protobuf descriptor using reflection to find exactly which resource is the one that’s stored.

of the oneof field by querying the field by name, which is just oneof_resource in the protobuf definition. This function looks something like:

func getContainedResourceOneOfField(cr *bcrpb.ContainedResource) protoreflect.FieldDescriptor {

  oneof := descriptor.Oneofs().ByName("oneof_resource")

Finally, we simply unwrap the ContainedResource, which is now possible by leveraging that FieldDescriptor:

func Unwrap(cr *bcrpb.ContainedResource) fhir.Resource {

  field := getContainedResourceOneOfField(cr)

  // If field is nil, it means we have no contained resource value set -- so

  // the only valid value to return while unwrapping is `nil`.

  return message.Interface().(fhir.Resource)

The message.Interface() converts this back into the original type, which we then assert back into the fhir.Resource interface to model all of our resources.

Using this wizardry, we can simply Unwrap(...) the ContainedResource and pass it off in the expression so that the next subexpression can extract the fields off of it, exactly as one would expect.

In a corporate job, working with Grammars with a need to focus on detailed optimizations doesn’t happen often. Implementing a FHIRPath engine in Go presented us with unique challenges, and offered opportunities to flex our performance and efficiency muscles. We hope the first three posts of Verily’s Tech Blog, which explored our implementation process, provide insight for developers working with FHIR and Go. We’ll keep improving and expanding the capabilities of 

. Please dig into the library and contribute to its development!



Lastly, keep an eye out — more deep dives into Verily’s 

Why Verily uses FHIRPath

How we implemented FHIRPath in Go

Explore the challenges Verily platform engineers faced building FHIRPath in Go—and how smart fixes made it faster, cleaner, and easier for developers to use.

Tech Blog #3: FHIRPath Implementation Series

Banner image

, we gave the rationale behind why we use FHIRPath, which provides a smarter way to navigate and collaborate on healthcare data. But getting it to work in code isn’t exactly plug-and-play. At Verily, our first attempt relied on regular expressions and quick fixes. It got messy and labor-intensive. So, we stepped back and did it better by utilizing ANTLR, a tool that turns a language’s grammar into working code. 

And now, we’re sharing this journey of product engineering enlightenment, and how in the end, we made FHIRPath parsing cleaner, faster, and much easier to extend.

: Reading this might give you scary flashbacks about your compilers class. (Just kidding — we hope!) 

FHIRPath is a programming language that can be represented by a 

. When developing FHIRPath grammar, our first challenge was implementing a compiler to convert the FHIRPath syntax into objects that our program understood how to process. Compilers consist of lexers and parsers, which are both programs that process some alphabet and match it to an understandable grammar. 

Lexers understand regular grammar. They process symbols from the input and produce tokens. On the other hand, parsers process these strings of tokens to match a context-free grammar, and produce a parse tree. Here’s one way to think of the difference:

Actually, this is our second attempt at implementing the FHIRPath grammar. First, we tried to use naive regular expressions and string operations to implement a small subset of the supported operations. To add a new type of operation, we had to modify our implementation to include a new regex. Without a proper language compiler, it became difficult to add new types of expressions that we needed, especially given how complex some FHIRPath expressions can be.

To save us from this “regular expression rabbit hole", we leveraged the open-source ANTLR tool, which is a parser generator. Given the FHIRPath grammar, ANTLR generates a parser object with methods that enable the implementer to walk the parse tree, unlocking the complete grammar for implementation. 

This boils things down into a simpler, albeit still difficult, problem. The question remains, how do we walk the parse tree and produce something evaluatable against FHIR resources?

Two examples of parse trees. On the left, an algebraic expression and on the right, a FHIRPath expression.

The figure below demonstrates three fundamental aspects of the FHIRPath language:


This sample expression demonstrates three fundamental aspects of the FHIRPath language.

// Compile parses and compiles the FHIRPath expression down to a single

// If there are any syntax or semantic errors, this will return an

// error indicating the compilation failure reason.

func Compile(expr string, options ...CompileOption) (*Expression, error)

// Evaluate the expression, returning either a collection of elements, or error

func (e *Expression) Evaluate(input []fhir.Resource, options ...EvaluateOption)

We decided to design the library so that FHIRPath evaluation is broken down into two steps:
     1. Compilation
     2. Evaluation

With this design, users can reuse expressions without the computational overhead of re-parsing each time. Additionally, it enables syntax errors to be caught earlier.

FHIRPath can process any of the FHIR-defined 

, as well as additional FHIRPath-defined 

The Golang implementation of FHIR doesn’t represent this type hierarchy as defined. Additionally, the FHIRPath spec implicitly converts between FHIR native and FHIRPath native types. In order to process all different types with one homogenous interface, we defined a custom type system.Collection as a type alias for [ ]any. This allowed flexibility of working with all types, while concurrently allowing us to override the type with useful custom methods to work with the returned type.

We broke down all FHIRPath expressions to follow one homogenous interface that accepts a system.Collection as input, does some processing, and produces a system.Collection as output.

Here’s how we modelled this in Golang:

// Expression is the abstraction for all FHIRPath expressions,

// ie. taking in some input collection and outputting some collection.

    Evaluate(*Context, system.Collection) (system.Collection, error)

Next, we defined a custom expression type for each type of FHIRPath expression that implements this interface. The result: the compiler parses the expression, and based on the expression that it is currently visiting, produces one of these expression types.


The figure below is an example implementation for the TypeExpression. The first path element to a FHIRPath expression is a filter on the resource name. We implemented this filter with the Google FHIR proto API.

After putting it all together, we produced these expressions while parsing the FHIRPath expression, and combined expressions by piping the output of one to the input of another. We’ll discuss the details of how this works in the next post of Verily’s Tech Blog.

// TypeExpression contains the FHIR Type identifier string,

// to be able to filter the items in the input collection that have the

// Evaluate filters the messages in the input that are identified by the Type

func (e *TypeExpression) Evaluate(ctx *Context, input system.Collection) (system.Collection, error) {

        message, ok := item.(proto.Message)

        // find message name, add to collection only if it matches

        pReflect := message.ProtoReflect()

        name := pReflect.Descriptor().Name()

FHIRPath defines a set of functions as part of the specification. Similarly to how expressions are defined, functions all fit within one common interface: they take in an input system.Collection and spit out an output system.Collection. However, they also support a set of parameters, from 0 to N. To massage the interface to support all different kinds of functions with varying input parameters, we defined the abstraction as shown in the figure below.

// FHIRPathFunc is the common abstraction for all function types

type FHIRPathFunc func(ctx *expr.Context, input system.Collection, args ...expr.Expression) (system.Collection, error)

// Function contains the FHIRPathFunction, along with metadata

To check the function arity at compile time, we defined a min and max arity value for each function. This was essential because catching at compile-time means less failure as run time. And because the function arguments can be any type, or even a FHIRPath expression in itself, we could not implement compile time type checking. Rather, we verified the types at run time.


Another unique implementation feature is that we supported the injection of custom golang functions at compile time. With some protoreflect 

, we can now convert regular Golang functions into the FHIRPathFunc abstraction, making it easy to leverage complete bespoke functions that the FHIRPath specification doesn’t support.

We did this because most of our users don’t know protoreflect. So, we thought allowing developers to define their function with Go code would be helpful, which could also seamlessly interoperate with the FHIR API.

, we turned our FHIRPath engine from a patchwork of hacks into something much more stable and flexible. Now, we catch errors early, reuse compiled expressions, and even support custom functions — all without battling the system. At Verily, it’s transformed the internal mechanics of how our product engineering team works with healthcare data. 


Watch for the next post of Verily’s Tech Blog that explores challenges we encountered during the FHIRPath implementation, and the tricks, including some that could be described as fancy, that we used to overcome issues and create persistent improvements.

Implementing FHIRPath in Go: How we tackled challenges

Learn how Verily rebuilt its FHIRPath engine using ANTLR to create a faster, more flexible, and extensible way to evaluate healthcare data in Go.

Tech Blog #2: FHIRPath Implementation Series

Verily Tech blog image

 — a look under the hood at product engineering, and into the key learnings, thought processes and capabilities that enable building 

 for making healthcare data AI-ready, along with powering other innovations to realize the promise of precision health.

 as the foundational data model across our work. FHIR, which stands for Fast Healthcare Interoperability Resources, is the data standard that enables secure, efficient exchange of electronic health data between healthcare systems and applications. But it isn’t just about data exchange — it’s about shared understanding. At Verily, we’ve seen firsthand how FHIR and specifically, FHIRPath, enables real collaboration across expertise and roles. From clinicians to engineers, product teams to data informaticists, 

FHIRPath ensures everyone can speak the same healthcare data language.

In this first post, we explore how FHIRPath supports that collaboration to accelerate product development and enable scaling that’s smarter, not just faster.

Largely, FHIR is a way for machines to talk to each other about healthcare data. Once in use, it’s also a way for people to talk to each other about healthcare data. With FHIR, medical doctors, software engineers, scientists, and product managers can collaborate using the same set of definitions to build systems that help make people healthier.

For the vast majority of cases, the simplest way to describe a matching record is to use FHIRPath, an expression language based on FHIR. Given this is a universal language used in health technology, its detailed nuances aside, many new Verily employees know what this means before their first day: `Patient.name.given = Jane`. 

At Verily, we build software. Ironically, a great way to scale and build products faster, is to remove the need for a software engineer to interpret the intention of domain experts, such as medical, data, or product experts. If these experts are independently able to configure a software system without engineer support to translate their ideas, they can roughly double the pace of implementation. And as expected, we like going faster.

At its core, FHIRPath is a filtering language. It takes resources, evaluates an expression on them, and returns any FHIR data type. This makes it perfect for quickly evaluating complicated sets of conditions on collections of resources, and iterating on that set of conditions. It can do other things, such as return booleans, but essentially, this is the same as filtering language.


We’ve had FHIRPath users try to specify queries. And certainly, it seems like it should work. They may ask themselves, “Can’t `Patient.name.given = Jane` return all the Janes in my database?” But in fact, although it’s possible in specific cases to create a query from a FHIRPath expression (and 

 exist), like “Patient’s given name”, it isn’t possible in the general case.

Take this simple modification on the previous example: `name.given`. Evaluated on an array of resources, that expression returns an array of given names. Most databases wouldn’t support this kind of query.

So you can map some FHIRPath expressions to queries. And for such sets, offering support may be useful in your system. But in general, it can’t be done.

What FHIRPath is, and isn’t —
in brief

In order to explain why we use FHIRPath the way we use it, we’ll need to give a brief overview of our platform stack.

At Verily, the entire system revolves around a central storage solution with foundations in the 

 API (Google’s FHIR API). Because GCP is at our core and our Alphabet DNA runs deep, we define all our network API (and indeed lots of internal APIs) through 

. This is great news for our native FHIR support, though, because Google maintains an excellent 

. This means that all our language-specific (Golang, Typescript, Swift, Kotlin, etc) internal representations of FHIR objects are all created from a shared definition.

We have dozens of sub-systems that perform independent and parallel data transformations directly in the shared central FHIR storage layer. This goes from internet-fundamentals such as Authz, to fancy algorithms running on unstructured Patient data. Our architecture is all event-driven, resulting in a fire-hose of events that all systems drink from. Each asynchronous sub-system routinely filters through ~10M (and growing…) events per hour.

 to publish all database changes on shared publishers. System components are then free to create subscriptions to these publishers, and trigger processes according to their own rules. This is where the filtering capabilities of FHIRPath really become useful for us.

Our stack: GCP Healthcare,
pub/sub, protocol buffers, golang


In our event-driven system, components need to filter out most of the events that come their way. Using FHIRPath, we configure them to listen only to a specific set of events. Essentially, we get a set of input resources, each with their set of FHIRPaths, and process events only if the required conditions match.

For example, take a look at a Golang filter for some Questionnaire Responses that might be relevant to us.

for _, extension := questionnaireResponse.GetExtension() {

    if extension.GetUrl() == "http://foo/url" {

        if extension.GetValue().GetReference().GetValue() == "foo-value" {

It isn’t too complicated, right? Here is its FHIRPath analog.

            - path: "meta.extension('http://foo/url').value.reference.value = 'foo-value'"

The differences themselves aren’t that stark. But imagine that you want to modify your existing filter, adding something like, `.meta.tag.code = 'foo-bar-baz'`, which is a very simple idea to express in FHIRPath.

In the Golang case, or any other programming language, you’d need to communicate it to an engineer, who would have to invoke the relevant accessor functions one-by-one, not to mention they’d have to get on their roadmap or iteration/sprint plan.

However, in the FHIRPath case, you simply add an item in the configuration file, and you’re good to go. It might even be de-coupled from your code’s release and deploy cycle. This means an informaticist, for example, could create, modify, test, and deploy configuration changes without needing to coordinate with an engineering team. This is a huge win for productivity.

More about Verily