Designing a Chimera Data Platform

Image by Gordon Johnson from Pixabay

One of the key roles of a software team at a Biotech organization is to enable the research program to scale without sacrificing flexibility and innovation. I’ll write more about this in upcoming posts, but today I want to explore how my understanding of the scope of this work has shifted since switching from a company that builds software as a product to a biotech startup that uses software to drive innovation.

Software vs IT

The standard perspective of software engineering, or at least the perspective that I initially adopted, is to define the scope of a project by the code that you write. The project may rely on externally-developed libraries and interact with other systems, but these are seen as defining external requirements. Within these requirements, writing code gives the project team both complete internal control and complete responsibility for the details.

In other words, the scope of a software engineering project is limited to what you have complete control over.

For an IT project, on the other hand, the scope of a project is typically to acquire and configure software that someone else has written. (Many IT teams have wider scope, so this is an oversimplification, but bear with me.) The project team in this setting only has control within the bounds defined by configuring the software. Their task is to shape the software to the organization’s processes and structures via this limited control.

So the scope of an IT-style project involves very narrowly defined control, but extends to how the organization will use the software.

Redefining Scope

Coming from a software product company, I initially viewed the role of the internal software team that I lead in terms of a standard software project, limited to the functionality defined by code that we wrote. However, our users were doing most of their work in externally developed software such as the lab team’s Electronic Lab Notebook (ELN).

To organize and optimize the overall exploration and innovation process, we would need to impact all these activities that currently happen in software where the scope of control is only at the level of configuration. We wouldn’t have the control that I was used to.

Expanding the Platform

My first instinct was to move key functionality from these configured tools into the custom tools we were building – to bring the users to us so we could have complete control over their experience.

But it quickly became clear this wouldn’t work: If a user can do 90% of their work in a single tool, they’ll be very reluctant to switch to something else for the 10% that we want to customize. And we’re not going to reproduce all 100% of the existing functionality.

I realized that I needed to adjust how I viewed the scope of our work to a hybrid between a software project and an IT project.

If the goal is to scale the organization’s research program without sacrificing flexibility then we need to leverage every interaction that a user has with any software, whether it’s custom tools we build or off-the-shelf software that we configure. Instead of putting artificial boundaries around the internally-written components, we need to design a single unified platform that spans and integrates all these pieces.

Re-scoping Design

Building a unified software platform requires applying a software design mentality to the larger scope covering both custom and configured software.

For example, we’ve started collecting user stories and user journeys that span multiple systems. This has pushed us to find ways to guide the user between these systems while minimizing context switching. It means creating a consistent experience no matter which system the user is interacting with.

As a result, instead of pushing users into our custom tools, we’ve started looking for ways to keep them in their domain-specific systems. Instead of our custom software replacing functionality from these systems, its role is to provide a central junction that coordinates data flows and guides users between them.

The Chimera Data Platform

Like the chimera – a mythical beast made up of parts from different animals – the resulting platform strives to be a single coherent whole made of different systems that were not designed to fit together.

We haven’t yet perfected this approach, and we still have plenty to learn, but I’m confident that this is a promising new direction that we can continue to refine.

Want to read more? Then subscribe to my weekly newsletter where I’ll announce new blog posts and explore ideas from earlier posts in more depth.

2 thoughts on “Designing a Chimera Data Platform

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: