Transforming an Enterprise Platform
Rebuilding an acquired product for horizontal scale, without ever stopping.
The story of who built the team that did this is a separate one. This is the story of what we built.
The situation
vRealize Operations came into VMware through acquisition. It monitored virtualized infrastructure and it was good at it, which was also the problem: enterprise customers were running estates far larger than anything the original design had targeted. A single deployment could be watching thousands of hosts and hundreds of thousands of virtual machines, and the product had been architected before that was the normal case.
Two things had to change together. The platform needed to scale horizontally rather than by being given a bigger machine. And it needed to be simpler to operate, because a monitoring product that requires expertise to run is asking customers to solve the problem they bought it to avoid.
The constraint was that large customers were already running it in production. There was no version of this where we stopped, rebuilt, and came back.
What made it hard
Strategically. We were rebuilding the foundation of a product with live enterprise commitments. Every quarter of architecture work was a quarter where customers on the current release saw less. The rebuild had to be worth more than the visible work it displaced, and defensible to the people feeling the displacement.
Technically. Horizontal scaling was the headline. The real difficulty was that different parts of the workload wanted fundamentally different things from storage. Time-series metrics at that volume, low-latency in-memory access for live analysis, and relational data for configuration and state are three different problems, and solving them with one store meant compromising all three.
Organizationally. Splitting a team’s attention across a rebuild and ongoing support is a known structure with a known failure mode: the sustaining track quietly becomes the lesser assignment, and the people on it know it before anyone says so.
What I did
I helped set the technical direction for the re-architecture and the structural split that made it possible. We separated the work into a future-architecture track and a sustaining track, so the rebuild had a team that was not being interrupted and customers had a team that was not being deferred.
On the architecture, I worked on the distributed foundation and the platform APIs, and on matching storage technology to workload rather than standardizing for its own sake. Cassandra, GemFire and PostgreSQL each did a job the others would have done worse. The stack was Java and Spring Boot, with the analytics that made the product valuable running on top.
I did not hold the engineering-management title in this period. The broader global organization of around two hundred engineers executed across multiple sites, and the sustaining track ran in parallel outside my direct focus. What I owned was technical direction and a share of the argument for why this shape was right.
Some of the most useful work was not architecture at all. A telecom operator asked to meet R&D rather than sales, so I went. What they needed was not a fix but a different shape: I proposed separating their environment across five datacenters for disaster recovery. They adopted it, expanding from one site to five, which required additional licensing.
A bank asked for something that ran against best practice and carried real risk. They had legitimate reasons and could not wait a year for the supported path. The easy answers were both wrong — refusing, or building what they asked for. I brought in security specialists to review the approach and we implemented it in a way that was safe. What that engagement involved is confidential and I can only describe it at this level.
The decisions
One team or two tracks
The fork. Rebuild inside the existing team, interleaving architecture work with customer commitments, or split into future-architecture and sustaining tracks.
The call. Split.
The cost. It landed on people rather than on the system. Sustaining work is necessary and it is less visible, and some strong engineers spent that period away from the rebuild everyone was watching. We managed it. I would not pretend the structure was neutral.
One store or several
The fork. Standardize on a single data store, or use polyglot persistence with each workload on the technology that suited it.
The call. Polyglot.
What it cost us was permanent. More systems to run, more failure modes, more to hire for, and fewer engineers who could hold the whole stack in their head. Every performance win came with an operational tax attached, and we paid it for years.
Preserve the workflow or redesign it
The fork. Keep the existing customer workflow intact and rebuild underneath it, or redesign the workflow around what the new architecture made possible.
The call. Redesign.
The cost. It was borne by customers and by the people supporting them. Large accounts had to be walked through a changed experience, which meant support load, account-management time, and a period where familiarity was worth less than it had been. Simpler is not the same as familiar, and the gap between them is expensive.
What changed
| Before | After |
|---|---|
| Complex acquired system, demanding to operate | Enterprise-scale platform |
| Vertically limited scaling | Horizontal, distributed scaling |
| One team, mixed priorities | Dedicated future-architecture and sustaining tracks |
| Undifferentiated customer workflow | Redesigned, simplified workflow |
The platform took on estates of thousands of hosts and hundreds of thousands of virtual machines as a normal case rather than an edge case.
What I carried forward
This is where I started asking what compute actually costs.
Customers were spending heavily on datacenter infrastructure and had almost no visibility into what any of it was costing them per workload. There was no invoice to read — they had paid upfront. The work I did on that question during these years produced three granted patents between 2017 and 2020, on computing a unit rate for power consumed in a datacenter, on reducing that consumption, and on placing workloads against policy and benchmark data.
I did not think of it as a specialism at the time. It turned out to be the question I kept returning to for the next decade, in environments that looked nothing like this one.