Software Development

Introducing a product delivery culture at Etsy

Etsy, an online marketplace for unique, handmade, and vintage items, has
seen high growth over the last five years. Then the pandemic dramatically
changed shoppers’ habits, leading to more consumers shopping online. As a
result, the Etsy marketplace grew from 45.7 million buyers at the end of
2019 to 90.1 million buyers (97%) at the end of 2021 and from 2.5 to 5.3
million (112%) sellers in the same period.

The growth massively increased demand on the technical platform, scaling
traffic almost 3X overnight. And Etsy had signifcantly more customers for
whom it needed to continue delivering great experiences. To keep up with
that demand, they had to scale up infrastructure, product delivery, and
talent drastically. While the growth challenged teams, the business was never
bottlenecked. Etsy’s teams were able to deliver new and improved
functionality, and the marketplace continued to provide a excellent customer
experience. This article and the next form the story of Etsy’s scaling strategy.

Etsy’s foundational scaling work had started long before the pandemic. In
2017, Mike Fisher joined as CTO. Josh Silverman had recently joined as Etsy’s
CEO, and was establishing institutional discipline to usher in a period of
growth. Mike has a background in scaling high-growth companies, and along
with Martin Abbott wrote several books on the topic, including The Art of Scalability
and Scalability Rules.

Etsy relied on physical hardware in two data centers, presenting several
scaling challenges. With their expected growth, it was apparent that the
costs would ramp up quickly. It affected product teams’ agility as they had
to plan far in advance for capacity. In addition, the data centers were
based in one state, which represented an availability risk. It was clear
they needed to move onto the cloud quickly. After an assessment, Mike and
his team chose the Google Cloud Platform (GCP) as the cloud partner and
started to plan a program to move their
many systems onto the cloud

While the cloud migration was happening, Etsy was growing its business and
its team. Mike identified the product delivery process as being another
potential scaling bottleneck. The autonomy afforded to product teams had
caused an issue: each team was delivering in different ways. Joining a team
meant learning a new set of practices, which was problematic as Etsy was
hiring many new people. In addition, they had noticed several product
initiatives that did not pay off as expected. These indicators led leadership
to re-evaluate the effectiveness of their product planning and delivery

Selecting a partner

Etsy had never worked with a software development consultancy. To be able
to scale product delivery and bring in some outside expertise, they started
looking for a vendor. After an extensive search, they selected Thoughtworks,
primarily because they perceived a close cultural fit and deep alignment
regarding modern software development.

Thoughtworks has a similar approach to Agile as Etsy – being Agile
rather than doing Agile. Etsy’s team was not looking to adopt a
methodology that blindly follows strict rituals and practices, and wanted
a partner that bases its approach on agile principles and culture.
Thoughtworks is known for technical excellence and leadership in DevOps,
Continuous Integration, and Continuous Delivery, all of which Etsy relies
on heavily. In addition, the two companies shared similar principles
regarding employee care, diversity, open-source, and technology

The Thoughtworks team started by embedding into product teams, working with
both buyer and seller teams, on various critical initiatives such as payment
model changes, Search Engine Optimization (SEO), and a notification platform.
This enabled Thoughtworks to understand how Etsy worked, created an immediate
impact on delivery, and improved development practices such as automated

For Thoughtworks, Etsy was quite different from the typical client.
They are a fully digital company, and their technical process and product
practises are very mature. Unlike most clients who engage Thoughtworks,
Etsy had no large transformation to undertake, and no ingrained habits
that needed a major shift. What they were seeking was outside perspective
and supplementive skills, to help them incrementally improve and tackle
their pressing initiatives. The culture and principles were already very
aligned with Thoughtworks’ own, so the partnership became more of an
exchange of ideas drawn from different experiences.

Scaling Product Delivery and Discovery

Thoughtworks and Etsy formed a cross-functional Product Delivery
Culture (PDC) team to analyze the current product delivery and discovery
process. They analyzed the entire software development value stream,
looking closely at the complaints of managers and team members. Selected
events were replayed to gain a deep understanding of “how” it happened
rather than who?. The Thoughtworks
team, because they hadn’t lived through the problems, were able to bring
an objective outsider perspective.

There were a number of things Etsy was doing really well:

  • Cross-functional teams: They structured their teams around “4 table legs”:
    Product, Design, Engineering, and Analytics. All planning and delivery practices
    happen with collaboration among the groups.
  • Incremental delivery of value to users: Continuous delivery is the core
    practice of their approach. A code change sitting in a repository and not
    deployed to users is not valuable, and effectively costs the company money. Etsy
    deploys hundreds of times a day, with a 30-minute commit to production lead
  • Features are prioritized using data: Product managers and data analysts
    collaborate to determine an expected value, measured by KPI changes. If the
    feature does not deliver enough value versus its cost, as evidenced by an A/B
    test, it is not rolled out, and the code is removed.

The team also uncovered opportunities for improvement:

  • Each team was doing delivery in different ways, having disparate processes
    and rituals, making onboarding new employees and changing teams difficult. Some
    teams were using a process like Scrum, while others did not use any formal
  • There were a number of initiatives that had not created value as
    they expected. For example, Etsy tried a gift wrap feature that had
    lower than expected adoption rates by sellers and buyers.
  • Prior to 2018, Etsy had been through a transitional period, with
    pivots in product strategy and leadership changes leading some product
    managers to expres a desire for greater ownership over their product

Time to learning

The team created an improvement program, based on lean thinking and the
ideas of Marty Cagan, who had recently presented at Etsy. The leadership team was
reading his book Inspired.

I promise you that at least half the ideas
on your roadmap are not going to deliver what you hope. (By the way, the
really good teams assume that at least three quarters of the ideas won’t
perform like they hope.)

Marty Cagan, Inspired (p19)

After some initial research the team came up with a metric they called Time
to Learning – the time it took for a product team to validate an idea with a
customer and gain a better understanding of its value. They had a baseline of
50 days that they wanted to reduce.

They were also looking at other metrics.

  • Improving the time to market
  • Improving the effectiveness of products (e.g. revenue, customer
  • Employee happiness

To impact the KPIs, they came up with a number of solution hypotheses:

Light-weight prototypes that incorporate more direct user research

Etsy has strong experimentation infrastructure and analytics capabilities.
They run many concurrent A/B tests on the marketplace. The problem the PDC
team observed is that the feedback cycle can be quite slow. For an idea to be
ready for an A/B test it had to be near production-level quality. It also had
to have enough data to show statistical relevance. A/B tests would run for
months in lesser-used parts of the marketplace.

To set your expectations, strong teams normally
test many product ideas each week—on the order of 10 to 20 or more per week. I
want to emphasize that these are experiments, typically run using prototypes.
A prototype is not something that’s ready for prime time and certainly not
something your company would try to sell and stand behind. But they’re
immensely useful, as they’re all about learning fast and cheap.

Marty Cagan, Inspired (p27)

To reduce the Time to Learning metric, the PDC team applied a lean UX
approach. First, they started by quickly creating lo-fi prototypes and
immediately showing them to users. Second, the team increased user feedback
sessions to every two weeks and reduced the level of ceremony. Third, they
created a dual-track system, with a design and research team continually
working ahead of a delivery team. The idea was to efficiently gain confidence
in an idea before it qualifies for an A/B test. By reducing the cost of
experimentation, they could increase the variety and quantity of experiments,
finding ideas that showed more value.

Figure 1: Experiment validation process

Blueprint for product delivery

The PDC, in conjunction with product team leaders, came up with a set of
product delivery principles and related delivery practice changes. The
practices included: smaller more focused stories, redesigning story walls to
better fit the team process, using spikes to reduce technical uncertainty, and
better ways of managing tech debt. They did not want to remove the autonomy of
teams. Continuous improvement is a big part of Etsy culture. These practices
are meant to be treated as sensible defaults that can be adapted and improved
for each team’s specific context.

The PDC team first tested the practice changes on pilot product teams to
establish their effectiveness and then iterated. Once the practice
demonstrated an improvement, the team added to a knowledge base and provided
light coaching to introduce the new practices to all product teams across

Lean Portfolio management

Some large initiatives were still being carried out, even though there was
not clear evidence of the initiative creating value. This is the typical sunk
cost problem. To solve this, the PDC team
suggested taking a lean approach to portfolio management.

Our runway should be a list of hypotheses to test, not a list of
requirements to build. When we reward our teams for their ability to deliver
requirements, it’s easy to rapidly bloat our products with unnecessary
features—leading to increased complexity, higher maintenance costs, and
limited ability to change. Features delivered are not a measure of success,
business outcomes are. Our runway is a series of questions that we need to
test to reduce uncertainty and improve our understanding of growth

Humble et al, Lean Enterprise (p96)

Instead of creating lengthy project specifications and plans, the PDC team
coached product teams to develop outcome statements for each initiative. With
outcome statements, starting with a customer or business outcome allows the
team to experiment with many different ideas to achieve the outcome by tying
them to measurable KPIs. The less precise an outcome is, including its KPIs,
the harder it is to specify. When preparing for possible initiatives, each
product team created a two-page brief summarizing the outcome, the measurable
impact, and the initial investment required.

The executive team prioritized the outcome statements, and each statement
was viewed as a “bet” amongst a portfolio. Inspired by the Lean Value Tree
concept, the executive team would practice a regular cadence to determine if a
bet was paying off, giving them the opportunity to pivot early.

Figure 2: An example initiative statement

Stronger product & engineering collaboration

Another improvement was to include engineers early in the product ideation
process and to have them join user feedback sessions. The team tested
engineers joining the hypothesis generation and validation sessions and
working closely with designers during design sprints. The results of the test
showed a number of benefits. Engineers improved their understanding of the
customer, building empathy for their needs. They could provide on-the-spot
feedback on the feasibility of an approach and quickly create prototypes to
demonstrate ideas to users. Cross-functional engagement allowed engineers to
contribute a number of great ideas that made it to the product.

After the initial PDC initiative Etsy continued to strengthen the
collaboration between groups. At every level of the organization, product and
engineering are partners, which means they’re able to have healthy trade-off
discussions. The platform group expanded the use of product managers for
infrastructure and developer tooling teams – encouraging teams to think about
technical capabilities as products and end-users being Etsy engineers. The
technical product managers do research and prioritization activities, just as
they would for a customer product.

What was the impact of product delivery culture?

Testing with pilot teams indicated that the Time to Learning went from 50
days to 5 days. During the pilot, one product manager said, “What I love about
the [direct user research sessions] pilot is that it lets us test our ideas
faster, it was really nice to not spend as much time diving deep into
something that we didn’t even know we were going to use.” An initiative on
inventory quality that was predicted to deliver a seven million dollar revenue
increase instead delivered eleven million dollars in incremental value using
the PDC suggested improvements. Following this pilot success, the team coached
the entire product development organization on the product delivery approach.

According to Mike Fisher, “It is very hard to isolate the benefits of a
process change on an organization of several hundred engineers since we don’t
run A/B experiments of this type. However, we did look at high level
productivity metrics after having PDC implemented for over 2 years, and all of
our metrics were very positive. We had improved our velocity by doing this
work, and we were able to produce ultimately more benefits for the buyers and
sellers – more features, better search, better processes; the whole
marketplace became better for buyers and sellers and that was the goal.”