Why We Chose Turbopuffer For Our Search Infrastructure

At Eve, we had previously used OpenSearch for our search infrastructure. OpenSearch has many good things going for it:

1. hardened open source technology that is actively maintained
2. managed option available AWS, docker container available for local dev
3. support for both keyword-based search and embeddings-based similarity search

We had adopted OpenSearch without too much consideration on our exact use cases, as it was a good enough option to get started with, and we were still early in trying to find product market fit.

Over time, we settled on more clear cut product functionality. Eve was a product to help plaintiff law firms with their case work. The vast majority of product functionality happened at the individual case level, and so it made sense to build our search indices on a per-case basis to give the most precise results.

OpenSearch became problematic as we scaled. Most of it boils down to 1 key reason: OpenSearch is inherently designed for larger index shards of 10–50GB each, while Eve was trying to run lots of small indices, often under 1GB in size. This led to multiple problems over time in OpenSearch:

1. Instability of metadata management of large number of shards → OpenSearch cluster could crash or lose data
2. Over-provisioned nodes in order to handle large number of shards → OpenSearch was extremely expensive to run, easily over $10,000/month on AWS.
3. Opensearch’s HNSW implementation for powering similarity search was generally very memory intensive, another contributor to high costs. Many other server vector database solutions face a similar problem, they are NOT cheap at scale.

We had two choices:

1. Look into merging indices so that shard size was in the recommended limits, and number of shards could be greatly reduced.
2. This would involve engineering some way to actively re-balance indexed data for cases, and was not something we wanted to spend time on. As important as search was to our stack, there were much higher value adds to the product we could be spending eng cycles on.
3. Look for an alternative technology that better suits our needs.

Along came Turbopuffer. At the time, we were exploring various serverless options for vector databases, and saw that both the Cursor and Notion teams had found success with this new kid on the block.

Turbopuffer was serverless search infrastructure that was well-suited for our needs of a large number of small search indices:

1. Basically no limits to the number of namespaces (aka indices) → natural fit for our per-case index use case
2. Data was kept in object storage when unused → Turbopuffer is cheap!
3. Data is cached into NVMe SSDs when actively used → Turbopuffer is still fast at query time!
4. Support for both keyword-based search and embeddings-based similarity search

From interacting with the Turbopuffer team, we were also impressed with their fast response times and general knowledge of search infrastructure. It’s always fun to move fast together with another early stage startup!

We ran a quick POC and within 1 week we had productionized a Turbopuffer beta. After another 2 weeks, most of it just spent baking and monitoring, all indices had been fully migrated to Turbopuffer. There was no downtime, and no issues throughout this entire process, just lots of successful API calls and a final monthly bill of under $300.

Turbopuffer continues to power our search stack today at Eve, and we’re glad we made the early investment to switch to a dedicated serverless search infrastructure provider. At Eve, we’re on a mission to build the best AI for plaintiff attorneys, and that’s we want to focus our in-house engineering resources.