Author: russasaurous

Russ Brown- January 10, 2026

You know that feeling when you’re three hours deep into an AWS console at 11 PM, staring at an error message that might as well be written in ancient Sumerian? Yeah. This post is about that journey.

I’ve always believed the best way to really learn a technology is to build something real with it — and then write down *exactly* how it went, including the parts where you questioned your career choices, Googled the same error five times, and finally cracked it with a solution so obvious you wanted to throw your laptop out the window.

So when I decided to create a “Recommended for You” feature for classic car enthusiasts — personalized recommendations powered by vector embeddings and semantic search — I knew I had to document everything: the clever architecture decisions, the AWS configuration nightmares, the 2 AM breakthroughs, and that sweet, sweet dopamine hit when the first real recommendation actually made sense.

**Disclaimer:** This is a personal, independent side project and technical demonstration. It is not affiliated with, endorsed by, or built for any specific company or real-world marketplace. All examples, data, scenarios, and car descriptions are hypothetical and used for educational purposes only. No actual muscle cars were harmed in the making of this tutorial.

This post walks through the architecture, the AWS OpenSearch domain creation (featuring **all** the validation errors I encountered — there were… several), k-NN indexing, embedding generation, and basic integration. Basically, all the practical details that Medium tutorials conveniently skip over because the author “definitely didn’t spend four hours troubleshooting that part, nope, worked first try.”

If you’re working on recommendation systems, vector search, or just trying to survive AWS’s passive-aggressive validation messages, I hope this saves you some time (or at least makes you feel less alone when things inevitably break).

Let’s get into it. 🚗💨

—

## The Goal: Semantic Recommendations for Classic Cars

Picture this: a marketplace packed with vintage beauties, roaring muscle cars, and pristine collector’s items. Users browse and buy based on *very* specific tastes — someone who just dropped $50K on a 1969 Ford Mustang Boss 429 isn’t randomly shopping for minivans next. They’re probably eyeing that cherry 1970 Dodge Charger R/T.

Here’s the thing: these are infrequent, high-value purchases. Traditional collaborative filtering (“users who bought this also bought…”) completely face-plants with sparse data. You can’t build a Netflix-style recommendation engine when most users buy maybe one or two cars total.

**Enter vector databases.**

The concept is elegant: represent each car and each user’s preferences as high-dimensional embeddings (basically, coordinates in semantic space), then use k-Nearest Neighbors (k-NN) to find the most similar items. It’s like playing matchmaker, but with math.

The system blends:
– **Content-based signals** (car attributes: make, model, year, price, category, those slightly purple prose descriptions like “chrome gleaming in the sunset”)
– **Behavioral signals** (past purchases, bids, views, the cars they favorited at 2 AM after watching *Bullitt*)

—

## Architecture Overview

Here’s what I cobbled together:

– **Embeddings:** Generated using Sentence Transformers (specifically `all-MiniLM-L6-v2` for text descriptions) with plans to add CLIP for image support because why not make things harder
– **Vector Store:** AWS OpenSearch Service with k-NN plugin (managed service = someone else deals with cluster health at 3 AM)
– **Backend:** AWS Lambda to generate user embeddings on-the-fly and query OpenSearch
– **Data Sources:** Hypothetical listings with classic car attributes; user behavior stored in DynamoDB
– **Freshness:** EventBridge triggers to index new listings as they appear (because recommendation staleness is so 2015)

Simple enough, right? *Narrator: It was not simple enough.*

—

## Step 1: Creating the OpenSearch Domain (AKA: Welcome to Hell)

AWS OpenSearch is actually pretty great — managed service, excellent k-NN support, integrates nicely with the rest of AWS. Creating the domain? Less great. More like negotiating with a very pedantic robot that rejects your application for reasons it will only vaguely hint at.

### Domain Configuration

– **Name:** `hemmings-cars` (name your domain after your dreams)
– **Version:** OpenSearch 2.17 (supports disk-based k-NN for better cost efficiency)
– **Instance Type:** `r6g.large.search` (memory-optimized for vector workloads — vectors are hungry little beasts)
– **Storage:** 100 GiB gp3 EBS per node
– **Security:** Fine-grained access control enabled, with an IAM role ARN set as master user

I clicked “Create.” I felt optimistic. I was a fool.

### The Validation Errors That Almost Broke Me

AWS domain creation validates *everything* before provisioning. Here are the two errors that consumed my entire Saturday:

#### Error #1: “General configuration validation failure”

Super helpful, AWS. Thanks for the specificity.

**Usual culprits:**
– IAM permissions issues (does your role actually have `AmazonOpenSearchServiceFullAccess`?)
– Instance type availability in your region (apparently some regions just… don’t have certain instance types? Cool system.)

**Fix:** Double-checked the role permissions, switched regions, sacrificed a rubber duck to the AWS gods, and retried.

#### Error #2: The IPv6 CIDR Nightmare

Oh, this one. This *beautiful* error message:

“`
IPv6CIDRBlockNotFoundForSubnet
“`

Translation: “Your VPC has an IPv6 CIDR block assigned at the VPC level, but the specific subnets you selected don’t have their own IPv6 CIDR blocks, and I am fundamentally incapable of dealing with this philosophical inconsistency.”

**Solution:** Assigned /64 IPv6 CIDR blocks to each subnet via the VPC console:

“`bash
aws ec2 associate-subnet-cidr-block \
–subnet-id subnet-xxxxx \
–ipv6-cidr-block “2600:1f13:xxxx:xxxx::/64”
“`

**Alternative** (if you don’t actually need IPv6 and enabled it by accident like me): Switch to IPv4-only subnets and remove the VPC’s IPv6 CIDR. Sometimes the best solution is admitting you didn’t need the fancy feature in the first place.

### When Things Get Stuck

At one point, my domain got stuck in “Processing” state for 45 minutes. Nothing moving. Just… processing. Processing what? Processing *feelings?*

Canceled the change:

“`bash
aws opensearch cancel-domain-config-change \
–domain-name hemmings-cars \
–region us-east-1
“`

**Pro tip that could’ve saved me hours:** Use dry runs to catch issues early:

“`bash
aws opensearch start-domain-dry-run \
–domain-name hemmings-cars \
–region us-east-1 \
–cli-input-json file://domain-config.json
“`

This feature is criminally underused. Be smarter than Past Me.

—

## Step 2: Setting Up the k-NN Index

Once the domain finally achieved sentience (or at least “Active” status), I created the index with k-NN enabled:

“`bash
curl -XPUT “https://hemmings-cars.us-east-1.opensearch.amazonaws.com/hemmings-cars” \
-H ‘Content-Type: application/json’ -d ‘{
“settings”: {
“index”: {
“knn”: true,
“knn.algo_param.ef_search”: 100
}
},
“mappings”: {
“properties”: {
“car_id”: { “type”: “keyword” },
“embedding”: { “type”: “knn_vector”, “dimension”: 384 },
“make”: { “type”: “keyword” },
“model”: { “type”: “keyword” },
“year”: { “type”: “integer” },
“price”: { “type”: “float” },
“category”: { “type”: “keyword” },
“description”: { “type”: “text” }
}
}
}’
“`

The `dimension: 384` matches the output of `all-MiniLM-L6-v2`. If you use a different model and get dimension mismatches, you’ll receive an error message that will make you feel like you’ve disappointed your computer personally.

—

## Step 3: Indexing Items with Embeddings

For each hypothetical listing, I generated embeddings and indexed them:

“`python
from opensearchpy import OpenSearch, AWSV4SignerAuth
from sentence_transformers import SentenceTransformer
import boto3

# AWS auth dance
credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, ‘us-east-1’, ‘es’)
client = OpenSearch(
hosts=[{‘host’: ‘hemmings-cars.us-east-1.opensearch.amazonaws.com’, ‘port’: 443}],
http_auth=auth,
use_ssl=True,
verify_certs=True
)

# Load the model (this takes a minute the first time)
model = SentenceTransformer(‘all-MiniLM-L6-v2’)

# Example car listing
car = {
“car_id”: “car_001”,
“description”: “1969 Ford Mustang Boss 429, V8, Candy Apple Red, excellent condition, numbers matching”,
“make”: “Ford”,
“model”: “Mustang”,
“year”: 1969,
“price”: 45000,
“category”: “muscle car”
}

# Generate embedding from description
embedding = model.encode(car[“description”]).tolist()

# Index it
document = {**car, “embedding”: embedding}
client.index(index=”hemmings-cars”, id=car[“car_id”], body=document)
“`

For **user embeddings**, I averaged the embeddings of their past interactions (purchases, views, items they lingered on for suspiciously long). It’s not perfect, but it works surprisingly well.

—

## Step 4: Querying for Recommendations

Simple k-NN query to find similar cars:

“`python
import numpy as np

# Generate user embedding (simplified version)
user_embedding = np.mean([
model.encode(“purchased 1969 Ford Mustang Boss 429”)
], axis=0).tolist()

# Query for 5 nearest neighbors
query = {
“size”: 5,
“query”: {
“knn”: {
“embedding”: {
“vector”: user_embedding,
“k”: 5
}
}
}
}

response = client.search(index=”hemmings-cars”, body=query)

for hit in response[‘hits’][‘hits’]:
print(f”{hit[‘_source’][‘year’]} {hit[‘_source’][‘make’]} {hit[‘_source’][‘model’]} – ${hit[‘_source’][‘price’]:,}”)
“`

The first time this returned sensible results, I literally said “oh DAMN” out loud to my empty apartment.

You can add filters for price range, category, year, etc. as needed. Want muscle cars under $50K from the ’70s? Easy.

—

## Step 5: Integration & Scaling Thoughts

For production (you know, if this weren’t just me messing around):

– **AWS Lambda handler** to fetch user behavior from DynamoDB, generate embedding on-the-fly, query OpenSearch, return results
– **EventBridge** for real-time indexing of new items (trigger Lambda on new listing creation)
– **CloudWatch monitoring** for `KNNGraphMemoryUsage` and cost control (k-NN can get *expensive* at scale)
– **Caching layer** (ElastiCache?) for frequently requested recommendations
– **A/B testing framework** to measure actual click-through and conversion rates

—

## What I Actually Learned (Besides Regex for AWS Error Messages)

The biggest surprises weren’t the ML parts — those were honestly pretty straightforward. It was the AWS domain validation quirks (especially that IPv6 CIDR issue) and how much small configuration details can completely derail your progress.

I also learned that documentation lies. Not maliciously — it’s just that tutorials show you the happy path, and real life is *never* the happy path. Real life is three wrong turns, a weird error message, and a Stack Overflow post from 2019 that almost applies to your problem but not quite.

Building and documenting this reminded me how valuable it is to capture the real process — errors, retries, small wins, moments of confusion, and all. This is the tutorial I wish I’d found on my first attempt.

—

## Next Steps

If I keep going with this (and I probably will, because I’m apparently incapable of leaving projects alone):

– **Image embeddings** with CLIP (because descriptions like “beautiful patina” are subjective, but pictures don’t lie)
– **Hybrid search** combining semantic similarity with traditional filters
– **A/B testing framework** to see if this actually performs better than random recommendations (spoiler: it does, but *how much better* matters)
– **Production deployment** with proper monitoring, error handling, and all those boring adult engineering things

—

## Final Thoughts

If you’ve tackled similar projects or hit the same AWS gotchas, I’d love to hear about it in the comments. Specifically, I want to know:

– Did you also lose hours to IPv6 CIDR blocks?
– What’s your favorite embedding model for niche domains?
– Have you found a way to make AWS error messages less cryptic, or is that just our collective burden to bear?

Thanks for reading, and may your validation errors be few and your k-NN queries be fast. 🏎️

Coming soon – how did I get the DATA for this endeavor?

—

*P.S. — If you actually work on a classic car marketplace and need a recommendation system, my DMs are open. Just saying.*