We have a small internal Streamlit app that helps our sales team — answer sheets, discovery prompts, objection rebuttals, all grounded in Abilytics decks and case studies. It's not customer-facing. It's the kind of tool where the "right" architecture is the one that costs the least to run and the least to keep alive.
The first version used AWS: Chroma on the app's local disk with hash-bag-of-words embeddings, Bedrock Nova Micro for generation, and an AssumeRole into an execution role to read the corpus from S3. It worked. Nova Micro is genuinely cheap. But the corpus lived in AWS, the app lived in Databricks, and the pipe between them was glued with boto3 and prayer.
Then Databricks Free Edition dropped Agent Bricks (Knowledge Assistant + Genie) and pay-per-token Foundation Model endpoints — all with a monthly quota. Which meant the whole stack could live in one workspace on one credential for zero recurring cost. Worth trying.
This is a straight comparison of what happened.
What Stayed the Same
The Streamlit UI, the modes (knowledge, discovery, solution, objection), and the guardrails. A double-gated IntentGuard evaluates raw input, resolves coreferences, then re-evaluates the rewritten query. A partial-allow layer handles mixed prompts like "give me three CFO objections and write the Python for it" by answering the sales half and refusing the code half. An output validator scrubs anything that slips through.
None of that changed. It's decoupled from the retrieval/generation backend, which is exactly why swapping the backend was tractable.
Trying Agent Bricks Knowledge Assistant
The first thing I tried was Databricks' managed answer — the Knowledge Assistant. On paper it's the ideal shape for us: point it at a Unity Catalog volume, KA handles retrieval + generation server-side, cites its sources, and the app just makes one HTTP call.
I went in fully committed. Provisioned the whole thing via CLI — schema, volume, KA instance, knowledge source, sync. Wrote the provider, wrote the tests, wired the double-gate guardrails around it, added a light Foundation Model provider for the cheap rewrite/summary calls so we wouldn't pay a full KA retrieval per pronoun. The plan was: KA handles docs, Genie handles the objection rebuttals table, guardrails wrap both.
Here's what happened.
The plan was to point the app at a Databricks-managed retrieval+generation endpoint. Create a Unity Catalog volume, drop the corpus in, create a Knowledge Assistant over it, and the KA endpoint does retrieval and generation server-side. One HTTP call, structured citations, no local vector store.
Provisioning was straight CLI. workspace.presales.knowledge volume (couldn't create a fresh catalog — the metastore has Default Storage on and needs an explicit storage location I didn't want to attach). Eleven files uploaded. KA created. Knowledge source attached. Sync triggered.
The API's payload shape wasn't obvious. KA rejects the OpenAI Chat Completions messages shape — it wants OpenAI Responses (input array + top-level instructions + max_output_tokens). A ten-minute change on our side; worth catching once.
Then indexing failed mid-build: Vector search index is not able to sync. Please try again later. Free Edition serverless compute is shared across Vector Search indexes, Apps, and SQL warehouses. Two always-on Databricks Apps in the workspace were holding compute slots continuously; the indexer couldn't get one. Retry got through eventually.
Then the endpoint 500'd on live queries. State READY / DEPLOYMENT_READY, vector index ready=true with 58 rows. Payload validated cleanly against the KA's Pydantic model. But every real query returned HTTP 500 "Internal error." I recreated the KA fresh, tried three payload variants, nudged a redeploy. Same result. The CLI marks the knowledge-assistants API as *Beta*, and this is a Beta bug I couldn't work around from the outside.
At that point I had two choices: file a support ticket and wait, or build the pieces myself out of the endpoints that do work on Free Edition.
What Actually Shipped
Free Edition has three things I could reach reliably today: Foundation Model chat endpoints, embedding endpoints, and the App runtime itself. That's enough for RAG if you assemble it yourself. Every piece is a serving endpoint from Databricks' shared multi-tenant pool — no dedicated compute, no long-lived warehouse, no monthly bill.
Retrieval: 100 chunks embedded via databricks-bge-large-en (1024-dim). Stored as a 2 MB JSONL and loaded into memory on app start. Query does a plain Python cosine over 100 vectors — fast enough not to matter.
Generation: databricks-meta-llama-3-3-70b-instruct receives the top-k chunks and the mode prompt. Answers come back cited.
Guardrails: unchanged from the original app. Total shape:
User → IntentGuard (raw + rewritten) → bge-large embedding → cosine over 100 chunks → Llama 3.3 70B chat → grounded answer with citations → validate_generation_output → Streamlit UI
https://youtu.be/2wi-Ol9NZVc
Two problems came up that were worth learning:
Chroma 1.0's embedding-function handling silently downgraded our 1024-dim vectors to a 256-dim built-in. Even after implementing the newer EmbeddingFunction interface with embed_documents / embed_query, storage was 256-dim. Not worth debugging further for 100 chunks — I dropped Chroma and wrote 50 lines of pure-Python cosine. Faster, no dim negotiation, no dependency.
Free Edition embedding endpoints throttle hard. Sending 100 chunks in a single batch returned REQUEST_LIMIT_EXCEEDED: Exceeded workspace QPS rate limit. The bge-large endpoint on Free Edition tolerates roughly one call per second. Now the embedder sends one text at a time with a 0.6s spacing and exponential backoff on 429/5xx/connection-drop. Cold-start reindex takes ~2 minutes. Once done, results persist to disk and ship with the app bundle — no reindex on subsequent deploys.
Bedrock + Chroma + S3 vs What We Ended Up With
Both stacks can run this app. The tradeoffs:
| Dimension |
Bedrock + Chroma + S3 |
Databricks Free Edition (native RAG) |
| Recurring cost |
Very low per-token + S3 |
$0 within monthly Free Edition quota |
| Credential surface |
AWS keys + AssumeRole + external ID; Databricks separately |
One Databricks identity (App SP) |
| Corpus updates |
Push to S3 → reindex Chroma at startup |
Local knowledge/ folder → bundle deploy |
| Retrieval quality |
Hash bag-of-words |
bge-large-en, real semantic embeddings |
| Where things live |
AWS + Databricks |
Databricks only |
| Ceiling |
Scales with AWS bill |
Free-tier QPS and monthly caps |
Bedrock is still perfectly fine if you're already on AWS or need Nova Micro's specific pricing curve at scale. For an internal tool where the corpus is small and traffic is a sales team, "one thing to maintain" won.
Deploying to the Team
The app is a Databricks App now, not a laptop process. databricks bundle deploy uploads the code plus the pre-embedded JSONL. databricks.yml binds CAN_QUERY permissions on the three endpoints (Llama, bge, KA) to the App's service principal — no PAT in the deployed config.
The URL lives at abilytics-presales-copilot-*.aws.databricksapps.com. Every request is SSO-gated to the Databricks workspace, so anyone the workspace admin invites can reach it from any laptop, any network. That's the shape "public to the sales team" takes on Free Edition: workspace membership becomes the access list.
What Didn't Make It
Agent Bricks Knowledge Assistant is in the deployed config as the on-deck answer path. Two KA instances were provisioned end-to-end during this migration — endpoint, knowledge source, indexed corpus, CAN_QUERY permissions to the app SP — and the provider code + tests are shipped. Both instances 500'd on live queries in the Beta; when Databricks resolves that, flipping LLM_PROVIDER=agent_bricks is a one-line env change and we're on the managed retrieval+generation path with no other code motion.
Genie is on the same footing — provider stub, config env vars (DATABRICKS_GENIE_SPACE_ID, DATABRICKS_GENIE_WAREHOUSE_ID), and a documented plan for the tabular path. Right now our only tabular file is Objection Rebuttals.xlsx with 8 rows, which the RAG stack already answers well enough. The day we add a real analytical table — deal history, usage telemetry, ARR by segment — Genie becomes the right tool: create a Genie space over the Delta table, drop the space id into the env, and route the modes that ask analytical questions through it. That's the next thing to test.
The Part That Actually Mattered
The interesting question about this migration wasn't "is Databricks better than AWS." It was "can we assemble a working system out of the pieces that a Free Edition workspace actually gives us today, and what does that force us to give up?"
Answer: we gave up the "one managed endpoint" convenience of KA. We gave up Chroma. We accepted a one-time 2-minute cold reindex against a throttled embedding endpoint. In exchange we got a stack with a real embedding model, real citations, zero recurring cost, and one credential to rotate. That trade was worth it.
If you're doing this on a paid workspace where KA works and warehouses spin up on demand, the answer will be different — you'll take the managed endpoint and skip the pure-Python cosine. That's fine. The point isn't that Free Edition is the right answer; it's that the constraints of the tier you're on shape which trades you should be making.
The rip-out is done. The rebuild is live. The sales team is on it now.
