How much will this system cost in terms of compute overhead and APIs?
Context: personally I think we probably want the best embeddings we can get, even at very high cost — at least I am happy to throw money to achieve a few percentage points of better quality. But then, for example, we will probably want to regenerate embeddings regularly and throw a lot of those expensive API calls away. So, iterating with a RAG system could add up. I have never built a system like this before that might depend so heavily on external models.
Eventually we will do fine-tuning and potentially even model training, but for now I'm just trying to get my head around the costs of a well-prepared RAG system.
-
For inference / runtime costs, if we had x users and x queries per session, how does it pencil out?
-
For training, how many round-trips requests do we need to get our metadata refined, split, summarize, vectorize, tag, etc. to arrive at the system that is ready for inference? I assume we will use OpenAI to generate embeddings, and there will be pre-processing steps needed to get quality embeddings.
-
For other compute costs, hosting and indexes etc, we probably need a spreadsheet with all of our SaaS tools, APIs and costs.
How much will this system cost in terms of compute overhead and APIs?
Context: personally I think we probably want the best embeddings we can get, even at very high cost — at least I am happy to throw money to achieve a few percentage points of better quality. But then, for example, we will probably want to regenerate embeddings regularly and throw a lot of those expensive API calls away. So, iterating with a RAG system could add up. I have never built a system like this before that might depend so heavily on external models.
Eventually we will do fine-tuning and potentially even model training, but for now I'm just trying to get my head around the costs of a well-prepared RAG system.
For inference / runtime costs, if we had x users and x queries per session, how does it pencil out?
For training, how many round-trips requests do we need to get our metadata refined, split, summarize, vectorize, tag, etc. to arrive at the system that is ready for inference? I assume we will use OpenAI to generate embeddings, and there will be pre-processing steps needed to get quality embeddings.
For other compute costs, hosting and indexes etc, we probably need a spreadsheet with all of our SaaS tools, APIs and costs.