I'm curious to know, based on your experience in production, how much does Chain-of-Though Reasoning typically cost in terms of tokens for frameworks like LlamaIndex, LangChain, CrewAI, etc.?
I understand it depends on many different factors including the complexity of the product and the architecture of the agents involved, but I'd love to hear about your experiences in creating real-world-class application.
tikkun4 months ago
tough to answer. depends on the llm, on the api provider or hosting method, on the prompt
but you could get a simple estimate by running a CoT prompt, copying the output tokens, and then using an online token counter to count the length & tokens of the CoT part of the completion
maniliop4 months ago
The issue is that I can give it a try on a set of toy problems, but finding a real-world-class problem is totally a different story IMHO. That is exactly the reason I asked about people's "experiences".
Anyway thanks for the response.
kingkongjaffa4 months ago
> real-world-class problem
The issue is that this notion is nonsense. There are just problems nothing about them makes them “world class” whatever that means.
maniliop4 months ago
Fair enough, it's "nonsense". Curious to know about YOUR experience. Sounds "nonsense" yet?