Local LLM alongside GPT-4
T
Tyler Wince
I would love to see one of these smaller, less powerful, local-only models be an option (like the new WebGPT) alongside GPT-4. I would use GPT-4 when I need a powerful model and don't mind sharing my data with OpenAI. I would use a local-first model when I don't need something super powerful OR I want to use all of my notes as inputs to the model.
Jeremy Jordan
Currently the app lets me input my own API key for OpenAI calls. Could you add an advanced setting where we can provide a custom endpoint for making the OpenAI api calls? Many local LLM tools offer servers which conform to the OpenAI endpoint schema, so this could be a nice simple way to allow people to tinker with running LLMs locally
Examples of local inference servers that would conform to the OpenAI endpoint schema
https://github.com/ggerganov/llama.cpp/tree/master/examples/server
https://www.ollama.com/blog/openai-compatibility
In the settings you could have a toggle for "Local LLM inference" and if enabled request an endpoint to send the AI requests to. This has the benefit of alleviating Reflect from the responsibility of keeping up with and supporting new models. For people that want something that works great out of the box, they can use the existing OpenAI integration. For people that like to keep things more private and explore running open source models locally, they can take the few extra steps to spin up a local OpenAI compatible server and point Reflect to using that.
cc sam c Alex MacCaw
s
sam c
open
s
sam c
planned
Alex is looking towards this as new options become available.
Juan David Garavito Espejo
I would like something like mem.ai, a master chat assistant that can look in all your notes and bring relevant info, also create notes for you if needed.
W
Working Online
Can we get more options for implementing a local vector database with Langchain to index custom documents (PDF, Folder, and others)?