The Call to run your own LLM

 I am not really calling anyone to do anything but it should be a wake up to start looking into running your own LLM. Why am I bringing this up. There was an article that has made the rounds in the software engineering circle called " AI is making a generation of Illiterate Programmers". Here is the link.[https://nmn.gl/blog/ai-illiterate-programmers]()

Now Its a really good article and comes up with good points, such has the sensation of always going to the LLM to troubleshoot or not being able to read the error state. The skills you develop over the years entropy(if I am using that word in this context correctly. Basically if you were going to the gym and you stop going for a long time, your muscles start to lose there build up and decay in a sense). In a lot of ways, we are in a stage of this technology were we recognize that its doing something that is not great and we need to find a balance. However, that is not the point of my post. The point of the post is in the beginning where they state that chatgpt was down for hours and they felt like they couldn't do there job. This is where my point is. At some point users should learn to run there own LLM. Now, its not cheap, fast or models are not that good, but we definitely are in a decent point where its time to try it out. Primeagen does make this point in his video related to this article. This hit home for me. Right now I rely on 2 LLMs ChatGPT for a lot of generic and professional usage. Along with Googles NotebookLM(Ill add gemini). To me these two do what I need for my needs. ChatGPT is one I pay for each month(not the pro version), but I do notice the fact that when it goes down it does hit hard. I don't rely on it to much I can still troubleshoot for my job well but there are certain things that I am realizing I do go to ChatGPT it does act as my replacement to a search engine(sometimes, I still google fu like a pro), but I find myself clawing at being upset when its down. This is the moment where I know changes need to be made.

First, people really need to learn how to document. As in my previous post about documentation no one does it well, but the internet is a glorified documentation hub. I won't say its a library cause libraries are organized a certain way, and there are other things that make libraries unique in my opinion, but internet is a giant wiki documentation hub. With that being said, you should learn how to do a better job at this, to the second part which is Run your documentation through an LLM(locally of course). This is not a promotion of any RAG style tools or anything but that are many out there to use to run against your documentation. Really what an LLM should be is a way to continue training your brain to problem solve. When you solve the problem, you should document and add it to your personal knowledgebase. This is what everyone does now with Stack overflow, forums etc. When we ran into a problem we spend hours print statements, taking screenshots and posting to find out the issue. It seems people are trying to use LLM in a similar fashion but they are not quite there yet. However, if you collect your own documentation, it can help over time to develop your own skillset. That does not mean you don't try to read and understand the trouble issues but you use the LLM especially your personal one as a way of self talk. Some programmers have a ducky as a stand in as they talk through a problem. LLMs gives you that plus feedback but an LLM that is generic is the same as doing a google search, you my focus on the first page of results without ever realizing the correct info my be on other pages because of whatever algorithm google implemented. This is something Primagen and TJ both youtubers talked about recently. A local LLM that you build its knowledge on from your experience might be useful when you supplment with some external factors. That external factor should be the same way a person would learn from there mistakes. Documenting what worked so they can go back and learning to read it. So the AI also has a way to understand. Usually documenting your process to your local LLM(even cloud based ones) can help understand code.

Third and fourth is cost and knowledge of the process. The cost part is that you will end up spending money for the LLM(s) plus whatever wrapper to work with your llm. Pay for chatgpt plus and then cursor. Then you might buy some other tool. By the end of it, you are probably spending close to 100-400 a month on LLMs that might go down(using hyperbole here it might not cost that much). If you are spending that type of money or close to it, you can build a rig, with an LLM, with a decent llm wrapper and electricity. Mind you that overtime, you will pay your rig down and recover that cost in time since if cloud LLMs go down, yours is still accessible, wrappers, and open source LLMs improve, but you are not covering that cost for those companies(unless you help donate to some of these folks), and your not covering compute power since you can shut off your own machine when not in use. I am not saying it is always least expensive running your own stuff. Your time troubleshooting, setting up etc are all there but you gain the knowledge of how these things work and that time is valuable in the long term. That is another benefit.

I am not saying we are all there yet. I am still mostly paying with localLLM vs using them outright. I am experimenting to see what works best. I eventually will move there and I have a few tools on a few machines where I will slowly start offloading some of my data off these platforms. In a world where this technology is ever evolving, I think some folks want to avoid having gatekeepers of data. I believe most if not all companies will have there own trained LLM locally ran vs hosting them in the cloud because company data and company IP will be much better safeguarded locally vs in the cloud. Companies are using cloud now but usually that is because the cost of that knowledge is worth it at this time, as not only employees began to use and understand how to use the tool, but it gives time for the data specialist and architects to design how that data should be formatted, shown and understood. Then at a flip, all that will start to come in house where the data can be specialized upon.