Replies (55)
Michelle, may i ask a question. If i were to use Starts and host my own llm, what is the easiest/best/cheapest set up? Also (sorry 2 questions) will i be able to create Orac?
You can run an LLM on pretty much anything. Two easy ways, either LM Studio or Ollama. The main constraint is the size of the RAM on your machine.
There are two things AI uses:
GPU to process the prompt, and (V)RAM to hold the model and read through it constantly to construct the output.
If you don't have a good GPU, then prompt processing will take longer, if you have slow RAM, the token generated output will take longer.
The smaller your RAM, the dumber your model 😂
Small models work surprisingly well if you supply enough context and you allow web search so the model can compensate its lack of knowledge with live data from the Internet.
If you want to train an AI, however, that's a much harder job and you need something like an Nvidia DGX Spark, which is currently over £5,000 in the UK.
If its just a jnos can it also be a nostr event?
I guess 🤷♂
If you made it a Kind 1, that would be the single largest note every posted in the history of NOSTR 😂
I think easiest is to download ollama, if not wrong it basically tells you what models you can run on your system and download and set it up for you
Cheers to both of you, even though Michelle could Shirley have elaborated?!
Michelle you don't mean that 😂
Sans les mot qui bon tres bien ensemble
mapple.ai
I should ask my chat, but figured I'd just ask you. What does this change do?
I looked it up and am still a bit murky. Is this a way for them to compact their stored data about you? Seems kinda like that, assuming they're doing away with the stored conversations (deleting them?).
When you asked it to "Remember this", it stored it in a flat file, which was injected into each context window as a pre-prompt.
So then in any new context window, if it became relevant, it would know that context.
Now, the memory files are being preserved for paid subscribers, but they are being deprecated. So for now, it can still find that information, but only when asked to, like here:
Their automatic use is being deprecated, or they'll go away entirely sometime?
Sadly, I don't think my history is worth downloading... bunch of random searches, mostly
Not stored conversations, although these are only retrievable by you fully.
Just the memory file it keeps to inject into every new conversation.
I don't know exactly, but I presume they are going to rely on context retrieval only in the future.
Ok yeah. I was thinking same. The storage is probably getting out of hand, and cost of including all that pre-prompt bloated, or something.
Unrelated, are you using agent swarms at all? I don't really know what they are, but I feel like all the smart people use them.
I did for my PageHold project, but as I don't do a lot of coding, I don't really need them.
I'm more a multi agent kinda guy 😂
idk, but when i'm starting at looking at some new thing very often automatically the harness spawns a bunch of agents to do work in parallel
I don't look closely at what is going on when using coding agent, but I _think_ it's all one agent. Granted, I'm not doing anything super deep, but I suspect I'm missing out by not explicitly knowing how and when to summon them
i have three letters:
DSH
also chatbox, with deepseek. depending on what it was you wanted to do. chatbox with some nice wingdings like brave search api and a playwright PWA MCP to get around needing to pay chatbox licence but still get proper web fetches on pages that are all script generated.
anyway, ahaha. have fun playing with those "frontier" apps, they love the end user so much more than the clowns who are gonna pay their IPOs
Don't start with the 3 letter thing.
It didn't work out too well for SBF or NVK and I have premonitions for DHH 😂
I saw there was a new kimi. I've never yet tried a Chinese model yet.
As for local, I'll prob look into it in about a year...just feels right
It's really the same agent. Think of it as Schizophrenia, but for machines 😂
Yeah, but maybe run a bit differently with compartmentalized contexts? Prob best to let the model decide for me, than break things up explicitly, for me anyway
If someone wants to meet Chinese models I am in several trading telegrams that I have been invited to by them. I could ask.
I haven't done so well with the trading, but these young ladies are hot. Let me tell you.
Yes, it was a joke.
They are the same model (normally), but they each take on a different task or role. They are normally coordinated by a supervisor who handles inter-model communication and coordination.
Oh hell yeah. Too many men born, thus the surplus of hot Chinese models reaching out. That makes perfect sense!
Yeah, I hear that and it sounds like I should be doing that, or at least claiming I do if anyone asks. "Yeah, my swarm is churning on some tasks right now. I definitely have a swarm or two cooking at all times...doing things... 👀 😅 "
The fun thing is you can just ask your agent to set this up for you, you don't even have to understand it 😂
DSH is deepseek harness. the best coding harness i've ever used.
Thats bad news considering the absolute bolloks I get from the big company’s models. They have virtually unlimited money for ram and it can’t remember what it said one prompt ago.
I can give you a serious answer if you like. No seriously 😳
RAM is not the issue, it's context window. The bigger the window (the longer the conversation), the harder it is for an LLM to keep track, because everytime you enter a new bit of text, everything in that prompt gets resubmitted.
So imagine, when a friend spoke to you, they had to repeat everything they've ever said to you before, ever. Your human context keeps a vague memory of your previous conversations with a specific person, but an LLM has no memory (yes ironic I know), so it has to be told everything every time. That's what makes memory hard, not RAM but keeping the context manageable.
Long term memories used to be stored in flat files and injected into the start of any new context window so the LLM could have the illusion of understanding you, but now OpenAI have changed this to gathering a summary of recent chats and injecting those into the start of the context window.
The LLM can, if prompted, still search this flat memory file or the full context of recent chats, but it doesn't store this in the current context window by default.
Superfluous Intelligence
You Jevin' at all? All the cool kids are or maybe were (it's been a week or two), jevin'
Yeah, jev is a decision engine. For decisions it whoops LLMs in speed and accuracy, but it's not AI.
Turns out a lot of what we ask AI to do, can be done by good old fashioned IF/THEN/ELSE statements for a fraction of the cost and time 😂
That’s why it’s important to hashing and cryptographic the text so the memory of agent can compressed 3 terabyte data without any single failure of memory
Hashing is not the same as encrypting.
I can hash has the entire bible into a single 256 character hash. I can use that hash to prove that the copy of the bible you have is the same as mine, but I can't use that hash to reconstruct the data.
Encrypting the text is done for privacy, normally for transmission across networks or if stored privately.
I should have explained compression while I was at it.
Compression is the technique of replacing commonly repeating patterns with indicator codes, so, in a non real example, you could replace the word "the", with the number "1", as the word "the" appears often, reducing 3 characters to 1, saves space. In reality, the patterns are often dynamically calculated and stored as a table.
There are two types of compression lossy and non lossy. In non lossy, you have to maintain the original content verbatim.
In lossy environments, such as photography, you can replace an area of approximately the same colour red, with a pattern, or mask, that describes the approximate colour and area of similar colour. So a lot of pixels get replaced by a simple vector area description and colour palette number.
Lets just say you’re answering a person that you think has an ai account instead of a person that doesn’t and the AI literally can’t remember the last prompt in a 3 prompt( like one sentence prompts) conversation.
Here what I have in mind :
Capture >Normalize >fingerprint >Hash
Capture : photon, scan , microscope , any object
Normalise : fix lightning , scan , edit , crop so two captured Look alike
Fingerprint ; extract the uncloneable features into small size byte string
Hash and anchor : SHA 256 , then published it as Nostr event or open time stamp proof
The only questions remains is : Rust or Tyoe script ?
Say the name and I working on it
Have you tried letting a fruit fly tell you a story recently 😂
I've seen a house fly
You're a superman 😂
I am pretty cool
I'm no coder, unless you count 1980's COBOL, so the choice is yours.
But yes, go build 🫡
depending on where they are viewing 7" phone or 55" TV
use ffmpeg - lossless <5MB files are good 7" small videos.
the llm is constantly writing little programs. all the code editing is sed and grep
sed. grep, and snek. the unholy trinity
or snek sed grep if you prefer
My OpenAI data has been exported, it's 403MB uncompressed 😱
View quoted note →
