I think “pure” intelligence by itself is an algorithm. And as such an algorithm has no desire to do anything. Just like a web server has no desire to output web pages - it just does because it’s what the logic happens to make. It’s possible we need some sort of a quantum computer to achieve brain-like performance of intelligence - but I think intelligence is an algorithm specifically because LLMs exist.

Being able to pick an option is not necessarily a hallmark of cognition either - again a web server does that all the time. So I have a desire to think because it will end up getting me food and shelter and all the things I want. If we give a computer - a capability to think - it would have no desire to do so unless we executed the thinking algorithm.

Based on my research the intelligence by itself has no wants. Our wants come from being biological and evolving basic instincts and feelings like pain is bad, hunger, want to reproduce. All our other wants come from intelligence applied and predicting outcomes that maximize the biological wants. Without biological wants there’s not much to build off of.

I like this path of reasoning. Let’s assume whoever invents the AGI wants “greater good”. However malicious humans exist so how does that interact with AGI technology. Let’s say the governments started implementing some sort of universal safety net, but I am malicious and want to be have better status. What can I do?

I can start stockpiling still-scarce resources. With AGI that’s land and natural resources. I can made my own AGI that aligns with my own values and start a smearing campaign that degrades values I don’t like… but if everyone has AGI access the latter is probably not very effective. I can set up traps and bully others to show how much control I have to get people to fear me… I suppose AGI can amplify this but hopefully it amplifies the “good guys” as well so it cancels out at a societal scales but local cliques will still get hurt.

I guess land and natural resource hoarding become most viable forms of greed… I wonder if this wood make feudal societies a thing again. Did modern governments happen because intellectual labor was more profitable and so it was more profitable to share land and natural resources in order to have intellectuals work together?

Your statement implies AGI is good only for bad things (things I view as bad). That’s not the case as I see AGI being good for just doing things I don’t like doing. (AGI I’m asking about are not LLMs that we have now)

This is along the same lines I imagined it as well. And yes modern LLMs are nothing like what I’ve described. We’d need kilowatts of power per every single LLM conversation if we were to gradient descend them per conversation…. Funny enough human brain runs at like 10 watts. So I imagine there’s some huge paradigm shift from what LLMs do to what human brain does. You don’t get 1000 time efficiency boost from just putting all the world’s ram into it.

Hmm I see your points and I generally agree with them, except I don’t think we will ever collectively agree to ban AGI research before we invented it. There’s just too much juicy power for the governments to agree to ban it. As a result, it will happen eventually - maybe not in our lifetime but it will happen.

Oh I agree that “if these guys make agi” then we’re screwed. But I also don’t think Altmans and Amodeos of the world will invent AGI. However in that case the road isn’t smooth either. I dream of Star Trek like society but to get there is still going to be hard because it’s a disruption.

What about the benefits like having super cheap doctors available at your home at your schedule? You doing think the benefit is worth the change?

The problem isn’t what can be done, the problem is what is being done. An AI in hands of a good developer can make that developer better - sure.

On the other hand what actually happens is equivalent to submitting code that doesn’t compile, except it’s not quite as easy to discover. The code does compile but it often is still bogus. Maybe it doesn’t solve the issue, maybe it does but in a round-about way, etc.

Legal and economical arguments aside I’m not saying “don’t use AI” - I’m saying “if you submit AI code - verify its work before making someone else verify its work”. And that means you need to know how to verify its work - just compiling and running isn’t enough.

I have no ethical problem with that but I can’t speak for the masses. But also it’s a bit in a grey area legally since it’s unclear who owns LLM outputs.

It doesn’t run doom, it plays doom. Very different. We don’t understand the neurons well enough to translate doom to neural networks, but we can train them to play doom, due some definition of playing.

The problem is that FOSS projects that are popular already have a a high quality standard because they are developed by very smart people who’s been developing for a long time.

There is inherently a problem with current AI - It doesn’t learn.

When a new developer joins a FOSS project they first spend weeks or even months learning this project’s specific coding practices, standards, and reasoning. I’ve been programming for 20 years and have a PhD. I can’t “just start contributing” to something like KDE because I’ve never looked at its codebase. I don’t know assumptions, methodology, or challenges that are baked into the codebase. I need to spend time to learn those to be a useful contributor.

AI can’t learn unless you do gradient descend. The problem that you mention about sounding elitist is that those FOSS developers spend a good amount of their time training new contributors and everyone is proud of their work. Now you have people who are not ready to be contributors tell AI to contribute and it’s doing its best but it just can’t learn and even having a 200K token context is not enough to fit all those project specific assumptions and rhetorics in.

No in the matrix, smart machines captured humans and used their body heat to work.

Here dumb humans captured useless human tissue in hopes to run dumb machines.

Apparently Nintendo switch 2 is using the standard already, so it might go over better than Sony.

Global interpreter lock

My friend, let me tell you a story during my studies when I had to help someone find a bug in their 1383-line long main() in C… on the other hand I think Ill spare you from the gruesome details, but it took me 30 hours.

The Test part of TDD isn’t meant to encompass your whole need before developing the application. It’s function-by function based. It also forces you to not have giant functions. Let’s say you’re making a compiler. First you need to parse text. Idk what language structure we are doing yet but first we need to tokenize our steam. You write a test that inputs hello world into your tokenizer then expects two tokens back. You start implementing your tokenizer. Repeat for parser. Then you realize you need to tokenize numbers too. So you go back and make a token test for numbers.

So you don’t need to make all the tests ahead of time. You just expand at the smallest test possible.

I see your edit but in case you’re interested - a capacitor is technically a 0 resistance battery for DC.

What was that about him doing twitter’s technology policing and leaving running the company to the new CEO?

Your own article says it’s VMs. The tpm itself can be bricked. Ok that sucks. Still not persistent like you describe.