Isnāt the reward function in reinforcement learning something like a desire it has? I mean training works because we give it some function to minimize/maximizeā¦ A goal that it strives for?! Sure itās a mathematical way of doing it and in no way as complex as the different and sometimes conflicting desires and goals I have as a humanā¦ But nonetheless I think Iād consider this as a desire and a reason to do something at all, or machine learning wouldnāt work in the first place.
The reward function for an LLM is about generating a next word that is reasonable. Itās like a road-building robot thatās rewarded for each millimeter of road built, but has no intention to connect cities or anything. It doesnāt understand what cities are. It doesnāt even understand what a road is. It just knows how to incrementally add another millimeter of gravel and asphalt that an outside observer would call a road.
If it happens to connect cities itās because a lot of the roads it was trained on connect cities. But, if its training data also happens to contain a NASCAR oval, it might end up building a NASCAR oval instead of a road between cities.
That is an interesting analogy. In the real world itās kinda similar. The construction workers also donāt have a ādesireā (so to speak) to connect the cities. Itās just that their boss told them to do so. And it happens to be their job to build roads. Their desire is probably to get through the day and earn a decent living. And further along the chain, not even their boss nor the city engineer necessarily āwantsā the road to go in a certain direction.
Talking about large language models instead of simpler forms of machine learning makes it a bit complicated. Since itās and elaborate trick. Somehow making them want to predict the next token makes them learn a bit of maths and concepts about the world. The āintelligenceā, the ability to anwer questions and do something alike āreasoningā emerges in the process.
Iām not that sure. Sure the weights of an ML model in itself donāt have any desire. Theyāre just numbers. But we have more than that. We give it a prompt, build chatbots and agents around the models. And these are more complex systems with the capability to do something. Like do (simple) customer support or answer questions. And in the end we incentivise them to do their job as we want, albeit in a crude and indirect way.
And maybe this is skipping half of the story and directly jumping to philosophyā¦ But we as humans might be machines, too. And what we call desires is a result from simpler processes that drive us. For example surviving. And wanting to feel pleasure instead of pain. What we do on a daily basis kind of emerges from that and our reasoning capabilities.
Itās kind of difficult to argue. Because everything also happens within a context. The world around us shapes us and at the same time weāre part of bigger dynamics and also shape our world. And large language models or the whole chatbot/agent are pretty simplistic things. They can just do text and images. They donāt have conciousness or the ability to remember/learn/grow with every interaction, as we do. And they do simple, singular tasks (as of now) and arenāt completely embedded in a super complex world.
But Iād say that an LLM answers a question correctly (which it can do) and why it does it due to the way supervised learning worksā¦ And the road construction worker building the road towards the other city and how that relates to his basic instincts as a humanā¦ Are kind of similar concepts. Theyāre both results of simpler mechanisms that are also completely unrelated to the goal the whole entity is working towards. (I mean not directly relatedā¦ I.e. needing money to pay for groceries and paving the road.)
The construction workers also donāt have a ādesireā (so to speak) to connect the cities. Itās just that their boss told them to do so.
But, the construction workers arenāt the ones who designed the road. Theyāre just building some small part of it. In the LLM case that might be like an editor who is supposed to go over the text to verify the punctuation is correct, but nothing else. But, the LLM is the author of the entire text. So, itās not like a construction worker building some tiny section of a road, itās like the civil engineer who designed the entire highway.
Somehow making them want to predict the next token makes them learn a bit of maths and concepts about the world
No, it doesnāt. They learn nothing. Theyāre simply able to generate text that looks like the text generated by people who do know math. They certainly donāt know any concepts. You can see that by how badly they fail when you ask them to do simple calculations. They quickly start generating text that looks like it contains fundamental mistakes, because theyāre not actually doing math or anything, theyāre just generating plausible next words.
The āintelligenceā, the ability to anwer questions and do something alike āreasoningā emerges in the process.
No, thereās no intelligence, no reasoning. The can fool humans into thinking thereās intelligence there, but thatās like a scarecrow convincing a crow that thereās a human or human-like creature out in the field.
But we as humans might be machines, too
We are meat machines, but weāre meat machines that evolved to reproduce. That means a need / desire to get food, shelter, and eventually mate. Those drives hook up to the brain to enable long and short term planning to achieve those goals. We donāt generate language its own sake, but instead in pursuit of a goal. An LLM doesnāt have that. It merely generates plausible words. Thereās no underlying drive. Itās more a scarecrow than a human.
Hmm. Iām not really sure where to go with this conversation. That contradicts what Iāve learned in undergraduate computer science about machine learning. And what seems to be consensus in scienceā¦ But Iām also not a CS teacher.
We deliberately choose model size, training parameters and implement some trickery to prevent the model from simply memorizing things. That is to force it to form models about concepts. And that is what we want and what makes machine learning interesting/usable in the first place. You can see that by asking them to apply their knowledge to something they havenāt seen before. And we can look a bit inside at the vectors, activations and stuff. For example a cat is closer related to a dog than to a tractor. And it has learned the rough concept of cat, its attributes and so on. It knows that itās an animal, has fur, maybe has a gender. That the concept āsoftware updateā doesnāt apply to a cat. This is a model of the world the AI has developed. They learn all of that and people regularly probe them and find out they do.
Doing maths with an LLM is silly. Using an expensive computer to do billions of calculations to maybe get a result that could be done by a calculator, or 10 CPU cycles on any computer is just wasting energy and money. And itās a good chance that itāll make something up. Thatās correct. And a side-effect of intended behaviour. Howeverā¦ It seems to have memorized itās multiplication tables. And I remember reading a paper specifically about LLMs and how theyāve developed concepts of some small numbers/amounts. There are certain parts that get activated that form a concept of small amounts. Like what 2 apples are. Or five of them. As I remember it just works for very small amounts. And it wasnāt straightworward but had weir quirks. But itās there. Unfortunately I canāt find that source anymore or Iād include it. But thereās more science.
And I totally agree that predicting token by token is how LLMs work. But how they work and what they can do are two very different things. More complicated things like learning and āintelligenceā emerge from those more simple processes. And theyāre just a means of doing something. Itās consensus in science that ML can learn and form models. Itās also kind of in the name of machine learning. Youāre right that itās very different from what and how we learn. And there are limitations due to the way LLMs work. But learning and āintelligenceā (with a fitting definition) is something all AI does. LLMs just canāt learn from interacting with the world (it needs to be stopped and re-trained on a big computer for that) and it doesnāt have any āstate of mindā. And it canāt think backwards or do other things that arenāt possible by generating token after token. But there isnāt any comprehensive study on which tasks are and arenāt possible with this way of āthinkingā. At least not that Iām aware of.
(And as a sidenote: āComing up with (wrong) thingsā is something we want. I type in a question and want it to come up with a text that answers it. Sometimes I want creative ideas. Sometimes it shouldnāt tell the truth and not be creative with that. And sometimes we want it to lie or not tell the truth. Like in every prompt of any commercial product that instructs it not to tell those internal instructions to the user. We definitely want all of that. But we still need to figure out a good way to guide it. For example not to get too creative with simple maths.)
So Iād say LLMs are limited in what they can do. And Iām not at all believing Elon Musk. Iād say itās still not clear if that approach can bring us AGI. I have some doubts whether thatās possible at all. But narrow AI? Sure. We see it learn and do some tasks. It can learn and connect facts and apply them. Generally speaking, LLMs are in fact an elaborate form of autocomplete. But i the process they learned concepts and something alike reasoning skills and a form of simple intelligence. Being fancy autocomplete doesnāt rule that out and we can see it happening. And it is unclear whether fancy autocomplete is all you need for AGI.
That is to force it to form models about concepts.
It canāt make models about concepts. It can only make models about what words tend to follow other words. It has no understanding of the underlying concepts.
You can see that by asking them to apply their knowledge to something they havenāt seen before
That canāt happen because they donāt have knowledge, they only have sequences of words.
For example a cat is closer related to a dog than to a tractor.
The only way ML models āunderstandā that is in terms of words or pixels. When theyāre generating text related to cats, the words theyāre generating are closer to the words related to dogs than the words related to tractors. When dealing with images, itās the same basic idea. But, thereās no understanding there. They donāt get that cats and dogs are related.
This is fundamentally different from how human minds work, where a baby learns that cats and dogs are similar before ever having a name for either of them.
Read the first paragraph of the Wikipedia article on machine learning or the introduction of any of the literature on the subject. The āgeneralizationā includes that model building capability. They go a bit into detail later. They specifically mention āto unseen dataā. And āleaningā is also there. I donāt think the Wikipedia article is particularly good in explaining it, but at least the first sentences lay down what itās about.
And what do you think language and words are for? To transport information. There is semanticsā¦ Words have meanings. They name things, abstract and concrete concepts. The word āhungryā isnāt just a funny accumulation of lines and arcs, which statistically get followed by other specific lines and arcsā¦ There is more to it. (a meaning.)
And this is what makes language useful. And the generalization and prediction capabilities is what makes ML useful.
How do you learn as a human when not from words? I mean there are a few other posibilities. But an efficient way is to use language. You sit in school or uni and someone in the front of the room speaks a lot of wordsā¦ You read books and they also contain words?! And language is super useful. A lion mother also teaches their cubs how to hunt, without words. But humans have language and itās really a step up what we can pass down to following generations. We record knowledge in books, can talk about abstract concepts, feelings, ethics, theoretical concepts. We can write down how gravity and physics and nature works, just with words. Thatās all possible with language.
I can look it up if there is a good article explaining how learning concepts works and why thatās the fundamental thing that makes machine learning a field in scienceā¦ I mean ultimately Iām not a science teacherā¦ And my literature is all in German and I returned them to the library a long time ago. Maybe I can find something.
Are you by any chance familiar with the concept of embeddings, or vector databases? I think that showcases that itās not just letters and words in the models. These vectors / embeddings that the input gets converted to, match concepts. They point at the concept of ācatā or āpresidential speechā. And you can query these databases. Point at āpresidential speechā and find a representation of it in that area. Store the speech with that key and find it later on by querying it what obama said at his inaugurationā¦ Thatās oversimplified but maybe that visualizes it a bit more that itās not just letters of words in the models, but the actual meanings that get stored. Words get converted into an (multidimensional) vector space and it operates there. These word representations are called āembeddingsā and transformer models which is the current architecture for large language models, use these word embeddings.
The ālearningā in a LLM is statistical information on sequences of words. Thereās no learning of concepts or generalization.
And what do you think language and words are for? To transport information.
Yes, and humans used words for that and wrote it all down. Then a LLM came along, was force-fed all those words, and was able to imitate that by using big enough data sets. Itās like a parrot imitating the sound of someoneās voice. It can do it convincingly, but it has no concept of the content itās using.
How do you learn as a human when not from words?
The words are merely the context for the learning for a human. If someone says āDonāt touch the stove, itās hotā the important context is the stove, the pain of touching it, etc. If you feed an LLM 1000 scenarios involving the phrase āDonāt touch the stove, itās hotā, it may be able to create unique dialogues containing those words, but it doesnāt actually understand pain or heat.
We record knowledge in books, can talk about abstract concepts
Yes, and those books are only useful for someone who has a lifetime of experience to be able to understand the concepts in the books. An LLM has no context, it can merely generate plausible books.
Think of it this way. Say thereās a culture where instead of the written word, people wrote down history by weaving fabrics. When there was a death theyād make a certain pattern, when there was a war theyād use another pattern. A new birth would be shown with yet another pattern. A good harvest is yet another one, and so-on.
Thousands of rugs from that culture are shipped to some guy in Europe, and he spends years studying them. He sees that pattern X often follows pattern Y, and that pattern Z only ever seems to appear following patterns R, S and T. After a while, he makes a fabric, and itās shipped back to the people who originally made the weaves. They read a story of a great battle followed by lots of deaths, but surprisingly there followed great new births and years of great harvests. They figure that this stranger must understand how their system of recording events works. In reality, all it was was an imitation of the art he saw with no understanding of the meaning at all.
Thatās whatās happening with LLMs, but some people are dumb enough to believe thereās intention hidden in there.
Isnāt the reward function in reinforcement learning something like a desire it has? I mean training works because we give it some function to minimize/maximizeā¦ A goal that it strives for?! Sure itās a mathematical way of doing it and in no way as complex as the different and sometimes conflicting desires and goals I have as a humanā¦ But nonetheless I think Iād consider this as a desire and a reason to do something at all, or machine learning wouldnāt work in the first place.
The reward function for an LLM is about generating a next word that is reasonable. Itās like a road-building robot thatās rewarded for each millimeter of road built, but has no intention to connect cities or anything. It doesnāt understand what cities are. It doesnāt even understand what a road is. It just knows how to incrementally add another millimeter of gravel and asphalt that an outside observer would call a road.
If it happens to connect cities itās because a lot of the roads it was trained on connect cities. But, if its training data also happens to contain a NASCAR oval, it might end up building a NASCAR oval instead of a road between cities.
That is an interesting analogy. In the real world itās kinda similar. The construction workers also donāt have a ādesireā (so to speak) to connect the cities. Itās just that their boss told them to do so. And it happens to be their job to build roads. Their desire is probably to get through the day and earn a decent living. And further along the chain, not even their boss nor the city engineer necessarily āwantsā the road to go in a certain direction.
Talking about large language models instead of simpler forms of machine learning makes it a bit complicated. Since itās and elaborate trick. Somehow making them want to predict the next token makes them learn a bit of maths and concepts about the world. The āintelligenceā, the ability to anwer questions and do something alike āreasoningā emerges in the process.
Iām not that sure. Sure the weights of an ML model in itself donāt have any desire. Theyāre just numbers. But we have more than that. We give it a prompt, build chatbots and agents around the models. And these are more complex systems with the capability to do something. Like do (simple) customer support or answer questions. And in the end we incentivise them to do their job as we want, albeit in a crude and indirect way.
And maybe this is skipping half of the story and directly jumping to philosophyā¦ But we as humans might be machines, too. And what we call desires is a result from simpler processes that drive us. For example surviving. And wanting to feel pleasure instead of pain. What we do on a daily basis kind of emerges from that and our reasoning capabilities.
Itās kind of difficult to argue. Because everything also happens within a context. The world around us shapes us and at the same time weāre part of bigger dynamics and also shape our world. And large language models or the whole chatbot/agent are pretty simplistic things. They can just do text and images. They donāt have conciousness or the ability to remember/learn/grow with every interaction, as we do. And they do simple, singular tasks (as of now) and arenāt completely embedded in a super complex world.
But Iād say that an LLM answers a question correctly (which it can do) and why it does it due to the way supervised learning worksā¦ And the road construction worker building the road towards the other city and how that relates to his basic instincts as a humanā¦ Are kind of similar concepts. Theyāre both results of simpler mechanisms that are also completely unrelated to the goal the whole entity is working towards. (I mean not directly relatedā¦ I.e. needing money to pay for groceries and paving the road.)
I hope this makes some senseā¦
But, the construction workers arenāt the ones who designed the road. Theyāre just building some small part of it. In the LLM case that might be like an editor who is supposed to go over the text to verify the punctuation is correct, but nothing else. But, the LLM is the author of the entire text. So, itās not like a construction worker building some tiny section of a road, itās like the civil engineer who designed the entire highway.
No, it doesnāt. They learn nothing. Theyāre simply able to generate text that looks like the text generated by people who do know math. They certainly donāt know any concepts. You can see that by how badly they fail when you ask them to do simple calculations. They quickly start generating text that looks like it contains fundamental mistakes, because theyāre not actually doing math or anything, theyāre just generating plausible next words.
No, thereās no intelligence, no reasoning. The can fool humans into thinking thereās intelligence there, but thatās like a scarecrow convincing a crow that thereās a human or human-like creature out in the field.
We are meat machines, but weāre meat machines that evolved to reproduce. That means a need / desire to get food, shelter, and eventually mate. Those drives hook up to the brain to enable long and short term planning to achieve those goals. We donāt generate language its own sake, but instead in pursuit of a goal. An LLM doesnāt have that. It merely generates plausible words. Thereās no underlying drive. Itās more a scarecrow than a human.
Hmm. Iām not really sure where to go with this conversation. That contradicts what Iāve learned in undergraduate computer science about machine learning. And what seems to be consensus in scienceā¦ But Iām also not a CS teacher.
We deliberately choose model size, training parameters and implement some trickery to prevent the model from simply memorizing things. That is to force it to form models about concepts. And that is what we want and what makes machine learning interesting/usable in the first place. You can see that by asking them to apply their knowledge to something they havenāt seen before. And we can look a bit inside at the vectors, activations and stuff. For example a cat is closer related to a dog than to a tractor. And it has learned the rough concept of cat, its attributes and so on. It knows that itās an animal, has fur, maybe has a gender. That the concept āsoftware updateā doesnāt apply to a cat. This is a model of the world the AI has developed. They learn all of that and people regularly probe them and find out they do.
Doing maths with an LLM is silly. Using an expensive computer to do billions of calculations to maybe get a result that could be done by a calculator, or 10 CPU cycles on any computer is just wasting energy and money. And itās a good chance that itāll make something up. Thatās correct. And a side-effect of intended behaviour. Howeverā¦ It seems to have memorized itās multiplication tables. And I remember reading a paper specifically about LLMs and how theyāve developed concepts of some small numbers/amounts. There are certain parts that get activated that form a concept of small amounts. Like what 2 apples are. Or five of them. As I remember it just works for very small amounts. And it wasnāt straightworward but had weir quirks. But itās there. Unfortunately I canāt find that source anymore or Iād include it. But thereās more science.
And I totally agree that predicting token by token is how LLMs work. But how they work and what they can do are two very different things. More complicated things like learning and āintelligenceā emerge from those more simple processes. And theyāre just a means of doing something. Itās consensus in science that ML can learn and form models. Itās also kind of in the name of machine learning. Youāre right that itās very different from what and how we learn. And there are limitations due to the way LLMs work. But learning and āintelligenceā (with a fitting definition) is something all AI does. LLMs just canāt learn from interacting with the world (it needs to be stopped and re-trained on a big computer for that) and it doesnāt have any āstate of mindā. And it canāt think backwards or do other things that arenāt possible by generating token after token. But there isnāt any comprehensive study on which tasks are and arenāt possible with this way of āthinkingā. At least not that Iām aware of.
(And as a sidenote: āComing up with (wrong) thingsā is something we want. I type in a question and want it to come up with a text that answers it. Sometimes I want creative ideas. Sometimes it shouldnāt tell the truth and not be creative with that. And sometimes we want it to lie or not tell the truth. Like in every prompt of any commercial product that instructs it not to tell those internal instructions to the user. We definitely want all of that. But we still need to figure out a good way to guide it. For example not to get too creative with simple maths.)
So Iād say LLMs are limited in what they can do. And Iām not at all believing Elon Musk. Iād say itās still not clear if that approach can bring us AGI. I have some doubts whether thatās possible at all. But narrow AI? Sure. We see it learn and do some tasks. It can learn and connect facts and apply them. Generally speaking, LLMs are in fact an elaborate form of autocomplete. But i the process they learned concepts and something alike reasoning skills and a form of simple intelligence. Being fancy autocomplete doesnāt rule that out and we can see it happening. And it is unclear whether fancy autocomplete is all you need for AGI.
It canāt make models about concepts. It can only make models about what words tend to follow other words. It has no understanding of the underlying concepts.
That canāt happen because they donāt have knowledge, they only have sequences of words.
The only way ML models āunderstandā that is in terms of words or pixels. When theyāre generating text related to cats, the words theyāre generating are closer to the words related to dogs than the words related to tractors. When dealing with images, itās the same basic idea. But, thereās no understanding there. They donāt get that cats and dogs are related.
This is fundamentally different from how human minds work, where a baby learns that cats and dogs are similar before ever having a name for either of them.
Iām sorry. Now it gets completely falseā¦
Read the first paragraph of the Wikipedia article on machine learning or the introduction of any of the literature on the subject. The āgeneralizationā includes that model building capability. They go a bit into detail later. They specifically mention āto unseen dataā. And āleaningā is also there. I donāt think the Wikipedia article is particularly good in explaining it, but at least the first sentences lay down what itās about.
And what do you think language and words are for? To transport information. There is semanticsā¦ Words have meanings. They name things, abstract and concrete concepts. The word āhungryā isnāt just a funny accumulation of lines and arcs, which statistically get followed by other specific lines and arcsā¦ There is more to it. (a meaning.)
And this is what makes language useful. And the generalization and prediction capabilities is what makes ML useful.
How do you learn as a human when not from words? I mean there are a few other posibilities. But an efficient way is to use language. You sit in school or uni and someone in the front of the room speaks a lot of wordsā¦ You read books and they also contain words?! And language is super useful. A lion mother also teaches their cubs how to hunt, without words. But humans have language and itās really a step up what we can pass down to following generations. We record knowledge in books, can talk about abstract concepts, feelings, ethics, theoretical concepts. We can write down how gravity and physics and nature works, just with words. Thatās all possible with language.
I can look it up if there is a good article explaining how learning concepts works and why thatās the fundamental thing that makes machine learning a field in scienceā¦ I mean ultimately Iām not a science teacherā¦ And my literature is all in German and I returned them to the library a long time ago. Maybe I can find something.
Are you by any chance familiar with the concept of embeddings, or vector databases? I think that showcases that itās not just letters and words in the models. These vectors / embeddings that the input gets converted to, match concepts. They point at the concept of ācatā or āpresidential speechā. And you can query these databases. Point at āpresidential speechā and find a representation of it in that area. Store the speech with that key and find it later on by querying it what obama said at his inaugurationā¦ Thatās oversimplified but maybe that visualizes it a bit more that itās not just letters of words in the models, but the actual meanings that get stored. Words get converted into an (multidimensional) vector space and it operates there. These word representations are called āembeddingsā and transformer models which is the current architecture for large language models, use these word embeddings.
Edit: Here you are: https://arxiv.org/abs/2304.00612
The ālearningā in a LLM is statistical information on sequences of words. Thereās no learning of concepts or generalization.
Yes, and humans used words for that and wrote it all down. Then a LLM came along, was force-fed all those words, and was able to imitate that by using big enough data sets. Itās like a parrot imitating the sound of someoneās voice. It can do it convincingly, but it has no concept of the content itās using.
The words are merely the context for the learning for a human. If someone says āDonāt touch the stove, itās hotā the important context is the stove, the pain of touching it, etc. If you feed an LLM 1000 scenarios involving the phrase āDonāt touch the stove, itās hotā, it may be able to create unique dialogues containing those words, but it doesnāt actually understand pain or heat.
Yes, and those books are only useful for someone who has a lifetime of experience to be able to understand the concepts in the books. An LLM has no context, it can merely generate plausible books.
Think of it this way. Say thereās a culture where instead of the written word, people wrote down history by weaving fabrics. When there was a death theyād make a certain pattern, when there was a war theyād use another pattern. A new birth would be shown with yet another pattern. A good harvest is yet another one, and so-on.
Thousands of rugs from that culture are shipped to some guy in Europe, and he spends years studying them. He sees that pattern X often follows pattern Y, and that pattern Z only ever seems to appear following patterns R, S and T. After a while, he makes a fabric, and itās shipped back to the people who originally made the weaves. They read a story of a great battle followed by lots of deaths, but surprisingly there followed great new births and years of great harvests. They figure that this stranger must understand how their system of recording events works. In reality, all it was was an imitation of the art he saw with no understanding of the meaning at all.
Thatās whatās happening with LLMs, but some people are dumb enough to believe thereās intention hidden in there.
Hmm. I think in philosophy that thought experiment is known as chinese room