I mean the real challenge is you still need to program all the different states. The LLM can help generate narrative, but that doesn’t change interactivness with the world. A great example is the Skyrim LLM mod a while back. In that user’s would “convince” an NPC to join them, but there would need to be programming behind the scenes that allows people to be recruited and the LLM dialouge would need to know to trigger that. There is the possibility of some interesting things, but it’s going to be hard to work it out.
It would be difficult, yes. I’m a software developer myself and have been working with LLMs on personal projects recently, so I’ve got some context on the challenges involved.
The hype around LLMs is obviously all “Yeah just throw AI at the problem! AI can do it!” but the reality is that you will always need a good amount of normal coding to wrap around that and make the LLM inputs and outputs sane and interoperable with the rest of your system. So I’m very aware.
My real wonder is that with an appropriate implementation, how much of the classical aspects of the game could you ultimately and eventually move to LLMs, which is what the patent seems to be suggesting.
For example, if you used LLM only for character dialogue and nothing else, it would go something like this:
You talk to an NPC and insult them
The convo is assessed and it tweaks some hidden classically programmed reputation and faction variables
You go to the base of a faction associated
Those variables determine the faction is hostile
But you could potentially use LLMs to manage more aspects directly, and that could look like this:
You talk to an NPC and insult them
An LLM summary of your actions is written to a world log
You go to the base of the faction associated
The controller LLM parses the entire world log for your actions relevant to the faction and determines the result as hostility, including extracting reasoning for that which members of the faction can confront you on if spoken to
Now that’s already a lot of work and the only bit of classic programming we really took out is how the rep system is managed. But we gained some flexibility in that the source of your relationship with the faction could come from any action anywhere, including ones the game designers never even dreamed up, not just certain things which were pre-known to update it.
Where decisions actually interact with the game world will always need to be classically programmed (like being hostile and what that means and how it causes the characters to act, do they shoot you, what it means to ‘shoot’ and ‘move’ etc) and there will need to be a way to interface with that, but LLMs could introduce some level of flexibility in places where that wasn’t possible before.
A reaulting problem though is that the more you give to LLMs, the more the entire thing is likely to unravel and become incoherent, without doing even more work to prevent that, and there will still be cracks.
Is it ultimately feasible? I don’t know, but it will be interesting for whoever gets to try.
Yeah, that’s exactly how I see it too. The biggest challenge to me is even if you can do it, can you make it feel reasonable and consistent.
In games today you know that there are good/bad options in a given scenario. With LLMs anything you do could cause an impact to a faction.
The other big issue is you run into user input. How many players want to be typing or speaking full conversations? Maybe it works for some games, but you only really get the full “flexibility” if users are fully interacting. This could greatly limit the games where LLMs could make an impact.
You can start to monitor actions, and not dialouge, and process events/actions through an LLM, but not sure how much LLMs would make that easier versus just programming those interactions.
Another big one is generation of inappropriate or sexual content. No publisher will dare to ship something when the next week there’s going to be videos all over social media of their game characters saying and doing wildly inappropriate stuff (as hilarious as that would be for us as players!)
I mean the real challenge is you still need to program all the different states. The LLM can help generate narrative, but that doesn’t change interactivness with the world. A great example is the Skyrim LLM mod a while back. In that user’s would “convince” an NPC to join them, but there would need to be programming behind the scenes that allows people to be recruited and the LLM dialouge would need to know to trigger that. There is the possibility of some interesting things, but it’s going to be hard to work it out.
It would be difficult, yes. I’m a software developer myself and have been working with LLMs on personal projects recently, so I’ve got some context on the challenges involved.
The hype around LLMs is obviously all “Yeah just throw AI at the problem! AI can do it!” but the reality is that you will always need a good amount of normal coding to wrap around that and make the LLM inputs and outputs sane and interoperable with the rest of your system. So I’m very aware.
My real wonder is that with an appropriate implementation, how much of the classical aspects of the game could you ultimately and eventually move to LLMs, which is what the patent seems to be suggesting.
For example, if you used LLM only for character dialogue and nothing else, it would go something like this:
But you could potentially use LLMs to manage more aspects directly, and that could look like this:
Now that’s already a lot of work and the only bit of classic programming we really took out is how the rep system is managed. But we gained some flexibility in that the source of your relationship with the faction could come from any action anywhere, including ones the game designers never even dreamed up, not just certain things which were pre-known to update it.
Where decisions actually interact with the game world will always need to be classically programmed (like being hostile and what that means and how it causes the characters to act, do they shoot you, what it means to ‘shoot’ and ‘move’ etc) and there will need to be a way to interface with that, but LLMs could introduce some level of flexibility in places where that wasn’t possible before.
A reaulting problem though is that the more you give to LLMs, the more the entire thing is likely to unravel and become incoherent, without doing even more work to prevent that, and there will still be cracks.
Is it ultimately feasible? I don’t know, but it will be interesting for whoever gets to try.
Yeah, that’s exactly how I see it too. The biggest challenge to me is even if you can do it, can you make it feel reasonable and consistent.
In games today you know that there are good/bad options in a given scenario. With LLMs anything you do could cause an impact to a faction.
The other big issue is you run into user input. How many players want to be typing or speaking full conversations? Maybe it works for some games, but you only really get the full “flexibility” if users are fully interacting. This could greatly limit the games where LLMs could make an impact.
You can start to monitor actions, and not dialouge, and process events/actions through an LLM, but not sure how much LLMs would make that easier versus just programming those interactions.
There’s all sorts of problems and roadblocks.
Another big one is generation of inappropriate or sexual content. No publisher will dare to ship something when the next week there’s going to be videos all over social media of their game characters saying and doing wildly inappropriate stuff (as hilarious as that would be for us as players!)
Haha, that’s a really good point.