Memory usage, garbage collecting, and unloading inactive cells
Current state
- Right after starting/loading a game only the active cell is loaded to RAM. During the game the engine loads to RAM the cells that are close to the player. There is no mechanism for unloading cells, so RAM usage monotonically increases until the game is restarted/reloaded.
- Removing an item simply sets
count
to 0 without removing itsLiveCellRef
.
Problems
- OpenMW can crash with "not enough memory" if too many cells were visited in one game session (can be illustrated by an mwscript that teleports the player to every interior cell one by one).
- Repeatedly adding and removing something to/from a container can cause unlimited memory usage increase even if only a single item exists at a time. It is quite bad for some scripting use cases.
- In multiplayer both problems will become even bigger because several players can visit more cells at a time, and restarting the server to unload inactive cells is inconvenient (not applicable to the current state of tes3mp because now tes3mp server doesn't work with mwworld at all).
Goal
We need a mechanism to unload inactive cells and free deleted instances.
The main difficulty is that MWWorld::Ptr
holds raw pointer to LiveCellRef
and if the LiveCellRef
was removed but the Ptr is still stored somewhere, every access to it can cause a segfault.
Proposal
LuaManager already has a mechanism to work with instances that may become unavailable.
It consists of
-
MWLua::ObjectRegistry
- a mappingESM::RefNum
->MWWorld::Ptr
for all active Ptrs. For instances without RefNum it adds a generated one. SoRefNum
is a unique identifier. -
MWLua::Object
- a wrapper aroundMWWorld::Ptr
that can update the Ptr (using ObjectRegistry) if the correspondingLiveCellRef
was unloaded, loaded again, or moved to another cell or container.
It looks reasonable to detach this mechanism from Lua and use it in every place where we store Ptrs between frames.
I propose to add a new class MWWorld::WorldModel
that combines MWLua::WorldView
, MWLua::ObjectRegistry
, MWWorld::Cells
.
WorldModel
should:
- Have memory ownership of all
CellStore
andContainerStore
. - Handle requests to load something to memory.
- Handle requests to add/remove instances to cells/containers or move something from one cell to another.
- Make decisions on which cells to unload from memory if a memory budget is exhausted.
- Manage the global mapping
ESM::RefNum
->MWWorld::Ptr
for all instances loaded to RAM. - (in more distant future) Manage sqlite database to search through all (including those that are not in memory) instances.
WorldModel
shouldn't do anything outside its responsibility. In particular:
- It shouldn't have any game logic.
- It shouldn't interact with rendering.
- It shouldn't modify
ESMStore
. Managing instances is a huge task on its own, better have it separated from managing records.
Detailed plan
-
[Done !2498 (merged), !2535 (merged)] Rename
MWWorld::Cells
->MWWorld::WorldModel
; MakeWorldModel
andMWWorld::Scene
to be accessible throughEnvironment
. -
[!2549 (merged)] Merge
MWLua::ObjectRegistry
intoMWWorld::WorldModel
. -
[!2803 (merged)] Extract non-lua-related part of
MWLua::Object
to apps/mwworld and call itMWWorld::SafePtr
. It is a wrapper aroundMWWorld::Ptr
that updates the Ptr if the LiveCellRef it refers to was moved to another cell or unloaded from memory. -
Change how the mapping
ESM::RefNum
->MWWorld::Ptr
is populated. Currently it is updated when Ptr is added to the scene, instead of it we need to update it immediately when something is added to a cell of container. -
Switch from
Ptr
toSafePtr
all usages of everything that can be potentially unloaded. -
Implement unloading inactive cells.