Using the most basic type of memory pooling can improve the performance of your AI engine dramatically. However, your memory usage may not be optimal at first if you have one big memory pool.
Problems arise when pooling AI tasks because of the following:
All tasks are different and they have various sizes in memory.
Some tasks use practically no memory while others require hundreds of bytes.
With one pool using a maximum chunk size, hundreds of bytes are wasted for each small allocation.
In fact, you’ll typically waste (max_size - average_size) per task — which is a pretty high price to pay if you have only one very large task. Very often, most of the tasks use a small amount of memory (say, around 32 bytes each), so you can use that to your advantage.
A better strategy is to split your allocations into two pools, one small pool for the average allocation, and a large pool for everything else. To figure out the best small pool size to use, try the following:
Collect a list of all your tasks sizes and their frequency of use by logging all the memory allocations.
Plot a graph of your allocations per-size, and see where you have the peak (typically around the median).
Pick a chunk size for the small pool that covers the majority of your small allocations, like twice the median size.
You could solve this as an optimization problem to work out precisely which size saves you the most memory! But doing it by hand will allow you to use a sensible size of chunk (e.g. multiple of 4), which you can set as a guideline for all task allocations.
Later, you can even introduce more pools with multiple chunk sizes if necessary, but watch out as this can also waste memory as certain pool become rarely used. (Fewer pools is more flexible, and you don’t have to worry too much about the total pool size). So two-three chunk sizes is typically sufficient.
Once you have this in place, your memory usage will drop drastically! Do you have any strategies that you use for memory pooling? Share them in the comments.