Memory Allocation Stress Tests

The problem

Memory allocation failure in some systems can be problematic in c and c++ even if 'proper programming procedures' are used.

The examples

svn source:

• source:trunk/alloctests/trunk

svn repository at:

• http://opensource.jdkoftinoff.com/jdks/svn/trunk/alloctests/trunk

Building

On a posix-like system, do the following:

svn co http://opensource.jdkoftinoff.com/jdks/svn/trunk/alloctests/trunk alloctests
cd alloctests
mkdir b
cd b
../configure --target-platform-posix=1
make
# results get put into build/tests
# if you are brave, run make test. It may freeze your system.
make test

See the MagicMakefileV3 page for info on how configure and magic.mak works.

The failures

The following test programs will typically fail due of exhaustion of virtual memory space:

• source:trunk/alloctests/trunk/tests/test_malloc1.c
• source:trunk/alloctests/trunk/tests/test_new1.cpp

When a memory allocation fails, the appropriate action occurs - either the allocation function returns 0 or an exception is thrown.

The following test programs on systems with lazy memory allocation fail in different ways:

• source:trunk/alloctests/trunk/tests/test_malloc2.c
• source:trunk/alloctests/trunk/tests/test_new2.cpp

These programs are different in that after virtual memory is allocated, the memory pages are touched. This causes the actual memory pages to be allocated on systems that have lazy memory allocation. If memory is exhausted at this point, typically the offending process, or sometimes a different process, may be killed by an OOM Killer (Out Of Memory Killer).

Unfortunately, most linux installations suffer from this behaviour.

Specifically, on Ubuntu 6, based on linux 2.6.18 with default configuration, both test programs intermittently cause Killed to be printed instead of any properly handled error code. This Killed being printed and the program being killed happens after malloc() or new() returns with success.

If you are unlucky, the lazy allocations perfectly fill up the memory and any super-user or admin process that then needs to indirectly allocate a page of memory, even if just for a stack enlargement due to a function call, then that process may be the victim of the OOM Killer.

see: http://www.win.tue.nl/~aeb/linux/lk/lk-9.html for information on linux OOM killer.

The Discussion

This page and minimal project was started in response to the following discussion thread in the newsgroup comp.lang.c++.moderated

• http://groups.google.ca/group/comp.lang.c++.moderated/browse_thread/thread/b7c199077e3e3d76/1867320eabbc2a1a?

A very important response is by James Kanze:

• http://groups.google.ca/group/comp.lang.c++.moderated/msg/83db143d4f5c7841

Addendum:

• It seems under linux 2.6.28 the /proc/sys/vm/overcommit_memory flags don't seem to stop OOM. will write more soon!