Windows CE >> How to increase heap memory for malloc()?

by Michael Schwab » Sat, 08 Jan 2005 05:20:54 GMT

As my program starts (in the first constructor that executes) I can malloc()
up to 26 Mb of space from the heap. While my program is initializing I can
watch this number drop dramatically, and by the time it finishes
initializing (loading all instances, bitmaps, etc.), I only can only
malloc() 5 Mb of space from the heap, and that's not enough to run a movie
using DirectShow.

I found the /HEAP:reserve[,commit] link option in MSDN, and I've tried
setting that to various numbers, but it seems to have no effect on the
amount of memory available to malloc(). For example, I've tried setting it
to "/HEAP:41820160,41820160" or "/HEAP:41820160" or "/HEAP:33554432" or
"/HEAP:50331648" (all numbers greater than 26 Mb), to no avail.

The /HEAP help also says the default heap is 1 Mb, and that's clearly not
true in my case, where it starts out at something over 26 Mb with no /HEAP
option.

If I set it to "/HEEP:..." the linker gives an "unrecognized option" error
message, so I know my link switches are actually making it to the linker, at
any rate! :-)

How am I supposed to increase my heap space?

Michael Schwab



Windows CE >> How to increase heap memory for malloc()?

by Steve Maillet (eMVP) » Sun, 09 Jan 2005 02:18:02 GMT


Windows CE processes are limited to 32M of virtual memory space. Therefore
unless your application is really small and doesn't load much in the way of
DLLs you won't be able to allocate anything that large from the heap. You
can however use memory mapped files without any backing file to get larger
amounts of memory from outside the 32M limit. and in V5.0 you can allocate
shared memory . But keep in mind that such memory is a free for all
(read/write by any process), except for the V5.0 shared memory which is read
only for other processes.

--
Steve Maillet
EmbeddedFusion
www.EmbeddedFusion.com
smaillet at EmbeddedFusion dot com

Windows CE >> How to increase heap memory for malloc()?

by Michael Schwab » Tue, 11 Jan 2005 04:45:11 GMT

es, now that I've read more about the heap I see I've been asking the wrong
question. The heap is limited by the 32Mb process limit (although I thought
that had been raised to 64Mb with CE.NET? - Guess not), and can't be
increased.

My brute force method of determining available heap space was to do a whole
bunch of malloc/free calls, which was probably exponentially increasing the
fragmentation of the heap! However, by that method I was able to
malloc/free a 26Mb chunk early on in my program, which said the initial
usage for code, etc, was about 6 Mb out of the 32Mb. By the end of
initialization, I could only allocate a 7.7Mb chunk, so I thought that meant
my program had used another 18.3 Mb of heap.

After faithfully searching the groups I came across the notion of using
CreateToolhelp32Snapshot(TH32CS_SNAPHEAPLIST) to determine how much heap
space is currently being used. I followed that up with Heap32ListFirst,
Heap32First, and Heap32Next to chase down the heap list to add up my heap
usage. (Heap32ListNext never returns anything under CE, so I guess my
process has only one heap list?)

Checking for dwFlags != LF32_FREE I added up the blocks in use and the size
of each block (dwBlockSize) to determine the heap in use. I also added up
the free blocks and found the largest free block, to see what that told me
about fragmentation.

But the results from this method don't jive with those of my brute-force
malloc/free method. At the point where the old method said I could alloc
26Mb of heap, the new method says I've only used 25 blocks for a total of
7344 bytes. So far so good. But, by the end of initialization, where my
old method said I'd used up 18Mb, the CreateToolhelp32Snapshot method says
I've only used 6.6 Mb (6631856) of heap space (in 785 blocks)! And there
was only about 30Kb in 37 free blocks, so it doesn't seem too fragmented,
either.

But I can still only run CoCreateInstance/Release six times, and the seventh
time it fails with an out-of-memory error. My CreateToolhelp32Snapshot
method says the first time it used another 608 bytes (and 4 blocks), and
times 2-6 only adds 64 bytes (and 2 blocks) each to the usage, bringing me
to 6632784 bytes in 799 blocks right before the first out-of-memory error
occurs. After that, even CreateToolhelp32Snapshot itself fails for lack of
memory.

If I add these numbers up I seem to using much less than half of my 32Mb
memory allocation. And the ControlPanel->System->Memory tab backs this up -
with my program running it only shows about 16Mb of program memory allocated
(out of 40Mb available).

So why am I running out of memory with these API calls? Or am I missing
something in the way I'm adding up the used memory blocks with
CreateToolhelp32Snapshot?

One of my problems is I'm not doing any of the malloc() calls myself - it's
all the API doing it for me (creating classes and threads, loading bitmaps
with SHLoadDIBitmap(), and the DirectShow calls). Therefore, I don't have
any control about which heap those calls draw memory from - they all seem to
get it from the main heap. None of the alternatives to the main heap (like
HeapAlloc) seem to have anything to do with where malloc() itself gets it's
memory. Is there any way to redirect these API calls to malloc() from
somewhere else?

Thanks,
Michael Schwab

"Steve Maillet (eMVP)" < XXXX@XXXXX.COM > wrote in message
news: XXXX@XXXXX.COM ...
of
read

Windows CE >> How to increase heap memory for malloc()?

by Steve Maillet (eMVP) » Tue, 11 Jan 2005 22:38:57 GMT

Have you looked into the implementation of the component you are CoCreating?
Perhaps it has a bug and is returning an out of memory indicator instead of
something more meaningful.

--
Steve Maillet
EmbeddedFusion
www.EmbeddedFusion.com
smaillet at EmbeddedFusion dot com

Windows CE >> How to increase heap memory for malloc()?

by Michael Schwab » Wed, 12 Jan 2005 01:02:04 GMT

It's not my component - it's the DirectShow IGraphBuilder object, like this:
hr = CoCreateInstance(CLSID_FilterGraph, NULL, CLSCTX_INPROC_SERVER,
IID_IGraphBuilder, (void **)&pGraph);
if (FAILED(hr))

(See my "DirectShow heap memory leak" for all the details of this related
problem).

and, like I said, it ran successfully six times (while eating up 1Mb of heap
space each time), and only died the seventh time. And, after that,
CreateToolhelp32Snapshot also returns error code 8 - Not enough storage to
process this command (when it's worked fine dozens of times before). I can
even run a small movie once or twice before running out of memory, so I know
the component is working correctly (except for the memory leak!). Anyway,
I'm going to give up and create a separate process to play the movie, to get
around the memory leak.

My main question here is why is there so much difference between my two heap
space checking methods? I can malloc() a 26 Mb buffer at the beginning of
the program (when CreateToolhelp32Snapshot shows almost no heap space used),
but I can only malloc() an 8 Mb buffer after initialization (when
CreateToolhelp32Snapshot only shows 6.6 Mb of heap space used). This
difference makes me wonder about heap fragmentation, but wouldn't that show
up as a lot of free "holes" in the heap list? I don't see that.

I'm also bothered by the fact that I only see one heap list for my
application under CE, when there are many under WinXP. Does that make
sense? Also, if I call the CreateToolhelp32Snapshot from other threads in
my program, they return a different processID (although they should be in
the same process), and the rest of the heap list calls fail. So I had to
put in a check for a different process ID and ignore those.

Even if I roll movie playing off to another process, I'm afraid my
application can't grow much more before some other API calls will start to
run out of memory. It would be nice to see a list of what API calls use
main heap memory (doing malloc()s, I presume?) and how much they use.

Michael Schwab



CoCreating?
of

Windows CE >> How to increase heap memory for malloc()?

by Michael Schwab » Wed, 12 Jan 2005 04:05:34 GMT

K, GlobalMemoryStatus() is my new friend (one of you eMVPs suggested it in
a thread I found in the Groups search). The TotalVirtual value comes out to
exactly 32Mb, and the AvailVirtual is always exactly 16,384 bytes greater
than what my brute force malloc()/free() method was showing me! (And it's
much faster than doing dozens of malloc()/free() calls!)

After initialization AvailVirtual=7798784. After running
CoCreateInstance()/Release() six times AvailVirtual is down to 458752.

So I guess CreateToolhelp32Snapshot isn't telling me the whole story.
Either there's more heap lists I'm not able to access, or a bunch of my 32
Mb memory space is getting eaten up for other reasons besides heap
allocations. Maybe it's the Module32 stuff?

But what's frustrating is that much of the "eating-up" of memory seems to be
just reserving the space rather than allocating it. I can see this by
looking at the difference between TotalPhys (41.8 Mb on my system) and
AvailPhys from GlobalMemoryStatus. After initialization TotalPhys-AvailPhys
is 13561856, and after I run CoCreateInstance six times TotalPhys-AvailPhys
rises to 13660160, or an increase of only 96 kb when AvailVirtual has
dropped 7 Mb!). TotalPhys and (TotalPhys-AvailPhys) match very closely with
the values shown under ControlPanel->System->Memory tab for program memory
allocated and in use.

Before my program runs, the Memory tab already shows 5Mb of program memory
allocated, which means my program has allocated less than 9 Mb of program
memory, and yet it claims that almost all of my 32 Mb program space is used
(AvailVirtual=458752), and gives out-of-memory errors! I don't understand
why this is happening!

What can be reserving 23 Mb of my available address space without actually
allocating program memory?

Michael Schwab

"Michael Schwab" < XXXX@XXXXX.COM > wrote in message
news:ubzlI9$ XXXX@XXXXX.COM ...
this:
heap
can
know
get
heap
used),
show



Similar Threads

1. Uncommitted heap increases / virtual memory reserved increases

I've got a server application on Windows XP with a memory problem. It
keeps claiming memory. The UNcommitted portion a the heap keeps
growing for no apparent reason. And the reserved part of Virtual
memory (as a result?) keeps growing too. No where in my program I
reserve virtual memory, and I've a minimum of new/delete operations to
avoid heap fragmentation.

Does any one know what the problem is? How I can fix it? Where I have
to look for?

Below is some memory info from the application running for 9 days,
it's start value -> and the value after 9 day.


Available Physical Memory
1576325120 -> 1579347968 bytes 

Available Virtual Memory
1891962880 -> 1756041216 bytes

Virtual Memory Reserved
92573696 -> 223338496 bytes

Virtual Memory Committed
163213312 -> 168407040 bytes

Heap Committed
3641344 -> 7995392 bytes

Heap Uncommitted
6647808 -> 128122880 bytes

Number of Heaps
9 -> 9

2. Problem with heap and malloc()/free()

3. How to increase malloc limit?

Hi all,

I have a little program that requires a lot of memory.
Currently I am finding that malloc is returning NULL when
I try to allocate over about 780 MB on a Windows XP Pro
machine that has 3 GB of usable RAM. I noticed that
the M$ website says "The virtual address space of
processes and applications is still limited to 2 GB"
but I am seeing half of that. Is the 780 MB limit perhaps
due to my using malloc instead of a Win32 function?

Thanks.

4. Increasing Desktop Heap

5. How to increase HEAP size in VC++ 6.0 environment

Hi All

My program needs a lot of memory and after allocating some using NEW
statement, it returns NULL for any new object created. I guess that is
becuase of the heap size limitation of the Visual Studio which stands
at 1MB. I need to increase it. One option I found was to set
/HEAP:memory in the Link options of project setting. Bur it didn't
seem to work. I am getting same problem again.
Dows anybody has a opinion about this.

regards
Nikhil

6. how to increase heap size for a process

7. How to free heap allocated my malloc?

Is it there a way in C# to free heap memory that has been allocated by an 
unmanaged component with malloc()?

I get an exception when trying to use Marshal.FreeCoTaskMem() on the managed 
pointer (of type IntPtr) returned from un-managed code as an "out" parameter. 
 In the C-dll this parameter is a double-pointer of my custom structure.  

I think the reason is that there is no CoTaskMemAlloc() in the ANSI-C-style 
DLL for allocating this pointer, just malloc().

Any suggestions for a good book or another source on PInvoke and Marshalling 
different data types (including pointers) back and forth between managed and 
unmanaged code (like C-style APIs)?

Any quide-lines for C# interoperability with ANSI-C code for multi-platforms 
would be appreciated.


Valentina


8. Simple malloc problem with Visual C++ 2005 {Heap}