increasing memory

Post any questions, ideas, or topics related to Jython and Python scripting.
Post Reply
User avatar
TJS
Posts: 90
Joined: Thu Mar 05, 2009 7:31 pm

increasing memory

Post by TJS »

Hi,

What's the best way to free-up memory while looping (say over spectral bands) in a script?

I've tried:

# trying to keep memory down on loop
collectGarbage()
layerVIS.doRemove()

and even

# taking out the boomstick...
boomstick()

but in both cases, the memory used just keeps increasing.

I'm a a PC, running mcidas-v version 1.4.

Thanks,
TJS
Attachments
Memory usage when loops over 4 bands.
Memory usage when loops over 4 bands.
User avatar
tommy.jasmin
Posts: 35
Joined: Tue Aug 17, 2010 6:56 pm

Re: increasing memory

Post by tommy.jasmin »

Tim - is the script not finishing as expected? Or are you just concerned that at some point as you add functionality there will be trouble?

Not knowing the details, it's a bit of a hack but if you can break the script up into multiple scripts, that's one way to be sure all resources are freed up each step.
User avatar
beckys
Posts: 172
Joined: Wed Jan 07, 2009 7:50 pm

Re: increasing memory

Post by beckys »

Hi Tim,

Tommy's first question is important here. If you're running out of memory, or if you're using a significantly larger amount of memory from the script than you do from the user interface, then we may have a problem in the code.

By default, McIDAS-V uses 80% of the available memory on your machine. Java will often allow the memory usage to approach that value before it attempts any cleanup that it can do. This is normal for Java, but very different than what we're used to with Fortran or C.

In a nutshell, if you're seeing other problems (actually running out of memory or using way too much memory), then we should investigate further. If you're not seeing other problems, then it's probably just Java being Java, and nothing to worry about.

I hope this helps!
Becky
User avatar
TJS
Posts: 90
Joined: Thu Mar 05, 2009 7:31 pm

Re: increasing memory

Post by TJS »

Thanks. The script does finish, but it becomes real slow, along with any other application running on my PC.
User avatar
TJS
Posts: 90
Joined: Thu Mar 05, 2009 7:31 pm

Re: increasing memory

Post by TJS »

Plus, even when the script finishes, I don't get the memory back. So, breaking up the script that does 16 bands into 16 scripts won't help, unless I kill mcidas after each band. TJS
User avatar
Rick
Posts: 404
Joined: Fri Jan 16, 2009 8:20 pm

Re: increasing memory

Post by Rick »

Hi Tim,

After your script has completed, go to the task manager and let us know if there are any leftover mcservl.exe or javaw.exe processes running. Java won't release the memory until these processes have finished.

Also, if you have not done so, please try running your script from the jython shell. From the main display window menu bar, Tools->Formulas->Jython Shell. Generally it's easiest to paste your script into the shell. Select the 2 down arrows to the right of the evaluate button. Paste your script into the that same text box and select evaluate. From the main display window, watch the memory allocation. If the numbers are in red, you might run out of memory.

Let us know your results.

Rick
User avatar
TJS
Posts: 90
Joined: Thu Mar 05, 2009 7:31 pm

Re: increasing memory

Post by TJS »

Hi,

javaw.exe stays around, but that might be since I'm running this in the jython shell.

That said, it takes up lots of memory (GBs) and so it seems like the 'remove all data' doesn't free the memory.

TJS
User avatar
Rick
Posts: 404
Joined: Fri Jan 16, 2009 8:20 pm

Re: increasing memory

Post by Rick »

When you exit your McIDAS-V session check to see if javaw.exe is still running. If it is not, do your other applications again behave normally?

Please post your entire script so we can give it a try.

Sounds like Java is behaving properly by taking all the memory it can grab. Removing a layer should release some of the memory, but not all the memory will be freed up.
Post Reply