java Memory leak /Threads

[henryb]

Hi There,
Ive been trying to get McIDAS v1.4 running in scripting mode on a CentOS 6.4 server with a GeForce GT610 Nvidia Graphics card.
When I do alot of processing the heap memroy goes up and the number of waiting threads increases dramatically to about 120.
I can see them using jstack - Also I can see the heap memory usage with "jconsole:

Below is list of the open threads from jstack. The program works fine in script mode both on the server and my local ubuntu box.
I've been to by a java guru that it could be that the threads are waiting because the Canvas3D display is not visible in script mode:

... Henry



"J3D-BehaviorScheduler-1" prio=10 tid=0x00007fbe141d2000 nid=0xe99 in Object.wait() [0x00007fbe5055c000]
"J3D-BehaviorScheduler-2" prio=10 tid=0x00007fbe141d5000 nid=0x1095 in Object.wait() [0x00007fbe031f0000]
"J3D-BehaviorScheduler-3" prio=10 tid=0x00007fbe141be000 nid=0x1494 in Object.wait() [0x00007fbdff068000]
"J3D-BehaviorScheduler-4" prio=10 tid=0x00007fbe14040000 nid=0x190f in Object.wait() [0x00007fbdfdd2a000]
"J3D-BehaviorScheduler-5" prio=10 tid=0x00007fbe141a3000 nid=0x1c0a in Object.wait() [0x00007fbdfd225000]
"J3D-BehaviorStructureUpdateThread-1" prio=10 tid=0x00007fbe141a8800 nid=0xe94 in Object.wait() [0x00007fbe03cfb000]
"J3D-BehaviorStructureUpdateThread-2" prio=10 tid=0x00007fbe14187800 nid=0x1090 in Object.wait() [0x00007fbe5035a000]
"J3D-BehaviorStructureUpdateThread-3" prio=10 tid=0x00007fbe141de000 nid=0x148f in Object.wait() [0x00007fbdfec35000]
"J3D-BehaviorStructureUpdateThread-4" prio=10 tid=0x00007fbe1403c000 nid=0x190a in Object.wait() [0x00007fbdfe32c000]
"J3D-BehaviorStructureUpdateThread-5" prio=10 tid=0x00007fbe1419d800 nid=0x1c05 in Object.wait() [0x00007fbdfe029000]
"J3D-GeometryStructureUpdateThread-1" prio=10 tid=0x00007fbe141b0800 nid=0xe95 in Object.wait() [0x00007fbe03bfa000]
"J3D-GeometryStructureUpdateThread-2" prio=10 tid=0x00007fbe141c2000 nid=0x1091 in Object.wait() [0x00007fbe02ceb000]
"J3D-GeometryStructureUpdateThread-3" prio=10 tid=0x00007fbe141e6000 nid=0x1490 in Object.wait() [0x00007fbdfe730000]
"J3D-GeometryStructureUpdateThread-4" prio=10 tid=0x00007fbe1403c800 nid=0x190b in Object.wait() [0x00007fbdfdb28000]
"J3D-GeometryStructureUpdateThread-5" prio=10 tid=0x00007fbe1419e000 nid=0x1c06 in Object.wait() [0x00007fbdfcf22000]
"J3D-InputDeviceScheduler-1" prio=10 tid=0x00007fbe141b2800 nid=0xe9c in Object.wait() [0x00007fbe037f6000]
"J3D-InputDeviceScheduler-2" prio=10 tid=0x00007fbe141ad000 nid=0x1098 in Object.wait() [0x00007fbe032f1000]
"J3D-InputDeviceScheduler-3" prio=10 tid=0x00007fbe1404d000 nid=0x1497 in Object.wait() [0x00007fbdff268000]
"J3D-InputDeviceScheduler-4" prio=10 tid=0x00007fbe14044000 nid=0x1912 in Object.wait() [0x00007fbe50158000]
"J3D-InputDeviceScheduler-5" prio=10 tid=0x00007fbe1400a800 nid=0x1c0d in Object.wait() [0x00007fbdfe62f000]
"J3D-MasterControl-1" prio=10 tid=0x00007fbe2006d000 nid=0xe90 in Object.wait() [0x00007fbe02bea000]
"J3D-NotificationThread" prio=10 tid=0x00007fbe20061800 nid=0xe8f in Object.wait() [0x00007fbe035f4000]
"J3D-Renderer-1" prio=10 tid=0x00007fbe141ce800 nid=0xe91 in Object.wait() [0x00007fbe03efd000]
"J3D-RenderingAttributesStructureUpdateThread" prio=10 tid=0x00007fbe20055000 nid=0xe8d in Object.wait() [0x00007fbe50a61000]
"J3D-RenderingEnvironmentStructureUpdateThread-1" prio=10 tid=0x00007fbe141dc800 nid=0xe97 in Object.wait() [0x00007fbe5045b000]
"J3D-RenderingEnvironmentStructureUpdateThread-2" prio=10 tid=0x00007fbe141c8800 nid=0x1093 in Object.wait() [0x00007fbe51074000]
"J3D-RenderingEnvironmentStructureUpdateThread-3" prio=10 tid=0x00007fbe1419a800 nid=0x1492 in Object.wait() [0x00007fbdfe831000]
"J3D-RenderingEnvironmentStructureUpdateThread-4" prio=10 tid=0x00007fbe1403e000 nid=0x190d in Object.wait() [0x00007fbdfdc29000]
"J3D-RenderingEnvironmentStructureUpdateThread-5" prio=10 tid=0x00007fbe141a0800 nid=0x1c08 in Object.wait() [0x00007fbdfd124000]
"J3D-RenderStructureUpdateThread-1" prio=10 tid=0x00007fbe141d3800 nid=0xe9a in Object.wait() [0x00007fbe50259000]
"J3D-RenderStructureUpdateThread-2" prio=10 tid=0x00007fbe141c6800 nid=0x1096 in Object.wait() [0x00007fbe51175000]
"J3D-RenderStructureUpdateThread-3" prio=10 tid=0x00007fbe1404b000 nid=0x1495 in Object.wait() [0x00007fbe5085f000]
"J3D-RenderStructureUpdateThread-4" prio=10 tid=0x00007fbe14041000 nid=0x1910 in Object.wait() [0x00007fbdfde2b000]
"J3D-RenderStructureUpdateThread-5" prio=10 tid=0x00007fbe141a4800 nid=0x1c0b in Object.wait() [0x00007fbdfd427000]
"J3D-SoundSchedulerUpdateThread-1" prio=10 tid=0x00007fbe14198800 nid=0xe9b in Object.wait() [0x00007fbe030ef000]
"J3D-SoundSchedulerUpdateThread-2" prio=10 tid=0x00007fbe141d7000 nid=0x1097 in Object.wait() [0x00007fbe02fee000]
"J3D-SoundSchedulerUpdateThread-3" prio=10 tid=0x00007fbe1404b800 nid=0x1496 in Object.wait() [0x00007fbdfea33000]
"J3D-SoundSchedulerUpdateThread-4" prio=10 tid=0x00007fbe14042800 nid=0x1911 in Object.wait() [0x00007fbdfeb34000]
"J3D-SoundSchedulerUpdateThread-5" prio=10 tid=0x00007fbe141a6000 nid=0x1c0c in Object.wait() [0x00007fbdfe52e000]
"J3D-SoundStructureUpdateThread-1" prio=10 tid=0x00007fbe141d0800 nid=0xe96 in Object.wait() [0x00007fbe036f5000]
"J3D-SoundStructureUpdateThread-2" prio=10 tid=0x00007fbe141ca800 nid=0x1092 in Object.wait() [0x00007fbe5065d000]
"J3D-SoundStructureUpdateThread-3" prio=10 tid=0x00007fbe1419a000 nid=0x1491 in Object.wait() [0x00007fbdfe932000]
"J3D-SoundStructureUpdateThread-4" prio=10 tid=0x00007fbe1403d800 nid=0x190c in Object.wait() [0x00007fbdfee35000]
"J3D-SoundStructureUpdateThread-5" prio=10 tid=0x00007fbe1419f000 nid=0x1c07 in Object.wait() [0x00007fbdfe22b000]
"J3D-TimerThread" prio=10 tid=0x00007fbe20079000 nid=0xe8e in Object.wait() [0x00007fbe02ae9000]

[bobc]

Hello Henry -

Thanks for reporting this problem. I sent this off to one of our programmers, and he had two requests:

1. It looks like you didn't provide all of the jstack output. Having the results of “jstack -l <PROCESS ID> > mcv-threads.out” would definitely help out.

2. Since you mentioned that you used jconsole, it might be worthwhile trying it again, going to the "Threads" tab, and hitting the "Detect Deadlock". Can you let us know what happens when you do this?

Thanks again -
Bob Carp
McIDAS Help Desk

[henryb]

HI Bob,
thank you for your help.
Attached is the full jstack out. There are NO deadlocks
The jython script I run - has processed four distinct countires
ie each time I load up a new country : this means sub-setting the netcdf data / mask data / map
jython script log output
DEBUG:root:Main: region argentina processed
DEBUG:root:Main: region beunos-aires processed
DEBUG:root:Main: region brazil processed
DEBUG:root:Main: region rio-de-janeiro processed
DEBUG:root:Main: region uruguay processed
DEBUG:root:Main: region rocha too small to be processed
DEBUG:root:Main: region france processed
DEBUG:root:Main: region lower-normandy processed



.... Henry

[henryb]

Hi Bob,
did you manage to pass the stack log to a programmer ?
.. thanks Henry

[bobc]

Hi Henry -

Sorry for the delayed response. I did pass this off to one of our java programmers, but he is on vacation until next Tuesday. I will write up a McIDAS-V inquiry to investigate this memory leak.

Thanks again -
Bob

[Jon]

Hello Henry,

I apologize for the delayed reply; I've been trying to replicate the problem…but it's proving difficult to do so without having the script and/or data to work with. Is there any way I can get those?

Also, I saw your post on the Jogamp Java3D forum. FWIW we've been working with Unidata to migrate VisAD/IDV/McV to the newer JOGL and Java3D as well as moving to Java 7 (we've also been testing Java 8, and will start Java 9 as soon as Oracle has something for me to build). Unfortunately the newer JOGL and Java 3D still appear to have some issues, but these are actively being discussed.

[henryb]

Hi Jon,
I tried the new JOGJ stuff and I still get the same problem.
So I guess the problem is further down the software stack -
The thing is the server is not to hand - its in Amsterdam
When we requested to add a Graphics card - the hosting company made a whole song and dance about it and then eventually installed an nvidia GT810

I am currently getting by, by keeping the number of areas processed by each script to between 5-10 - so the thread count doesn't go ballistic

.. Henry

[Jon]

Hi Henry,

Without knowing the details of your script, you may want to try the "boomstick" function (the terrible name signifies that it shouldn't be used lightly!). It removes all data sources, displays, and attempts to do garbage collection. The following is some very limited pseudocode to demonstrate how/when the function might be used.

Code: Select all

# load area 1
# display area 1
# capture the display's contents
boomstick()

# load area 2
# display area 2
# capture the display's contents
boomstick()

# ...and so on!

[henryb]

Hi Jon,
I think I have found the problem. In my script I resize the the active display a multiple number of times with the command setDimensions() and doing this create multiple dormant threads
(nb the program is being run in script mode)

Code: Select all

runMcV -userpath  -debug -nogui -script /home/hmb/Documents/SURFTWEETER/mcv/run/support.py

The output from my program below is.

DEBUG:root:/**** log start ****/
DEBUG:root:total number of Threads=32
DEBUG:root:total number of Threads=41
DEBUG:root:total number of Threads=50
DEBUG:root:total number of Threads=59
DEBUG:root:total number of Threads=68
DEBUG:root:total number of Threads=77
DEBUG:root:total number of Threads=86
DEBUG:root:total number of Threads=95
DEBUG:root:total number of Threads=104
DEBUG:root:total number of Threads=113
DEBUG:root:/**** log end ****/

Below is my program

Code: Select all

import os
import logging
import sys
import time
homeDir=os.getenv("HOME")
print homeDir

def numThreads():
    numT=java.lang.management.ManagementFactory.getThreadMXBean().getThreadCount();
    return numT

#open a logging file
logging.basicConfig(filename="support.log", filemode='w', level=logging.DEBUG);     
logging.debug("/**** log start ****/");


# Open any bundle that has a display
loadBundle("auswaveG2.mcv",None,500,400)

for sizeY in range(400,500,10):
  logging.debug("total number of Threads=%d\n"% numThreads() );
  activeDisplay().setDimensions(0,0,500,(sizeY));


logging.debug("Final number of Threads=%d\n"% numThreads() );
logging.debug("/**** log end ****/");

[Jon]

Hi Henry,

First, I apologize for letting this sit, unanswered, for so long. I had very much wanted to just follow up with a "it's fixed in the new nightly", but I'm sorry to say that I've tried a number of different approaches and consistently hit a brick wall. Out of curiosity, is this something that impacts your day-to-day use of McV?

We've written this up as Inquiry 1679: http://mcidas.ssec.wisc.edu/inquiry-v/?inquiry=1679

And FWIW, this bug is one of those that really bothers me, so I'll continue to try things as I think of them.