ENTRY

"Linux, Runtime.exec, fork = java.lang.ProcessBuilder.start, java.io.IOException: “cannot allocate memory”?"

Date:May 4th, 2010
Tags:
Comments: 0
  • Join date: 11-30-99
  • Posts: 23
View Counter:
  • 371 views
Rate it
Ad
Poll
  • Best shells (or command line language) are?

    View Results

    Loading ... Loading ...
Feeds:
  • bodytext bodytext bodytext

Do you also have some huge server? We are running both Hudson and TeamCity CI’s. They have 20-100 simultaneous builds. And here it comes:

Caused by: java.io.IOException: Cannot run program "svn": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:328)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.execute(DefaultGroovyMethods.java:6672)
at org.codehaus.groovy.runtime.dgm$132.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:270)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:121)
at jetbrains.buildserver.groovyPlug.RevisionExtractionHelper.getLatestChangeRevFromVcsRootUsingCommandLine(RevisionExtractionHelper.groovy:175)
at jetbrains.buildserver.groovyPlug.RevisionExtractionHelper$getLatestChangeRevFromVcsRootUsingCommandLine.callCurrent(Unknown Source)
at jetbrains.buildserver.groovyPlug.RevisionExtractionHelper.addLastModificationsRevisions(RevisionExtractionHelper.groovy:237)
at jetbrains.buildserver.groovyPlug.RevisionExtractionHelperI$addLastModificationsRevisions$0.call(Unknown Source)
at jetbrains.buildserver.groovyPlug.GroovyReferencePropertiesProvider.getParameters(GroovyReferencePropertiesProvider:62)
at sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

The problem seems to be that java forks all the TeamCity process! So, if TeamCity currently allocates 1gb, java will require 2gb to use ANY command with Runtime.exec. So, what to do?

Ideas:

  • You run java like “java -Xmx6g -Xms6g …”. That means, that you will need to increase available swap size to at least 12gb (better ca 15gb). That should work for single threaded system;
  • Use proxy. Create a separate small java process which does Runtime.exec and returns the output to main process;
  • Avoid multithreading if possible;
  • If one thread definitely works, but you have many, you can use following (pseudocode here):
1
2
3
4
5
6
7
8
9
for(int i=0;i<10;i++) {
try {
.. your Runtime.exec
} catch (java.io.IOException ex ) {
int sleepFor = (int)new math.random()*10000;
 
log.warn("Catched out of memory, (linux forking of teamcity), sleep "+sleepFor+" and try once more" ;
     sleep (sleepFor):
}

The idea is to avoid that many copies of huge process collide and eat all the memory…

Here is more about the problem:
http://developers.sun.com/solaris/articles/subprocess/subprocess.html

Be Sociable, Share!
 
Interesting entry?
VN:F [1.8.5_1061]
Rating: 0 (from 0 votes)
0 votes 'YES'  0 votes 'NO'


TAGS
No tags for this post.

RELATED
Pages
Posts

SOCIAL
Be Sociable, Share!

INCOMING SEARCH TERMS



Responces

Leave a Reply