Skip to main content

Java ExecutorService and Linux Max Process Limit

Here's a little post about the Java ExecutorService, and a problem I ran into on Ubuntu. Consider this Java code


public static void main(String[] args) throws InterruptedException {
    int tasksSubmitted = 0;
    int nTasks = 800;
    int taskDur = 2000;
    int nThreads = 400;
    ExecutorService executor = Executors.newFixedThreadPool(nThreads);
    try {
        for (int i = 0; i < nTasks; i++) {
            executor.submit(() -> {
                try {
                    Thread.sleep(taskDur);
                } catch (InterruptedException e) {
                    System.out.println(e);
                }
            });
            tasksSubmitted++;
        }
    }
    catch (Error | Exception e) {
        System.out.println(e);
    }
    System.out.println("Tasks submitted: " + tasksSubmitted);
    executor.shutdown();
    executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
    if (tasksSubmitted != nTasks) {
        System.out.println("Only submitted " + tasksSubmitted + " of " + nTasks + " tasks");
    }
}

This little program uses a fixed thread pool executor service of 400 worker threads, executes 800 tasks that just wait 2 seconds, and waits for all tasks to complete.

On some EC2 instances, this code runs perfectly fine, and all 800 tasks execute. But on other EC2 instances, this code was failing. Took me a while to figure out that it was failing, since it was failing with an OutOfMemoryError, which I wasn't handling so my tasks were failing silently.

The documentation for ExecutorService.submit() doesn't mention that an OutOfMemoryError can be thrown, only RejectedExecutionExcpetion or NullPointerException.

Once I caught and logged Errors, I found the culprit: java.lang.OutOfMemoryError : unable to create new native Thread.

That was a pretty strange error, since my EC2 instance has 8 GB of RAM. Assuming a 1MB stack size @ 400 threads, that would only chew up 400 MB of RAM. So it's not a memory limitation.

With a quick Google search I found that on Linux there's a setting for maximum number of processes per user. You can check the process limit via ulimit -u. On the servers that this code failed,  ulimit -u was coming back as 50, whereas on the servers where it worked, ulimit -u was coming back with 64K.

Increasing the ulimit -u option to 1024 fixed the problem.

The ulimit options are stored in /etc/security/limits.conf, at least on my Ubuntu EC2 instance.

If you're using ExecutorService, be aware that submit can throw an OutOfMemoryError in the event the max process limit is reached.

Comments

Post a Comment

Popular posts from this blog

Basic Web Performance Testing With JMeter and Gatling

Introduction In this post I'll give a quick way to get some basic web performance metrics using both JMeter and Gatling . JMeter is a well known, open source, Java based tool for performance testing. It has a lot of features, and can be a little confusing at first. Scripts (aka Test Plans), are XML documents, edited using the JMeter GUI.  There are lots of options, supports a wide variety of protocols, and produces some OK looking graphs and reports. Gatling is a lesser known tool, but I really like it. It's a Scala based tool, with scripts written in a nice DSL. While the scripts require some basic Scala, they are fairly easy to understand and modify. The output is a nice looking, interactive, HTML page. Metrics   Below are the basic metrics gathered by both JMeter and Gatling . If you are just starting performance testing, these might be a good starting point . Response Time – Difference between time when request was sent and time when response has been fully rec

Generating Java Mixed Mode Flame Graphs

Overview I've seen Brendan Gregg's talk on generating mixed-mode flame graphs  and I wanted to reproduce those flamegraphs for myself. Setting up the tools is a little bit of work, so I wanted to capture those steps. Check out the Java in Flames post on the Netflix blog for more information. I've created github repo ( github.com/jerometerry/perf )  that contains the scripts used to get this going, including a Vagrantfile, and JMeter Test Plan. Here's a flame graph I generated while applying load (via JMeter) to the basic arithmetic Tomcat sample application. All the green stacks are Java code, red stacks are kernel code, and yellow stacks are C++ code. The big green pile on the right is all the Tomcat Java code that's being run. Tools Here's the technologies I used (I'm writing this on a Mac). VirtualBox 5.1.12 Vagrant 1.9.1 bento/ubuntu-16.04 (kernel 4.4.0-38) Tomcat 7.0.68 JMeter 3.1 OpenJDK 8 1.8.111 linux-tools-4.4.0-38 linux-to

Multi Threaded NUnit Tests

Recently I needed to reproduce an Entity Framework deadlock issue. The test needed to run in NUnit, and involved firing off two separate threads. The trouble is that in NUnit, exceptions in threads terminate the parent thread without failing the test. For example, here's a test that starts two threads: the first thread simply logs to the console, while the other thread turfs an exception. What I expected was that this test should fail. However, the test actually passes. readonly ThreadStart[] delegates = { () => { Console.WriteLine("Nothing to see here"); }, () => { throw new InvalidOperationException("Blow up"); } }; [Test] public void SimpleMultiThreading() { var threads = delegates.Select(d => new Thread(d)).ToList(); foreach (var t in threads) { t.Start(); } foreach (var t in threads) { t.Join(); } } Peter Provost posted an article that describes how to make this test fail. It