Java ExecutorService and Linux Max Process Limit

Here's a little post about the Java ExecutorService, and a problem I ran into on Ubuntu. Consider this Java code

public static void main(String[] args) throws InterruptedException {
    int tasksSubmitted = 0;
    int nTasks = 800;
    int taskDur = 2000;
    int nThreads = 400;
    ExecutorService executor = Executors.newFixedThreadPool(nThreads);
    try {
        for (int i = 0; i < nTasks; i++) {
            executor.submit(() -> {
                try {
                    Thread.sleep(taskDur);
                } catch (InterruptedException e) {
                    System.out.println(e);
                }
            });
            tasksSubmitted++;
        }
    }
    catch (Error | Exception e) {
        System.out.println(e);
    }
    System.out.println("Tasks submitted: " + tasksSubmitted);
    executor.shutdown();
    executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
    if (tasksSubmitted != nTasks) {
        System.out.println("Only submitted " + tasksSubmitted + " of " + nTasks + " tasks");
    }
}

This little program uses a fixed thread pool executor service of 400 worker threads, executes 800 tasks that just wait 2 seconds, and waits for all tasks to complete.

On some EC2 instances, this code runs perfectly fine, and all 800 tasks execute. But on other EC2 instances, this code was failing. Took me a while to figure out that it was failing, since it was failing with an OutOfMemoryError, which I wasn't handling so my tasks were failing silently.

The documentation for ExecutorService.submit() doesn't mention that an OutOfMemoryError can be thrown, only RejectedExecutionExcpetion or NullPointerException.

Once I caught and logged Errors, I found the culprit: java.lang.OutOfMemoryError : unable to create new native Thread.

That was a pretty strange error, since my EC2 instance has 8 GB of RAM. Assuming a 1MB stack size @ 400 threads, that would only chew up 400 MB of RAM. So it's not a memory limitation.

With a quick Google search I found that on Linux there's a setting for maximum number of processes per user. You can check the process limit via ulimit -u. On the servers that this code failed,  ulimit -u was coming back as 50, whereas on the servers where it worked, ulimit -u was coming back with 64K.

Increasing the ulimit -u option to 1024 fixed the problem.

The ulimit options are stored in /etc/security/limits.conf, at least on my Ubuntu EC2 instance.

If you're using ExecutorService, be aware that submit can throw an OutOfMemoryError in the event the max process limit is reached.