Skip to main content

Posts

Postgres Query Stats using pg_stat_statements

Just wanted to share how to configure Postgres to capture query statistics using pg_stat_statements, including how often queries are run, how long they take to execute, along with a whole host of other useful information.

First, you need to edit the Postgres configuration, adding pg_stat_statements to preshared libraries

shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.track = all
After editing postgresql.conf, restart postgres. On Ubuntu

sudo /etc/init.d/postgresql stop
sudo /etc/init.d/postgresql start
After restarting Postgres, you need to enabled the pg_stat_statements extension. You can enable it per database, or for all databases on the same Postgres server by installing on the postgres database.

CREATE EXTENSION pg_stat_statements; Once pg_stat_statements is added to the preshared libraries and the extension is enabled, you can query pg_stat_statements to get useful information about running queries.

SELECT
  query,
  sum(calls) AS calls,
  sum(total_t…
Recent posts

Java ExecutorService and Linux Max Process Limit

Here's a little post about the Java ExecutorService, and a problem I ran into on Ubuntu. Consider this Java code
public static void main(String[] args) throws InterruptedException { int tasksSubmitted = 0; int nTasks = 800; int taskDur = 2000; int nThreads = 400; ExecutorService executor = Executors.newFixedThreadPool(nThreads); try { for (int i = 0; i < nTasks; i++) { executor.submit(() -> { try { Thread.sleep(taskDur); } catch (InterruptedException e) { System.out.println(e); } }); tasksSubmitted++; } } catch (Error | Exception e) { System.out.println(e); } System.out.println("Tasks submitted: " + tasksSubmitted); executor.shutdown(); executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS); if (tasksSubmitted != nTasks) { System.out.println("Only…

Openers vs Closers

I've read an article in the past about the 2 types of developers: openers and closers. I can't remember the exact link, but this distinction is weighing on my mind lately.

In my mind these are roles, not developer types. A developer might be filling either role, depending on the circumstances.

Openers

The openers are the developers that are on the front line, taking feature requests from managers and customers, and turning them into real applications that can be interacted with. The openers are free to use whatever means necessary to get the job done. They aren't concerned with performance or scalability. Their job is to make the app work.

Closers

Once the app features solidify, the closers are the ones who fix the performance and scalability issues. They care deeply about runtime performance, about efficient algorithms, about profiling tools, memory allocation, all all the deep technical issues. While all developers are typically nerds, the closes are typically the uber-ner…

Relationships Between The DevOps Handbook and Systems Thinking

I've recently finished reading The DevOps Handbook, and Thinking in Systems, and have just started reading The Fifth Discipline. Below are a few of my thoughts after reading them.

The DevOps Handbook is a really great book and I highly recommend anyone participating in software development to give it a read. In the book, the authors describe the Three Ways (aka principles):
FlowFeedback Continual Learning and Experimentation
The first way focuses on getting software through the work stream, the second way focuses on getting feedback, and the third way focuses on organization learning. 
The DevOps Handbook's visualization of the 3 ways reminds me of systems diagrams in Thinking in Systems by Donella Meadows. 

Stuart Rimell has a nice 2 part post on systems thinking in software development, that draws on ideas in Thinking in Systems (Part 1, Part 2)
After reading The DevOps Handbook and Thinking in Systems, I got curious about systems thinking, and how that applies to software de…

Java: throws Exception

I have a pet peeve in Java, and it's methods that use the java.lang.Exception class in the throws clause.

For example

public void doSomething() throws Exception {
}

This forces callers of the doSomething method to either catch the raw Exception, or add it to throws clause of the consuming method. I think that this encourages developers to do the wrong thing. If you're going to use checked exceptions, then callers should only be catching specific exceptions that can be handled. Otherwise, the exception should propagate up the call stack, potentially terminating the thread if no one handles the exception.

Throws clauses are part of the specification of your method. If you're going to use checked exceptions, wouldn't it be nice if the caller could see the explicit list of exceptions to be handled? From a contract perspective, there's no value in specifying that your method can throw any Exception. You're better off removing the throws Exception clause, and just th…

2016 Reading List

As part of my own personal professional development, I publish the books I've read for the year. I'm on GoodReads, and here is my reading challenge.

Looking back, there were a few themes
Performance - I'm staring to get serious about performance analysis of systems. Brendan Gregg and Martin Thompson are influential in my understanding of systems and how to design for performance. It's also clear that what I thought I knew about computers, specifically OS kernels, is severely lacking. Operations - Site Reliability Engineering opened my eyes as to how large companies such as Google support applications in production. Philosophy and religion interest me. The more I think about it, I'm probably an atheist. The scientific method resonates with me, as well as Occam's Razor. I question my Catholic upbringing. The Bill Nye / Ken Ham debate gets you thinking. 
Systems Performance: Enterprise and the Cloud, Prentice Hall, 2013, Brendan GreggExtreme Ownership: How U.S. Navy…

Generating Java Mixed Mode Flame Graphs

Overview I've seen Brendan Gregg's talk on generating mixed-mode flame graphs and I wanted to reproduce those flamegraphs for myself. Setting up the tools is a little bit of work, so I wanted to capture those steps. Check out the Java in Flames post on the Netflix blog for more information.

I've created github repo (github.com/jerometerry/perf)  that contains the scripts used to get this going, including a Vagrantfile, and JMeter Test Plan.

Here's a flame graph I generated while applying load (via JMeter) to the basic arithmetic Tomcat sample application. All the green stacks are Java code, red stacks are kernel code, and yellow stacks are C++ code. The big green pile on the right is all the Tomcat Java code that's being run.


Tools Here's the technologies I used (I'm writing this on a Mac).
VirtualBox 5.1.12Vagrant 1.9.1bento/ubuntu-16.04 (kernel 4.4.0-38)Tomcat 7.0.68JMeter 3.1OpenJDK 8 1.8.111linux-tools-4.4.0-38linux-tools-commonBrendan Gregg's Fla…