Connection Pools.

While thinking through where I want to take Logjammin', given it has been neglected for a while, I was contemplating the different types of connection pools that a server has, and basically broke it down in to three different pools:

1. Connections to the clients.
2. Connections to peers.
3. Connections to other (dependent) services.

Now, I can easily argue the first set isn't a "pool" in the traditional sense, but it is a collection of connections that must be managed in some fashion. The second one is normally used for replication or synchronization. And the last one is the one we normally make connection pools for -- typically for database connections.

Take a database server for example. Knowing your clients will be intentionally keeping long running connections, it makes sense to pre-allocate resources for connections or keep the connections open and idle. The only real difference on the two sides is who initiated the connection. Keep-alive, long polling, and server sent events, the client as a connection pool starts to make a lot more sense.

That leaves the peers. I like to think of peers as essentially client AND server initiated connection pools. They require solutions on both sides, as some peer will connect to them, and they, in term, will connect to some other peer.

Right now, I am looking at the logjammin' code, neglected for a year, and questioning why I treat all three pools distinctly different? The Connections to clients is handled through the normal listen. The connection to the peers is handled in a different thread. And the dependent connections aren't implemented at all.

I have another trip to Dulles this weekend, and I want to play around with creating a pool of connections concept to replace the "Server" concept I have built right now. Something like each Logjammin' instance is a collection of 1 or more connection pools which deal with resource management, and initiating a configurable series of stages or events.

Update 10/10/2014: It took a while, but finally pushed it.

logjammin update

I moved logjammin over to github from bitbucket. Had more to do with the usual social networking stuff -- people I wanted to work with are on github, don't really use bitbucket).

In addition to that, I have been doing some major surgery on the project to change out some of the dependencies. Specifically, I have switched out CryptoPP for the Nettle cryptography library, and I am eventually going to switch out OpenSSL for gnutls.

On the nettle front, I had a hard time finding good examples for the GCM mode encryption. I had to piece things together form the information in the Nettle documentation, source, and playing with results. You can see the resulting encrypt and decrypt methods in the lj::Document class.

The most confusing part, at least for me, was that in GCM mode, everything uses the encrypt function of the underlying cipher. If you don't pay attention and accidentally pass in a decryption function, it will not work as expected. I also, stupidly, thought I had to chunk my inputs into the block size. I didn't realize until the end that I could just pass the entire message in and get the expected result. You only need to chunk up the message if you need to call encrypt / decrypt multiple times.

V8 and Lua

I haven't posted in a long time. I have been investing a lot of time into the nosql database I was working on. The biggest change their lately has been integrating v8 and lua into the code base. One of the things I really liked about integration of Lua was something called Lunar. It is an incredibly easy way to simplify Lua integration in c++. I don't remember all of the changes I have made to my version over years: stuff like meta method support, etc. If you are trying to do some Lua integration, it is a great place to start.

But in order to get some friends to help me work on Logjammin', I had to start incorporating v8 (silly JS developers). I have to say that I found certain parts of the v8 documentation and examples really hard to follow. After spending an entire afternoon just trying to get a print function into logjamd, I realized that it would be pretty straightforward to create a v8 version of Lunar. It is actually slightly easier because of the cached template objects that normally get used. And because I am currently reading Quicksilver by Neil Stephenson, I decided to call the helper file "Jesuit". To be honest, I was thinking of Janissaries, since that is how book two ends. But that was too long of a word.

Anyway. The Lua version is here:

https://bitbucket.org/ibd71mav/logjammin/src/f1c2487e2053/src/lua/lunar.h

And the v8 version is here:

https://bitbucket.org/ibd71mav/logjammin/src/f1c2487e2053/src/js/jesuit.h

Concurrency approaches

When you start developing a service or system, it is much easier to make it work as a single thread in a single process. Single threading allows rapid development of concept at the expense of concurrency. Single threaded just makes things so much more predictable.

But eventually a service needs the ability to handle more than a single user at a time. Typically, the process either forks itself or utilizes some threading library (like pthreads). Multithreading an application introduces new complexities. Things like IPC, mutexes, semaphores, race conditions, and atomicity become troublesome concepts. For some applications, multi-threading is the only way to go. Applications we use every day would be unusable as a single threaded application. Complex AJAX websites would be horrible -- the A in AJAX stands for Asynchronous after all.

Most things accomplished with threading can also be accomplished with multiple processes. IPC can be done through a resource other than memory. Creating a socket connection between two processes is a very simple method of implementing IPC between two processes and it avoids some of the complications inherent in semaphores. So why don't developers leverage this approach more often? It is typically viewed as wasteful of resources and it typically performs slower than two threads communicating inside the same process. For these reasons I have avoided processes unless the functionality is intended for physically different hosts.

Then Google Chrome came around and I had to partially rethink my above beliefs. Chrome is not a single process running multiple threads. It is a collection of multiple processes running multiple threads per process. Google decided more upfront memory consumption was an acceptable exchange for more crash isolation. Check out the Chrome book to see a crash course on chrome in a comic-book format.

So now I am trying to write a database service. Like all modern database servers, it needs to support replication. It particularly must support replication between hosts. In development, I normally run two processes on the same host and pretend those processes are running on different machines. It was at this point the lightbulb lit up and I started thinking about Chrome's architecture. What if a database server was a collection of single threaded processes running on a single host. The database equivalent of the prefork Apache MPM. The model is not new, and it isn't even unique, but it isn't common in the DB world. kdb+ is one of the few databases I have heard of that tries this approach.

Unlike the prefork model, I am thinking of leaving them completely unrelated processes. Essentially leveraging the replication functionality as my IPC model and leveraging some other technology to do the load balancing. The simplicity of the approach looks appealing at first glance. The theoretical redundancy and fault tolerance in the model is intriguing.

Static Blocks and Static Inheritence.

I was using a library that made heavy use of the static inheritance behavior of Java.

This example is rather contrived, but stick with me. So imagine this situation, You have a super class named Base:

public class Base {
    public static String searchEngine = "http://www.google.com";
    public static void search() { System.out.println(searchEngine); }
}
public class Child {
    static {
        searchEngine = "http://www.bing.com";
    }
}

Given the above classes, execute the following program code:

Base.search();
Child.search();

Some of you noted that I called a static method on Child that does not exist. Technically that is true. The Java compiler understands that Child inherited Base.search() and properly compiles. Nice feature. Especially helpful when you are doing static imports and what all of the inherited static methods as well.

So here is the resulting output:

http://www.google.com
http://www.google.com

This is where things get interesting. While the compiler understands the inheritance tree, the Java Runtime doesn't. So for the purposes of bytecode generation, the call to "Child.search()" gets replaced with a call to "Base.search()". That is fine. Totally acceptable behavior. A lot of constructs in java are syntactical sugar for more elaborate byte-code.

But now we get to the part I consider a bug, or at least an ambiguous behavior. For "Child.search()", I obviously referenced the Child class. As such, I expected the Child static block to be executed. But the resulting bytecode contains a reference to Base; the class loader has no reason to load the class Child in that example thus the Child static blokc is not executed.

Now, there are a couple of ways to force the Class loader to do what I want:

    Class.forName("Child"); Child.search(); //jdbc example, force reference before.
    new Child(); Child.search();  //hard coded example, same as above.
    new Child().search(); // Unholy. don't call static methods against instances.

None of these are optimal solutions. I especially think the last one is unholy. Back to my problem, I am going to redesign the library in question to remove the super class dependency on a child static block being executed. I don't like telling the JVM or the class loader what to do based on hints, and I think depending on the order of static block execution is equivalent to not vaccinating your kids for Polio -- sure, things might just work out fine, but do you really want to risk them getting Polio?

Tokyo Cabinet Tuning Part 1 - Bucket Array Size

I have been playing around with Tokyo Cabinet for a few weeks now, and I wanted to share some of the tuning hints I have found.

I was loading a database with just shy of two billion records, and speed would become unacceptably slow after about the 500 million mark. In order to improve the database performance, I have begun experimenting with different tuning options available through the tcbdbtune method. The first tuning option I experimented with was the number of members in the bucket array.

Putting records into the B+ tree database will be much slower than you expect unless you increase the number of elements in the bucket array. Some of my runs took over 30 minutes to load 100 million records. I performed over 200 tests with different bucket, leaf/non-leaf member values, and record counts. In the end I found the bucket array should be between one-tenth to six-tenths the expected data-set size. Anything smaller or larger results in longer loads. The leaf/non-leaf values had very little impact on the performance of linear record writing.

I am still collecting data on the performance of different leaf/non-leaf settings for random writes, and I will post about those findings in part 2.

Anonymous Logger Memory Leak

I was helping a development group uncover the cause of a memory leak a while back, and figured I should share the finding here.

The memory leak in this case was induced by the static method "getAnonymousLogger()" on the class "java.util.logging.Logger"; although it looks like it could be caused by any logger that sets a parent in code. When an anonymous logger is created, it is created from scratch, and the root looger is set as the parent.

When a logger has its parent set in code, a weak reference is created on the parent for its list of "kids". This design allows short lived loggers to be garbage collected even though the parent maintains a reference to the logger. Great in theory, but missing a component in the current implementation. The current implementation never cleans up the weak reference objects. That is, the memory used by the Logger is freed, but the weak reference held by the parent is not. Those interested in the bug can get the JDK source code, and look at the private doSetParent(Logger) method on the Logger class.

Calling setParent or getting an anonymous logger leaks a minuscule amount of memory per invocation. In applications that run for a very short period of time (like an applet), this is probably unnoticeable. When called per invocation of a servlet in a long running j2ee container, it will eventually kill your JVM instance.

So how do you get around it? My first suggestion is to never use getAnonymousLogger unless you have an exceptionally good reason (probably involving the security manager). My second suggestion is to save a reference to the Logger object as a member field, and always reuse that reference.