Home | All Posts

[14 Nov 2013] Lein Uberjar with Titan Dependency

If you’re working with Titan in Clojure there’s a good chance you’re using Lein and if you are there’s also a chance that you are building a “jar with dependencies” via lein uberjar. I was doing just that yesterday, when I was suddenly tossed into a point of deep frustration. I watched happily as my code ran perfectly in my local environment with lein run, but after doing a lein uberjar and pushing my fat jar to a server instance in EC2 I got this:

Caused by: java.lang.IllegalArgumentException: A SPI class of type org.apache.lucene.codecs.
  PostingsFormat with name 'Lucene41' does not exist. You need to add the corresponding JAR 
  file supporting this SPI to your classpath.The current classpath supports the 
  following names: [Pulsing41, SimpleText, Memory, BloomFilter, Direct]
	at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:109)
	at org.apache.lucene.codecs.PostingsFormat.forName(PostingsFormat.java:100)
	at org.elasticsearch.index.codec.postingsformat.PostingFormats.<clinit>(
	  PostingFormats.java:76)

I ended up unlocking the solution to the problem by noticing an issue in the ElasticSearch GitHub Repo named “Conflicting META-INF between elasticsearch & lucene-core/lucene-codecs for packaging into standalone binary”. At that point, I realized that Lein was overwriting these files over and over again in the uberjar process:

META-INF/services/org.apache.lucene.codecs.Codec
META-INF/services/org.apache.lucene.codecs.DocValuesFormat
META-INF/services/org.apache.lucene.codecs.PostingsFormat

For my “jar with dependencies” to be functional, I would need those files to merge with one another as opposed to simply performing an overwrite. In Maven, you would likely use the Shade Plugin configured to do a resource trasformation. Up until yesterday, I wasn’t aware of how to do that with Lein, but after some research realized that a recently added feature to Lein (as of 2.3.3) provided a similar feature set. I simply added the following snippet to my project.clj:

:uberjar-merge-with {#"org\.apache\.lucene\.codecs\.*" [slurp str spit]}

which merged those various files together instead of writing one over the other. At that point, my “jar with dependencies” was good to go. You can read more about uberjar-merge-with here.

Home | All Posts