LArray alternatives and similar packages
Based on the "Extensions" category.
Alternatively, view LArray alternatives based on common mentions on social networks and blogs.
9.7 9.6 LArray VS catsLightweight, modular, and extensible library for functional programming.
8.5 1.1 LArray VS CassovaryCassovary is a simple big graph processing library for the JVM
8.3 9.6 LArray VS scala.metaLibrary to read, analyze, transform and generate Scala programs
7.8 7.8 LArray VS Scala-LoggingConvenient and performant logging library for Scala wrapping SLF4J.
7.5 0.0 LArray VS ChimneyScala library for boilerplate-free, type-safe data transformations
7.0 0.0 LArray VS FreestyleA cohesive & pragmatic framework of FP centric Scala libraries
6.8 6.8 LArray VS Scala GraphGraph for Scala is intended to provide basic graph functionality seamlessly fitting into the Scala Collection Library. Like the well known members of scala.collection, Graph for Scala is an in-memory graph library aiming at editing and traversing graphs, finding cycles etc. in a user-friendly way.
6.5 8.5 L2 LArray VS tinylogtinylog is a lightweight logging framework for Java, Kotlin, Scala, and Android
5.8 9.0 LArray VS scribeThe fastest logging library in the world. Built from scratch in Scala and programmatically configurable.
5.3 0.0 LArray VS EachA macro library that converts native imperative syntax to scalaz's monadic expressions
4.8 0.0 LArray VS Stateless FutureAsynchronous programming in fully featured Scala syntax.
4.7 0.0 LArray VS Scala BlitzScala framework for efficient sequential and data-parallel collections -
4.4 0.0 LArray VS Records for ScalaLabeled records for Scala based on structural refinement types and macros.
4.2 0.0 LArray VS Play monadic actionsA simple scala DSL to allow clean and monadic style for Play! Actions
3.3 7.8 LArray VS enableIf.scalaA library that toggles Scala code at compile-time, like #if in C/C++
2.1 0.0 LArray VS FreedslPractical effect composition library based on abstract wrapping type and the free monad
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of LArray or a related project?
A library for managing large off-heap arrays that can hold more than 2G (231) entries in Java and Scala. Notably LArray is disposable by calling
LArray.free. Even if you forget to release it, GC will automatically deallocate the memory acquired by LArray. LArray also supports
mmap (memory-mapped file) whose size is more than 2GB.
- LArray can create arrays with more than 2G(231) entries.
- 231 -1 (2G) is the limitation of the default Java/Scala array size, because these arrays use 32-bit signed integer (int) as indexes. LArray uses long type indexes of 64-bit signed integers to resolve this limitation.
- For example, the entire human genome data (3GB) can be stored in LArray.
- LArray can be released immediately from the memory.
- The default arrays in Java/Scala stay in JVM heaps until they are collected by GC, so it is generally difficult to avoid
OutOfMemoryExceptionwhen working with large amount of data. For example, call
new Array[Int](1000)x 10,000 times. You are lucky if you don't see OutOfMemoryException.
- LArray can be collected by Garbage Collector (GC)
- Even if you forget to call LArray.free, the acquired memory will be released when GC sweeps LArray instances.
- To prevent accidental memory release, keep a reference to LArray somewhere (e.g., in List) as in the standard Java/Scala program.
- LArray resides in off-heap memory
- LArray uses a memory space outside the JVM heap, so creating LArrays with more than -Xmx(maximum heap size) is possible. This is useful when you need large amount of memory, or it is unknown how much memory is required in your application.
- Fast memory allocation
- LArray internally uses a concurrent memory allocator suited to multi-threaded programs, which is faster than the default JVM memory allocator.
- LArray by default skips the array initialization (zero-filling), which improves the memory allocation speed significantly.
- LArray can be used as DirectBuffer
- Enables zero-copy transfer to/from files, network, etc.
- Zero-copy compression with snappy-java (supported since version 1.1.0-M4. Pass LArray.address to Snappy.rawCompress etc.)
- Rich set of operations for LArray[A]
- map, filter, reduce, zip, etc. Almost all collection operations in Scala are already implemented for LArray[A].
- Supports Memory-mapped file larger than 2GB
- It can create memory regions that can be shared between processes.
- LArray[A] of generic objects (e.g., LArray[String], LArray[AnyRef]) cannot be released immedeately from the main memory, because objects other than primitive types need to be created on JVM heaps and they are under the control of GC.
- To release objects from main memory, you need to create off-heap objects. For example, create a large
LArray[Byte], then align your object data on the array. Object parameters can be retrieved with
- To release objects from main memory, you need to create off-heap objects. For example, create a large
Here is a simple benchmark result that compares concurrent memory-allocation performances of LArray (with or without zero-filling), java arrays,
ByteBuffer.allocateDirect, using Mac OS X with 2.9GHz Intelli Core i7. This test allocates 100 x 1MB of memory space concurrently using multiple threads, and repeats this process 20 times.
-concurrent allocation total:2.426 sec. , count: 10, avg:0.243 sec. , core avg:0.236 sec. , min:0.159 sec. , max:0.379 sec. -without zero-filling total:0.126 sec. , count: 20, avg:6.279 msec., core avg:2.096 msec., min:1.405 msec., max:0.086 sec. -with zero-filling total:0.476 sec. , count: 20, avg:0.024 sec. , core avg:0.023 sec. , min:0.017 sec. , max:0.037 sec. -java array total:0.423 sec. , count: 20, avg:0.021 sec. , core avg:0.021 sec. , min:0.014 sec. , max:0.029 sec. -byte buffer total:1.028 sec. , count: 20, avg:0.051 sec. , core avg:0.044 sec. , min:0.014 sec. , max:0.216 sec. -direct byte buffer total:0.360 sec. , count: 20, avg:0.018 sec. , core avg:0.018 sec. , min:0.015 sec. , max:0.026 sec.
All allocators except LArray are orders of magnitude slower than LArray, and consumes CPUs because they need to fill the allocated memory with zeros due to their specification.
In a single thread execution, you can see more clearly how fast LArray can allocate memories.
-single-thread allocation total:3.655 sec. , count: 10, avg:0.366 sec. , core avg:0.356 sec. , min:0.247 sec. , max:0.558 sec. -without zero-filling total:0.030 sec. , count: 20, avg:1.496 msec., core avg:1.125 msec., min:0.950 msec., max:8.713 msec. -with zero-filling total:0.961 sec. , count: 20, avg:0.048 sec. , core avg:0.047 sec. , min:0.044 sec. , max:0.070 sec. -java array total:0.967 sec. , count: 20, avg:0.048 sec. , core avg:0.037 sec. , min:0.012 sec. , max:0.295 sec. -byte buffer total:0.879 sec. , count: 20, avg:0.044 sec. , core avg:0.033 sec. , min:0.014 sec. , max:0.276 sec. -direct byte buffer total:0.812 sec. , count: 20, avg:0.041 sec. , core avg:0.041 sec. , min:0.032 sec. , max:0.049 sec.
LArray (and LBuffer) has memory address that can be used for seamlessly interacting with fast native methods through JNI. Here is an example of using
rawCompress(...) in snappy-java, which can take raw-memory address to compress/uncompress the data using C++ code, and is generally faster than Dain's pure-java version of Snappy.
[SnappyCompressTest] -compress total:0.017 sec. , count: 10, avg:1.669 msec., core avg:0.769 msec., min:0.479 msec., max:0.010 sec. -LBuffer -> LBuffer (raw) total:1.760 msec., count: 50, avg:0.035 msec., core avg:0.030 msec., min:0.024 msec., max:0.278 msec. -Array -> Array (raw) total:1.450 msec., count: 50, avg:0.029 msec., core avg:0.027 msec., min:0.023 msec., max:0.110 msec. -Array -> Array (dain) total:0.011 sec. , count: 50, avg:0.225 msec., core avg:0.141 msec., min:0.030 msec., max:4.441 msec. [SnappyCompressTest] -decompress total:7.722 msec., count: 10, avg:0.772 msec., core avg:0.473 msec., min:0.418 msec., max:3.521 msec. -LBuffer -> LBuffer (raw) total:1.745 msec., count: 50, avg:0.035 msec., core avg:0.029 msec., min:0.020 msec., max:0.331 msec. -Array -> Array (raw) total:1.189 msec., count: 50, avg:0.024 msec., core avg:0.021 msec., min:0.018 msec., max:0.149 msec. -Array -> Array (dain) total:2.571 msec., count: 50, avg:0.051 msec., core avg:0.027 msec., min:0.025 msec., max:1.240 msec.
- [Test code](larray/src/test/scala/xerial/larray/SnappyCompressTest.scala)
LArray consists of three-modules.
- larray-buffer (Java) Off-heap memory buffer
LBufferand its allocator with GC support.
- larray-mmap (Java + JNI (C code)) Memory-mapped file implementation
- larray (Scala and Java API) Provides rich set of array operations through
You can use each module independently. For example, if you only need an off-heap memory allocator that collects memory upon GC, use
LBuffer in larray-buffer.
Simply you can include larray to the dependency in Maven or SBT so that all modules will be added to your classpaths.
A standard JVM, (e.g. Oracle JVM (standard JVM, HotSpotVM) or OpenJDK) must be used since
larray-buffer depends on
sun.misc.Unsafe class to allocate off-heap memory.
larray-mmap (MMapBuffer and LArray.mmap) uses JNI and is available for the following major CPU architectures:
- Windows (32/64-bit)
- Linux (i368, amd64 (Intel 64-bit), arm, armhf)
- Mac OSX (Intel 64bit)
- 2016-12-13: vesrion 0.4.0 - Fix #52. Support Scala 2.12. Use wvlet-log for internal logging.
- 2016-05-12: version 0.3.4 - Minor performance improvement release
- 2016-03-04: version 0.3.3 - Add Scala 2.11.7, 2.10.6 support
- March 4th, 2016 version 0.3.0 - Scala 2.11.7 support
- November 11, 2013 version 0.2.1 - Use orgnization name
org.xerial.larray. Add LBuffer.view.
- November 11, 2013 version 0.2 - Extracted pure-java modules (larray-buffer.jar and larray-mmap.jar) from larray.jar (for Scala).
- August 28, 2013 version 0.1.2 - improved memory layout
- August 28, 2013 version 0.1.1 (for Scala 2.10.2)
- Apr 23, 2013 Released version 0.1
Add the following sbt dependency to your project settings:
libraryDependencies += "org.xerial.larray" %% "larray" % "0.4.0"
- Using snapshot versions:
resolvers += "Sonatype shapshot repo" at "https://oss.sonatype.org/content/repositories/snapshots/" libraryDependencies += "org.xerial.larray" %% "larray" % "0.4.1-SNAPSHOT"
LArray can be used in the same manner with the standard Scala Arrays:
import xerial.larray._ val l = LArray(1, 2, 3) val e = l(0) // 1 println(l.mkString(", ")) // 1, 2, 3 l(1) = 5 println(l.mkString(", ")) // 1, 5, 3 // Create an LArray of Int type val l2 = LArray.of[Int](10000L) // Release the memory resource l2.free l2(0) // The result of accessing released LArray is undefined
For more examples, see [xerial/larray/example/LArrayExample.scala](larray/src/main/scala/xerial/larray/example/LArrayExample.scala)
Add the following dependency to your pom.xml (Maven):
<dependency> <groupId>org.xerial.larray</groupId> <artifactId>larray_2.12</artifactId> <version>0.4.0</version> </dependency>
In Java we cannot provide concise syntaxes as in Scala. Instead, use
update methods to read/write values in an array.
import xerial.larray.japi.LArrayJ; import xerial.larray.*; LIntArray l = LArrayJ.newLIntArray(10000L); l.update(0L, 20); // Set l[0L] = 20 int e0 = l.apply(0L); // Get l[0L] // release l.free();
For more examples, see [xerial/larray/example/LArrayJavaExample.java](larray/src/main/scala/xerial/larray/example/LArrayJavaExample.java)
- Building LArray:
- Run tests:
- Creating IntelliJ IDEA project: