Time Series for Spark (distributed as the spark-ts package) is a Scala / Java / Python library for analyzing large-scale time series data sets. It is hosted here.

Post questions and comments to the Google group, or email them directly to <mailto:spark-ts@googlegroups.com.

Note: The spark-ts library is no longer under active development by me (Sandy). I unfortunately no longer have bandwidth to develop features, answer all questions on the mailing list, or fix all bugs that are filed. That said, I remain happy to review pull requests and do whatever I can to aid others in advancing the library.

Check out the examples repository for a taste of what it’s like to use the library.

Check out the Scaladoc, Javadoc, or Python doc.

Time Series for Spark offers:

• A set of abstractions for manipulating time series data, similar to what’s provided for smaller data sets in Pandas, Matlab, and R’s zoo and xts packages.
• Models, tests, and functions that enable dealing with time series from a statistical perspective, similar to what’s provided in StatsModels and a variety of Matlab and R packages.

The library is geared towards use cases in finance (munging tick data, building risk models), but intends to be general enough that other fields with continuous time series data, like meteorology, can make use of it.

The library currently expects that individual univariate time series can easily fit in memory on each machine, but that collections of univariate time series may need to be distributed across many machines. While time series that violate this expectation pose a bunch of fun distributed programming problems, they don’t tend to come up very often in finance, where an array holding a value for every minute of every trading day for ten years needs less than a couple million elements.