Scio alternatives and similar packages
Based on the "Big Data" category.
Alternatively, view Scio alternatives based on common mentions on social networks and blogs.
-
-
-
Deeplearning4J
Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learning using automatic differentiation. -
-
-
-
Reactive-kafka
Alpakka Kafka connector - Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka. -
-
-
-
-
-
-
-
-
-
-
Schemer
Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API. -
-
GridScale
Scala library for accessing various file, batch systems, job schedulers and grid middlewares. -
Spark Utils
Basic framework utilities to quickly start writing production ready Apache Spark applications -
WorkOS - The modern identity platform for B2B SaaS
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of Scio or a related project?
README
Scio
[
](./LICENSE)
Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o] Verb: I can, know, understand, have knowledge.
Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.
Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow). See this page for a list of breaking changes.
Features
- Scala API close to that of Spark and Scalding core APIs
- Unified batch and streaming programming model
- Fully managed service*
- Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable
- JDBC, TensorFlow TFRecords, Cassandra, Elasticsearch and Parquet I/O
- Interactive mode with Scio REPL
- Type safe BigQuery
- Integration with Algebird and Breeze
- Pipeline orchestration with Scala Futures
- Distributed cache
* provided by Google Cloud Dataflow
Quick Start
Download and install the Java Development Kit (JDK) version 8.
Install sbt.
Use our giter8 template to quickly create a new Scio job repository:
sbt new spotify/scio.g8
Switch to the new repo (default scio-job) and build it:
cd scio-job
sbt stage
Run the included word count example:
target/universal/stage/bin/scio-job --output=wc
List result files and inspect content:
ls -l wc
cat wc/part-00000-of-00004.txt
Documentation
Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.
Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.
- Scio Docs - main documentation site
- Scio Scaladocs - current API documentation
- Scio Examples - examples with side-by-side explanation
Artifacts
Scio includes the following artifacts:
scio-core: core libraryscio-test: test utilities, add to your project as a "test" dependencyscio-avro: add-on for Avro, can also be used standalonescio-google-cloud-platform: add-on for Google Cloud IO's: BigQuery, Bigtable, Pub/Sub, Datastore, Spannerscio-cassandra*: add-ons for Cassandrascio-elasticsearch*: add-ons for Elasticsearchscio-extra: extra utilities for working with collections, Breeze, etc., best effort supportscio-jdbc: add-on for JDBC IOscio-neo4j: add-on for Neo4J IOscio-parquet: add-on for Parquetscio-tensorflow: add-on for TensorFlow TFRecords IO and predictionscio-redis: add-on for Redisscio-smb: add-on for Sort Merge Bucket operationsscio-repl: extension of the Scala REPL with Scio specific operations
License
Copyright 2021 Spotify AB.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
*Note that all licence references and agreements mentioned in the Scio README section above
are relevant to that project's source code only.