All Versions
26
Latest Version
Avg Release Cycle
27 days
Latest Release
1583 days ago

Changelog History
Page 2

  • v0.8.3 Changes

    March 19, 2020

    "Draconifors"

    🔋 Features

    • ➕ Add CsvIO (#2701)
    • Implement hashSubtractByKey (#2769)
    • ➕ Add directory treatment and compression option to readFiles (#2756)
    • ➕ Add FileNaming and prefix support to BinaryIO (#2759)
    • ➕ Add Header and Footer support to TextIO (#2758)
    • ➕ Add GenericJson coder (#2742)
    • ➕ Add ProtobufUtil for Message<->Avro conversions (#2743)

    🐛 Bug fixes & improvements

    • ➕ Add schema cache support in BigQueryType (#2775)
    • ⚡️ Optimize KeyGroupIterator (#2779)
    • 👉 Make CoGbkResult Iterable slightly lazy (#2749)
    • 👉 Make java Enum coder deterministic #2761 #2764
    • 🛠 Fix saveAsTypedBigQuery deprecation message (#2763)
    • Patch SortValues doFn to avoid re-enconding bytes (#2748)
    • 🛠 Fix: remove redundant withExtendedErrorInfo in dynamic BigQuery (#2744)

  • v0.8.2 Changes

    March 02, 2020

    "Capacious Extremis"

    🔋 Features

    • ⚡️ Update Beam to 2.19.0 (#2665)
    • ➕ Add SCollection#tap (#2713)
    • ➕ Add SortedBucketTransform (#2693)
    • ➕ Add API bindings for SortedBucketTransform (#2712)
    • ➕ Add counters to SMB read transforms (#2704)
    • ➕ Add ParquetExampleIO (#2696)
    • 👍 Allow setting MIME type on RemoteFileUtil.upload (#2711)
    • ➕ Add BigQuery macro Avro Schema and toAvro support (#2646)

    🐛 Bug fixes & improvements

    • ⚡️ Ignore updates on google libs that we need to keep in sync with beam (#2728)
    • ➕ Add all scio compile deps (#2634)
    • ⬆️ Bump circe to 0.13.0 (#2717)
    • 🛠 Fix BigQuery Storage default params (#2692)
    • 🛠 Fix: TableRow#getRecord returns Map (#2710)
    • 🛠 Fix: eval coder eagerly for sealead traits (#2699)
    • ➕ Add readFiles alternative to deprecated readAll (#2697)
    • ⚡️ Minor update topByKey comments (#2695)
    • ✂ Remove duplicate stage files (#2678)
    • 👉 Make files to stage list mutable (#2676)
    • 👉 Make Coder[Set] throw NonDeterministicException (#2674)
    • Prevent sharded-sparkey shards from having negative indices. (#2667)
    • ⬇️ Reduce SMB Dataflow graph size for many-bucketed joins (#2666)
    • ➕ Add test for #2649, Avro map field in SMB sink (#2664)
    • 👮 Force GenericRecord writer in SMB Avro, fix 2649 (#2650)
    • 🛠 Fix: Add missing toAvro to BigQueryType (#2657)
    • ✂ Remove patched bigquery rows iterator (#2647)

  • v0.8.1 Changes

    January 31, 2020

    "Bombarda Maxima"

    🔋 Features

    • ⚡️ Update Beam to 2.18 (#2631)
    • ➕ Add BigQuery Geography support (#2620)
    • ➕ Add simple sorter method (#2542)
    • ➕ Add Avro BigQuery logical types conversion (#2602)

    🐛 Bug fixes & improvements

    • ➕ Add Map/MultiMap singleton side input, fix #2628 (#2629)
    • 🛠 Fix ClassCastEx in PubSubIO (#2583)
    • 🛠 Fix sharded Sparkey side input with missing values. (#2617)
    • ➕ Add specialized coders for Elasticsearch, fix #2626 (#2637)
    • 👉 Make KryoAtomicCoder.instanceId an incrementing integer, not a UUID (#2589)
    • 🛠 Fix BigQuery caching, by using dry-run when extracting tables (#2613)
    • 🛠 Fix recursive coders (#2605)
    • Avoid NPE when finding rt.jar under jdk11 (#2610)
    • ✅ More tests on Coders (#2639)
    • 🤡 Consolidate IT mock data generation (#2624)
    • 🛠 Fix sql script query generator
    • Avro Message record from scio-schemas could hide protobuf Message (#2609)
    • 📇 Rename metricsLocation option parameter (#2603)
    • ➕ Add comment on scio-extra support level
    • ➕ Add test case for avro conversion with bytes (#2594)
    • ➕ Add lisp-case arg parsing removal (#2592) (#2221)
    • ✂ Remove unused ClassTag (#2590)
    • 🛠 Fix BigQuery deprecated method message (#2586)
    • ➕ Add scalafmt to scalafix rules (#2491), rename join names (#2455), rename named args: that to rhs (#2468)
    • 🛠 Fix BigQuery deprecated method message (#2586)
    • ➕ Add elasticsearch example (#2388)
    • ✂ Remove mdoc annotation for macro-based Parquet.md examples (#2581)
    • ⚡️ Update doc and migration guides (#2585)
    • 👉 Use weakTypeOf in scio-bigquery (#2641)
    • ⬇️ Drop FastEval in scio-sql (#2622)

  • v0.8.0 Changes

    July 16, 2019

    🔋 Features

    • ⚡️ Update beam to 2.13.0 (#1952)
    • ➕ Add partitionToMap function (#1968)
    • 🆕 New async DoFns (#1971)
    • 🔦 Expose Beam's AvroIO.parseGenericRecords (#1973)(#2051)
    • ➕ Add batchByKey function (#2009)
    • ➕ Add elasticsearch7 module (#2004)
    • ➕ Add support for recursive Coders (#2032)(#2048)
    • ➕ Adding filterValues function (#1964)

    🐛 Bug Fixes & Improvements

    • 🛠 Fix equality for mutable bf sparse instances (#1972)
    • ➕ Add bq not implemented tap when write append (#2042)
    • 🛠 Fix schema inference from BigQueryType (#2027)
    • Explicitly name the transform in close() that initializes counters (#2047)
    • Rework ClosedScioContext into ScioRunner (#2036)(#2055)
    • ➕ Add incorrect table aliases hint (#2053)
    • 👍 Allow app/job name override in scio-repl (#2046)
    • 👉 Use BigtableOptions param for BigTable utils (#2003)
    • 👻 InMemorySink will now create an empty collection in the cache so empty SCollections also return an iterable instead of throwing an exception (#2039)
    • ➕ Add WrappedArray coder (#1945)
    • 🛠 Fix sql interpolator tag index (#1961)
    • 📜 Use composite transform for sparse transforms (#1947)
    • 🛠 Fix benchmarks (#1976)(#2005)
    • 👷 Set location when fetching Job in JobOps (#1966)
    • Cleanup materialize and set coder (#2016)
    • 🔨 Refactor applyPerKey (#2011)(#1978)
    • 🛠 Fix BigtableDoFn (#2001)
    • 👌 Support passing Assertions directly to JobTest output (#1993)(#1995)
    • ⚡️ Update docs (#2012)
    • ➕ Add deprecation dates to Apache-Beam.md
    • ✂ Remove deprecation warnings (#2041)(#2045)
    • ✂ Remove uneed implicit context and duplicated logger (#2018)
    • 🚀 Warn only about latest stable release (non prerelease, draft) (#1980)
    • ⚡️ Dependency updates (#1962)(#1960)(#1956)(#1957)(#1948)(#1965)(#1990)(#2023)(#2033)(#1977)(#1998(#2052)

  • v0.8.0-beta2 Changes

    October 16, 2019

    🔋 Features

    • ⬆️ Bump Beam to 2.16.0 (#2292)
    • ⬆️ Bump featran to 0.4.0
    • ➕ Add DelegatingSideInput and TypedSparkeySideInput. (#2287)
    • ➕ Add initial BigQuery methods with Beam schema (#2197)
    • 👍 Allow user to pass in Bigtable GcRule (#2256)
    • 👌 Support accessing all counters within JobTest (fix #2007) (#2247)
    • ➕ Add BigQuery Source type (#2214)
    • ➕ Add Storage API read from query (#2204)

    🐛 Bug Fixes & Improvements

    • 🗄 Segregate and deprecate avro reflective records support (#2300)
    • 🛠 Fix lambda ser/de in SCollectionMatchers, #2294 (#2296)
    • 📜 Use PubsubIO.readMessagesWithCoderAndParseFn from Beam 2.16 (#2295)
    • Wrap PubsubWithAttributes write in composite PTransform (fix #2286) (#2288)
    • AvroType macro shouldn't generate duplicate record classes fix #2264) (#2267)
    • ➕ Add Paradise dependency back to scio-repl classpath (fix #2265) (#2266)
    • 👉 Use JavaSerializer for encoding Throwable (fix #2249) (#2261)
    • Check name uniqueness without running on Dataflow (#2252)
    • ➕ Added fix for BinaryIO output. (#2251)
    • Truncate app args if long (#2244)
    • 🛠 Fix: run repl with jdk11 (#2246)
    • ⬇️ Reduce guava usage #2223 #2242
    • 👉 Use upstream magnolia for scala >= 2.12, fix #2192 (#2241)
    • 🛠 Fix #2096: restore sbt < 1.3.0 behaviour by not closing classloader (#2224)
    • ⚡️ Update version check message (#2227)
    • Don't wrap exceptions in WrappedBCoder (#2266)
    • Bypass nullable coder for ObjectFileIO, fix #2187 (#2188)

    💥 Breaking Changes

    • Decouple SQL extension from scio-core (#2201)

  • v0.8.0-beta1

    August 23, 2019

  • v0.8.0-alpha2 Changes

    July 16, 2019

    🔋 Features

    • ⚡️ Update beam to 2.13.0 (#1952)
    • ➕ Add partitionToMap function (#1968)
    • 🆕 New async DoFns (#1971)
    • 🔦 Expose Beam's AvroIO.parseGenericRecords (#1973)(#2051)
    • ➕ Add batchByKey function (#2009)
    • ➕ Add elasticsearch7 module (#2004)
    • ➕ Add support for recursive Coders (#2032)(#2048)
    • ➕ Adding filterValues function (#1964)

    🐛 Bug Fixes & Improvements

    • 🛠 Fix equality for mutable bf sparse instances (#1972)
    • ➕ Add bq not implemented tap when write append (#2042)
    • 🛠 Fix schema inference from BigQueryType (#2027)
    • Explicitly name the transform in close() that initializes counters (#2047)
    • Rework ClosedScioContext into ScioRunner (#2036)(#2055)
    • ➕ Add incorrect table aliases hint (#2053)
    • 👍 Allow app/job name override in scio-repl (#2046)
    • 👉 Use BigtableOptions param for BigTable utils (#2003)
    • 👻 InMemorySink will now create an empty collection in the cache so empty SCollections also return an iterable instead of throwing an exception (#2039)
    • ➕ Add WrappedArray coder (#1945)
    • 🛠 Fix sql interpolator tag index (#1961)
    • 📜 Use composite transform for sparse transforms (#1947)
    • 🛠 Fix benchmarks (#1976)(#2005)
    • 👷 Set location when fetching Job in JobOps (#1966)
    • Cleanup materialize and set coder (#2016)
    • 🔨 Refactor applyPerKey (#2011)(#1978)
    • 🛠 Fix BigtableDoFn (#2001)
    • 👌 Support passing Assertions directly to JobTest output (#1993)(#1995)
    • ⚡️ Update docs (#2012)
    • ➕ Add deprecation dates to Apache-Beam.md
    • ✂ Remove deprecation warnings (#2041)(#2045)
    • ✂ Remove uneed implicit context and duplicated logger (#2018)
    • 🚀 Warn only about latest stable release (non prerelease, draft) (#1980)
    • ⚡️ Dependency updates (#1962)(#1960)(#1956)(#1957)(#1948)(#1965)(#1990)(#2023)(#2033)(#1977)(#1998(#2052)

  • v0.8.0-alpha1 Changes

    May 29, 2019

    Highlights

    🔋 Features

    🐛 Bug Fixes & Improvements

    • ➕ Add defaults for BinaryIO (#1919)
    • ➕ Add BigQuery NUMERIC missing cases (#1827)
    • ➕ Add Spanner syntax and instances package (#1863)
    • ➕ Add bigquery syntax package (#1838)
    • ➕ Add dynamic syntax package (#1920)
    • 🔄 Change default fpProb of sparseIntersectByKey (#1935)
    • Wrap unsafe access of nullable fields and methods in try (#1930)
    • 🛠 Fix FileStorage implicit access modifier (#1909)
    • ✅ Remote sys.exit from code path which is used in tests (#1900)
    • 🔨 Refactor bigquery schema/table cache (#1886)
    • Correctly format dataset id (#1907)
    • ➕ Add header/footer/prefix/suffix to BinaryIO, fix #1785 (#1884)
    • ➕ Add table type to disambiguate spec and reference (#1866)
    • 👍 Replace IllegalStateException with UnsupportedOperationException (#1861)
    • 🔨 Refactor sparse intersect by key (#1881)
    • scio-tensorflow cleanup (#1879)
    • 🎉 Initialize counters from ScioContext (#1873)
    • 🛠 Fix #1866 bigquery it tests (#1878)
    • Add warn combine*/sum* transform usage, fix #1620 (#1856)
    • Create empty Sparkey file on empty collection (#1834)
    • 👉 Use java.io.tmpdir and username as cache dir (#1843)
    • ⚡️ Update WriteDynamicExample, fix #1786 (#1802)
    • 🛠 Fix KryAtomicCoder null encoding (#1841)
    • ➕ Add jmh benchmarks for bloom filters (#1913)
    • 🛠 Fix AvroIO type bounds (#1799)
    • ➕ Add more join tests (#1797)
    • 👻 Avro map string key cast exception (#1832)
    • ➕ Add toString on Coders impl (#1819)
    • ➕ Add a hand written Coder for pairs (#1775)
    • ✂ Remove usage of deprecated api (#1812)
    • Enable higher kinds and partial unification (#1810)
    • 👕 Enable more Xlint options (#1816)
    • Enable -Ywarn-value-discard (#1815)
    • 🛠 Fix schema parsing in BQ TypeProvider
    • 🛠 Fix NPE in coders with nullableCoders=true (#1820)
    • 👉 Make BQ annotations serializable (#1773)
    • 🛠 Fix: use same protoc (#1781)
    • 👉 Use camelCase for typed arguments, fix #1770 (#1780)
    • ⬆️ Bump versions (#1801)
    • 🛠 Fix scalac options for 2.11
    • ⏪ Revert auto-service to 1.0-RC2. Same version as beam (#1837)
    • ⚡️ Dependency updates (#1807)(#1803)(#1928)(#1924)(#1916)(#1914)(#1868)(#1875)(#1836)
    • 🛠 Fix more scalac warns (#1844)
    • 🔌 Quieted down code dumper for IntelliJ Scala Plugin (#1854)
    • 👉 Make ensureSerializable return Either (#1798)
    • ✅ Disable checkVersion in Scio's tests (#1908)
    • 🔨 Refactor jdbc io (#1910)
    • ⚡️ Updated overall documentaion (#1906)(#1941)(#1890)(#1936)(#1817)(#1825)

    💥 Breaking changes

    • ✂ Remove tensorflow methods related to schema inference (#1849)
    • ✂ Remove the usage of Future around ScioContext and Tap's (#1666)
    • ✂ Remove references of Future[Tap[T]] (#1835)

  • v0.7.4 Changes

    March 25, 2019

    "Watsonula wautieri"

    🔋 Features

    • ➕ Add sequence example support to scio-tensorflow (#1757)
    • ➕ Add compile time warning when Coder fallsback to Kryo on GenericRecord (#1768)
    • ⬆️ Bump Beam to 2.11 (#1739)
    • ⚡️ Update probuf version to 3.7.0 (#1752)
    • ⬆️ Bump GCS connector version (#1766)
    • ⬆️ Bump casssandra-driver-core to 3.7.1 (#1760)

    🐛 Bug Fixes & Improvements

    • ⚡️ Update coursier (#1751)
    • 🛠 Fix project ID in Spanner admin client (#1771)
    • 🛠 Fix spanner IT test
    • 🔨 Refactor Tensorflow SCollection syntax (#1763)
    • 📇 Rename Avro ScioContext ops class (#1765)
    • ➕ Add explict project to fix failing ScioIOBenchmark (#1764)
    • 📇 Rename unify TF save function names (#1762)
    • ➕ Add FAQ for IntelliJ SBT heap
    • ✂ Remove outdated maintainers notes in FAQ.md
    • Minor doc site fix (#1759)
    • 🔨 Refactor avro syntax (#1753)
    • Avro IO with type bound (#1737)
    • 🚀 Update release docs

  • v0.7.3 Changes

    March 12, 2019

    "Vulpes Vulpes"

    🐛 Bug Fixes & Improvements

    • 🛠 Fix FileStorage.avroFile (#1727)
    • 🛠 Fix perf regression in Coder (#1729)
    • ⬇️ Reduce the size of the captured stacktrace in WrappedBCoder (#1745)
    • 🛠 Fix #1734: Limit job graph size by not wrapping native beam coders (#1741)
    • 👀 Explicit reset position on SeekableInput (#1747)
    • 👌 Support scalatest NotWord (#1743)
    • 👉 make BigQuery priority sysprop case-insensitive (#1736)
    • 👉 Use getSchema and avoid reflection when creating AvroCoder (#1724)
    • 👷 Clarify error message when a job uses an input multiple times. (#1720)
    • ✏️ tiny typos in Coders.md (#1732)
    • Incorrect generic type in ScalaDoc (#1725)
    • 👉 Use BenchmarkResult as entity (#1712)