All Versions
12
Latest Version
Avg Release Cycle
26 days
Latest Release
91 days ago

Changelog History
Page 1

  • v0.8.0

    July 16, 2019

    🔋 Features

    • ⚡️ Update beam to 2.13.0 (#1952)
    • ➕ Add partitionToMap function (#1968)
    • 🆕 New async DoFns (#1971)
    • 🔦 Expose Beam's AvroIO.parseGenericRecords (#1973)(#2051)
    • ➕ Add batchByKey function (#2009)
    • ➕ Add elasticsearch7 module (#2004)
    • ➕ Add support for recursive Coders (#2032)(#2048)
    • ➕ Adding filterValues function (#1964)

    🐛 Bug Fixes & Improvements

    • 🛠 Fix equality for mutable bf sparse instances (#1972)
    • ➕ Add bq not implemented tap when write append (#2042)
    • 🛠 Fix schema inference from BigQueryType (#2027)
    • Explicitly name the transform in close() that initializes counters (#2047)
    • Rework ClosedScioContext into ScioRunner (#2036)(#2055)
    • ➕ Add incorrect table aliases hint (#2053)
    • 👍 Allow app/job name override in scio-repl (#2046)
    • 👉 Use BigtableOptions param for BigTable utils (#2003)
    • 👻 InMemorySink will now create an empty collection in the cache so empty SCollections also return an iterable instead of throwing an exception (#2039)
    • ➕ Add WrappedArray coder (#1945)
    • 🛠 Fix sql interpolator tag index (#1961)
    • 📜 Use composite transform for sparse transforms (#1947)
    • 🛠 Fix benchmarks (#1976)(#2005)
    • 👷 Set location when fetching Job in JobOps (#1966)
    • Cleanup materialize and set coder (#2016)
    • ♻️ Refactor applyPerKey (#2011)(#1978)
    • 🛠 Fix BigtableDoFn (#2001)
    • 👌 Support passing Assertions directly to JobTest output (#1993)(#1995)
    • ⚡️ Update docs (#2012)
    • ➕ Add deprecation dates to Apache-Beam.md
    • ✂ Remove deprecation warnings (#2041)(#2045)
    • ✂ Remove uneed implicit context and duplicated logger (#2018)
    • 🚀 Warn only about latest stable release (non prerelease, draft) (#1980)
    • ⚡️ Dependency updates (#1962)(#1960)(#1956)(#1957)(#1948)(#1965)(#1990)(#2023)(#2033)(#1977)(#1998(#2052)
  • v0.8.0-beta1

    August 23, 2019
  • v0.8.0-alpha2

    July 16, 2019

    🔋 Features

    • ⚡️ Update beam to 2.13.0 (#1952)
    • ➕ Add partitionToMap function (#1968)
    • 🆕 New async DoFns (#1971)
    • 🔦 Expose Beam's AvroIO.parseGenericRecords (#1973)(#2051)
    • ➕ Add batchByKey function (#2009)
    • ➕ Add elasticsearch7 module (#2004)
    • ➕ Add support for recursive Coders (#2032)(#2048)
    • ➕ Adding filterValues function (#1964)

    🐛 Bug Fixes & Improvements

    • 🛠 Fix equality for mutable bf sparse instances (#1972)
    • ➕ Add bq not implemented tap when write append (#2042)
    • 🛠 Fix schema inference from BigQueryType (#2027)
    • Explicitly name the transform in close() that initializes counters (#2047)
    • Rework ClosedScioContext into ScioRunner (#2036)(#2055)
    • ➕ Add incorrect table aliases hint (#2053)
    • 👍 Allow app/job name override in scio-repl (#2046)
    • 👉 Use BigtableOptions param for BigTable utils (#2003)
    • 👻 InMemorySink will now create an empty collection in the cache so empty SCollections also return an iterable instead of throwing an exception (#2039)
    • ➕ Add WrappedArray coder (#1945)
    • 🛠 Fix sql interpolator tag index (#1961)
    • 📜 Use composite transform for sparse transforms (#1947)
    • 🛠 Fix benchmarks (#1976)(#2005)
    • 👷 Set location when fetching Job in JobOps (#1966)
    • Cleanup materialize and set coder (#2016)
    • ♻️ Refactor applyPerKey (#2011)(#1978)
    • 🛠 Fix BigtableDoFn (#2001)
    • 👌 Support passing Assertions directly to JobTest output (#1993)(#1995)
    • ⚡️ Update docs (#2012)
    • ➕ Add deprecation dates to Apache-Beam.md
    • ✂ Remove deprecation warnings (#2041)(#2045)
    • ✂ Remove uneed implicit context and duplicated logger (#2018)
    • 🚀 Warn only about latest stable release (non prerelease, draft) (#1980)
    • ⚡️ Dependency updates (#1962)(#1960)(#1956)(#1957)(#1948)(#1965)(#1990)(#2023)(#2033)(#1977)(#1998(#2052)
  • v0.8.0-alpha1

    May 29, 2019

    Highlights

    🔋 Features

    🐛 Bug Fixes & Improvements

    • ➕ Add defaults for BinaryIO (#1919)
    • ➕ Add BigQuery NUMERIC missing cases (#1827)
    • ➕ Add Spanner syntax and instances package (#1863)
    • ➕ Add bigquery syntax package (#1838)
    • ➕ Add dynamic syntax package (#1920)
    • 🔄 Change default fpProb of sparseIntersectByKey (#1935)
    • Wrap unsafe access of nullable fields and methods in try (#1930)
    • 🛠 Fix FileStorage implicit access modifier (#1909)
    • ✅ Remote sys.exit from code path which is used in tests (#1900)
    • ♻️ Refactor bigquery schema/table cache (#1886)
    • Correctly format dataset id (#1907)
    • ➕ Add header/footer/prefix/suffix to BinaryIO, fix #1785 (#1884)
    • ➕ Add table type to disambiguate spec and reference (#1866)
    • 👍 Replace IllegalStateException with UnsupportedOperationException (#1861)
    • ♻️ Refactor sparse intersect by key (#1881)
    • scio-tensorflow cleanup (#1879)
    • 🎉 Initialize counters from ScioContext (#1873)
    • 🛠 Fix #1866 bigquery it tests (#1878)
    • Add warn combine*/sum* transform usage, fix #1620 (#1856)
    • Create empty Sparkey file on empty collection (#1834)
    • 👉 Use java.io.tmpdir and username as cache dir (#1843)
    • ⚡️ Update WriteDynamicExample, fix #1786 (#1802)
    • 🛠 Fix KryAtomicCoder null encoding (#1841)
    • ➕ Add jmh benchmarks for bloom filters (#1913)
    • 🛠 Fix AvroIO type bounds (#1799)
    • ➕ Add more join tests (#1797)
    • 👻 Avro map string key cast exception (#1832)
    • ➕ Add toString on Coders impl (#1819)
    • ➕ Add a hand written Coder for pairs (#1775)
    • ✂ Remove usage of deprecated api (#1812)
    • Enable higher kinds and partial unification (#1810)
    • 👕 Enable more Xlint options (#1816)
    • Enable -Ywarn-value-discard (#1815)
    • 🛠 Fix schema parsing in BQ TypeProvider
    • 🛠 Fix NPE in coders with nullableCoders=true (#1820)
    • 👉 Make BQ annotations serializable (#1773)
    • 🛠 Fix: use same protoc (#1781)
    • 👉 Use camelCase for typed arguments, fix #1770 (#1780)
    • ⬆️ Bump versions (#1801)
    • 🛠 Fix scalac options for 2.11
    • ⏪ Revert auto-service to 1.0-RC2. Same version as beam (#1837)
    • ⚡️ Dependency updates (#1807)(#1803)(#1928)(#1924)(#1916)(#1914)(#1868)(#1875)(#1836)
    • 🛠 Fix more scalac warns (#1844)
    • 🔌 Quieted down code dumper for IntelliJ Scala Plugin (#1854)
    • 👉 Make ensureSerializable return Either (#1798)
    • ✅ Disable checkVersion in Scio's tests (#1908)
    • ♻️ Refactor jdbc io (#1910)
    • ⚡️ Updated overall documentaion (#1906)(#1941)(#1890)(#1936)(#1817)(#1825)

    💥 Breaking changes

    • ✂ Remove tensorflow methods related to schema inference (#1849)
    • ✂ Remove the usage of Future around ScioContext and Tap's (#1666)
    • ✂ Remove references of Future[Tap[T]] (#1835)
  • v0.7.4

    March 25, 2019

    "Watsonula wautieri"

    🔋 Features

    • ➕ Add sequence example support to scio-tensorflow (#1757)
    • ➕ Add compile time warning when Coder fallsback to Kryo on GenericRecord (#1768)
    • ⬆️ Bump Beam to 2.11 (#1739)
    • ⚡️ Update probuf version to 3.7.0 (#1752)
    • ⬆️ Bump GCS connector version (#1766)
    • ⬆️ Bump casssandra-driver-core to 3.7.1 (#1760)

    🐛 Bug Fixes & Improvements

    • ⚡️ Update coursier (#1751)
    • 🛠 Fix project ID in Spanner admin client (#1771)
    • 🛠 Fix spanner IT test
    • ♻️ Refactor Tensorflow SCollection syntax (#1763)
    • 📇 Rename Avro ScioContext ops class (#1765)
    • ➕ Add explict project to fix failing ScioIOBenchmark (#1764)
    • 📇 Rename unify TF save function names (#1762)
    • ➕ Add FAQ for IntelliJ SBT heap
    • ✂ Remove outdated maintainers notes in FAQ.md
    • Minor doc site fix (#1759)
    • ♻️ Refactor avro syntax (#1753)
    • Avro IO with type bound (#1737)
    • 🚀 Update release docs
  • v0.7.3

    March 12, 2019

    "Vulpes Vulpes"

    🐛 Bug Fixes & Improvements

    • 🛠 Fix FileStorage.avroFile (#1727)
    • 🛠 Fix perf regression in Coder (#1729)
    • ⬇️ Reduce the size of the captured stacktrace in WrappedBCoder (#1745)
    • 🛠 Fix #1734: Limit job graph size by not wrapping native beam coders (#1741)
    • 👀 Explicit reset position on SeekableInput (#1747)
    • 👌 Support scalatest NotWord (#1743)
    • 👉 make BigQuery priority sysprop case-insensitive (#1736)
    • 👉 Use getSchema and avoid reflection when creating AvroCoder (#1724)
    • 👷 Clarify error message when a job uses an input multiple times. (#1720)
    • ✏️ tiny typos in Coders.md (#1732)
    • Incorrect generic type in ScalaDoc (#1725)
    • 👉 Use BenchmarkResult as entity (#1712)
  • v0.7.2

    March 04, 2019

    "Ursus t. Ussuricus"

    🔋 Features

    • ⚡️ Update Beam to 2.10 (#1674, #1676)
    • Clearer Coder exceptions (#1672)
    • 👉 Use new HadoopFormatIO (#1675)
    • ➕ Add spanner MutationGroup coder (#1704)
    • ⚡️ Optimize CombineFn's (speeds up aggregate-, reduce-, and combine-based operations!) (#1699)
    • 👉 Use list side input on cross product (#1691)
    • 🛠 Fix DistinctBy serialization for Scala Classes (#1710, #1715)
    • ✂ Remove deprecation warning on tfRecordExampleFileWithSchema (#1714)
    • Cleanup around scio context (#1679)
    • 🔖 Version bumps: cassandra-all -> 2.2.14 (#1677), 3.11.4 (#1678); Sparkey -> 3.0.0 (#1690), ES5 -> 5.6.15, ES6 -> 6.6.1 (#1700); tensorflow -> 1.13.1 (#1707); scalatest -> 3.0.6 (#1709); featran-* -> 0.3.0 (#1713)

    🐛 Bug fixes

    • 🛠 Fix Magnolia generated tree annotations removal to ensure Derived coders are serializable (#1673)
  • v0.7.1

    February 08, 2019

    "Taxidea Taxus"

    🔋 Features

    • 🆕 New HashCode-based partitioning method for keyed SCollections (#1654)
    • 🆕 New Coder for java.lang.ArrayList (#1649), and more space-efficient coders for small ADTs like Either and Try (#1652)
    • 🆕 new BinaryIO output (#1663)
    • Simpler, clearer toString method for Coders (#1671)
    • 📦 Custom Assertions for unit testing Coders added to scio-test package (#1642)
    • 🆕 New SideMap and SideSet SideInput types, usable in hashFullOuterJoin, hashIntersectByKey, and hashFilter methods
    • Library version bumps: mysql-connector-java -> 8.0.15 (#1653), mysql-socket-factory -> 1.0.12 (#1627), protobuf-java -> 3.6.1 (#1633), hadoop-client -> 2.7.7 (#1634), jackson-module-scala -> 2.9.8 (#1632), parquet-avro -> 1.10.1 (#1648), kantan.csv -> 0.5.0 (#1647)

    🐛 Bug fixes & Improvements

    • ⚡️ Optimized Bloom filter aggregations in sparse joins (#1644)
    • 📦 Spanner-specific Coders repackaged from scio-core to scio-spanner (#1630)
    • 🚚 Fallback coder always uses Kryo (#1668) and RichCoderRegistry is removed (#1670)
  • v0.7.0

    January 18, 2019

    "Suricata suricatta"

    💥 Breaking changes

    • 👀 See v0.7.0 Migration Guide for detailed instructions
    • 🆕 New Magnolia based Coders derivation replaces ClassTag and Kryo
    • 🆕 New ScioIO replaces TestIO[T] to simplify IO implementation and stubbing in JobTest
    • ⚡️ Update dynamic file destination API #1305
    • ✂ Remove deprecated TensorFlow graph prediction method #1370
    • Object file IO is no longer backwards compatible due to coder changes
    • ♻️ Refactor bigquery client (#1439)

    🔋 Features

    🐛 Bug fixes & Improvements

    • 👉 Make PTransform names unique #1355 #1387
    • Fail for unknown args in ContextAndArgs.typed[T] (#1413)
    • 🛠 Fix verifyNondeterministic exception in coders (#1418)
    • 🛠 Fix BigQueryType on refined types (#1424)
    • 🛠 Fix mergeAccumulators crash (#1428)
    • ✅ Set timestamp attribute in JobTest for PubSubIO (#1417)
    • Rework Coder's implicit not found message (again) (#1469)
    • 🛠 Fix KryoRegistrar scope widenning (#1462)
    • 👉 Make compression options in ExtractOps typed (#1449) (#1457)
    • ➕ Add back BigQuery schema caching, regression of #1439 (#1458)
    • ✅ Register default file systems in Scio test context (fix #1455) (#1463)
    • 0️⃣ Use coherent defaults accross IO (#1478)
    • 🛠 Fix scio-repl to use refactored BigQuery client (#1459)
    • 📜 Typed argument parsing is broken when name contains camelCase. (#1460)
    • Pubusb topic name was not being set for Messages (#1568)
    • 🛠 Fix macro generated class directory (#1558)
    • 🛠 Fix stack overflow when maxByKey is used with explicit ordering (#1560)
    • 🛠 Fix id and timestamp attributes not being passed in saveAsPubsub (#1559)
    • 🛠 Fix flatten type inference changing the coder context bound to an implicit parameter(#1551)
    • 🛠 Fix: use CodeMaterializer in SideOutputCollections (#1548)
    • ⚠ Default to disabled warning on coders (#1588)
    • 🗄 Use alternative to deprecated write method (#1592)
    • 📜 Simplify BigQueryType query method arg type parsing (#1585)
    • ➕ Add rules for TextIO, AvroIO, PubsubIO and BigQueryIO (#1577)
    • #1587: Fix sideoutput potentialy missing coder (#1598)
    • ➕ Add region to DataflowResult (#1479)
    • ✂ Remove unused autovalue dependency (#1575)
  • v0.7.0-beta3

    January 08, 2019

    🐛 Bug fixes & Improvements

    • ⚠ Default to disabled warning on coders (#1588)
    • 🗄 Use alternative to deprecated write method (#1592)
    • 📜 Simplify BigQueryType query method arg type parsing (#1585)
    • ➕ Add rules for TextIO, AvroIO, PubsubIO and BigQueryIO (#1577)
    • #1587: Fix sideoutput potentialy missing coder (#1598)
    • ⚡️ Update beam-runners-direct-java, ... to 2.9.0 (#1580)
    • ⚡️ Update annoy4s to 0.8.0 (#1579)
    • ⚡️ Update zoltar-api, zoltar-tensorflow to 0.5.1 (#1578)
    • ⚡️ Update circe-core, circe-generic, ... to 0.11.0 (#1586)
    • ⚡️ Update guava to 25.1-jre (#1589)
    • ✂ Remove unused autovalue dependency (#1575)

    🔋 Features

    • ➕ Add elasticsearch 6 (#1572)
    • ➕ Add Numeric type support in scio-bigquery (#1599)