Scio v0.8.0 Release Notes

Release Date: 2019-07-16 // 3 months ago
  • 🔋 Features

    • ⚡️ Update beam to 2.13.0 (#1952)
    • ➕ Add partitionToMap function (#1968)
    • 🆕 New async DoFns (#1971)
    • 🔦 Expose Beam's AvroIO.parseGenericRecords (#1973)(#2051)
    • ➕ Add batchByKey function (#2009)
    • ➕ Add elasticsearch7 module (#2004)
    • ➕ Add support for recursive Coders (#2032)(#2048)
    • ➕ Adding filterValues function (#1964)

    🐛 Bug Fixes & Improvements

    • 🛠 Fix equality for mutable bf sparse instances (#1972)
    • ➕ Add bq not implemented tap when write append (#2042)
    • 🛠 Fix schema inference from BigQueryType (#2027)
    • Explicitly name the transform in close() that initializes counters (#2047)
    • Rework ClosedScioContext into ScioRunner (#2036)(#2055)
    • ➕ Add incorrect table aliases hint (#2053)
    • 👍 Allow app/job name override in scio-repl (#2046)
    • 👉 Use BigtableOptions param for BigTable utils (#2003)
    • 👻 InMemorySink will now create an empty collection in the cache so empty SCollections also return an iterable instead of throwing an exception (#2039)
    • ➕ Add WrappedArray coder (#1945)
    • 🛠 Fix sql interpolator tag index (#1961)
    • 📜 Use composite transform for sparse transforms (#1947)
    • 🛠 Fix benchmarks (#1976)(#2005)
    • 👷 Set location when fetching Job in JobOps (#1966)
    • Cleanup materialize and set coder (#2016)
    • ♻️ Refactor applyPerKey (#2011)(#1978)
    • 🛠 Fix BigtableDoFn (#2001)
    • 👌 Support passing Assertions directly to JobTest output (#1993)(#1995)
    • ⚡️ Update docs (#2012)
    • ➕ Add deprecation dates to Apache-Beam.md
    • ✂ Remove deprecation warnings (#2041)(#2045)
    • ✂ Remove uneed implicit context and duplicated logger (#2018)
    • 🚀 Warn only about latest stable release (non prerelease, draft) (#1980)
    • ⚡️ Dependency updates (#1962)(#1960)(#1956)(#1957)(#1948)(#1965)(#1990)(#2023)(#2033)(#1977)(#1998(#2052)

Previous changes from v0.8.0-alpha1

  • Highlights

    🔋 Features

    🐛 Bug Fixes & Improvements

    • ➕ Add defaults for BinaryIO (#1919)
    • ➕ Add BigQuery NUMERIC missing cases (#1827)
    • ➕ Add Spanner syntax and instances package (#1863)
    • ➕ Add bigquery syntax package (#1838)
    • ➕ Add dynamic syntax package (#1920)
    • 🔄 Change default fpProb of sparseIntersectByKey (#1935)
    • Wrap unsafe access of nullable fields and methods in try (#1930)
    • 🛠 Fix FileStorage implicit access modifier (#1909)
    • ✅ Remote sys.exit from code path which is used in tests (#1900)
    • ♻️ Refactor bigquery schema/table cache (#1886)
    • Correctly format dataset id (#1907)
    • ➕ Add header/footer/prefix/suffix to BinaryIO, fix #1785 (#1884)
    • ➕ Add table type to disambiguate spec and reference (#1866)
    • 👍 Replace IllegalStateException with UnsupportedOperationException (#1861)
    • ♻️ Refactor sparse intersect by key (#1881)
    • scio-tensorflow cleanup (#1879)
    • 🎉 Initialize counters from ScioContext (#1873)
    • 🛠 Fix #1866 bigquery it tests (#1878)
    • Add warn combine*/sum* transform usage, fix #1620 (#1856)
    • Create empty Sparkey file on empty collection (#1834)
    • 👉 Use java.io.tmpdir and username as cache dir (#1843)
    • ⚡️ Update WriteDynamicExample, fix #1786 (#1802)
    • 🛠 Fix KryAtomicCoder null encoding (#1841)
    • ➕ Add jmh benchmarks for bloom filters (#1913)
    • 🛠 Fix AvroIO type bounds (#1799)
    • ➕ Add more join tests (#1797)
    • 👻 Avro map string key cast exception (#1832)
    • ➕ Add toString on Coders impl (#1819)
    • ➕ Add a hand written Coder for pairs (#1775)
    • ✂ Remove usage of deprecated api (#1812)
    • Enable higher kinds and partial unification (#1810)
    • 👕 Enable more Xlint options (#1816)
    • Enable -Ywarn-value-discard (#1815)
    • 🛠 Fix schema parsing in BQ TypeProvider
    • 🛠 Fix NPE in coders with nullableCoders=true (#1820)
    • 👉 Make BQ annotations serializable (#1773)
    • 🛠 Fix: use same protoc (#1781)
    • 👉 Use camelCase for typed arguments, fix #1770 (#1780)
    • ⬆️ Bump versions (#1801)
    • 🛠 Fix scalac options for 2.11
    • ⏪ Revert auto-service to 1.0-RC2. Same version as beam (#1837)
    • ⚡️ Dependency updates (#1807)(#1803)(#1928)(#1924)(#1916)(#1914)(#1868)(#1875)(#1836)
    • 🛠 Fix more scalac warns (#1844)
    • 🔌 Quieted down code dumper for IntelliJ Scala Plugin (#1854)
    • 👉 Make ensureSerializable return Either (#1798)
    • ✅ Disable checkVersion in Scio's tests (#1908)
    • ♻️ Refactor jdbc io (#1910)
    • ⚡️ Updated overall documentaion (#1906)(#1941)(#1890)(#1936)(#1817)(#1825)

    💥 Breaking changes

    • ✂ Remove tensorflow methods related to schema inference (#1849)
    • ✂ Remove the usage of Future around ScioContext and Tap's (#1666)
    • ✂ Remove references of Future[Tap[T]] (#1835)