All Versions
26
Latest Version
Avg Release Cycle
27 days
Latest Release
1225 days ago
Changelog History
Page 3
Changelog History
Page 3
-
v0.7.2 Changes
March 04, 2019"Ursus t. Ussuricus"
๐ Features
- โก๏ธ Update Beam to 2.10 (#1674, #1676)
- Clearer
Coder
exceptions (#1672) - ๐ Use new HadoopFormatIO (#1675)
- โ Add spanner MutationGroup coder (#1704)
- โก๏ธ Optimize CombineFn's (speeds up
aggregate
-,reduce
-, andcombine
-based operations!) (#1699) - ๐ Use list side input on cross product (#1691)
- ๐ Fix DistinctBy serialization for Scala Classes (#1710, #1715)
- โ Remove deprecation warning on tfRecordExampleFileWithSchema (#1714)
- Cleanup around scio context (#1679)
- ๐ Version bumps: cassandra-all -> 2.2.14 (#1677), 3.11.4 (#1678); Sparkey -> 3.0.0 (#1690), ES5 -> 5.6.15, ES6 -> 6.6.1 (#1700); tensorflow -> 1.13.1 (#1707); scalatest -> 3.0.6 (#1709); featran-* -> 0.3.0 (#1713)
๐ Bug fixes
- ๐ Fix Magnolia generated tree annotations removal to ensure Derived coders are serializable (#1673)
-
v0.7.1 Changes
February 08, 2019"Taxidea Taxus"
๐ Features
- ๐ New HashCode-based partitioning method for keyed SCollections (#1654)
- ๐ New Coder for
java.lang.ArrayList
(#1649), and more space-efficient coders for small ADTs likeEither
andTry
(#1652) - ๐ new
BinaryIO
output (#1663) - Simpler, clearer
toString
method for Coders (#1671) - ๐ฆ Custom
Assertion
s for unit testing Coders added toscio-test
package (#1642) - ๐ New
SideMap
andSideSet
SideInput types, usable inhashFullOuterJoin
,hashIntersectByKey
, andhashFilter
methods - Library version bumps: mysql-connector-java -> 8.0.15 (#1653), mysql-socket-factory -> 1.0.12 (#1627), protobuf-java -> 3.6.1 (#1633), hadoop-client -> 2.7.7 (#1634), jackson-module-scala -> 2.9.8 (#1632), parquet-avro -> 1.10.1 (#1648), kantan.csv -> 0.5.0 (#1647)
๐ Bug fixes & Improvements
-
v0.7.0 Changes
January 18, 2019"Suricata suricatta"
๐ฅ Breaking changes
- ๐ See v0.7.0 Migration Guide for detailed instructions
- ๐ New Magnolia based Coders derivation replaces
ClassTag
and Kryo - ๐ New ScioIO replaces
TestIO[T]
to simplify IO implementation and stubbing inJobTest
- โก๏ธ Update dynamic file destination API #1305
- โ Remove deprecated TensorFlow graph prediction method #1370
- Object file IO is no longer backwards compatible due to coder changes
- ๐จ Refactor bigquery client (#1439)
๐ Features
- Macro based coder derivation for magnitudes faster (de)serialization (#1454)(#1394)(#1440)(#1434)(#1427)(#1412)(#1401)(#1429)(#1554)(#1494)(#1438)(#1605)(#1612)
- Redesigned unified
ScioIO[T]
for all IO modules - โ Add
SCollection#{readAll,readAllBytes}
(splittable DoFn support) #796 #1363 - ๐ Sparse lookups (#1398)(#1354)(#1393)
- โ Add sparse left and right outer joins #1386
- Check and warn chained joins #1362
- ๐ Support Parquet compression #1189 #1318
- Port Parquet IO to Parquet 1.10 #1340 #1345
- ๐ง Configurable fetch and batch size for JDBC IO #1314
- โ Add PubSubIO batch size write params (#1433)
- ๐ Improve coders messages
- โ Add BigQueryType typesafe args (#1476)(#1431)
- ๐ support PubsubMessage in PubsubIO (#1395)
- โ Add subscription function to PubSubAdmin(#1483)
- Register sys.props (#1404)(#1406)
- โ Make typed and default args parsing logic more test friendly (#1421)
- โ Add Google Spanner package (#1491)
- โ Add BigQuery TimePartitioning support, fix #1419 (#1466)
- โ Add Numeric type support in scio-bigquery (#1599)
- โ Add scalafix rules (#1435)(#1464)(#1474)(#1468)(#1470)
- ๐ฆ Expose transform function (#1492)(#1487)
- ๐ Allow creating
DataflowResult
from dfJob
(#1481) - โ Remove Future.failed in IOs (#1482)
- โ Add better error messages when missing sys.props (#1488)(#1461)
- Avoid second sql legacy check when using extractTables query op (#1508)
- โ Add support for more WriteDisposition's in bigquery writeRows (#1511)
- โ Add call site transform name in union all (#1499)
- โก๏ธ Update apache beam to 2.9.0 (#1580)
- โก๏ธ Updated other dependencies (#1589)(#1586)(#1578)(#1579)(#1489)(#1544)(#1520)(#1539)(#1534)(#1533)(#1512)(#1517)(#1531)(#1521)(#1532)(#1538)(#1540)(#1526)(#1529)(#1518)(#1519)(#1536)(#1513)(#1530)(#1535)(#1527)(#1525)(#1524)(#1523)(#1514)(#1515)(#1516)(#1537)(#1509)(#1510)(#1565)(#1432)(#1614)
- โ Add elasticsearch 6 (#1572)
- ๐ Improve AvroType.toSchema annotation error if a case class is not provided (#1609)
- ๐ New scio website (#1610)
๐ Bug fixes & Improvements
- ๐ Make
PTransform
names unique #1355 #1387 - Fail for unknown args in
ContextAndArgs.typed[T]
(#1413) - ๐ Fix verifyNondeterministic exception in coders (#1418)
- ๐ Fix BigQueryType on refined types (#1424)
- ๐ Fix mergeAccumulators crash (#1428)
- โ Set timestamp attribute in JobTest for PubSubIO (#1417)
- Rework Coder's implicit not found message (again) (#1469)
- ๐ Fix KryoRegistrar scope widenning (#1462)
- ๐ Make compression options in ExtractOps typed (#1449) (#1457)
- โ Add back BigQuery schema caching, regression of #1439 (#1458)
- โ Register default file systems in Scio test context (fix #1455) (#1463)
- 0๏ธโฃ Use coherent defaults accross IO (#1478)
- ๐ Fix scio-repl to use refactored BigQuery client (#1459)
- ๐ Typed argument parsing is broken when name contains camelCase. (#1460)
- Pubusb topic name was not being set for Messages (#1568)
- ๐ Fix macro generated class directory (#1558)
- ๐ Fix stack overflow when maxByKey is used with explicit ordering (#1560)
- ๐ Fix id and timestamp attributes not being passed in saveAsPubsub (#1559)
- ๐ Fix flatten type inference changing the coder context bound to an implicit parameter(#1551)
- ๐ Fix: use CodeMaterializer in SideOutputCollections (#1548)
- โ Default to disabled warning on coders (#1588)
- ๐ Use alternative to deprecated write method (#1592)
- ๐ Simplify BigQueryType query method arg type parsing (#1585)
- โ Add rules for TextIO, AvroIO, PubsubIO and BigQueryIO (#1577)
- #1587: Fix sideoutput potentialy missing coder (#1598)
- โ Add region to DataflowResult (#1479)
- โ Remove unused autovalue dependency (#1575)
-
v0.7.0-beta3 Changes
January 08, 2019๐ Bug fixes & Improvements
- โ Default to disabled warning on coders (#1588)
- ๐ Use alternative to deprecated write method (#1592)
- ๐ Simplify BigQueryType query method arg type parsing (#1585)
- โ Add rules for TextIO, AvroIO, PubsubIO and BigQueryIO (#1577)
- #1587: Fix sideoutput potentialy missing coder (#1598)
- โก๏ธ Update beam-runners-direct-java, ... to 2.9.0 (#1580)
- โก๏ธ Update annoy4s to 0.8.0 (#1579)
- โก๏ธ Update zoltar-api, zoltar-tensorflow to 0.5.1 (#1578)
- โก๏ธ Update circe-core, circe-generic, ... to 0.11.0 (#1586)
- โก๏ธ Update guava to 25.1-jre (#1589)
- โ Remove unused autovalue dependency (#1575)
๐ Features
-
v0.7.0-beta2 Changes
December 06, 2018๐ Bug Fixes
- Pubusb topic name was not being set for Messages (#1568)
- ๐ Fix macro generated class directory (#1558)
- ๐ Fix stack overflow when maxByKey is used with explicit ordering (#1560)
- ๐ Fix id and timestamp attributes not being passed in saveAsPubsub (#1559)
- ๐ Fix flatten type inference changing the coder context bound to an implicit parameter(#1551)
- ๐ Fix: use CodeMaterializer in SideOutputCollections (#1548)
๐ Features
- ๐ Improve the implicitNotFound message on Coder (#1554)
- โ Add coursier for fast dep resolution (#1546)
- Avoid second sql legacy check when using extractTables query op (#1508)
- โ Add support for more WriteDisposition's in bigquery writeRows (#1511)
- โ Add call site transform name in union all (#1499)
- Avoid using kryo coder for TF Schema and Feature (#1494)
- โก๏ธ Updated dependencies (#1544) (#1520) (#1539) (#1534) (#1533(#1512) (#1517) (#1531) (#1521) (#1532)(#1538) (#1540) (#1526) (#1529) (#1518) (#1519) (#1536) (#1513) (#1530) (#1535)(#1527) (#1525) (#1524)(#1523) (#1514) (#1515) (#1516) (#1537) (#1509) (#1510) (#1565)
-
v0.7.0-beta1 Changes
December 06, 2018๐ Bug fixes
- Rework Coder's implicit not found message (again) (#1469)
- ๐ Fix KryoRegistrar scope widenning (#1462)
- ๐ Make compression options in ExtractOps typed (#1449) (#1457)
- โ Add back BigQuery schema caching, regression of #1439 (#1458)
- โ Register default file systems in Scio test context (fix #1455) (#1463)
- 0๏ธโฃ Use coherent defaults accross IO (#1478)
- ๐ Fix scio-repl to use refactored BigQuery client (#1459)
- ๐ Typed argument parsing is broken when name contains camelCase. (#1460)
๐ Features
- โ Add Google Spanner package (#1491)
- โ Add BigQuery TimePartitioning support, fix #1419 (#1466)
- โ Add subscription function to PubSubAdmin(#1483)
- โฌ๏ธ Bump Beam to 2.8.0 (#1493)
- โก๏ธ Update dependencies (#1489)
- โ Add scalafix rules (#1435)(#1464)(#1474)(#1468)(#1470)
- โ Add BigQueryType typesafe args (#1476)
- โ Add region to DataflowResult (#1479)
- ๐ฆ Expose transform function (#1492)(#1487)
- ๐ Allow creating
DataflowResult
from dfJob
(#1481) - โ Remove Future.failed in IOs (#1482)
- โ Add better error messages when missing sys.props (#1488)(#1461)