Scio v0.7.0 Release Notes
Release Date: 2019-01-18 // over 5 years ago-
"Suricata suricatta"
💥 Breaking changes
- 👀 See v0.7.0 Migration Guide for detailed instructions
- 🆕 New Magnolia based Coders derivation replaces
ClassTagand Kryo - 🆕 New ScioIO replaces
TestIO[T]to simplify IO implementation and stubbing inJobTest - ⚡️ Update dynamic file destination API #1305
- ✂ Remove deprecated TensorFlow graph prediction method #1370
- Object file IO is no longer backwards compatible due to coder changes
- 🔨 Refactor bigquery client (#1439)
🔋 Features
- Macro based coder derivation for magnitudes faster (de)serialization (#1454)(#1394)(#1440)(#1434)(#1427)(#1412)(#1401)(#1429)(#1554)(#1494)(#1438)(#1605)(#1612)
- Redesigned unified
ScioIO[T]for all IO modules - ➕ Add
SCollection#{readAll,readAllBytes}(splittable DoFn support) #796 #1363 - 📜 Sparse lookups (#1398)(#1354)(#1393)
- ➕ Add sparse left and right outer joins #1386
- Check and warn chained joins #1362
- 👌 Support Parquet compression #1189 #1318
- Port Parquet IO to Parquet 1.10 #1340 #1345
- 🔧 Configurable fetch and batch size for JDBC IO #1314
- ➕ Add PubSubIO batch size write params (#1433)
- 👌 Improve coders messages
- ➕ Add BigQueryType typesafe args (#1476)(#1431)
- 👌 support PubsubMessage in PubsubIO (#1395)
- ➕ Add subscription function to PubSubAdmin(#1483)
- Register sys.props (#1404)(#1406)
- ✅ Make typed and default args parsing logic more test friendly (#1421)
- ➕ Add Google Spanner package (#1491)
- ➕ Add BigQuery TimePartitioning support, fix #1419 (#1466)
- ➕ Add Numeric type support in scio-bigquery (#1599)
- ➕ Add scalafix rules (#1435)(#1464)(#1474)(#1468)(#1470)
- 🔦 Expose transform function (#1492)(#1487)
- 👍 Allow creating
DataflowResultfrom dfJob(#1481) - ✂ Remove Future.failed in IOs (#1482)
- ➕ Add better error messages when missing sys.props (#1488)(#1461)
- Avoid second sql legacy check when using extractTables query op (#1508)
- ➕ Add support for more WriteDisposition's in bigquery writeRows (#1511)
- ➕ Add call site transform name in union all (#1499)
- ⚡️ Update apache beam to 2.9.0 (#1580)
- ⚡️ Updated other dependencies (#1589)(#1586)(#1578)(#1579)(#1489)(#1544)(#1520)(#1539)(#1534)(#1533)(#1512)(#1517)(#1531)(#1521)(#1532)(#1538)(#1540)(#1526)(#1529)(#1518)(#1519)(#1536)(#1513)(#1530)(#1535)(#1527)(#1525)(#1524)(#1523)(#1514)(#1515)(#1516)(#1537)(#1509)(#1510)(#1565)(#1432)(#1614)
- ➕ Add elasticsearch 6 (#1572)
- 👌 Improve AvroType.toSchema annotation error if a case class is not provided (#1609)
- 🆕 New scio website (#1610)
🐛 Bug fixes & Improvements
- 👉 Make
PTransformnames unique #1355 #1387 - Fail for unknown args in
ContextAndArgs.typed[T](#1413) - 🛠 Fix verifyNondeterministic exception in coders (#1418)
- 🛠 Fix BigQueryType on refined types (#1424)
- 🛠 Fix mergeAccumulators crash (#1428)
- ✅ Set timestamp attribute in JobTest for PubSubIO (#1417)
- Rework Coder's implicit not found message (again) (#1469)
- 🛠 Fix KryoRegistrar scope widenning (#1462)
- 👉 Make compression options in ExtractOps typed (#1449) (#1457)
- ➕ Add back BigQuery schema caching, regression of #1439 (#1458)
- ✅ Register default file systems in Scio test context (fix #1455) (#1463)
- 0️⃣ Use coherent defaults accross IO (#1478)
- 🛠 Fix scio-repl to use refactored BigQuery client (#1459)
- 📜 Typed argument parsing is broken when name contains camelCase. (#1460)
- Pubusb topic name was not being set for Messages (#1568)
- 🛠 Fix macro generated class directory (#1558)
- 🛠 Fix stack overflow when maxByKey is used with explicit ordering (#1560)
- 🛠 Fix id and timestamp attributes not being passed in saveAsPubsub (#1559)
- 🛠 Fix flatten type inference changing the coder context bound to an implicit parameter(#1551)
- 🛠 Fix: use CodeMaterializer in SideOutputCollections (#1548)
- ⚠ Default to disabled warning on coders (#1588)
- 🗄 Use alternative to deprecated write method (#1592)
- 📜 Simplify BigQueryType query method arg type parsing (#1585)
- ➕ Add rules for TextIO, AvroIO, PubsubIO and BigQueryIO (#1577)
- #1587: Fix sideoutput potentialy missing coder (#1598)
- ➕ Add region to DataflowResult (#1479)
- ✂ Remove unused autovalue dependency (#1575)