BigDL v0.5.0 Release Notes
Release Date: 2018-03-30 // about 6 years ago-
Highlights
- 💅 Bring in a Keras-like API(Scala and Python). User can easily run their Keras code (training and inference) on Apache Spark through BigDL. For more details, see this link.
- Support load Tensorflow dynamic models(e.g. LSTM, RNN) in BigDL and support more Tensorflow operations, see this page.
- 👌 Support combining data preprocessing and neural network layers in the same model (to make model deployment easy )
- Speedup various modules in BigDL (BCECriterion, rmsprop, LeakyRelu, etc.)
- ➕ Add DataFrame-based image reader and transformer
🆕 New Features
- Tensor can be converted to OpenCVMat
- Bring in a new Keras-like API for scala and python
- 👌 Support load Tensorflow dynamic models(e.g. LSTM, RNN)
- 👌 Support load more Tensorflow operations(InvertPermutation, ConcatOffset, Exit, NextIteration, Enter, RefEnter, LoopCond, ControlTrigger, TensorArrayV3,TensorArrayGradV3, TensorArrayGatherV3, TensorArrayScatterV3, TensorArrayConcatV3, TensorArraySplitV3, TensorArrayReadV3, TensorArrayWriteV3, TensorArraySizeV3, StackPopV2, StackPop, StackPushV2, StackPush, StackV2, Stack)
- 👍 ResizeBilinear support NCHW
- 👍 ImageFrame support load Hadoop sequence file
- 👍 ImageFrame support gray image
- ➕ Add Kv2Tensor Operation(Scala)
- ➕ Add PGCriterion to compute the negative policy gradient given action distribution, sampled action and reward
- 👌 Support gradual increase learning rate in LearningrateScheduler
- ➕ Add FixExpand and add more options to AspectScale for image preprocessing
- ➕ Add RowTransformer(Scala)
- 👌 Support to add preprocessors to Graph, which allows user combine preprocessing and trainable model into one model
- 👍 Resnet on cifar-10 example support load images from HDFS
- ➕ Add CategoricalColHashBucket operation(Scala)
- 👍 Predictor support Table as output
- ➕ Add BucketizedCol operation(Scala)
- 👌 Support using DenseTensor and SparseTensor together to create Sample
- ➕ Add CrossProduct Layer (Scala)
- 👻 Provide an option to allow user bypass the exception in transformer
- 📜 DenseToSparse layer support disable backward propagation
- ➕ Add CategoricalColVocaList Operation(Scala)
- 👌 Support imageframe in python optimizer
- 👌 Support get executor number and executor cores in python
- ➕ Add IndicatorCol Operation(Scala)
- ➕ Add TensorOp, which is an operation with Tensor[T]-formatted input and output, and provides shortcuts to build Operations for tensor transformation by closures. (Scala)
- 🐳 Provide a docker file to make it easily to setup testing environment of BigDL
- ➕ Add CrossCol Operation(Scala)
- ➕ Add MkString Operation(Scala)
- ➕ Add a prediction service interface for concurrent calls and accept bytes input
- ➕ Add SparseTensor.cast & SparseTensor.applyFun
- ➕ Add DataFrame-based image reader and transformer
- 👌 Support load tensoflow model files saved by tf.saved_model API
- 📜 SparseMiniBatch supporting multiple TensorDataTypes
✨ Enhancement
- 👍 ImageFrame support serialization
- 0️⃣ A default implementation of zeroGradParameter is added to AbstractModule
- 👌 Improve the style of the document website
- Models in different threads share weights in model training
- Speed up leaky relu
- Speed up Rmsprop
- Speed up BCECriterion
- 👌 Support Calling Java Function in Python Executor and ModelBroadcast in Python
- ➕ Add detail instructions to run-on-ec2
- ⚡️ Optimize padding mechanism
- 🛠 Fix maven compiling warnings
- Check duplicate layers in the container
- 🚀 Refine the document which introduce how to automatically Deploy BigDL on Dataproc cluster
- Refactor adding extra jars/python packages for python user. Now only need to set env variable BIGDL_JARS & BIGDL_PACKAGES
- Implement appendColumn and avoid the error caused by API mismatch between different Spark version
- ➕ Add python inception training on ImageNet example
- ⚡️ Update "can't find locality partition for partition ..." to warning message
API change
- 📦 Move DataFrame-based API to dlframe package
- 🚚 Refine the Container hierarchy. The add method(used in Sequential, Concat…) is moved to a subclass DynamicContainer
- Refine the serialization code hierarchy
- Dynamic Graph has been an internal class which is only used to run tensorflow models
- Operation is not allowed to use outside Graph
- The getParamter method as final and private[bigdl], which should be only used in model training
- ✂ remove the updateParameter method, which is only used in internal test
- Some Tensorflow related operations are marked as internal, which should be only used when running Tensorflow models
🐛 Bug Fix
- 🛠 Fix Sparse sample batch bug. It should add another dimension instead of concat the original tensor
- 🛠 Fix some activation or layers don’t work in TimeDistributed and RnnCell
- 🛠 Fix a bug in SparseTensor resize method
- 🛠 Fix a bug when convert SparseTensor to DenseTensor
- 🛠 Fix a bug in SpatialFullConvolution
- 🛠 Fix a bug in Cosine equal method
- 🛠 Fix optimization state mess up when call optimizer.optimize() multiple times
- 🛠 Fix a bug in Recurrent forward after invoking reset
- 🛠 Fix a bug in inplace leakyrelu
- 🛠 Fix a bug when save/load bi-rnn layers
- 🛠 Fix getParameters() in submodule will create new storage when parameters has been shared by parent module
- 🛠 Fix some incompatible syntax between python 2.7 and 3.6
- 🛠 Fix save/load graph will loss stop gradient information
- 🛠 Fix a bug in SReLU
- 🛠 Fix a bug in DLModel
- 🛠 Fix sparse tensor dot product bug
- 🛠 Fix Maxout ser issue
- 🛠 Fix some serialization issue in some customized faster rcnn model
- 🛠 Fix and refine some example document instructions
- Fix a bug in export_tf_checkpoint.py script
- 🛠 Fix a bug in set up python package.
- 🛠 Fix picklers initialization issues
- 🛠 Fix some race condition issue in Spark 1.6 when broadcasting model
- 🛠 Fix Model.load in python return type is wrong
- 🛠 Fix a bug when use pyspark-with-bigdl.sh to run jobs on Yarn
- 🛠 Fix empty tensor call size and stride not throw null exception