tecton.SparkStreamConfig
Summary​
Configuration used to define a stream source using a Data Source Function.
The SparkStreamConfig
class is used to configure a stream source using a user
defined Data Source Function.
This class is used as an input to a
StreamSource
’s parameter
stream_config
. Declaring this configuration class alone will not register a
Data Source. Instead, declare as a part of StreamSource
that takes this
configuration class instance as a parameter.
warning
Do not instantiate this class directly. Use
tecton.spark_stream_config()
instead.
Methods​
__init__(...)​
Instantiates a new SparkBatchConfig.
Parameters​
data_source_function
(Callable
[[SparkSession
],DataFrame
]) – User-defined Data Source Function that takes in aSparkSession
and returns a streamingDataFrame
.
Returns​
A SparkStreamConfig class instance.