tecton.HiveConfig
Summary
Configuration used to reference a Hive table.
The HiveConfig class is used to create a reference to a Hive Table.
This class is used as an input to a
BatchSource
’s parameter
batch_config
. Declaring this configuration class alone will not register a
Data Source. Instead, declare as a part of BatchSource
that takes this
configuration class instance as a parameter.
Example
from tecton import HiveConfig
import pyspark
def convert_temperature(df: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame:
from pyspark.sql.functions import udf, col
from pyspark.sql.types import DoubleType
# Convert the incoming PySpark DataFrame temperature Celsius to Fahrenheit
udf_convert = udf(lambda x: x * 1.8 + 32.0, DoubleType())
converted_df = df.withColumn("Fahrenheit", udf_convert(col("Temperature"))).drop("Temperature")
return converted_df
# declare a HiveConfig instance, which can be used as a parameter to a BatchSource
batch_config = HiveConfig(
database="global_temperatures", table="us_cities", timestamp_field="timestamp", post_processor=convert_temperature
)
Attributes
data_delay
: This attribute is the same as thedata_delay
parameter of the__init__
method. See below.
Methods
__init__(...)
Instantiates a new HiveConfig.
Parameters
-
table
(str
) – A table registered in Hive MetaStore. -
database
(str
) – A database registered in Hive MetaStore. -
timestamp_field
(Optional
[str
]) – The timestamp column in this data source that should be used by FilteredSource to filter data from this source, before any feature view transformations are applied. Only required if this source is used with FilteredSource. (Default:None
) -
timestamp_format
(Optional
[str
]) – Format of string-encoded timestamp column (e.g. “yyyy-MM-dd’T’hh:mm:ss.SSS’Z’”). If the timestamp string cannot be parsed with this format, Tecton will fallback and attempt to use the default timestamp parser. (Default:None
) -
datetime_partition_columns
(Optional
[List
[DatetimePartitionColumn
])] – List of DatetimePartitionColumn the raw data is partitioned by. (Default:None
) -
post_processor
(Optional
[Callable
]) – Python user defined functionf(DataFrame) -> DataFrame
that takes in raw PySpark data source DataFrame and translates it to the DataFrame to be consumed by the Feature View. (Default:None
) -
data_delay
(timedelta
) – By default, incremental materialization jobs run immediately at the end of the batch schedule period. This parameter configures how long they wait after the end of the period before starting, typically to ensure that all data has landed. For example, if a feature view has a batch_schedule of 1 day and one of the data source inputs hasdata_delay=timedelta(hours=1)
set, then incremental materialization jobs will run at 01:00 UTC. (Default:datetime.timedelta(0)
)
Returns
A HiveConfig class instance.