Limits and Safety Guardrails

Tecton's feature materialization comes with several preconfigured limits and safety guardrails.

These limits reduce the risk that an accidental deployment of features results in massive unexpected infrastructure costs or in unexpected pressure on the infrastructure components that Tecton is integrated with (e.g., the online store and raw data sources).

Please file a support ticket if you need to modify these limits for your deployment.

Limits

• Parallel Spark Materialization Job Instance Limit: Tecton schedules at most 150 parallel Spark job instances for materialization jobs. Please note that 1 Spark job can use more than 1 instance. The number of instances per job is defined by the Spark cluster configuration. If a materialization job is waiting in line, its status will be Scheduled. To increase this limit, contact Tecton Support.

• Minimum Spark Streaming Materialization Job Instance Capacity: 15% of the “Parallel Spark Materialization Job Instance Limit” is reserved for Streaming materialization jobs. E.g. if the limit is configured to be 150, and 127 instances are used by materialization jobs, the remaining 23 are reserved exclusively for Streaming jobs. This limit ensures that there is always enough available capacity for streaming jobs, which typically have higher freshness needs than batch jobs.

• Parallel Spark Jobs per Feature View limit: Tecton schedules at most 16 parallel materialization jobs for a single Feature View.

• Redis write throughput per Feaure View Limit: Tecton throttles the online writes to a Redis cluster to a max of 3000 writes per second. This limit ensures that the Redis cluster can retain enough capacity for high scale and low latency serving.

• Online Read Throughput: Tecton's online serving capacity can in its default configuration handle roughly ~200 QPS. Please be aware that you need to scale up Tecton's online serving capacity to serve high-volume work loads. When provisioned properly, Tecton can easily serve traffic in excess of 1,000,000 QPS.

• Request Source Maximum Payload Size: When providing request data for On-Demand Feature Views, at most 4mb of data can be included in a single API request.

Limit Principles

The limits listed above are set with the following principles in mind:

• Online Serving before Online Writing: It's paramount to never exhaust the online serving infrastructure with too many parallel writes.
• Streaming Jobs before Batch Jobs: Streaming materialization jobs are generally more important than batch materialization jobs because streaming features have typically higher freshness needs
• Forward Fills before Backward Fills: When it comes to feature materialization, it's more important to execute a feature's ongoing forward fills than a feature's backfills.
• Recent History before Ancient History: When executing backfills, it's more important to materialize the more recent data before older data.