Secrets

Public Preview

This feature is currently in Public Preview.

This feature has the following limitations:

• Available for Rift. Coming to Spark in a future release.

If you have questions or want to share feedback, please file a feature request.

In order to connect your Tecton account to certain Data Sources and infrastructure, you may need to provide Tecton with authentication credentials. Tecton Secrets is a secure mechanism for configuring those credentials, and defining their use in feature pipelines.

This guide describes how to configure and manage Secrets in the Tecton Platform.

Secure secrets management

Tecton Secrets is designed from the ground up to enable best practices for handling sensitive credentials in your feature pipelines.

• Encryption at Rest and in Transit: Tecton Secrets are encrypted both at rest and during transmission so your secrets are safe.
• Secrets-specific storage: Tecton Secrets leverages Cloud-provider secret management infrastructure with strictly limited access.
• Access Controls: Management of and access to Secrets is governed by access controls. Administrators have fine-grained controls to determine when secrets can be used for development or production.

Concepts

At a high level, you'll create a secret for a data source, grant a workspace access to that secret, and then reference that secret inside your Data Source configuration code.

Secrets: A secret is an individual key-value pair of (secret key, secret value).

Secret scopes: Scopes represent a group of related secrets. You can tailor access at the scope level, so it's a good idea to keep secrets that are used in similar feature views or by similar groups of users together.

Assigning access to scopes: Assign roles to your Users, Service Accounts, Principal Groups to manage or access secrets in that scope during development. Assign roles to Workspaces to enable secret read access during materialization.

Using Secrets

Creating and managing secrets and scopes

To create, update, and delete secrets, use the secrets command family or Secrets Service HTTP APIs.

For example, the following commands create a scope, grant my user management of the scope, and insert the first secret.

% tecton secrets create-scope -s my-demo-scope
% tecton secrets put -s my-demo-scope -k my-demo-secret -f secretvalue.txt

note

This example uses the secretvalue.txt file to pass in the secret value, so that we avoid having the secret live in our shell command history. See the put command reference for alternative options.

Accessing secrets in Data Source definitions

Once a secret has been configured, it can be referenced in a Data Source definition.

To leverage an existing secret in your local development environment, the authenticated User or Service Account needs to have the secret_scope_reader role.

note

Any place that the Tecton SDK expects a Secret object, you can pass in a String instead when testing a definition locally in a Python or Notebook environment. Using a String can be convenient for quickly prototyping before going through the full Secret creation flow. As a best practice, Tecton recommends avoiding saving plain-text secrets.

To apply the same definition to a live workspace, that workspace must have the secret_scope_reader role.

Here are some examples for accessing and using secrets in your feature definitions:

• Use secrets in a Snowflake BatchConfig to set the user and password for connecting to the Snowflake database (example).
• Use secrets in a pandas_batch_config decorator and within the data source function to load your data (example).

Manage access to secrets

To grant and revoke roles on secret scopes, use the access-control command family or Secrets Service HTTP APIs.

There are 3 levels of access to a secret scope: the manager, writer, and reader roles. The table below details what permissions each role has on a secret scope.

Everyone in a Tecton deployment can create a secret scope. The creator of that secret scope is automatically granted the Manager role on the secret scope. This grant can always be removed by other scope Managers or Tecton Admins as needed.

Scope-Level Permission	Manager role	Writer role	Reader role	Everyone
Create a secret scope in a cluster (instance-level action)				x
View and edit role assignments on a secret scope	x
Delete a secret scope	x
Create, update, and delete secret values in a secret scope	x	x
List secret keys in a secret scope	x	x
Read secret values in a secret scope	x	x	x

End-to-end example

In this example, we'll walk through the end-to-end workflow of configuring secrets for a new Snowflake Data Source for development and production.

Create the scope, secrets, and grant access

First, create the scope and secrets in Tecton. In this example we'll register a Snowflake user name and password as secrets.

% tecton secrets create-scope -s snowflake-scope
% tecton secrets put -s snowflake-scope -k snowflake-password -f snowflake_password.txt
% tecton secrets put -s snowflake-scope -k snowflake-user -f snowflake_user.txt

Substitute the new secret in the notebook definition

To test that we created the secrets correctly, we'll now reference them in our Notebook Data Source definition.

snowflake_ds = BatchSource(
    name="click_stream_snowflake_ds_dev",
    batch_config=SnowflakeConfig(
        url="https://yourcluster.snowflakecomputing.com",
        database="DS_DEV",
        schema="CLICKSTREAM_DEV",
        table="CLICKSTREAM_EVENTS",
        warehouse="DEV_WH",
        user=Secret(scope="snowflake-scope", key="snowflake-user"),
        password=Secret(scope="snowflake-scope", key="snowflake-password"),
    ),
)

snowflake_ds.validate()

# Read sample data to test the connection
snowflake_ds.get_dataframe().to_pandas().head(10)

Grant role to workspace and apply

Finally, we'll grant our workspace read access to the scope so that materialization jobs can leverage the secret. After updating the feature repo, we can apply to the workspace.

% tecton access-control assign-role -w prod -r secret_scope_reader -c snowflake-scope
% tecton apply