Deploying Features in a Repository to a Workspace

Your Tecton Repository contains Tecton Object Definitions which define feature pipelines and other dataflows.

When you make changes to your Repository locally and apply them to a selected Target Workspace (using the tecton apply CLI command), Tecton does the following:

• Tecton identifies the diff between the Target Workspace's current state vs. the newly-applied state and validates any new or updated Object Definitions
• Tecton updates the Workspace Configuration accordingly (e.g. by setting up required pipelines, spinning up online or offline table(s), modifying Feature View definitions, connecting to new Data Sources, etc.).
• The changes are applied to the current workspace that is selected in your Tecton CLI. (To see the selected Workspace, run tecton workspace show. To change the selected Workspace, run tecton workspace select your_workspace).
• Changes that have been applied become available on your Tecton instance; for example, feature pipelines run against applied features and write the results to the online and offline stores.

Making a change to a Tecton Repository

As an example, add the file my_entity.py, which contains an Entity object, to a local Tecton Repository:

# my_repo/my_entity.py
from tecton import Entity

user = Entity(name="user", join_keys=["user_id"], description="My first entity!")

It's important to declare Tecton Objects as global variables in your Python module. When plan or apply commands are run, the Tecton CLI references all Python objects instantiated in the global scope to identify objects in the Tecton Repository.

Previewing changes using tecton plan

To preview changes made to your local Tecton Repository, run tecton plan. These are the changes that will be applied if tecton apply is run (the next step).

$ tecton plan
Using workspace "my_workspace" on cluster https://my_app.tecton.ai
✅ Imported 1 Python module from the feature repository
✅ Collecting local feature declarations
✅ Performing server-side validation of feature declarations
 ↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓

  + Create Entity
    name:            user
    description:     My first entity!

 ↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑

Pushing changes using tecton apply

Running tecton apply will generate the same output as tecton plan, along with a final prompt to apply the changes.

$ tecton apply
Using workspace "my_workspace" on cluster https://my_app.tecton.ai
✅ Imported 1 Python module from the feature repository
✅ Collecting local feature declarations
✅ Performing server-side validation of feature declarations
 ↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓

  + Create Entity
    name:            user
    description:     My first entity!

 ↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Are you sure you want to apply this plan? [y/N]> y
🎉 all done!

If you see the message "all done!", as shown above, then your changes have been pushed to to the Tecton Repository.

note

If your Repository contains multiple changes, tecton apply will apply all changes at once.

Running tecton plan before tecton apply is not required.

Types of Repository changes

There are 5 types of changes that can be applied to a Repository:

• + Create: A new object is being created for the first time.
• - Delete: A previously created object is being deleted.
• ~ Recreate: A change to an existing Tecton Object that requires it to be recreated in the remote Tecton Repository. This often occurs when transformations are updated or dependencies change between Objects. For example, changing a Data Source definition may require any Feature Views that depend on it to be recreated and re-materialized. This is also known as a destructive update.
• ~ Update: An update to an existing Tecton Object's properties which doesn't require it to be recreated (e.g. changing the description of a Feature View). This is also known as a non-destructive update.
• ~ Upgrade: No-op updates of Objects to meet the latest Tecton API version. These are sometimes observed after upgrading the Tecton SDK using pip3 and should be considered safe.

Examples

Below are examples of create, delete, and update changes:

$ tecton apply
Using workspace "prod"
✅ Imported 3 Python modules from the feature repository
✅ Collecting local feature declarations
✅ Performing server-side validation of feature declarations
↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓
  - Delete Entity
    name:            my_entity
    owner:           alice

  + Create Entity
    name:            my_new_entity
    owner:           alice

  ~ Update FeatureView
    name:            my_feature_view
    owner:           alice
    description:  -> Description of this FeatureView!
↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Are you sure you want to apply this plan? [y/N]> y
🎉 all done!

Other actions related to tecton apply

Viewing the apply history for a Workspace

The tecton log command will display a list of previously applied ids for the remote Tecton Repository in your current Workspace.

$ tecton log
Using workspace "prod"
Apply ID:	006ad43e0000000000000107
Author:	drake
Date:	2020-05-20 23:19:41.829000

Apply ID:	83a205340000000000000105
Author:	rihanna
Date:	2020-05-20 18:00:01.858000

Apply ID:	56e8a66a00000000000000fd
Author:	jayz
Date:	2020-05-19 15:13:35.083000

Apply ID:	4bfe16ea00000000000000f4
Author:	alicakeys
Date:	2020-05-18 18:45:21.232000

Restoring a previous apply

Tecton stores a snapshot of your Workspace's local Tecton Repository each time tecton apply is run. The tecton restore command makes it possible to overwrite your local Tecton Repository with a previous applied version.

To restore the most recently applied version, run tecton restore without an apply ID:

$ tecton restore
Using workspace "prod"
This operation may remove or modify the following files:
/Users/drake/Tecton/my-git-repo/feature_repo/entities.py
/Users/drake/Tecton/my-git-repo/feature_repo/data_sources.py
/Users/drake/Tecton/my-git-repo/feature_repo/feature_views.py
Ok? [y/N]>y

To restore previous version of your local Tecton Repository, first run tecton log to determine which apply ID to restore, then run tecton restore <apply ID>.

$ tecton log
Using workspace "prod"
Apply ID:	006ad43e0000000000000107
Author:	jayz
Date:	2020-05-20 23:19:41.829000

Apply ID:	**83a205340000000000000105**
Author:	rihanna
Date:	2020-05-20 18:00:01.858000

Apply ID:	4bfe16ea00000000000000f4
Author:	alicakeys
Date:	2020-05-18 18:45:21.232000

$ tecton restore **83a205340000000000000105**
tecton restore
Using workspace "prod"
This operation may remove or modify the following files:
/Users/drake/Tecton/my-git-repo/feature_repo/entities.py
/Users/drake/Tecton/my-git-repo/feature_repo/data_sources.py
/Users/drake/Tecton/my-git-repo/feature_repo/feature_views.py
Ok? [y/N]>y

Skipping files using .tectonignore

Tecton supports a .tectonignore file that can specify files or path expressions to ignore when running plan or apply. It's similar to Git's .gitignore configuration. .tectonignore should be declared in the feature repo root directory.

When running tecton plan or tecton apply, Tecton's CLI processes all objects in .py files within a folder containing a .tecton file (created by running tecton init). For example, consider the following repo with objects declared in transactions_batch.py, entities.py, fraud_detection.py, and fraudulent_transactions_count.py.

├── data_sources
│   └── transactions_batch.py
├── entities.py
├── feature_services
│   └── fraud_detection.py
└── features
    └── fraudulent_transactions_count.py

Suppose everything under feature_services/ needed to be temporarily ignored. Adding .tectonignore to the repo root with the following glob expression will ignore the file fraud_detection.py altogether.

# Ignore everything under feature_services/
feature_services/*.py

# Alternatively, include nested directories under feature_services/
feature_services/**/*.py

# Alternatively, ignore a specific file
feature_services/fraud_detection.py

When running tecton plan or tecton apply, Tecton's CLI would see the following file structure:

├── .tectonignore
├── data_sources
│   └── transactions_batch.py
├── entities.py
└── features
    └── fraudulent_transactions_count.py

Since fraud_detection.py is now ignored, Tecton's CLI will flag that any objects that were defined in the file and previously applied would be deleted upon running tecton apply.

Cloning an existing Workspace

Running tecton apply within a workspace simply applies the local feature repo to the Workspace. Suppose you would like to iterate on a live Workspace in an experimental development Workspace using a new git branch.

1. Select the existing Workspace using tecton workspace select:

$ tecton workspace select existing_workspace
Switched to workspace "existing_workspace".

2. (Optional) Restore the existing Workspace to your local repo

$ tecton restore

3. Create a new workspace

$ tecton workspace create my_workspace
Created workspace "my_workspace".
Switched to workspace "my_workspace".
You have created a new, empty workspace. Workspaces let
you create and manage an isolated feature repository.
Running "tecton plan" will compare your local repository
against the remote repository, which is initially empty.

4. Apply the current state to your new workspace.

   The new workspace my_workspace is empty. Run tecton apply so that the new workspace reflects the current prod state, which has been captured in your local Git branch.

   $ tecton apply
   ✅ Imported 11 Python modules from the feature repository
   ✅ Collecting local feature declarations
   ↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓
   ...
   ↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
   Are you sure you want to apply this plan? [Y/n] Y
   All done!

5. Create a new Git branch

   To develop using the existing Workspace state as a base, create a new Git branch from main in which to store changes made in your workspace:

   # Create a new git branch
   git checkout -b "my_workspace_branch"
   Switched to new branch "my_workspace_branch"

You now have a development environment with a workspace and a Git branch that reflect the current state of existing_workspace.