Connecting CI/CD

Overview

We recommend creating a new CI/CD pipeline to automatically run tecton plan and tecton apply upon changes to your Tecton feature repo.

Example: Tecton Plan

For example, to run a tecton plan and tecton apply in GitHub actions, you can add the following file to the .github/workflows directory underneath your Github repository, assuming that your features repository starts at the base of your Github repository. Note that this example uses Github environments:

name: Tecton Feature Repo CI/CD

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      planid: ${{ steps.planid.outputs.planid }}
    env:
      TECTON_API_KEY: ${{ secrets.TECTON_API_KEY }}
      API_SERVICE: https://app.tecton.ai/api
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: "3.9"
          cache: "pip" # caching pip dependencies
      - run: pip install -r requirements.txt

      - name: Run tecton plan
        run: tecton plan | tee output.log

      - name: Extract plan output
        id: planid
        run: |
          PLANID=$(cat output.log | perl -wne  '/Generated plan ID is ([0-9a-f]+)/i and print $1')
          echo "planid=$PLANID" >> "$GITHUB_OUTPUT"

  apply:
    runs-on: ubuntu-latest
    needs: plan
    environment: prod-workspace
    steps:
      - env:
          PLANID: ${{needs.plan.outputs.planid}}
        run: echo "$PLANID"

Your requirements.txt file can look something like the below:

tecton[pyspark]~=0.7.3

To authenticate your Github Action, you'll need to create a Service Account to obtain a Tecton API key, and assign the Service Account the Editor role for the appropriate workspace:

1. Create the Service Account to obtain the API key.

tecton service-account create \
      --name "cicd-service-account" \
      --description "A cicd example"

Output:

Save this API Key - you will not be able to get it again.
API Key:            <Your-api-key>
Service Account ID: <Your-Service-Account-Id>

2. Assign the Editor role for the workspace to the Service Account.

tecton access-control assign-role --role editor \
      --workspace <Your-workspace> \
      --service-account <Your-Service-Account-Id>

Output:

Successfully updated role.

Validate Plan with JSON output

You can output a JSON version of a to-be-applied diff using the --json-out flag.

tecton plan --json-out <path>

This can be useful in a CI/CD pipeline to prevent applying unintended changes by running a custom script on the output.

Example json file output:

{
  "objectDiffs": [
    {
      "transitionType": "DELETE",
      "objectMetadata": {
        "name": "transaction_user_has_good_credit",
        "objectType": "FEATURE_VIEW",
        "owner": "john@doe.com",
        "description": "Whether the user had a good credit score (over 670) as of the time of a transaction."
      }
    },
    {
      "transitionType": "RECREATE",
      "objectMetadata": {
        "name": "continuous_feature_service",
        "objectType": "FEATURE_SERVICE",
        "owner": "john@doe.com",
        "description": "A Feature Service providing continuous features."
      }
    }
  ]
}

See Types of Repository Changes doc to help understand the plan output.

Apply Generated Plan

When a plan is successfully generated with tecton plan, an ID for that plan is printed to the console after the plan contents.

...
↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Generated plan ID is a25e9516ebde475690ef3806e1f12e1e

If tecton plan was run with the --json-out flag, the plan ID is also included as a field in the JSON file:

{
  "objectDiffs": [
    ...
  ],
  "planId": "a25e9516ebde475690ef3806e1f12e1e"
}

After this plan is approved through your team's workflow (whether automated or manual), you can directly apply the plan by passing the plan ID through the --plan-id parameter:

tecton apply --plan-id=a25e9516ebde475690ef3806e1f12e1e

This will apply the plan directly without recomputing a new plan.

If any changes have been made to the feature repo since the plan was generated (i.e. someone ran tecton apply), then you will get an error and must generate a new plan on top of the current repo state.

Protecting Critical Objects from Destruction

Changes to Tecton objects may trigger destructive changes which can lead to serving downtime or expensive rematerializations. You can set the prevent_destroy on critical objects to avoid accidentally destroying them. See this guide for more information about prevent_destroy.