Configure Customer-managed keys for Encryption
Tecton always encrypts feature data and metadata, such as feature definitions. Tecton accounts can optionally be configured to use customer-managed keys for feature data and metadata encryption. By using customer-managed keys, a customer has the control to revoke access to this data from both Tecton and the Cloud Provider.
Customer-managed keys must be enabled during Tecton account creation as they cannot be enabled for existing accounts.
Configure Customer-managed keys with AWS KMS​
The following steps illustrate how to configure Customer-managed keys for DynamoDB, S3, and the Tecton Control Plane.
- Notify Tecton that you would like to use Customer-managed keys when creating your Tecton account.
- Tecton will provide the AWS Account ID associated with your Tecton account.
- Grant permissions for Tecton to use the key in the key policy. See instructions below for details.
- Provide Tecton with the KMS key ARN.
The instructions below show how to configure permissions using Terraform or the AWS console.
Configure Key Policy in Terraform​
-
The Tecton representative will provide the Tecton control plane AWS account ID
-
Note the ARN of the KMS key to be used
-
Add the KMS key ARN to the Tecton deployment module used to provision the data plane
-
Add
kms_key_id = <your_kms_key_arn>
to match the appropriate KMS key ARN -
For example:
locals {
# Deployment name must be less than 22 characters (AWS limitation)
deployment_name = "my-deployment-name"
# The region and account_id of this Tecton account you just created
region = "my-region"
account_id = "1234567890"
# Get from your Tecton rep
tecton_control_plane_root_principal = "arn:aws:iam::987654321:root"
# get from your tecton rep
cross_account_external_id = "tecton-external-id"
key_kms_id = "arn:aws:kms:us-west-2:1234567890:key/aaaaaaa-bbbb-cccc-dddd-eeeeeeeee"
# OPTIONAL for EMR notebook clusters in a different account (see optional block at end of file)
# cross_account_arn = "arn:aws:iam::9876543210:root"
}
module "tecton" {
source = "../deployment"
deployment_name = local.deployment_name
account_id = local.account_id
region = local.region
cross_account_external_id = local.cross_account_external_id
kms_key_id = local.key_kms_id
create_emr_roles = true
s3_read_write_principals = [local.tecton_control_plane_root_principal]
}
-
-
Run
terraform apply
to apply the changes -
Provide the KMS key ARN to the Tecton representative to complete the configuration process in the control plane
Configure Key Policy in the AWS console​
-
Open AWS Console, go to Key Management Service page
- Click on the key that is to be used
-
Open the Key Policy tab, and click on Edit to open the key policy editor
-
Paste in the following JSON policy, replacing:
${ACCOUNT_ID}
 with the account ID of your Tecton Data Plane account${CROSS_ACCOUNT_ROLE_NAME}
with the name of the cross-account role.${SPARK_ROLE_NAME}
 with the name of the Databricks/EMR IAM role. For Rift compute, please disregard this principal altogether.${TECTON_CONTROL_PLANE_ACCOUNT}
with the account ID of the Tecton Control Plane provided by the Tecton representative
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${ACCOUNT_ID}:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::${ACCOUNT_ID}:role/${CROSS_ACCOUNT_ROLE_NAME}",
"arn:aws:iam::${ACCOUNT_ID}:role/${SPARK_ROLE_NAME}",
"arn:aws:iam::${TECTON_CONTROL_PLANE_ACCOUNT}:root"
]
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::${ACCOUNT_ID}:role/${CROSS_ACCOUNT_ROLE_NAME}",
"arn:aws:iam::${ACCOUNT_ID}:role/${SPARK_ROLE_NAME}",
"arn:aws:iam::${TECTON_CONTROL_PLANE_ACCOUNT}:root"
]
},
"Action": ["kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant"],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
} -
Provide the KMS key ARN to Tecton representative to complete the rest of the cluster deployment process
Other optional components​
Databricks Cross-account Access​
In the case where Databricks runs in a separate account from the Tecton data plane, the following steps illustrate how to grant the appropriate permissions to the Databricks workers.
-
If the Terraform method was used, pass in the Databricks account principal to
var.kms_key_additional_principals
variable in step 3.-
For example:
locals {
# Deployment name must be less than 22 characters (AWS limitation)
deployment_name = "my-deployment-name"
# The region and account_id of this Tecton account you just created
region = "my-region"
account_id = "1234567890"
# Get from your Tecton rep
tecton_control_plane_root_principal = "arn:aws:iam::987654321:root"
# get from your tecton rep
cross_account_external_id = "tecton-external-id"
key_kms_id = "arn:aws:kms:us-west-2:1234567890:key/aaaaaaa-bbbb-cccc-dddd-eeeeeeeee"
# OPTIONAL for Databricks notebook clusters in a different account (see optional block at end of file)
cross_account_arn = "arn:aws:iam::9876543210:root"
}
module "tecton" {
source = "../deployment"
deployment_name = local.deployment_name
account_id = local.account_id
tecton_assuming_account_id = local.tecton_assuming_account_id
region = local.region
cross_account_external_id = local.cross_account_external_id
kms_key_id = local.key_kms_id
create_emr_roles = true
s3_read_write_principals = [local.tecton_control_plane_root_principal]
}
-
-
If the AWS method was used, add the Databricks account to the list of principals in step 3,
- With
${DATABRICKS_ACCOUNT_ID}
being the account id of the Databricks account
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${ACCOUNT_ID}:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::${ACCOUNT_ID}:role/${CROSS_ACCOUNT_ROLE_NAME}",
"arn:aws:iam::${ACCOUNT_ID}:role/${SPARK_ROLE_NAME}",
"arn:aws:iam::${TECTON_CONTROL_PLANE_ACCOUNT}:root",
"arn:aws:iam::${DATABRICKS_ACCOUNT_ID}:root"
]
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::${ACCOUNT_ID}:role/${CROSS_ACCOUNT_ROLE_NAME}",
"arn:aws:iam::${ACCOUNT_ID}:role/${SPARK_ROLE_NAME}",
"arn:aws:iam::${TECTON_CONTROL_PLANE_ACCOUNT}:root",
"arn:aws:iam::${DATABRICKS_ACCOUNT_ID}:root"
]
},
"Action": ["kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant"],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
} - With
-
In the Databricks account, attach the following policy to the role used by Databricks instance profile,
- With
${KMS_KEY_ARN}
being the ARN of your KMS key
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "${ROLE_ARN}"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey",
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "${KMS_KEY_ARN}"
}
]
} - With
Redis​
If you choose Redis as the online feature store, at-rest encryption must be enabled at replication group creation time. This means you must bring an Elasticache Redis cluster with at-rest encryption already enabled prior to connecting to Tecton.
See the Redis documentation for how to configure at-rest encryption.