IAM support for Glue in Cloud #1633
+58
−17
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -69,6 +69,13 @@ When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to | |
|
|
||
| == Authorize access to AWS Glue | ||
|
|
||
| ifdef::env-cloud[] | ||
| For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the Redpanda broker role when Iceberg is enabled. You don't need to manually create IAM policies or roles for Glue access. | ||
|
|
||
| For clusters created before March 2026, you must run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the broker role with the necessary Glue permissions. | ||
| endif::[] | ||
|
|
||
| ifndef::env-cloud[] | ||
| You must allow Redpanda access to AWS Glue services in your AWS account. You can use the same access credentials that you configured for S3 (IAM role, access keys, and KMS key), as long as you have also added read and write access to AWS Glue Data Catalog. | ||
|
|
||
| For example, you could create a separate IAM policy that manages access to AWS Glue and attach it to the IAM role that Redpanda also uses to access S3. Add all AWS Glue API actions in the policy (`"glue:*"`) on the following resources: | ||
|
|
@@ -100,6 +107,7 @@ Your IAM policy should include a statement similar to the following: | |
| ---- | ||
|
|
||
| For more information on configuring IAM permissions, see the https://docs.aws.amazon.com/glue/latest/dg/configure-iam-for-glue.html[AWS Glue documentation^]. | ||
| endif::[] | ||
|
|
||
| == Configure authentication and credentials | ||
|
|
||
|
|
@@ -115,12 +123,14 @@ You must configure credentials for the AWS Glue Data Catalog integration in eith | |
| endif::[] | ||
|
|
||
| ifdef::env-cloud[] | ||
| You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways: | ||
|
|
||
| * Allow Redpanda to use the same `cloud_storage_*` credential properties already configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach, especially in BYOC deployments where the broker's IAM role already includes the necessary Glue permissions. | ||
| * If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: | ||
| ** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`]. To use the broker's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`. | ||
| ** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only) | ||
| ** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] (static credentials only), added as a secret value (see the <<update-cluster-configuration,next section>> for details) | ||
| ** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] | ||
| endif::[] | ||
|
|
||
| == Update cluster configuration | ||
|
|
@@ -140,8 +150,8 @@ iceberg_delete: false | |
| iceberg_catalog_type: rest | ||
| iceberg_rest_catalog_endpoint: https://glue.<glue-region>.amazonaws.com/iceberg | ||
| iceberg_rest_catalog_authentication_mode: aws_sigv4 | ||
| # Because Tiered Storage does not support the use of distinct buckets for Iceberg, | ||
| # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket | ||
| iceberg_rest_catalog_base_location: s3://<bucket-name>/<warehouse-path> | ||
| # Use the iceberg_rest_catalog_aws_* properties if you want to | ||
| # use separate AWS credentials for the catalog, or omit these lines to reuse S3 | ||
|
|
@@ -163,37 +173,64 @@ Use your own values for the following placeholders: | |
| -- | ||
| endif::[] | ||
| ifdef::env-cloud[] | ||
| Use `rpk` like in the following examples, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. | ||
| + | ||
| [tabs] | ||
| ====== | ||
| IAM role (instance metadata):: | ||
| + | ||
| -- | ||
| [,bash] | ||
| ---- | ||
| # Glue requires Redpanda Iceberg tables to be manually deleted | ||
| # so iceberg_delete is set to false. | ||
| rpk cloud login | ||
|
|
||
| rpk profile create --from-cloud <cluster-id> | ||
|
|
||
| rpk cluster config set \ | ||
| iceberg_enabled=true \ | ||
| iceberg_delete=false \ | ||
| iceberg_catalog_type=rest \ | ||
| iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \ | ||
| iceberg_rest_catalog_authentication_mode=aws_sigv4 \ | ||
| iceberg_rest_catalog_credentials_source=aws_instance_metadata \ | ||
| iceberg_rest_catalog_aws_region=<glue-region> \ | ||
| iceberg_rest_catalog_base_location=s3://<bucket-name>/<warehouse-path> | ||
| ---- | ||
kbatuigas marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| -- | ||
|
|
||
| Static credentials:: | ||
| + | ||
| -- | ||
| [,bash] | ||
| ---- | ||
| # Glue requires Redpanda Iceberg tables to be manually deleted | ||
| # so iceberg_delete is set to false. | ||
| rpk cluster config set \ | ||
| iceberg_enabled=true \ | ||
| iceberg_delete=false \ | ||
| iceberg_catalog_type=rest \ | ||
| iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \ | ||
| iceberg_rest_catalog_authentication_mode=aws_sigv4 \ | ||
| iceberg_rest_catalog_credentials_source=config_file \ | ||
| iceberg_rest_catalog_aws_region=<glue-region> \ | ||
| iceberg_rest_catalog_aws_access_key=<glue-access-key> \ | ||
| iceberg_rest_catalog_aws_secret_key='${secrets.<glue-secret-key-name>}' \ | ||
| iceberg_rest_catalog_base_location=s3://<bucket-name>/<warehouse-path> | ||
| ---- | ||
| -- | ||
| ====== | ||
| + | ||
| Use your own values for the following placeholders: | ||
| + | ||
| -- | ||
| - `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. | ||
| - `<bucket-name>` and `<warehouse-path>`: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3://<bucket-name>/iceberg`. For BYOC clusters, the bucket name is `redpanda-cloud-storage-<cluster-id>`. For BYOVPC clusters, use the name of the bucket you created as a customer-managed resource. | ||
|
Contributor
Author
There was a problem hiding this comment. @simon0191 Could you confirm that this is OK to add here, and should we update the table in this doc as well? |
||
| + | ||
| As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. | ||
| - `<glue-access-key>` (static credentials only): The AWS access key ID for your Glue service account. | ||
| - `<glue-secret-key-name>` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. | ||
| -- | ||
| endif::[] | ||
| + | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@simon0191 Is this correct -- Cloud users won't have to do anything special for IAM, so the lines that follow this should display for Self-managed only?