diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index f25ab71625..6d8970d9ca 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -69,6 +69,13 @@ When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to == Authorize access to AWS Glue +ifdef::env-cloud[] +For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the Redpanda broker role when Iceberg is enabled. You don't need to manually create IAM policies or roles for Glue access. + +For clusters created before March 2026, you must run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the broker role with the necessary Glue permissions. +endif::[] + +ifndef::env-cloud[] You must allow Redpanda access to AWS Glue services in your AWS account. You can use the same access credentials that you configured for S3 (IAM role, access keys, and KMS key), as long as you have also added read and write access to AWS Glue Data Catalog. For example, you could create a separate IAM policy that manages access to AWS Glue and attach it to the IAM role that Redpanda also uses to access S3. Add all AWS Glue API actions in the policy (`"glue:*"`) on the following resources: @@ -100,6 +107,7 @@ Your IAM policy should include a statement similar to the following: ---- For more information on configuring IAM permissions, see the https://docs.aws.amazon.com/glue/latest/dg/configure-iam-for-glue.html[AWS Glue documentation^]. +endif::[] == Configure authentication and credentials @@ -115,12 +123,14 @@ You must configure credentials for the AWS Glue Data Catalog integration in eith endif::[] ifdef::env-cloud[] -You must configure credentials for the AWS Glue Data Catalog integration using the following properties: +You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways: -* config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] set to `config_file` -* config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] -* config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`], added as a secret value (see the <> for details) -* config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] +* Allow Redpanda to use the same `cloud_storage_*` credential properties already configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach, especially in BYOC deployments where the broker's IAM role already includes the necessary Glue permissions. +* If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: +** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`]. To use the broker's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`. +** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only) +** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] (static credentials only), added as a secret value (see the <> for details) +** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] endif::[] == Update cluster configuration @@ -140,8 +150,8 @@ iceberg_delete: false iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://glue..amazonaws.com/iceberg iceberg_rest_catalog_authentication_mode: aws_sigv4 - # Because Tiered Storage does not support the use of distinct buckets for Iceberg, - # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket +# Because Tiered Storage does not support the use of distinct buckets for Iceberg, +# always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket iceberg_rest_catalog_base_location: s3:/// # Use the iceberg_rest_catalog_aws_* properties if you want to # use separate AWS credentials for the catalog, or omit these lines to reuse S3 @@ -163,37 +173,64 @@ Use your own values for the following placeholders: -- endif::[] ifdef::env-cloud[] -Use `rpk` like in the following example, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. +Use `rpk` like in the following examples, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. + +[tabs] +====== +IAM role (instance metadata):: ++ +-- [,bash] ---- +# Glue requires Redpanda Iceberg tables to be manually deleted +# so iceberg_delete is set to false. rpk cloud login rpk profile create --from-cloud rpk cluster config set \ iceberg_enabled=true \ + iceberg_delete=false \ + iceberg_catalog_type=rest \ + iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ + iceberg_rest_catalog_authentication_mode=aws_sigv4 \ + iceberg_rest_catalog_credentials_source=aws_instance_metadata \ + iceberg_rest_catalog_aws_region= \ + iceberg_rest_catalog_base_location=s3:/// +---- +-- + +Static credentials:: ++ +-- +[,bash] +---- +# Glue requires Redpanda Iceberg tables to be manually deleted +# so iceberg_delete is set to false. +rpk cluster config set \ + iceberg_enabled=true \ + iceberg_delete=false \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ iceberg_rest_catalog_authentication_mode=aws_sigv4 \ - # Because Tiered Storage does not support the use of distinct buckets for Iceberg, - # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket - iceberg_rest_catalog_base_location=s3:/// \ - # Set credentials source to config_file if using - # iceberg_rest_catalog_aws_access_key and iceberg_rest_catalog_aws_secret_key iceberg_rest_catalog_credentials_source=config_file \ iceberg_rest_catalog_aws_region= \ iceberg_rest_catalog_aws_access_key= \ - iceberg_rest_catalog_aws_secret_key='${secrets.}' + iceberg_rest_catalog_aws_secret_key='${secrets.}' \ + iceberg_rest_catalog_base_location=s3:/// ---- +-- +====== + Use your own values for the following placeholders: + -- - ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. -- `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. -- ``: The AWS access key ID for your Glue service account. -- ``: The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. +- `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the bucket you created as a customer-managed resource. ++ +As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. +- `` (static credentials only): The AWS access key ID for your Glue service account. +- `` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. -- endif::[] + diff --git a/modules/reference/partials/properties/cluster-properties.adoc b/modules/reference/partials/properties/cluster-properties.adoc index ccbbe20671..0c78a76b41 100644 --- a/modules/reference/partials/properties/cluster-properties.adoc +++ b/modules/reference/partials/properties/cluster-properties.adoc @@ -7087,6 +7087,7 @@ endif::[] // end::redpanda-cloud[] +// tag::redpanda-cloud[] === iceberg_rest_catalog_aws_credentials_source *Accepted values*: `aws_instance_metadata`, `azure_aks_oidc_federation`, `azure_vm_instance_metadata`, `config_file`, `gcp_instance_metadata`, `sts`. @@ -7147,6 +7148,7 @@ endif::[] |=== +// end::redpanda-cloud[] // tag::redpanda-cloud[] === iceberg_rest_catalog_aws_region diff --git a/modules/reference/partials/properties/object-storage-properties.adoc b/modules/reference/partials/properties/object-storage-properties.adoc index 01d278d44d..def0af9e5b 100644 --- a/modules/reference/partials/properties/object-storage-properties.adoc +++ b/modules/reference/partials/properties/object-storage-properties.adoc @@ -1519,6 +1519,7 @@ endif::[] |=== +// tag::redpanda-cloud[] === cloud_storage_credentials_source The source of credentials used to authenticate to object storage services. @@ -1574,6 +1575,7 @@ endif::[] |=== +// end::redpanda-cloud[] === cloud_storage_crl_file