Skip to content
Customer Portal
Contact us
  • There are no suggestions because the search field is empty.

Having issues creating a workspace in BYOC?

You create a workspace on your BYOC cluster and it never becomes healthy.  Either the workspace shows Create failed in the portal, the creation returns a 422 error or maybe the workspace stays in a creating state and never turns green.

Causes and fixes

Work through these in order; the first two cover most cases.

1. The agent is offline

A workspace can only be created on an agent that is Online in the Ververica portal.

  • Check the agent status in the portal and that the agent pods are running: 
    kubectl get pods -n <agent-namespace>

     

2. Agent resources were applied in the wrong order (GitOps)

If you deploy the agent with Argo CD or another GitOps tool, a sync can apply resources out of order and creation fails with a 422 error.

  • Avoid full syncs of the agent application. Sync only the resources that are out of sync.
  • Do not sync Secrets, ConfigMaps or custom resources that the agent generates and rotates itself (auth secret, TLS certificates, workspace resources). Exclude them from reconciliation, e.g. with Argo CD ignoreDifferences.
  • If resources were already applied out of order, delete the affected resources and sync them again with parent resources first.

3. The IAM roles are misconfigured

The agent uses two IAM roles. If a role name, policy or trust relationship is wrong, the workspace cannot start. A typical sign is the vvp-agent-0 pod crash-looping with an access error:

kubectl get pods -A | grep vvp-agent
kubectl logs vvp-agent-0 -n <namespace-from-above> 
# .. is not authorized to perform: s3:GetBucketLocation on resource.."

Check the role names. Use the default names VVCAdminRole and VVCTenantRole exactly (names are case-sensitive). If you use custom names, pass both when you install or upgrade the agent:

helm upgrade --install vv-agent oci://registry.ververica.cloud/agent-charts/vv-agent \
   -n <agent-namespace> --create-namespace \
   --version <agent-version> \
   --set global.pyxisRoleArn="arn:aws:iam::<account-id>:role/<custom-admin-role>" \
   --set pyxis-controlplanes.vvcTenantRoleName=<custom-tenant-role> \
   --values <your-values>.yaml

pyxisRoleArn takes the full ARN; vvcTenantRoleName takes the role name only.

Check the policies and trust relationships in the AWS IAM console:

Role

Trusted entity

Permissions

VVCAdminRole

Web identity: your EKS cluster's OIDC provider, audience sts.amazonaws.com

sts:AssumeRole on the tenant role's ARN

VVCTenantRole

AWS account: this account

s3:PutObject, s3:GetObject, s3:DeleteObject on <bucket>/*; s3:ListBucket, s3:GetBucketLocation on <bucket>

Common mistakes: the admin policy still points at VVCTenantRole after you renamed the tenant role; the tenant policy does not include the bucket selected for the workspace; the admin role trusts the OIDC provider of an old or different cluster.

Check the pyxis-admin service account annotation. It must carry the admin role's ARN:

kubectl get serviceaccount pyxis-admin -n <agent-namespace> -o yaml | grep role-arn 
# eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<admin-role-name>

If it is missing or wrong, set global.pyxisRoleArn correctly, redeploy the agent and restart the agent pods so they pick up the new role.

Check the RAM security tokens are ready. The agent creates a RAMSecurityToken in each workspace namespace once it can assume the tenant role. All of them must report ready:

kubectl get ramsecuritytokens -A

A token that is not ready points to the admin role, its trust relationship or the service account annotation above. kubectl describe ramsecuritytoken <name> -n <namespace> shows the error.

Check the STS secrets exist in the workspace namespaces. Each workspace has two namespaces, <workspace-id> and flink-<workspace-id>-vvp-deploy. Both must contain the STS credential secret created from the token:

kubectl get secrets -n <workspace-id> | \
  grep sts kubectl get secrets -n flink-<workspace-id>-vvp-deploy | \
  grep sts

If the secrets are missing, fix the token first; the secrets are created once it is ready.

4. The workspace stays failed after you fixed the cause

After repeated failed attempts, a workspace is permanently marked as failed and does not retry on its own. Once the cause is fixed, either delete it and create a new one (see below), or contact Ververica Support to retry the creation.

Clean up a workspace that never became healthy

Deleting a failed workspace in the portal may leave its resources on your cluster. Each leftover keeps a crash-looping pod and reserves about 1 CPU and 12 GiB. To check and remove them:

  1. List workspace resources on the cluster and compare the IDs with the workspaces shown in the portal:

     

    kubectl get controlplane -n <agent-namespace>

     

  2. For each workspace ID that no longer exists in the portal, delete its resource. The agent removes the remaining namespaces and pods within about a minute. 
    kubectl delete controlplane <workspace-id> -n <agent-namespace>

Only delete IDs that are not active workspaces. The list also contains agent-level entries (e.g. flink, flink-byoc); leave those in place.

Still stuck?

Open a ticket with Ververica Support and include:

  • The workspace ID and the time of the creation attempt.
  • The agent status shown in the portal.
  • Output of kubectl get pods -A | grep -E 'vvp|agent'.
  • Logs from any pod that is not running, e.g. kubectl logs vvp-agent-0 -n <namespace>.
  • Whether you deploy the agent with Helm directly or through a GitOps tool.