Skip to content
Customer Portal
Contact us
  • There are no suggestions because the search field is empty.

Multiple Jobs are Started from one Ververica Platform Deployment

Issue

When a deployment starts in Ververica Platform, multiple Flink jobs may be started unexpectedly. This occurs because the appmanager controller times out while writing job state to a persistent volume backed by Amazon EFS, then retries and creates a duplicate job. The following TimeoutException may also appear in Ververica Platform appmanager logs:

2020-04-14 15:15:23.873 DEBUG 1 --- [eduler-worker-1] c.d.a.c.c.drivers.ScheduledController : Exception while invoking the controller DeploymentControllerLogic{deploymentId=12d6b0e7-b9e8-4736-9ff4-d63688cc6882}. Will retry after backing off for 3210 milliseconds

java.lang.RuntimeException: java.util.concurrent.TimeoutException
 at com.google.common.base.Throwables.propagate(Throwables.java:241) ~[guava-28.0-jre.jar!/:na]
 at com.dataartisans.appmanager.controller.api.MoreFutures.deref(MoreFutures.java:41) ~[appmanager-controller-api-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.JobRepositoryFacade.deref(JobRepositoryFacade.java:147) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.JobRepositoryFacade.create(JobRepositoryFacade.java:58) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.domain.deployment.ToRunning.createJob(ToRunning.java:248) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.domain.deployment.ToRunning.tryTransition(ToRunning.java:145) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.domain.deployment.DeploymentControllerLogic.dispatchOnSpecState(DeploymentControllerLogic.java:84) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.domain.deployment.DeploymentControllerLogic.invoke(DeploymentControllerLogic.java:72) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.drivers.ScheduledController.invoke(ScheduledController.java:56) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.drivers.SchedulerWorker.invokeSilently(SchedulerWorker.java:50) [appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.drivers.SchedulerWorker.run(SchedulerWorker.java:27) [appmanager-controller-2.1.0.jar!/:na]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_242]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_242]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
Caused by: java.util.concurrent.TimeoutException: null
 at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) ~[na:1.8.0_242]
 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[na:1.8.0_242]
 at com.dataartisans.appmanager.controller.api.MoreFutures.deref(MoreFutures.java:39) ~[appmanager-controller-api-2.1.0.jar!/:na]
 ... 12 common frames omitted

Environment

This happens in Ververica Platform 2.1.0 with persistent volumes backed by Amazon Elastic File System (EFS).

This may happen in Ververica Platform 2.1.1 or later with EFS if a longer timeout is needed (see how to increase the timeout value as described below).

Resolution

Upgrade to Ververica Platform 2.1.1 or later and, if necessary, set a longer timeout in your values.yaml, e.g.,:

vvp:
  appmanager:
    persistence.repository.timeout.ms: 15000

The time unit here is millisecond. The default value is 10000 (10 seconds). Increase the timeout if necessary.

To verify the fix, monitor the appmanager logs after upgrading and adjusting the timeout. If the TimeoutException no longer appears and only a single Flink job is started per deployment, the issue is resolved.

Cause

Accessing persistent volumes backed by Amazon EFS can be slow. When the appmanager's internal repository call exceeds the configured timeout (default: 10 seconds), the controller treats the job creation as failed, backs off, and retries — potentially creating a duplicate Flink job.

Related Information

 

Written by Jun Qin · Published 24 Apr 2020 · Last updated 11 Nov 2025