Need some advice, any help really appreciated.
Trying to create a standalone service fabric cluster as https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-creation-for-windows-server
To be specific a Windows.X509.MultiMachine one.
So, i have an Active Directory domain, it's certificate. One machine as domain controller and three nodes I want to create a cluster of.
A TestConfiguration.ps1 powershell script indicates that everything is ok, but CreateServiceFabricCluster.ps1 takes a lot time and throws an error "Timed out waiting for Installer Service to complete for machine DEV1." etc. DiagnosticsStore is empty.
DeploymentTrace:
2018/02/14-13:58:51.837,Info,5084,SystemFabricDeployer.SFDeployer,Running Best Practices Analyzer...
2018/02/14-13:58:51.844,Verbose,5084,SystemFabricDeployer.SFDeployer,Validating executing user is an Administrator.
2018/02/14-13:58:51.850,Verbose,5084,SystemFabricDeployer.SFDeployer,Converting JSON config to model.
2018/02/14-13:58:52.200,Error,5084,SystemFabricDeployer.SFDeployer,Config validation: Server Certificate Thumbprint contains invalid characters
2018/02/14-13:58:52.201,Verbose,5084,SystemFabricDeployer.SFDeployer,Validating CAB file is valid at C:\Users\Administrator\Documents\mssf\DeploymentRuntimePackages\MicrosoftAzureServiceFabric.6.1.456.9494.cab.
2018/02/14-13:58:52.591,Error,5084,SystemFabricDeployer.SFDeployer,Best Practices Analyzer determined environment has an issue. Please see additional BPA log output in DeploymentTraces folder.
2018/02/14-13:58:52.592,Error,5084,SystemFabricDeployer.SFDeployer,Cluster Setup cancelled due to validation error(s) found by Best Practices Analyzer. Inspect details in DeploymentTraces log folder local to executing location.
EventLog:
2/14/2018 6:52:33 AM - DEV1 - Error - Timed out waiting for Installer Service to complete for machine DEV1. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
2/14/2018 6:52:33 AM - DEV1 - Error - Timed out waiting for Installer Service to complete for machine DEV2. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
2/14/2018 6:52:33 AM - DEV1 - Error - Timed out waiting for Installer Service to complete for machine DEV3. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
2/14/2018 6:52:49 AM - DEV1 - Error - federation open failed with FABRIC_E_TIMEOUT
2/14/2018 6:52:49 AM - DEV1 - Error - Fabric Node open failed with error code = FABRIC_E_TIMEOUT
2/14/2018 6:52:52 AM - DEV1 - Error - Target information file exists. This would indicate that Fabric node open or Fabric uninstall didn't happen successfully. Rolling back..
2/14/2018 6:57:50 AM - DEV1 - Error - federation open failed with FABRIC_E_TIMEOUT
2/14/2018 6:57:50 AM - DEV1 - Error - Fabric Node open failed with error code = FABRIC_E_TIMEOUT
2/14/2018 7:02:51 AM - DEV1 - Error - federation open failed with FABRIC_E_TIMEOUT
2/14/2018 7:02:51 AM - DEV1 - Error - Fabric Node open failed with error code = FABRIC_E_TIMEOUT
2/14/2018 7:04:24 AM - DEV1 - Error - CreateCluster Error: System.AggregateException: One or more errors occurred. ---> System.ServiceProcess.TimeoutException: Timed out waiting for Installer Service to complete for machine DEV3. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController installerSvc)
at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.RunFabricServices(List`1 machines, FabricPackageType fabricPackageType)
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.<CreateClusterAsyncInternal>d__1.MoveNext()
---> (Inner Exception #0) System.ServiceProcess.TimeoutException: Timed out waiting for Installer Service to complete for machine DEV3. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController installerSvc)
at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )<---
---> (Inner Exception #1) System.ServiceProcess.TimeoutException: Timed out waiting for Installer Service to complete for machine DEV1. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController installerSvc)
at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )<---
---> (Inner Exception #2) System.ServiceProcess.TimeoutException: Timed out waiting for Installer Service to complete for machine DEV2. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController installerSvc)
at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )<---
My nodes resolvable names are DEV1, DEV2, DEV3, domain "hp.dev", it's netbios name HPDEV. All of them are on Windows Server 2016 Standard if it'll help.
Here is a ClusterConfig.json:
{
"name": "hP Secure Cluster",
"clusterConfigurationVersion": "1.0.0",
"apiVersion": "10-2017",
"nodes": [
{
"nodeName": "Node1",
"iPAddress": "DEV1",
"nodeTypeRef": "NodeTypeDefault",
"faultDomain": "fd:/dc1/r1",
"upgradeDomain": "UD1"
},
{
"nodeName": "Node2",
"iPAddress": "DEV2",
"nodeTypeRef": "NodeTypeDefault",
"faultDomain": "fd:/dc1/r2",
"upgradeDomain": "UD2"
},
{
"nodeName": "Node3",
"iPAddress": "DEV3",
"nodeTypeRef": "NodeTypeDefault",
"faultDomain": "fd:/dc1/r3",
"upgradeDomain": "UD3"
}
],
"properties": {
"diagnosticsStore": {
"metadata": "Please replace the diagnostics file share with an actual file share accessible from all cluster machines.",
"dataDeletionAgeInDays": "7",
"storeType": "FileShare",
"connectionstring": "C:\\ProgramData\\SF\\DiagnosticsStore"
},
"security": {
"metadata": "The Credential type X509 indicates this is cluster is secured using X509 Certificates. The thumbprint format is - d5 ec 42 3b 79 cb e5 07 fd 83 59 3c 56 b9 d5 31 24 25 42 64.",
"ClusterCredentialType": "Windows",
"ServerCredentialType": "X509",
"WindowsIdentities": {
"ClusterIdentity": "HPDEV\\Administrator"
},
"CertificateInformation": {
"ServerCertificateCommonNames": {
"CommonNames": [
{
"CertificateCommonName": "HPCA",
}
],
"X509StoreName": "My"
}
}
},
"no
