vSphere daemon bootstrap

[antonOO]

For some reason the daemon fails to bootstrap on vSphere.
The daemon-config.yaml is :

providers:
  vsphere:
    - name: my-vs
      vsphere_user: [email protected]
      vsphere_password: ********
      vsphere_url: https://10.160.**.**
      datastore: datastore1
      datacenter: unik_dc
      network: VM Network

I can see that the VsphereUnikInstanceListener is being created and in the VM console log I see that it is available on port 3000 with its id:

But on the terminal from which I run $ unik daemon , I reach:

EBU[0121] Powering on VirtualMachine:vm-159... OK      
INFO[0121] listening for udp heartbeat...               
DEBU[0121] ARE WE LISTENING ON THE SOCKET YET?&{{0xc8202b6a10}} 
INFO[0121] UDP Server listening on 0.0.0.0:9967        

and then:

WARN[0181] error encountered, ensuring vm and disks are destroyed  error=[vsphere/deploy_instance_listener.go:150] failed to retrieve instance listener ip. is unik instance listener running?: {[common/get_instance_listener_ip.go:20] getting instance listener ip timed out after 1m0s}

...

ERRO[0201] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:92] initializing vsphere provider: {[vsphere/vsphere_provider.go:53] deploying virtualbox instance listener: {[vsphere/deploy_instance_listener.go:59] launching instance of instance listener: {[vsphere/deploy_instance_listener.go:150] failed to retrieve instance listener ip. is unik instance listener running?: {[common/get_instance_listener_ip.go:20] getting instance listener ip timed out after 1m0s}}}}} 

In vSphere i have created an ESXi host responsible for running the deamon and storing it in datastore1. The connection between the running VsphereUnikInstanceListener and the PC from which I am running unik daemon cannot be established.

[gberginc]

@antonOO I've got no experience with vSphere, but I wonder if port 3000 is accessible from where the daemon is trying to reach it?

@miha-plesko did you perhaps experience something similar?

[antonOO]

Well, when I start unik daemon --debug and the curl the VsphereUnikInstanceListener id I receive - 404 page not found , if I turn down the VM running the instance listener I do not receive a response, therefore the listener is reachable, but I am not querying the required endpoint maybe?

Thank you fro the response!

[miha-plesko]

Hi @antonOO ,

from the terminal on your local computer (where unik deamon) is running, are you able to get response to:

$ curl 10.160.90.10:3000/instances

The 10.160.90.10 is an IP of your VsphereUnikInstanceListener as I can see from the PNG that you've attached. If you can't get a JSON response there, then vSphere may actually be blocking incoming TCP traffic on port 3000 of VsphereUnikInstanceListener.

[antonOO]

I received {} , therefore the VsphereUnikInstanceListener is running, the problem is it is not attached to a terminal on the PC i am working on and I am not quite sure if everything is working fine ...

anton@anton-Latitude-E7270:~$ curl 10.160.79.226:3000
404 page not found
anton@anton-Latitude-E7270:~$ curl 10.160.79.226:3000/instances
{}

...

anton@anton-Latitude-E7270:~$ unik target --host 10.160.79.226
INFO[0000] target set: 10.160.79.226:3000               
anton@anton-Latitude-E7270:~$ unik instances
INFO[0000] listing instances                             host=10.160.79.226:3000
ERRO[0000] failed listing instances: [client/instances.go:28] response body {} did not unmarshal to type []*types.Instance: {json: cannot unmarshal object into Go value of type []*types.Instance} 

...

anton@anton-Latitude-E7270:~/unik_projects/go$ unik build --name goImage --path ./ --base rump --language go --provider vsphere
INFO[0000] running unik build                            args= base=rump force=false host=10.160.79.226:3000 language=go mountPoints=[] name=goImage path=./ provider=vsphere
INFO[0000] App packaged as tarball: /tmp/sources.tar.gz.192148109
 
ERRO[0000] build failed: [cmd/build.go:105] building image failed: {[client/images.go:63] failed with status 404: 404 page not found
} 

[miha-plesko]

Could it be that you've pointed your unik client (the one on your local machine) to the IP of VsphereUnikInstanceListener instead to the IP of the machine where unik daemon is running? They are both using port 3000 so one can easily mix them :D

I must admit, however, that I haven't tested myself if running unik client on different machine that unik deamon actually works (in theory it should, but...). Is it possible for you to just copy-paste the daemon-config.yaml to your local machine (where you run client) and try to run daemon there? If you try this, don't forget to do:

$ unik target --host localhost

[antonOO]

Just to clarify my understanding of how this deamon works. From my local machine I execute unik daemon --debug -> then via the daemon-config.yaml the sphereUnikInstanceListener is created on the server via vsphere (in this case). The daemon is running in the terminal where I executed unik daemon --debug and is used to talk to the VsphereUnikInstanceListener. In another terminal I point out the IP of the daemon responsible for the communication with the instance listener - $ unik target --host localhost. Through that terminal I initiate the communication.

If my understanding is correct the initial problem, though, is still apparent - the connection between the daemon and instance listener cannot be established. The instance listener cannot track back to the daemon and I have no idea why. The daemon-config.yaml is local.

@miha-plesko

[ilackarms]

@antonOO the daemon needs to be on the same subnet as the instance listener. the instance listener uses udp broadcast on port 9967 to be discovered by the daemon. if udp broadcast does not work for whatever reason between the two nodes (instance listener and daemon) then you'll get the error that unik timed out waiting for instance listener

[shadjiiski]

Hi, @ilackarms. The usecase that we are trying to validate with @antonOO is managing a remote unikernel provider (vSphere in this case). This means that in general the daemon and the instance listener cannot be on the same subnet. Is there any other way to make the connection between the two nodes, e.g. providing IPs as command line arguments when executing the unik daemon command? Alternatively, is it possible to use only one of the nodes, e.g. sending API calls directly to the instance listener or having the daemon work with no instance listener?

[antonOO]

Hi @ilackarms , thank you for the response. The instance listener and daemon are on the same subnet, but I still get the error that unik timed out waiting for instance listener...

[ilevine]

@antonOO, Can I close this issue ?