APIcast hostname resolution issue

[estevaobk]

[APIcast doesn't resolve hostname without a TLD (.)]

Version

Containers from both APIcast v3.2.0 and v3.3.0 (possibly reproduced on earlier versions as well):

• registry.access.redhat.com/3scale-amp22/apicast-gateway
• registry.access.redhat.com/3scale-amp23/apicast-gateway

Steps To Reproduce

NOTE: The steps below are being reproduced using the Docker implementation of APIcast.

1. Create a new Internal Network:

$ docker network create --internal internal_network

2. Deploy an Apache (httpd) dummy container with the name "webhost" attached to this network:

$ docker container run --name webhost --network internal_network httpd

3. Create a new API calling "webhost" and using a self-managed APIcast:
   3.1. Private Base URL: http://webhost:80
   3.2. Staging and Production Public Base URL: http://localhost:8080
   3.3. Update the Staging Environment and Promote to Production.

4. Deploy an APIcast self-managed container based on any of the Versions described above (in the example below, 'registry.access.redhat.com/3scale-amp23/apicast-gateway:latest' is being used):

$ docker container run --name apicast -e THREESCALE_PORTAL_ENDPOINT=https://<token>@<user>-admin.3scale.net registry.access.redhat.com/3scale-amp23/apicast-gateway:latest

5. Attach the Internal Network to the container:

$ docker network connect internal_network apicast

6. Open a shell (bash) into the currently running apicast:

$ docker exec -it apicast bash

7. Curl the hostname "webhost" in order to ensure that the O.S. level name resolution is working:

$ curl -X GET -I http://webhost:80

HTTP/1.1 200 OK
Date: Fri, 19 Oct 2018 18:18:01 GMT
Server: Apache/2.4.34 (Unix)
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Content-Length: 45
Content-Type: text/html

8. Use the same command on APIcast:

$ curl -X GET -I http://localhost:8080?user_key=<user_key>

HTTP/1.1 503 Service Temporarily Unavailable
Server: openresty/1.13.6.2
Date: Fri, 19 Oct 2018 18:18:46 GMT
Content-Type: text/html
Content-Length: 219
Connection: keep-alive`

9. Observe the following lines on the APIcast Stdout/Stderr:

2018/10/19 18:19:31 [error] 34#34: *86 [lua] balancer.lua:41: get_peer(): could not select peer: empty peers while connecting to upstream, client: 127.0.0.1, server: _, request: "GET /?user_key=<user_key>
HTTP/1.1", host: "webhost"
[19/Oct/2018:18:19:33 +0000] localhost:8080 127.0.0.1:56488 "GET /?user_key=<user_key> HTTP/1.1" 503 219 (4.100) 2.1

10. The results above differs when using a full domain name with dots - e.g. "webhost.internal_network" as the Private Base URL:

$ curl -X GET -I http://webhost.internal_network

HTTP/1.1 200 OK
Date: Fri, 19 Oct 2018 18:23:11 GMT
Server: Apache/2.4.34 (Unix)
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Content-Length: 45
Content-Type: text/html

$ curl -X GET -I http://localhost:8080?user_key=<user_key>

HTTP/1.1 200 OK
Server: openresty/1.13.6.2
Date: Fri, 19 Oct 2018 18:24:03 GMT
Content-Type: text/html
Content-Length: 45
Connection: keep-alive
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes

[19/Oct/2018:18:24:03 +0000] localhost:8080 127.0.0.1:56610 "GET /?user_key=<user_key> HTTP/1.1" 200 45 (0.900) 0`

Current Result

APIcast doesn't resolve hostname, only hostname.domain.

Expected Result

APIcast should resolve both hostname and hostname.domain.

Additional Information

• Since the O.S. resolves both hostname and hosnamet.domain, we are able to isolate this issue to the APIcast name resolution.

• Adding the host in /etc/hosts doesn't solve the issue.

[mikz]

@estevaobk thank you! Great steps to reproduce.

Here is some additional research:

Using docker nameserver:

# /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0

;; QUESTION SECTION:
;webhost.internal_network.	IN	A

;; ANSWER SECTION:
webhost.internal_network. 600	IN	A	172.18.0.2

;; Query time: 1 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Mon Oct 22 09:24:14 UTC 2018
;; MSG SIZE  rcvd: 82

;; QUESTION SECTION:
;webhost.			IN	A

;; ANSWER SECTION:
webhost.		600	IN	A	172.18.0.2

;; Query time: 2 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Mon Oct 22 09:26:17 UTC 2018
;; MSG SIZE  rcvd: 48

And using our dnsmasq:

;; QUESTION SECTION:
;webhost.			IN	A

;; Query time: 1 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1)
;; WHEN: Mon Oct 22 09:26:41 UTC 2018
;; MSG SIZE  rcvd: 25

;; QUESTION SECTION:
;webhost.internal_network.	IN	A

;; ANSWER SECTION:
webhost.internal_network. 600	IN	A	172.18.0.2

;; Query time: 2 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1)
;; WHEN: Mon Oct 22 09:26:57 UTC 2018
;; MSG SIZE  rcvd: 82

We are starting dnsmasq with: --listen-address=127.0.0.1 --port=5353 --all-servers --no-host --no-hosts --cache-size=1000 --no-negcache --domain-needed to not resolve naked domains.

That was done in b58bab8. We need to verify if that is still needed.

[mikz]

@estevaobk as a workaround you can set proper search scope in /etc/resolv.conf like:

search internal
nameserver 127.0.0.11
options ndots:0

In plain docker that is possible by using --dns-search flag when starting the contianer.

[ghost]

@mikz That didn't work for me. I've added the nginx 'resolver' directive to conf/nginx.conf (127.0.0.11) and now my Docker hostnames are properly resolved. We are still on v3.0.0. Maybe my issue is not the same as above. Docker hostnames were not resolved by Lua. Similar to this issue: openresty/lua-resty-redis#59 Setting the RESOLVER env var to 127.0.0.11:53 did not work.

@mikz I've also found your comment here: nmasse-itix/apicast-logger#2 (comment) Sorry for the confusion. Your answer was meant for newer versions.

[mikz]

@ddevaal v3.0.0 is significantly different from now. The logic of resolving changed several times since then.

[akostadinov]

Yes, this documentation issue is related #1373