[THREESCALE-9301] Fix dns cache miss

[tkan145]

What

Fix https://issues.redhat.com/browse/THREESCALE-9301

Notes

Before

Description	qname	queries	answer	cached key
shortname	backend-listener	backend-listener. backend-listener.default.svc.cluster.loca backend-listener.svc.cluster.local backend-listener.cluster.local	name: backend-listener.default.svc.cluster.local address: 10.7.0.1 type: A	backend-listener.default.svc.cluster.local
FQDN	backend-listener.default.svc.cluster.local	backend-listener.default.svc.cluster.local. backend-listener.default.svc.cluster.local.default.svc.cluster.local backend-listener.default.svc.cluster.local.svc.cluster.local backend-listener.default.svc.cluster.local.cluster.local	name: backend-listener.default.svc.cluster.local. address:10.7.0.1 type: A	backend-listener.default.svc.cluster.local.
FQDN	foo.example.com	foo.example.com. foo.example.com.default.svc.cluster.local foo.example.svc.cluster.local foo.example.cluster.local	name: foo.example.com. address: 10.7.0.1	foo.example.com.
Absolute FQDN	foo.example.com.	foo.example.com.	name: foo.example.com. address: 10.7.0.1	foo.example.com.

Now

Description	qname	queries	answer	cached key
shortname	backend-listener	backend-listener. backend-listener.default.svc.cluster.loca backend-listener.svc.cluster.local backend-listener.cluster.local	name: backend-listener.default.svc.cluster.local address: 10.7.0.1 type: A	backend-listener:1
FQDN	backend-listener.default.svc.cluster.local	backend-listener.default.svc.cluster.local. backend-listener.default.svc.cluster.local.default.svc.cluster.local backend-listener.default.svc.cluster.local.svc.cluster.local backend-listener.default.svc.cluster.local.cluster.local	name: backend-listener.default.svc.cluster.local. address:10.7.0.1 type: A	backend-listener.default.svc.cluster.local:1
FQDN	foo.example.com	foo.example.com. foo.example.com.default.svc.cluster.local foo.example.svc.cluster.local foo.example.cluster.local	name: foo.example.com. address: 10.7.0.1	foo.example.com:1
Absolute FQDN	foo.example.com.	foo.example.com.	name: foo.example.com. address: 10.7.0.1	foo.example.com.:1

Verification steps

• Checkout master branch
• Start development environment

make development

• Create a apicast-config.json file with the following content

cat <<EOF >apicast-config.json
{
    "services": [
        {
            "id": "1",
            "backend_version": "1",
            "proxy": {
                "hosts": [
                    "one"
                ],
                "api_backend": "https://echo-api.3scale.net:443",
                "backend": {
                    "endpoint": "http://127.0.0.1:8081",
                    "host": "backend"
                },
                "policy_chain": [
                    {
                        "name": "apicast.policy.apicast"
                    }
                ],
                "proxy_rules": [
                    {
                        "http_method": "GET",
                        "pattern": "/",
                        "metric_system_name": "hits",
                        "delta": 1,
                        "parameters": [],
                        "querystring_parameters": {}
                    }
                ]
            }
        }
    ]
} 
EOF

• Start APIcast

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0 THREESCALE_CONFIG_FILE=apicast-config.json ./bin/apicast

• Capture APIcast IP

APICAST_IP=$(docker inspect apicast_build_0-development-1 | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)

• Send request

curl -i -k -H "Host: one" -H "Accept: application/json" "http://${APICAST_IP}:8080/?user_key="

• Send another request

curl -i -k -H "Host: one" -H "Accept: application/json" "http://${APICAST_IP}:8080/?user_key="

• Check the log, you should see something like the follow

22 resolver.lua:350: lookup(): resolver query: echo-api.3scale.net, requestID=41877adab1e263e2d68f62c699a4aea2                                          
22 cache.lua:115: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries, requestID=41877adab1e263e2d68f62c699a4aea2                        
22 cache.lua:115: fetch_answers(): resolver cache read echo-api.pro-saas.3sca.net 3 entries, requestID=41877adab1e263e2d68f62c699a4aea2                 
22 cache.lua:190: get(): resolver cache hit: echo-api.3scale.net 3.213.166.152, 54.157.201.86, 52.72.133.221, requestID=41877adab1e263e2d68f62c699a4aea2
22 resolver.lua:370: lookup(): resolver query: echo-api.3scale.net finished with 4 answers, requestID=41877adab1e263e2d68f62c699a4aea2                  

• Stop APIcast

CTRL-C

• Checkout this branch
• Start APIcast again

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0 THREESCALE_CONFIG_FILE=apicast-config.json ./bin/apicast

• Send 2 requests
• Check the log, it should use key with <qname:qtype> format

cache.lua:130: fetch_answers(): resolver cache read echo-api.3scale.net:1 3 entries, requestID=a48f515c452e71c0fafb79c1f75d0205                        
cache.lua:205: get(): resolver cache hit: echo-api.3scale.net:1 3.213.166.152, 54.157.201.86, 52.72.133.221, requestID=a48f515c452e71c0fafb79c1f75d0205
resolver.lua:405: get_servers(): resolver query: echo-api.3scale.net finished with 3 answers, requestID=a48f515c452e71c0fafb79c1f75d0205               

[eguzki]

LGTM

verfification steps working

There was some gdoc explaining the DNS resolution process in APIcast. Not for this PR, but we want some content of it in README of this repo. Whenever you have some time.