Performance issues with deadwood

[Sam Trenholme]

> The result is rather disappointing, the mean value over several runs
> is around
>
>  8 min for deadwood and
>  1 min for unbound.

At this point in the development cycle, the important thing is to get
Deadwood to work and resolve all the hostnames we can out there.  I
am, right now, more interested in seeing what hostnames Deadwood can
not resolve that Unbound and other DNS servers can resolve.
Optimizing Deadwood for speed with an empty cache definitely matters,
but I’m a little uncomfortable optimizing Deadwood for speed at this
point in the development cycle of MaraDNS 2.0.

There are a few things to think about if Deadwood speed is important:

* Increase maxprocs to 128 or 256:

maxprocs = 256

* Increase the cache size from 1024 elements to 65536 elements:

maximum_cache_elements = 65536

* Disable in-flight merging.  This may be a little less secure, and
will make Deadwood strain other DNS servers on the net more, but may
help speed up Deadwood:

max_inflights = 1

* It may or may not speed things up to decrease timeout_seconds to 1,
its minimum possible value:

timeout_seconds = 1

> If I've repeated the test _without_ restarting deadwood, we have less
> timeouts in the second run, the third run is finally clean.

One thing to keep in mind is that Deadwood is capable of storing the
cache between Deadwood sessions, so Deadwood is usually not “started
cold”.  To do this:

cd /etc/deadwood
mkdir cache
chown 99:99 cache

And then add this to dwood3rc:

cache_file = "cache/dw_cache"

Please be sure to remove the persistent cache file if making changes
to Deadwood’s configuration.

> deadwood has problems resolving some host names immediately.

I have been seeing similar things.  Here are some changes in the code
that may speed things up:

* Always choose a name server that isn’t glueless, if one is available.

* If a given name server gives us a reply we can’t use, always
immediately contact the next nameserver.  There are places in
Deadwood’s code when we wait timeout_seconds after getting a reply and
before sending the next query (such as if we get an empty DNS packet
marked “server fail”).

* We may speed things up by accepting a direct answer to our question
in the AR section of a reply.  This kind of thing has to be done with
some care, in order to ensure that it doesn’t make us more susceptible
to spoofing attacks.

* Have Deadwood use sub-second timestamps.  Right now, Deadwood’s
timestamps allow a single second to be divided in to 256 “ticks”

> hope this helps and best regards

It does help.  I have been getting the general sense that Deadwood
seems a bit slow with an empty cache, but Deadwood seems generally
fine once entries are in the cache.  Again, Deadwood has a persistent
cache, so the slowdown in real-world use will not be so marked.

Actually, It’s very encouraging to see the issues we’re having are
speed issues and not crash issues or memory leak issues, or even “this
host does not resolve” issues.  Around this time with MaraDNS 1.0’s
development, Franky and myself were still hunting down and plugging
memory leaks.  Indeed, “this host does not resolve” (as well as
“MaraDNS uses too many resources on my 486” issues) have always
plagued MaraDNS 1.0’s recursive code.

The sense I am getting is that Deadwood is quite solid at this point.
In terms of web surfing, the issues I’m seeing with hostnames that
don’t resolve at all are issues like “This broken DNS server sent us
an empty packet; should we interpret that as meaning the hostname
doesn’t exist, or as meaning this DNS server is broken and to contact
the next one”.

The plan I have right now is to do some minor tweaking with Deadwood
to try and speed things up; it’s trivial to add code to give Deadwood
more preference for NS records that are not glueless, for example.  I
will also consider cases where Deadwood doesn’t quickly contact
another nameserver once she gets a reply, but instead waits
timeout_seconds before contacting the next nameserver, as being bugs.

More significant changes, such as having fraction-of-a-second
timestamps and timeouts or accepting IPs for questions in the AR
section of a reply, will have to be deferred until after Deadwood
3.0/MaraDNS 2.0.  Quite frankly, I will have to look at my work and
economic situation before making any plans to do significant
development after I get MaraDNS 2.0 with Deadwood 3.0 out the door.

Again, thank you very much for your bug report.  It gives me an idea
of where to focus Deadwood development.  Now, let me get back to a
paid gig I have moving a friend’s blog from b2evolution to Wordpress.

- Sam