From strenholme.usenet at gmail.com Fri Sep 3 11:46:41 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Fri, 3 Sep 2010 08:46:41 -0700 Subject: Monthly posting: MaraDNS progress report; I'm still looking for work Message-ID: I have just released Deadwood 2.9.06 today. It can be downloaded in source code form and for Windows here: http://maradns.org/deadwood/testing/ I also want everyone to know that, while my involvement with MaraDNS has given me a handful of phone screens, as well as two face-to-face interviews, I haven?t gotten any offers yet and am still available for work. I really would like a job that allows me to remotely telecommute from Mexico about a week every month (yes, there is high-speed internet down there) so I can spend time with my wife until she has permission to come to the United States?a process that might take a year or longer. My resume is available here: http://samiam.org/resume/ People here know exactly how well I can code; I wrote MaraDNS (as well as Deadwood, MaraDNS 2.0?s recursive resolver) after all. If anyone has any potential leads, please contact me via private email. This is a monthly posting; if I do not find a job in September, I will post an announcement in early October. - Sam From juergen.daubert at t-online.de Mon Sep 6 08:41:43 2010 From: juergen.daubert at t-online.de (Juergen Daubert) Date: Mon, 6 Sep 2010 14:41:43 +0200 Subject: Performance issues with deadwood Message-ID: <20100906124143.GA30816@jue.netz> Hi Sam, reading in your blog that some test with deadwood might be helpful, I took the opportunity to do a simple comparison with unbound [1], IMHO the best recursive resolver currently available. The test I've done is very simple, it just records the time to resolve around 100 host-names, see [2] for the list. Using this list I've done the following after a fresh restart of the deadwood resp. unbound resolver: $:> time $(for h in $(cat hosts.txt); do printf "\n\n\n"; dig $h; done > test.log) The result is rather disappointing, the mean value over several runs is around 8 min for deadwood and 1 min for unbound. Note that there are several out-timed queries for deadwood but not for unbound. See [3] for the log of deadwood, [4] the one of unbound. If I've repeated the test _without_ restarting deadwood, we have less timeouts in the second run, the third run is finally clean. Unbound is not optimized for performance it has all kind of dnssec stuff enabled, without the query time is around 12 sec lower. Some system infos: CRUX linux system with glibc 2.10.1, deadwood compiled with gcc 4.4.4 without ipv6 support. Deadwood 2.9.06 and H-20100905-2 without any noticeable difference. Unbound 1.4.6. Deadwood config: $:> grep -Ev '^#|^$' /etc/deadwoodrc bind_address="127.0.0.2" chroot_dir = "/var/lib/deadwood" maximum_cache_elements = 10000 recursive_acl = "127.0.0.1/16" The above is for sure not very scientifically, but it represents the feeling I have while browsing with the two resolvers, in respect of performance and timeouts, and shows that deadwood has problems resolving some host names immediately. hope this helps and best regards Juergen [1] http://www.unbound.net/ [2] http://jue.li/tmp/deadwood/hosts.txt [3] http://jue.li/tmp/deadwood/deadwood-H-20100905-2.log [4] http://jue.li/tmp/deadwood/unbound-1.4.6.log -- Juergen Daubert | mailto:jue at jue.li Korb, Germany | http://jue.li/crux From strenholme.usenet at gmail.com Mon Sep 6 11:44:52 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Mon, 6 Sep 2010 08:44:52 -0700 Subject: Performance issues with deadwood In-Reply-To: <20100906124143.GA30816@jue.netz> References: <20100906124143.GA30816@jue.netz> Message-ID: > The result is rather disappointing, the mean value over several runs > is around > > ?8 min for deadwood and > ?1 min for unbound. At this point in the development cycle, the important thing is to get Deadwood to work and resolve all the hostnames we can out there. I am, right now, more interested in seeing what hostnames Deadwood can not resolve that Unbound and other DNS servers can resolve. Optimizing Deadwood for speed with an empty cache definitely matters, but I?m a little uncomfortable optimizing Deadwood for speed at this point in the development cycle of MaraDNS 2.0. There are a few things to think about if Deadwood speed is important: * Increase maxprocs to 128 or 256: maxprocs = 256 * Increase the cache size from 1024 elements to 65536 elements: maximum_cache_elements = 65536 * Disable in-flight merging. This may be a little less secure, and will make Deadwood strain other DNS servers on the net more, but may help speed up Deadwood: max_inflights = 1 * It may or may not speed things up to decrease timeout_seconds to 1, its minimum possible value: timeout_seconds = 1 > If I've repeated the test _without_ restarting deadwood, we have less > timeouts in the second run, the third run is finally clean. One thing to keep in mind is that Deadwood is capable of storing the cache between Deadwood sessions, so Deadwood is usually not ?started cold?. To do this: cd /etc/deadwood mkdir cache chown 99:99 cache And then add this to dwood3rc: cache_file = "cache/dw_cache" Please be sure to remove the persistent cache file if making changes to Deadwood?s configuration. > deadwood has problems resolving some host names immediately. I have been seeing similar things. Here are some changes in the code that may speed things up: * Always choose a name server that isn?t glueless, if one is available. * If a given name server gives us a reply we can?t use, always immediately contact the next nameserver. There are places in Deadwood?s code when we wait timeout_seconds after getting a reply and before sending the next query (such as if we get an empty DNS packet marked ?server fail?). * We may speed things up by accepting a direct answer to our question in the AR section of a reply. This kind of thing has to be done with some care, in order to ensure that it doesn?t make us more susceptible to spoofing attacks. * Have Deadwood use sub-second timestamps. Right now, Deadwood?s timestamps allow a single second to be divided in to 256 ?ticks? > hope this helps and best regards It does help. I have been getting the general sense that Deadwood seems a bit slow with an empty cache, but Deadwood seems generally fine once entries are in the cache. Again, Deadwood has a persistent cache, so the slowdown in real-world use will not be so marked. Actually, It?s very encouraging to see the issues we?re having are speed issues and not crash issues or memory leak issues, or even ?this host does not resolve? issues. Around this time with MaraDNS 1.0?s development, Franky and myself were still hunting down and plugging memory leaks. Indeed, ?this host does not resolve? (as well as ?MaraDNS uses too many resources on my 486? issues) have always plagued MaraDNS 1.0?s recursive code. The sense I am getting is that Deadwood is quite solid at this point. In terms of web surfing, the issues I?m seeing with hostnames that don?t resolve at all are issues like ?This broken DNS server sent us an empty packet; should we interpret that as meaning the hostname doesn?t exist, or as meaning this DNS server is broken and to contact the next one?. The plan I have right now is to do some minor tweaking with Deadwood to try and speed things up; it?s trivial to add code to give Deadwood more preference for NS records that are not glueless, for example. I will also consider cases where Deadwood doesn?t quickly contact another nameserver once she gets a reply, but instead waits timeout_seconds before contacting the next nameserver, as being bugs. More significant changes, such as having fraction-of-a-second timestamps and timeouts or accepting IPs for questions in the AR section of a reply, will have to be deferred until after Deadwood 3.0/MaraDNS 2.0. Quite frankly, I will have to look at my work and economic situation before making any plans to do significant development after I get MaraDNS 2.0 with Deadwood 3.0 out the door. Again, thank you very much for your bug report. It gives me an idea of where to focus Deadwood development. Now, let me get back to a paid gig I have moving a friend?s blog from b2evolution to Wordpress. - Sam From wayne.kroncke at tiscali.co.uk Mon Sep 6 14:55:18 2010 From: wayne.kroncke at tiscali.co.uk (wayne at tiscali) Date: Mon, 06 Sep 2010 19:55:18 +0100 Subject: Performance issues with deadwood In-Reply-To: References: <20100906124143.GA30816@jue.netz> Message-ID: <4C853916.5010100@tiscali.co.uk> hi sam, & the list. i'm using the latest deadwood on my win7 ultimate x64 pc as a client side cache rather than as a lan dns server, and have over the last few weeks noticed that the uncached response times are quite large, been meaning to mention this here, but have been considering how to phrase it all. i'm glad that i am not the only one who has noticed, but am also glad it will be addressed. i have been using a freeware windows based dns test program from gibson research co. at http://www.grc.com/dns/benchmark.htm which tests a given list of public (and private) dns servers for cached and uncached performance. it also tests your pc's dns resolvers as set in the adapter settings for tcp/ipv4, allowing you to pick the fastest servers to adjust your pc's settings accordingly. deadwood always comes out top on cached performance, with it not being measureable at 0.00 sec, but it's uncached response time is terrible, can be as high as 3 secs, similar to as noted below, with time getting worse if you fiddle with increasing the timeout. with the same upsteam servers set in another caching dns server working out about 0.094 for the uncached response (and also 0.00 cached). i've now tweaked deadwood's config as suggested in the earlier message which has helped a bit, but uncached response times are still 1.76 secs. how this test would relate to deadwood running on other OS's i am not sure, but it may be usefull as a rough guide for us windows users. best regards, wayne kroncke On 06/09/2010 16:44, Sam Trenholme wrote: >> The result is rather disappointing, the mean value over several runs >> is around >> >> 8 min for deadwood and >> 1 min for unbound. > At this point in the development cycle, the important thing is to get > Deadwood to work and resolve all the hostnames we can out there. I > am, right now, more interested in seeing what hostnames Deadwood can > not resolve that Unbound and other DNS servers can resolve. > Optimizing Deadwood for speed with an empty cache definitely matters, > but I?m a little uncomfortable optimizing Deadwood for speed at this > point in the development cycle of MaraDNS 2.0. > > There are a few things to think about if Deadwood speed is important: > > * Increase maxprocs to 128 or 256: > > maxprocs = 256 > > * Increase the cache size from 1024 elements to 65536 elements: > > maximum_cache_elements = 65536 > > * Disable in-flight merging. This may be a little less secure, and > will make Deadwood strain other DNS servers on the net more, but may > help speed up Deadwood: > > max_inflights = 1 > > * It may or may not speed things up to decrease timeout_seconds to 1, > its minimum possible value: > > timeout_seconds = 1 > >> If I've repeated the test _without_ restarting deadwood, we have less >> timeouts in the second run, the third run is finally clean. > One thing to keep in mind is that Deadwood is capable of storing the > cache between Deadwood sessions, so Deadwood is usually not ?started > cold?. To do this: > > cd /etc/deadwood > mkdir cache > chown 99:99 cache > > And then add this to dwood3rc: > > cache_file = "cache/dw_cache" > > Please be sure to remove the persistent cache file if making changes > to Deadwood?s configuration. > >> deadwood has problems resolving some host names immediately. > I have been seeing similar things. Here are some changes in the code > that may speed things up: > > * Always choose a name server that isn?t glueless, if one is available. > > * If a given name server gives us a reply we can?t use, always > immediately contact the next nameserver. There are places in > Deadwood?s code when we wait timeout_seconds after getting a reply and > before sending the next query (such as if we get an empty DNS packet > marked ?server fail?). > > * We may speed things up by accepting a direct answer to our question > in the AR section of a reply. This kind of thing has to be done with > some care, in order to ensure that it doesn?t make us more susceptible > to spoofing attacks. > > * Have Deadwood use sub-second timestamps. Right now, Deadwood?s > timestamps allow a single second to be divided in to 256 ?ticks? > >> hope this helps and best regards > It does help. I have been getting the general sense that Deadwood > seems a bit slow with an empty cache, but Deadwood seems generally > fine once entries are in the cache. Again, Deadwood has a persistent > cache, so the slowdown in real-world use will not be so marked. > > Actually, It?s very encouraging to see the issues we?re having are > speed issues and not crash issues or memory leak issues, or even ?this > host does not resolve? issues. Around this time with MaraDNS 1.0?s > development, Franky and myself were still hunting down and plugging > memory leaks. Indeed, ?this host does not resolve? (as well as > ?MaraDNS uses too many resources on my 486? issues) have always > plagued MaraDNS 1.0?s recursive code. > > The sense I am getting is that Deadwood is quite solid at this point. > In terms of web surfing, the issues I?m seeing with hostnames that > don?t resolve at all are issues like ?This broken DNS server sent us > an empty packet; should we interpret that as meaning the hostname > doesn?t exist, or as meaning this DNS server is broken and to contact > the next one?. > > The plan I have right now is to do some minor tweaking with Deadwood > to try and speed things up; it?s trivial to add code to give Deadwood > more preference for NS records that are not glueless, for example. I > will also consider cases where Deadwood doesn?t quickly contact > another nameserver once she gets a reply, but instead waits > timeout_seconds before contacting the next nameserver, as being bugs. > > More significant changes, such as having fraction-of-a-second > timestamps and timeouts or accepting IPs for questions in the AR > section of a reply, will have to be deferred until after Deadwood > 3.0/MaraDNS 2.0. Quite frankly, I will have to look at my work and > economic situation before making any plans to do significant > development after I get MaraDNS 2.0 with Deadwood 3.0 out the door. > > Again, thank you very much for your bug report. It gives me an idea > of where to focus Deadwood development. Now, let me get back to a > paid gig I have moving a friend?s blog from b2evolution to Wordpress. > > - Sam From strenholme.usenet at gmail.com Mon Sep 6 21:00:57 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Mon, 6 Sep 2010 18:00:57 -0700 Subject: Performance issues with deadwood In-Reply-To: <4C853916.5010100@tiscali.co.uk> References: <20100906124143.GA30816@jue.netz> <4C853916.5010100@tiscali.co.uk> Message-ID: >It may be useful as a rough guide for us windows users. For the record, I do appreciate people piping up on the list about Deadwood?s issues. As it turns out, the main platform I test and use Deadwood on is Windows, so anything that affects how it runs in Windows is something I?ll notice right away. Indeed, the fact that the slowdown issues affects how things run in Windows means I can more easily test tuning things to try and reduce the slowdown. Again, the fact is that I?ve been concentrating a lot more on getting everything out there to resolve because the big issue with MaraDNS 1's resolver which was never fully solved was hosts not resolving. I call Deadwood a success because people aren?t seeing Deadwood crash, people aren?t finding memory leaks, and people aren?t finding hosts Deadwood can?t resolve. - Sam Note: I do not answer MaraDNS (including Deadwood) support requests sent by private email without being compensated for my time. A MaraDNS support request is any and all discussion you may wish to have about MaraDNS in private email; if you want to email me to talk about MaraDNS then, yes, that is a support request. I will discuss rates if you want this kind of support. Thank you for your understanding. MaraDNS security vulnerability reports, however, will be dealt with without charge and kept confidential. If you don't know what Bugtraq is, then, no, your email is not a security report. It is not a security report unless you've done due diligence to determine how the security bug you think you found can reasonably be exploited. From strenholme.usenet at gmail.com Mon Sep 6 23:47:07 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Mon, 6 Sep 2010 20:47:07 -0700 Subject: Performance issues with deadwood In-Reply-To: <20100906124143.GA30816@jue.netz> References: <20100906124143.GA30816@jue.netz> Message-ID: > $:> time $(for h in $(cat hosts.txt); do printf "\n\n\n"; dig $h; done > test.log) > > The result is rather disappointing, the mean value over several runs > is around > > ?8 min for deadwood and > ?1 min for unbound. OK, I?ve found and fixed a bug in Deadwood which accounted for about two thirds of the slowdown compared to unbound. I just did the same test you did, and this the timings I saw: * 7 minutes 29 seconds without the bug patched (about the same you saw, with a fairly large cache in place) * 2 minutes 57 seconds with the bug patched (from a ?cold start?: No cache) Now, the reason I?m getting slower numbers than you may be caused by me having a slower internet connection than you. Or, it may be caused because there are other bugs slowing down resolution which I still need to stamp out. Anyway, deadwood-H-20100906-2.tar.bz2 at http://maradns.org/deadwood/snap/ has the bug patched. In addition, I will add the patch after my rather long sig. Thank you for the bug report and let me know how much of a speedup you see. - Sam Note: I do not answer MaraDNS (including Deadwood) support requests sent by private email without being compensated for my time. A MaraDNS support request is any and all discussion you may wish to have about MaraDNS in private email; if you want to email me to talk about MaraDNS then, yes, that is a support request. I will discuss rates if you want this kind of support. Thank you for your understanding. MaraDNS security vulnerability reports, however, will be dealt with without charge and kept confidential. If you don't know what Bugtraq is, then, no, your email is not a security report. It is not a security report unless you've done due diligence to determine how the security bug you think you found can reasonably be exploited. --- deadwood-H-20100906-1/src/DwUdpSocket.c.orig 2010-09-06 20:01:26.000000000 -0700 +++ deadwood-H-20100906-1/src/DwUdpSocket.c 2010-09-06 20:03:49.000000000 -0700 @@ -179,8 +179,9 @@ dw_str *query, SOCKET x_socket_num) { sockaddr_all_T server; SOCKET s = 0; - int32_t rnum = -1; + int_fast32_t rnum = -1; ip_addr_T addr = {0,{0,0},0,0}; + int counter = 0; socklen_t inet_len = sizeof(struct sockaddr_in); if(rem[n].socket != x_socket_num || /* Already used (sanity check) */ @@ -195,7 +196,12 @@ } /* Get IP of remote server and open a socket to them */ - addr = get_upstream_ip(query,n); + for(counter = 0; counter < 7; counter++) { /* Don't stop on bad IP */ + addr = get_upstream_ip(query,n); + if(addr.len > 0) { + break; + } + } dw_log_ip("Making connection to IP",&addr,128); if(addr.len == 0) { /* Failed to get upstream IP */ return; From juergen.daubert at t-online.de Tue Sep 7 06:56:00 2010 From: juergen.daubert at t-online.de (Juergen Daubert) Date: Tue, 7 Sep 2010 12:56:00 +0200 Subject: Performance issues with deadwood In-Reply-To: References: <20100906124143.GA30816@jue.netz> Message-ID: <20100907105600.GA29763@jue.netz> On Mon, Sep 06, 2010 at 08:47:07PM -0700, Sam Trenholme wrote: > > $:> time $(for h in $(cat hosts.txt); do printf "\n\n\n"; dig $h; done > test.log) > > > > The result is rather disappointing, the mean value over several runs > > is around > > > > ?8 min for deadwood and > > ?1 min for unbound. > > OK, I?ve found and fixed a bug in Deadwood which accounted for about > two thirds of the slowdown compared to unbound. > > I just did the same test you did, and this the timings I saw: > > * 7 minutes 29 seconds without the bug patched (about the same you > saw, with a fairly large cache in place) > > * 2 minutes 57 seconds with the bug patched (from a ?cold start?: No cache) Yeah, it's much better now :) I've done 5 measurements after a cold start and got values in the range of 1 min 40 sec and 2 min 49 sec with a mean value of 2 min 15 sec. We have fewer time-outs as well now, range between 0 and 5, 1.6 mean. -- Juergen Daubert | mailto:jue at jue.li Korb, Germany | http://jue.li/crux From wayne.kroncke at tiscali.co.uk Tue Sep 7 07:34:22 2010 From: wayne.kroncke at tiscali.co.uk (wayne at tiscali) Date: Tue, 07 Sep 2010 12:34:22 +0100 Subject: Performance issues with deadwood In-Reply-To: <20100907105600.GA29763@jue.netz> References: <20100906124143.GA30816@jue.netz> <20100907105600.GA29763@jue.netz> Message-ID: <4C86233E.4030207@tiscali.co.uk> hi y'all; sounds good, looking forward to the windows version being updated to reflect the fix. best regards, wayne kroncke On 07/09/2010 11:56, Juergen Daubert wrote: > > Yeah, it's much better now :) > > I've done 5 measurements after a cold start and got values in the range > of 1 min 40 sec and 2 min 49 sec with a mean value of 2 min 15 sec. > We have fewer time-outs as well now, range between 0 and 5, 1.6 mean. > > From strenholme.usenet at gmail.com Wed Sep 8 03:29:04 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Wed, 8 Sep 2010 00:29:04 -0700 Subject: Performance issues with deadwood In-Reply-To: <4C853916.5010100@tiscali.co.uk> References: <20100906124143.GA30816@jue.netz> <4C853916.5010100@tiscali.co.uk> Message-ID: > i have been using a freeware windows based dns test program from gibson > research co. at http://www.grc.com/dns/benchmark.htm which tests a given > list of public (and private) dns servers for cached and uncached > performance. Thank you for the pointer. This program has helped me find some bugs in Deadwood. It is my conclusion that, while this program has been useful for finding a bug in Deadwood, it is not a reliable tool for testing a DNS server running on one?s local connection. The reason why this tool was reporting slow replies from Deadwood was because of a bug in Deadwood (with hostnames and case sensitivity; a case which does not affect things with most real-world queries but delayed the responses Gibson?s program makes by timeout_seconds) which I have fixed. Once I patched the bug, while Gibson?s DNS benchmark tool reports much higher speeds with Deadwood, it also *incorrectly* reports that Deadwood drops between 80 and 95% of DNS packets, which plain simply is not true. Since these numbers are so distressing, which I did was carefully look through Deadwood?s code to see if something was amiss. I didn?t find anything, and just verified that Deadwood is perfectly responsive while being subjected to Gibson?s test, and that Deadwood replies to almost all queries sent to it by Gibson?s test. One thing that was *very* frustrating for me dealing with Gibson?s tool is that it doesn?t give me enough information to troubleshoot its error reports. It doesn?t tell me which particular DNS packets were ?dropped?, so I was pulling my hair out all afternoon making damn sure the problem was Gibson?s broken tool and not Deadwood. My theory?again, Gibson?s tool is a black box, so this is a guess?is that Gibson?s test looks at the average reply for a cache, and considers any reply more than N standard deviations from that average reply time to be a dropped packet. As it turns out, Unbound run on my system against this test reports 5% to 10% of packets being dropped, and the older buggy versions of Deadwood (which delay replies with the wrong case by timeout_seconds) are reported as having no dropped packets with Gibson?s tool. I would guess that Gibson?s tool is optimized to benchmark queries from a recursive DNS server with a high-speed internet connection, where the round trip time to do a recursive DNS query is a fraction of the response time of a home internet connection. It is obvious, based on how the tool incorrectly states that Deadwood drops most packets, that the tool can not handle very well home network connections with a more variable response time. Gibson also has an interesting DNS spoof-ability test which I used to verify that recurse_min_bind_port and recurse_number_ports work correctly in Deadwood; however the page at https://www.grc.com/dns/gallery.htm claims there is something wrong with MaraDNS?s default settings since I only use 28 instead of 32 bits of entropy for the query ID + source port?MaraDNS? default settings allow people to use MaraDNS with random source ports without completely removing one?s UDP firewall and allowing one to readily run other UDP services on the same machine. MaraDNS was immune from the Kaminsky attack as far back as 2001 because MaraDNS (and Deadwood) have never allowed packets in the AR section of a reply in to their cache. If you ask for abwjde.paypal.com in an attempt to poison www.paypal.com with an AR packet, a successful spoof will only spoof abwjde.paypal.com?*not* www.paypal.com. One would have to send on average over 100 million packets that would need to arrive *before* paypal.com?s legitimate DNS packet to successfully spoof MaraDNS or Deadwood. This is how it was in 2001; this is how it is today in 2010. So, in conclusion, Gibson?s tools are useful, but after spending all day using them and looking over Deadwood?s replies to it with a microscope, the tool is just not a very good tool for testing a recursive resolver on a home internet connection. - Sam From strenholme.usenet at gmail.com Wed Sep 8 03:58:01 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Wed, 8 Sep 2010 00:58:01 -0700 Subject: Performance issues with deadwood In-Reply-To: <4C86233E.4030207@tiscali.co.uk> References: <20100906124143.GA30816@jue.netz> <20100907105600.GA29763@jue.netz> <4C86233E.4030207@tiscali.co.uk> Message-ID: > sounds good, looking forward to the windows version being updated to reflect > the fix. http://maradns.org/deadwood/snap/Deadwood-H-20100908-win32.zip Now, please note: * Gibson?s tool reports this being a lot faster than the last Deadwood * Gibson?s tool reports that this release of Deadwood is unreliable. Gibson?s tool is lying. I?m aware of the issue and my official reply is ?Bug Steve Gibson and tell him to write better software?. :) (It is good software, it?s just not the best tool for testing a program like Deadwood being run on a home network) Everyone else: The cool patch that further randomizes the hash by adding a random constant broke other code, so has been withdrawn. I know what the issue is and have updated the patch to mostly fix the issue, but it doesn?t work. So, I?m withdrawing the patch for now. And, oh, hash_magic_number is broken for the same reasons. One idea is to remove the hash_magic_number parameter, and add some code that will, *before* reading /etc/dwood3rc (we have to do it before because the hash is used to parse a mararc file), read /dev/urandom (?secret.txt? on Windows machines) and get entropy from a combination of that and the timestamp when Deadwood is started for the additive constant. - Sam Note: I am still available and am looking for work. I appreciate all of the people who have given me leads via private email. Note also: If you want MaraDNS help or to report a MaraDNS bug, send it to the list, not to my private email unless you?re willing to pay me. Thank you for your understanding. From wayne.kroncke at tiscali.co.uk Wed Sep 8 05:00:22 2010 From: wayne.kroncke at tiscali.co.uk (wayne at tiscali) Date: Wed, 08 Sep 2010 10:00:22 +0100 Subject: Performance issues with deadwood In-Reply-To: References: <20100906124143.GA30816@jue.netz> <20100907105600.GA29763@jue.netz> <4C86233E.4030207@tiscali.co.uk> Message-ID: <4C8750A6.6000601@tiscali.co.uk> hi sam, thanx for that. i've been taking gibson's tests with a grain of salt anyway as it's still a work in progress & i think of it as at best a beta & the docs are unfinished :). it is fun to watch even tho as you note it comes up with some funny results on dns sites i know are good reliable ones. best to make a few runs and see the differences. not aware of a similar dns benchmark for windows tho. :) best regards, wayne On 08/09/2010 08:58, Sam Trenholme wrote: >> sounds good, looking forward to the windows version being updated to reflect >> the fix. > http://maradns.org/deadwood/snap/Deadwood-H-20100908-win32.zip From sysadmin at mrgnetwork.com.br Wed Sep 8 11:35:54 2010 From: sysadmin at mrgnetwork.com.br (Marlon) Date: Wed, 08 Sep 2010 12:35:54 -0300 Subject: Performance issues with deadwood In-Reply-To: (sfid-H20100907-003401-+039.94-1@spamfilter.osbf.lua) References: <20100906124143.GA30816@jue.netz> (sfid-H20100907-003401-+039.94-1@spamfilter.osbf.lua) Message-ID: <4C87AD5A.9010702@mrgnetwork.com.br> Guys, > OK, I?ve found and fixed a bug in Deadwood which accounted for about > two thirds of the slowdown compared to unbound. > > I just did the same test you did, and this the timings I saw: > > * 7 minutes 29 seconds without the bug patched (about the same you > saw, with a fairly large cache in place) > > * 2 minutes 57 seconds with the bug patched (from a ?cold start?: No cache) > > Now, the reason I?m getting slower numbers than you may be caused by > me having a slower internet connection than you. Or, it may be caused > because there are other bugs slowing down resolution which I still > need to stamp out. > > Anyway, deadwood-H-20100906-2.tar.bz2 at > http://maradns.org/deadwood/snap/ has the bug patched. In addition, I > will add the patch after my rather long sig. > > Thank you for the bug report and let me know how much of a speedup you see. > > - Sam > I compiled deadwood-H-20100906-2.tar.bz2 with ICC instead GCC on a Mac mini dual core (running Linux) with 1 GB of ram and ran the test: time $(for h in $(cat hosts.txt); do printf "\n\n\n"; dig $h; done > test.log) and got: Deadwood: real 2m26.541s user 0m0.120s sys 0m0.020s I ran the test using another DNS recursors too: PowerDNS recursor: real 1m49.019s user 0m0.109s sys 0m0.030s Unbound: real 1m17.721s user 0m0.115s sys 0m0.023s my internet connection is an ADSL with 3 MB of download and ~ 700 Kbps of upload. and finally I ran another test using namebench ( http://code.google.com/p/namebench/ ), please see results attached. ( I don' t know if the pages will render properly to you. ) Deadwood: Avg (ms) = 691.34 Min = 0.39 Max = 7,118.85 Err = 8 NoAns = 52 PowerDns Recursor: Avg (ms) = 392.99 Min = 0.41 Max = 3305.29 Err = 3 NoAns = 2 Unbound: Avg (ms) = 442.41 Min = 0.49 Max = 2494.06 Err = 0 NoAns = 4 regards, Marlon From strenholme.usenet at gmail.com Wed Sep 8 14:20:13 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Wed, 8 Sep 2010 11:20:13 -0700 Subject: Performance issues with deadwood In-Reply-To: <4C87AD5A.9010702@mrgnetwork.com.br> References: <20100906124143.GA30816@jue.netz> <4C87AD5A.9010702@mrgnetwork.com.br> Message-ID: > and finally I ran another test using namebench ( http://code.google.com/p/namebench/ ), please see results attached. ( I don' t know if the pages will render properly to you. ?) Thanks for the pointer. I ran namebench myself and got some really good results against the latest Deadwood snapshot (see below) > Deadwood: > > Avg (ms) ? ? ? ?= ? ? ? 691.34 > Min ? ? ? ? ? ? = ? ? ? 0.39 > Max ? ? ? ? ? ? = ? ? ? 7,118.85 > Err ? ? ? ? ? ? = ? ? ? 8 > NoAns ? = ? ? ? 52 I just ran namebench 1.3.1 myself against deadwood-H-20100908-2, running 250 queries. 249 of the queries were answered within the first 3.5 seconds after sending the query; the one query that wasn?t answered resolved just fine when I asked Deadwood the query after running namebench. The average response time was 302ms. Compared to Unbound 1.4.6, Deadwood is doing a lot better; Unbound missed 54 out of 250 queries and took on average 365ms to answer the queries it was able to answer. To be fair to Unbound, Deadwood, unlike Unbound, has a persistent cache (stored to a file between Deadwood sessions; note that this needs to be set up in CentOS Linux) and wasn?t ?started cold? when running this test. I, however, still contend it was a fair benchmark: Since Unbound does *not* have a persistent cache but Deadwood does, Deadwood will be started with no cache much less frequently than Unbound will be. I much prefer namebench over Gibson?s tool; namebench gives out a verbose log file with the names of every hostname queried for and the response time needed to query the host name in question. In addition, the tool is open-source, so it is possible to add tests, such as Gibson?s <>.<> test, if needed. At this point, it looks like Deadwood?s performance is comparable to other DNS servers. While Deadwood is a bit slower before having a cache, this is more than compensated for by Deadwood?s persistent cache. I have fixed the bug causing most of Deadwood?s slowdown people saw before. In other news, this morning I rewrote the code that creates a random add_constant for Deadwood?s core hashing algorithm to get its entropy before loading the mararc file. My plan today is to finish up writing the code (use the timestamp to get more entropy and update the documentation) and then to release Deadwood-2.9.07 today or tomorrow. - Sam Note: I am looking for work. Please send me a private email if you have something for my skill set (writing a fully functional recursive DNS server in C). Note also: Don?t email me personally any MaraDNS support concerns or bug reports; post your concern to the mailing list. Otherwise, I *will* ask you for money. From strenholme.usenet at gmail.com Wed Sep 8 16:04:28 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Wed, 8 Sep 2010 13:04:28 -0700 Subject: New Deadwood snapshot: Resolve speeds comparable to Unbound Message-ID: It looks like I have solved the issues people have reported with Deadwood being sluggish. The two bugs I fixed were one where Deadwood would delay timeout_seconds (normally 2 seconds) if there was a problem getting an upstream IP from its cache, and another where Deadwood would delay timeout_seconds if there were upper case letters in the name of a DNS reply. I just ran Juergen?s test with Deadwood and Unbound. Deadwood is able to resolve the list of domain names on my network in 2:42 (two minutes forty-two seconds) and 2:54 (the 2:42 figure was done starting Deadwood with no cache); Unbound needed 2:57 to resolve the same list. In addition, namebench shows Deadwood doing a lot better than Unbound (far fewer lost queries and better average response time to the queries Deadwood does resolve) on my machine. Steve Gibson?s DNS benchmark tool is buggy and incorrectly reports that Deadwood drops most DNS packets; I have set up Deadwood to report every reply it sends as well as every incoming query, and have verified that Deadwood replies to most, if not all, of its incoming queries when Gibson?s DNS benchmark is run against it. In addition, I have revised Deadwood?s internal hash algorithm to be more secure: The algorithm, in addition to using a multiplication constant that randomly changes every time Deadwood is compiled in CentOS Linux or a new Deadwood release is made (which Deadwood has always done), now uses an additive constant that is different every time Deadwood is started. Deadwood is hard-coded to use /dev/urandom (secret.txt in Windows systems) for entropy to generate this additive constant from (we also get a bit of entropy from the timestamp); if this file is not found, Deadwood will fallback to using a hard-coded additive constant for its core hash. I would like to thank Juergen Daubert, Wayne, and Marlon for their bug reports, feedback, and suggestions getting programs to help debug Deadwood?s performance. The snapshot can be downloaded here: http://maradns.org/deadwood/snap/deadwood-H-20100908-3.tar.bz2 Windows users can use this snapshot that has the improved performance (but not the improvements to Deadwood?s internal hashing function): http://maradns.org/deadwood/snap/Deadwood-H-20100908-win32.zip Please report any bugs to the mailing list [1]. - Sam [1] Crash reports are useless unless they are accompanied by a stack trace or a recipe for reproducing the crash; Valgrind errors are invalid unless Deadwood is compiled with -DVALGRIND_NOERRORS Note: Yeah, I?m still looking for work. Send me an email if you have a job for me. Note also: No, I won?t answer your MaraDNS support concerns via private email unless you pay me. Also, please don?t email me bug reports; post them to the list. From strenholme.usenet at gmail.com Wed Sep 8 21:46:50 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Wed, 8 Sep 2010 18:46:50 -0700 Subject: New Deadwood snapshot: Resolve speeds comparable to Unbound In-Reply-To: References: Message-ID: > The snapshot can be downloaded here: > > http://maradns.org/deadwood/snap/deadwood-H-20100908-3.tar.bz2 Update: Anyone who downloaded deadwood-H-20100908-3 should download deadwood-H-20100908-4; this update is a hotfix that fixes dictionary variables which I broke in deadwood-H-20100908-3. Note also that dictionary variables work with the most recent Windows binary; it was the improved hash compression security which temporarily broke dictionary variables. http://maradns.org/deadwood/snap/deadwood-H-20100908-4.tar.bz2 - Sam Please don?t email me your bug reports or support concerns. Instead, post them to the list. From strenholme.usenet at gmail.com Thu Sep 9 04:16:09 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Thu, 9 Sep 2010 01:16:09 -0700 Subject: Deadwood 2.9.07 released: Better performance and security Message-ID: I have released Deadwood 2.9.07 today, which has (on my system) slightly better performance than Unbound?I have fixed the performance problems recently discussed on the mailing list. In addition, I have improved the security a little by revising how Deadwood gets some entropy for its ?hash compression function?; this better protects Deadwood against some obscure denial of service attacks. Deadwood 2.9.07 can be downloaded as source code and as a Windows binary here: http://maradns.org/deadwood/testing/ I encourage people to download this release and report bugs here on the mailing list. - Sam Note: Sorry, I don?t accept bug reports or answer MaraDNS support concerns via private email. Note also: Yes, I?m looking for work. http://samiam.org/resume/ From spamcatch-maradns.org at messageme.de Thu Sep 9 17:24:48 2010 From: spamcatch-maradns.org at messageme.de (=?UTF-8?B?U2ViYXN0aWFuIE3DvGxsZXI=?=) Date: Thu, 09 Sep 2010 23:24:48 +0200 Subject: Deadwood@OpenWRT Message-ID: <4C8950A0.2000907@messageme.de> Hi, just want to pass my Makefile for a OpenWRT Backfire (MIPS) cross-compile. Maybe someone is interested in this. I also put a pre-compiled Deadwood 2.9.07 on the following link, for those who don't want to have to compile around 3GB of OpenWRT's buildroot. http://files.messageme.de/deadwood Cheers, Sebastian From strenholme.usenet at gmail.com Thu Sep 9 22:26:20 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Thu, 9 Sep 2010 19:26:20 -0700 Subject: Deadwood@OpenWRT In-Reply-To: <4C8950A0.2000907@messageme.de> References: <4C8950A0.2000907@messageme.de> Message-ID: > just want to pass my Makefile for a OpenWRT Backfire (MIPS) > cross-compile. Maybe someone is interested in this. > I also put a pre-compiled Deadwood 2.9.07 on the following link, for > those who don't want to have to compile around 3GB of OpenWRT's buildroot. It looks really good. This is actually the kind of target I had in mind when designing and coding Deadwood: I wanted a tiny recursive DNS server small enough to work on an embedded system. Hence, no threads. Hence, lots of code optimized to run on a 32-bit system (The secure PRNG was deliberately chosen to be one using 32-bit words, but, as it turns out, also has a version using 64-bit words. The has compression function is designed around 32-bit words. I use int32_t when I mean ?An integer that can fit over 16 bits?.) Hence, a small code size and memory footprint. I?m very curious how well Deadwood works on an embedded OpenWRT host. I do apologize for not using $HOSTCC correctly; I?ve never used a cross compiler to compile Deadwood, so I neglected to set up the makefile to always use $HOSTCC to compile the RandomPrime tool (which is used to compile other programs). Note that while I have written Deadwood to be endian-neutral, I have never had the chance to test Deadwood on a big-endian system like the MIPS ISA. Does Deadwood work without problem on a big-endian system? If not, let me know, and I?ll see if I can get QEMU or whatever set up to debug the big-endian issues. I also wonder if there is a MIPS version of the ?strip? tool? I know x86_32 binaries are about 90 or 100k in size, but squeeze down to being 64k binaries when stripped. For example, Deadwood 2.9.07 compiles to be 100,589 bytes in size in Windows32, but ?strip? reduces its size to 64,512 bytes (the x86_64 binary is about 82k in size after -Os + strip). 118k seems to be about right for an unstripped binary; see if you can strip it to make it 70k or 80k in size (I know MIPS? ISA does result in larger binaries than x86?s ISA; there?s a reason the ARM ISA has the ?thumb? instruction set). Besides that, it looks really good. Is is OK if I add your Makefile to the Deadwood distribution once we figure out how to strip the binary? - Sam Standard disclaimer: If you want to report a bug/ask for help via private email, do not forget to bring out your credit card first. From spamcatch-maradns.org at messageme.de Fri Sep 10 01:12:43 2010 From: spamcatch-maradns.org at messageme.de (=?UTF-8?B?U2ViYXN0aWFuIE3DvGxsZXI=?=) Date: Fri, 10 Sep 2010 07:12:43 +0200 Subject: Deadwood@OpenWRT In-Reply-To: References: <4C8950A0.2000907@messageme.de> Message-ID: <4C89BE4B.8010203@messageme.de> Am 10.09.2010 04:26, schrieb Sam Trenholme: > I?m very curious how well Deadwood works on an embedded OpenWRT host. > I just used the host.txt Juergen Daubert posted a few days ago and ran it from a client. I am connected on a cable connection with a latency of 10ms to my upstream DNS Server. It took 0m57.597s to finish, before the MaraDNS upstream server cached the results and 0m13.621s after the upstream MaraDNS cached it. Both times I flushed Deadwood's cache. > I do apologize for not using $HOSTCC correctly; I?ve never used a > cross compiler to compile Deadwood, so I neglected to set up the > makefile to always use $HOSTCC to compile the RandomPrime tool (which > is used to compile other programs). > > Note that while I have written Deadwood to be endian-neutral, I have > never had the chance to test Deadwood on a big-endian system like the > MIPS ISA. Does Deadwood work without problem on a big-endian system? > If not, let me know, and I?ll see if I can get QEMU or whatever set up > to debug the big-endian issues. > I didn't ran into any trouble yet, using Deadwood since a few weeks (2.9.03) on my TP-Link TL-WR1043ND. One time I got a segfault, but after recompiling (and changing nothing) everything worked. It is possible that I forgot to shutdown DNSMasq which is the default resolver on OpenWRT before running Deadwood. > I also wonder if there is a MIPS version of the ?strip? tool? I know > x86_32 binaries are about 90 or 100k in size, but squeeze down to > being 64k binaries when stripped. For example, Deadwood 2.9.07 > compiles to be 100,589 bytes in size in Windows32, but ?strip? reduces > its size to 64,512 bytes (the x86_64 binary is about 82k in size after > -Os + strip). 118k seems to be about right for an unstripped binary; > see if you can strip it to make it 70k or 80k in size (I know MIPS? > ISA does result in larger binaries than x86?s ISA; there?s a reason > the ARM ISA has the ?thumb? instruction set). > Yes, there is a strip tool, after striping the file sizes 100768 bytes. I used -s for striping >> -s --strip-all Remove all symbol and relocation information > Besides that, it looks really good. Is is OK if I add your Makefile > to the Deadwood distribution once we figure out how to strip the > binary? I just updated makefile and binary, to give it a easier look/understanding for you about the PATH hierarchy and added striping. Yeah, of course can you add it if you want. I mainly posted here to reach people who are interested in getting Deadwood running off x86. Sebastian From m.ferlitsch at gmail.com Fri Sep 10 04:15:30 2010 From: m.ferlitsch at gmail.com (Markus Ferlitsch) Date: Fri, 10 Sep 2010 10:15:30 +0200 Subject: Deadwood@OpenWRT In-Reply-To: <4C89BE4B.8010203@messageme.de> References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> Message-ID: Cool. I also use maradns on OpenWRT routers. It works fine and very stabil! greets, Markus 2010/9/10, Sebastian M?ller : > Am 10.09.2010 04:26, schrieb Sam Trenholme: >> I?m very curious how well Deadwood works on an embedded OpenWRT host. >> > I just used the host.txt Juergen Daubert posted a few days ago and ran > it from a client. I am connected on a cable connection with a latency of > 10ms to my upstream DNS Server. > > It took 0m57.597s to finish, before the MaraDNS upstream server cached > the results > and 0m13.621s after the upstream MaraDNS cached it. > > Both times I flushed Deadwood's cache. > >> I do apologize for not using $HOSTCC correctly; I?ve never used a >> cross compiler to compile Deadwood, so I neglected to set up the >> makefile to always use $HOSTCC to compile the RandomPrime tool (which >> is used to compile other programs). >> >> Note that while I have written Deadwood to be endian-neutral, I have >> never had the chance to test Deadwood on a big-endian system like the >> MIPS ISA. Does Deadwood work without problem on a big-endian system? >> If not, let me know, and I?ll see if I can get QEMU or whatever set up >> to debug the big-endian issues. >> > I didn't ran into any trouble yet, using Deadwood since a few weeks > (2.9.03) on my TP-Link TL-WR1043ND. One time I got a segfault, but after > recompiling (and changing nothing) everything worked. It is possible > that I forgot to shutdown DNSMasq which is the default resolver on > OpenWRT before running Deadwood. > >> I also wonder if there is a MIPS version of the ?strip? tool? I know >> x86_32 binaries are about 90 or 100k in size, but squeeze down to >> being 64k binaries when stripped. For example, Deadwood 2.9.07 >> compiles to be 100,589 bytes in size in Windows32, but ?strip? reduces >> its size to 64,512 bytes (the x86_64 binary is about 82k in size after >> -Os + strip). 118k seems to be about right for an unstripped binary; >> see if you can strip it to make it 70k or 80k in size (I know MIPS? >> ISA does result in larger binaries than x86?s ISA; there?s a reason >> the ARM ISA has the ?thumb? instruction set). >> > Yes, there is a strip tool, after striping the file sizes 100768 bytes. > I used -s for striping > >>> -s --strip-all Remove all symbol and relocation information > > >> Besides that, it looks really good. Is is OK if I add your Makefile >> to the Deadwood distribution once we figure out how to strip the >> binary? > > I just updated makefile and binary, to give it a easier > look/understanding for you about the PATH hierarchy and added striping. > Yeah, of course can you add it if you want. I mainly posted here to > reach people who are interested in getting Deadwood running off x86. > > Sebastian > > From strenholme.usenet at gmail.com Fri Sep 10 04:28:37 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Fri, 10 Sep 2010 01:28:37 -0700 Subject: Deadwood@OpenWRT In-Reply-To: <4C89BE4B.8010203@messageme.de> References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> Message-ID: >> I?m very curious how well Deadwood works on an embedded OpenWRT host. >> > I just used the host.txt Juergen Daubert posted a few days ago and ran > it from a client. I am connected on a cable connection with a latency of > 10ms to my upstream DNS Server. > > It took 0m57.597s to finish, before the MaraDNS upstream server cached > the results and 0m13.621s after the upstream MaraDNS cached it. Nice, and very fast. Is Deadwood able to do its own recursion with reasonable performance on an embedded OpenWRT host? I know that, on Windows, Deadwood needs three or four megs of memory to work as a fully recursive nameserver with 1024 entries in the cache; I?m thinking that Deadwood would use quite a bit of the processing power and memory of a WRT54G (8 megs of ram, 2 megs of flash); it may be necessary to reduce the cache size to 512 entries to stop Deadwood from hogging all of the memory (quick test: Deadwood uses 2.2 megs of memory with a 512-entry cache in Windows). It?s a good thing Deadwood?s LRU cache design helps Deadwood run well when there are more entries being processed than what will fit in the cache. > I didn't ran into any trouble yet, using Deadwood since a few weeks > (2.9.03) on my TP-Link TL-WR1043ND. One time I got a segfault, but after > recompiling (and changing nothing) everything worked. It is possible > that I forgot to shutdown DNSMasq which is the default resolver on > OpenWRT before running Deadwood. Hmmm...maybe. I?ll blame it on cosmic rays unless we find a way to consistently reproduce it. > Yes, there is a strip tool, after striping the file sizes 100768 bytes. > I used -s for striping That?s bigger than I thought it would be. It?s amazing how compact the x86 instruction set is; my guess was that the overhead for RISC is making a binary 50% bigger but it?s closer to 55% bigger. I wonder how large the AMD thumb ISA binary would be? (OK, time to install an ARM cross-compiler to find out...) > I mainly posted here to > reach people who are interested in getting Deadwood running off x86. I?m very interested in that target also. Deadwood is about as small as a full-featured recursive DNS server can get, and I think it makes as much sense to put it in an embedded router as on a desktop PC. From strenholme.usenet at gmail.com Fri Sep 10 04:29:36 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Fri, 10 Sep 2010 01:29:36 -0700 Subject: Deadwood@OpenWRT In-Reply-To: References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> Message-ID: > I also use maradns on OpenWRT routers. It works fine and very stable! Have you had a chance to try Deadwood (MaraDNS 2.0?s recursive core) on an OpenWRT router? - Sam From m.ferlitsch at gmail.com Fri Sep 10 08:53:25 2010 From: m.ferlitsch at gmail.com (Markus Ferlitsch) Date: Fri, 10 Sep 2010 14:53:25 +0200 Subject: Deadwood@OpenWRT In-Reply-To: References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> Message-ID: Hi, no I didn't try because maradns works very good. But now Deadwood also seems to run stabil, so I will try it in next time (next holidays) on one of my OpenWrt routers. greets, Markus. 2010/9/10, Sam Trenholme : >> I also use maradns on OpenWRT routers. It works fine and very stable! > > Have you had a chance to try Deadwood (MaraDNS 2.0?s recursive core) > on an OpenWRT router? > > - Sam > From spamcatch-maradns.org at messageme.de Fri Sep 10 10:37:13 2010 From: spamcatch-maradns.org at messageme.de (=?UTF-8?B?U2ViYXN0aWFuIE3DvGxsZXI=?=) Date: Fri, 10 Sep 2010 16:37:13 +0200 Subject: Deadwood@OpenWRT In-Reply-To: References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> Message-ID: <4C8A4299.5060104@messageme.de> Am 10.09.2010 10:28, schrieb Sam Trenholme: >>> I?m very curious how well Deadwood works on an embedded OpenWRT host. >>> >> I just used the host.txt Juergen Daubert posted a few days ago and ran >> it from a client. I am connected on a cable connection with a latency of >> 10ms to my upstream DNS Server. >> >> It took 0m57.597s to finish, before the MaraDNS upstream server cached >> the results and 0m13.621s after the upstream MaraDNS cached it. > > Nice, and very fast. Is Deadwood able to do its own recursion with > reasonable performance on an embedded OpenWRT host? I know that, on > Windows, Deadwood needs three or four megs of memory to work as a > fully recursive nameserver with 1024 entries in the cache; I?m > thinking that Deadwood would use quite a bit of the processing power > and memory of a WRT54G (8 megs of ram, 2 megs of flash); it may be > necessary to reduce the cache size to 512 entries to stop Deadwood > from hogging all of the memory (quick test: Deadwood uses 2.2 megs of > memory with a 512-entry cache in Windows). It?s a good thing > Deadwood?s LRU cache design helps Deadwood run well when there are > more entries being processed than what will fit in the cache. > I did the test 3 times without using an upstream server. It took real 2m40.678s real 2m17.158s real 2m16.010s (I didn't change the order) Deadwood uses 5% of memory, my router got 32mb overall / displayed 29mb in top. (29mb|32mb) * 0.05 = 1.45mb|1.6mb #free total used free shared buffers Mem: 29484 20736 8748 0 1100 during the test, top displayed, a CPU usage of 1-3%. But my router got a fast 400Mhz CPU and DDR2. I got an old Linksys WRT54GS running OpenWRT, too. It got a 200MHz CPU which wasn't able to handle QoS on a 25mbit line (CPU overloaded at 14mbit). I will revive it later and test Deadwood performance on it. >> I didn't ran into any trouble yet, using Deadwood since a few weeks >> (2.9.03) on my TP-Link TL-WR1043ND. One time I got a segfault, but after >> recompiling (and changing nothing) everything worked. It is possible >> that I forgot to shutdown DNSMasq which is the default resolver on >> OpenWRT before running Deadwood. > > Hmmm...maybe. I?ll blame it on cosmic rays unless we find a way to > consistently reproduce it. > Cosmic is a good reason for now. If it happens again I will search a better reason. >> Yes, there is a strip tool, after striping the file sizes 100768 bytes. >> I used -s for striping > > That?s bigger than I thought it would be. It?s amazing how compact > the x86 instruction set is; my guess was that the overhead for RISC is > making a binary 50% bigger but it?s closer to 55% bigger. I wonder > how large the AMD thumb ISA binary would be? (OK, time to install an > ARM cross-compiler to find out...) > >> I mainly posted here to >> reach people who are interested in getting Deadwood running off x86. > > I?m very interested in that target also. Deadwood is about as small > as a full-featured recursive DNS server can get, and I think it makes > as much sense to put it in an embedded router as on a desktop PC. From strenholme.usenet at gmail.com Sat Sep 11 22:15:05 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Sat, 11 Sep 2010 19:15:05 -0700 Subject: Deadwood@OpenWRT In-Reply-To: <4C8A4299.5060104@messageme.de> References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> <4C8A4299.5060104@messageme.de> Message-ID: > I did the test 3 times without using an upstream server. It took > real ? ?2m40.678s > real ? ?2m17.158s > real ? ?2m16.010s > (I didn't change the order) > > Deadwood uses 5% of memory, my router got 32mb overall / displayed 29mb > in top. > (29mb|32mb) * 0.05 = 1.45mb|1.6mb First of all, thanks for doing these tests. One use case I was never able to test for in Deadwood was to see how it ran on a big-endian ISA. While I have written Deadwood in an endian-neutral manner, it?s nice to see reports of Deadwood actually working on a big-endian processor without problem. Second of all, it looks like OpenWRT uses smaller malloc() pages or what not; on Windows, a single entry in the cache uses about two kilobytes of memory, but here it looks like each cache entry uses less memory. Anyway, I had as one of the ?known issues? for Deadwood the fact it hasn?t been tested on a big-endian machine; I have removed that issue. Another use case I would like to see tested is using Deadwood for DNS lookups delivering mail on a mailhub (lots of MX lookups). Deadwood, by default, disables MX lookups (on an average consumer machine, it?s far more likely an MX lookup is coming from a spambot than from a legitimate request to deliver email), but there is a way to enable it which I will spell out if anyone comes forward and says they want to test Deadwood doing DNS lookups so email can be delivered. - Sam No, I do not answer MaraDNS email privately. Not unless you pay me for it. From spamcatch-maradns.org at messageme.de Sun Sep 12 08:53:02 2010 From: spamcatch-maradns.org at messageme.de (=?UTF-8?B?U2ViYXN0aWFuIE3DvGxsZXI=?=) Date: Sun, 12 Sep 2010 14:53:02 +0200 Subject: Deadwood@OpenWRT In-Reply-To: References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> <4C8A4299.5060104@messageme.de> Message-ID: <4C8CCD2E.8070804@messageme.de> Am 12.09.2010 04:15, schrieb Sam Trenholme: > Anyway, I had as one of the ?known issues? for Deadwood the fact it > hasn?t been tested on a big-endian machine; I have removed that issue. > > Another use case I would like to see tested is using Deadwood for DNS > lookups delivering mail on a mailhub (lots of MX lookups). Deadwood, > by default, disables MX lookups (on an average consumer machine, it?s > far more likely an MX lookup is coming from a spambot than from a > legitimate request to deliver email), but there is a way to enable it > which I will spell out if anyone comes forward and says they want to > test Deadwood doing DNS lookups so email can be delivered. > I revived my WRT54GS, after that I noticed that it's using MIPSEL instead of MIPS. I am currently building a buildroot for the following Arch CPUs. It will take a little time, to build the crosscompilers, since it runs on an old AMD64 3200+ with 1,5gb Ram. Arch Board ==================================== ARM kirkwood, orion ARMEB ixp4xx AVR32 avr32 MIPS ar71xx, atheros, brcm63xx, ifxmips, octeon MIPSEL ar7, brcm-2.4, brcm47xx, cobalt, rb532, xburst POWERPC ppc40x, ppc44x Sebastian From sysadmin at mrgnetwork.com.br Sun Sep 12 22:02:19 2010 From: sysadmin at mrgnetwork.com.br (Marlon) Date: Sun, 12 Sep 2010 23:02:19 -0300 Subject: Deadwood 2.9.07 released: Better performance and security In-Reply-To: (sfid-H20100909-050333-+030.23-1@spamfilter.osbf.lua) References: (sfid-H20100909-050333-+030.23-1@spamfilter.osbf.lua) Message-ID: <4C8D862B.7090700@mrgnetwork.com.br> On 09/09/2010 05:16 AM, Sam Trenholme wrote: > I have released Deadwood 2.9.07 today, which has (on my system) > slightly better performance than Unbound?I have fixed the performance > problems recently discussed on the mailing list. > > In addition, I have improved the security a little by revising how > Deadwood gets some entropy for its ?hash compression function?; this > better protects Deadwood against some obscure denial of service > attacks. > > Deadwood 2.9.07 can be downloaded as source code and as a Windows binary here: > > http://maradns.org/deadwood/testing/ > > I encourage people to download this release and report bugs here on > the mailing list. > > - Sam > Hi, I repeat here the tests with Deadwood 2.9.07, please see the results below: time $(for h in $(cat hosts.txt); do printf "\n\n\n"; dig $h; done > test.log) Deadwood: 3m25.948s Unbound: 1m13.523s Namebench: Deadwood - AVG(ms): 830.77 Min: 0.44 Max: 7604.16 Err: 12 NoAns: 1 Unbound - AVG(ms): 457.22 Min: 0.57 Max: 7002.80 Err: 1 NoAns: 0 Regards, Marlon From strenholme.usenet at gmail.com Sun Sep 12 22:42:33 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Sun, 12 Sep 2010 19:42:33 -0700 Subject: Deadwood 2.9.07 released: Better performance and security In-Reply-To: <4C8D862B.7090700@mrgnetwork.com.br> References: <4C8D862B.7090700@mrgnetwork.com.br> Message-ID: > I repeat here the tests with Deadwood 2.9.07, please see the results below: > > > time $(for h in $(cat hosts.txt); do printf "\n\n\n"; dig $h; done > > test.log) > > Deadwood: ? ?3m25.948s > Unbound: ? ? ?1m13.523s > > > Namebench: > > Deadwood - AVG(ms): 830.77 ? ? Min: 0.44 ? ? Max: 7604.16 ? ? Err: 12 > NoAns: ?1 > Unbound - ? AVG(ms): 457.22 ? ? Min: 0.57 ? ? Max: 7002.80 ? ? Err: 1 > ? ?NoAns: ?0 > > Regards, > > Marlon Thank you very much for running this test. I?m going to have to close this issue as ?works for me?; these are not the numbers I am seeing. I?m seeing slightly better numbers for Deadwood (under three minutes) and worse numbers for Unbound (nearly three minutes). Indeed, Deadwood is faster than Unbound on my system (Windows XP service pack 3 and all updates). Looking at the number of errors you?re getting with Deadwood, perhaps maxprocs is not set high enough for this test and you?re overloading Deadwood (which can only have 32 pending queries at once unless this is increased by setting maxprocs). Maybe someone will come forward and submit a patch that will improve Deadwood?s performance?there might be something non-standard about how Linux does sockets that allows considerable speedup if someone knows the secret recipe, just as Linux has a longstanding ?wont fix? issue that sometimes select() says a socket is non-blocking but that does not mean the socket is really non-blocking. When Deadwood was taking 8 minutes to go through the list, it was a fairly critical issue. When Deadwood has better performance than Unbound on some systems but worse performance than Unbound on other systems, it?s a wishlist issue. What I would be interested in seeing is particular hostnames that consistently resolve more slowly in Deadwood thank in Unbound. If you can find a single hostname in Unbound that resolves, say, three times as fast in Unbound with an empty cache than in Deadwood with an empty cache, I want to know about it. I?m even more interested in seeing hostnames that Deadwood can not resolve at all but that Unboung, BIND, or whatever can resolve. If people have any of these to report, please let us know on the list. Again, thank you for your Deadwood performance numbers and for helping test Deadwood. - Sam No, I don?t answer private email about MaraDNS or Deadwood. Well, I don?t unless you pay me for my time. From strenholme.usenet at gmail.com Sun Sep 12 22:44:28 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Sun, 12 Sep 2010 19:44:28 -0700 Subject: Deadwood@OpenWRT In-Reply-To: <4C8CCD2E.8070804@messageme.de> References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> <4C8A4299.5060104@messageme.de> <4C8CCD2E.8070804@messageme.de> Message-ID: > I revived my WRT54GS, after that I noticed that it's using MIPSEL > instead of MIPS. I am currently building a buildroot for the following > Arch CPUs. It will take a little time, to build the crosscompilers, > since it runs on an old AMD64 3200+ with 1,5gb Ram. > > Arch ? ? ? ? ? ?Board > ==================================== > ARM ? ? ? ? ? ? kirkwood, orion > ARMEB ? ? ? ? ? ixp4xx > AVR32 ? ? ? ? ? avr32 > MIPS ? ? ? ? ? ?ar71xx, atheros, brcm63xx, ifxmips, octeon > MIPSEL ? ? ? ? ?ar7, brcm-2.4, brcm47xx, cobalt, rb532, xburst > POWERPC ? ? ? ? ppc40x, ppc44x I would be interested in being a mirror site hosting these files. While I can?t support Deadwood on OpenWRT, I?m perfectly happy mirroring Deadwood OpenWRT ports. - Sam From spamcatch-maradns.org at messageme.de Tue Sep 14 04:37:53 2010 From: spamcatch-maradns.org at messageme.de (=?UTF-8?B?U2ViYXN0aWFuIE3DvGxsZXI=?=) Date: Tue, 14 Sep 2010 10:37:53 +0200 Subject: Deadwood@OpenWRT In-Reply-To: References: <4C8950A0.2000907@messageme.de> <4C89BE4B.8010203@messageme.de> <4C8A4299.5060104@messageme.de> <4C8CCD2E.8070804@messageme.de> Message-ID: <4C8F3461.8090604@messageme.de> Am 13.09.2010 04:44, schrieb Sam Trenholme: >> I revived my WRT54GS, after that I noticed that it's using MIPSEL >> instead of MIPS. I am currently building a buildroot for the following >> Arch CPUs. It will take a little time, to build the crosscompilers, >> since it runs on an old AMD64 3200+ with 1,5gb Ram. >> >> Arch Board >> ==================================== >> ARM kirkwood, orion >> ARMEB ixp4xx >> AVR32 avr32 >> MIPS ar71xx, atheros, brcm63xx, ifxmips, octeon >> MIPSEL ar7, brcm-2.4, brcm47xx, cobalt, rb532, xburst >> POWERPC ppc40x, ppc44x > > I would be interested in being a mirror site hosting these files. > While I can?t support Deadwood on OpenWRT, I?m perfectly happy > mirroring Deadwood OpenWRT ports. > > - Sam Do that as you like, the server I am hosting the files got a free-traffic limit of 100gbyte/month. I will update the binaries to stay on the newest version, but I won't compile new binaries from the Hotfixes as long as it's not a problem on a certain arch listed above. I talked yesterday to the devs of OpenWRT, they suggested me to remove -mtune= from the CFLAGS, since there are too many different CPU revisions/command-sets. Due to that, the binary is 6kbyte bigger than tuned to a single MIPS CPU revision. Additionally I am currently unable to build AVR32 and ARMEB due to buildroot compile errors. I repeated the test I ran on the TL-WR1034ND on the old WRT54GS running bcrm-2.4. wrt54gs running bcrm-2.4 MIPSEL: real 1m24.800s TL-WR1034ND running ar71xx MIPS: real 2m40.678s both ran own recursion. Cheers, Sebastian From strenholme.usenet at gmail.com Tue Sep 21 16:20:59 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Tue, 21 Sep 2010 15:20:59 -0500 Subject: I'm providing a mirror of Deadwood OpenWRT files Message-ID: I am making a mirror of Sebastian M?ller?s Deadwood port to various OpenWRT platforms available. The ARM port has the smallest RISC binary (Sebastian: Was this compiled , the PowerPC binary is about as big as Deadwood?s x86_64 binary, and the MIPS binary is the largest. Sebastian: I note that the ARM v5t CPU does have ARM Thumb support. It looks like there isn?t a "-mthumb" options in the compile flags for Deadwood ARM, and the GCC manual for the ARM processor notes that ?The default is to use the 32-bit ARM instruction set.?. Right now the ARM binary is 73772 bytes in size?only 15% larger than the x86_32 binary?and we may be able to make is as small as or smaller than the x86_32 binary by enabling Thumb instructions. - Sam From strenholme.usenet at gmail.com Tue Sep 21 16:21:40 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Tue, 21 Sep 2010 15:21:40 -0500 Subject: I'm providing a mirror of Deadwood OpenWRT files In-Reply-To: References: Message-ID: > I am making a mirror of Sebastian M?ller?s Deadwood port to various > OpenWRT platforms available. It?s here: http://maradns.org/deadwood/OpenWRT/ - Sam From spamcatch-maradns.org at messageme.de Wed Sep 22 09:53:50 2010 From: spamcatch-maradns.org at messageme.de (=?UTF-8?B?U2ViYXN0aWFuIE3DvGxsZXI=?=) Date: Wed, 22 Sep 2010 15:53:50 +0200 Subject: I'm providing a mirror of Deadwood OpenWRT files In-Reply-To: References: Message-ID: <4C9A0A6E.7020106@messageme.de> Am 21.09.2010 22:20, schrieb Sam Trenholme: > Sebastian: I note that the ARM v5t CPU does have ARM Thumb support. > It looks like there isn?t a "-mthumb" options in the compile flags for > Deadwood ARM, and the GCC manual for the ARM processor notes that ?The > default is to use the 32-bit ARM instruction set.?. Right now the ARM > binary is 73772 bytes in size?only 15% larger than the x86_32 > binary?and we may be able to make is as small as or smaller than the > x86_32 binary by enabling Thumb instructions. > > - Sam Hi Sam, I talked to the OpenWRT Devs at IRC a before I made the binaries. They suggested me to set as few compile options as possible to guarantee that it is working on all CPU's. Thats why I removed all -m instructions I made before and left only -Os. I am not very firm with ARM CPU's and don't know the differences between ARMv5t and ARMv5te, but of course I can recompile the binary with -mthumb if you want. Cheers, Sebastian From strenholme.usenet at gmail.com Wed Sep 22 12:43:07 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Wed, 22 Sep 2010 11:43:07 -0500 Subject: I'm providing a mirror of Deadwood OpenWRT files In-Reply-To: <4C9A0A6E.7020106@messageme.de> References: <4C9A0A6E.7020106@messageme.de> Message-ID: > I am not very firm with ARM CPU's and > don't know the differences between ARMv5t and ARMv5te, but of course I > can recompile the binary with -mthumb if you want. OK, I?m thinking maybe we should then have two binaries: A conservative non-thumb compile that is guaranteed to work on pretty much any ARM CPU out there, and a slightly less conservative thumb binary which showcases how small Deadwood is and that the ARM Thumb ISA can make binaries as small as or smaller than x86_32 binaries (possibly ?nearly as small as?). We can put the ?thumb? binary in a separate directory. As an aside, I?m fixing a number of minor bugs in the parser. Things like the ?+=? (append) operator not working for upstream_servers/root_servers, the parser not strictly enforcing the use of only unused dictionary variables, the parser not requiring dictionary variables being initialized with a line like "root_servers = {}", things like that. - Sam From spamcatch-maradns.org at messageme.de Wed Sep 22 18:56:28 2010 From: spamcatch-maradns.org at messageme.de (=?UTF-8?B?U2ViYXN0aWFuIE3DvGxsZXI=?=) Date: Thu, 23 Sep 2010 00:56:28 +0200 Subject: I'm providing a mirror of Deadwood OpenWRT files In-Reply-To: References: <4C9A0A6E.7020106@messageme.de> Message-ID: <4C9A899C.6060302@messageme.de> Am 22.09.2010 18:43, schrieb Sam Trenholme: >> I am not very firm with ARM CPU's and >> don't know the differences between ARMv5t and ARMv5te, but of course I >> can recompile the binary with -mthumb if you want. > > OK, I?m thinking maybe we should then have two binaries: A > conservative non-thumb compile that is guaranteed to work on pretty > much any ARM CPU out there, and a slightly less conservative thumb > binary which showcases how small Deadwood is and that the ARM Thumb > ISA can make binaries as small as or smaller than x86_32 binaries > (possibly ?nearly as small as?). > > We can put the ?thumb? binary in a separate directory. > /I will ask at IRC#openwrt if there would be any trouble using -mthumb for v5te./ I asked in IRC#openwrt one of the main-devs said: 00:30 < *> the 't' in ARMv5t/te stands for 'thumb' So, for now, I got this (visual) difference for you #ls -l ARM*/* -rwxr-xr-x 1 sebastian sebastian 73772 23. Sep 00:53 ARM/Deadwood -rwxr-xr-x 1 sebastian sebastian 49608 23. Sep 00:53 ARM-thumb/Deadwood > As an aside, I?m fixing a number of minor bugs in the parser. Things > like the ?+=? (append) operator not working for > upstream_servers/root_servers, the parser not strictly enforcing the > use of only unused dictionary variables, the parser not requiring > dictionary variables being initialized with a line like "root_servers > = {}", things like that. > > - Sam I will update the binaries after you released the fixed version. Do you see any exigence in archiving the binaries? Cheers, From strenholme.usenet at gmail.com Fri Sep 24 22:52:42 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Fri, 24 Sep 2010 21:52:42 -0500 Subject: I'm providing a mirror of Deadwood OpenWRT files In-Reply-To: <4C9A899C.6060302@messageme.de> References: <4C9A0A6E.7020106@messageme.de> <4C9A899C.6060302@messageme.de> Message-ID: > -rwxr-xr-x 1 sebastian sebastian 73772 23. Sep 00:53 ARM/Deadwood > -rwxr-xr-x 1 sebastian sebastian 49608 23. Sep 00:53 ARM-thumb/Deadwood That is what I was curious to see; it's nice to see that x86 doesn't make the smallest code out there, and that ARM thumb makes even smaller code. > I will update the binaries after you released the fixed version. I have just released Deadwood 3.0.01; this is a stable release and I, at this point, plan on only applying bug fixes to Deadwood. It is available here: http://maradns.org/deadwood/stable/ > Do you see any exigence in archiving the binaries? No, I do not. Once you have the 3.0 binaries up, I will mirror them and remove the 2.9.07 binaries. - Sam From spamcatch-maradns.org at messageme.de Sun Sep 26 23:27:00 2010 From: spamcatch-maradns.org at messageme.de (=?UTF-8?B?U2ViYXN0aWFuIE3DvGxsZXI=?=) Date: Mon, 27 Sep 2010 05:27:00 +0200 Subject: I'm providing a mirror of Deadwood OpenWRT files In-Reply-To: References: <4C9A0A6E.7020106@messageme.de> <4C9A899C.6060302@messageme.de> Message-ID: <4CA00F04.5090603@messageme.de> Am 25.09.2010 04:52, schrieb Sam Trenholme: > No, I do not. Once you have the 3.0 binaries up, I will mirror them > and remove the 2.9.07 binaries. > > - Sam Just updated. From strenholme.usenet at gmail.com Tue Sep 28 04:30:43 2010 From: strenholme.usenet at gmail.com (Sam Trenholme) Date: Tue, 28 Sep 2010 01:30:43 -0700 Subject: I'm providing a mirror of Deadwood OpenWRT files In-Reply-To: <4CA00F04.5090603@messageme.de> References: <4C9A0A6E.7020106@messageme.de> <4C9A899C.6060302@messageme.de> <4CA00F04.5090603@messageme.de> Message-ID: >> Once you have the 3.0 binaries up, I will mirror them >> and remove the 2.9.07 binaries. >> >> - Sam > > Just updated. I?ve updated my mirror: http://maradns.org/deadwood/OpenWRT I am keeping a copy of the pre-thumb ARM binary of Deadwood 2.9.07 around just in case some ancient router out there doesn?t support ARM thumb extensions. I?ve also just released MaraDNS 2.0. This is simply MaraDNS 1.4 with the old deprecated MaraDNS 1 recursive code torn out of the maradns daemon, to be replaced by the separate Deadwood daemon. More to the point, the documentation has been updated to reflect the fact that recursion should now be done with the Deadwood daemon I have spent the last three years making a reality. - Sam