Get Headers by Date Problem

John Elliott · Experienced Joined: 10 May 2005 Posts: 121

Get Headers by Date for alt.binaries.teevee indicates that it's getting 120000001 headers for only 1 day's range, from 6/10/11 to 6/11/11.

It's been downloading headers since 7:14am, and now it's 7:44am. Is this normal to take so long?

Is it really going to download 120000001 headers just to get 1 days worth?

administrator · Posted: Sat Jun 11, 2011 11:59 am Post subject:

NMP performs a binary search to locate the lowest and highest headers (by article index) matching your date range. It's possible that the headers in this group are out of sequence on the server, but it's also possible that with all of the spam attacks lately that there really are that many headers.

Have you taken a look at the headers you are getting to see if they are in the date range?

Regards

BZB · Expert Joined: 26 Apr 2005 Posts: 1387

That group is one of the massive ones, so you'll always be dealing with a large number of headers even for just a day.

administrator · Posted: Sat Jun 11, 2011 12:24 pm Post subject:

Based on the log you sent, it looks like your server failed to respond to an XOVER request correctly after header compression was enabled. The server does support header compression (according to the log), but the XOVER request returned garbage and NMP defaulted to single-header requests.

However, I can't tell from the log alone what NMP is doing. If it is retrying "missing" headers from a previous session, that could also explain the single-header requests. You should be able to tell by watching the connection itself. If it isn't retrying missing headers, then I'd stop the job and retry.

Regards

John Elliott · Experienced Joined: 10 May 2005 Posts: 121

I am using header compression, and I can see by the speed of these downloads that it works. Does this still not work with SSL enabled?

John Elliott · Experienced Joined: 10 May 2005 Posts: 121

NMP shows the entire header count for this newsgroup as 120,148,449. It was going do download all of them except for 148,448 per the connection window. I did a screen .jpg, but I can't see how to upload it.[/img]

administrator · Posted: Sat Jun 11, 2011 2:40 pm Post subject:

administrator · Posted: Sat Jun 11, 2011 2:43 pm Post subject:

John Elliott · Experienced Joined: 10 May 2005 Posts: 121

I'm using Message ID instead of Article Index. I thought that would give greater flexibility in downloading NZB's & using multiple servers. It also says below that the Message-ID is unique, so I can't see why it would go back 120,000,001 headers. Using a fallback server, should I or could I use Article Index?

Use Message-ID instead of article index - If your server supports it, enabling this option allows Newsman to ask for articles using their unique Message-ID rather than the article number.

Enable "Use Message-ID instead of article index" - In each news server's setup page, this option can be enabled to tell Newsman Pro not to try to download articles using their article index. Some news servers claiming long retention times do not maintain the article index spool at the same retention as the actual articles. If Newsman Pro attempts to retrieve an older article using its article index, the download will fail and Newsman Pro will disconnect...wait...retry. After the second attempt, Newsman Pro will retry using the article's Message-ID automatically. Enabling this option will force Newsman Pro to skip the download attempts using the article index. Most commercial news servers support downloading using the Message-ID.

This seems to indicate that Message-ID is superior. Guess I have to live with this....

Thanks anyway.

BZB · Expert Joined: 26 Apr 2005 Posts: 1387

BZB · Expert Joined: 26 Apr 2005 Posts: 1387

John Elliott · Experienced Joined: 10 May 2005 Posts: 121

I got new headers, then did a catchup. Using XFeature & SSL I then repeated the get 1 day's headers. Still wanted to get 120000001 headers.

I got new headers again, then did a catchup. Turned compression to Astraweb's version. SSL on, did get 1 days headers. Same thing, 120000001 coming my way.

Turned compression off, got new headers, did a catchup, did get 1 days headers, same thing again. This is all using Message ID. This is only happening on alt.binaries.teevee. Perhaps something's wrong with it.

administrator · Posted: Sat Jun 11, 2011 10:32 pm Post subject:

The Message-ID setting does not have any impact on getting headers. This setting is only used when downloading articles from previously downloaded headers.

Regards

RobertWright · Experienced Joined: 13 Feb 2006 Posts: 131

I have run into a similar thing. I use Newshosting and added alt.binaries.erotica to a workgroup. I'm using Article Index, instead of Message-ID, SSL is not enabled and neither is header compression. I tried to download headers by date, selecting 20 as the number of days to get, but not a single header was downloading after running overnight. The total number of headers was showing as 120000001. I stopped this header download and tried 10 days and still didn't get a single header after 10 minutes. I stopped this download and switched to Last xxx headers. I entered 10,000,000 headers and clicked the OK button. Headers immediately started downloading. On the workgroups screen I know see the Message count showing as 120000001, the High is 1154176170, and the Low is 1034176170. This was the first time I had tried to download headers for this group.

administrator · Posted: Wed Jul 20, 2011 10:04 pm Post subject:

I just tried to get the last 2 days and it worked just fine. I suspect that there are conditions that can occur where the binary search just can't determine what headers are included in a date range simply due to header volume. Getting the last xxx headers will always work because the NNTP protocol supports this directly. There is no NNTP support for getting headers in a date range, so NMP has to walk the header index searching for starting and ending points. I've never had it fail for me, but I only use it in an emergency.

Regards