Compressed headers.
Goto page 1, 2  Next
 
Post new topic   Reply to topic    newsmanpro.com Forum Index : Feature Requests
View previous topic :: View next topic  
Author Message
virtuacard
Experienced
Experienced


Joined: 18 Jul 2005
Posts: 75
Location: Spain, EU

PostPosted: Mon Jun 29, 2009 12:29 pm    Post subject: Compressed headers. Reply with quote

Hi!

I'm back, after a lot of time...(I have been in Ubuntu's Hell... Wink )

Well, one question...Can you implement this? :

http://forums.newsbin.com/viewtopic.php?t=23489

I think that would be really useful!

Best regards.
_________________
Hispanic causing panic!
Back to top
View user's profile Send private message
administrator
Developer
Developer


Joined: 24 Jul 2004
Posts: 4824
Location: King William, VA

PostPosted: Mon Jun 29, 2009 12:42 pm    Post subject: Reply with quote

I'll look into it...

Regards
Back to top
View user's profile Send private message Send e-mail
virtuacard
Experienced
Experienced


Joined: 18 Jul 2005
Posts: 75
Location: Spain, EU

PostPosted: Mon Jun 29, 2009 12:51 pm    Post subject: Reply with quote

Update... Technical info about that:

http://helpdesk.astraweb.com/index.php?_m=news&_a=viewnews&newsid=9

Regards.
_________________
Hispanic causing panic!
Back to top
View user's profile Send private message
administrator
Developer
Developer


Joined: 24 Jul 2004
Posts: 4824
Location: King William, VA

PostPosted: Mon Jun 29, 2009 1:04 pm    Post subject: Reply with quote

Looks like there are at least two methods that are available out there, but only Astraweb's is open to any newsreader. Giganews and Newshosting compress headers through their "accelerators". The Astraweb method seems pretty straightforward, but I'll have to see what's involved on the coding side to support it.

Regards
Back to top
View user's profile Send private message Send e-mail
BZB
Expert
Expert


Joined: 26 Apr 2005
Posts: 1408

PostPosted: Mon Jun 29, 2009 1:41 pm    Post subject: Reply with quote

It seems that there is actually some organisation between the news hosting companies to developer the XZHDR and XZVER as NNTP extensions, so following the Astraweb spec may very well allow NMP to support the same implementations from Giganews and Newshosting.

That's certainly worthwhile I think, and other news readers seem to be implementing it for the same reasons.

Not sure how it will work with SSL though.
Back to top
View user's profile Send private message
administrator
Developer
Developer


Joined: 24 Jul 2004
Posts: 4824
Location: King William, VA

PostPosted: Mon Jun 29, 2009 10:57 pm    Post subject: Reply with quote

Well, I built a test bed to download headers using XZVER and can retrieve and decode them, but the ZLib call to inflate the result is failing, so I think I'm either missing something, or Astraweb is doing something unusual. I'm voting for the latter and have sent a message to their helpdesk. We'll see how far that gets me.

Regards
Back to top
View user's profile Send private message Send e-mail
BZB
Expert
Expert


Joined: 26 Apr 2005
Posts: 1408

PostPosted: Tue Jun 30, 2009 10:40 am    Post subject: Reply with quote

administrator wrote:
Well, I built a test bed to download headers using XZVER and can retrieve and decode them, but the ZLib call to inflate the result is failing, so I think I'm either missing something, or Astraweb is doing something unusual. I'm voting for the latter and have sent a message to their helpdesk. We'll see how far that gets me.

Regards


Out of interest, what is the error?

I don't know if it will be any help, but if you get stalled at the helpdesk, try contacting "steve" @ astraweb.com . He was a very competent technical guy who dealt with some backend stuff when I was reporting header corruption problems on one of their server farms. I don't know if he still works there, but he was much more useful than the helpdesk.
Back to top
View user's profile Send private message
administrator
Developer
Developer


Joined: 24 Jul 2004
Posts: 4824
Location: King William, VA

PostPosted: Tue Jun 30, 2009 11:08 am    Post subject: Reply with quote

Generic "Data Error". Funny you should mention Steve...I received an email from him this morning with some suggested solutions.

Regards
Back to top
View user's profile Send private message Send e-mail
administrator
Developer
Developer


Joined: 24 Jul 2004
Posts: 4824
Location: King William, VA

PostPosted: Mon Jul 06, 2009 11:34 pm    Post subject: Reply with quote

BZB wrote:
It seems that there is actually some organisation between the news hosting companies to developer the XZHDR and XZVER as NNTP extensions, so following the Astraweb spec may very well allow NMP to support the same implementations from Giganews and Newshosting.


Well, this isn't going to be a quick addition real soon. The Astraweb XZVER method does work, but Steve over at Astraweb has suggested that this implementation is going away in the near future. Apparently, there is another method for handling compressed headers that most providers are working towards, including Astraweb. I've experimented with it, but in order to implement it cleanly into NMP, I will have to modify some low-level NNTP stuff and that will take some serious time and effort to ensure that it a) works and b) doesn't break anything else. I'm going to back-burner this temporarily until the final specification is ready. In the meantime, I'll take a look at what will be needed to implement the spec as it has been relayed to me.

Sorry for the delay, but I don't take these mods lightly.

Regards
Back to top
View user's profile Send private message Send e-mail
BZB
Expert
Expert


Joined: 26 Apr 2005
Posts: 1408

PostPosted: Mon Jul 06, 2009 11:55 pm    Post subject: Reply with quote

administrator wrote:


Sorry for the delay, but I don't take these mods lightly.


If the current spec is a lot of work, and it's being replaced soon, it's probably not a good use of your time. Having said that, these things usually take a lot longer to implement than people tend to imply, unless Astraweb has convinced you the new spec is coming real soon.

Have you any links to the new spec?
Back to top
View user's profile Send private message
administrator
Developer
Developer


Joined: 24 Jul 2004
Posts: 4824
Location: King William, VA

PostPosted: Tue Jul 07, 2009 1:05 am    Post subject: Reply with quote

BZB wrote:
administrator wrote:


Sorry for the delay, but I don't take these mods lightly.


If the current spec is a lot of work, and it's being replaced soon, it's probably not a good use of your time. Having said that, these things usually take a lot longer to implement than people tend to imply, unless Astraweb has convinced you the new spec is coming real soon.

Have you any links to the new spec?


No, it's all pretty hush hush at this point. Steve was pretty clear that it was coming relatively soon. The spec I'm talking about is the new one, but it is not "official" yet. It will replace the XZVER method altogether.

Ironically, for me, the new spec is more work than Astraweb's XZVER implementation.

Regards
Back to top
View user's profile Send private message Send e-mail
BZB
Expert
Expert


Joined: 26 Apr 2005
Posts: 1408

PostPosted: Tue Jul 07, 2009 9:32 am    Post subject: Reply with quote

administrator wrote:

Ironically, for me, the new spec is more work than Astraweb's XZVER implementation.


If it's going to become industry standard and all the major usenet providers are going to support it, it's worth doing given the large amount of time and data that can be involved in header downloads. It all looks very compressable given it's all text.

I was looking forward to testing it, so I guess I'll just have to wait a bit longer.
Back to top
View user's profile Send private message
administrator
Developer
Developer


Joined: 24 Jul 2004
Posts: 4824
Location: King William, VA

PostPosted: Tue Jul 07, 2009 11:02 am    Post subject: Reply with quote

BZB wrote:
administrator wrote:

Ironically, for me, the new spec is more work than Astraweb's XZVER implementation.


If it's going to become industry standard and all the major usenet providers are going to support it, it's worth doing given the large amount of time and data that can be involved in header downloads. It all looks very compressable given it's all text.

I was looking forward to testing it, so I guess I'll just have to wait a bit longer.


That, in theory, is what's supposed to happen, but no one is openly admitting yet.

Regards
Back to top
View user's profile Send private message Send e-mail
administrator
Developer
Developer


Joined: 24 Jul 2004
Posts: 4824
Location: King William, VA

PostPosted: Mon Aug 24, 2009 6:36 pm    Post subject: Reply with quote

An update:

I am currently running NMP with native header compression for servers supporting the new compression mode command. I've tested with Newshosting and Astraweb and it appears to be working great, though I want to run it a lot longer before putting out a beta. I will likely also add support for Astraweb's XZVER implementation, eventually, as well (but not Diablo's XZVER implementation).

On my better-than-dialup 512kbs DSL connection, 3,000,000 headers take about an hour (instead of 2.5). YMMV.

Regards
Back to top
View user's profile Send private message Send e-mail
BZB
Expert
Expert


Joined: 26 Apr 2005
Posts: 1408

PostPosted: Mon Aug 24, 2009 9:58 pm    Post subject: Reply with quote

Ooh, I was just thinking the other day of poking you with a stick to see if any progress had been made! Laughing It's been so quiet around here, we need some extra functionality to wake us up.

I'm looking forward to testing it out. Do you know if Highwinds supports header compression? There's not really much of a way to get hold of them as they seem to be very distanced from end-users and primarily interested in outsourcing NNTP servers for ISPs, so I though it would be easier just to try it out with a client.

If it doesn't work I could always request support through my ISP.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    newsmanpro.com Forum Index : Feature Requests All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Copyright 2003-2006, Daniel Cumpian
NMP Default By ::Dementeddogz.com::