TOC
From joevan@vanudity.com Wed May 22 08:38:37 2002 Subject: Re: To the Group...about Agent-Interesting reading From: joevan Date: Wed, 22 May 2002 08:38:37 -0400
The enclosed text takes care of a lot of the misconceptions about encoding. All binaries posted here [in the newsgroup] are encoded. Read the following through and then you will know about it. I remember a few months ago there were people who were posting or trying to post yEncode using faulty programs. The person who created the encoding system tried to get people to wait until it was perfected which it is now. Any hoo some just plowed ahead and messed it up.
yEnc links and information:
http://www.winews.net/yenc/develop.htm
Newsreaders that support yEnc for Windows Computers: Agent 1.91 (http://www.forteinc.com/main/homepage.php)
(not supported in Freeagent 1.21)
Giganews Binary Newsreader: http://www.uwekeller.de/index.html
Gravity (Build 2034 or 2036):
http://lightning.prohosting.com/~tbates/gravity/ (freeware)
http://lightning.prohosting.com/~tbates/gravity/ (freeware)
NewsBin Version V4.0B22: www.newsbin.com
XNews: http://xnews.3dnews.net/#download (freeware)
NovusNews: http://www.novusnews.com
Nographer: http://nographer.com
NewsRover: http://www.newsrover.com
NewsPro: http://www.usenetopia.com
NewsPro: http://www.usenetpro.com
MyNews: http://www.winews.net (freeware)
BNR Binary News Reader: http://www.co.jyu.fi/~ap/bnr.html
Binary Boy http://www.binaryboy.com
Newsreaders that support yEnc for LINUX:
BNR2: http://www.co.jyu.fi/~ap/bnr.html
Pan: http://pan.rebelbase.com/roadmap.html
http://www.yenc.org
Newsreaders that support yEnc for Apple Computers:
MT News Watcher (OS-X): http://www.smfr.org/mtnw/downloading.html
Thoth - newsreader: http://www.thothsw.com
Rosetta - decoder: http://www.thothsw.com
An article on yEnc:
ALL binaries are encoded in one way or another. Either in UUencode, Base 64, BinHex, Apple Single or Double. Usenet only supported 7-bit ASCII text postings using the standard ASCII set. Since a JPG, MPG or AVI is not a 7-bit text format it must be converted. There is no other way of attaching a file to a posted message other than encoding it in some fashion.
YEnc only adds about 4% overhead to encode a file using it's method. All other forms of encoding use an extra 30% - 40% of overhead to accomplish this task.
See the from-worst-to-best example below of a 10MB file transfer.

Original File: 10,447,908 bytes -------------------------------------------------------------- Format: Base 64 (Windows attempt: less encoding, larger file size) Lines of code: 193,529 Download & Upload Size: 14,141,556 bytes -------------------------------------------------------------- Format: UUencode (With MIME support for error correction) Lines of code: 232,228 Download & Upload Size: 14,412,520 bytes -------------------------------------------------------------- Format: UUencode (No MIME) (The original Unix encoding for binaries) Lines of code: 232,213 Download & Upload Size: 14,412,052 bytes -------------------------------------------------------------- Format: BinHex (Apple's attempt but only a small savings) Lines of code: 218,798 Download & Upload Size: 14,234,196 bytes -------------------------------------------------------------- Format: Apple Double (The best you can get with 7-bit, Windows Support) Lines of code: 193,543 Download & Upload Size: 14,142,042 bytes -------------------------------------------------------------- Format: Apple Single (The best you can get with 7-bit) Lines of code: 193,531 Download & Upload Size: 14,141,805 bytes -------------------------------------------------------------- Format: yEnc (only the first 8-bit encoding method?) Lines of code: 83,997 Download & Upload Size: 10,844,185 bytes

As you can plainly see yEnc adds only 396,277 bytes! to the binary. A savings on your end, the posters end, and to the news servers in general. While all other methods add around 4,000,000 bytes (4MB) to every 10MB you've downloaded. 14MB downloaded for only 10MB of data isn't a good ratio.
Usenet has been able to handle 8-bit text for a long time. This allows for non ASCII characters. Here is a short explanation from the Software Engineers at Toth (Mac newsreader) & Rosetta about why yEnc works and why it doesn't with some programs.

yEnc is different from the usual encoding methods in that it uses non-ASCII characters in the encoded text. Therefore to properly download such attachments prior to decoding, the downloading program has to make sure that it does not strip out control characters or convert 8-bit characters or modify the attachment text in any way. This will rule out using many programs to download yEnc posts before decoding as they will modify the text being downloaded and therefore corrupt it.
Rosetta can properly decode temp files only if they have not been corrupted in this manner during downloading. Because Rosetta is primarily designed as a decoder for usenet binary attachments, where corruption may occur for a variety of reasons, the program does its best to recover as much of the original file as possible without reporting errors or refusing to decode imperfect files. Normally this is an advantage, as attachments may be incomplete or have a bad line of text and still yield a useful file. But in the case of a yEnc attachment downloaded with a program that corrupts yEnc posts, a corrupted yEnc attachment will probably not yield a useful file.

All other attempts at different encoding schemes have fallen short of real savings on bandwidth. Mostly because of 7-bit encoding. YEnc is the first to offer true time savings at encoding time but also on uploads, downloads and extracting. Your gonna have to do it anyway so you might as well use the fastest method. That is the method with the smallest overhead, and that is well dare I say yEnc!
And since it is 8-bit it will also support crc32 verification which will stop a file that even with all it's parts available from being corrupt. Nothing will help with a file that is missing parts, this occurs most of the time on propigation to other newservers. That is why a poster will usually find all parts on the server they posted too but not complete on others. Propigation can be helped by reducing the load on servers. Which at this point can only be done by reducing bandwidth used to support each file or part of a file.
Now if your newsreader doesn't support it, don't come whining to the posters. They are doing us all a big favor by posting the files. Take your act to your software manufacturer and hound them, it is what we pay them for. That's what I did. I was told that yEnc isn't at all difficult to implement, within two weeks I recieved a version that supported yEnc. As a software engineer I've looked into this method and have found it very easy to use and implement. If they tell you otherwise it's due to laziness on their part not with the difficulty in implementaion. If your newsreader forces you to manually decode files, trash it. A computer is here to help you with repetative tasks not add more. If they don't understand that simple concept they shouldn't be writing software in the first place.
Hogwasher 3 will be released shortly for you Mac users out there, with OS X and yEnc support. It's carbon so it will run under OS 9 & X and supports long file names in X. Thoth (commercial) and MT-NewsWatcher (freeware) now support it, but get the newest version. www.versiontracker.com. Windows has been covered ad nausium so I won't go into that.
Make yourself a nuisance in Microsoft's or AOL's house and see how fast they react and support it. As if they actually respect their software users needs. Then when they don't, go find another piece of software to use. If what you're using is free, well you get what you payed for. In the future you will see other 8-bit encoding methods or even 16-bit, and if your software is slow to support new methods you will be left out in the cold again and again.
In the future this type of encoding may also lead to an additional savings of 10% - 25% or more in file size since 8-bit and 16-bit encoding in theory should allow for compression of graphic files.