Setup Guide - Usenet

Before starting downloading from Newsgroups, you must sign up to a Usenet access service, these would have been included with your ISP long ago but with declining popularity and the sheer amount of data the service has to store forced many companies discontinued the option, or if it remains will be limited to text only newsgroups.

Newsdemon is recommended as a low cost option that offers reliable connections and a long retention (how long the files remain available after they posted). There are free options available if you’re prepared to search, but most come with slow download speeds, unreliable and generally don’t last longer than a few weeks at best.

Software

The Usenet service is accessed via a specialised program that can connect and download information from the servers. Some Usenet service providers offer a free or discounted copy of a downloader when signing up, but if not a free downloader called SABnzbd is available to get started with downloading.

SABnzbd essentially runs as a service in the background on the computer which makes it great for home servers due to it using a minimal amount of system resources. What makes it different and a bit strange is that the service is controlled via its own build in web server and accessed via a web browser, but when used to it the process is simple, plus with build in PAR correction and RAR extracting SABnzbd is the easiest approach to getting files from Usenet.

SABnzbd user interface

If more control and personal vetting of data is wanted then its best to stick with a traditional program that is dedicated to downloading, then adding separate correction and extraction utilities to the system. Unfortunately free programs for this are hard to come by, but as a paid option I recommend Newsbin for its easy to understand layout that’s also easy to control.

Whichever option chosen, the setup involves creating a connection to the service provider, this is easily done by using the information provided when signing up to a service.

Downloading

Choosing what to download from Usenet can be achieved in one of two ways. The first and more traditional is to select a group and download headers to see what’s available, where the groups and thought of as a genre of data and the headers are similar to the subject field on an email, where a brief description of what’s in the post is made.

This used to be the only way to use Usenet, but to the huge amount of data involved today it may be time consuming and a bandwidth hog to get headers for a popular group. More recently the use of .nzb files have made Usenet much easier to get data from, instead of searching raw data yourself a nzb file acts a pointer that instructs a downloader exactly what to download and from which location.

It’s more likely that an nzb file will be found for a download rather than referencing a Usenet group, so downloading is as easy as opening a nzb file in the downloader, or copying the file to the downloader’s watch folder, that actively scans the folder for nzb files, and when found starts the download automatically.

If using a Usenet program as a downloader only then there are a few more steps to convert the downloaded files into the data you want. It also comes in handy if the all in one solution like SABNZB fails to extract files...

Handling the Data

When download completes what's usually left a number of Rar files and possibly a few Par files. Due to the way the Usenet was originally a text based messaging system that got adapted to include files, larger content needs to be spit up across a number of "posts" to allow users to get all the data. This is where Rar files comes into play, Rar's are like .zip files that compress data but also have the ability to be split into multiple files that when all are available, can be decoded into a single copy of the original files.

File view of a typical Usenet download

Accessing the real data you want from a download can be handles by 7zip, an open source program that can compress files in a number of formats, but for this case can decompress and rebuild Rar files into usable data.

Install this program, and on the first run allow it to associate with Rar files. Then to extract files simple double click on the file with a .Rar or .Part.01 extension to see the archive, then copy it to another location just like a zip file.

The Usenet system isn't perfect, with high amounts of data flowing between different servers some data may be lost, and for Rar files to work all the sections or parts need to be available to extract the data. Par files here sit atop of the Rar files to act as a kind of redundancy to fill in the gaps where rar files are missing data. All the Rar files are segregated into blocks and if some of those blocks are missing, as long as there are an equal or greater  quantity of Par blocks available, the data is recreated to fill in the gaps where data is missing.

The ability to recreate missing data can only be attributed to magic or witchcraft, but can be a powerful tool that can allow the extraction of data that would otherwise be rendered useless.

Working with Par files requires installing QuickPar, another open source program, while it hasn't left beta stage and with the last update made in 2004 it's likely that it's never going to, it still works perfectly in reconstructing Rar files. Install and again on the first run allow it to associate with .par files.