Linux How-to use the cli to download large files from the web to your 'Glider Server'

This quick how-to should help you download large files from the web where no torrent file is available or trustworthy. For example downloading a game demo from the publisher’s site to your server without having to open vnc or leave your main computer running. For this were going to use screen and wget. Both are cli so ssh into your box and we can get started.
First we need to install screen, screen is a tool to create virtual terminals which aren’t connected to either your xterm or console.

Code:

 sudo apt-get install screen 

Once its downloaded and installed you can call it using

Code:

 screen 

this shows a brief message introducing screen with an option to terminate or continue for now just hit return to terminate.

Next job is to find the file you want. Using your desktop and browser find the file you want to download for this example we'll download a fairly small file but this can be used for anything for example that 2 gig Bioshock demo. For this to work you need a direct link to the file to download so for a demo this will usually end .zip or .exe. For this example we'll just use a picture from the bit front page... Gliders flux desktop will do http://www.bit-tech.net/content_images/2007/11/bashing_through_scripts/for_l.jpg.

Were going to use the command line program wget, this is a basic but powerful tool which gets things from the web and is installed by default on all Debian based machines. Couple this with screen and you have a background-able and more importantly re-selectable download manager which can tell you how far along your downloads are.

so cd to your shared files folder

Code:

 cd /home/shared/ 

and run

Code:

 wget http://www.bit-tech.net/content_images/2007/11/bashing_through_scripts/for_l.jpg.

this should save the image to your shared directory. Obviously this isn't terribly useful so we need a larger file to show off screen.

Still in the shared dir run

Code:

 screen wget http://us.download.nvidia.com/downloads/nZone/demos/nzd_BioShockPC.zip 

this is a sufficiently large file that we can play. wget will start in its default stat you'll see the percentage downloaded, a visual display of percentage download, the volume downloaded, the current speed and an eta. The power of screen is that sessions is not attached directily to the session you called it from, now you can close your connection and shutdown your noisy gaming computer your session of screen is still running quite happily. Now you get up in the morning and want to find out what the progress of your file is simply ssh back in to the server and run

Code:

 screen -r

this will reconnect you to your session pretty much where your left it though hopefully further along with the download. If it fails to reconnect the file is probably complete and screen will have terminated.

The old hands will probably be familiar with these tools and may have there own methods, but i thought i might help some of the newer users get a bit more from there shiny new server. Comments criticism all welcome.

 

I love things like this: the kind of info thats nearly impossible to look up unless you've heard of the program before, that new linux user guides won't mention, and experienced linux users would assume you knew

 

can also use links2 the text web browser over ssh.

Just log in via SSH and the command is.

Code:

links2 http://blablabla

just press d to download. it will save to the current directory as per when you launched it.

 

great guide. i've been looking for exactly this ever since i set up my server last month.

 

a couple of quick questions about screen and wget, can you have multiple sessions of screen running? and does wget support downloading from multiple servers

 

The answer to that question is yes and yes...

As 1 user, you can have multiple screens running (you can list them with "screen -ls")
AND you can attach a screen at multiple places (attach using "screen -x")

Given the right config you can even have fancy stuff like multiuser screens, or tabbed screens (with statusbars and stuff)

Gentoo to the rescue!

http://gentoo-wiki.com/TIP_Using_screen

PS: I'm an absolute screen-whore... Use it everywhere with multiple tabs open at the same time...

 

Screen supports multiple screens. Run

Code:

screen -S NAME wget ...

to run screen with a name of NAME

ctrl+a, d will detach this if needs be
then running

Code:

screen -r

will list windows and show how to reconnect.

Usually just running

Code:

screen -r NAME

is enough

 

i use screen on everything, our media box, my disk server, a remote shell account. it holds my irc client so that i can reconnect and check if ive missed anything, lets me set something playing remotely on the media box without havign to keep the session open to it. i've got several apps running in screens, such as a vnc server thats then running azureus on my file server, one for a usenet client and web interface etc etc.

for me it's one of the fundamental advantages o running linux. if only you could have screened xapps

 

You could start X in a screen

 

I'm not intending to downplay the usefulness of screen (I use it a lot myself), but wget has a -b option which makes it to run as a background process

Edit - sounds like we need a 'show use your .screenrc + plus picy' thread