1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Linux How-to use the cli to download large files from the web to your 'Glider Server'

Discussion in 'Software' started by steveo_mcg, 6 Dec 2007.

  1. steveo_mcg

    steveo_mcg New Member

    Joined:
    26 May 2005
    Posts:
    5,841
    Likes Received:
    80
    This quick how-to should help you download large files from the web where no torrent file is available or trustworthy. For example downloading a game demo from the publisher’s site to your server without having to open vnc or leave your main computer running. For this were going to use screen and wget. Both are cli so ssh into your box and we can get started.
    First we need to install screen, screen is a tool to create virtual terminals which aren’t connected to either your xterm or console.
    Code:
     sudo apt-get install screen 
    Once its downloaded and installed you can call it using
    Code:
     screen 
    this shows a brief message introducing screen with an option to terminate or continue for now just hit return to terminate.

    Next job is to find the file you want. Using your desktop and browser find the file you want to download for this example we'll download a fairly small file but this can be used for anything for example that 2 gig Bioshock demo. For this to work you need a direct link to the file to download so for a demo this will usually end .zip or .exe. For this example we'll just use a picture from the bit front page... Gliders flux desktop will do http://www.bit-tech.net/content_images/2007/11/bashing_through_scripts/for_l.jpg.

    Were going to use the command line program wget, this is a basic but powerful tool which gets things from the web and is installed by default on all Debian based machines. Couple this with screen and you have a background-able and more importantly re-selectable download manager which can tell you how far along your downloads are.

    so cd to your shared files folder
    Code:
     cd /home/shared/ 
    and run
    Code:
     wget http://www.bit-tech.net/content_images/2007/11/bashing_through_scripts/for_l.jpg.
    this should save the image to your shared directory. Obviously this isn't terribly useful so we need a larger file to show off screen.

    Still in the shared dir run
    Code:
     screen wget http://us.download.nvidia.com/downloads/nZone/demos/nzd_BioShockPC.zip 
    this is a sufficiently large file that we can play. wget will start in its default stat you'll see the percentage downloaded, a visual display of percentage download, the volume downloaded, the current speed and an eta. The power of screen is that sessions is not attached directily to the session you called it from, now you can close your connection and shutdown your noisy gaming computer your session of screen is still running quite happily. Now you get up in the morning and want to find out what the progress of your file is simply ssh back in to the server and run
    Code:
     screen -r
    this will reconnect you to your session pretty much where your left it though hopefully further along with the download. If it fails to reconnect the file is probably complete and screen will have terminated.

    The old hands will probably be familiar with these tools and may have there own methods, but i thought i might help some of the newer users get a bit more from there shiny new server. Comments criticism all welcome.
     
    Last edited: 7 Dec 2007
  2. cyrilthefish

    cyrilthefish New Member

    Joined:
    15 Apr 2004
    Posts:
    1,363
    Likes Received:
    99
    I love things like this: the kind of info thats nearly impossible to look up unless you've heard of the program before, that new linux user guides won't mention, and experienced linux users would assume you knew :)
     
  3. woodshop

    woodshop UnSeenly

    Joined:
    14 Oct 2003
    Posts:
    1,408
    Likes Received:
    8
    can also use links2 the text web browser over ssh.

    Just log in via SSH and the command is.
    Code:
    links2 http://blablabla
    
    just press d to download. it will save to the current directory as per when you launched it.
     
  4. notatoad

    notatoad pretty fing wonderful

    Joined:
    25 Jun 2004
    Posts:
    3,213
    Likes Received:
    60
    great guide. i've been looking for exactly this ever since i set up my server last month.
     
  5. Woodstock

    Woodstock So Say We All

    Joined:
    10 Sep 2006
    Posts:
    1,783
    Likes Received:
    2
    a couple of quick questions about screen and wget, can you have multiple sessions of screen running? and does wget support downloading from multiple servers
     
  6. Glider

    Glider /dev/null

    Joined:
    2 Aug 2005
    Posts:
    4,173
    Likes Received:
    21
    The answer to that question is yes and yes...

    As 1 user, you can have multiple screens running (you can list them with "screen -ls")
    AND you can attach a screen at multiple places (attach using "screen -x")

    Given the right config you can even have fancy stuff like multiuser screens, or tabbed screens (with statusbars and stuff)

    Gentoo to the rescue!

    http://gentoo-wiki.com/TIP_Using_screen

    PS: I'm an absolute screen-whore... Use it everywhere with multiple tabs open at the same time...
     
  7. DougEdey

    DougEdey I pwn all your storage

    Joined:
    5 Jul 2005
    Posts:
    13,933
    Likes Received:
    33
    Screen supports multiple screens. Run

    Code:
    screen -S NAME wget ...
    to run screen with a name of NAME

    ctrl+a, d will detach this if needs be
    then running
    Code:
    screen -r
    will list windows and show how to reconnect.

    Usually just running

    Code:
    screen -r NAME
    is enough
     
  8. adrock

    adrock Caninus Nervous Rex

    Joined:
    5 Dec 2006
    Posts:
    1,265
    Likes Received:
    43
    i use screen on everything, our media box, my disk server, a remote shell account. it holds my irc client so that i can reconnect and check if ive missed anything, lets me set something playing remotely on the media box without havign to keep the session open to it. i've got several apps running in screens, such as a vnc server thats then running azureus on my file server, one for a usenet client and web interface etc etc.

    for me it's one of the fundamental advantages o running linux. if only you could have screened xapps :(
     
  9. Glider

    Glider /dev/null

    Joined:
    2 Aug 2005
    Posts:
    4,173
    Likes Received:
    21
    You could start X in a screen ;)
     
  10. simon w

    simon w New Member

    Joined:
    3 Nov 2003
    Posts:
    1,302
    Likes Received:
    0
    I'm not intending to downplay the usefulness of screen (I use it a lot myself), but wget has a -b option which makes it to run as a background process :)

    Edit - sounds like we need a 'show use your .screenrc + plus picy' thread :geek: :p
     
    Last edited: 13 Dec 2007
Tags:

Share This Page