1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Linux What is this Linux thing?

Discussion in 'Software' started by Glider, 27 Jul 2006.

  1. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    Would be good if i explained what i was trying to achieve, wouldn't it. :)

    Basically I'm getting together the commands needed to copy files off of a USB stick, write an ISO to it, recreate a partition and copy the files back to it. I've ended up with this method because GUI based programs write an ISO9660 partition to the entire 64GB drive leaving it unusable as a storage medium, whereas dd left me with 60 odd GB of unpartitioned space afterwards.

    Because i don't want to do it all by hand and risk forgetting the commands when i need to update the live USB/install medium in years to come i though making script to update the USB when needs be would be better.
    Not even when the file ends with an ".iso" extension, like they do? Either way I've also tried simply using "manjaro*" (without the quotations) and it still claims the file "
    manjaro-kde-18.1.1-191015-linux53.iso" that's in the directory it's being run from and it can find when using the full filename isn't there.
     
    Last edited: 30 Oct 2019
  2. Gareth Halfacree

    Gareth Halfacree WIIGII! Staff Administrator Super Moderator Moderator

    Joined:
    4 Dec 2007
    Posts:
    13,135
    Likes Received:
    2,244
    Not even. With "*." you're saying "any string of characters plus any single character," which is redundant unless you're in the very odd scenario of needing to match any string of two or more characters - which you're not.

    You won't be able to do what you're trying to do with dd, 'cos dd doesn't like wildcards (so you don't screw everything up.) What you *can* do is take a different approach - this is Linux, so there's always a dozen ways to do something.

    You could...

    for i in manjaro*iso; do dd if=$i of=/dev/sdb status=progress; done

    That'd do what you're trying to do, but if you've more than one manjaro*iso file it'll write the first, then go back and overwrite it with the second, then go back and overwrite it with the third, and so on and so forth.

    You could...

    find . -name manjaro*iso -exec dd if={} of=/dev/sdb status=progress \;

    But that has the same problem.

    You could...

    for i in $(ls -r manjaro*iso | head -n1); do dd if=$i of=/dev/sdb status=progress; done

    That has the advantage of only doing *one* matching file - the one that's furthest down in the alphabetical sorting order, which means if you have manjaro-2019-08-11.iso and manjaro-2019-10-30.iso it'll write the latter rather than the former.

    In any of these cases, though, you're going to need to be really, really, really careful. Scripting dd is usually a bad idea, 'cos it won't think twice before wiping everything in its path. I'd especially recommend avoiding using /dev/sdX device names as the output file target: there's no guarantee that what was /dev/sdb today will be /dev/sdb tomorrow. If you insist on doing it this way, you're going to want to specify the target via UUID so you don't programmatically nuke something you care about. You can snag 'em via sudo blkid.

    EDIT:
    Or just use Balena Etcher, which is point-'n'-drool. Or right-click on the ISO and choose Disk Image Writer, if your DE has it - and if not, it'll probably have an equivalent.
     
    Corky42 likes this.
  3. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    Top man! :)

    I suspected it was something to do with dd not liking wildcards because of the chance of blowing things up, great to have it confirmed as it can be hard to know if it's my own dumbness or a limitation, I'll look into your suggestions, either one will probably do as there will only ever be one manjaro-whatever.iso in the directory the script will be run from.
    Yea, i have to say I'm really enjoying the freedom, i feel like a little kid who's just learnt something new that he want's to share with everyone even though it's probably old hat to them. :)
    I was going to do that but doesn't the UUID change when the partition table is changed/overwritten as would happen when dd'ing a bootable ISO to a USB.
    It does but it's a bit bloated (something like 100Mb) and their collection of data on what images you've used put me right off, plus I'm guessing it will do what most GUI image writing programs do and write a 2-3GB ISO to the entire 64GB drive leaving me with no space to store anything else.
     
  4. Gareth Halfacree

    Gareth Halfacree WIIGII! Staff Administrator Super Moderator Moderator

    Joined:
    4 Dec 2007
    Posts:
    13,135
    Likes Received:
    2,244
    Fixed that for you. Always assume the worst!
    You'd guess wrong. It's basically dd with an Electron UI (hence the size.) In fact, neither Etcher nor Disk Image Writer will make any attempt to resize a written image: if you write a 1GB image to a 32GB drive, you'll have a 1GB image on a 32GB drive and around 31GB of free space.
     
    Corky42 likes this.
  5. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    Really, I'm going to have to go back and have another look at that then as by the sounds of it i missed something.

    EDIT: quick update, you're correct. They do only write a 1GB image to a 1GB partition, GParted misreports the partition layout.
     
    Last edited: 30 Oct 2019
  6. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    So it's taken me all day to workout what's going on hear and i just wanted to dbl check I've understood correctly, apologies if my wording isn't exact as I've got zero knowledge of programing.

    The section before the ; is setting an environment variable for the program that follows the ; in this case you associated (by using for) the letter i with any (all?) files starting with manjaro and ending in iso, then dd runs and we tell it to use the EV we just set by using the $.
     
  7. Gareth Halfacree

    Gareth Halfacree WIIGII! Staff Administrator Super Moderator Moderator

    Joined:
    4 Dec 2007
    Posts:
    13,135
    Likes Received:
    2,244
    It'll probably make more sense like this:

    Code:
    for i in manjaro*iso;
       do dd if=$i of=/dev/sdb status=progress;
    done
    So, effectively, there are three commands. The first creates a list of filenames matching "manjaro*iso" - so anything which starts "manjaro" and ends "iso" and has something inbetween. The for loop then iterates over the list (which is why we used "i" as the variable name - to indicate it's an iterative variable.) The second then takes one of these filenames, which has been assigned the variable name i, and runs a command with said filename in it as a string ($i). When that command has finished, it'll move onto the next filename that matched, and the next, and the next, then finish (done) when it's run out of matching filenames.

    This, by contrast:

    Code:
    for i in $(ls -r manjaro*iso | head -n1);
        do dd if=$i of=/dev/sdb status=progress;
    done
    Does something similar, except everything in the $() bit is a separate command whose output is treated as a string. In other words, $(echo boo) is functionally equivalent to "boo" - and $(ls -r manjaro*iso | head -n1)" is functionally equivalent to "manjaro-whateverthelastoneinthefolderis.iso".

    Because we're only using one result (head -n1), though, it's a bit of a waste of a for loop. The exact same command could be written as:

    Code:
    dd if=$(ls -r manjaro*iso | head -n1) of=/dev/sdb status=progress
    We're only working on one file, so no loop's needed.

    Alternatively...

    Code:
    dd if=$(ls manjaro*iso | tail -n1) of=/dev/sdb status=progress
    In the first version, we're asking ls to give us a list of files in reverse alphanumeric order then using head to take the first entry only; in this version, we're asking ls to give us a list of files in normal alphanumeric order then using tail to take the last entry only. Functionally, there's no difference. Told you there's always multiple ways to do things in Linux!
     
    Corky42 likes this.
  8. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    Wow, that's way more than i was expecting, thanks for taking the time to go into such detail, that's going to take some time to digest as it's taken me an entire day just trying to workout the first example you gave. :)

    Not that I'm complaining as like i said it's quiet fun trying to work these sort of things out.
     
  9. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    Quick update: I tried your first example and it fell over moaning about something, something, "do" so I've ended up with the following..
    Code:
    myvar=$( ls manjaro*iso ) ; sudo dd bs=4M if=$myvar of=/dev/disk/by-id/usb-Samsung_Flash_Drive_0000000000000000-0:0 status=progress oflag=sync ; done
    I changed the id of the drive in the above into "000.." in the above because serial number. :)

    Now i just need to workout how to recreate the partition from where ^^that^^ ends to the end of the drive, i think I'll leave that for another day though.

    Thanks again for pointing me in the right direction Mr H.
     
  10. Gareth Halfacree

    Gareth Halfacree WIIGII! Staff Administrator Super Moderator Moderator

    Joined:
    4 Dec 2007
    Posts:
    13,135
    Likes Received:
    2,244
    Odd - seems to work for me. You using something other than bash for your shell?

    Code:
    blacklaw@shodan:/media/RAM Disk$ for i in manjaro*iso; do echo dd if=$i of=/dev/sdb status=progress; done
    dd if=manjaro-test.iso of=/dev/sdb status=progress
    
    I'd still recommend using the $(ls manjaro*iso | tail -n1) approach, mind, 'cos as much as you'd like to think you'll remember to remove old ISOs I bet there'll come a time when you forget.

    Probably best using parted for that - 's pretty scriptable.
     
  11. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    Not afaik, 'echo $0' reports /bin/bash, if it's of interest the exact working of the error was "bash: syntax error near unexpected token `do'".
     
  12. Gareth Halfacree

    Gareth Halfacree WIIGII! Staff Administrator Super Moderator Moderator

    Joined:
    4 Dec 2007
    Posts:
    13,135
    Likes Received:
    2,244
    Sounds to me like there's something screwy going on. What happens if you copy and paste from this text file? (It won't do anything, there's an "echo" before "dd" so it'll just print what it *would* have done if the echo wasn't there.)
     
  13. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    It prints "dd if=manjaro-kde-18.1.1-191015-linux53.iso of=/dev/sdb status=progress" to the command line...Something is going on as if i add sudo to the start of it it errors again so I'm guessing it's something to do with that.

    Just to cast aside any did he enter sudo wrongly a copy and past from the cli...
    Code:
    [peter@peter-pc Software]$ sudo for i in manjaro*iso; do echo dd if=$i of=/dev/sdb status=progress; done
    :)
     
  14. Gareth Halfacree

    Gareth Halfacree WIIGII! Staff Administrator Super Moderator Moderator

    Joined:
    4 Dec 2007
    Posts:
    13,135
    Likes Received:
    2,244
    That's not where you want the sudo. Remember, it's three commands: "for," "do", and "done". "sudo for" leaves "do" running as you, not as the root user. You want "do sudo".

    Code:
    for i in manjaro*iso; do sudo echo dd if=$i of=/dev/sdb status=progress; done
    
    Want confirmation it's running as root?

    Code:
    for i in manjaro*iso; do sudo whoami; done
    
     
  15. Corky42

    Corky42 Where's walle?

    Joined:
    30 Oct 2012
    Posts:
    9,532
    Likes Received:
    359
    D'oh!

    I suspected it was something to do with where sudo was placed so i tried it with 'for i ...; sudo do' and got the same error, if only I'd thought about it a bit more as the moment you mentioned "do sudo" i had one of those how did i miss that moments.
     

Share This Page