# refracta.freeforums.org

place to talk about refracta

Skip to content

Welcome
Welcome to refracta You are currently viewing our boards as a guest, which gives you limited access to view most discussions and access our other features. By joining our free community, you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content, and access many other special features. In addition, registered members also see less advertisements. Registration is fast, simple, and absolutely free, so please, join our community today!

## refractasnapshot re-run on a previously saved filesystem

Stuff that has not yet gone into the official build.
Post a reply
11 posts • Page 1 of 2 • 1, 2

### refractasnapshot re-run on a previously saved filesystem

by dzz » Sat Apr 06, 2013 12:05 am

I need ability to rebuild a live image from a saved filesystem, such as refractasnapshot creates.
For example my current Wheezy system. When it was fairly new I made a snapshot. Now I want the live-image updated and tailored specifically for mobile use (not in the same way as the current running system) This involves some manual additions/deletions and chroot for apt stuff (including dist-upgrade)
Previously this was done manually or with custom scripts. However refracta snapshot itself can have a "nocopy" option with some quite minor code additions. Posted is a diff against 9.0.8-2 showing what I did.

Code: Select all
  `179a180,185
  >
  > else
  > if [ "$nocopy" = "yes" ]; then
  > $DIALOG --$ERROR --title="Refracta Snapshot" --text=" Error.. No saved filesystem copy was found.. "
  > exit 1
  > fi
  190a197
  >
  196a204,220
  >
  > # Don't use /media/* for $snapshot_dir unless it is a mounted filesystem
  > snapdir_is_remote=$(echo ${snapshot_dir}| awk -F / '{print "/" $2 "/" $3}' |grep /media/)
  >
  > if [ -n "$snapdir_is_remote" ] && cat /proc/mounts|grep -q ${snapdir_is_remote}; then
  > echo "$snapdir is mounted"
  > else
  > $DIALOG --$ERROR --title="Refracta Snapshot" --text=" Error.. The selected snapshot work directory cannot be accessed.. "
  > exit 1
  > fi
  >
  >
  > if [ "$nocopy" = "yes" ]; then
  > return
  > fi
  >
  >
  278c302
  < $DIALOG --$INFO --text="Excludes file was set to default values.
  ---
  > $DIALOG --$INFO --title="Refracta Snapshot" --text="Excludes file was set to default values.
  281c305
  < $DIALOG --$WARNING --text="Something went wrong. You should check your excludes file.
  ---
  > $DIALOG --$WARNING --title="Refracta Snapshot" --text="Something went wrong. You should check your excludes file.
  296a321
  > FALSE 04 "Use a previously saved filesystem copy." \
  309a335,342
  >
  > if $(echo $opts | grep -q 04); then
  > nocopy="yes"
  > save_work="yes"
  > kernel_image="${work_dir}/myfs/vmlinuz"
  > initrd_image="${work_dir}/myfs/initrd.img"
  > fi
  >
  335c368
  < $DIALOG --$INFO --text="This may take a moment while the program checks for free space. "
  ---
  > $DIALOG --$INFO --title="Refracta Snapshot" --text="This may take a moment while the program checks for free space. "
  347a381,383
  >
  > copy_filesystem () {
  >
  352a389,401
  > # Copy the filesystem
  > rsync -av / myfs/ --delete --exclude="$work_dir" \
  > --exclude="$snapshot_dir" --exclude-from="$snapshot_excludes" \
  > | tee >($DIALOG --title="Copying filesystem..." --progress --pulsate --width 300)
  > kill $(pgrep $DIALOG)
  >
  > }
  >
  >
  > if ! [ "$nocopy" = "yes" ]; then
  > copy_filesystem
  > fi
  >
  361,367d409
  < # Copy the filesystem
  < rsync -av / myfs/ --delete --exclude="$work_dir" \
  < --exclude="$snapshot_dir" --exclude-from="$snapshot_excludes" \
  < | tee >($DIALOG --title="Copying filesystem..." --progress --pulsate --width 300)
  < kill $(pgrep $DIALOG)
  <
  <
  428c470
  < $DIALOG --$QUESTION --title=Edit Boot Menu --${BUTTON0}="Yes"${BUTTON0NUM} --${BUTTON1}="No"${BUTTON1NUM} \
  ---
  > $DIALOG --$QUESTION --title="Edit Boot Menu" --${BUTTON0}="Yes"${BUTTON0NUM} --${BUTTON1}="No"${BUTTON1NUM} \
  `

Another advantage: The original snapshot was 3.5 GB. After dealing manually with some excess stuff which is probably beyond the reasonable scope of snapshot's rsync-exclude list I got it down to 1.1 GB (amazing what lurks in those hidden "dot" files)
One word of warning to anyone who tries this.. Be careful what and how you delete, there might be symlinked directories to outside the saved FS in (most likely but could be elsewhere) $FS/home.. "rm -rf" horror stories are well documented elsewhere.
Note also my patch is "experimental" (tested here and one time only with zero guarantee against data loss) however the idea at least might be a useful addition to official Refractasnapshot. It includes a few other things like fixing some dialog windows that didn't display well and a check for, if a custom "work" dir on another filestem is used, it is actually plugged and mounted.

dzz
  Posts: 647
  Joined: Wed Apr 27, 2011 11:53 am
  Location: Devon, England

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by fsmithred » Sat Apr 06, 2013 1:02 pm

Code: Select all
  `> kernel_image="${work_dir}/myfs/vmlinuz"
  > initrd_image="${work_dir}/myfs/initrd.img"`

I think this should cause errors with the cp command trying to copy a file to itself. Are you seeing that, and if so, is it a problem?
Actually, I think I'll just go ahead and make copy_kernel a function like you did with copy_filesystem, and they both can run (or not) together. Then kernel_image and initrd_image don't need to be redefined.

Code: Select all
  `# Let iso/, vmlinuz and initrd.img get copied, even if work_dir was saved,
  # in case they have changed, unless $nocopy = yes.
  copy_kernel () {
  rsync -a "$iso_dir"/ "$work_dir"/iso/
  cp "$kernel_image" "$work_dir"/iso/live/
  cp "$initrd_image" "$work_dir"/iso/live/
  }
  # Copy the filesystem
  copy_filesystem () {
  rsync -av / myfs/ --delete --exclude="$work_dir" \
  --exclude="$snapshot_dir" --exclude-from="$snapshot_excludes" \
  | tee >($DIALOG --title="Copying filesystem..." --progress --pulsate --width 300)
  kill $(pgrep $DIALOG)
  }
  if ! [ "$nocopy" = "yes" ]; then
  copy_kernel
  copy_filesystem
  fi`

fsmithred
  Posts: 2101
  Joined: Wed Mar 09, 2011 9:13 pm

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by dzz » Sun Apr 07, 2013 1:09 am

> I think this should cause errors with the cp command trying to copy a file to itself. Are you seeing that, and if so, is it a problem?

Not sure what that means, it's getting copied from $work_dir/myfs/ to $work_dir/iso/live/ .. if it's a link a straight cp uses the link's target and overwrites one with the same name in the destination. No such errors in the log, their date stamps show they were copied at that time and the iso boots
But it's wrong anyway because $work_dir/myfs/initrd.img is absolute link to /boot/whatever.img (= the running system's /boot .. in this scenario we want the one from /boot of the saved fs). Curiously vmlinuz link is relative (boot/whatever). In fact this running system has the same kernel but could have been different.Thanks for the prompt to review that.
Something like this might be better, will try later:

Code: Select all
  `# nocopy mode
  kernel_image="$(ls ${work_dir}/myfs/boot|grep -m 1 vmlinuz)"
  initrd_image="$(ls ${work_dir}/myfs/boot|grep -m 1 initrd)"
  # later
  cp ${work_dir}/myfs/boot/${kernel_image} ${work_dir}/iso/live/vmlinuz
  cp ${work_dir}/myfs/boot/${initrd_image} ${work_dir}/iso/live/initrd.img`

Later to work out how to deal with a second kernel, might need to boot something that can't handle PAE
Experimental only so far!

dzz
  Posts: 647
  Joined: Wed Apr 27, 2011 11:53 am
  Location: Devon, England

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by fsmithred » Sun Apr 07, 2013 1:34 pm

Oh. It means that I wasn't looking at the code. Sorry.
Would it help to split up $kernel_image and $kernel_path, so that $kernel_image is the file name (vmlinuz for a default) and the script could adjust $kernel_path for nocopy?
These lines should also be dealt with for nocopy:

Code: Select all
  `# @@@@ Warning: This will replace these files in custom iso_dir @@@@@
  #copy some isolinux stuff from the system to the snapshot
  rsync -a /usr/lib/syslinux/isolinux.bin "$iso_dir"/isolinux/
  rsync -a /usr/lib/syslinux/vesamenu.c32 "$iso_dir"/isolinux/`

Is rsync really needed at all for nocopy? Would it make sense to skip that and go right to mksquashfs?

fsmithred
  Posts: 2101
  Joined: Wed Mar 09, 2011 9:13 pm

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by fsmithred » Sun Apr 07, 2013 3:07 pm

And should the nocopy option make the script pause right before mksquashfs (same as edit_boot_menu=yes) for manual tweaks?

fsmithred
  Posts: 2101
  Joined: Wed Mar 09, 2011 9:13 pm

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by fsmithred » Mon Apr 08, 2013 1:05 am

Do you want the package list to be based on the previously saved copy? Right now, 'dpkg -l' is run on the host system. I think it could chroot $work_dir for the nocopy option.
change this:

Code: Select all
  `# Prepend the dir name with a constant,
  # so you can find and delete the old ones
  # that might have different snapshot basenames.
  dir_prefix="pkglist"
  rm -r "$work_dir"/iso/"$dir_prefix"*
  mkdir -p "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"
  dpkg -l | grep "ii" | awk '{ print $2 }' > "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"/package_list`

to this?

Code: Select all
  `# Prepend the dir name with a constant,
  # so you can find and delete the old ones
  # that might have different snapshot basenames.
  dir_prefix="pkglist"
  rm -r "$work_dir"/iso/"$dir_prefix"*
  mkdir -p "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"
  if [[ $nocopy = "yes" ]] ; then
  chroot "$work_dir" dpkg -l | grep "ii" | awk '{ print $2 }' > "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"/package_list
  else
  dpkg -l | grep "ii" | awk '{ print $2 }' > "$work_dir"/iso/"${dir_prefix}_${filename%.iso}"
  fi
  `

fsmithred
  Posts: 2101
  Joined: Wed Mar 09, 2011 9:13 pm

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by fsmithred » Mon Apr 08, 2013 1:13 am

Here's the rsync section. I think it's all set - no rsync will run if $nocopy = yes. There are a few other preparations that are done after that, for pmount, updatedb, clamav, root ssh, snapshot name, and then it gets squashed.

Code: Select all
  `# The real work starts here
  cd "$work_dir"
  # @@@@ Warning: This will replace these files in custom iso_dir @@@@@
  #copy some isolinux stuff from the system to the snapshot
  copy_isolinux () {
  rsync -a /usr/lib/syslinux/isolinux.bin "$iso_dir"/isolinux/
  rsync -a /usr/lib/syslinux/vesamenu.c32 "$iso_dir"/isolinux/
  }
  # Let iso/, vmlinuz and initrd.img get copied, even if work_dir was saved,
  # in case they have changed, unless $nocopy = yes.
  copy_kernel () {
  rsync -a "$iso_dir"/ "$work_dir"/iso/
  cp "$kernel_image" "$work_dir"/iso/live/
  cp "$initrd_image" "$work_dir"/iso/live/
  }
  # Copy the filesystem
  copy_filesystem () {
  rsync -av / myfs/ --delete-excluded --exclude="$work_dir" \
  --exclude="$snapshot_dir" --exclude-from="$snapshot_excludes" \
  | tee >($DIALOG --title="Copying filesystem..." --progress --pulsate --width 300)
  kill $(pgrep $DIALOG)
  }
  if ! [ "$nocopy" = "yes" ]; then
  copy_isolinux
  copy_kernel
  copy_filesystem
  fi
  `

Another thought was to force a pause before squashing, but I didn't do this one:

Code: Select all
  `if [[ $nocopy = "yes" ]] ; then
  edit_boot_menu="yes"
  fi
  `

fsmithred
  Posts: 2101
  Joined: Wed Mar 09, 2011 9:13 pm

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by dzz » Mon Apr 08, 2013 2:41 pm

Don't see a reason to pause, the whole idea is ability do do some manual tweaks in the saved fs before rebuild, presumable that is already done (including menus if needed) It shouldn't matter if those few preperations get run again
The package lists would be different only if a user had chrooted it with bind mounts and run apt (rather than simple file additions/deletions).
Doing that is possible but gets more complex. The bind mounts can be difficult to umount (but a reboot sorts them) because services might have been started in the chroot. Also some stuff listed in the rsync excludes gets written (log files, apt files, bash history..) and would need finding and deleting manually.
If you know how to handle that you don't need help updating a package list!

dzz
  Posts: 647
  Joined: Wed Apr 27, 2011 11:53 am
  Location: Devon, England

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by fsmithred » Tue Apr 09, 2013 1:58 pm

refractasnapshot-9.0.9 (base and gui packages)
http://distro.ibiblio.org/refracta/testing/

Code: Select all
  `* Added test for remote, unmounted snapshot_dir or work_dir
  * Added more items to excludes, especially root's hidden files.
  * Changed rsync option '--delete' to '--delete-excludes'
  * Added nocopy option to bypass rsync.
  `

To use the nocopy option with the text-only version, run it with the -n (--nocopy) option on the command-line.

Code: Select all
  `refractasnapshot -n`

fsmithred
  Posts: 2101
  Joined: Wed Mar 09, 2011 9:13 pm

Top

### Re: refractasnapshot re-run on a previously saved filesystem

by thwak » Tue Sep 03, 2013 9:00 am

the gui version begs inclusion of some explanatory text.
When it pauses, a yad(?) screen presents 5 checkboxes with 5 too-short "labels".
User wonders "Can I checkmark more than one?" and (later) "why am I seeing this same screen again?"
Also, it would be helpful if some text (technical, or non-technical, in addition to progress bar) displayed what step/action the script is currently performing.

thwak
  Posts: 174
  Joined: Tue Nov 20, 2012 3:58 am

Top

Next Display posts from previous:
Sort by

Post a reply
11 posts • Page 1 of 2 • 1, 2

Return to Experimental

Jump to:

### Who is online

Users browsing this forum: No registered users and 0 guests

Donate Now
ProNight Style by ModphpBB3
Hosted by © 2017 FreeForums.org | Create a free forum | Powered by phpBB
A Member of the Crowdgather Forum Community
About FreeForums | Legal | Advertise Here | Investors | Contact FreeForums.org
Report Violation
suspicion-preferred