Spiffy Way To Migrate All Installed Packages Between Linux Ubuntu Servers (Upgrades Too!)

Migrate Linux Servers

Are you looking for ways to Migrate Linux Servers between multiple installations? I’ve experimented with various Linux tools & configurations over the years & sometimes there is a need to migrate between Virtual Machines or hardware. It takes a considerable amount of time to have to reconfigure a Linux Ubuntu server from scratch & fine tune everything, over & over again.

One of the better guides that I’ve seen to help get started out with the process of getting all the installed packages from the old Linux Ubuntu Server to the new one can be found at: https://www.cyberciti.biz/faq/how-to-list-all-installed-packages-debian-ubuntu

What I like the most about this guide is the variety of options & possibilities it provides to accomplish installing (or perhaps re-installing, in some cases) the packages from the original Linux Ubuntu Server setup.

The Missing Steps to Migrate Linux Servers

Over the years, I’ve come across a TON of tutorials on this subject that seem to leave out the method for cleaning up the output file for parsing. Unfortunately, the Cyberciti.biz guide I shared is no exception.

For the section:

How to save a list to a text file

The syntax is as follows on server1:

$ dpkg --get-selections | grep -v deinstall > installed_packages_list.txt

This ends up giving you a text file formatted like this:

 

You will get gripes about “install” not being a package in APT (or YUM for CentOS & Red Hat) if you skip straight to using:

 

How to Fix Exported Installed Packages List File For Use With ‘apt-get install’

You need to get rid of all the “orphaned” ‘install’ strings between the tabs & newline chars, & then properly space the text of the installed package list first. There may also be a difference in platform such as amd64 vs i386 that you need to account for. If your platform requires a specific platform version of a package, you don’t (typically?) need to include the xxxxx:amd64 or xxxxx:i386 in the package name when installing, the system will (or should!) determine that for you “automagically”. You WILL need to pick out the packages that you DO NOT want to install all in one shot, still. You will likely get some “nags” about specific packages when there is a problem.

To do that, I’m a fan of this Perl5 one-liner:

NOTICE: The above command CREATES A SECOND FILE. You can’t try to RegEx it like that without creating the new file because it will just blank it out completely. So PLEASE take note of that. You can either save one (or both) to keep an reference of the installed packages on your system(s) or delete them. I will leave those decisions in your very capable hands.

NOTE FOR THE “EDGE CASE” FOLKS: The above command DOES require ‘perl’ to be installed on your system! I’m 99.99% sure Perl comes as part of the “standard installation packages” in (probably) every Debian-based (even *Unix-based?) distros with very few exceptions like those you might find with a SUPER tiny footprint like you might use for a Raspberry Pi or something for a specific “dedicated purposes” like that; eg. DSL – “Damn Small Linux” appears to have limited Perl, which may or may NOT be enough – *Digression: Activated!*

This aforementioned Perl one-liner tweaks the contents of the ORIGINAL file with your “unclean list” into a NEW file with a list of installed packages to look like this instead:

... dpkg e2fslibs e2fsprogs ed eject emacsen-common fail2ban ...


Now that this file is all cleaned up & ‘all on one line’ it will actually work with that final step using “apt-get install -y”, like so:

CAPTAIN OBVIOUS SAYS: You will have to work out how you want to move this file over to your new server.

WARNING: You don’t really NEED to go through all the junk involved in setting up your FTP/SCP or whatever other thing some of you weirdos like to use (eg. WebDav? Really? For THIS?…). If this is just for a local VM installed on the same machine or in the subnet of your network: you *COULD* just go through your SSH console (I assume 99.9999% of readers are going to some kind of VM environment) & replace this section: ‘$(< packages_list.txt)’ with a big, gigantic, nasty, old-fashioned COPY+PASTE to get ‘r’ dun. I’m just warning you though: It’s NOT awesome to do that large of a copy/paste through certain kinds of VNC-type connections (eg. TeamViewer) but *COULD* work through PuTTY/KiTTY/ConEmu/Putty SSH Commander or whatever your annoyingly obscure & hipster SSH tool of choice is. I am going to recommend you just move the file over to your new server & FINISHED READING THE REST OF THIS POST BEFORE YOU PERFORM THAT MASSIVE INSTALL.

Let Me Guess: Your Situation Has a Caveat Not Mentioned

There has yet to be a guide or even a “migration script” that can account for every single nuance involved in migrating between Linux Ubuntu Server installations. Guides like these are just that: Guides. I have tried to create posts that fulfill the glaring holes in available information on the web. With as many websites & forum pages as there are on the internet today, it’s still kind of incredible to stumble upon issues & questions that have either non-existent information for them, or its splintered & scattered around in multiple locations & needs congealed into an organized resource.

The Manual Intervention Required for Your Unique Linux Server Migration Requirements

You will STILL NEED TO “cherry pick” the installed packages list from the properly formatted version the above Linux command-line commands will result in! For example, if you use both Apache & Nginx together in a local reverse proxy configuration, you won’t be able to install both Apache (package name: ‘apache2’ in Linux Ubuntu Server 16.04.3 LTS & 17.04+) & Nginx (package name: ‘nginx’, respectively) at the same time because upon initial installation, they both require port 80 to be open. The quickest fix for getting them both installed is to install one of them first, ‘stop’ the service running, then run the apt-get install -y ‘package-name’ after. For example, if we have installed ‘apache2’ first (which seems to be the case, most of the time) then we would need to do the following to get ‘nginx’ installed:

In the above command line examples, you may swap ‘apache2’ with ‘nginx’ for your needs. Before you will be able to run them both together on the same server, you will need to change at least one of them to not use port 80 first. It gets even more tricky when you are running SSL connections on port 443, etc, but since this is a guide to help migrate between servers, there is a good chance that you will simply be able to move your OLD configuration files onto the new server (eg: SCP/SFTP/FTP the old /etc/apache2/ & /etc/nginx/ directories over to the new server in the identical directory paths) & simply reboot afterwards.

|_|1+!|\\/|4+3 1!|\\||_|% |-|4(|(3|2 (|-|34+! => (Ultimate Linux Hacker Cheat!)

You can actually just re-run the ENTIRE ‘installed_packages.txt’ file-reading / copy-pasted list installation line again & again until all the packages are “properly cherry picked” & installed. Unless you are doing something super goofy, it shouldn’t mess anything up. If the packages are already installed it will just skippi-dee-do-dah right over all those that are already installed & try to install anything that had issues before. So no need to hunt down all the package names individually. You may have a few non-apt Repository packages, but you should be able to figure out how to get them onto your new machine if you were rockin’ them in your last setup.

To make things mildly less tedious in re-configuring a new IP address / hostname / mirror you could dive into your /etc/ directory & pluck out the relevant configuration files on your old server to use for use in overwriting the new machines settings. Doing this *CAN* be a time saver since you could then just grep/eyeball scan each config file for the few lines you may need to change. If you are migrating between DIFFERENT VERSIONS of a Linux OS, this becomes an EXTREMELY bad idea though. If the environments match though, you could quickly crank out a mirror for something like a development, testing / QA, staging and/or load-balanced server situation (though using a disk cloned image or restoring from a full backup might be quite a bit easier).

Final Thoughts About (Debian) Linux Ubuntu Server Migrations

I also ran into issues with applications like EHCP (Easy Hosting Control Panel), EHCP (Force Edition) or Webmin where the installation packages need to be manually downloaded from outside the APT repositories. You will likely need to manually download those packages again & then you might be able to import the configuration to the new server that matches the old one. There is a chance you will have to manually reconfigure a few things yourself again though. Such as the /etc/network/interfaces file to configure a static IP address for the new server that may be on a different network from the old server.

If anyone reading this comes across an issue that they would like some help with regarding Linux Ubuntu Server migration (or any Debian-based Linux Server distro, really) post your questions in the comments section & I will try to help you out.

UPDATE: The well respected (and frankly brilliant) Vivek Gite accepted my comment into the very popular blog page that I linked to above (along with a kind note of gratitude): https://www.cyberciti.biz/faq/how-to-list-all-installed-packages-debian-ubuntu/#comment-920788