Switching a mail provider is often combined with one major thought holding one back: What will happen to all my e-mails? I don't want to lose my e-mails!
Sure, if you used POP3 to download the e-mails to your mail client, the e-mails are not stored on the server anymore. This worked fine in the 90s and 00s but since mobile Internet there are usually multiple devices connected to your mailbox: Notebook, Workstation, Smartphone or even a Tablet. In such a scenario, only IMAP (IMAPv4 to be precise) can do the job: All the e-mails remain on the server (in the mailbox) but are synced to the local mail client. Every change in the local mail client is synced back to the server. Hence all the devices see the same status.
For a customer of Infiniroot we recently needed to find a way to migrate all the existing e-mails (more than 28'000) from a long-existing mailbox to a new mailbox located on the dedicated server of this customer. This is where we came across imapsync.
imapsync is a migration tool for mailboxes using the IMAP protocol. On the official website, imasync is described as:
Imapsync is an IMAP transfer tool. The purpose of imapsync is to migrate IMAP accounts or to backup IMAP accounts. IMAP, IMAP4 in fact (started December 1994), is one of the three current standard protocols used to access mailboxes, the two other being POP3 (started November 1988) and HTTP (started May 1996) with webmails, webmails are often tied to an IMAP server.
The best thing: imapsync is developed as Open Source Software. It is therefore free to use. However the author, Gilles Lamiral, offers additional support and quickly available imapsync for Windows (imapsync.exe), too. Windows users without knowledge how to build an executable exe file from a Perl source might want to consider this.
imapsync can be downloaded right from the public Git repository on GitHub. It is recommended to use the latest release. At the time of this writing 1.977 was the newest available released version. The newest tar.gz release is downloaded using wget into a directory called “build” and is then extracted using tar:
ck@linux ~ $ mkdir build
ck@linux ~ $ cd build/
ck@linux ~/build $ wget https://github.com/imapsync/imapsync/archive/1.977.tar.gz
ck@linux ~/build $ tar -xzf 1.977.tar.gz
Change into the unpacked imapsync folder and there are a couple of sub folders and files:
ck@linux ~/build $ cd imapsync-1.977/
ck@linux ~/build/imapsync-1.977 $ ls -la
total 1912
drwxrwxr-x 10 ckadm ckadm 4096 Jun 17 2020 ./
drwxrwxr-x 7 ckadm ckadm 4096 Dec 29 13:53 ../
-rw-rw-r-- 1 ckadm ckadm 197011 Jun 17 2020 ChangeLog
-rw-rw-r-- 1 ckadm ckadm 29457 Jun 17 2020 CREDITS
lrwxrwxrwx 1 ckadm ckadm 5 Jun 17 2020 dist -> dist2
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 doc/
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 examples/
lrwxrwxrwx 1 ckadm ckadm 21 Jun 17 2020 FAQ -> FAQ.d/FAQ.General.txt
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 FAQ.d/
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 .github/
-rw-rw-r-- 1 ckadm ckadm 177 Jun 17 2020 .gitignore
-rwxrwxr-x 1 ckadm ckadm 152 Jun 17 2020 i3*
-rwxrwxr-x 1 ckadm ckadm 647598 Jun 17 2020 imapsync*
-rwxrwxr-x 1 ckadm ckadm 604052 Jun 17 2020 imapsync-1.945*
-rw-rw-r-- 1 ckadm ckadm 29958 Jun 17 2020 index.shtml
lrwxrwxrwx 1 ckadm ckadm 25 Jun 17 2020 INSTALL -> INSTALL.d/INSTALL.ANY.txt
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 INSTALL.d/
-rw-rw-r-- 1 ckadm ckadm 320 Jun 17 2020 LICENSE
-rw-rw-r-- 1 ckadm ckadm 30815 Jun 17 2020 Makefile
-rw-rw-r-- 1 ckadm ckadm 320 Jun 17 2020 NOLIMIT
-rw-rw-r-- 1 ckadm ckadm 36936 Jun 17 2020 README
-rw-rw-r-- 1 ckadm ckadm 37780 Jun 17 2020 README.md
lrwxrwxrwx 1 ckadm ckadm 6 Jun 17 2020 README.txt -> README
-rw-rw-r-- 1 ckadm ckadm 5800 Jun 17 2020 README_Windows.txt
drwxrwxr-x 3 ckadm ckadm 4096 Jun 17 2020 S/
-rw-rw-r-- 1 ckadm ckadm 193577 Jun 17 2020 tests.sh
-rw-rw-r-- 1 ckadm ckadm 41612 Jun 17 2020 TODO
-rw-rw-r-- 1 ckadm ckadm 6 Jun 17 2020 VERSION
drwxrwxr-x 12 ckadm ckadm 4096 Jun 17 2020 W/
-rwxrwxr-x 1 ckadm ckadm 4725 Jun 17 2020 webserver*
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 X/
The most important file is already existing: imapsync, the program (a Perl script) itself. No need to compile anything. However there are a lot of Perl modules which must be installed as requirements of imapsync. Besides already having had a few Perl modules installed, the following modules needed to be installed additionally (package names for Debian and Ubuntu):
ck@linux ~ $ sudo apt-get install libio-tee-perl libmail-imapclient-perl libterm-readkey-perl libunicode-string-perl libreadonly-perl libsys-meminfo-perl libregexp-common-perl libfile-tail-perl libencode-imaputf7-perl
If there are still Perl modules missing, imapsync will exit with an error pointing to the missing module. Once all the required Perl modules are installed, imapsync can be launched and will by default show the help/usage information:
ck@linux ~/build/imapsync-1.977 $ ./imapsync
Name:
imapsync - Email IMAP tool for syncing, copying, migrating and archiving
email mailboxes between two imap servers, one way, and without duplicates.
Version:
This documentation refers to Imapsync $Revision: 1.977 $
Usage:
To synchronize the source imap account
"test1" on server "test1.lamiral.info" with password "secret1"
to the destination imap account
"test2" on server "test2.lamiral.info" with password "secret2"
do:
imapsync \
--host1 test1.lamiral.info --user1 test1 --password1 secret1 \
--host2 test2.lamiral.info --user2 test2 --password2 secret2
Options:
[...]
If you are used to work with rsync, you quickly know how to use imapsync, too. Just as rsync, imapsync transfers (migrates) the mails from a source to a target. A source is defined by using the parameters --host1, --user1 and --password1. The target is defined by, you guessed it already, by the parameters --host2, --user2 and --password2. Additional options (such as verbose output) are available, too.
| |
Source Mailbox |
Target Mailbox |
| IMAP Host |
--host1 | --host2 |
| IMAP Username | --user1 | --user2 |
| IMAP Password |
--password1 | --password2 |
It is always wise to first start a mailbox migration with a simulation a.k.a. dry-run. The parameter --dry allows to simulate an IMAP migration. This also helps to verify the given credentials for the mailboxes. At the end of each imapsync run (with or without --dry), statistics of the migration will be shown.
ck@linux ~/build/imapsync-1.977 $ ./imapsync --dry --host1 source.example.com --user1 source@example.com --password1 12345678 --host2 target.example.com --user2 target@example.com --password2 87654321
[...]
++++ Statistics
Transfer started on : Wed Dec 30 10:07:32 2020
Transfer ended on : Wed Dec 30 10:08:13 2020
Transfer time : 40.9 sec
Folders synced : 262/262 synced
Messages transferred : 0 (could be 49 without dry mode)
Messages skipped : 0
Messages found duplicate on host1 : 0
Messages found duplicate on host2 : 0
Messages found crossduplicate on host2 : 0
Messages void (noheader) on host1 : 0
Messages void (noheader) on host2 : 0
Messages found in host1 not in host2 : 49 messages
Messages found in host2 not in host1 : 5 messages
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 0 (0.000 KiB)
Total bytes skipped : 0 (0.000 KiB)
Message rate : 0.0 messages/s
Average bandwidth rate : 0.0 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption at the end : 263.6 MiB (started with 221.6 MiB)
Load end is : 1.42 1.43 1.35 4/1434 on 4 cores
Biggest message : 0 bytes (0.000 KiB)
Memory/biggest message ratio : NA
Start difference host2 - host1 : -28558 messages, -8405244919 bytes (-7.828 GiB)
Final difference host2 - host1 : -28558 messages, -8405244919 bytes (-7.828 GiB)
The sync is not finished, there are 49 identified messages in host1 that are not on host2.
There is no unidentified message
The sync is not strict, there are 5 messages in host2 that are not on host1. Use --delete2 to delete them and have a strict sync. (5 identified messages in host2)
Detected 0 errors
The statistics of the dry-run also show how many mailbox folders (262) and e-mails (28558) will be synced. The simulated run obviously runs pretty quickly – it took around 40s to complete.
Time to do the real sync now. As, in this situation, more than 28k mails need to be transferred, this can take a while...
ck@linux ~/build/imapsync-1.977 $ ./imapsync --host1 source.example.com --user1 source@example.com --password1 12345678 --host2 target.example.com --user2 target@example.com --password2 87654321
Host1 Nb folders: 262 folders
Host2 Nb folders: 262 folders
Host1 Nb messages: 28568 messages
Host2 Nb messages: 28538 messages
Host1 Total size: 8405740214 bytes (7.828 GiB)
Host2 Total size: 8385683883 bytes (7.810 GiB)
Host1 Biggest message: 72919837 bytes (69.542 MiB)
Host2 Biggest message: 72919837 bytes (69.542 MiB)
Time spent on sizing: 21.5 seconds
++++ Statistics
Transfer started on : Wed Dec 30 10:09:36 2020
Transfer ended on : Wed Dec 30 11:51:20 2020
Transfer time : 6103.7 sec
Folders synced : 262/262 synced
Messages transferred : 28532
Messages skipped : 36
Messages found duplicate on host1 : 22
Messages found duplicate on host2 : 0
Messages found crossduplicate on host2 : 0
Messages void (noheader) on host1 : 14 in order to sync those 14 unidentified messages, add option --addheader
Messages void (noheader) on host2 : 0
Messages found in host1 not in host2 : 0 messages
Messages found in host2 not in host1 : 6 messages
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 8385026114 (7.809 GiB)
Total bytes skipped : 351682 (343.439 KiB)
Message rate : 4.7 messages/s
Average bandwidth rate : 1341.6 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption at the end : 411.2 MiB (started with 221.6 MiB)
Load end is : 1.41 1.60 1.85 1/1461 on 4 cores
Biggest message : 72919837 bytes (69.542 MiB)
Memory/biggest message ratio : 5.9
Start difference host2 - host1 : -28558 messages, -8405244919 bytes (-7.828 GiB)
Final difference host2 - host1 : -30 messages, -20056331 bytes (-19.127 MiB)
The sync looks good, all 28528 identified messages in host1 are on host2.
There are 14 unidentified messages (usually Sent or Draft messages). To sync them add option --addheader
The sync is not strict, there are 6 messages in host2 that are not on host1. Use --delete2 to delete them and have a strict sync. (28534 identified messages in host2)
Detected 0 errors
Finally after 6103 seconds (~1h 41min), the IMAP migration with more than 7.8 GB was completed and the target mailbox contained 28558 mails.
Important to note: All e-mail headers remain unchanged. This also means that the dates of each e-mail (received, sent) remain the same. The same views and the same order of the e-mails are showing up in the new mailbox.
Infiniroot provides professional and dedicated mail servers. This means the mailboxes are stored on dedicated servers, no other mailboxes from other customer are located on this mailbox server. We will help migrating your e-mails from your current mail hosting - obviously using imapsync. Contact us for more information.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder