imapsync couldn't append / NO APPEND failed zimbra invalid name

I am in the process of migrating user mail from a Zimbra ZCS v6.x FOSS BSD server to a Zimbra ZCS v8.x FOSS VMWare appliance Linux server. I used the following Zimbra command line tool to produce a compressed archive for each user via SOAP/REST calls:

/opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL "//?fmt=tgz" > /tmp/account.tgz

..on the source server, and:

/opt/zimbra/bin/zmmailbox -z -m user@domain.com postRestURL "//?fmt=tgz&resolve=reset" /tmp/account.tgz

.. on the destination server. Whilst this appeared to work and was pretty fast, when I did some verifications I got the following results:

Decomissioned server:

[zimbra@kronenbourg /usr/home/user1]$ zmmailbox -z -m user1@somedomain.co.uk
mailbox: user1@somedomain.co.uk, size: 6.80 GB, messages: 105766, unread: 49210
mbox user1@somedomain.co.uk>

[zimbra@kronenbourg /usr/home/user2]$ zmmailbox -z -m user2@somedomain.co.uk
mailbox: user2@somedomain.co.uk, size: 2.15 GB, messages: 23920, unread: 210
mbox user2@somedomain.co.uk>

New VMWare appliance server:

zimbra@kronenbourg:~/redolog/archive$ zmmailbox -z -m user1@somedomain.co.uk
mailbox: user1@somedomain.co.uk, size: 8.00 GB, messages: 112836, unread: 52558
authenticated as user1@somedomain.co.uk
mbox user1@somedomain.co.uk>

zimbra@kronenbourg:~/redolog/archive$ zmmailbox -z -m user2@somedomain.co.uk
mailbox: user2@somedomain.co.uk, size: 2.10 GB, messages: 22823, unread: 210
authenticated as user2@somedomain.co.uk

A little worrying to say the least, so I decided to revisit my old "friend" imapsync and perform a "live" mail migration between servers. First of all I had to go ahead and git clone the latest imapsync source and build it on the VMWare appliance Ubuntu box as apt-get installed a very old version. I also had to grab and install an updated Perl IMAPClient module (http://search.cpan.org/~plobbes/Mail-IMAPClient-3.29/) before imapsync would build. I'm using imapsync v1.558 btw. Here is the command line I settled upon:

nohup imapsync --nofoldersizes --usecache --tmpdir /var/tmp --buffersize 81920000 --nosyncacls --syncinternaldates --subscribe --host1 192.168.0.66 --port1 993 --host2 192.168.0.64 --port2 993 --user1 user1\@somedomain.co.uk --password1 password1 --user2 user2\@somedomain.co.uk --password2 password2 --ssl1 --ssl2 --authmech1 PLAIN --authmech2 PLAIN --reconnectretry1 100000 --reconnectretry2 100000 &

Then I just tailed the nohup.out and went to bed :-)... It was around 6 times slower than the REST export option (however this was over a Powerline network) so YMMV, but it's definitely slower as expected. Reviewing the log I noticed hundreds of errors along the lines of:

msg Flagged/63672 {15067}       copied to Flagged/6801       11.80 msgs/s  314.255 KiB/s
msg Flagged/98800 {30349}       copied to Flagged/6803       11.80 msgs/s  314.254 KiB/s
- msg Flagged/99159 {24629} couldn't append  (Subject:[Re: Join my network on LinkedIn]) to folder Flagged: 6552 NO APPEND failed
msg Flagged/111586 {40960}      copied to Flagged/6804       11.80 msgs/s  314.262 KiB/s
msg Flagged/140168 {25505}      copied to Flagged/6806       11.80 msgs/s  314.255 KiB/s

.. as you can see we had trouble around message ID 6804; and digging through the zimbra.log I found the corresponding entry:

2013-08-07 11:33:38,907 INFO  [ImapSSLServer-12] [name=user1@somedomain.co.uk;mid=19;ip=192.168.0.64;] mailop - Adding Message: id=6803, Message-ID=, parentId=-1, folderId=6745, folderName=Flagged.

2013-08-07 11:33:38,986 INFO  [ImapSSLServer-6] [name=user1@somedomain.co.uk;mid=19;ip=192.168.0.64;] imap - APPEND failed: invalid name: wellknown:FLAG0

So some of my mails are tagged, and some of those tag names contain spaces (bad practice for a developer right? :/). So the solution is to SSH onto the old VMWare server and rename the tags with spaces:

mbox user1@somedomain.co.uk> rt "well known" "well_known"

Since we used the --usecache option with imapsync, next time we re-run the migration it will just copy the missing mails (i.e. the ones with a tag whose name contains a space), and now we are closer:

root@kronenbourg:/opt/zimbra/log# su zimbra
zimbra@kronenbourg:~/log$ zmmailbox -z -m user1@somedomain.co.uk
mailbox: user1@somedomain.co.uk, size: 6.76 GB, messages: 104775, unread: 49035

.. but still missing 991 messages!! Looking back at the imapsync output I see other errors:

Host1 uid 63620 no header by parse_headers so taking whole header with BODY.PEEK[HEADER]
Host1 _SOME_FOLDER_/63620 size 2278 ignored (no wanted headers so we ignore this message)

Seemingly imapsync needs help with some headers (possibly emails without a Message-ID according to one of my friends) so adding the switch (--useheader ALL) fixes that:

nohup imapsync --nofoldersizes --usecache --tmpdir /var/tmp --buffersize 81920000 --nosyncacls --syncinternaldates --subscribe --host1 192.168.0.66 --port1 993 --host2 192.168.0.64 --port2 993 --user1 user1\@somedomain.co.uk --password1 password1 --user2 user2\@somedomain.co.uk --password2 password2 --ssl1 --ssl2 --authmech1 PLAIN --authmech2 PLAIN --reconnectretry1 100000 --reconnectretry2 100000 --useheader ALL &

After all this, one issue still remains, that being certain folders are "virtual" and can't be sync'd via normal IMAP. In my case I had two folders to deal with: "Contacts" and "Emailed Contacts". For these folders we fall back to the REST calls detailed above:

/opt/zimbra/bin/zmmailbox -z -m user1@somedomain.co.uk getRestURL '//?fmt=tgz&query=under:"Emailed Contacts"' >emailedcontacts.tgz
/opt/zimbra/bin/zmmailbox -z -m user1@somedomain.co.uk getRestURL '//?fmt=tgz&query=under:"Contacts"' >contacts.tgz

..on the source server, and:

/opt/zimbra/bin/zmmailbox -z -m bec@xzer0.co.uk postRestURL "//?fmt=tgz&resolve=skip" emailedcontacts.tgz
/opt/zimbra/bin/zmmailbox -z -m bec@xzer0.co.uk postRestURL "//?fmt=tgz&resolve=skip" contacts.tgz

.. on the destination server. Notice the use of the "skip" argument, make sure to use this and not the "reset" argument or else all existing mails will be deleted first! Time to do a final verfication between the servers to get some confidence... we'll use zmmailbox and imapsync and see where we're at:

zmmailbox:

zimbra@kronenbourg:~/log$ zmmailbox -z -m user2@somedomain.co.uk
mailbox: user2@somedomain.co.uk, size: 2.15 GB, messages: 23912, unread: 210

vs

zimbra@kronenbourg:~/log$ zmmailbox -z -m user2@somedomain.co.uk
mailbox: user2@somedomain.co.uk, size: 2.15 GB, messages: 23024, unread: 210

imapsync:

/usr/bin/imapsync --host1 192.168.0.66 --port1 993 --host2 192.168.0.64 --port2 993 --user1 user2@somedomain.co.uk --password1 password1 --user2 user2@somedomain.co.uk --password2 password2 --ssl1 --ssl2 --authmech1 PLAIN --authmech2 PLAIN --dry --justfolders

...

Host1 Nb messages:           23893 messages
Host1 Total size:       2306660541 bytes (2.148 GiB)
Host1 Biggest message:    33794974 bytes (32.229 MiB)

vs

Host2 Nb messages:           23893 messages
Host2 Total size:       2306685481 bytes (2.148 GiB)
Host2 Biggest message:    33794974 bytes (32.229 MiB)

So zmmailbox tells us we still have missing messages, whereas imapsync tells us we're good (well apart from a 25k discrepency). I haven't discovered why I get these differing results but as it stands right now I'm happy enough with the migration, I just hope this helps someone else.

Conclusion: Don't really have an awful lot of faith in the Zimbra CLI tools!