stuartgillibrand.co.uk

Saturday, June 07, 2014

4096x4096 framebuffer on X windows VM

I've been doing a lot of Linux development on a Virtual Machine (ESX) recently and figured it would be nice to leverage the virtual VGA adapter to produce a 4096x4096 X resolution on my Xubuntu install.

There are two ways I utilise this massive desktop, one way is just using the scrollbars on the console view to facilitate my development environment with loads of terminal windows / etc. The other way is to stack three laptops side by side, each running VSphere client and each having a console open to the same VM instance, thereby creating a super wide "remoted" monitor session (with three keyboards/mice too!). You could even add a second or third monitor to each laptop and have a total of six / nine screens to better make use of the super extended vertical aspect (budget permitting of course ;-)).

Here are the command lines I used to coerce X windows into 4096x4096:

$ xrandr --newmode 4096x4096_60 1460.75 4096 4464 4920 5744 4096 4099 4109 4239 -hsync +vsync

$ xrandr --addmode Virtual1 4096x4096_60

The 4096x4096 mode will now be available in your display properties.

FYI the modeline was generated using the cvt tool:

$ cvt 4096 4096 60

Friday, June 06, 2014

Free digital certificates

A colleague of mine kindly pointed me to http://www.startssl.com as a source of FREE digital certificates. Fill your boots...

Friday, January 31, 2014

Using inkscape to generate svg objects from a PDF

I have a client who has a collection of storyboard / wireframe PDF documents describing end-user interaction with a company website, and my client wanted me to design and build a website that could closely match the PDFs. There is a particular page which shows a kind of user/vehicle history in a vertical format, where each individual historical event is rendered as a "bubble":

I could see no other way to create such a result without using Canvas or SVG. In the end I went with SVG for this particular application and leveraged a great application called Inkscape to pull the individual "bubble" out of the PDF and export it into SVG format:

This produced a fairly verbose SVG file:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->

<svg
   xmlns:dc="http://purl.org/dc/elements/1.1/"
   xmlns:cc="http://creativecommons.org/ns#"
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
   width="744.09448819"
   height="1052.3622047"
   id="svg19746"
   version="1.1"
   inkscape:version="0.48.4 r9939"
   sodipodi:docname="New document 6">
  <defs
     id="defs19748">
    <clipPath
       clipPathUnits="userSpaceOnUse"
       id="clipPath8551">
      <path
         inkscape:connector-curvature="0"
         d="m 0,5177.253 3126.964,0 L 3126.964,0 0,0 0,5177.253 z"
         id="path8553" />
    </clipPath>
  </defs>
  <sodipodi:namedview
     id="base"
     pagecolor="#ffffff"
     bordercolor="#666666"
     borderopacity="1.0"
     inkscape:pageopacity="0.0"
     inkscape:pageshadow="2"
     inkscape:zoom="0.35"
     inkscape:cx="375"
     inkscape:cy="520"
     inkscape:document-units="px"
     inkscape:current-layer="layer1"
     showgrid="false"
     inkscape:window-width="463"
     inkscape:window-height="453"
     inkscape:window-x="225"
     inkscape:window-y="225"
     inkscape:window-maximized="0" />
  <metadata
     id="metadata19751">
    <rdf:RDF>
      <cc:Work
         rdf:about="">
        <dc:format>image/svg+xml</dc:format>
        <dc:type
           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
        <dc:title></dc:title>
      </cc:Work>
    </rdf:RDF>
  </metadata>
  <g
     inkscape:label="Layer 1"
     inkscape:groupmode="layer"
     id="layer1">
    <g
       transform="matrix(1.25,0,0,-1.25,-1385.6255,4051.4283)"
       id="g8547">
      <g
         id="g8549"
         clip-path="url(#clipPath8551)">
        <g
           id="g8555"
           transform="translate(1275.4824,2776.2529)">
          <path
             inkscape:connector-curvature="0"
             d="m 0,0 c -5.79,0 -10.482,5.003 -10.482,11.344 l 0,56 C -10.482,73.428 -6.084,78 0,78 l 188,0 c 5.99,0 13.449,-3.583 16.981,-8.057 L 221.802,48.552 C 225.049,44.438 232.106,41 237.211,41 l 39.307,0 0,-4 -39.307,0 c -5.356,0 -12.27,-3.316 -15.411,-7.296 L 204.983,8.317 C 201.345,3.709 193.728,0 188,0 L 0,0 z"
             style="fill:#c1212c;fill-opacity:1;fill-rule:nonzero;stroke:none"
             id="path8557" />
        </g>
        <g
           id="g8559"
           transform="translate(1463.4824,2855.2529)">
          <path
             inkscape:connector-curvature="0"
             d="m 0,0 -188,0 c -6.556,0 -11.482,-5.1 -11.482,-11.656 l 0,-56 c 0,-6.893 5.141,-12.344 11.482,-12.344 l 188,0 c 5.994,0 13.964,3.84 17.766,8.656 l 16.822,21.471 C 37.511,-46.17 44.209,-43 49.211,-43 l 20.789,0 17.5,0 2.018,0 0,1.844 0,2 0,2.156 -2.018,0 -17.5,0 -20.789,0 c -4.842,0 -11.54,3.225 -14.627,7.135 L 17.77,-8.402 C 14.065,-3.711 6.262,0 0,0 m 0,-2 c 5.5,0 12.788,-3.359 16.196,-7.677 L 33.015,-31.065 C 36.423,-35.383 43.711,-39 49.211,-39 l 20.789,0 17.518,0 0,-2 -17.518,0 -20.789,0 c -5.5,0 -12.788,-3.359 -16.196,-7.677 L 16.196,-70.065 C 12.788,-74.383 5.5,-78 0,-78 l -188,0 c -5.5,0 -9.482,4.844 -9.482,10.344 l 0,56 c 0,5.5 3.982,9.656 9.482,9.656 l 187.518,0"
             style="fill:#d7d6d4;fill-opacity:1;fill-rule:nonzero;stroke:none"
             id="path8561" />
        </g>
      </g>
    </g>
  </g>
</svg>

After some common sense analysis, I trimmed it down to the following:

<svg
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   >
  <g
     id="layer1">
    <g
       transform="matrix(1.25,0,0,-1.25,-1369.1969,3999.9997)"
       id="g8547">
      <g
         id="g8549"
         clip-path="url(#clipPath8551)">
        <g
           id="g8555"
           transform="translate(1275.4824,2776.2529)">
          <path
             d="m 0,0 c -5.79,0 -10.482,5.003 -10.482,11.344 l 0,56 C -10.482,73.428 -6.084,78 0,78 l 188,0 c 5.99,0 13.449,-3.583 16.981,-8.057 L 221.802,48.552 C 225.049,44.438 232.106,41 237.211,41 l 39.307,0 0,-4 -39.307,0 c -5.356,0 -12.27,-3.316 -15.411,-7.296 L 204.983,8.317 C 201.345,3.709 193.728,0 188,0 L 0,0 z"
             style="fill:#c1212c;fill-opacity:1;fill-rule:nonzero;stroke:none"
             id="path8557" />
        </g>
        <g
           id="g8559"
           transform="translate(1463.4824,2855.2529)">
          <path
             d="m 0,0 -188,0 c -6.556,0 -11.482,-5.1 -11.482,-11.656 l 0,-56 c 0,-6.893 5.141,-12.344 11.482,-12.344 l 188,0 c 5.994,0 13.964,3.84 17.766,8.656 l 16.822,21.471 C 37.511,-46.17 44.209,-43 49.211,-43 l 20.789,0 17.5,0 2.018,0 0,1.844 0,2 0,2.156 -2.018,0 -17.5,0 -20.789,0 c -4.842,0 -11.54,3.225 -14.627,7.135 L 17.77,-8.402 C 14.065,-3.711 6.262,0 0,0 m 0,-2 c 5.5,0 12.788,-3.359 16.196,-7.677 L 33.015,-31.065 C 36.423,-35.383 43.711,-39 49.211,-39 l 20.789,0 17.518,0 0,-2 -17.518,0 -20.789,0 c -5.5,0 -12.788,-3.359 -16.196,-7.677 L 16.196,-70.065 C 12.788,-74.383 5.5,-78 0,-78 l -188,0 c -5.5,0 -9.482,4.844 -9.482,10.344 l 0,56 c 0,5.5 3.982,9.656 9.482,9.656 l 187.518,0"
             style="fill:#d7d6d4;fill-opacity:1;fill-rule:nonzero;stroke:none"
             id="path8561" />
        </g>
      </g>
    </g>
  </g>
</svg>

Finally, I removed some un-needed "g" container elements, removed the clip path (as it's not needed since we have a single object) and corrected the "translate" (x/y offsetting) by normalising the lowest pair to 0,0 (in this case 1275.4824,2776.2529) and then subtracted those values from the remaining translate transforms (1463.4824,2855.2529). This now gives us a shape that is drawn from a ZERO offset. One final task was to tweak the transform matrix (1.25,0,0,-1.25,-1369.1969,3999.9997) so we take the scale down to 1.0 and again correct the x/y positioning so the whole SVG is now at a zero offset. Most of these tweaks are needed simply because of the way Inkscape exported the original object to SVG:

<svg
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   >
  <g
       transform="matrix(1.0,0,0,-1.0,12,79)"
       >
        <g
   id="g8555">
          <path
             d="m 0,0 c -5.79,0 -10.482,5.003 -10.482,11.344 l 0,56 C -10.482,73.428 -6.084,78 0,78 l 188,0 c 5.99,0 13.449,-3.583 16.981,-8.057 L 221.802,48.552 C 225.049,44.438 232.106,41 237.211,41 l 39.307,0 0,-4 -39.307,0 c -5.356,0 -12.27,-3.316 -15.411,-7.296 L 204.983,8.317 C 201.345,3.709 193.728,0 188,0 L 0,0 z"
             style="fill:#c1212c;fill-opacity:1;fill-rule:nonzero;stroke:none"
             id="path8557" />
   </g>
        <g
           id="g8559"
           transform="translate(188,79)">
          <path
             d="m 0,0 -188,0 c -6.556,0 -11.482,-5.1 -11.482,-11.656 l 0,-56 c 0,-6.893 5.141,-12.344 11.482,-12.344 l 188,0 c 5.994,0 13.964,3.84 17.766,8.656 l 16.822,21.471 C 37.511,-46.17 44.209,-43 49.211,-43 l 20.789,0 17.5,0 2.018,0 0,1.844 0,2 0,2.156 -2.018,0 -17.5,0 -20.789,0 c -4.842,0 -11.54,3.225 -14.627,7.135 L 17.77,-8.402 C 14.065,-3.711 6.262,0 0,0 m 0,-2 c 5.5,0 12.788,-3.359 16.196,-7.677 L 33.015,-31.065 C 36.423,-35.383 43.711,-39 49.211,-39 l 20.789,0 17.518,0 0,-2 -17.518,0 -20.789,0 c -5.5,0 -12.788,-3.359 -16.196,-7.677 L 16.196,-70.065 C 12.788,-74.383 5.5,-78 0,-78 l -188,0 c -5.5,0 -9.482,4.844 -9.482,10.344 l 0,56 c 0,5.5 3.982,9.656 9.482,9.656 l 187.518,0"
             style="fill:#d7d6d4;fill-opacity:1;fill-rule:nonzero;stroke:none"
             id="path8561" />
        </g>
    </g>
</svg>

... and the resultant individual SVG object:

Quite a nice result... all that is left now is to get the rest of the site working and the positioning correct ;-)

Thursday, August 22, 2013

using existing legacy geom mirror disks with nas4free / freenas / later versions of BSD

I am in the process of retiring my old FreeBSD 5.5 based file server and replacing it with a more modern BSD distro that is aimed specifically at file serving duties. The setup of the current storage is all the disks are identical pairs and are configured using geom_mirror (RAID1) for redundancy. I'll be using a new motherboard and memory on the new server but will be re-using the existing disks. This got me to thinking if it would be a simple case of just plugging in the disks and having the new OS recognise them as containing GEOM metadata and thus allowing me to make use of them with minimal hassle?

I decided to prove or disprove this theory using my VMWare ESXi server.

Firstly, I created a FreeBSD 5.5 VM, and then a FreeBSD 9.0 VM. I decided to do it this way instead of using the actual NAS4Free distro as I know the NAS4Free distro is based upon FreeBSD9, so the end result should be the same!

I add two small extra disks to the 5.5 box and then go ahead and configure a RAID1 mirror:

$ su
Password:
bsd55# uname -a
FreeBSD bsd55 5.5-RELEASE FreeBSD 5.5-RELEASE #0: Tue May 23 14:58:27 UTC 2006     root@perseus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
bsd55# kldload /boot/kernel/geom_mirror.ko
bsd55# gmirror label gm055 da1 da2
bsd55# gmirror status
        Name    Status  Components
mirror/gm055  COMPLETE  da1
                        da2
bsd55# newfs /dev/mirror/gm055
/dev/mirror/gm055: 512.0MB (1048572 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 128.00MB, 8192 blks, 16384 inodes.
super-block backups (for fsck -b #) at:
 160, 262304, 524448, 786592
bsd55# mount /dev/mirror/gm055 /mnt
bsd55# echo 'proof of conecpt' >/mnt/poc
bsd55# ls -l /mnt
total 4
drwxrwxr-x  2 root  operator  512 Aug 21 22:43 .snap
-rw-r--r--  1 root  wheel      17 Aug 21 22:49 poc
bsd55# cat /mnt/poc
proof of conecpt
bsd55# shutdown now
Shutdown NOW!
shutdown: [pid 574]
bsd55#                                                                          
*** FINAL System shutdown message from nf@bsd55 ***
System going down IMMEDIATELY



System shutdown time has arrived

Next I turn the power off on the 5.5 VM and then edit the settings for the 9.0 VM using vSphere client. What I am doing here is attaching the 5.5 mirrored disks to the 9.0 box:

Now I proceed to power on the 9.0 box and ssh into it to see what the new kernel makes of the old disks:

$ su
Password:
You have mail.
root@hg:/usr/home/nf # uname -a
FreeBSD hg 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:10 UTC 2012     root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
root@hg:/usr/home/nf # kldload /boot/kernel/geom_mirror.ko
root@hg:/usr/home/nf # dmesg
GEOM_MIRROR: Upgrading metadata on da2 (v3->v4).
GEOM_MIRROR: Device mirror/gm055 launched (2/2).
GEOM_MIRROR: Upgrading metadata on da1 (v3->v4).
root@hg:/usr/home/nf # fsck -t ufs /dev/mirror/gm055
** /dev/mirror/gm055
** Last Mounted on /mnt
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
3 files, 3 used, 247891 free (27 frags, 30983 blocks, 0.0% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****
root@hg:/usr/home/nf # mount /dev/mirror/gm055 /mnt
root@hg:/usr/home/nf # ls -l /mnt
total 4
drwxrwxr-x  2 root  operator  512 Aug 22 00:51 .snap
-rw-r--r--  1 root  wheel      17 Aug 22 00:51 poc
root@hg:/usr/home/nf # cat /mnt/poc
proof of conecpt

As can be seen above, the kernel recognised the legacy geom mirrored disks and performed a metadata upgrade from v3 to v4. To further this I prove that the original data can be read back.

So if anyone is thinking of moving GEOM based disks from an old BSD box to a newer distribution, fear not - as usual BSD is rock steady and can be relied upon!

Wednesday, August 07, 2013

imapsync couldn't append / NO APPEND failed zimbra invalid name

I am in the process of migrating user mail from a Zimbra ZCS v6.x FOSS BSD server to a Zimbra ZCS v8.x FOSS VMWare appliance Linux server. I used the following Zimbra command line tool to produce a compressed archive for each user via SOAP/REST calls:

/opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL "//?fmt=tgz" > /tmp/account.tgz

..on the source server, and:

/opt/zimbra/bin/zmmailbox -z -m user@domain.com postRestURL "//?fmt=tgz&resolve=reset" /tmp/account.tgz

.. on the destination server. Whilst this appeared to work and was pretty fast, when I did some verifications I got the following results:

Decomissioned server:

[zimbra@kronenbourg /usr/home/user1]$ zmmailbox -z -m user1@somedomain.co.uk
mailbox: user1@somedomain.co.uk, size: 6.80 GB, messages: 105766, unread: 49210
mbox user1@somedomain.co.uk>

[zimbra@kronenbourg /usr/home/user2]$ zmmailbox -z -m user2@somedomain.co.uk
mailbox: user2@somedomain.co.uk, size: 2.15 GB, messages: 23920, unread: 210
mbox user2@somedomain.co.uk>

New VMWare appliance server:

zimbra@kronenbourg:~/redolog/archive$ zmmailbox -z -m user1@somedomain.co.uk
mailbox: user1@somedomain.co.uk, size: 8.00 GB, messages: 112836, unread: 52558
authenticated as user1@somedomain.co.uk
mbox user1@somedomain.co.uk>

zimbra@kronenbourg:~/redolog/archive$ zmmailbox -z -m user2@somedomain.co.uk
mailbox: user2@somedomain.co.uk, size: 2.10 GB, messages: 22823, unread: 210
authenticated as user2@somedomain.co.uk

A little worrying to say the least, so I decided to revisit my old "friend" imapsync and perform a "live" mail migration between servers. First of all I had to go ahead and git clone the latest imapsync source and build it on the VMWare appliance Ubuntu box as apt-get installed a very old version. I also had to grab and install an updated Perl IMAPClient module (http://search.cpan.org/~plobbes/Mail-IMAPClient-3.29/) before imapsync would build. I'm using imapsync v1.558 btw. Here is the command line I settled upon:

nohup imapsync --nofoldersizes --usecache --tmpdir /var/tmp --buffersize 81920000 --nosyncacls --syncinternaldates --subscribe --host1 192.168.0.66 --port1 993 --host2 192.168.0.64 --port2 993 --user1 user1\@somedomain.co.uk --password1 password1 --user2 user2\@somedomain.co.uk --password2 password2 --ssl1 --ssl2 --authmech1 PLAIN --authmech2 PLAIN --reconnectretry1 100000 --reconnectretry2 100000 &

Then I just tailed the nohup.out and went to bed :-)... It was around 6 times slower than the REST export option (however this was over a Powerline network) so YMMV, but it's definitely slower as expected. Reviewing the log I noticed hundreds of errors along the lines of:

msg Flagged/63672 {15067}       copied to Flagged/6801       11.80 msgs/s  314.255 KiB/s
msg Flagged/98800 {30349}       copied to Flagged/6803       11.80 msgs/s  314.254 KiB/s
- msg Flagged/99159 {24629} couldn't append  (Subject:[Re: Join my network on LinkedIn]) to folder Flagged: 6552 NO APPEND failed
msg Flagged/111586 {40960}      copied to Flagged/6804       11.80 msgs/s  314.262 KiB/s
msg Flagged/140168 {25505}      copied to Flagged/6806       11.80 msgs/s  314.255 KiB/s

.. as you can see we had trouble around message ID 6804; and digging through the zimbra.log I found the corresponding entry:

2013-08-07 11:33:38,907 INFO  [ImapSSLServer-12] [name=user1@somedomain.co.uk;mid=19;ip=192.168.0.64;] mailop - Adding Message: id=6803, Message-ID=, parentId=-1, folderId=6745, folderName=Flagged.

2013-08-07 11:33:38,986 INFO  [ImapSSLServer-6] [name=user1@somedomain.co.uk;mid=19;ip=192.168.0.64;] imap - APPEND failed: invalid name: wellknown:FLAG0

So some of my mails are tagged, and some of those tag names contain spaces (bad practice for a developer right? :/). So the solution is to SSH onto the old VMWare server and rename the tags with spaces:


mbox user1@somedomain.co.uk> rt "well known" "well_known"

Since we used the --usecache option with imapsync, next time we re-run the migration it will just copy the missing mails (i.e. the ones with a tag whose name contains a space), and now we are closer:

root@kronenbourg:/opt/zimbra/log# su zimbra
zimbra@kronenbourg:~/log$ zmmailbox -z -m user1@somedomain.co.uk
mailbox: user1@somedomain.co.uk, size: 6.76 GB, messages: 104775, unread: 49035

.. but still missing 991 messages!! Looking back at the imapsync output I see other errors:

Host1 uid 63620 no header by parse_headers so taking whole header with BODY.PEEK[HEADER]
Host1 _SOME_FOLDER_/63620 size 2278 ignored (no wanted headers so we ignore this message)

Seemingly imapsync needs help with some headers (possibly emails without a Message-ID according to one of my friends) so adding the switch (--useheader ALL) fixes that:

nohup imapsync --nofoldersizes --usecache --tmpdir /var/tmp --buffersize 81920000 --nosyncacls --syncinternaldates --subscribe --host1 192.168.0.66 --port1 993 --host2 192.168.0.64 --port2 993 --user1 user1\@somedomain.co.uk --password1 password1 --user2 user2\@somedomain.co.uk --password2 password2 --ssl1 --ssl2 --authmech1 PLAIN --authmech2 PLAIN --reconnectretry1 100000 --reconnectretry2 100000 --useheader ALL &

After all this, one issue still remains, that being certain folders are "virtual" and can't be sync'd via normal IMAP. In my case I had two folders to deal with: "Contacts" and "Emailed Contacts". For these folders we fall back to the REST calls detailed above:

/opt/zimbra/bin/zmmailbox -z -m user1@somedomain.co.uk getRestURL '//?fmt=tgz&query=under:"Emailed Contacts"' >emailedcontacts.tgz
/opt/zimbra/bin/zmmailbox -z -m user1@somedomain.co.uk getRestURL '//?fmt=tgz&query=under:"Contacts"' >contacts.tgz

..on the source server, and:

/opt/zimbra/bin/zmmailbox -z -m bec@xzer0.co.uk postRestURL "//?fmt=tgz&resolve=skip" emailedcontacts.tgz
/opt/zimbra/bin/zmmailbox -z -m bec@xzer0.co.uk postRestURL "//?fmt=tgz&resolve=skip" contacts.tgz

.. on the destination server. Notice the use of the "skip" argument, make sure to use this and not the "reset" argument or else all existing mails will be deleted first! Time to do a final verfication between the servers to get some confidence... we'll use zmmailbox and imapsync and see where we're at:

zmmailbox:

zimbra@kronenbourg:~/log$ zmmailbox -z -m user2@somedomain.co.uk
mailbox: user2@somedomain.co.uk, size: 2.15 GB, messages: 23912, unread: 210

zimbra@kronenbourg:~/log$ zmmailbox -z -m user2@somedomain.co.uk
mailbox: user2@somedomain.co.uk, size: 2.15 GB, messages: 23024, unread: 210

imapsync:

/usr/bin/imapsync --host1 192.168.0.66 --port1 993 --host2 192.168.0.64 --port2 993 --user1 user2@somedomain.co.uk --password1 password1 --user2 user2@somedomain.co.uk --password2 password2 --ssl1 --ssl2 --authmech1 PLAIN --authmech2 PLAIN --dry --justfolders

...

Host1 Nb messages:           23893 messages
Host1 Total size:       2306660541 bytes (2.148 GiB)
Host1 Biggest message:    33794974 bytes (32.229 MiB)

vs

Host2 Nb messages:           23893 messages
Host2 Total size:       2306685481 bytes (2.148 GiB)
Host2 Biggest message:    33794974 bytes (32.229 MiB)

So zmmailbox tells us we still have missing messages, whereas imapsync tells us we're good (well apart from a 25k discrepency). I haven't discovered why I get these differing results but as it stands right now I'm happy enough with the migration, I just hope this helps someone else.

Conclusion: Don't really have an awful lot of faith in the Zimbra CLI tools!

Tuesday, July 16, 2013

mirror/scrape/backup/download/archive all Atlassian JIRA issues (and attachments) present in my activity stream

I was looking for a way to make a full local backup of all JIRAs that are mentioned in my JIRA activity feed (which equates to all JIRA issues I have ever worked on or updated/commented/etc.).

Pre-requisites:

To accomplish this I've used standard Unix/POSIX tools, so you'll need to use Cygwin if you're doing this on Windows (everybody that runs Windows has Cygwin installed anyway, right? :)).

wget: Make sure you are using a recent version of wget as we use quite a few command line switches, I'm using v1.13.4.
xmllint: Used for executing XPath against the returned XML DOM of our activity feed.
tr, awk: For standard stream processing / etc.
Atlassian JIRA: My tests are against v6.0 of the "On Demand" version of the product (in other words hosted by Atlassian), I'm hoping/guessing this will work for a locally managed "Download" version. You must have the "activity stream" gadget installed and accessible for the user profile for which you are performing this against.

Commands:

Make a directory to store the output:

mkdir jiraSuck

Login to Atlassian JIRA website by providing your username and password via POST data, saving the cookies so we can maintain the session (obviously replace my username with the your required username).
(please note in my example the JIRA server I am querying is the "On Demand" type which means it is hosted by Atlassian as a subdomain of atlassian.net):

wget --keep-session-cookies --max-redirect 0 --no-check-certificate --save-cookies cookies.txt --post-data 'username=sgillibrand&password=YOURPASSWORD' https://JIRASUBDOMAIN.atlassian.net/login

Get an XML stream of ALL your activity by asking for a large max results figure (999999):

wget --no-check-certificate --load-cookies cookies.txt -O jiraActivity.xml "https://JIRASUBDOMAIN.atlassian.net/activity?maxResults=999999&streams=user+IS+sgillibrand&os_authType=basic&title=undefined"

We accomplish quite a few things with this next line. Note, as the returned XML representation of our activity is littered with numerous namespaces I'm using the local-name() functionality of XPath to enable us to operate in a namespace agnostic way. So first we extract all the HREFs/URLs for each JIRA issue mentioned in each activity entry, then we place each URL on a seperate line, delete the 'href=' and remove all double quotes. Next we remove all duplicate URLs and finally create an alternate URL based upon the current URL which will give us a printable version of the JIRA issue (this is handy is it contains all field data expanded) - all the output is redirected to a file jiraUrls.txt:

xmllint --xpath //\*\[local\-name\(\)\=\'entry\'\]/\*\[local\-name\(\)\=\'target\'\]/\*\[local\-name\(\)\=\'link\'\]/@href jiraActivity.xml | tr " " \\n | awk 'sub(/href=/, "")' | awk 'gsub(/"/, "")' | awk '!x[$0]++' | awk -F "/" '{printf "%s\n%s/%s/%s/si/jira.issueviews:issue-html/%s/%s.html\n",$0,$1,$2,$3,$5,$5}' >jiraUrls.txt

Change into our newly created directory:

cd jiraSuck

Now scrape all the JIRA issue standard HTML, printable HTML and specified attachments - this can take some time!
(change the acceptable file extensions and domains to suit your needs):

wget --no-check-certificate -nc -r -k -p -l 1 -E --accept=.jpg,.png,.zip,.7z,.rar,.html,.htm,.xls,.ppt,.xlsx,.doc,.docx,.pptx --restrict-file-names=windows domains=atlassian.net --load-cookies ../cookies.txt -i ../jiraUrls.txt

Some time later..............

FINISHED --2013-07-16 16:30:46--
Total wall clock time: 30m 49s
Downloaded: 856 files, 373M in 25m 6s (253 KB/s)

Explanation of the resultant directory structure:

browse directory contains the normal HTML view of each JIRA issue.
si directory contains the printable view of each JIRA issue.
secure contains any attachments associated with each JIRA issue.

All pertinent links have been converted to point relatively to your local directory structure.

Job done :)

Wednesday, July 03, 2013

Tracking focus in Java

After a lot of experimenting I found a reliable way to track which controls are getting focus:

Component currFocus = null;

KeyboardFocusManager.getCurrentKeyboardFocusManager().addPropertyChangeListener(
  new PropertyChangeListener(){
    public void propertyChange(PropertyChangeEvent e){
      String prop = e.getPropertyName();
      if ((prop.equals("focusOwner"))){
        if(e.getNewValue() != null){
          currFocus = (Component)e.getNewValue();
        }
      }
    }
  }
);

Tuesday, February 19, 2013

Create a windows form initially hidden

I've recently had the need to create a hidden windows .NET form (from creation) and found numerous suggestions, however the most simple and elegant way seems to be:

this->ShowInTaskbar = false;
this->WindowState = System::Windows::Forms::FormWindowState::Minimized

.. before you create the window/message pump with Application::Run().

Wednesday, August 22, 2012

ODL vs IDL

Copied for longevity

(from the great Igor Tandetnik):

The difference is not between ODL and IDL - you can change the file
extension to .idl and it will behave exactly the same way. The
difference is in the file structure.

The way MIDL compiler works, .h file is generated based on the
declarations _outside_ the library block. TLB is generated based on the
declarations inside the library block. The usual approach is to defined
all the interfaces (and structs and enums) outside the library block,
then mention them inside (usually as part of coclass statement). This
way, you would get them both in the header file and in TLB.

Your ODL file, I bet, defines everything inside a library block.

(from the great Alexander Nickolov):

If you deselect ODL mode for your MFC project, you must also
rename it with IDL extension since it's no longer an ODL file.
ODL = Object Description Language - only describes type libraries
IDL = Interface Definition Language - describes both interfaces and
type libraries (contains ODL as a subset).

Historically, initially IDL coudn't be used for type libraries, so
Microsoft invented ODL. Later they joined the two languages
under the IDL name. IDL itself is inherited from OSF DCE RPC
spec and extended to support COM interfaces. It does contain
the original IDL as a subset - you can still use it to define RPC
interfaces (e.g. don't specify [object] for the interface).

Wednesday, August 08, 2012

Catching HTTP 401/403 using Java SWT IE Browser

I've been working on a bit of Java code which needs to handle an HTTP proxy authentication failure, the code in question uses IE via the SWT browser and unfortunately I've discovered that you cannot chain onto the NavigateError event!

The event is handled inside IE.java (SWT source) but just does some specialised UNC handling without giving you the opportunity to do something useful (like catch the HTTP status code). As such I've had to restort to using an additional connection using HttpURLConnection and Authenticator.setDefault() / getPasswordAuthentication() - which is wasteful as I'm now having to make two connections to the webserver...

It's not even as if you can derive a new class from org.eclipse.swt.browser.IE because the required member variables have default visibility which means anything outside the org.eclipse.swt.browser package can't access them :(

Wednesday, May 30, 2012

a useful reminder for converting wide strings to narrow (ATL)

LPCSTR pszNarrow = NULL;
USES_CONVERSION;

pszNarrow = W2CA(pszWide);

Tuesday, March 20, 2012

howto pause for a finite number of seconds in a dos batch file

You can leverage the choice command for pausing for <x> number of seconds:

choice /D y /T 1 >NUL

(substituting the '1')

Tuesday, March 13, 2012

Tortoise SVN password decryption

Need to retrieve the locally stored encrypted password that Tortoise SVN stores for each of your repositories?

http://www.leapbeyond.com/ric/TSvnPD/

Saved my bacon today :)

Friday, February 24, 2012

User profile migration XP/Vista/Windows 7

My client was recently taken over by another company and as such all of the network infastructure was migrated over to a new Active Directory domain. As ever the instructions provided the "IT team" fell miserably short of what was required for a smooth transition. Namely, they offered no way of migrating your user profile, so our old AD accounts were disabled and then we had to login to our new accounts and face a blank profile (i.e registry / software nightmares as well as any customisations you may of had / etc.) - so I went Googling' and came across this:

http://www.forensit.com/domain-migration.html - User Profile Wizard by ForensiT.

Worked flawlessly first time and it's free. Very nice product and highly reccomended.

Thursday, January 26, 2012

Debug release mode crash

Useful information on debugging problems in release builds when all your customer has is an instruction pointer address as the error.

Gives detailed information on best practices for generating the MAP file and then using this to determine which line number and source file is causing the exception / APPCRASH / issue.

http://www.codeproject.com/Articles/3472/Finding-crash-information-using-the-MAP-file

Wednesday, January 25, 2012

Bluetooth via the command line

Need to manipulate Windows bluetooth via the command line?

Bluetooth command line tools:
http://bluetoothinstaller.com/bluetooth-command-line-tools/

Friday, December 02, 2011

Using Image File Execution Options (IEFO) to invoke a debugger automatically

A handy tip for invoking a debugger (or anything else) in place of a normal executable at image load time is to set the following registry key:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\<image name>\Debugger = <other image name> (REG_SZ)

For example, to invoke WinDBG when the OS loads notepad.exe:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe\Debugger = "C:\Program Files\Debugging Tools for Windows (x86)\windbg.exe"

Very handy for short lived processes that are causing trouble, but you have no other means of attaching to them / etc!

Friday, September 23, 2011

Having trouble setting the height of a .NET Winform TextBox control?

Make sure multiline is set to true!

GetManifestResourceStream() returns NULL

If like me you're having problems accessing an embedded (assembly) resource using GetManifestResourceStream(), then read on:

GetManifestResourceStream was returning null even though I was pre-fixing the resource name with the namespace, i.e. NameSpace.Resource.ext.

The problem turned out to be the sub directory was missing from the name, in my example my resource lived under a folder (in the solution explorer) called Resources, so changing the qualified name to be:

NameSpace.Resources.Resource.ext fixed the issue.

Hope this helps someone else!

Wednesday, August 31, 2011

Useful information on using MFC in a C++/CLI project

http://msdn.microsoft.com/en-US/library/ms235211(v=VS.80).aspx
http://social.msdn.microsoft.com/Forums/en-AU/vcgeneral/thread/f46b5972-6cb5-4417-8cf9-33df56ccf2c4

Compiler errors in C++/CLI 2008 / VC9.0 toolset ('f:\dd\vctools\compiler\utc\src\p2\main.c line 182) when turning off pre-compiled headers?

Try this hotfix:

http://support.microsoft.com/kb/976656

Friday, August 26, 2011

List of MSXML versions

For reference, here you can find a list of all versions of MSXML:

http://support.microsoft.com/kb/269238

.. and just in case Microsoft decide to once again change their site structure without providing adequate redirects here is the content for prosperity:

MSXML versions that have been released by Microsoft

Version	File name	File version number
1.0	Msxml.dll	4.71.1712.5
1.0a	Msxml.dll	4.72.2106.4
1.0 SP1	Msxml.dll	4.72.3110.0
2.0	Msxml.dll	5.0.2014.0206
2.0a	Msxml.dll	5.0.2314.1000
2.0b	Msxml.dll	5.0.2614.3500
2.5 Beta 2	Msxml.dll	5.0.2919.38
2.5a	Msxml.dll	5.0.2919.6303
2.5	Msxml.dll	5.0.2920.0
2.5 SP1	Msxml.dll	8.0.5226
2.5 SP2	Msxml.dll, Msxmlr.dll (the last DLL is a resource-only DLL)	8.0.5718.1
2.5 SP3	Msxml.dll, Msxmlr.dll (the last DLL is a resource-only DLL)	8.00.6730.0
2.6 Web Release	Msxml2.dll	7.50.4920.0
2.6 SP1	Msxml2.dll, Msxml2a.dll, Msxml2r.dll (the last two DLLs are resource-only DLLs)	8.1.7502.0
2.6 SP2	Msxml2.dll, Msxml2a.dll, Msxml2r.dll (the last two DLLs are resource-only DLLs)	8.2.8307.0
3.0 Web Release	Msxml3.dll, Msxml3a.dll, Msxml3r.dll (the last two DLLs are resource-only DLLs)	7.50.5108.0
3.0	Msxml3.dll, Msxml3a.dll, Msxml3r.dll (the last two DLLs are resource-only DLLs)	8.0.7820.0
3.0 SP1	Msxml3.dll, Msxml3a.dll, Msxml3r.dll (the last two DLLs are resource-only DLLs)	8.10.8308.0
3.0 SP2	Msxml3.dll, Msxml3a.dll, Msxml3r.dll (the last two DLLs are resource-only DLLs)	8.20.8730.1
3.0 SP3	Msxml3.dll, Msxml3a.dll (8.20.8730.1), Msxml3r.dll (8.20.8730.1) (the last two DLLs are resource-only DLLs)	8.30.9926.0
3.0 SP4	Msxml3.dll, Msxml3a.dll (8.20.8730.1), Msxml3r.dll (8.20.8730.1) (the last two DLLs are resource-only DLLs)	8.40.9419.0
3.0 SP5	Msxml3.dll, Msxml3a.dll (8.20.8730.1), Msxml3r.dll (8.20.8730.1) (the last two DLLs are resource-only DLLs)	8.50.2162.0
3.0 SP7	Msxml3.dll, Msxml3a.dll, Msxml3r.dll (the last two DLLs are resource-only DLLs)	8.70.1104
3.0 Post SP7 MS06-061 (http://www.microsoft.com/technet/security/bulletin/ms06-061.mspx)	Msxml3.dll	8.70.1113.0
3.0 Post SP8	Msxml3.dll, Msxml3a.dll (8.20.8730.1), Msxml3r.dll (8.20.8730.1) (the last two DLLs are resource-only DLLs)	8.80.1185.0
3.0 SP9	Msxml3.dll, Msxml3a.dll (8.20.8730.1), Msxml3r.dll (8.20.8730.1) (the last two DLLs are resource-only DLLs)	8.90.1002.0
3.0 Post SP9 MS06-061 (http://www.microsoft.com/technet/security/bulletin/ms06-061.mspx)	Msxml3.dll	8.90.1101.0
3.0 SP10 (On Vista SP1)	Msxml3.dll, Msxml3r.dll (8.20.8730.1) (the last DLL is resource-only DLL)	8.100.1043.0
3.0 SP10 (On Vista SP2)	Msxml3.dll, Msxml3r.dll (8.20.8730.1) (the last DLL is resource-only DLL)	8.100.5000.0
3.0 Post SP10 KB973687 (http://support.microsoft.com/kb/973687)	Msxml3.dll	8.100.5002.0
3.0 SP11	Msxml3.dll, Msxml3r.dll (8.110.7600.16385) (the last DLL is resource-only DLL)	8.110.7600.16385
3.0 Post SP11 MS10-051 (http://www.microsoft.com/technet/security/Bulletin/MS10-051.mspx)	Msxml3.dll	8.110.7600.16605
4.0	Msxml4.dll, WinHTTP5.dll, Msxml4a.dll, Msxml4r.dll (the last two DLLs are resource-only DLLs)	4.0.9004.0
4.0 SP1	Msxml4.dll, WinHTTP5.dll, Msxml4a.dll, Msxml4r.dll (the last two DLLs are resource-only DLLs)	4.10.9404.0
4.0 SP2	Msxml4.dll, WinHTTP5.dll (5.0.2613.0), Msxml4a.dll (4.10.9404.0), Msxml4r.dll (4.10.9404.0) (the last two DLLs are resource-only DLLs)	4.20.9818.0
4.0 Post SP2 MS06-061 (http://www.microsoft.com/technet/security/bulletin/ms06-061.mspx)	Msxml4.dll, Msxml4r.dll (4.10.9404.0) (the last DLL is a resource-only DLL)	4.20.9839.0
4.0 Post SP2 MS06-071 (http://www.microsoft.com/technet/security/bulletin/ms06-071.mspx)	Msxml4.dll	4.20.9841.0
4.0 Post SP2 MS07-042 (http://www.microsoft.com/technet/security/bulletin/ms07-042.mspx)	Msxml4.dll	4.20.9848.0
4.0 SP3	Msxml4.dll, Msxml4r.dll (4.30.2100.0) (the last DLL is a resource-only DLL)	4.30.2100.0
5.0 (Office 2003)	Msxml5.dll, Msxml5r.dll (5.0.2916.0) (the last DLL is a resource-only DLL)	5.0.2916.0
5.0 (Office 2003 SP1)	Msxml5.dll, Msxml5r.dll (5.10.2925.0) (the last DLL is a resource-only DLL)	5.10.2925.0
5.0 (Office 2003 SP2)	Msxml5.dll, Msxml5r.dll ( 5.10.2927.0) (the last DLL is a resource-only DLL)	5.10.2927.0
5.0 (Office 2003 Post SP2) MS06-061 (http://www.microsoft.com/technet/security/bulletin/ms06-061.mspx)	Msxml5.dll	5.10.2930.0
5.0 (Office 2007)	Msxml5.dll	5.20.1076
5.0 (Office 2007 Post release) MS07-042 (http://www.microsoft.com/technet/security/bulletin/ms07-042.mspx)	Msxml5.dll	5.20.1081.0
6.0	Msxml6.dll, Msxml6r.dll (6.0.3883.0) (the last DLL is a resource-only DLL)	6.0.3883.0
6.0 Post release MS06-061 (http://www.microsoft.com/technet/security/bulletin/ms06-061.mspx)	Msxml6.dll, Msxml6r.dll (6.0.3883.0) (the last DLL is a resource-only DLL)	6.0.3888.0
6.0 Post release MS06-071 (http://www.microsoft.com/technet/security/bulletin/ms06-071.mspx)	Msxml6.dll	6.0.3890.0
6.0 SP1	Msxml6.dll	6.10.1129.0
6.0 Post SP1 MS07-042 (http://www.microsoft.com/technet/security/bulletin/ms07-042.mspx)	Msxml6.dll	6.10.1200.0
6.0 Post SP2 MS08-069 (http://www.microsoft.com/technet/security/bulletin/ms08-069.mspx)	Msxml6.dll	6.20.1099.0
6.0 Post SP2 KB973687 (http://support.microsoft.com/kb/973687)	Msxml6.dll	6.20.5002.0
6.0 SP3	Msxml6.dll, Msxml6r.dll (the last DLL is a resource-only DLL)	6.30.7600.16385

Notes

MSXML versions 1.x and 2.x are no longer supported by Microsoft.
MSXML 3.0 support follows the support policy of the OS into which it is built.
MSXML 4.0 is supported by Microsoft only if you are running Service Pack 3 (SP3).
MSXML 5.0 is supported by the Microsoft Office lifecycle policy.
MSXML 6.0 support follows the support policy of the OS into which it is built or onto which it is installed.
This table includes the version of MSXML that is included in security updates that were released since October 10, 2006. For more information about the security updates, click the following article numbers to view the articles in the Microsoft Knowledge Base:
924191
(http://support.microsoft.com/kb/924191/ )
MS06-061: Vulnerabilities in Microsoft XML Core Services could allow remote code execution
928088
(http://support.microsoft.com/kb/928088/ )
MS06-071: Vulnerability in Microsoft XML Core Services could allow remote code execution
936227
(http://support.microsoft.com/kb/936227/ )
MS07-042: Vulnerability in Microsoft XML Core Services could allow remote code execution
955218
(http://support.microsoft.com/kb/955218/ )
MS08-069: Vulnerabilities in Microsoft XML Core Services could allow remote code execution
2079403
(http://support.microsoft.com/kb/2079403/ )
MS10-051: Vulnerability in Microsoft XML Core Services Could allow remote code execution
32-bit versions of MSXML 3.0 (Wmsxml3.dll.) on 64-bit Windows Server 2003 uses the same MSXML and file version numbers that are listed in this table.
If MSXML 4.0 SP2 (out-of-support in April 2010) is installed on a computer that is running Windows NT, Windows 2000 SP2 and earlier, Windows 2003 RTM, or Windows XP RTM, the WinHTTP5.dll file is installed.

MSXML versions that are included with other Microsoft products and software updates

Replaced MSXML security update - MS02-008

Article number	Msxml version	File name	File version number
318202 (http://support.microsoft.com/kb/318202/EN-US/ )	2.6 SP2 (including hotfix)	Msxml2.dll, Msxml2a.dll, Msxml2r.dll (the last two DLLs are resource-only DLLs)	8.20.8307.0
317244 (http://support.microsoft.com/kb/317244/EN-US/ )	3.0 SP2 (including hotfix)	Msxml3.dll	8.20.9415.0
317244 (http://support.microsoft.com/kb/317244/EN-US/ )	4.0 (including hotfix)	Msxml4.dll, WinHTTP5.dll, Msxml4a.dll, Msxml4r.dll (the last two DLLs are resource-only DLLs)	4.00.9406.0

MSXML versions that are included with Microsoft Internet Explorer

Microsoft Internet Explorer version	Microsoft XML version (file version)
4.0	1.0 (4.71.1712.5)
4.0a	1.0a (4.72.2106.4)
4.01 (SP1)	2.0a (5.0.2014.0206)
5.0a	2.0a (5.0.2314.1000)
5.0b	2.0b (5.0.2614.3500)
5.01	2.5a (5.0.2919.6303)
5.01 SP1	2.5 SP1 (8.0.5226)
5.5	2.5 SP1 (8.0.5226)
5.5 SP2	2.5 Post-SP2 (8.00.6611.0)
6.0	3.0 SP2 (8.20.8730.1)
6.0 SP1	3.0 SP3 (8.30.9926.0)

MSXML versions that are included with MDAC

MDAC version (products)	Microsoft XML version (file version)
MDAC 2.5 (Windows 2000)	2.5 (5.0.2920.0)
MDAC 2.5 SP1 (Windows 2000 SP1)	2.5 SP1 (8.0.5226)
MDAC 2.5 SP2 (Windows 2000 SP2)	2.5 SP2 (8.0.5718.1)
MDAC 2.5 SP3 (Windows 2000 SP3)	2.5 SP3 (8.00.6730.0)
MDAC 2.6 (SQL Server 2000)	2.6 (8.0.6518.1)
MDAC 2.6 SP1 (SQL Server 2000 SP1)	2.6 SP1 (8.1.7502.0)
MDAC 2.6 SP2 (SQL Server 2000 SP2)	2.6 SP2 (8.2.8307.0)
MDAC 2.7 (Windows XP)	3.0 SP2 (8.20.8730.1)
MDAC 2.7 SP1 (Windows XP SP1)	3.0 SP3 (8.30.9926.0)
MDAC 2.8 (Windows Server 2003)	3.0 SP4 (8.40.9419.0)
MDAC 2.8 SP1	3.0 SP7 (8.70.1104.0)

MSXML versions that are included with other Microsoft products

This is not a complete list of MSXML versions that are included with other Microsoft products.

Other products	Microsoft XML version (file version)
Microsoft Security Essentials (Client)	4.0
BizTalk Server 2004	4.0
Microsoft Office Project Server 2003	4.0
Encarta 2003, 2004, 2005, and Student/Encarta 2006	4.0
Works Suite 2006, Works Suite 2005, Works Suite 2004, Works Suite 2003	4.0
Live Communications Server	4.0, 6.0
Application Compatibility Toolkit	4.0, 6.0
URGE	6.0
Microsoft .NET Framework 3.0	6.0
Microsoft Visual Studio 2005	6.0
Microsoft SQL Server 2005	6.0
Microsoft SQL Server 2008	6.0
Student/Encarta 2007	6.0
Zune software version 1.0	6.0
Windows SharePoint Services 3.0	6.0

Note If the MSXML parser is updated on a computer that has Microsoft Application Center 2000 installed, you must reinstall Application Center 2000. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

289510

(http://support.microsoft.com/kb/289510/ )

Application Center 2000 installs a special version of MSXML 3.0

Anyone looking for the Platform SDK Feb 2003 (aka the last Platform SDK to support Visual C++ 6)?

Here it is:

http://groups.google.com/group/microsoft.public.platformsdk.sdk_install/msg/087b0178f5d8159e?&hl=en

Thursday, August 25, 2011

More fun with C++/CLI DLLs and native MFC Extension DLLs!

I've recently been busy working on an ActiveX DLL written in C++/CLI, I chose C++/CLI (instead of C#) because we have "legacy" code in native C++ DLLs, some of which contain MFC exported classes (i.e. MFC Extension DLLs). As such there would have been a horrible amount of "glue" code to get this working in C# so the C++/CLI was the best choice for this sort of project.

Everything seems to have been going very well, until I noticed regasm hanging up on Vista during interop registration, the debug version of the DLL doesn't exhibit the problem, only the release version.

So I fired up WinDbg and setup the Windows symbol server path and launched regasm through the debugger. The stack trace revealed an invalid pointer usage inside AfxTermExtensionModule inside MFC42U. After some research I learned the way that MFC extension DLLs work internally (via CDynLinkLibrary), and some further digging produced this:-

http://support.microsoft.com/kb/154126

My next point of call was the DllMain entry points for the 4 MFC extension DLLs I'm making use of, which revealed that the CDynLinkLibrary instantiation was being carried out inside DllMain()'s DLL_PROCESS_ATTACH switch branch. Following the advice from the Microsoft link above, it becomes obvious that we do indeed need to create a separate exported function for the CDynLinkLibrary instantiation, this way we can call it and wire into the link library chain of our own DLL.

Severe headache, but at least I found a solution!

Finalize and Dispose in C#

I came across this succinct (well as succinct as you can be when discussing this topic) and comprehensive description of Dispose and Finalize semantics and thought I'd re-post it for reference (the author mentions specific C# functions, but the underlying mechanism applies to all .NET languages):-

(credits go to itowlson @ codeproject)

Finalize is the backstop method, called by the garbage collector when it reclaims an object. Dispose is the "deterministic cleanup" method, called by applications to release valuable native resources (window handles, database connections, etc.) when they are no longer needed, rather than leaving them held indefinitely until the GC gets round to the object.

As the user of an object, you always use Dispose. Finalize is for the GC. As the implementer of a class, if you hold managed resources that ought to be disposed, you implement Dispose. If you hold native resources, you implement both Dispose and Finalize, and both call a common method that releases the native resources. These idioms are typically combined through a private Dispose(bool disposing) method, which Dispose calls with true, and Finalize calls with false. This method always frees native resources, then checks the disposing parameter, and if it is true it disposes managed resources and calls GC.SuppressFinalize.

Wednesday, August 24, 2011

Visual Studio embedded manfiest targetting 9.0.21022.8 CRT even though 9.0.30729.1 (SP1) is in the redist

Something else noticed whilst developing my C++/CLI ActiveX control.

I'm using the CRT (v9.0) and for deployment I'm shipping the DLLs as private assemblies (i.e. 'xcopy deployment'). However what I noticed is that the embedded manifest inside my DLL was pointing toward version 9.0.210228.8 of the CRT, when the actual DLLs in the redist folder (which I'm using for packaging) are the SP1 version of the CRT (v9.0.30729.1).

Problem is, if the target installation PC doesn't already have the SP1 Visual C++ 2008 Redistributable SP1 package installed, then the runtime probe will fallback to my private assembly, however, because my private assembly version is 9.0.30729.1 but the manifest is asking for 9.0.210228.8 then we get a load failure.

Solution is to add a #define into the source (stdafx.h) to tell Visual Studio to generate a manifest which points to the current vclibs installed for the development environment (i.e. 9.0.30729.1):-

#define _BIND_TO_CURRENT_VCLIBS_VERSION 1

Tuesday, August 23, 2011

C++/CLI ActiveX automatic registration

I've been working on an ActiveX control written in C++/CLI recently and I thought I'd share the COM registration function ([ComRegisterFunction()] attribute) I've written (which is called by regasm.exe when you're registering the assembly for COM Interop).

It does the following:

Creates the necessary keys to register as an ActiveX control.
Adds in the codebase keys so there is no need to pass the /codebase flag to regasm.exe.

Please note, if you're not including oleidl.h as part of your include hierarchy (which I am as I'm using the Windows SDK), then you'll need to define the following constants:

// Ref: http://msdn.microsoft.com/en-us/library/ms678497.aspx

const int OLEMISC_RECOMPOSEONRESIZE = 1;
const int OLEMISC_CANTLINKINSIDE = 16;
const int OLEMISC_INSIDEOUT = 128;
const int OLEMISC_ACTIVATEWHENVISIBLE = 256;
const int OLEMISC_SETCLIENTSITEFIRST = 131072;

Credit is given to the authors of the CSActiveX control upon which this code was originally based.

//
// we need to add some extra registry keys on top of the regasm.exe generated
// keys. this is because we are an ActiveX control (not just a plain COM control)
// and as such require certain keys and values to be present.
//
// we also add in the codebase entries (instead of passing /codebase to regasm.exe)
// this is so we avoid the overkill warnings that regasm.exe generates.
//
// as such to register this .NET assembly as a fully fledged ActiveX control
// we need to run:
//
// regasm.exe /tlb
//
void FTEngine::Register( Type^ t )
{
  try
  {
    // Open the CLSID key of the control
    RegistryKey^ keyCLSID = Registry::ClassesRoot->OpenSubKey( "CLSID\\" +
                             t->GUID.ToString("B"), true );

    //////////////////////////////////////////////////////////////////////////
    // Set "InprocServer32" to register a 32-bit in-process server.
    // InprocServer32 =
    // Ref: http://msdn.microsoft.com/en-us/library/ms683844.aspx
    //
    RegistryKey^ subkey = keyCLSID->OpenSubKey( "InprocServer32", true );
    if( subkey )
    {
      // .NET runtime engine (mscoree.dll) for .NET assemblies
      subkey->SetValue( nullptr, Environment::SystemDirectory + "\\mscoree.dll" );

      // setup codebase flag as we're not in the GAC
      subkey->SetValue( "CodeBase", Assembly::GetExecutingAssembly()->CodeBase );

      // setup the version specific codebase flag
      Version^ ver = Assembly::GetExecutingAssembly()->GetName()->Version;
      subkey = subkey->OpenSubKey( String::Format( "{0}.{1}.{2}.{3}",
                                   ver->Major, ver->Minor, ver->Build,
                                   ver->Revision ), true );

      if( subkey )
      {
        // setup codebase flag as we're not in the GAC
        subkey->SetValue( "CodeBase",
                          Assembly::GetExecutingAssembly()->CodeBase );
      }
   }

   //////////////////////////////////////////////////////////////////////////
   // Create "Control" to identify it as an ActiveX Control.
   // Ref: http://msdn.microsoft.com/en-us/library/ms680056.aspx
   //
   keyCLSID->CreateSubKey( "Control" );

   //////////////////////////////////////////////////////////////////////////
   // Create "MiscStatus" to specify how to create/display an object.
   // MiscStatus =
   // Ref: http://msdn.microsoft.com/en-us/library/ms683733.aspx
   //
    subkey = keyCLSID->CreateSubKey( "MiscStatus" );

    int nMiscStatus = OLEMISC_RECOMPOSEONRESIZE +
                        OLEMISC_CANTLINKINSIDE + OLEMISC_INSIDEOUT +
                        OLEMISC_ACTIVATEWHENVISIBLE + OLEMISC_SETCLIENTSITEFIRST;

   subkey->SetValue( "", nMiscStatus.ToString(), RegistryValueKind::String );

   //////////////////////////////////////////////////////////////////////////
   // Create "ToolBoxBitmap32" to identify the module name and the resource
   // ID for a 16 x 16 bitmap as the toolbar button face.
   // ToolBoxBitmap32 = .,
   // Ref: http://msdn.microsoft.com/en-us/library/ms687316.aspx
   //
   subkey = keyCLSID->CreateSubKey( "ToolBoxBitmap32" );

   // If you want different icons for each control in the assembly you
   // can modify this section to specify a different icon each time.
   // Each specified icon must be embedded as a win32 resource in the
   // assembly; the default one is at the index 101, but you can use
   // additional ones.
   subkey->SetValue( "",
                     Assembly::GetExecutingAssembly()->Location + ", 101",
                     RegistryValueKind::String );

   //////////////////////////////////////////////////////////////////////////
   // Create "TypeLib" to specify the typelib GUID associated with the class.
   //
   subkey = keyCLSID->CreateSubKey("TypeLib");

   Guid libId = Marshal::GetTypeLibGuidForAssembly( t->Assembly );
   subkey->SetValue( "",
                     libId.ToString("B"),
                     RegistryValueKind::String );  


   //////////////////////////////////////////////////////////////////////////
   // Create "Version" to specify the version of the control.
   // Ref: http://msdn.microsoft.com/en-us/library/ms686568.aspx
   //
   subkey = keyCLSID->CreateSubKey( "Version" );

   int nMajor, nMinor;
   Marshal::GetTypeLibVersionForAssembly( t->Assembly, nMajor, nMinor);
   subkey->SetValue( "",
                     String::Format( "{0}.{1}", nMajor, nMinor ) );
   }
 catch( Exception^ ex )
 {
   LogException( ex );
   throw;
 }
}

Custom attributes are not consistent!

error LNK2022: metadata operation failed (80131195) : Custom attributes are not consistent: (0x0c00007d).

This is caused by specifying a managed attribute on the implementation of a C++/CLI function (i.e .cpp file), but not on the definition (i.e .h file).

Change your code so just the header file contains the attribute.

Javascript can't pass by reference into ActiveX/COM component

Useful information:
JScript / Javascript can't pass values by reference into a COM component, whereas VBScript can...

Thursday, July 28, 2011

Visual Studio (all editions including Express) on 64bit Windows may cause headaches when using "Register for COM Interop"

Just experienced a wasted 60 minutes trying to figure out why a C# project I've created (which is registered for COM Interop / CCW) couldn't be created from my test harness .vbs file (via CreateObject).

Background: I installed Visual C# 2010 Express on a 64bit Windows 7 platform, created my demo C# 'COM project', told VS to 'Register for COM Interop' - it all built without fail. However when I try and call into the COM object via a test.vbs file COM was complaining!

After some investigation (the /regfile switch on regasm.exe helped me out) I noticed that VS was using the 32bit version of regasm.exe which was subsequently writing the registry entries under the 'virtualised' WOW64 hierarchy (i.e. HKEY_LOCAL_MACHINE\Software\Wow6432Node) . Thus when I execute my .vbs (which internally uses the 64bit cscript.exe) then it couldn't find the COM object as registered (as it was only registered for the 32bit world).

Solution is to manually run the 64bit regasm.exe so the registry entries are put into the normal (and correct) place. Then your COM object will be visible to the 64bit world!

See here:-
http://support.microsoft.com/kb/956933