7 Days of Urchin 7: Day 3 – Performance

In the last installment, I went over my experience in upgrading from Urchin 6 to Urchin 7. Some of you may wonder what are the benefits of moving to Urchin 7. Well, one of the big benefits is speeeed. Speed of processing that is.

The move from 32-bit to 64-bit

Earlier versions of Urchin were limited to 32-bit operating systems. That means that Urchin could only use a maximum of 2 GBs of memory. With small log files, this wasn’t a major problem because you would never use up all your memory, and processing profiles would be relatively speedy. The problem came with really large log files or if you increased the data row limit. If you had either of these two situations, Urchin would run out of RAM and would begin swapping data in and out of RAM. This would slow down the processing process dramatically.

With the latest Urchin, if you have a 64-bit processor and a 64-bit operating system, you can utilize a maximum of 16 GBs of memory. The result? Much speedier processing of large logs. I wish I had the resources to get a new Xeon 4-core, 32 GB RAM Urchin mega-machine to test Urchin 7 on, but unfortunately, there was no budget for this, and plus, I didn’t have any log files large enough to tax the system. However, from the performance benchmarks we saw from Google during the beta period, the increase in performance was dramatic when increasing RAM limits from 2 to 4 GBs.

Welcome the multi-core processors

With the advent of multi-core processors, multi-tasking has truly been realized. And to take advantage of this, Urchin now has a setting that will allow you to enable concurrent profile processing. So instead of taking 33 hours to process 1000 profiles at 2 minutes/profile, you can theoretically cut that down to half the time. Of course, to take advantage of all those cores, you will need to ensure your other hardware is up to the task. Some things to take into account: I/O speed and RAM. It is recommended to have profiles that will be processed at the same time reside on different physical disks. Doing so will optimize the processing speed. Also, each profile will require it’s own block of dedicated RAM. So if you are planning on running concurrent profiles, make sure you install enough RAM in your box to handle:

• the OS
• the database and
• the number of concurrent profiles you will be processing

My recommendations are:

• 2 GBs for OS
• 1 GB for database
• 2 – 4 GBs of RAM per concurrent process (depending on the size of the log files)

Stay tuned for the next installment in the series: 7 Days of Urchin 7: Day 4 – Advanced Segments

If you would like to read more about Urchin 7, please visit http://www.google.com/urchin.