Poor RAID 0 Performance in PS CS2/CS3 BETA

(With my 8gb RAM, Photoshop is caching scratch into memory at this point anyway)

Not necessarily…

Photoshop creates the Scratch Disk the moment you open a file or create one, based on certain assumptions it makes about the file itself and your workflow, number of history states set in preferences, etc.

It’s not unheard of for the scratch disk to grow by 35 to 50 times the size of your largest file or more.

A 969MB file, like the one you cite, can create a scratch that is easily several times the amount of your 8GB of installed RAM.

Just a thought.

"A 969MB file, like the one you cite, can create a scratch that is easily several times the >amount of your 8GB of installed RAM." -Ramon

Yes, but Photoshop doesn’t create a scratch disk of that size if I open and then immediately ‘save as’ in the same folder, which is the basis for my timing on this. I wasn’t trying to measure scratch speed efficiency, or CPU tests on filters, just the speed at which the files would load and save based on the added efficiency of the 4-disk RAID 0.

In the case of the 969mb file, it consumed only 2.3 gb of scratch as reported by Photoshop’s info window; 2.71gb as reported by the OS finder.

I’ve provided minimally 3x the information bandwidth for PS to open and save files and it barely notices it’s even there. Is this a limitation of Photoshop, or is this situation true of all applications access to the hard drives vs the OS Finder?

It has been my experience that operations involving the hard drive, regardless of what scheme you can come up with, can not be dramatically accelerated.

but Photoshop doesn’t create a scratch disk of that size if I open and then immediately ‘save as’ in the same folder

It nevertheless has written to the scratch disk, regardless of your available RAM.

It has been my experience that operations involving the hard drive, regardless of what >scheme you can come up with, can not be dramatically accelerated.

Ok Ramon, that’s almost valuable information. I’m querying this forum to find out exactly why, in specifics, why minimally 3x faster hard drives produce no significant performance gain in a program that clearly makes tons of use of the hard drive. What exactly have you tried to do to speed up your hard drives and what exactly was the outcome? Have you set up 4-disk RAID 0’s and seen no performance increase like I have?

For literally a decade in Photoshop use, I’ve always seen or heard of the ideal of the premier Photoshop stations set up with 10k RPM Cheeta drives in RAID 0 setups, purportedly to increase speed, but based on my experience that kind of investment or investment in a Fiberchannel X-RAID is pointless because Photoshop isn’t or can’t make use of that extra bandwidth.

Why is this? How is this? Before posting this, if found only 3 incidences of ‘RAID’ on these forums, but when it did appear, people were discussing setting up monster Photoshop-handy computers. Well, I went and did it, and there was no success, so I’d like these words to act as warning for anyone considering a RAID for Photoshop use and a question for those who already have.

It nevertheless has written to the scratch disk, regardless of your available RAM.

Yes, I KNOW. 2.71gb of Scratch Disk used to cache the opening of the 969 mb file to be EXACT as I’ve already stated. I’m unclear what your point is on this topic? The scratch disk is on the RAID as well. By your insistence on the scratch disk tangent, shouldn’t that action have been sped up as well? I saw ZERO speed difference in moving my scratch disk from my local 250gb drive in it’s own 8gb partition, to having the scratch disk on the RAID.

Could any part of this issue be because I have a SOFTWARE based RAID handled by the OS instead of a HARDWARE based RAID completely handled externally? Again, everyone, including Apple, talking about making crazy fast boxes are talking about RAID 0 STRIPING of 3 internal drives of a new MacPro (4th drive, un-RAIDed for the OS) and using the OS to manage the RAID.

Does the OS somehow limit the speed of hard drive access to applications? As I stated in the beginning of this thread, Finder actions regarding this new drive are 3x-5x faster than on the non-RAIDed drives.

Anyone with knowledge in these matters who’s able to articulate a clear course of action based on the specifics of this situation, please speak up. This isn’t mystical or magical; there’s a cause & effect there that I’m trying desperately to understand. Allen Wicks, please feel free to chime in here. 😉

I’m sorry to sound insistent, but I hate getting my hopes up when I’m notified by email that there’s been a response to this topic and it’s a discussion of absolutely no practical use. I’ve just spent $1k on this endeavor and have a huge amount of client work upcoming that I was hoping/needing to use the potential of this new speed on. Another $1k will get me a 5-disk Hardware RAID, and if that’s the direction I need to go, I will, but it would seem currently that would also be money wasted.

You don’t really think for a moment that I read that long verbiage, do you? Imagine how many files could have been saved while you typed all that. 😀

Sorry, no offense, but I skipped it. Too hard on my old, tired eyes.

The point I had in mind is that once you achieved performance gains by using the fastest CPU, RAM, etc., The reduction in time writing and reading files rapidly approaches the point of diminishing returns.

Support Knowledgebase / Optimize performance in Photoshop (CS2 on Mac OS) <http://www.adobe.com/support/techdocs/332270.html>

You don’t really think for a moment that I read that long verbiage, do you? Imagine how many files could have been saved while you typed all that.

With 8 machines here, I was copying files while I typed, and yes, I’d think anyone over a working IQ of 80 would take the time to read the words the author of the thread posted before responding. What is your point here if you’re not participating in the actual discussion? Go have coffee with someone if you want banter… maybe yell at random people from a street corner, that might help.

That Support Knowledgebase article you just posted says"

Note that RAID 0 partitions provide the best performance as Photoshop scratch disks.

I’ve seen no performance difference in RAIDing my scratch disk, so what was I to glean from an article that I, of course, already searched out and took the time to read before ever posting to a forum when I’m involving other peoples valuable time and expertise?

Sorry, no offense, but I skipped it. Too hard on my old, tired eyes.

Sorry, offense taken. (Command +) to increase the type size on your screen. Also, there are plenty of websites with large pictures to occupy you if the idea of reading tires you.

I’m sorry if I write more than you want to read, but perhaps this thread would be best served by people who actually do take the time to read because banter like this is a waste of time. Perhaps if you had read the section related to RAID performance in the article you so flippantly reposted, you’d see it does nothing to bring light to this discussion.

If you have private discussion you’d like to have with me, feel free to email or telephone me, I’m listed, but please leave this forum for the transference of valid knowledge and ideas. Any response you post after this point that doesn’t state unique information on Software/Hardware RAIDs, mb/s rates, or Photoshop specific program code, is unnecessary and unwanted.

Stuff it, you prig!

Hasn’t it dawned on you that the minute you hit Save, the image file has to be processed, the composite has to be built, the whole thing compressed, etc.?

No, there’s absolutely no discussion, private or public, that I’d like to have with you. I have been posting to the topic for the benefit of anyone who is interested. That is the purpose of the forums, for everyone to benefit from ALL posts, not just the original poster.

I don’t know if threads where Chris Cox has commented on the issue have scrolled off the Earth by now, but a forum or Google search might turn them up.

Here’s one post by Chris Cox:

Chris Cox, "20 GB file" #8, 17 Aug 2006 1:05 pm </cgi-bin/webx?14/7>

Chris Cox – 1:05pm Aug 17, 06 PST (#8 of 11)

20 Gig doesn’t take that long.
250 Gig takes about 3 hours to open or save (with consumer disks and a current machine).

But Photoshop can handle 20 Gig (or 250Gig) without a problem – as long as you have a good amount of RAM and lots of scratch disk space available. What you’ll end up waiting on is your hard disks (reading and scratch).

It’s just NOT the same as copying a closed file in the Finder.

Stuff it, you prig!

he yells from his street corner..

lol.

Hasn’t it dawned on you that the minute you hit Save, the image file has to be processed, the composite has to be built, the whole thing compressed, etc.?

Yes, you dolt. From my THREAD STARTING POST:

I’m glad I didn’t spend the extra 300% on 10k RPM Raptor drives, because if this 4 disk RAID 0 that’s minimally 3x faster doesn’t speed up Photoshop large file management, clearly disk access doesn’t really play into this mix anymore.

I don’t expect a RAID to increase the speed of applying a filter. I don’t expect a RAID to increase the speed of generating a full resolution composite.

I do, however, expect a RAID to increase the load and save speeds of large Photoshop files.

If my expectations are wrong, I’m looking for ACTUAL INFORMATION of why I’m wrong. Of the ~2 minute save time, how much is spent with the actual process of writing information to the disk? Really, I want to know, that’s why I posted the question.

Ramon, you’ve replied 7 times to this thread, admittedly not even reading what I’ve written, and then posting either emotional, redundant or useless information. Get a life and quit hijacking my thread.

The smart thing you can do here is NOT respond, but your history pretty much assures that we’ll hear from you again, and your response will be scintillating… perhaps to a twelve year old.

BK,
You started the "emotional" trend.
Ramón was only trying to help
(even when he wasn’t reading your [long/interesting] posts)

You own nothing here but the need to be helped.

Good luck!

…isn’t mystical or magical; there’s a cause & effect there that I’m trying desperately to understand.

Me too. My intent has been to be (hopefully soon) building a MP setup very similar to what you describe. The only thing I can suggest is to try to configure a reasonably replicable test that is very close to an actual job and see what happens.

When I get a chance I may experiment with various configurations of my C2D Macbook Pro and its FW800 RAID 0 vs. a non-RAID FW800 drive. However the MBP is limited to 3 GB RAM so that variable I cannot approximate.

Allen – thank you for weighing in. I’ve started a dialog directly with Adobe and will post my findings here.

LTi – Agreed, I did become emotional first, but I hate seeing where many of these threads degenerate to. When Ramon responded to the thread that I started, stating that he hadn’t bothered to read my entire post, it clicked wrong in my caffeinated head and I had to respond. Sorry for being snippy.

I’ve been very specific in both my questions and the stated method of my testing the program and devices. When someone bulldozes in and responds just for the sake of responding without answering or at the very least broadening the scope of what the real question is in a forum that has so much potential for problem solving, it really irks me.

While I have made lasting relationships in the real-world via relationships started in online discussions, chats and even games, my motivations in posting to the Adobe site had nothing to do with social banter.

Let’s hope this thread now continues in a purely forward direction with everyone being adults and taking the time to fully read and comprehend the discussion and where it needs to go to be beneficial for all taking the time to read it.

Hi Bradley,

I hear your frustration. Often, Open and Save performance can be a significant bottleneck when working with larger files.

If I had your hardware setup, I would optimize performance as follows (I assume this a dual 2.5 G5, not a G5 Quad):

1. Assuming they are identical, either stripe the existing two internal drives or replace the two internal drives with two 150GB 10K Raptors (you can use them in your Mac Pro later on). Stripe those drives as a single RAID0 volume with Disk Utility (32k stripe size) and install the system on it. Use it for everything except Photoshop’s scratch disk and your image files.
You need the fastest possible system drive not only for better overall Photoshop performance but also because when you work really large files, it isn’t hard to start actively paging the OS swapfile- even with 8GB RAM installed. Be sure to maintain a lot of free space on the startup volume. The less data on there, the better- preferably less than 50% used at all times.

2. Use the external 4-drive SATA as follows (I assume they are identical 400GB drives? I also assume you’ve got the SATA card installed in the 133MHz PCI-X slot): With Disk Utility or SoftRAID, create two volumes, each one striping all four drives.

The first volume should be 100GB and should be used exclusively as Photoshop’s one and only scratch disk. The second massive volume would store all your image files.

For even faster Open & Save times, you can also use the "work disk/scratch disk method" with the designated scratch disk: copy the image file you want to work on to the scratch disk and open it and save it to the scratch disk. When you’re done working on it, either save it off to the original location on the second volume as a ‘version’ or as a replacement of original. then erase the scratch volume or trash the version on the scratch volume. This will give you the benefit of faster open & save times from the 4-drive stripe and particularly- from the fastest outer parts of the drives.

When you setup the striped volumes, you should use the 128k stripe block size for the scratch disk volume, and the 32k stripe block size for the image file volume.

3. Backup your startup disk and your image file volume every day to separate standard volumes on completely separate drives. While your mirror is a nice protection from mechanical failure, it doesn’t protect from file or directory corruption; any data that gets florfed is immediately florfed on the mirrored drive. If you do opt for a mirror, it’s safer if you hot-swap out one part of the mirrored set to an off-line status periodically, then swap it back in and use the auto-rebuilding feature to update it.

"Testing within Photoshop, however, has resulted in less than exciting results."

I know you are presently under the gun, but when you have some time- it is probably better to use a Photoshop-specific test- such as PshopTest2.1 <http://www.macgurus.com/forums/showthread.php?t=20218>. It will make it easier to fine tune your setup.

Some additional items of interest and to doublecheck:

A. I assume you have the Bigger Tiles plug-in installed and enabled. BT can make a significant performance improvement with large image files.

B. I assume you do not have the DisableVMBuffering plugin installed or enabled. It can help with the painting bug, but at the price of disabling scratch data buffering in the extra RAM installed beyond 4GB.

C. The CS3 beta does not yet have a Bigger Tiles plugin, nor is scratch data buffering in extra RAM installed beyond 4GB enabled.

D. I assume you are using Mac OSX 10.4.8 which has improved Photoshop performance over previous Tiger point updates.

E. I assume you have Processor Performance set to Highest in Energy Saver prefs.

F. Dragging the scratch disk volume into the Privacy pane of Spotlight prefs will also help.

G. With your 8GB of installed RAM, setting Photoshop’s memory pref to 100% will give you the fastest overall performance with big files, but Open and Save times will be somewhat impacted negatively because the OS is being robbed a bit. Finding a balance somewhere between 70% and 100% will improve your Open and Save times.

I hope this helps!

-George

George, "Use the external 4-drive SATA … create two volumes, each one striping all four drives"

If you do create two volumes, don’t you lose the benefit of the speed of a RAID, as the reading/writing of the scratch volume will be competing with the reading/writing of files?

I’d rather go with separated disks, especially in the case of batch operations: a source disk, a destination one, and maybe a raid0 for the Scratch

Bradley, I am also wondering, like you if is it the fact that the processor had to "compete" for the calculations of the saving of the file, and those for the raid 0 operations. But investing in a hardware RAID controller will be expensive, and I understand that you’d like to get performance returns.

Hi PE,

Yes, ideally his image file storage would be on a separate set of non-RAID drives and the scratch disk would be an external 4-drive 15k SCSI or Raptor stripe dedicated just for scratch.

But that would mean spending even more $.

As an economic compromise-

Since his external SATA box probably does 200+ MB p/s, contention between a relatively small first volume for scratch and the second larger volume for storage will be far outweighed in performance by the general speed of that stripe. And he could use the "work disk/scratch disk method" for no contention at all.

In any event, it should not be taking him nearly that long to open or save those files; something else is amiss.

Bradley, have you looked at using SoftRAID? I find it speeds up both RAID and non-RAID volumes.

I’ve used it for years on servers and Photoshop workstations without a single hiccup.

Maybe there is a disk driver problem at work here.

First off Bradley, it was YOUR tude that brought this all on…the fact is while you’ve thrown around a lot of pretty hardware specs, you haven’t even bothered to state some pretty basics on yourt set up. For example, are you running with or without Bigger Tiles plug-in? What is your Photoshop % set to? Where is your primary scratch disk? Are you specing only one scratch disk?

An internal raid 0 using your internal SATA ports ain’t gonna be all that faster since you need to add ports to keep up with the badwidth requirements that raid 0 demands. So, you can stripe the drives but you are port limited.

As far as the external, what’s your SATA card? You have 4 drives but how many ports? Again your speed will be port limited…the PCI bus has a limit on data transfer and you need a master port for each drive not shared SATA ports. So, unless you have a 5 or 8 port card, you’re in the same boat with the external as you are with the internal.

While adding raid 0 is useful it ain’t gonna give you radical improvements but incremental improvements based on the REAL speed improvements of the drives.

George, wow, thanks for the thorough and specific response. Let me add to the information here by sharing the responses I had directly from the Senior Lead Computer Scientist at Adobe–

From what it sounds like, you’ve got what I would consider a near-optimal system. I’d use the 4-disk RAID0 for scratch and the 3-disk RAID0 for current project access. When you get the MacPro system, I’d shoot for getting 16GB (or more) of RAM if you can afford it.

As far as what’s taking the time during loading and saving, Photoshop is spending a bunch of time during load allocating scratch, dividing the image up into scratch tiles, getting pyramid levels (cache levels) ready for display. If you get a strip-chart style resource monitor (my favorite is X Resource Monitor) you’ll be able to get a sense of if it’s the CPU crunching during load time that’s keeping things from going faster. And, once all the layers are loaded and divvyed up that way, the image has to be composited to the screen.

For saving, the composite image has to be computed for saving with maximum compatibility. There’s also the re-assembly of the layers of the image from the scratch tiles, compression of the data, and preparation of a bunch of other data. Again, a resource monitor will probably show some CPU usage which will represent that stuff.

Save/Load performance is something we know we need to look at. For CS3 much of that bubbled to down below other things on the performance priority list – launch time, healing brush performance among other things.

Here’s the part I found very enlightening and it’s sure to spur a torrent of conversation after I share it:

– Make sure that Photoshops’ memory usage percentage isn’t too high. With 8GB of RAM backing up reads/writes to/from scratch, less can be more in this situation – I’d shoot for Photoshop taking about 1.5GB of RAM. If set too high, things in Photoshops’ address space can get paged out to the system disk by the OS, in which case things will happen at the speed of what I’d guess is the slowest drive in the system.

So there you have it. Reduce memory allocation to Photoshop to 1.5GB… among other things. I’ve been full on multi-tasking today with portfolio showings, construction, retouching, motor-heading my HEMI and rubbing my wife’s very pregnant belly and I’m off to a Hockey game right now, so I’ll respond to some of your other questions and confirmations in the next round. Again, George, thank you so much for the input. I look forward to playing and implementing these ideas.

In other news:

First off Bradley, it was YOUR tude that brought this all on…the fact is while you’ve thrown around a lot of pretty hardware specs, you haven’t even bothered to state some pretty basics on yourt set up. For example, are you running with or without Bigger Tiles plug-in? What is your Photoshop % set to? Where is your primary scratch disk? Are you specing only one scratch disk?

First off, Jeff, I already apologized for being snippy and I really don’t think my response was all that harsh considering there was an individual weighing in that didn’t even take the time to thoroughly read what I had written… Like you, for instance.

For one, whether I have the Bigger Tiles plug-in or not, (I do) doesn’t matter if I’m timing the same situation with a relative respect to the only change being the RAID 0 in the mix.

If you had read my most, without having the beginnings of an emotional hissy on your own, you’d see that I already stated that my Primary Scratch Disk was set to the RAID 0 and from the first line you’d see that I described my 4-channel RAID, which if you deliberated on it a moment, you’d probably conclude that I’ve got 4 eSATA cables running from the external unit to the 4-ports on the PCI card.

To be perfectly explicit about my hardware:
-4, 320 7200RPM 16mb cache Barracuda drives
-1 SansDigital MS4T External RAID/JBOD Housing
-4 eSATA cables connecting to
-1 TeraCard TC-PCI-4S, 133MHz 64-bit PCI ATA Generation 2 Placed within a dual 2.5 G5 box running 10.4.8 with 8GB PC3200 DDR SDRAM, twin 7200GB 16MB Cache Barracuda drives, GeForce 6800 256mb Ultra AGP nVIDIA card connecting 30"&20" Apple displays with 7TB of external drive storage sitting on my desk, not including the new RAID.

To answer your other questions, I had been running 100% memory allocation to PS CS2, but based on Adobe engineering advice, I’m now down to 49%

Hey, thanks for the research Bradley, I’ve just backed PS CS2 off down to 50% memory use and it’s quicker here too. I’d always believed 70% was the optimum.

Optimum memory allocation has always varied for each individual, very much trial-and-error.

Optimum memory allocation has always varied for each individual, very much trial-and-error.

Precisely. If there were no such variations, you can bet the Photoshop engineers would have made it an automatic or automatically calculated setting long ago.

Check out MacGurus.com and go to there photoshop acceleration and also the RAID forum. There is alot of good knowledge there concerning raids and photoshop. Also barefeats.com has some test with various raids and photoshop.

Well Bradley, if you had started out indicating you were running at 100%, I suspect somebody else would have been able to point out that was suboptimal…you still haven’t stated whether you have a single scratch disk of a couple of partitions…it’s been my experience that multiple scratches is also suboptimal. And from the wording, I suspect the fellow ansering was Scott B? If so, I think you misread his suggestion…I think you can indeed run your ram allocation up…somewhere between 60-75% max should be fine. The ideal is to make sure you have at least a couple hundred MBs free at all times to avoid the OS paging in/out. You do this with Activity Monitor or X Resource Monitor…

And I see you still haven’t really lost the tude, have you? So, you want help or you want to act like a jerk? Up to you…and yes, your external array will be the fastest-the first partition on it anyway…