Frustrating Decimal Issues

DG
Posted By
Dave_Gouveia
Oct 23, 2008
Views
2547
Replies
20
Status
Closed
Hi. I am using PS2 and have run into a problem that I’ve run into before, but now it’s not seeming to rectify itself. When I select my Marquee and choose ‘fixed size’, I enter the value – in this case i needed 1/8", so I entered .125 for width and .125 for height. I click in my live area, and I see the values round off to .1 x .1. So I try .25. It rounds to .2. Now, I’m no math genius, but I’m going to say that the majority of people using this program and the inches measurements likely do things in standard fractions – 1/8th, a quarter inch, etc. So why is this rounding down everytime I try to change it? It’s quite frustrating. I tried a restart – no go. In the past when this happened, I have retyped the number, hit enter, and after a few goes it seemed to stick, but today it just won’t change.

Any ideas on this issue?

D

How to Master Sharpening in Photoshop

Give your photos a professional finish with sharpening in Photoshop. Learn to enhance details, create contrast, and prepare your images for print, web, and social media.

B
Buko
Oct 23, 2008
do you mean Photoshop CS2?

what system and computer?
DG
Dave_Gouveia
Oct 23, 2008
oops.. haha.. yeah. In my frustration I meant PS CS2. I’m on a Mac G5 tower using 10.4.11
P
PeterK.
Oct 23, 2008
It’s probably because your resolution is not able to give you exactly .125 inches. Some measurements can’t be broken down as finely on, say, a 72 px/inch image as on a 300px/inch image. Photoshop can’t cut a pixel in half, so your crop will jump to the edge of the closest pixel. Imagine if you had a resolution of 5 pixels/inch. Now try to do a quarter inch crop. If you had 4 pixels/inch you could cut exactly one of those pixels out to get a measurement of .25 inches, but at a resolution of 5px/inch each pixel is taking up .2 inches, and you can’t get any closer than that.
PR
Paul_R
Oct 23, 2008
When entering your values did you enter ‘in’? IE: .125in
P
PeterK.
Oct 23, 2008
Even with 72 px/inch .125 inches is possible. Paul R is on to it in that if you enter ".125 in" it works, whereas just ".125" changes it to .1.
DG
Dave_Gouveia
Oct 23, 2008
Just tried by adding the ‘in’ at the end, and it works. However, what a pain in the ass. If i’m using inches as my main measurement, I shouldn’t have to type ‘in’. If I go into the fixed size area, and erase it and type .125 and hit enter, it adds the ‘in’ already – because my document is set to inches. But as soon as i click within my pasteboard, it defaults to .1 in. (this happens regardless of resolution setting).

If the entire point of having ‘fixed width’ is so you can enter a value, Photoshop should’nt be rounding up or down. Obviously in the case of something like pixels, i couldnt’ put 1.5 pixels – it automatically rounds it so it’s correct… but .125" IS a correct measurement – in fact, it’s even a standard – 1/8" – the usual bleed on most documents. If I’ve hit enter, and the value is there, it shouldn’t default as soon as i click in my live area. What’s the point of stating a number after I’ve hit enter that is going to change to something else when i click in my live area anyway?

Just seems lame to me. And honestly, sounds like a glitch.
JM
J_Maloney
Oct 23, 2008
Ruler to inches (control/right click the ruler) and then you can just type a number. How’s it supposed to know which you want? 🙂 And why can’t you type "1/8"?

But as soon as i click within my pasteboard, it defaults to .1 in. (this happens regardless of resolution setting).

Maybe try resetting your prefs, or the tool?
DG
Dave_Gouveia
Oct 23, 2008
Ummm… my ruler IS set to inches. And it’s not supposed to know the number i want, that’s why I AM typing it in. The problem is that the program is CHANGING it after I’ve typed it in. And this has nothing to do with a specific tool – its the marquee tools in general. The ‘fixed width’ preference is in the top menu. And any designer that works with inches frequently doesn’t use fractions – they use the decimals: .125, .25, .5, .75, .0625, and so on. Computers are mathematical, not fractional.
P
Phosphor
Oct 24, 2008
This brings up a question I wondered about a couple months ago. I mentioned here somewhere, but I forget in what thread, and that is:

We know—because Adobe Illustrator engineer Teri Pettit told us so and explained it—that Illustrator calculates internally using decimals out to 12 (maybe more) places, but it only displays 3 or 4, just for user and UI convenience.

I’ve never heard any discussion about the calculation accuracy vs. accuracy display in regards to Photoshop.

Has anyone? Got links to back it up?
AR
alan_ruta
Oct 24, 2008
I just did a test i which I rotated a 1 pixel line (1200ppi file) 1.234567 degrees and then 1.234565 degrees and they were identical.

I just wonder that even if photoshop calculate to a precision of 12 decimal places whether it can be seen at 1200ppi.

alan
P
Phosphor
Oct 24, 2008
"1.234567 degrees and then 1.234565 degrees and they were identical."

So is it a matter of perceptible difference, or is it a matter of truncated calculation?
B
Buko
Oct 24, 2008
The problem is that the program is CHANGING it after I’ve typed it in

You were given the explanation but you refuse to accept it. You can’t have a fraction of a pixel.
AR
alan_ruta
Oct 24, 2008
So is it a matter of perceptible difference, or is it a matter of truncated calculation?

I’d like to know the answer from an adobe programmer also, but in photoshop my guess is the answer is both. At 1200ppi .001 is approx. 1 pixel so rotating .000002 degrees more (please correct, er I know someone will, if I’m wrong) is 2 thousanths of a pixel (the millions place vs. the 1000s place).

12 decimals places, I think that is a trillionth.

alan
DG
Dave_Gouveia
Oct 24, 2008
We’re a bit off topic here.. I don’t need to go to a fourth/fifth/tenth decimal place. We’re talking about standard sizing here. Standard Imperial measurements.

So Buko, you’re not correct in saying i’m ‘refusing’ to accept the explanation. I KNOW you can’t have half a pixel. But can you have 1/8"? I would say yes… but the program is saying no UNLESS i type in the word ‘in’, which, to me, is redundant. If I’m working in pixels, i don’t have to type ‘px’. If I’m in Millimeters, I don’t have to type ‘mm’. So why in inches?

And these standard measurements are used by designers worldwide: 1/8ths of an inche, 1/32nd of an inch… these aren’t magical numbers no one has heard of. They are STANDARDS.

I appreciate the responses from people, but please know that I’ve been in the industry using photoshop since version 3… so I’m not a idiot. Saying I’m ‘refusing’ to accept what clearly does not address WHY this issue exists is downright ridiculous. And NOWHERE did I state you can have half a pixel.. i said you CAN’T have half a pixel.
P
Phosphor
Oct 24, 2008
Here’s what I know:

Using HotDoor’s CADtool plugin with Illustrator 8, I’m afforded the ability to work with accuracy down to 1/10,000th of a point.

A point is 1/72 of 1 inch, or 0.0138888". In order to calculate 1/10,000th of that value we add 4 zeros just to the right of the decimal point, and we get an accuracy of 0.00000139". I kinda like knowing where I stand where accuracy’s concerned.

Do I use that level of accuracy all the time? Nope. But I like knowing that I can.
PR
Paul_R
Oct 24, 2008
I have just tested Photoshop CS2/3 and 4 all when rulers are set to inches accept .125 without the ‘in’
Maybe the tool or the preferences needs to be reset?
JM
J_Maloney
Oct 24, 2008
No, what I’m saying is, when I type in .125, it resets to .1. Stop trying to change the subject.
P
Phosphor
Oct 24, 2008
For what it’s worth, I’m seeing the same problem in PS CS2.

• Check ruler units—they’re set to use inches.
• Reset Marquee Tool, just for "good measure." 😉 • Set Option to "Fixed Size."
• Type "0.125" in Width field. Tab to the Height field and see that "in" is added to the value I typed in the Width field.
• Type "0.125" in Height field. Tab to Feather field and see that "in" is added to the value I typed in Height field.
• A value of "0" is already in the Feather field, so I tap my Enter key to deselect/dehighlight it.
• Click in document to create a 0.125" × 0.125" marquee. • Info Palette shows me that the actual size is 0.093" × 0.093". • I measure it with the Measure Tool. Same 0.093" value appears.

Next, I perform the same operations, except that I manually type in the "in" designation after "0.125."

Readings are accurate.
JM
J_Maloney
Oct 24, 2008
I can confirm it too, resetting prefs and tools doesn’t help. Hitting the switch w/h button does though.
P
Phosphor
Oct 24, 2008
I’d call it a bug, I think.

So it goes. I’ll live with it, and not complain, now that I know about it.

How to Master Sharpening in Photoshop

Give your photos a professional finish with sharpening in Photoshop. Learn to enhance details, create contrast, and prepare your images for print, web, and social media.

Related Discussion Topics

Nice and short text about related topics in discussion sections