sas >> Slow Data Operations, is SAS using all Memory?

by frotty22 » Wed, 30 Aug 2006 22:26:36 GMT

Hi,

I have Windows XP SP2 PC equipped with 2.5 GB RAM.
But when I look at the Task Manager Performance window while SAS is
working on data operations, it shows that there is still half of the
physical memory available but the page file size increases and the
harddrive LED on the computer shows the drive working all the time.
After finishing the operation, the CPU time is always only a fraction
of the overall time.

Is it possible that SAS is not using all of the physical RAM? Do I have
to set some options for that?

Thanks for any hints!



sas >> Slow Data Operations, is SAS using all Memory?

by Reeza » Wed, 30 Aug 2006 23:16:05 GMT


Check if your machine has hyperthreading.

It's supposed to prevent a process from using all the available
resources, and allow you to multitask better, but a drawback is that it
slows down larger processes.

Slight warning though, I've seen some interesting reactions when it's
been taken off. One computer wouldn't even boot up afterwards, and
others are faster.

Good Luck,
Reeza





sas >> Slow Data Operations, is SAS using all Memory?

by davidlcassell » Thu, 31 Aug 2006 13:14:16 GMT




[1] Your OS will *never* let SAS have all the physical RAM.
Even Windows is too smart for that. You have to have lots of
RAM available for all the other tasks your OS has to do, and you
need to have RAM available for other processes.

[2] Your RAM is not the problem. You say:


This is almost always an indication that your disk I/O is the
real bottleneck. Your RAM sits around, and your CPU chip sits
idle a lot of the time, while huge amounts of disk I/O get
done. You do not say what you are doing, but you may not
be using the features of SAS as efficiently as you could.

[3] There are things you can do to tune your system. But it
sounds so far like you need to tune your code a lot first, so
I would not even bother going in that direction. Focus on
what you are doing that makes your disk drive work so darn hard
that your CPU ends up sitting around taking a nap.

HTH,
David
_________________________________________________________________
Call friends with PC-to-PC calling -- FREE
http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline


Slow Data Operations, is SAS using all Memory?

by frotty22 » Thu, 31 Aug 2006 21:59:27 GMT




Thanks for you answers!

I still think it is a bit strange that from 2.5 GB there is half of it
sitting idle. SAS itself uses only a bit more than 100 MB. If the
system would require 1.25 GB for other stuff, why did my computer work
before the upgrade when it only had .5GB ?

I am aware that joins with databases of 1.5 or more GB are likely to
need some virtual memory. But the swapfile is too small during
processing. It seems like SAS is directly working on the harddrive
without trying to use the memory.

Thanks again,
Bob



Slow Data Operations, is SAS using all Memory?

by ChrisW75 » Fri, 01 Sep 2006 10:22:12 GMT

i Bob,
Check out the System options, you may have a constraint put onto the
memory allocation. The specific optiosn you want to check out are
MEMSIZE, SORTSIZE MAXMEM etc. (this is just off the top of my head, so
not all fo these may be correct, but I'm pretty certain on MEMSIZE).
Check out those options on the SAS Doco site and then tweak your .CFG
file to adjust them. I actually made some changes to my sortsize
option and it sped up sorts in quite a big way, as it reduced the
amount of disk access required.
To check your current options use:-
proc options;
run;
And scroll through your log.
ChrisW75

XXXX@XXXXX.COM wrote:



Slow Data Operations, is SAS using all Memory?

by davidlcassell » Fri, 01 Sep 2006 13:35:31 GMT

XXXX@XXXXX.COM wrote back:

I think that the problem is I/O rather than memory usage.

Since I don't know what code you are using to do this large
join, I'll have to speculate that you are (either implicitly or
explicitly) doing a sort-sort-merge, and probably doing it in
PROC SQL. So you are suffering from massive sort-o-mania.
This is often best addressed by re-design of the process.
It may be that you would do better designing this as a hash,
or perhaps avoiding the current process flow completely.
But I can't tell that from way over here.

If you really believe that you need to allocate more RAM,
then look into the REALMEMSIZE system option. Be careful
with it. Don't allocate more RAM than will actually be available
to SAS at invocation, or you'll end up causing performance
problems with thrashing of your virtual memory.

Once you have that, you can muck around with the
SORTSIZE system option. Perhaps a simple -SORTSIZE MAX
given your *careful* choice of REALMEMSIZE .

You can mess with MEMSIZE too, but optimal choices
for that vary according to your OS, so be careful. How
you assess a good choice for MEMSIZE is dependent on
your available system monitoring tools.

There are other things you can do to mess up your SAS
system (:-) but I always recommend coding over system
options, unless the system options have been messed up
already and need to be fixed.

HTH,
David
_________________________________________________________________
All-in-one security and maintenance for your PC. Get a free 90-day trial!
http://www.windowsonecare.com/trial.aspx?sc_cid=msn_hotmail


Slow Data Operations, is SAS using all Memory?

by ben.powell » Fri, 01 Sep 2006 16:59:18 GMT

If my surname was Dorfman or Venezia I could easily tell you the sort of
Hash index you could create that would instantly devour all that memory and
reassure you SAS uses it when it needs to. But its not and I can't. It is
possible much RAM isn't required given the procedures you're currently
using. I/O can be such a bottleneck that only truly ram resident operations
will make use of large amounts of ram(!). Bear in mind there has been some
research on the list in the past that suggests options to load datasets into
memory using sasload may not be that effective. The same goes for using ram
disks.

HTH.


Similar Threads

1. We have an AIX lpar and sas isn't using the cpu/memory/etc

Is it a particular query or application that is running slow? More details
might help to isolate where the potential bottleneck is.

Rgds

2. We have an AIX lpar and sas isn't using the cpu/memory/etc capacity available

3. Arithmetic operations using DFSORT

4. Using macro variables in arthimetic operations

5. Time is AM, instead of PM, when using ODS RTF

To comp.soft-sys.sas --

I'm using ODS RTF to write SAS/GRAPH results to an RTF file.  The date-
time stamp that gets written in the header of the RTF file is:

05:33  Wednesday, April 09, 2008

The only problem is, that the time is really 5:33 PM -- so it should
say either "5:33 PM" or (preferably) "17:33".  Does anyone know how to
fix this?  Thanks much!

       --Tom

6. Watch out for SAS keeping old data in memory while running

7. Fwd: Watch out for SAS keeping old data in memory while

Peter ,

I fear you have missd the lesson here, you should have looked at your log.
Rule of thumb, never ever ever ever ever ....... to infinity..... ever....
let a program go out with errors.  If there is an error always and I mean
always fix it.



Toby Dunn





From: Peter Constantinidis < XXXX@XXXXX.COM >
Reply-To: Peter Constantinidis < XXXX@XXXXX.COM >
To:  XXXX@XXXXX.COM 
Subject: Fwd: Watch out for SAS keeping old data in memory while running
Date: Tue, 6 Jun 2006 22:34:24 -0400

Sigh. I made a test program so I could demonstrate to interested
parties but alas, the problem in this case really was a combination of
my fault, and misleading behavior (my opinion) on the part of SAS.

Here is what's happened, my comments in brackets:

(beginning, 3 sets are made, and run, with no errors. now for output
portion, a set has been deleted on purpose.)

32   data report;
33   set perm.set1 perm.set2 perm.set3;
ERROR: File PERM.SET2.DATA does not exist.

(so far so good, it recognizes that set1 and set3 exist, but set2 has
been removed.)

34   ods html body='c:\test.htm' ;
NOTE: Writing HTML Body file: c:\test.htm
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.REPORT may be incomplete.  When this step
was stopped there were 0 observations and 2 variables.
WARNING: Data set WORK.REPORT was not replaced because this step was
stopped.

(here it is saying that work.report was not replaced with updated data
because a missing set in a set statement is enough to stop the step,
it does not skip over like I expected).

35   proc print data=report noobs label split='/';
36   var name bananas;
37   run;
NOTE: There were 9 observations read from the data set WORK.REPORT.
NOTE: PROCEDURE PRINT used (Total process time):
      real time           0.06 seconds
      cpu time            0.01 seconds
38   ods html close;

(misleading newbie trap! even though the previous step was STOPPED, it
still went ahead and printed what it had in memory. sas is doing what
it was told to do, and work.report was never replaced but it was never
deleted either.)

And that is what happened and it is my fault. The moral here for
students, is that you have to write a new data statement every time if
you're going to have a different # of files, otherwise it'll use the
old data it still has in memory. :(

Well, I'll certainly never make that mistake again.

8. Fwd: Watch out for SAS keeping old data in memory while running