6

I have four variables, each saved in 365 mat-files (size: 8 x 92 x 240). I try to load these into my function in a for-loop day=1:365, one variable file per day. However, the two first variables always take abnormally long time to load. My code for loading looks like this: 

load([eraFolder sprintf('Y%dD%d-tempSD.mat',year,day)], 'tempSD'); % took 5420 s to load

load([eraFolder sprintf('Y%dD%d-tempDewSD.mat',year,day)], 'tempDewSD') 

load([eraFolder sprintf('Y%dD%d-eEraSD.mat',year,day)], 'eEraSD'); % took 6 seconds to load

load([eraFolder sprintf('Y%dD%d-pEraSD.mat',year,day)], 'pEraSD');

Using Profiler, I could see that the first two variables took 5420 seconds to load in 365 calls, whereas the the last two variables took 6 and 4 seconds to load respectively over 365 calls. When I swap the order in which variables are loaded, e.g. eEraSD before tempSD, it is still the first two loads that take more time.

When using tic toc to track the time spent on loading, it appears that the time to load a the first or second variable exponentially increases with the number of calls (with the last calls taking 50 seconds to run) . For the third and fourth variable, the loading time stays around 0.02-0.04 seconds per file, more or less independent on how far in the for loop I have gone. See figures below.

enter image description here enter image description here

When using importdata instead of 'load', the first line takes about 8000 seconds to load 365 times (with the loading exponentially increasing as shown for T in the second figure). The other lines then take about 10 seconds to load 365 times.

I can't understand why it looks this way and what I can do to decrease the loading time. Would greatly appreciate any idea of a possible solution for this.

Shai
  • 111,146
  • 38
  • 238
  • 371
LaWa
  • 183
  • 7
  • Are these files stored on a network drive by any chance? – George Skoptsov Dec 14 '12 at 11:14
  • 2
    Does it repeat if you run the load-loop twice? (without any operations in between) – Xyand Dec 14 '12 at 11:14
  • The files are stored locally. – LaWa Dec 14 '12 at 12:13
  • Thanks Albert for pointing this out. If I run the the for-loop only, without any other code, I don't experience this problem at all... I tested in a separate function file (with the for-loop inside the function file once, and outside once), and the 365 days were loaded in no time. So it seems connected to something I do in the rest of the script, even though the tic toc and profile only points to the loading operations. I guess I should just try to expand the code stepwise from there, and see when the problem resurface. Or do you possibly have any idea of possible connections? – LaWa Dec 14 '12 at 12:36
  • 1
    Can you please also post your loading function? What size are your files? What is their format? – Barney Szabolcs Dec 14 '12 at 15:22
  • 1
    Is it possible you dynamically allocate memory for the loaded variables? i.e., you change the size of the variables inside your load loop? If so, try to pre-allocate all memory needed for all the variables in advance. – Shai Dec 16 '12 at 08:49
  • Another question: does your `mat` files has any additional variables you are not loading? – Shai Dec 16 '12 at 08:50
  • 1
    @Shai, in my function, I only load the variables and send them out of the function to the main script. I'm trying now to use `importdata` instead and see if it changes anything. I don't have any additional variables in the file that I don't load. – LaWa Dec 20 '12 at 14:56
  • @Faisal-Shahzad , I am not entirely sure what you mean. But the files are all 8X92X240 in size. – LaWa Dec 20 '12 at 14:58
  • Can you tell which save format you use? I remember the 7.3 format (hdf5 based) having serious performance issues. – bdecaf Jan 09 '13 at 13:01

2 Answers2

2

I suppose your data sets are in the same directory(over network or local) and with same attributes e.g. access properties and so on.

Then the only option left is with the charateristics of the vairbales stored in those matfiles. Can you check how much those variables appear in size e.g. by loading a sample one. This will narrow down to solve your problem.

Hope that help.

FS

1

Thanks for your help. I finally found out what caused the problem. In a 'for' loop later in the script, I saved other data to a folder I called temp. After renaming that folder to something else (e.g. temporary), the data loading problem disappeared.

(Doesn't matter so much now that the practical problem is solved, but I can't really say I understand why there was this peculiar relationship between the later 'save' call and this 'importdata' or 'load' call.)

Please see new question about the temp folder

Community
  • 1
  • 1
LaWa
  • 183
  • 7
  • very bizarre behavior. I saw your other question as well. Can you post a *very short* code snippet that reproduces this behavior (load and save to `temp`)? – Shai Jan 15 '13 at 10:20