Dec 2, 2013

Linx 4: Writing large files

We recently had an issue reported that Linx 4 takes a long time to build strings. The problem showed up when trying to write a large file in Linx 4 took a lot longer than it used to in Linx 2. The process to prove it was


  1. Create a string variable.
  2. Read a database table with 1 million records.
  3. Loop through the records and add to the string variable i.e. for each record string = string + record data.
  4. Write the string to file.


The problem turned out to be the creation of a very large string by appending to the same string in a loop. We have improved the performance to Linx 2 levels by using a different string concatenation mechanism when we detect the process wants to append to a string. Even though this solves the speed problem it does not mean that building large strings in memory is a good idea.

To illustrate the problem I built a small application that writes 500 character long rows to build 3 different file sizes. One method uses the string concatenation method and the other writes directly to file. There was not much difference in speed but memory usage was drastically different:


So be careful when building very large strings in memory, especially if they can happen simultaneously. At least now you won't have to wait very long for the crash...

2 comments:

  1. Thanks for this update guys. We've recently discovered the same issue on of our solutions and are glad it has been addressed.

    ReplyDelete
  2. It is a great website.. The Design looks very good.. Keep working like that!. online

    ReplyDelete