Linx 4 memory usage

Every now and again we get questions like "How much RAM do I need for my Linx Server?" or related questions about how Linx uses RAM. This post tries to shed some light on these questions.

Infrastructure

Linx uses the automatic memory management supplied by the .Net framework. Memory is allocated to Linx as required and freed by .Net using a garbage collection algorithm. The garbage collector balances the memory requirements of the operating system with the cost of freeing and allocating memory. You can read more about the technical details here.

Linx Server

The Linx Server hosts each Linx Solution in its own Windows process. Each solution takes up approximately 34MB of RAM for its basic operations. RAM usage above that depends entirely on how the solution is implemented. One can implement the parsing of a big text file and only use 2MB of RAM or you can implement the same thing slightly differently and use GBs of RAM.

Components that may use a lot of RAM

FileRead

Reading files where OutputToTable = True or ReadType = Complete uses much more RAM than reading files line by line and not outputting to a table. The amount of RAM consumed by using ReadType = Complete or OutputToTable = True is directly proportional to the size of the file that is being read.

Database read components

All database components  where OutputToTable = True or GetAllDataToClientFirst = True use much more RAM than when both are False. Either one of these properties, when set to True, will read all the data from the database and store it in memory.

Components where OutputToTable = True

All components where this property is True will collect all the data in memory.

Assign

Assigning to a string variable where the operator = Add. This keeps on increasing the string size in memory.

NewRow

Adding a new row to a table variable increases the memory used by the table.

Image processing components

Image processing components may be RAM and CPU intensive depending on the type of function that is being performed. Test these in separate solutions to determine the effect your particular implementation has on RAM and CPU.

Multiple processes running at the same time

Linx can run multiple processes at the same time. None of the processes might use a lot of RAM on its own but together they can have a significant effect on RAM usage e.g. A directory watch service fires an event for a new file. Each file takes 5 seconds to process and uses 10MB of RAM. Imagine dumping 100 files in the directory in 4 seconds. That's 1GB of memory required for at least a second.

Testing some scenarios

I tested some of the scenarios mentioned above. The tests all use the same text file of 10MB with 10000 lines, each 1000 characters in length.

• BuildFileInMemory builds a string using a string Variable and the Assign component and then writes it to a file.
• BuildFileOnDisk builds the same file but writes it to disk line by line.
• ReadFileComplete reads a file with ReadType = Complete.
• ReadFileLineByLine reads the file line by line.
• ReadFileLineByLineToTable reads the file line by line but OutputToTable = True.
• StoreDataInTable reads the file line by line and then uses NewRow to add it to a table Variable.

The results

1. Building a 10MB file in memory uses 99MB RAM vs 3MB for the direct to disk version.
2. Reading a 10MB file completely uses 42MB of RAM vs 10MB if read line by line.
3. Memory was not immediately freed up by the garbage collector on my system but simply re-used when I ran the same thing again (if my system required the RAM it would have been freed).
4. The garbage collector frees up memory as the rest of my system requires it.

A common reason to use RAM rather than go directly to disk is speed. Here are the results of two of the tests from above writing to a SSD and a USB drive.

Using memory to build up a large string and then writing it to disk as one item is approximately 60% faster than going repeatedly to disk on a very slow USB drive. But it still only takes 1.5s to write a 10Mb file to a lowly USB drive. My requirements will have to be pretty extraordinary to justify building large strings in RAM for speed reasons alone.

Linx Designer

Do not test RAM usage by using the Linx Designer. The debugger stores a lot of additional data for debugging purposes that will give you false results. Test the RAM and CPU usage of a solution in a Linx Server.

Things to consider when writing Linx Processes

• Number of files processed.
• Size of the files.
• Concurrency. Can processing happen simultaneously?
• Number of database transactions we are reading.

If any of these are large numbers try to use RAM sparingly by taking into account the comments above. If you are unsure experiment by putting the processes you are concerned about in its own solution and test it on a Linx Server.

How much RAM do I need for my Linx Server?

Follow one of these strategies to determine an answer:

1. Look at similar implementations and see what works for them.
2. Write a solution to simulate the load and try it on a Linx Server.
3. Take as much RAM as possible. RAM is cheap, your time is not.