Feb 25, 2015

Linx 5.0.680.6896 released

Today marks the release of another Linx 5 Beta (5.0.680.6896).  If you have an existing Linx 5 installation, your designer will prompt you to upgrade to the new version.  If you don't have Linx 5 installed yet, you can download the latest version from the Linx website.

It has been a while since our previous public beta.  A lot of work has gone into making the back-end more robust and we hope that you will notice the difference.  We will continue to make improvements and add new features using the feedback that we receive.

On the component side, we have improved on existing functions and services, as well as create a lot of new items (e.g. Zip, ExecuteStoredProcedure, etc.).  Head on over to the Linx help site to read more about all of these new (and existing) items, check out our video tutorials, or install the necessary packages in Linx and try them out for yourself.

In upcoming releases, you will (among other things) start to see some of the styling work that we are busy with, so please be sure to update when there are new versions available.

Feb 24, 2015

Linx 4.0.867.4040 released

Changelog

  • Add ActiveDirectory component.
  • Add PDFExtractImages component. This component is not distributed with Linx.
  • Validator updated with performance improvements and stricter DateTime validation.
  • Linx Server logs shown in grid style.
  • Linx Server logs filtered by from and to dates.
  • Linx Server header colour and name can be changed to make server identification easier.


Server name and header colour

It is now possible to specify a header colour and server name per Linx server instance.  This can be useful in environments where multiple Linx servers are hosted.  Please note that this functionality is only supported on modern browsers.

Feb 9, 2015

Linx 4 memory usage

Every now and again we get questions like "How much RAM do I need for my Linx Server?" or related questions about how Linx uses RAM. This post tries to shed some light on these questions.

Infrastructure


Linx uses the automatic memory management supplied by the .Net framework. Memory is allocated to Linx as required and freed by .Net using a garbage collection algorithm. The garbage collector balances the memory requirements of the operating system with the cost of freeing and allocating memory. You can read more about the technical details here.

Linx Server


The Linx Server hosts each Linx Solution in its own Windows process. Each solution takes up approximately 34MB of RAM for its basic operations. RAM usage above that depends entirely on how the solution is implemented. One can implement the parsing of a big text file and only use 2MB of RAM or you can implement the same thing slightly differently and use GBs of RAM.

Components that may use a lot of RAM


FileRead

Reading files where OutputToTable = True or ReadType = Complete uses much more RAM than reading files line by line and not outputting to a table. The amount of RAM consumed by using ReadType = Complete or OutputToTable = True is directly proportional to the size of the file that is being read.

Database read components

All database components  where OutputToTable = True or GetAllDataToClientFirst = True use much more RAM than when both are False. Either one of these properties, when set to True, will read all the data from the database and store it in memory.

Components where OutputToTable = True

All components where this property is True will collect all the data in memory.

Assign

Assigning to a string variable where the operator = Add. This keeps on increasing the string size in memory.

NewRow

Adding a new row to a table variable increases the memory used by the table.

Image processing components

Image processing components may be RAM and CPU intensive depending on the type of function that is being performed. Test these in separate solutions to determine the effect your particular implementation has on RAM and CPU.

Multiple processes running at the same time


Linx can run multiple processes at the same time. None of the processes might use a lot of RAM on its own but together they can have a significant effect on RAM usage e.g. A directory watch service fires an event for a new file. Each file takes 5 seconds to process and uses 10MB of RAM. Imagine dumping 100 files in the directory in 4 seconds. That's 1GB of memory required for at least a second.

Testing some scenarios


I tested some of the scenarios mentioned above. The tests all use the same text file of 10MB with 10000 lines, each 1000 characters in length.

  • BuildFileInMemory builds a string using a string Variable and the Assign component and then writes it to a file.
  • BuildFileOnDisk builds the same file but writes it to disk line by line.
  • ReadFileComplete reads a file with ReadType = Complete.
  • ReadFileLineByLine reads the file line by line.
  • ReadFileLineByLineToTable reads the file line by line but OutputToTable = True.
  • StoreDataInTable reads the file line by line and then uses NewRow to add it to a table Variable.


The results



  1. Building a 10MB file in memory uses 99MB RAM vs 3MB for the direct to disk version.
  2. Reading a 10MB file completely uses 42MB of RAM vs 10MB if read line by line.
  3. Memory was not immediately freed up by the garbage collector on my system but simply re-used when I ran the same thing again (if my system required the RAM it would have been freed).
  4. The garbage collector frees up memory as the rest of my system requires it.

A common reason to use RAM rather than go directly to disk is speed. Here are the results of two of the tests from above writing to a SSD and a USB drive.


Using memory to build up a large string and then writing it to disk as one item is approximately 60% faster than going repeatedly to disk on a very slow USB drive. But it still only takes 1.5s to write a 10Mb file to a lowly USB drive. My requirements will have to be pretty extraordinary to justify building large strings in RAM for speed reasons alone.

Linx Designer


Do not test RAM usage by using the Linx Designer. The debugger stores a lot of additional data for debugging purposes that will give you false results. Test the RAM and CPU usage of a solution in a Linx Server.


Things to consider when writing Linx Processes



  • Number of files processed.
  • Size of the files.
  • Concurrency. Can processing happen simultaneously?
  • Number of database transactions we are reading.

If any of these are large numbers try to use RAM sparingly by taking into account the comments above. If you are unsure experiment by putting the processes you are concerned about in its own solution and test it on a Linx Server.


How much RAM do I need for my Linx Server?


Follow one of these strategies to determine an answer:

  1. Look at similar implementations and see what works for them.
  2. Write a solution to simulate the load and try it on a Linx Server.
  3. Take as much RAM as possible. RAM is cheap, your time is not.



Feb 6, 2015

Linx 4.0.858.3983 released

Changelog


  • ReadMailMapi now returns empty string instead of null for emails with no body
  • LinxServer Service icons changed
  • LinxDesigner scaling of fonts for different screen sizes improved
  • CallSoapWebService now supports a union structure for a message request
  • CallSoapWebService improved handling of dates and times
  • Validator add AllowExtraFields property
  • Validator add LogEveryLine property
  • DecodeFormzBarcode can now distinguish between Signature and other barcodes
  • Add Ctrl-A shortcut to Text Editor
  • LinxServer fallback to file upload dialog for IE11 when using NTLM authentication.
  • Fix bug: CallSoapWebService not showing list/array in correct format
  • Fix bug: CallSoapWebService incorrect or missing use of XmlElementAttributes
  • Fix bug: CallSoapWebService empty xml request string
  • Fix bug: Validation of disabled Service Events
  • Fix bug: Validator not correctly validating a DateTime
  • Fix bug: Validator using previous row values
  • Fix bug: DateTimeFormat using incorrect CultureInfo
  • Fix bug: Validator shows error when no delimiter specified for fixed length format
  • Fix bug: RabbitMQService properties could not be set to Constants
  • Fix bug: RabbitMQService crashes when the Rabbit Server is down


Validator


We've made some major changes to the Validator and there are still more on the way. Please let me know if you are using it. I would like to include your usage scenario in our tests so we don't inadvertently break something...