2024-02-14 ChromSword Delphi And Lazarus Compillation Instructions

Summary

ChromSword 5.X versions for Developer and ReportViewer Integration tasks use Delphi 2005/2009 Pascal built code that is using COM, Windows DLL libraries. This particular type of DLLs allow to Pascal call C# code from other end and C# also is able to call Pascal code from this end. It allows code execution in both directions. In future, we would need to change this to work with Pascal Lazarus, which is the modern version that supports x64. The old Delphi Pascal only supports x86. In theory, the calling from the Pascal side is not that necessary. We could just ask Pascal to give us necessary method parameters without itself calling C# code.

ChromSword Offline 5 uses the TCP socket JSON service that is implemented as Lazarus Pascal. It is compatible with old Pascal code, but it adds a modern layer so that it would be easier to communicate without any kind of C# COM libraries.

Prequisites

1. Install Borland Delphi 2009 (do not install any .net additions it offers, even if installation fails it usually works with Pascal projects afterwards)

2. To install and build Chromsword.Logic which is Delphi 2009 project, install http://share.yellowrobot.xyz/1631869217-chromsword/delphi_2009.zip

3. Also install Lazarus FreePascal x64 latest version https://www.lazarus-ide.org/index.php?page=downloads (for Offline Service)

ChromSword.Logic.dll (ChromSword 5.x Delphi Pascal)

1. Open in ChromSword5 repo ChromSword.Logic following project group in Delphi. Make sure you use exactly this file not other file as the base project workspace.

2. After opening you will see multiple Homsford projects. You need only ChromSword.logic project. Other projects are legacy projects that were developed to support Hasp encryption key which is not used anymore. We don't need them anymore.

   

3. You can run or build them by clicking the green arrow on top bar or on the project left click and then compile and then build. It will produce then the DLL file.

   

4. Caution! For debugging purposes from C# side, the breakpoints will not work. Instead, you need to use show message boxes. In the code, you will see that there are such debugging codes.

5. Most important part for exposing functions and changing functions to C# side is using UX-based very buggy interface in Delphi called View > Type Library, where you can define com.dll type descriptions and these will be exposed to the Pascal and C# code. It is buggy because when you press save, often it shows error that it cannot save even though it does save these settings. So be very careful. Do not change them very fast. Change them step by step and make sure that they are reflected in Pascal and TLB files.

⚠️ At this moment you don't need to change existing COM functions, but please make another test function for yourself so you can make sure that you can send messages send data to the Pascal and receive it back to the C# just for your own testing purposes. Also please study a little bit Windows COM DLL interaces how they work.

🔴 Make new function “DummyTest2023(string something): string” that modifies input string in some way, for example makes it All caps letters. String data type is tricky in these interfaces, that's why I've chosen especially this one for you to test.

4. After you compile the DLL in Delphi it is not possible to directly use it in C#. First you need to create the TLB file from the Delphi code and then you need to create from TLB the proxy DLL file. Make sure that you have all of the necessary SDKs installed.

First legacy_generate_midl.bat:

xxxxxxxxxx
3
1
midl "Chromsword.Logic.ridl" /tlb "Chromsword.Logic.tlb"
2
"C:\Program Files (x86)\CodeGear\RAD Studio\6.0\bin\tlibimp.exe" "Chromsword.Logic.tlb
3
pause

Second proxy_tlbimp.bat:

xxxxxxxxxx
2
1
"C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\tlbimp.exe" Chromsword.Logic.tlb /out:Chromsword.Logic.Proxy.dll
2
pause

Third copy.bat (Copy to write C#.NET directories and also include some requirements from the Pascal side.)

xxxxxxxxxx
4
1
copy .\Chromsword.Logic.dll ..\Chromsword.Common\Chromsword.Logic.dll
2
copy .\Sol_on_Real_Data_1.txt ..\Chromsword.Common\Sol_on_Real_Data_1.txt
3
copy .\ExactOptimGradSolution.dll ..\Chromsword.Common\ExactOptimGradSolution.dll
4
pause

5. Then in the Visual Studio, make sure that these DLLs are included in projects and very important after any changes you need to reload projects so that they would be visible or close the whole solution and open again.

6. Here are some of the imported comb objects in different parts of the Chromsword project.

⚠️ This is the special type of inherited object in COM where the Delphi is able to call these functions from that end.

7. Remember to debug your own new function. On the Delphi side you will need to have message boxes. The debugger on Delphi side will not work.

Delphi Pascal Issue - Re-Optimization does not work for 2 detectors in specific Developer tasks

https://www.notion.so/evalds/for-2-detectors-_neither-Restart-nor-Re-optimize-worked-6b18ad97e5504bd4bf282f25fcbe44e6?pvs=4

We have a notion, issue with reported problem that re-optimization and restart doesn't work for particular types of projects and this is happening because the Pascal code does not have at all re-optimization procedure for some of developer tasks. Most likely in near future you will need to add those and here are the C# location of these settings and also in the Pascal side usually Ukrainians who provide Pascal code send the code and you need to either adapt it or replace it in Delphi code so that it would work together with C# code.

The main Pascal code that usually changes is the protocol unit and math unit.

🔴 Your task! Ukrainians Oleg got back to us and explained that for any of these special optimization tasks, for re-optimization we should use different task with some of the settings as described here. Please change it and test it. We need to make sure that re-optimization for two detectors does work for all of the types of the optimizations.

Lazarus Offline Compilation

ChromSword Offline uses Lazarus 3 Pascal compiled executable that runs TCP server, which takes the JSON inputs and produces back the JSON outputs for the offline simulations and optimizations. Caution! There are different encoding schemes for JSON inputs and outputs as a security feature. It is not just clear text JSON. But the good news is that debugging in Lazarus works very well and you can work it together with offline C# debug in both ends.

To compile these project you will need following GIT repos:

1. https://bitbucket.org/evaldsurtans/chromsword_offline_5

2. https://bitbucket.org/evaldsurtans/chromsword-offline-service-fpc

1. Open this project file in Lazarus

2. using run menu you can build compile run and debug

3. In project options you can change between Debug and release modes. In the debug mode you can also put breakpoints.

   

The main difference between modes is that it will change the compiler constants and compiler constants enable some parts of the code and disable other parts of the code. Previously the code was built as Windows Service but it eat up a lot of CPU resources that's why now it is a simple EXE which is executed in the background without showing it and it works better for CPU resources. Remember that the code for Windows Service exists but we do not use it anymore.

4. The main function for executing all of the requests to JSON service is located here. TCSSession.Process(jRequest:TJSONObject) : TJSONObject

5. 

6. On C# side in the offline Program here is the TCP connection controller

7. As a test task, please create a new function in JSON service and try to call it from C# code, do some modification with the inputs in Lazarus Pascal and return it back to the C# code. Please make a careful documentation of this task. {message: "dummy_action input:string} => {output:string}

8. General description how this service works:

   1. You first insert all of the parameters for the model in the offline application and it sends to the offline service build model command. There are multiple types of models to build. It is chosen and then it returns back the model coefficients.

   2. using model co-options which you can initialize using the same JSON service so you don't need to build it every time. You can then simulate chromatograms.

   3. Or, optionally, after you initialize the model coefficients, you can also run optimization command which will find the best concentration gradient to separate the peaks. It is also happening on the Lazarus site right now.

9. Please study carefully documentation of Offline Service here: http://share.yellowrobot.xyz/quick/2024-2-14-D679BF67-A1E4-4FE6-AB5F-6105BDE9311F.html

Issue with Lazarus Offline Service - Optimization

🔴 Most likely we will need to implement the optimization not in Lazarus but in C# but then use for each of the optimization steps the chromatogram simulation command from the offline service. There are already ready-made Bayesian optimization libraries in C# which works way better than this Lazarus version.

We investigated in detail MC Optimzation problem in Offline (https://www.notion.so/evalds/ChromSword-5-2-Off-line-Optimization-time-problem-5189fe2721a64fd0adb5496104a367ce?pvs=4) and I can confirm that problem is with Alexei code unfortunatelly

These lines in screenshot below triggers premature optimization stop.

I can offer 2 solutions:

A. Below are all information needed for Alexei to fix his code himself (send all atached files to him and screenshots)

B. We implement our own C# based optimization algorithm to find best gradients (I advise to use Gaussian Process optimization rather MC)

Whenever you want to get more information from Offline data, you need to change offline.config and then you will see new menus in Offline Application!

Problem in Alexei pascal code: