Date   

Re: Increasing delay of task-execution times

Avgerinos
 

Keith and Frederik

Yes, the magic debug log was one of the the first I though,  but I did not have much luck at this time:
  • The programs have to run for 1-2 hours (or to be called for 9.000-10.000 times) until the delay becomes  significant, so the magic debug log becomes really huge and very difficult to manage.
  • The magic debug log, gives you no way to focus on specific sub-tasks or to be activated after certain amount of execution time: You can select only specific execution levels and aspects, like gateway, program flow etc
  • The actual information I managed to get from the magic debug log, was only confirming this increasing delay: like everything was gradually slowing down with no apparent reason. The flow was correct, the database was not slowing down etc...
Maybe, after one month of unsuccessful debugging, I should give it one more try. I need to add a new hard disk to my dev-machine only for hosting this debug-log. Plus I need to find a way to clean it up from all the unnecessary information.

Regards
Avgerinos

On 29/10/2019 6:55 π.μ., Frederik Soete wrote:

Hi, Avgerinos,

I assume Keith means the magic profiling that can be activated via Windows environment variables: look up the "MGPROF" setting on this group, it has been mentioned some times before, and can give some insight into task calls, file opens/closes and their durations (milliseconds). Magic is able to write two such log files on closing a runtime process. Can be handier than writing your own logging.


Op di 29 okt. 2019 05:49 schreef Avgerinos <mento@...>:
Hi Keith

Are you referring to the MSSQL Profiler? Because the problem exists even with calculation-subtasks that are not accessing any database

Regards
Avgerinos

On 29/10/2019 5:51 π.μ., Keith Canniff wrote:

Avgerinos,

 

Have you turned on the Profiler rather than using the mTime function inside the application? The Profiler for the most part has almost no “drag” on the application, yet can track all the details you’re indicating.

 

Might give you a different prospective on what’s going on.

 

Keith

 

From: main@magicu-l.groups.io <main@magicu-l.groups.io> On Behalf Of Avgerinos
Sent: Monday, October 28, 2019 11:47 PM
To: main@magicu-l.groups.io
Subject: Re: [magicu-l] Increasing delay of task-execution times

 

Thank you Andreas.
A memory possible leak was one of my first suspicions. But from what I can see in the task manager, only a very small increase in memory usage can be noticed after 200.000 event-calls, while the system still has plenty of memory available.

Another strange thing I noticed last night, was when I left a 2nd instance of magic running a different batch task at the same time, on the same host machine. This affected the behavior of the primary instance (the one running the timer-event):

  • The event-program starts executing 30% faster, so it finally takes 30% more time to reach the critical delay I mentioned before (the event bottleneck).
  • The mTime function, which I use for measuring the elapsed times, brings more accurate results:  task execution times are reported with 1 ms increments instead of 15-16 ms increments returned otherwise.

Funny thing, is that I got similar results, when instead of running a parallel magic instance, I simply opened GoogleChrome and left it open, on a simple, not refreshing page.

I wonder whether this is an indicator that the problem relates to the way Windows are handling W32 apps and their execution-time allowance.

Regards
Avgerinos

On 28/10/2019 9:14 μ.μ., Andreas Sedlmeier wrote:

Hi Avgerinos,


Sounds like a memory leak in Xpa to me what you describe. Did you monitor memory usage of your process ? You can do that f.i. with Windows Performance Monitor (https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/using-performance-monitor-to-find-a-user-mode-memory-leak).
If you find out that you do suffer from such a leak you are however doomed. Magic is not open source. In the past Magic was leaking when you did something assign to Blob variables and in order to avoit this you had to update the Blob variable wiith NULL() - to force Magic to free the memory.
Maybe this bug is back ? First check if you do suffer from a memory leak.

Best regards,

Andreas


Virus-free. www.avast.com



Re: Increasing delay of task-execution times

Frederik Soete
 

Hi, Avgerinos,

I assume Keith means the magic profiling that can be activated via Windows environment variables: look up the "MGPROF" setting on this group, it has been mentioned some times before, and can give some insight into task calls, file opens/closes and their durations (milliseconds). Magic is able to write two such log files on closing a runtime process. Can be handier than writing your own logging.


Op di 29 okt. 2019 05:49 schreef Avgerinos <mento@...>:

Hi Keith

Are you referring to the MSSQL Profiler? Because the problem exists even with calculation-subtasks that are not accessing any database

Regards
Avgerinos

On 29/10/2019 5:51 π.μ., Keith Canniff wrote:

Avgerinos,

 

Have you turned on the Profiler rather than using the mTime function inside the application? The Profiler for the most part has almost no “drag” on the application, yet can track all the details you’re indicating.

 

Might give you a different prospective on what’s going on.

 

Keith

 

From: main@magicu-l.groups.io <main@magicu-l.groups.io> On Behalf Of Avgerinos
Sent: Monday, October 28, 2019 11:47 PM
To: main@magicu-l.groups.io
Subject: Re: [magicu-l] Increasing delay of task-execution times

 

Thank you Andreas.
A memory possible leak was one of my first suspicions. But from what I can see in the task manager, only a very small increase in memory usage can be noticed after 200.000 event-calls, while the system still has plenty of memory available.

Another strange thing I noticed last night, was when I left a 2nd instance of magic running a different batch task at the same time, on the same host machine. This affected the behavior of the primary instance (the one running the timer-event):

  • The event-program starts executing 30% faster, so it finally takes 30% more time to reach the critical delay I mentioned before (the event bottleneck).
  • The mTime function, which I use for measuring the elapsed times, brings more accurate results:  task execution times are reported with 1 ms increments instead of 15-16 ms increments returned otherwise.

Funny thing, is that I got similar results, when instead of running a parallel magic instance, I simply opened GoogleChrome and left it open, on a simple, not refreshing page.

I wonder whether this is an indicator that the problem relates to the way Windows are handling W32 apps and their execution-time allowance.

Regards
Avgerinos

On 28/10/2019 9:14 μ.μ., Andreas Sedlmeier wrote:

Hi Avgerinos,


Sounds like a memory leak in Xpa to me what you describe. Did you monitor memory usage of your process ? You can do that f.i. with Windows Performance Monitor (https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/using-performance-monitor-to-find-a-user-mode-memory-leak).
If you find out that you do suffer from such a leak you are however doomed. Magic is not open source. In the past Magic was leaking when you did something assign to Blob variables and in order to avoit this you had to update the Blob variable wiith NULL() - to force Magic to free the memory.
Maybe this bug is back ? First check if you do suffer from a memory leak.

Best regards,

Andreas


Virus-free. www.avast.com


Re: Increasing delay of task-execution times

Avgerinos
 

Hi Keith

Are you referring to the MSSQL Profiler? Because the problem exists even with calculation-subtasks that are not accessing any database

Regards
Avgerinos

On 29/10/2019 5:51 π.μ., Keith Canniff wrote:

Avgerinos,

 

Have you turned on the Profiler rather than using the mTime function inside the application? The Profiler for the most part has almost no “drag” on the application, yet can track all the details you’re indicating.

 

Might give you a different prospective on what’s going on.

 

Keith

 

From: main@magicu-l.groups.io <main@magicu-l.groups.io> On Behalf Of Avgerinos
Sent: Monday, October 28, 2019 11:47 PM
To: main@magicu-l.groups.io
Subject: Re: [magicu-l] Increasing delay of task-execution times

 

Thank you Andreas.
A memory possible leak was one of my first suspicions. But from what I can see in the task manager, only a very small increase in memory usage can be noticed after 200.000 event-calls, while the system still has plenty of memory available.

Another strange thing I noticed last night, was when I left a 2nd instance of magic running a different batch task at the same time, on the same host machine. This affected the behavior of the primary instance (the one running the timer-event):

  • The event-program starts executing 30% faster, so it finally takes 30% more time to reach the critical delay I mentioned before (the event bottleneck).
  • The mTime function, which I use for measuring the elapsed times, brings more accurate results:  task execution times are reported with 1 ms increments instead of 15-16 ms increments returned otherwise.

Funny thing, is that I got similar results, when instead of running a parallel magic instance, I simply opened GoogleChrome and left it open, on a simple, not refreshing page.

I wonder whether this is an indicator that the problem relates to the way Windows are handling W32 apps and their execution-time allowance.

Regards
Avgerinos

On 28/10/2019 9:14 μ.μ., Andreas Sedlmeier wrote:

Hi Avgerinos,


Sounds like a memory leak in Xpa to me what you describe. Did you monitor memory usage of your process ? You can do that f.i. with Windows Performance Monitor (https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/using-performance-monitor-to-find-a-user-mode-memory-leak).
If you find out that you do suffer from such a leak you are however doomed. Magic is not open source. In the past Magic was leaking when you did something assign to Blob variables and in order to avoit this you had to update the Blob variable wiith NULL() - to force Magic to free the memory.
Maybe this bug is back ? First check if you do suffer from a memory leak.

Best regards,

Andreas


Virus-free. www.avast.com


Re: Increasing delay of task-execution times

Keith Canniff
 

Avgerinos,

 

Have you turned on the Profiler rather than using the mTime function inside the application? The Profiler for the most part has almost no “drag” on the application, yet can track all the details you’re indicating.

 

Might give you a different prospective on what’s going on.

 

Keith

 

From: main@magicu-l.groups.io <main@magicu-l.groups.io> On Behalf Of Avgerinos
Sent: Monday, October 28, 2019 11:47 PM
To: main@magicu-l.groups.io
Subject: Re: [magicu-l] Increasing delay of task-execution times

 

Thank you Andreas.
A memory possible leak was one of my first suspicions. But from what I can see in the task manager, only a very small increase in memory usage can be noticed after 200.000 event-calls, while the system still has plenty of memory available.

Another strange thing I noticed last night, was when I left a 2nd instance of magic running a different batch task at the same time, on the same host machine. This affected the behavior of the primary instance (the one running the timer-event):

  • The event-program starts executing 30% faster, so it finally takes 30% more time to reach the critical delay I mentioned before (the event bottleneck).
  • The mTime function, which I use for measuring the elapsed times, brings more accurate results:  task execution times are reported with 1 ms increments instead of 15-16 ms increments returned otherwise.

Funny thing, is that I got similar results, when instead of running a parallel magic instance, I simply opened GoogleChrome and left it open, on a simple, not refreshing page.

I wonder whether this is an indicator that the problem relates to the way Windows are handling W32 apps and their execution-time allowance.

Regards
Avgerinos

On 28/10/2019 9:14 μ.μ., Andreas Sedlmeier wrote:

Hi Avgerinos,


Sounds like a memory leak in Xpa to me what you describe. Did you monitor memory usage of your process ? You can do that f.i. with Windows Performance Monitor (https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/using-performance-monitor-to-find-a-user-mode-memory-leak).
If you find out that you do suffer from such a leak you are however doomed. Magic is not open source. In the past Magic was leaking when you did something assign to Blob variables and in order to avoit this you had to update the Blob variable wiith NULL() - to force Magic to free the memory.
Maybe this bug is back ? First check if you do suffer from a memory leak.

Best regards,

Andreas


Virus-free. www.avast.com


Re: Increasing delay of task-execution times

Avgerinos
 

Thank you Andreas.
A memory possible leak was one of my first suspicions. But from what I can see in the task manager, only a very small increase in memory usage can be noticed after 200.000 event-calls, while the system still has plenty of memory available.

Another strange thing I noticed last night, was when I left a 2nd instance of magic running a different batch task at the same time, on the same host machine. This affected the behavior of the primary instance (the one running the timer-event):
  • The event-program starts executing 30% faster, so it finally takes 30% more time to reach the critical delay I mentioned before (the event bottleneck).
  • The mTime function, which I use for measuring the elapsed times, brings more accurate results:  task execution times are reported with 1 ms increments instead of 15-16 ms increments returned otherwise.
Funny thing, is that I got similar results, when instead of running a parallel magic instance, I simply opened GoogleChrome and left it open, on a simple, not refreshing page.

I wonder whether this is an indicator that the problem relates to the way Windows are handling W32 apps and their execution-time allowance.

Regards
Avgerinos

On 28/10/2019 9:14 μ.μ., Andreas Sedlmeier wrote:

Hi Avgerinos,



Sounds like a memory leak in Xpa to me what you describe. Did you monitor memory usage of your process ? You can do that f.i. with Windows Performance Monitor (https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/using-performance-monitor-to-find-a-user-mode-memory-leak).
If you find out that you do suffer from such a leak you are however doomed. Magic is not open source. In the past Magic was leaking when you did something assign to Blob variables and in order to avoit this you had to update the Blob variable wiith NULL() - to force Magic to free the memory.
Maybe this bug is back ? First check if you do suffer from a memory leak.

Best regards,

Andreas
_._,_._,_




Re: Pervasive SQL V8.0 to a newer version

marwin mesa
 

Hi Steven

Thank you for your reply,
 
this would be the first time for me to install Pervasive/Actian Zen,  and i'm not sure of goldstar team  would actually assist us to the point of connecting it to our Unipaas 1.9C.. and if they are familiar with the unipaas systems
 
are you offiering that service or know someone who might be ?


Re: Pervasive SQL V8.0 to a newer version

Steven Blank
 

Marwin,

I would recommend at least v13, if not the latest version, v14 (released in August), and definitely 64-bit. The product has been rebranded as Actian Zen in v14, but it's fully backward-compatible with Magic's "Pervasive ISAM" and "Pervasive SQL" gateways.

For licenses and support, I recommend Bill Bach at Goldstar Software for both.

www.goldstarsoftware.com

Steve Blank

On 10/28/2019 6:36 PM, marwin mesa via Groups.Io wrote:

Hi Magic u/l group:

1) our current setup is we are using Pervasive SQL v8.10  in a Virtual Server running Windows Server 2003
2) We are using Unipaas 1.9C in  separate Windows Server 2003 32bit / Windows Server 2008 32 bit/ Windows 10 64 bit  to access that Pervasive SQL
 
3) In the future we would like to use a higher version of Magic XPA than the 1.9 C
4) we would probably install  The newer Pervasive SQL probably v12 or higher into a windows server 2012 64 bit
 
 
So the questions are:
 
1)  What is the latest Pervasive  SQL version and model that you have worked on before and/or you would suggest for us to use that can be installed on Windows Server 2012/2016  and  can be connected to Unipaas 1,9c/ Magic XPA
 
2) Which vendor/reseller do you suggest for us to have the Pervasive SQL?
 
3) Do someone offer Services in guiding us to  install/configure and connecting the new Pervasive SQL to Unipaas 1.9C  ?
 
Regards


Pervasive SQL V8.0 to a newer version

marwin mesa
 

Hi Magic u/l group:

1) our current setup is we are using Pervasive SQL v8.10  in a Virtual Server running Windows Server 2003
2) We are using Unipaas 1.9C in  separate Windows Server 2003 32bit / Windows Server 2008 32 bit/ Windows 10 64 bit  to access that Pervasive SQL
 
3) In the future we would like to use a higher version of Magic XPA than the 1.9 C
4) we would probably install  The newer Pervasive SQL probably v12 or higher into a windows server 2012 64 bit
 
 
So the questions are:
 
1)  What is the latest Pervasive  SQL version and model that you have worked on before and/or you would suggest for us to use that can be installed on Windows Server 2012/2016  and  can be connected to Unipaas 1,9c/ Magic XPA
 
2) Which vendor/reseller do you suggest for us to have the Pervasive SQL?
 
3) Do someone offer Services in guiding us to  install/configure and connecting the new Pervasive SQL to Unipaas 1.9C  ?
 
Regards


Re: Increasing delay of task-execution times

David Hudson
 

would declaring the tables help, or is using declares in the parent task just a way to speed  files opening in parent tasks so it doesnt have to open them over and over as the subtasks run.
I defer to the experts on that.

David Hudson
President GA Business Solutions
404-435-8630 

Meaningful data leads to better decisions.


On Mon, Oct 28, 2019 at 11:51 AM Avgerinos <mento@...> wrote:
Hi magicians

It's been 4 weeks that I 'm struggling with a weird behavior of XPA-runtime engine, noticed in all version versions from 3.3c and above, up 4.5a:

In a desktop app, there is a program executed by a timer event, once every second. This base-program is calling 20 resident sub-tasks (reading some data from database and making some calculations).
Some of the sub-tasks tasks read SQL2014 tables (on SSD drive), while the rest do only arithmetic calculations without accessing the database. 

During runtime, the whole process starts by taking around 350 milliseconds to execute, and this is quite acceptable for an event called every 1 second.
While executing the program keeps on slowing down: after 2 hours it will exceed the 1000 millisecond limit and will create an event-bottleneck.
Note that, this delay will not change if I stop and restart the event, or if I exit the program. The only way to reset back to normal execution times is to exit system and restart Magic.

While trying to locate the cause, I added time-logging code in all critical points. And this is where the strangest part comes:
  • Database accessing subtasks that were taking some time to execute (e.g. 15-ms), they still report the same execution time
  • Calculation subtasks with no access to database (exiting after a single record-suffix execution), report increasing execution times: they start from 0 ms and they gradually jump to 15-16 ms.
  • Note that these 15-16 ms seems to be the maximum accuracy I can get using the mTime() function for logging.
Some more remarks:
  • All subtasks are "non-abortable" (no events allowed)
  • I tried both options: resident or not resident tasks, and saw no difference
  • The behavior does not change whether executing from within the XPA-Studio (debug on or off), or directly with XPA-Runtime.
  • Tried many type of database locking schemes and saw no difference. Anyway the delay seems to happen more with non-database accessing subtasks
  • Tried replacing some of the SQL tables with memory tables, and saw no difference.
  • When tasks have already slowed down, the windows task-manager does not show any increase in CPU usage (always below 10%) or Memory usage (at least 40% of total memory free)

If anyone has similar experience or ideas, please let me know.

Thanks in advance

Avgerinos


Re: Data Source Properties - Resident option

Marc Gauthier
 

Thank You very much Todd

Best regards,


Re: Increasing delay of task-execution times

Andreas Sedlmeier
 

Hi Avgerinos,



Sounds like a memory leak in Xpa to me what you describe. Did you monitor memory usage of your process ? You can do that f.i. with Windows Performance Monitor (https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/using-performance-monitor-to-find-a-user-mode-memory-leak).
If you find out that you do suffer from such a leak you are however doomed. Magic is not open source. In the past Magic was leaking when you did something assign to Blob variables and in order to avoit this you had to update the Blob variable wiith NULL() - to force Magic to free the memory.
Maybe this bug is back ? First check if you do suffer from a memory leak.

Best regards,

Andreas


Re: Increasing delay of task-execution times

Frederik Soete
 

Hi Avgerinos,

Alas, I cannot confirm or deny your last question with any absolute certainty. If I were you I would check every tasks' transaction settings, including the one on top that is calling the subtasks. When no SQL is involved, e.g. for calculating tasks, I myself prefer physical transactions. I consider those the most sane default in that situation. But I have had the impression before that even virtuals (e.g. a success indicator) may take part in transactions, so that a rollback of the db may also cause a rollback for implicated virtuals.

In the particular case I was reminiscing about (trying to parse some log files), the topmost task of my program still had 'Within Transaction' and contained only virtal variables. Then again, some other tasks that did access datasources (memory tables), did too have 'Within Transaction'. After all those settings were changed to physical, the creeping slowness fortunately vanished. In my case, no SQL was involved, only virtuals and memory tables.

Op ma 28 okt. 2019 19:01 schreef Avgerinos <mento@...>:

Thank you Frederik

I will try. Do you think transaction settings may affect execution of tasks that are not accessing any database tables?

Regards
Avgerinos


On 28/10/2019 7:37 μ.μ., Frederik Soete wrote:
Hi, Avgerinos,

I have seen something like what you describe. In my case I was importing text files into memory tables, and the process of running my program seemed to take more and more time, although the text files in essence did not grow at the time.

In my case it turned out I had forgotten to adjust the transaction settings in some newly and hastily created tasks (e.g. the default 'Within Transaction) to something more reasonable. After some adjustments the tasks took a 'normal/expected' time again.

So, my idea is for you to check your tasks' transaction settings. Those can matter for performance.

HTH,

Frederik Soete

Op ma 28 okt. 2019 17:49 schreef Avgerinos <mento@...>:
Yes, I thought of this already: tried boosting magic's priority but it does not affect execution time at all

On 28/10/2019 6:31 μ.μ., Jeroen Bos via Groups.Io wrote:

Just a thought, can it be that the program gets a lower execution priority or maybe that the program memory is replaced to the virtual memory?

 

Met vriendelijke groet / Yours sincerely



De bronafbeelding bekijken
 

 


Jeroen Bos | Technisch Consultant (Technical Consultant)

 

Van: main@magicu-l.groups.io <main@magicu-l.groups.io> Namens Avgerinos via Groups.Io
Verzonden: maandag 28 oktober 2019 16:51
Aan: main@magicu-l.groups.io
Onderwerp: [magicu-l] Increasing delay of task-execution times

 

Hi magicians

It's been 4 weeks that I 'm struggling with a weird behavior of XPA-runtime engine, noticed in all version versions from 3.3c and above, up 4.5a:

In a desktop app, there is a program executed by a timer event, once every second. This base-program is calling 20 resident sub-tasks (reading some data from database and making some calculations).
Some of the sub-tasks tasks read SQL2014 tables (on SSD drive), while the rest do only arithmetic calculations without accessing the database. 

During runtime, the whole process starts by taking around 350 milliseconds to execute, and this is quite acceptable for an event called every 1 second.
While executing the program keeps on slowing down: after 2 hours it will exceed the 1000 millisecond limit and will create an event-bottleneck.
Note that, this delay will not change if I stop and restart the event, or if I exit the program. The only way to reset back to normal execution times is to exit system and restart Magic.

While trying to locate the cause, I added time-logging code in all critical points. And this is where the strangest part comes:

  • Database accessing subtasks that were taking some time to execute (e.g. 15-ms), they still report the same execution time
  • Calculation subtasks with no access to database (exiting after a single record-suffix execution), report increasing execution times: they start from 0 ms and they gradually jump to 15-16 ms.
  • Note that these 15-16 ms seems to be the maximum accuracy I can get using the mTime() function for logging.

Some more remarks:

  • All subtasks are "non-abortable" (no events allowed)
  • I tried both options: resident or not resident tasks, and saw no difference
  • The behavior does not change whether executing from within the XPA-Studio (debug on or off), or directly with XPA-Runtime.
  • Tried many type of database locking schemes and saw no difference. Anyway the delay seems to happen more with non-database accessing subtasks
  • Tried replacing some of the SQL tables with memory tables, and saw no difference.
  • When tasks have already slowed down, the windows task-manager does not show any increase in CPU usage (always below 10%) or Memory usage (at least 40% of total memory free)

If anyone has similar experience or ideas, please let me know.

Thanks in advance

Avgerinos

-------------------------------------------------------------------------------------------------
This e-mail may contain confidential and/or privileged information. If you are not the
intended receipient (or have received this e-mail in error) please notify the sender
immediately and destroy this e-mail.
Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.

Deze e-mail kan vertrouwelijke en/of persoonlijke informatie bevatten. Als u niet de bedoelde
ontvanger bent, of deze e-mail per ongeluk heeft gekregen, breng dan de verzender
ogenblikkelijk hiervan op de hoogte en vernietig dit bericht.
Het kopieren, in de openbaarheid brengen of verspreiden van de inhoud van deze e-mail is ten
strengste verboden.

Ce courriel peut contenir des informations confidentielles et/ou privilegiees. Si ce courriel ne vous est pas destine
(ou vous avez recu ce courriel par erreur) svp, avertissez immediatement la personne qui a envoye ce courriel et detruisez le.
Toute copie non autorisee, divulgation des informations ou distribution du contenu de ce courriel sont strictement interdites.
-------------------------------------------------------------------------------------------------



Re: Increasing delay of task-execution times

Avgerinos
 

Thank you Frederik

I will try. Do you think transaction settings may affect execution of tasks that are not accessing any database tables?

Regards
Avgerinos


On 28/10/2019 7:37 μ.μ., Frederik Soete wrote:

Hi, Avgerinos,

I have seen something like what you describe. In my case I was importing text files into memory tables, and the process of running my program seemed to take more and more time, although the text files in essence did not grow at the time.

In my case it turned out I had forgotten to adjust the transaction settings in some newly and hastily created tasks (e.g. the default 'Within Transaction) to something more reasonable. After some adjustments the tasks took a 'normal/expected' time again.

So, my idea is for you to check your tasks' transaction settings. Those can matter for performance.

HTH,

Frederik Soete

Op ma 28 okt. 2019 17:49 schreef Avgerinos <mento@...>:
Yes, I thought of this already: tried boosting magic's priority but it does not affect execution time at all

On 28/10/2019 6:31 μ.μ., Jeroen Bos via Groups.Io wrote:

Just a thought, can it be that the program gets a lower execution priority or maybe that the program memory is replaced to the virtual memory?

 

Met vriendelijke groet / Yours sincerely



 

 


Jeroen Bos | Technisch Consultant (Technical Consultant)

 

Van: main@magicu-l.groups.io <main@magicu-l.groups.io> Namens Avgerinos via Groups.Io
Verzonden: maandag 28 oktober 2019 16:51
Aan: main@magicu-l.groups.io
Onderwerp: [magicu-l] Increasing delay of task-execution times

 

Hi magicians

It's been 4 weeks that I 'm struggling with a weird behavior of XPA-runtime engine, noticed in all version versions from 3.3c and above, up 4.5a:

In a desktop app, there is a program executed by a timer event, once every second. This base-program is calling 20 resident sub-tasks (reading some data from database and making some calculations).
Some of the sub-tasks tasks read SQL2014 tables (on SSD drive), while the rest do only arithmetic calculations without accessing the database. 

During runtime, the whole process starts by taking around 350 milliseconds to execute, and this is quite acceptable for an event called every 1 second.
While executing the program keeps on slowing down: after 2 hours it will exceed the 1000 millisecond limit and will create an event-bottleneck.
Note that, this delay will not change if I stop and restart the event, or if I exit the program. The only way to reset back to normal execution times is to exit system and restart Magic.

While trying to locate the cause, I added time-logging code in all critical points. And this is where the strangest part comes:

  • Database accessing subtasks that were taking some time to execute (e.g. 15-ms), they still report the same execution time
  • Calculation subtasks with no access to database (exiting after a single record-suffix execution), report increasing execution times: they start from 0 ms and they gradually jump to 15-16 ms.
  • Note that these 15-16 ms seems to be the maximum accuracy I can get using the mTime() function for logging.

Some more remarks:

  • All subtasks are "non-abortable" (no events allowed)
  • I tried both options: resident or not resident tasks, and saw no difference
  • The behavior does not change whether executing from within the XPA-Studio (debug on or off), or directly with XPA-Runtime.
  • Tried many type of database locking schemes and saw no difference. Anyway the delay seems to happen more with non-database accessing subtasks
  • Tried replacing some of the SQL tables with memory tables, and saw no difference.
  • When tasks have already slowed down, the windows task-manager does not show any increase in CPU usage (always below 10%) or Memory usage (at least 40% of total memory free)

If anyone has similar experience or ideas, please let me know.

Thanks in advance

Avgerinos

-------------------------------------------------------------------------------------------------
This e-mail may contain confidential and/or privileged information. If you are not the
intended receipient (or have received this e-mail in error) please notify the sender
immediately and destroy this e-mail.
Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.

Deze e-mail kan vertrouwelijke en/of persoonlijke informatie bevatten. Als u niet de bedoelde
ontvanger bent, of deze e-mail per ongeluk heeft gekregen, breng dan de verzender
ogenblikkelijk hiervan op de hoogte en vernietig dit bericht.
Het kopieren, in de openbaarheid brengen of verspreiden van de inhoud van deze e-mail is ten
strengste verboden.

Ce courriel peut contenir des informations confidentielles et/ou privilegiees. Si ce courriel ne vous est pas destine
(ou vous avez recu ce courriel par erreur) svp, avertissez immediatement la personne qui a envoye ce courriel et detruisez le.
Toute copie non autorisee, divulgation des informations ou distribution du contenu de ce courriel sont strictement interdites.
-------------------------------------------------------------------------------------------------



Re: Data Source Properties - Resident option

Todd Baremore
 

Marc,

In magic.ini
LoadResidentFiles = Y
Todd
On 10/28/2019 1:25 PM, Marc Gauthier wrote:

Hi,

Anybody was able to make the Resident option with the DBMS Btrieve ? I'm using  XPA 3.2e

I'm trying to get a small table to be resident. But according to the Monitor (PSQL Control Center Monitor)  the file is still getting open more than once...

I try all the option (Immediate, On Demand and Immediate and on Client) and I restart XPA completely. Always the same result, the table is still getting open more than once by my user.

Thank You very much!

Marc




Re: Increasing delay of task-execution times

Frederik Soete
 

Hi, Avgerinos,

I have seen something like what you describe. In my case I was importing text files into memory tables, and the process of running my program seemed to take more and more time, although the text files in essence did not grow at the time.

In my case it turned out I had forgotten to adjust the transaction settings in some newly and hastily created tasks (e.g. the default 'Within Transaction) to something more reasonable. After some adjustments the tasks took a 'normal/expected' time again.

So, my idea is for you to check your tasks' transaction settings. Those can matter for performance.

HTH,

Frederik Soete

Op ma 28 okt. 2019 17:49 schreef Avgerinos <mento@...>:

Yes, I thought of this already: tried boosting magic's priority but it does not affect execution time at all

On 28/10/2019 6:31 μ.μ., Jeroen Bos via Groups.Io wrote:

Just a thought, can it be that the program gets a lower execution priority or maybe that the program memory is replaced to the virtual memory?

 

Met vriendelijke groet / Yours sincerely



 

 


Jeroen Bos | Technisch Consultant (Technical Consultant)

 

Van: main@magicu-l.groups.io <main@magicu-l.groups.io> Namens Avgerinos via Groups.Io
Verzonden: maandag 28 oktober 2019 16:51
Aan: main@magicu-l.groups.io
Onderwerp: [magicu-l] Increasing delay of task-execution times

 

Hi magicians

It's been 4 weeks that I 'm struggling with a weird behavior of XPA-runtime engine, noticed in all version versions from 3.3c and above, up 4.5a:

In a desktop app, there is a program executed by a timer event, once every second. This base-program is calling 20 resident sub-tasks (reading some data from database and making some calculations).
Some of the sub-tasks tasks read SQL2014 tables (on SSD drive), while the rest do only arithmetic calculations without accessing the database. 

During runtime, the whole process starts by taking around 350 milliseconds to execute, and this is quite acceptable for an event called every 1 second.
While executing the program keeps on slowing down: after 2 hours it will exceed the 1000 millisecond limit and will create an event-bottleneck.
Note that, this delay will not change if I stop and restart the event, or if I exit the program. The only way to reset back to normal execution times is to exit system and restart Magic.

While trying to locate the cause, I added time-logging code in all critical points. And this is where the strangest part comes:

  • Database accessing subtasks that were taking some time to execute (e.g. 15-ms), they still report the same execution time
  • Calculation subtasks with no access to database (exiting after a single record-suffix execution), report increasing execution times: they start from 0 ms and they gradually jump to 15-16 ms.
  • Note that these 15-16 ms seems to be the maximum accuracy I can get using the mTime() function for logging.

Some more remarks:

  • All subtasks are "non-abortable" (no events allowed)
  • I tried both options: resident or not resident tasks, and saw no difference
  • The behavior does not change whether executing from within the XPA-Studio (debug on or off), or directly with XPA-Runtime.
  • Tried many type of database locking schemes and saw no difference. Anyway the delay seems to happen more with non-database accessing subtasks
  • Tried replacing some of the SQL tables with memory tables, and saw no difference.
  • When tasks have already slowed down, the windows task-manager does not show any increase in CPU usage (always below 10%) or Memory usage (at least 40% of total memory free)

If anyone has similar experience or ideas, please let me know.

Thanks in advance

Avgerinos

-------------------------------------------------------------------------------------------------
This e-mail may contain confidential and/or privileged information. If you are not the
intended receipient (or have received this e-mail in error) please notify the sender
immediately and destroy this e-mail.
Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.

Deze e-mail kan vertrouwelijke en/of persoonlijke informatie bevatten. Als u niet de bedoelde
ontvanger bent, of deze e-mail per ongeluk heeft gekregen, breng dan de verzender
ogenblikkelijk hiervan op de hoogte en vernietig dit bericht.
Het kopieren, in de openbaarheid brengen of verspreiden van de inhoud van deze e-mail is ten
strengste verboden.

Ce courriel peut contenir des informations confidentielles et/ou privilegiees. Si ce courriel ne vous est pas destine
(ou vous avez recu ce courriel par erreur) svp, avertissez immediatement la personne qui a envoye ce courriel et detruisez le.
Toute copie non autorisee, divulgation des informations ou distribution du contenu de ce courriel sont strictement interdites.
-------------------------------------------------------------------------------------------------


Data Source Properties - Resident option

Marc Gauthier
 

Hi,

Anybody was able to make the Resident option with the DBMS Btrieve ? I'm using  XPA 3.2e

I'm trying to get a small table to be resident. But according to the Monitor (PSQL Control Center Monitor)  the file is still getting open more than once...

I try all the option (Immediate, On Demand and Immediate and on Client) and I restart XPA completely. Always the same result, the table is still getting open more than once by my user.

Thank You very much!

Marc



Re: Increasing delay of task-execution times

Avgerinos
 

Yes, I thought of this already: tried boosting magic's priority but it does not affect execution time at all

On 28/10/2019 6:31 μ.μ., Jeroen Bos via Groups.Io wrote:

Just a thought, can it be that the program gets a lower execution priority or maybe that the program memory is replaced to the virtual memory?

 

Met vriendelijke groet / Yours sincerely



De bronafbeelding bekijken
 

 


Jeroen Bos | Technisch Consultant (Technical Consultant)

 

Van: main@magicu-l.groups.io <main@magicu-l.groups.io> Namens Avgerinos via Groups.Io
Verzonden: maandag 28 oktober 2019 16:51
Aan: main@magicu-l.groups.io
Onderwerp: [magicu-l] Increasing delay of task-execution times

 

Hi magicians

It's been 4 weeks that I 'm struggling with a weird behavior of XPA-runtime engine, noticed in all version versions from 3.3c and above, up 4.5a:

In a desktop app, there is a program executed by a timer event, once every second. This base-program is calling 20 resident sub-tasks (reading some data from database and making some calculations).
Some of the sub-tasks tasks read SQL2014 tables (on SSD drive), while the rest do only arithmetic calculations without accessing the database. 

During runtime, the whole process starts by taking around 350 milliseconds to execute, and this is quite acceptable for an event called every 1 second.
While executing the program keeps on slowing down: after 2 hours it will exceed the 1000 millisecond limit and will create an event-bottleneck.
Note that, this delay will not change if I stop and restart the event, or if I exit the program. The only way to reset back to normal execution times is to exit system and restart Magic.

While trying to locate the cause, I added time-logging code in all critical points. And this is where the strangest part comes:

  • Database accessing subtasks that were taking some time to execute (e.g. 15-ms), they still report the same execution time
  • Calculation subtasks with no access to database (exiting after a single record-suffix execution), report increasing execution times: they start from 0 ms and they gradually jump to 15-16 ms.
  • Note that these 15-16 ms seems to be the maximum accuracy I can get using the mTime() function for logging.

Some more remarks:

  • All subtasks are "non-abortable" (no events allowed)
  • I tried both options: resident or not resident tasks, and saw no difference
  • The behavior does not change whether executing from within the XPA-Studio (debug on or off), or directly with XPA-Runtime.
  • Tried many type of database locking schemes and saw no difference. Anyway the delay seems to happen more with non-database accessing subtasks
  • Tried replacing some of the SQL tables with memory tables, and saw no difference.
  • When tasks have already slowed down, the windows task-manager does not show any increase in CPU usage (always below 10%) or Memory usage (at least 40% of total memory free)

If anyone has similar experience or ideas, please let me know.

Thanks in advance

Avgerinos

-------------------------------------------------------------------------------------------------
This e-mail may contain confidential and/or privileged information. If you are not the
intended receipient (or have received this e-mail in error) please notify the sender
immediately and destroy this e-mail.
Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.

Deze e-mail kan vertrouwelijke en/of persoonlijke informatie bevatten. Als u niet de bedoelde
ontvanger bent, of deze e-mail per ongeluk heeft gekregen, breng dan de verzender
ogenblikkelijk hiervan op de hoogte en vernietig dit bericht.
Het kopieren, in de openbaarheid brengen of verspreiden van de inhoud van deze e-mail is ten
strengste verboden.

Ce courriel peut contenir des informations confidentielles et/ou privilegiees. Si ce courriel ne vous est pas destine
(ou vous avez recu ce courriel par erreur) svp, avertissez immediatement la personne qui a envoye ce courriel et detruisez le.
Toute copie non autorisee, divulgation des informations ou distribution du contenu de ce courriel sont strictement interdites.
-------------------------------------------------------------------------------------------------


Re: Increasing delay of task-execution times

Jeroen Bos
 

Just a thought, can it be that the program gets a lower execution priority or maybe that the program memory is replaced to the virtual memory?

 

Met vriendelijke groet / Yours sincerely

De bronafbeelding bekijken
 

 


Jeroen Bos | Technisch Consultant (Technical Consultant)

 

Van: main@magicu-l.groups.io <main@magicu-l.groups.io> Namens Avgerinos via Groups.Io
Verzonden: maandag 28 oktober 2019 16:51
Aan: main@magicu-l.groups.io
Onderwerp: [magicu-l] Increasing delay of task-execution times

 

Hi magicians

It's been 4 weeks that I 'm struggling with a weird behavior of XPA-runtime engine, noticed in all version versions from 3.3c and above, up 4.5a:

In a desktop app, there is a program executed by a timer event, once every second. This base-program is calling 20 resident sub-tasks (reading some data from database and making some calculations).
Some of the sub-tasks tasks read SQL2014 tables (on SSD drive), while the rest do only arithmetic calculations without accessing the database. 

During runtime, the whole process starts by taking around 350 milliseconds to execute, and this is quite acceptable for an event called every 1 second.
While executing the program keeps on slowing down: after 2 hours it will exceed the 1000 millisecond limit and will create an event-bottleneck.
Note that, this delay will not change if I stop and restart the event, or if I exit the program. The only way to reset back to normal execution times is to exit system and restart Magic.

While trying to locate the cause, I added time-logging code in all critical points. And this is where the strangest part comes:

  • Database accessing subtasks that were taking some time to execute (e.g. 15-ms), they still report the same execution time
  • Calculation subtasks with no access to database (exiting after a single record-suffix execution), report increasing execution times: they start from 0 ms and they gradually jump to 15-16 ms.
  • Note that these 15-16 ms seems to be the maximum accuracy I can get using the mTime() function for logging.

Some more remarks:

  • All subtasks are "non-abortable" (no events allowed)
  • I tried both options: resident or not resident tasks, and saw no difference
  • The behavior does not change whether executing from within the XPA-Studio (debug on or off), or directly with XPA-Runtime.
  • Tried many type of database locking schemes and saw no difference. Anyway the delay seems to happen more with non-database accessing subtasks
  • Tried replacing some of the SQL tables with memory tables, and saw no difference.
  • When tasks have already slowed down, the windows task-manager does not show any increase in CPU usage (always below 10%) or Memory usage (at least 40% of total memory free)

If anyone has similar experience or ideas, please let me know.

Thanks in advance

Avgerinos

-------------------------------------------------------------------------------------------------
This e-mail may contain confidential and/or privileged information. If you are not the
intended receipient (or have received this e-mail in error) please notify the sender
immediately and destroy this e-mail.
Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.

Deze e-mail kan vertrouwelijke en/of persoonlijke informatie bevatten. Als u niet de bedoelde
ontvanger bent, of deze e-mail per ongeluk heeft gekregen, breng dan de verzender
ogenblikkelijk hiervan op de hoogte en vernietig dit bericht.
Het kopieren, in de openbaarheid brengen of verspreiden van de inhoud van deze e-mail is ten
strengste verboden.

Ce courriel peut contenir des informations confidentielles et/ou privilegiees. Si ce courriel ne vous est pas destine
(ou vous avez recu ce courriel par erreur) svp, avertissez immediatement la personne qui a envoye ce courriel et detruisez le.
Toute copie non autorisee, divulgation des informations ou distribution du contenu de ce courriel sont strictement interdites.
-------------------------------------------------------------------------------------------------


Re: Increasing delay of task-execution times

Andy Jerison
 

I'd look very closely at every expression in the problem tasks. 


On Mon, Oct 28, 2019, 11:51 Avgerinos <mento@...> wrote:
Hi magicians

It's been 4 weeks that I 'm struggling with a weird behavior of XPA-runtime engine, noticed in all version versions from 3.3c and above, up 4.5a:

In a desktop app, there is a program executed by a timer event, once every second. This base-program is calling 20 resident sub-tasks (reading some data from database and making some calculations).
Some of the sub-tasks tasks read SQL2014 tables (on SSD drive), while the rest do only arithmetic calculations without accessing the database. 

During runtime, the whole process starts by taking around 350 milliseconds to execute, and this is quite acceptable for an event called every 1 second.
While executing the program keeps on slowing down: after 2 hours it will exceed the 1000 millisecond limit and will create an event-bottleneck.
Note that, this delay will not change if I stop and restart the event, or if I exit the program. The only way to reset back to normal execution times is to exit system and restart Magic.

While trying to locate the cause, I added time-logging code in all critical points. And this is where the strangest part comes:
  • Database accessing subtasks that were taking some time to execute (e.g. 15-ms), they still report the same execution time
  • Calculation subtasks with no access to database (exiting after a single record-suffix execution), report increasing execution times: they start from 0 ms and they gradually jump to 15-16 ms.
  • Note that these 15-16 ms seems to be the maximum accuracy I can get using the mTime() function for logging.
Some more remarks:
  • All subtasks are "non-abortable" (no events allowed)
  • I tried both options: resident or not resident tasks, and saw no difference
  • The behavior does not change whether executing from within the XPA-Studio (debug on or off), or directly with XPA-Runtime.
  • Tried many type of database locking schemes and saw no difference. Anyway the delay seems to happen more with non-database accessing subtasks
  • Tried replacing some of the SQL tables with memory tables, and saw no difference.
  • When tasks have already slowed down, the windows task-manager does not show any increase in CPU usage (always below 10%) or Memory usage (at least 40% of total memory free)

If anyone has similar experience or ideas, please let me know.

Thanks in advance

Avgerinos


Increasing delay of task-execution times

Avgerinos
 

Hi magicians

It's been 4 weeks that I 'm struggling with a weird behavior of XPA-runtime engine, noticed in all version versions from 3.3c and above, up 4.5a:

In a desktop app, there is a program executed by a timer event, once every second. This base-program is calling 20 resident sub-tasks (reading some data from database and making some calculations).
Some of the sub-tasks tasks read SQL2014 tables (on SSD drive), while the rest do only arithmetic calculations without accessing the database. 

During runtime, the whole process starts by taking around 350 milliseconds to execute, and this is quite acceptable for an event called every 1 second.
While executing the program keeps on slowing down: after 2 hours it will exceed the 1000 millisecond limit and will create an event-bottleneck.
Note that, this delay will not change if I stop and restart the event, or if I exit the program. The only way to reset back to normal execution times is to exit system and restart Magic.

While trying to locate the cause, I added time-logging code in all critical points. And this is where the strangest part comes:
  • Database accessing subtasks that were taking some time to execute (e.g. 15-ms), they still report the same execution time
  • Calculation subtasks with no access to database (exiting after a single record-suffix execution), report increasing execution times: they start from 0 ms and they gradually jump to 15-16 ms.
  • Note that these 15-16 ms seems to be the maximum accuracy I can get using the mTime() function for logging.
Some more remarks:
  • All subtasks are "non-abortable" (no events allowed)
  • I tried both options: resident or not resident tasks, and saw no difference
  • The behavior does not change whether executing from within the XPA-Studio (debug on or off), or directly with XPA-Runtime.
  • Tried many type of database locking schemes and saw no difference. Anyway the delay seems to happen more with non-database accessing subtasks
  • Tried replacing some of the SQL tables with memory tables, and saw no difference.
  • When tasks have already slowed down, the windows task-manager does not show any increase in CPU usage (always below 10%) or Memory usage (at least 40% of total memory free)

If anyone has similar experience or ideas, please let me know.

Thanks in advance

Avgerinos

4941 - 4960 of 196382