From Newsgroup: comp.lang.tcl


Hi,

I have the following program that create and then delete
a large data set in 3 manners.

I tested it under tcltk-8.6.10-barebones-x64 , tcl-9.0.1-installer-2.0.3-x64.msi
and with struct::matrix 2.0.4, struct::matrix 2.2, tests crossing :

1) "unset" usage in global space
2) "unset" usage in a procedure
3) "namespace delete"

What puzzles me is that I often read that Tcl does not release memory.
But according to the tasks manager ( Windows 11), with an accuracy of 0.1 Go, "unset" perfectly release memory, while "namespace delete" makes nothing.

So I wonder if :

1) Can it be an Ashok's "magic" spell (binaries are from him) ?
2) Is it normal that what can be done in global name space cannot
be done in another namespace ? ( I have also added some"array unset"
or "unset -nocomplain" to eradicate any possibility in
::struct::matrix::_destroy , wich I give below the original code)

For the brave I have an extra question :

In the matrix package there are 18 "catch {unset xyz}", if I replace
them by "unset -nocomplain xyz", the execution time is reduced
by a factor 8-10. Unless it has been forgotten after the apparition
of "-complain", could be there a trick to use catch / unset, in the
place of unset -nocomplain ?

Best regards, Olivier.

oooooooooo START ::struct::matrix::_destroy oooooooooooooo

proc ::struct::matrix::_destroy {name} {
variable ${name}::link

Unlink all existing arrays before destroying the object so that
we don't leave dangling references / traces.

foreach avar [array names link] {
_unlink $name $avar
}

namespace delete $name
interp alias {} $name {}
}

oooooooooo END ::struct::matrix::_destroy oooooooooooooo

oooooooooo START MEMORY TEST oooooooooooooo

# 1) UNSET USAGE IN GLOBAL SPACE

global localDir ; set localDir [file dirname [info script]]

source [ file join "$localDir" matrix.tcl ]

for { set i 0 } { $i < 5 } { incr i } {
set _BIG [ lrepeat 500000000 "dummy" ]
unset _BIG
}

tk_messageBox -title "Memory Test" -message "Check mem 1 !"

# 2) UNSET USAGE IN A PROCEDURE

proc testmem {} {
set _BIG [ lrepeat 500000000 "dummy" ]
}

for { set i 0 } { $i < 5 } { incr i } {
testmem
}

tk_messageBox -title "Memory Test" -message "Check mem 2 !"

# 3) NAMESPACE DELETE

set _BIG [ lrepeat 5000000 "dummy" ]

# if {[namespace exists _matrix_in]} {
# _matrix_in destroy
# }

::struct::matrix _matrix_in

_matrix_in add columns 1
_matrix_in add rows [ llength $_BIG ]
_matrix_in set column 0 $_BIG
_matrix_in destroy

# an extra delete !

namespace delete ::struct::matrix

tk_messageBox -title "Memory Test" -message "Check mem 3 !"

exit

oooooooooo END MEMORY TEST oooooooooooooo
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Hi Oliver,
others may answer in more detail, no time, some remarks:

Am 09.09.2025 um 12:02 schrieb Olivier:

Hi,

What puzzles me is that I often read that Tcl does not release memory.
But according to the tasks manager ( Windows 11), with an accuracy of 0.1 Go, "unset" perfectly release memory, while "namespace delete" makes nothing.

It is not true, that Tcl never releases memory.
There are different memory managers controlled by compile options.
And this is platform specific.

In some circumstances, there is the optimization, that memory was never released. This is true for some internal memory managers.

But the release moment is often a challenge.

Example. If you type in the console:
set test [string repeat 1000000 10000000]
delete test

Then, the history package still keeps a copy of the data and it is not released.

So, the release moment may be tricky.
If you delete a namespace: are you sure, nothing has a copy of your huge
data?

So I wonder if :

1) Can it be an Ashok's "magic" spell (binaries are from him) ?

Absolutly YES, but others have the magic too.

2) Is it normal that what can be done in global name space cannot
be done in another namespace ? ( I have also added some"array unset"
or "unset -nocomplain" to eradicate any possibility in
::struct::matrix::_destroy , wich I give below the original code)

For the brave I have an extra question :

In the matrix package there are 18 "catch {unset xyz}", if I replace
them by "unset -nocomplain xyz", the execution time is reduced
by a factor 8-10. Unless it has been forgotten after the apparition
of "-complain", could be there a trick to use catch / unset, in the
place of unset -nocomplain ?

I think, the catch version is slower in the error case. It saves the
call stack and the error message in global variables.

Best regards, Olivier.

oooooooooo START ::struct::matrix::_destroy oooooooooooooo

proc ::struct::matrix::_destroy {name} {
variable ${name}::link

Unlink all existing arrays before destroying the object so that
we don't leave dangling references / traces.

foreach avar [array names link] {
_unlink $name $avar
}

namespace delete $name
interp alias {} $name {}
}

oooooooooo END ::struct::matrix::_destroy oooooooooooooo

oooooooooo START MEMORY TEST oooooooooooooo

# 1) UNSET USAGE IN GLOBAL SPACE

global localDir ; set localDir [file dirname [info script]]

source [ file join "$localDir" matrix.tcl ]

for { set i 0 } { $i < 5 } { incr i } {
set _BIG [ lrepeat 500000000 "dummy" ]
unset _BIG
}

tk_messageBox -title "Memory Test" -message "Check mem 1 !"

# 2) UNSET USAGE IN A PROCEDURE

proc testmem {} {
set _BIG [ lrepeat 500000000 "dummy" ]
}

for { set i 0 } { $i < 5 } { incr i } {
testmem
}

tk_messageBox -title "Memory Test" -message "Check mem 2 !"

# 3) NAMESPACE DELETE

set _BIG [ lrepeat 5000000 "dummy" ]

# if {[namespace exists _matrix_in]} {
# _matrix_in destroy
# }

::struct::matrix _matrix_in

_matrix_in add columns 1
_matrix_in add rows [ llength $_BIG ]
_matrix_in set column 0 $_BIG
_matrix_in destroy

# an extra delete !

namespace delete ::struct::matrix

tk_messageBox -title "Memory Test" -message "Check mem 3 !"

exit

oooooooooo END MEMORY TEST oooooooooooooo

--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 9/9/2025 3:02 AM, Olivier wrote:

What puzzles me is that I often read that Tcl does not release memory.
But according to the tasks manager ( Windows 11), with an accuracy of 0.1 Go, "unset" perfectly release memory, while "namespace delete" makes nothing.

How are you measuring the current memory? Since it seems you are on windows, then there are several items this could be. I don't think tcl can release virtual memory allocated back to the operating system by shrinking the process at the end. And it's almost certain that it can't do that if the memory freed up is somewhere in the middle of the heap. In this case, it would put the freed up memory into a list of free memory, to be reused on the next allocation.

In windows, there is also working set, which is the amount of physical memory being used. This can shrink, even if you don't free up any virtual memory. The working set is managed completely by the OS, although there may be some system calls that can increase or decrease that. But that's not the same as the virtual memory space.

I don't think windows 11 has changed any of this, but then anything is possible.




--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 9/9/2025 3:08 PM, et99 wrote:

On 9/9/2025 3:02 AM, Olivier wrote:

What puzzles me is that I often read that Tcl does not release memory.
But according to the tasks manager ( Windows 11), with an accuracy of 0.1 Go,
"unset" perfectly release memory, while "namespace delete" makes nothing.

How are you measuring the current memory? Since it seems you are on windows, then there are several items this could be. I don't think tcl can release virtual memory allocated back to the operating system by shrinking the process at the end. And it's almost certain that it can't do that if the memory freed up is somewhere in the middle of the heap. In this case, it would put the freed up memory into a list of free memory, to be reused on the next allocation.

In windows, there is also working set, which is the amount of physical memory being used. This can shrink, even if you don't free up any virtual memory. The working set is managed completely by the OS, although there may be some system calls that can increase or decrease that. But that's not the same as the virtual memory space.

I don't think windows 11 has changed any of this, but then anything is possible.

I stand corrected on this. Apparently both tcl and windows CAN release memory back to the OS that was part of the virtual address space, even if it was in the middle of the process address space. However, windows task monitor memory is physical memory, not virtual memory.

To see this, you might want to get the program process explorer from microsoft,

https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer

Here you can monitor many aspects of memory, in particular, private bytes might be what you want to look at, although, virtual memory can also grow and shrink, but that's occupied by many other things besides program memory, such as dll's etc.





--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl


Hi Harald,

others may answer in more detail, no time, some remarks:

There are enough details, to help me !

I'm dealing with 1d simulation ( Spice) or finite elements analysis,
and parametrics analysis, this explains the use of large datasets.

It is not true, that Tcl never releases memory.
....
Then, the history package still keeps a copy of the data and it is not >released.

It is a valuable answer because it means, I would better use the results
of simulation as persistent database instead of make use of "read" command,
and load the whole file . I mean : opening the file, take the values of one signal, make usage of "seek", check for another signal, etc. Closing the file.

This will prevent too much memory use, in case of switch in compilation
that disable memory optimization.

Absolutly YES, but others have the magic too.

NOT ME , :-D !

So, the release moment may be tricky.
If you delete a namespace: are you sure, nothing has a copy of your huge data?

Well, in the package, there are some usage of "upvar #0", and "trace",
so I am not 100% sure. But I used "info vars", and saw nothing
suspect in the global space.

I think, the catch version is slower in the error case. It saves the
call stack and the error message in global variables.

Ah ! I did not think of the error stack. I have logs, and I will
pay attention for this.

Thank you for your time,

Olivier.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl


Hi et99,

How are you measuring the current memory?

Only by visual check. In fact the number I used are more important,
and it saturates the memory, the one defined as "Memory" in the tasks
manager, the RAM ... without details. The accuracy is only 0.1 Go.

I use only pure Tcl (Tcllib) and "barebones" binaries from Magicsplat,
it should limit the usage of external dll.

That said, I only use Ashok's binaries, I can try with other distributions
to see if "namespace" has a concern in memory management ?

Thank for the link, I will not used it, but it gave me the idea to use
Ashok's Windows Inspection Tool Set, it seems to have a more detailed
analysis than Process Explorer.

Have a nice day,

Olivier.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Am 10.09.2025 um 10:27 schrieb Olivier:

That said, I only use Ashok's binaries, I can try with other distributions
to see if "namespace" has a concern in memory management ?

I don't think it is the distribution.
Ashoks binaries are build using MS-VS 2022. This should be ok.

Better craft an example and post it here.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl


Harald Oehlmann <wortkarg3@yahoo.com> posted:

Am 10.09.2025 um 10:27 schrieb Olivier:

That said, I only use Ashok's binaries, I can try with other distributions to see if "namespace" has a concern in memory management ?

I don't think it is the distribution.
Ashoks binaries are build using MS-VS 2022. This should be ok.

Better craft an example and post it here.

No need to take more of your time, I put the "set _BIG [ lrepeat 500000000 "dummy" ]"
in a namespace, delete it, and the memory was released.
It is definitively something inside the matrix package.

I wish you a nice day,

Olivier.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl


Harald Oehlmann <wortkarg3@yahoo.com> posted:

Am 10.09.2025 um 10:27 schrieb Olivier:

That said, I only use Ashok's binaries, I can try with other distributions to see if "namespace" has a concern in memory management ?

I don't think it is the distribution.
Ashoks binaries are build using MS-VS 2022. This should be ok.

Better craft an example and post it here.

Harald,

(Once again the same post, it seemed to have a problem this morning in Newsgrouper)

Your suggestion made me wrote a test with "set _BIG [ lrepeat 500000000 "dummy" ]"
inside a namespace. And by deleting the namespace, the memory was releaed ! It is
definitively a problem from "matrix" package that I have to solve.

I wish you a nice end of day,

Olivier.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Am 11.09.2025 um 14:54 schrieb Olivier:

Harald Oehlmann <wortkarg3@yahoo.com> posted:

Am 10.09.2025 um 10:27 schrieb Olivier:

That said, I only use Ashok's binaries, I can try with other distributions >>> to see if "namespace" has a concern in memory management ?

I don't think it is the distribution.
Ashoks binaries are build using MS-VS 2022. This should be ok.

Better craft an example and post it here.

Harald,

(Once again the same post, it seemed to have a problem this morning in Newsgrouper)

Your suggestion made me wrote a test with "set _BIG [ lrepeat 500000000 "dummy" ]"
inside a namespace. And by deleting the namespace, the memory was releaed ! It is
definitively a problem from "matrix" package that I have to solve.

I wish you a nice end of day,

Olivier.

Yes, TCL 9.0 has a lot of referencing instead of copying. This often
delays the memory release.

If you want to debug this, you may use: tcl::unsupported::representation
The returned refcount informs you, how often the item is referenced.I am always surprised, how quick the number gets high.
Stack traces, history, error logs may all keep references.

--- Synchronet 3.21a-Linux NewsLink 1.2