- Ideas and Plans for Memory Profiling with NYTProf
- Email threads
- Possibly Relevant Perl Modules
- Other Items of Interest
Ideas and Plans for Memory Profiling with NYTProf
XXX This mostly a dumping ground for thoughts, and links at the moment. Somewhere to gather the info so it'll be easier to find when someone wants to work on it.
It's also focussed on runtime profiling of total memory usage over time (rather than a detailed analysis of who 'owns' what memory as some particular point in time, like the end of the program).
NYTProf currently only measures time and supports a limited range of "clocks" (realtime vs cputime). For profiling memory we need to add a new kind of "clock" that measures memory usage. Since we're generalizing the concept of what gets measured (and how we get the info from the system) a better name than "clock" would be "probe".
"Probe" means some measuring mechanism like get_clock(), times(), getrusage(), that may yield multiple pieces of information with a single call. "Measure" is one specific item generated by a probe. Probe "time" uses times(), measures: "time.user", "time.user+sys" etc Probe "clock" uses clock_gettime(), measures: "clock.realtime", "clock.monotonic" etc Probe "rusage" uses getrusage(), measures: "rusage.majflt", "rusage.nvcsw" etc Probe "memory" measures: "memory.bytes", "mem.allocs" etc Probe "arena" measures: "arena.svs", "arena.bytes" etc
Generalize the concepts of probes. Have a structure defining a 'probe' with pointers to functions to get the values, subtract values to get relative ticks, return the tick units etc. Give them names and attributes (cpu, realtime etc). User could then pick a probe by name. By default we'd pick the best available realtime probe. Use the subtraction logic where we currently handle times in the statement and subroutine profilers.
"Memory profiling in Devel::NYTProf?: - June 2009 http://groups.google.com/group/develnytprof-dev/browse_frm/thread/1df4cba3001cd4e4/136812b44e9f7631 Talking about the problems of measuring memory usage of the whole process re: http://blog.robin.smidsrod.no/index.php/2009/05/26/memory-footprint-of-popular-cpan-modules
"Memory profiling possibilities in NYTProf" - September 2009 http://groups.google.com/group/develnytprof-dev/browse_frm/thread/c711c132216a3cea/035012e3dc2971ec This includes a detailed overview of the issues.
"profiling memory" - Dec 2009 http://groups.google.com/group/develnytprof-dev/browse_frm/thread/5ffd24200866b0c1/201b58c18d826aaa Nicholas Clark offers an experimental patch that intercepts malloc and free and makes NYTProf measure memory usage.
Possibly Relevant Perl Modules
Per-process memory information:
Arena, stash, and pad based memory reporters:
http://search.cpan.org/perldoc?Devel::Gladiator http://search.cpan.org/dist/Internals-DumpArenas/ http://search.cpan.org/dist/Internals-GraphArenas/ http://search.cpan.org/perldoc?Devel::DumpSizes http://search.cpan.org/perldoc?Devel::Arena
Per-object memory size reporters:
Other Items of Interest
"Memory Efficient Perl" slides by jjore
Other profile/memory visualization tools