Monday, February 18, 2008

Python considered harmful

I used to think that using java was like being in league with memory suppliers.

However...

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
10946 fred 1 60 0 3692M 270M sleep 25:34 0.00% python
19559 joe 1 59 0 1950M 836M sleep 17:21 0.00% python
20738 bill 1 59 0 1787M 1608M sleep 3:28 0.00% python

That's on a machine with 4G of physical memory. And given that python is being used more widely, and that data volumes are increasing, I need to do a couple of things. First, order more memory; and second, work out how to build a 64-bit copy of python with all the modules we use.

4 comments:

Unknown said...

Did you run into this? http://evanjones.ca/python-memory-part3.html

Anonymous said...

We had a terrible time building a 64-bit version of Python 2.5 on Solaris 10 (SPARC). If you go down that path then please post info on your progress!

Anonymous said...

One way would be to use debian amd64, and just to apt-get install python. Not rocket science, when you choose decent tools.

On memory... a) it's hard to measure b) who's to say it's not using that memory very efficiently for some huge task, or using it as a cache?

The only real problem I can see here is the one that always bugged me: scripts show up in task lists as the interpreter, not the script itself. That's more of a unix design flaw than a python issue though, and it affects many languages besides python.

Peter Tribble said...

I wouldn't blame python - I suspect it's the script that's not freeing anything.

I was just going to base the build on the spec file that the JDS build uses. It's not going to work straight off, because Sun only ship 2.4.x and I need at least 2.5, and the complicated bit (I imagine) is going to be integrating with all the modules that are used.