Python Profiling for Babies

Posted on Fri 05 October 2018 in technology

If you have ever written a big stack of Python code and wondered why it is stupidly slow, you need a profiler. My go-to choice is pyinstrument, which samples your program while it is running and prints out pretty trees of where your program is spending time.

Installing it is pretty easy:

pip-3.6 install pyinstrument --user

And then you pop into your python code and have it sample the code you want to profile:

def big_terrible_piece_of_code()

    from pyinstrument import Profiler
    profiler = Profiler()

    # Do a whole of stuff

    print(profiler.output_text(unicode=True, color=True))

That'll output a big old graph like this:

2.449 None  None
└─ 2.431 _handle_and_close_when_done  gevent/
   └─ 2.431 handle  gevent/
      └─ 2.431 handle  gevent/
         └─ 2.431 handle_one_request  gevent/
            └─ 2.431 handle_one_response  gevent/
               └─ 2.431 run_application  gevent/
                  └─ 2.431 __call__  flask/
                     └─ 2.431 wsgi_app  flask/
                        └─ 2.431 full_dispatch_request  flask/
                           └─ 2.431 dispatch_request  flask/
                              └─ 2.431 wrapfn
                                 └─ 2.431 view_home_thread
                                    └─ 1.452 <listcomp>
                                       └─ 1.426 __next__  openarc/
                                          └─ 1.397 next  openarc/
                                             └─ 1.209 _set_attrs_from_cframe  openarc/
                                                └─ 0.818 add  openarc/

The most expensive code paths are near the top of the tree -- optimize those first if you can.

There are other tools in my optimization toolkit, but this is the one I find myself turning to most often. It is insanely easy to use, and in my book that trumps sophisticated tools any day of the week: sometimes a general, birdseye view is all you need to notice that your program is doing something stupid.

Previously: performance is not an accident