yappi

rs could actually read. After installing pylint. On a package level for instance. Say package name is foo and follows standard hierarchy with `foo/foo`: cd foo pyreverse -o png . # generates classes.png and packages.png # You can also see pyreverse -o png foo - Profiling in the context of Flask: Using Werkzeug's ProfilerMiddleware[^5] helps you see what's going on with each request. What functions are called, number of calls, total time, per call, which line, etc. If the example in the documentation does not work, try the following: ... try: from werkzeug.middleware.profiler import ProfilerMiddleware except ModuleNotFoundError: # Older version from werkzeug.contrib.profiler import ProfilerMiddleware ... # Assuming you have an app object app.config['PROFILE'] = True app.wsgi_app = ProfilerMiddleware(app.wsgi_app, restrictions=[50]) General things: it is very helpful to extract as much code from the routes. This helps making the functions usable elsewhere, and not rely on Flask's testing client which can be pretty frustrating when dealing with the app context, especially in test suites involving database actions, and weird connections in setUp and tearDown if you're using unittest*. As I said, this is general and not very specific for "big data" or "billions of rows", but these small things lead to bigger things in my opinion: making the code easier to read and extend, easier to test and cover, easier to profile and improve, compounds to a point you may postpone more involved approaches. [^1]: https://github.com/sumerc/yappi [^2]: https://kcachegrind.github.io/ [^3]: https://www.logilab.org/blogentry/6883 [^4]: https://github.com/PyCQA/pylint [^5]: https://werkzeug.palletsprojects.com/en/1.0.x/middleware/pro...