pylint

rs could actually read. After installing pylint. On a package level for instance. Say package name is foo and follows standard hierarchy with `foo/foo`: cd foo pyreverse -o png . # generates classes.png and packages.png # You can also see pyreverse -o png foo - Profiling in the context of Flask: Using Werkzeug's ProfilerMiddleware[^5] helps you see what's going on with each request. What functions are called, number of calls, total time, per call, which line, etc. If the example in the documentation does not work, try the following: ... try: from werkzeug.middleware.profiler import ProfilerMiddleware except ModuleNotFoundError: # Older version from werkzeug.contrib.profiler import ProfilerMiddleware ... # Assuming you have an app object app.config['PROFILE'] = True app.wsgi_app = ProfilerMiddleware(app.wsgi_app, restrictions=[50]) General things: it is very helpful to extract as much code from the routes. This helps making the functions usable elsewhere, and not rely on Flask's testing client which can be pretty frustrating when dealing with the app context, especially in test suites involving database actions, and weird connections in setUp and tearDown if you're using unittest*. As I said, this is general and not very specific for "big data" or "billions of rows", but these small things lead to bigger things in my opinion: making the code easier to read and extend, easier to test and cover, easier to profile and improve, compounds to a point you may postpone more involved approaches. [^1]: https://github.com/sumerc/yappi [^2]: https://kcachegrind.github.io/ [^3]: https://www.logilab.org/blogentry/6883 [^4]: https://github.com/PyCQA/pylint [^5]: https://werkzeug.palletsprojects.com/en/1.0.x/middleware/pro...

es/48 [2]: https://github.com/psf/black/issues/1178 [3]: https://github.com/psf/black/issues/1336 [4]: https://github.com/PyCQA/pylint/issues/289 [5]: https://discuss.python.org/t/pep-8-clarify-if-multiline-argu...

s. I think this is one of the main reasons for Python's success. It can be used as a high level interface to a huge low level ecosystem. ML wouldn't exist in Python without that. Python is a great language to create simple scripts as well. I use it as a bash replacement. But beware: Python can get really ugly if you rely too heavily on OOP/Inheritance. There is a lot of ugly Python code out there. I think Python shines the most if you use it in a procedural/semi-functional style and describe data-types for your business logic (almost) entirely with (dump) dataclasses [1] and namedtuples [2]. Beware2: Python's packaging story (package manager, etc) is currently in a messy state. There are good solutions, but it's difficult to find the right tools for a newcomer. I think it will take a couple of years for the dust to clear, but you might want to check out pipenv [7] or poetry [8] These packages will make your life a lot easier. I use them for every project: - black [3] or yapf [4] (auto-formatter) - mypy [0] (statically typing with editor support for VSCode) - pylint [5] or flake8 [6] (linter) [0] https://github.com/python/mypy [1] https://docs.python.org/3/library/dataclasses.html#module-da... [2] https://docs.python.org/3/library/typing.html#typing.NamedTu... [3] https://github.com/psf/black [4] https://github.com/google/yapf [5] https://github.com/PyCQA/pylint/ [6] https://github.com/PyCQA/flake8 [7] https://github.com/pypa/pipenv [8] https://github.com/sdispater/poetry