format | lint | pre-commit

✨ 🐍 ✨

# Tools ## 🧰 --- ## Formatters 🧼 ✨ [isort](https://pycqa.github.io/isort/): isort your imports, so you don't have to. [black](https://black.readthedocs.io/en/stable/): The Uncompromising Code Formatter. [flynt](https://github.com/ikamensh/flynt): String formatting converter. Note: Formatters will make modifications to your code in place. So, if you want to run them manually, I would suggest you to stage your changes first (`git add`) so that you can see the changes easily with `git diff`. Why are formatters important? - Keep the code reviews focused on the logic and structure of the code - And remove all the "nit" comments about styling issues - Consistency increases code readability Tools: - isort: Helps you keep your imports ordered and grouped properly: Standard library, third party, local/application. - black: Max line length, line breaks, quotes, spacing between function and class definitions, trailing commas, etc. - flynt: Transform as many string formatting as possible into f strings. --- ## Linters 👮‍♀️👮‍♂️ [bandit](https://bandit.readthedocs.io/en/latest/): Find common security issues. [flake8](https://flake8.pycqa.org/en/latest/index.html#): Your Tool For Style Guide Enforcement. [mypy](https://mypy.readthedocs.io/en/stable/index.html): Static type checker. Note: - bandit: 1. api key, password, token leaks. 2. usage of "unsafe" packages (random, pickle) or algorithms (md5) 3. bad file permissions 4. mark safe 5. etc. - flake8: 1. Unused/undefined variable 2. Unused/missing import 3. wildcard import 4. missing formatting argument 5. Function redefinition (copy/paste test function) 5. etc. - flake8 plugins: flake8-docstrings, flake8-fastapi, flake8-eradicate, etc. Search for `awesome flake8 extentions`. Linters help you identify problems before they become one and most of all can help with code completion and hints if you use an IDE or editor that supports it. --- ## Automation 👨‍🔧🤖 [pre-commit](https://pre-commit.com/): A framework for managing and maintaining multi-language pre-commit hooks. Note: All sorts of hooks are supported, not just pre-commit, but that's the only one I personally have a use for.

# Configuration ![ross shush](images/ross_shushing.jpg) Keep it to a minimum Note: If you can, try not to change any of the defaults, especially if you have multiple projects. They'll get out of sync quickly. And what makes it somewhat worse is that there is not yet a consensus on using pyproject.toml for configuration, only black and isort are using it at the moment. --- ## black ``` # pyproject.toml [tool.black] line-length = 120 ``` Note: Black is opinionated for a reason and I agree with most of their decisions so I would recommend to stick to the defaults, especially if you don't want to justify yourself, you can just blame it on Łukasz Langa and cut the conversation short. Here you can see that I changed the line length but it's based on my code style. --- ## isort ``` # pyproject.toml [tool.isort] profile = "black" line_length = 120 skip_gitignore = true ``` Note: - Skiping gitignore is critical if you place your virtualenv in your repo or have a node_modules folder lying around for example. - profile = "black" is important to make it compatible otherwise they'll fight each other for the formatting of the imports. - line_length is just there to keep things consistent across tools. --- ## flynt Nothing to see here 👀 ... Note: Flynt just magically works all the time. I haven't had a single issue with it after 2 years of use. --- ## bandit ``` # .bandit [bandit] targets: src,tests skips: B101 ``` [B101](https://bandit.readthedocs.io/en/latest/plugins/index.html#complete-test-plugin-listing): assert used [Suppressing individual lines](https://bandit.readthedocs.io/en/latest/config.html#suppressing-individual-lines): `# nosec: B101` Note: The configuration for bandit is a bit painful... But it might catch big mistakes like commiting a token, api key or password. So worth having IMO. - ⚠️ Don't mixup the "ini" file **AND/OR** the "yaml" config file. ⚠️ - Assert is removed with compiling to optimized byte code. - Bandit likes to complain about random, use the secrets module instead if you can or silent individual lines with `# nosec: B311` --- ## flake8 and plugins ``` # .flake8 [flake8] max_line_length = 120 extend-ignore = # Whitespace before ":" E203 # Missing docstring D1 docstring-convention = google ``` [Ignore errors](https://flake8.pycqa.org/en/3.1.1/user/ignoring-errors.html): `# noqa: E201` Note: Gives you great insights like unused imports, undefined variables, etc. - [E203](https://www.flake8rules.com/rules/E203.html): Incompatible with black for things like ranges. - [D1](http://www.pydocstyle.org/en/stable/error_codes.html): Used with flake8-docstrings. I don't care about missing docstrings as I try to keep them to a minimum. Instead I prefer to rely on explicit variable and function names, as well as type hints. Use `extend-ignore` instead of `ignore` to keep defaults. Doesn't support pyproject.toml because of legacy code around configuration that needs to be cleaned up first. --- ## Mypy ```ini # pyproject.toml [tool.mypy] mypy_path = ["src"] show_error_codes = true [[tool.mypy.overrides]] module = [ "package.*", "otherpackage.*", ] ignore_missing_imports = true ``` [Silencing errors](https://mypy.readthedocs.io/en/stable/error_codes.html#silencing-errors-based-on-error-codes): `# type: ignore[arg-type]` Note: - Mypy supports pyproject.toml only since the latest release couple weeks ago (0.901) - `mypy_path`: May not be required if you work on a library or package but you might need it for an application.

pre-commit


                        repos:
                          - repo: local
                            hooks:
                              - id: isort
                                name: isort
                                entry: isort
                                types: [python]
                                language: python
                              - id: flynt
                                name: flynt
                                entry: flynt
                                args: [--fail-on-change]
                                types: [python]
                                language: python
                              - id: black
                                name: black
                                entry: black
                                language: python
                                types: [python]
                                exclude: node_modules
                              - id: flake8
                                name: flake8
                                entry: flake8
                                language: python
                                types: [python]
                                exclude: .serverless,node_modules
                              - id: bandit
                                name: bandit
                                entry: bandit
                                language: python
                                types: [python]
                                args:  [--ini, .bandit, --recursive]
                              - id: mypy
                                name: mypy
                                entry: mypy
                                types: [python]
                                language: python
                          - repo: https://github.com/pre-commit/pre-commit-hooks
                            rev: v4.0.1
                            hooks:
                              - id: end-of-file-fixer
                                exclude: .*(min.js|min.css|html|svg|css.map|js.map)
                              - id: trailing-whitespace
                                exclude: .*(md)

## Dev requirements ``` # dev-requirements.in bandit==1.7.0 black==21.4b2 flake8-docstrings==1.6.0 flake8==3.9.1 flynt==0.64 isort==5.8.0 mypy==0.901 pre-commit==2.12.1 ``` Note: We use pip-tools to compile our requirements.txt files with pinned libraries for dependencies.

# Usage --- ## Setup your IDE/Editor ![pycharm](images/pycharm.png) ![vim](images/vim.png) ![vscode](images/vscode.png) ![sublime](images/sublime.png) ![emacs](images/emacs.png) ![atom](images/atom.png) Note: This is the first place where having those tools running in the background helps tremendously. Take the time to setup your IDE or editor, this is your most important environment. --- ## Install pre-commit hook ```bash # Optional if installed via dev-requirements.txt pip install pre-commit pre-commit install git add . git commit -m "Some bad code" ``` Note: - pre-commit runs only for the staged files, not the entire project. And only relevant linters/formatters are applied. - Also, because pre-commit will run everytime you commit, make sure you keep it fast otherwise nobody will want to use it.

![git commit](images/git_commit.png) --- ![git commit skip](images/git_commit_skip.png) --- ## Continuous Integration ```bash pre-commit run --all-files ``` Note: Devs can skip hooks with `git commit --no-verify`.

# Recap 1. Discuss it with your team and plan your rollout. 2. Run the formatters and linters. 3. Add pre-commit to your dev requirements and CI pipeline. 4. Do a brown bag with your team to get their Editor/IDE setup as well as pre-commit Note: Based on the size of your codebase you have different rollout strategies, but I would recommend doing one formatter at a time, then add the linters and fix remaining issues. You can ignore the code formatting commit for git blame by adding the rev to a .git-blame-ignore-revs file and do: `git config blame.ignoreRevsFile .git-blame-ignore-revs`.

## Additional Resources [Ignore bulk change with git blame](https://www.moxio.com/blog/43/ignoring-bulk-change-commits-with-git-blame) [Flake 8 plugins list](https://github.com/DmytroLitvinov/awesome-flake8-extensions) [Pre-commit ci](https://pre-commit.ci) [Test and code podcast](https://testandcode.com) (episodes 156 and 157) [Python bytes podcast](https://pythonbytes.fm) (episode 237) Note: Anthony Sottile Brian Okken Michael Kennedy

Q&A

linkedin.com/in/arthurio

notivize.com

github.com/arthurio/pre-commit