Trace python program from log perspective

刚回答了用 -vvv debug pip：


Just a reminder to whom google this error and come here.

Let's say I get this error:

$ python3 example.py
Traceback (most recent call last):
  File "example.py", line 7, in <module>
    import aalib
ModuleNotFoundError: No module named 'aalib'

Since it mentions aalib, I was thought to try aalib:

$ python3.8 -m pip install aalib
ERROR: Could not find a version that satisfies the requirement aalib (from versions: none)
ERROR: No matching distribution found for aalib

But it actually wrong package name, ensure pip search(service disabled at the time of writing), or google, or search on pypi site to get the accurate package name:

Then install successfully:

$ python3.8 -m pip install python-aalib
Collecting python-aalib
  Downloading python-aalib-0.3.2.tar.gz (14 kB)
...

---

As pip --help stated:

$ python3.8 -m pip --help
...
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.

To have a systematic way to figure out the root causes instead of rely on luck, you can append -vvv option of pip command to see details, e.g.:

$ python3.8 -u -m pip install aalib -vvv
User install by explicit request
Created temporary directory: /tmp/pip-ephem-wheel-cache-b3ghm9eb
Created temporary directory: /tmp/pip-req-tracker-ygwnj94r
Initialized build tracking at /tmp/pip-req-tracker-ygwnj94r
Created build tracker: /tmp/pip-req-tracker-ygwnj94r
Entered build tracker: /tmp/pip-req-tracker-ygwnj94r
Created temporary directory: /tmp/pip-install-jfurrdbb
1 location(s) to search for versions of aalib:
* https://pypi.org/simple/aalib/
Fetching project page and analyzing links: https://pypi.org/simple/aalib/
Getting page https://pypi.org/simple/aalib/
Found index url https://pypi.org/simple
Getting credentials from keyring for https://pypi.org/simple
Getting credentials from keyring for pypi.org
Looking up "https://pypi.org/simple/aalib/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/aalib/ HTTP/1.1" 404 13
[hole] Status code 404 not in (200, 203, 300, 301)
Could not fetch URL https://pypi.org/simple/aalib/: 404 Client Error: Not Found for url: https://pypi.org/simple/aalib/ - skipping
Given no hashes to check 0 links for project 'aalib': discarding no candidates
ERROR: Could not find a version that satisfies the requirement aalib (from versions: none)
Cleaning up...
Removed build tracker: '/tmp/pip-req-tracker-ygwnj94r'
ERROR: No matching distribution found for aalib
Exception information:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 357, in run
    resolver.resolve(requirement_set)
  File "/usr/lib/python3/dist-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/lib/python3/dist-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/lib/python3/dist-packages/pip/_internal/legacy_resolve.py", line 281, in _get_abstract_dist_for
    req.populate_link(self.finder, upgrade_allowed, require_hashes)
  File "/usr/lib/python3/dist-packages/pip/_internal/req/req_install.py", line 249, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python3/dist-packages/pip/_internal/index/package_finder.py", line 926, in find_requirement
    raise DistributionNotFound(
pip._internal.exceptions.DistributionNotFound: No matching distribution found for aalib

From above log, there is pretty obvious the URL https://pypi.org/simple/aalib/ 404 not found. Then you can guess the possible reasons which cause that 404, i.e. wrong package name. Another thing is I can modify relevant python files of pip modules to further debug with above log. To edit .whl file, you can use wheel command to unpack and pack.

由于答案篇幅有点长且可以多种 debug 方式，我就没说更进一阶的方式。

由于 pip 使用 logging 来 print 的，更改 logging module 的输出函数可以更直接知道这个 log 是谁 print 的， 从而轻松修改相应的 python 脚本。

  

即:

    print(''.join(['\n\x1b[7;36m[holeL] name: ', repr(name), ' lvl: ', repr(level), ' fn: \x1b[0m\x1b[K\x1b[17;36m', repr(fn), '\x1b[0m\x1b[K\x1b[7;36m lno: ', repr(lno), ' msg: ', repr(msg), ' args: ', repr(args), ' exc_info: ', repr(exc_info), ' func: ', repr(func), ' sinfo: ', repr(sinfo), '\x1b[0m\x1b[K' ]))

比起 trace 模块方便输出大量的 log 要去想排除哪一个又怕排除错，轻松得多：

之后想到修改 trace 模块以包含 print 和 write,  之后想到有 stack 才能清楚 log 与程序的上下关系。

不过为了想到避免重复的 stacks， 尝试了各种复杂的代码， 之后想到 Prev 重复的 Prev 重复很难 refer。

最后想到用 [NEW] 和 [EQU] 不同 tags 标记新部分和重复部分，一目了然且不会看起来重复冗长。

 `NEW` 和 `EQU` 同样三个字的宽度才容易区分上下同一个文件。




... 省略



字体颜色也并非随着个人喜好而随便选的。

绿表示新/diff 的 +(显示时间都是新的，所以也是绿)，和平深蓝表示打平/相等/重复。

白底黑字代码比较一目了然阅读(原先的水绿背景虽然美但太难阅读了)，且与一般黑底白字的 log 相反，容易区分。

至于文件名字和行数都是同一个水绿色（不会太明显而扰乱代码的阅读，也不会太深而很难看见）。

行数连着主打背景白色的源代码， 所以水绿用在背景色。

不过不同终端的颜色可能有点不同。gnome-terminal 的标记和时间会闪， konsole 不会， 之前在 konsole 做的时候没发现不同。

由于冗长的代码需要长时间的阅读，我倾向于不闪， 不过就给用户两种选择吧。




源代码: https://github.com/limkokhole/pylogtrace