Reply to topic

problem with python code completion using anaconda

adamc
KDE Developer
Posts
46
Karma
0
Hi,
my problem is that code completion is not working for the pytorch module.
Code: Select all
import torch

torch.zero*Ctrl+Space*

gives only ZeroDivisionError, but not the torch.zeros() function, which is in the torch module. Strangely, torch.tensor works (torch.tensor is a class, while torch.zeros is a function afaik). but after creating a torch.tensor object, i can't complete on its methods. I can't complete on method parameters either. I believe that it has worked on my machine before, but i might be mistaken.

my environment:
I'm using kdevelop 5.4.2 from the kubuntu repositories, python 3.7 and anaconda. anaconda is installed into ~/bin/anaconda.
$ conda --version
conda 4.7.12
$ which python
/home/me/bin/anaconda3/bin/python
$ python --version
Python 3.7.4


what I have tried so far:
* code completion works on standard modules like math
* starting via PYTHONPATH=/home/me/bin/anaconda3/lib/python3.7/site-packages/ kdevelop
* setting the python interpreter path in the project configuration to /home/me/bin/anaconda3/bin/python
* in kdevelop settings / python documentation generate docs for the torch module

generating the documentation module looks successful, but i'm not sure it runs correctly as the return type is 'None' for most (or all) functions:
$cat ..kdevpythonsupport/documentation_files/torch.py
...
def zeros():
"""
zeros(*size, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) -> Tensor

Returns a tensor filled with the scalar value `0`, with the shape defined
... more docs ...
"""
return None
...


some suspicious shell outputs while running Kdevelop are:
kdevplatform.serialization: repository "/home/madam/.cache/kdevduchain/kdevelop-{fdc4bc3a-3044-436f-aba1-f80bddb27fab}" was write-locked, it probably is inconsistent
kdevplatform.serialization: "The data-repository at /home/madam/.cache/kdevduchain/kdevelop-{fdc4bc3a-3044-436f-aba1-f80bddb27fab} has to be cleared."
qrc:/qml/storage.js:24: TypeError: Cannot call method 'openDatabaseSync' of undefined
kdevelop.plugins.python.duchain: [ !!! ] did not get a typetrack container object when expecting one! Fix code / setup.
kdevelop.plugins.python.duchain: tuple type object is not available
kdevelop.plugins.python.parser: Discarding parts of the code to be parsed because of previous errors ## i fixed those later on ##
SyntaxError: ('invalid syntax', ('<kdev-editor-contents>', 13609, 21, 'def tpass#rch.ops():\n'))
---- Parsing FAILED ----
OverflowError: Python int too large to convert to C long
qrc:/qml/storage.js:24: TypeError: Cannot call method 'openDatabaseSync' of undefined
OverflowError: Python int too large to convert to C long
kdevplatform.language: invalid item for index 57 0 0
TypeError: can't convert complex to int
OverflowError: Python int too large to convert to C long
kdevplatform.language: Could not open file "/usr/share/kdevpythonsupport/documentation_files/PyQt4/__init__.py" (path "/usr/share/kdevpythonsupport/documentation_files/PyQt4/__init__.py" )

(but those didn't appear every time i started kdevelop)
flherne
Registered Member
Posts
20
Karma
0
I could reproduce this locally.

Because pytorch is mostly not written in Python, kdev-python can't parse it directly to find functions or types and relies on the stubfile produced by the 'Generate Documentation' tool.

There appears to be a bug in the documentation generator, which causes a syntax error in the generated Python file.

If you edit `~/.local/share/kdevpythonsupport/documentation_files/torch.py`, and edit line 13609 to replace `def torch.ops()` with just `def ops()`it should work.

torch.tensor works anyway because that's one of the few parts of Torch that actually *does* exist in Python code - there's a `site-packages/torch/tensor.py` file, and kdev-python parses it fine.

EDIT: The return types of the generated stubs are indeed `None`, which is an expected limitation of the generator.
At some point, kdev-python might be able to use the `.pyi` stubfile shipped with pytorch instead of trying to generate one itself, which would solve both of these issues.
adamc
KDE Developer
Posts
46
Karma
0
Thanks a lot!

flherne posted a workaround in the irc chat, namely copying the .pyi files into documentation_files/torch.py
this gives not perfect, but okayish completion. Certainly much better than before :)

not sure this is overkill or not, but i wrote a short script to do that for all torch modules:
Code: Select all
cd $CONDA_PREFIX/lib/python3.7/site-packages
dest="$HOME/.local/share/kdevpythonsupport/documentation_files/"
for i in $(find ./torch -iname "*__init__.pyi"); do
    mkdir -p ${dest}$(dirname $i) &&
    cp $(pwd)/$i $dest${i::-1};
done;

for i in $(find ./torch -iname "*.py"); do
    mkdir -p ${dest}$(dirname $i) &&
    ln -s $(pwd)/$i $dest${i};
done;

use at your own risk!


not perfect: torch.tensor doesn't auto complete for instance
Code: Select all
import torch
M = torch.tensor([[0]])
# M. doesn't show anything useful

 
Reply to topic

Bookmarks



Who is online

Registered users: Baidu [Spider], Bing [Bot], boudewijn, bovender, claydoh, colstar, Google [Bot], grinder, Majestic-12 [Bot], mcoudert, quisimodo, sandro4912, TheraHedwig, Voronwe13