This forum has been archived. All content is frozen. Please use KDE Discuss instead.

.csv import

Tags: None
(comma "," separated)
Conker
Registered Member
Posts
7
Karma
0

.csv import

Thu Apr 20, 2023 3:40 pm
A little background... I looked at kexi a few years ago but my linux OS only offered an old version so I went with libreoffice base.
My current OS (Kubuntu 22.04) offers kexi 3.2 so I am trying again (libreoffice is very demanding).
I have lots of data in base and my approach is to export a data table into Calc and then save as .csv ready for import into kexi.
A few questions if anyone can help.
1. I have empty cells in my table which messes up the .csv import. In Calc I try a Find/Replace - leave Find empty and NULL in Replace - but it just brings over the text 'NULL'.
Is there a solution?
2. My base tables have some columns formatted as Currency - like $22.50 - in kexi it shows as 22.5 Any recommendation (without formatting the price columns in base as text and attendant retyping of 1000s of records ?
3. Is there a limit on the size of a .csv file that can be imported into kexi?
User avatar
jstaniek
Moderator
Posts
1027
Karma
2
OS

Re: .csv import

Thu Apr 20, 2023 4:51 pm
Conker, thanks a lot for trying KEXI!!!

Question 1. In general, while KEXI may lack in some areas (like lack of subforms), it's very well possible to have it as the top quality CSV import tool.
Your case will be addressed. So I encourage you to send some report or test data, more info at https://community.kde.org/Kexi/File_a_bug_or_wish.
If you have a confidential file you can contact me and send me data privately, and it won't be published anywhere.
Or, to make private data, you can just reduce it to much smaller size but still demonstrating the issue we're talking about. It's often possible.

Question 2. Again please send (https://community.kde.org/Kexi/File_a_bug_or_wish) a small test CSV file and explain what result that you expect. Please report one aspect in each wish or bug.
While we miss the "UPDATE" queries, which would be ideal for processing of data formats on import, it's possible that we come up with alternative solution. For example pre-processing a file using a simple external Python script prior to importing the files into KEXI. In some distant future in-KEXI Python scripting can make it even more convenient.

Question 3: there's no limit of this kind, but there are following limits just for the needs of the format detector and previewing (information from the kexicsvimportdialog.cpp file):
MAX_ROWS_TO_PREVIEW = 100 -- max rows to preview on import (this is only preview, finally all rows are imported)
MAX_BYTES_TO_PREVIEW = 10240 -- like above, 10KB or characters are analyzed, it should be sufficient except for extreme cases when you have e.g. >10KB text in a single cell
MAX_CHARS_TO_SCAN_WHILE_DETECTING_DELIMITER = 4096 -- self-explanatory


Best regards,
Jarosław Staniek
• Qt Certified Specialist
KEXI - Open Source Visual DB Apps Builder
• Request a feature or fix for KEXI here
May I help you? Please mention your app's version and OS when asking for help
Conker
Registered Member
Posts
7
Karma
0

Re: .csv import

Thu Apr 20, 2023 6:08 pm
Thanks for prompt response.
I have prepared a test file, 36.6KB in libreoffice 7.4 Calc with .ods file extension.
After conversion to .csv and import into KEXI my objective is to have the blank cells blank, and to have the value data show as e.g. $12.34 -with dollar sign and two decimal places.
I prefer to send to you privately but how to do this?
Regards.
User avatar
jstaniek
Moderator
Posts
1027
Karma
2
OS

Re: .csv import

Thu Apr 20, 2023 6:43 pm
I've contacted you via the PM


Best regards,
Jarosław Staniek
• Qt Certified Specialist
KEXI - Open Source Visual DB Apps Builder
• Request a feature or fix for KEXI here
May I help you? Please mention your app's version and OS when asking for help
User avatar
jstaniek
Moderator
Posts
1027
Karma
2
OS

Re: .csv import

Fri Apr 21, 2023 10:45 pm
Update.
I've sent you the CSV exported in LibreOffice Calc with default settings using File->Save as, and a kexi file with the imported CSV.
I see no traces of NULL text, the data looks OK.

Currency indeed imports as floating point, that is because KEXI does not support DECIMAL or CURRENCY type. SQLite introduced it much later, and we have not caught up...
Regarding the display of the $, it can be solved by using one of these features, both not implemented (sorry):
1. better: a format property prepending the $
2. prepending the $ using a SQL query (printf() function).

Or even better, using the currency type - not implemented unfortunately. When it is, the CSV importer would also be able to detect currencies.


Best regards,
Jarosław Staniek
• Qt Certified Specialist
KEXI - Open Source Visual DB Apps Builder
• Request a feature or fix for KEXI here
May I help you? Please mention your app's version and OS when asking for help
Conker
Registered Member
Posts
7
Karma
0

Re: .csv import

Sat Apr 22, 2023 1:00 pm
Thanks for this. After starting again I was able to duplicate what you did with the default .csv import. Still shows 3 issues;
1. Where there are blank number fields the default import shows '0' in KEXI table
2. Cannot bring in currency symbol and trailing zero; as you mention it is a presently missing feature...
3. Does not bring in 1000s comma separator - easy to fix by using ";" as file separator

Next I set up a defined database and designated the currency columns as Text. This successfully brings in the data but with implications for future data entry as currency symbol must be typed with each addition.

Next I successfully imported 147,011 records. This is good, especially when I can go from first to last almost instantly (after first slower attempt to get to last record - I assume indexing is taking place). Also very good is the tabbing GUI which allows instant switching between table and query.
I have made a couple of queries with SQL - like the SQL checker (now needs some useful feedback when it fails as a future feature!).

Now to report some not so good points
1. Many random freezes and finally shutdown. Sorry, bug reporting looks to be beyond my present understanding.
2. Ascending/Descending seems to cause some of the freezes? My expectation is that you select a column and then select Ascending or Descending order but cannot select column. Perhaps as new user I do not yet understand correct use. You can perhaps make suggestions?
3. Tried to use Compact Database but get "Unknown Error' message. Does this mean it has not performed compact?
Have made first Form but again I need more time to understand everything.
4. When editing data, only Form allows fixing cursor anywhere in the text. In Table and Query you must reenter complete string. Why this restriction?

Enough for now.
User avatar
jstaniek
Moderator
Posts
1027
Karma
2
OS

Re: .csv import

Sat Apr 22, 2023 2:56 pm
Thanks for so great explanation. I trust it helps us to further improve KEXI!

One initial remark, because there are about 150,000 rows, we may lack the optimization and therefore: too much memory is used because all data is loaded to the main memory, processing takes too much so your operating system reports the app is "not responsive" which is true but definitely it's not a crash caused by invalid implementation. However it is a weak explanation from the user's point of view, I admit.

We just do not check the memory limits at all. It's also a TO-DO for us (to defer loading all the data) but it's perfectly possible. It is for now so much more convenient to load entire table (or query result) beforehand. The use case of handling huge data sets should be a typical for KEXI.

Noted down here: https://invent.kde.org/office/kexi/-/issues/139

Regards!


Best regards,
Jarosław Staniek
• Qt Certified Specialist
KEXI - Open Source Visual DB Apps Builder
• Request a feature or fix for KEXI here
May I help you? Please mention your app's version and OS when asking for help
Conker
Registered Member
Posts
7
Karma
0

Re: .csv import

Mon Apr 24, 2023 12:57 am
OK but when I review my original KEXI test
viewtopic.php?f=221&t=129257
it is mentioned that users exist with one million records (880000 actual limit) and also currency possibilities so we have unusual situation of newer software version having reduced specification?
I am happy to anytime test anything you like to suggest for such large data set situations and of course will continue to evaluate the software overall.


Bookmarks



Who is online

Registered users: Bing [Bot], Google [Bot], Sogou [Bot]