I've never used Skrooge before, but am a longterm KMM user, originally in Windows, then in Linux since 2014. It's possible my unfamiliarity with Skrooge is the cause of these failures [ie, maybe i am doing something wrong?], but if not then this seems to be a potential bug with Skrooge's Import function.

I have made multiple attempts to import my 830 kB KMM `.kmy` file into Skrooge:
1. Linux Manjaro KDE Stable-branch; Skrooge 2.16.2
2. ArchLabs OpenBox; Skrooge 2.17.0
3. Linux Manjaro KDE Unstable-branch; Skrooge 2.17.0
4. Linux Mint 19.1 Cinnamon; Skrooge 2.11.0

There's a 100% failure rate. The progress bar halts at ~56% for maybe 5' - 10' each time, before eventually failing with error messages like these:

Code: Select all

[ERR-5]: Import of file named 'file:///home/steffie/Documents/Steffie_2003+.kmy' failed
[ERR-10001]: unrecognized token: "'" Unable to execute statement
[ERR-10001]: INSERT INTO suboperation ('d_date','rd_operation_id','t_comment','r_category_id','f_value') VALUES ('2017-05-04','8702','
[ERR-10001]: INSERT INTO suboperation ('t_comment','d_date','f_value','rd_operation_id','r_category_id') VALUES ('
[ERR-10001]: INSERT INTO suboperation ('d_date','t_comment','rd_operation_id','f_value','r_category_id') VALUES ('2017-05-04','

As best i can understand this error message it seems to be saying that Skrooge fell into a heap when it encountered an apostrophe mark in my KMM database. I've now searched my KMM file for such occurrences & found 3171 transactions which include an apostrophe. This is a ridiculous failure of Skrooge if it cannot handle basic English punctuation symbols -- a bug?

By the way, an apparent subsidiary bug is that no version of Skrooge i tried correctly copied the full error text contents to clipboard; the only way i can actually show you is via a screenshot picture:



I then experimented with the [impractical, deeply disadvantageous] CSV option. I picked my smallest KMM account [comprising the least transactions (my file includes many accounts; several Open, & several Closed), each holding considerably more transactions than the account i tried here], exported it as CSV, then tried importing it into Skrooge. Again it failed:

Code: Select all

[ERR-5]: Import of file named 'file:///home/steffie/Documents/kmm cash acct export.csv' failed
[ERR-5]: Columns date and amount not found. Set import parameters in settings (Configure Skrooge... / Import/Export / CSV / Edit regular expressions...).

Do i need to give up on Skrooge [before i've even been able to test & evaluate it], or is a solution possible please?

steffie wrote:I've never used Skrooge before, but am a longterm KMM user, originally in Windows, then in Linux since 2014. It's possible my unfamiliarity with Skrooge is the cause of these failures [ie, maybe i am doing something wrong?], but if not then this seems to be a potential bug with Skrooge's Import function.

I have made multiple attempts to import my 830 kB KMM `.kmy` file into Skrooge:
1. Linux Manjaro KDE Stable-branch; Skrooge 2.16.2
2. ArchLabs OpenBox; Skrooge 2.17.0
3. Linux Manjaro KDE Unstable-branch; Skrooge 2.17.0
4. Linux Mint 19.1 Cinnamon; Skrooge 2.11.0

There's a 100% failure rate. The progress bar halts at ~56% for maybe 5' - 10' each time, before eventually failing with error messages like these:

Code: Select all

[ERR-5]: Import of file named 'file:///home/steffie/Documents/Steffie_2003+.kmy' failed
[ERR-10001]: unrecognized token: "'" Unable to execute statement
[ERR-10001]: INSERT INTO suboperation ('d_date','rd_operation_id','t_comment','r_category_id','f_value') VALUES ('2017-05-04','8702','
[ERR-10001]: INSERT INTO suboperation ('t_comment','d_date','f_value','rd_operation_id','r_category_id') VALUES ('
[ERR-10001]: INSERT INTO suboperation ('d_date','t_comment','rd_operation_id','f_value','r_category_id') VALUES ('2017-05-04','

As best i can understand this error message it seems to be saying that Skrooge fell into a heap when it encountered an apostrophe mark in my KMM database. I've now searched my KMM file for such occurrences & found 3171 transactions which include an apostrophe. This is a ridiculous failure of Skrooge if it cannot handle basic English punctuation symbols -- a bug?

Hi,
This is clearly a bug but not so ridiculous.
This is not due to apostrophe. I have many test case using apostrophes.
I just did a test with return line but I didn't reproduced the error.
There is something strange in your data but I don't know what.

steffie wrote:By the way, an apparent subsidiary bug is that no version of Skrooge i tried correctly copied the full error text contents to clipboard; the only way i can actually show you is via a screenshot picture:

As you can see, the sql order displayed seems to be correct.
I tested it without error.
Again, there is something strange in your data but I don't know what.

Could you provide me a sample file to reproduce the error?
This can be a kmy file with only one operation (2017-05-04).
My email is in the "About" of Skrooge.

steffie wrote:I then experimented with the [impractical, deeply disadvantageous] CSV option. I picked my smallest KMM account [comprising the least transactions (my file includes many accounts; several Open, & several Closed), each holding considerably more transactions than the account i tried here], exported it as CSV, then tried importing it into Skrooge. Again it failed:

Code: Select all

[ERR-5]: Import of file named 'file:///home/steffie/Documents/kmm cash acct export.csv' failed
[ERR-5]: Columns date and amount not found. Set import parameters in settings (Configure Skrooge... / Import/Export / CSV / Edit regular expressions...).

In settings, you have to map the columns of your CSV file with the operations' properties.

steffie wrote:Do i need to give up on Skrooge [before i've even been able to test & evaluate it], or is a solution possible please?

Thank you for giving a chance to Skrooge.

Hello

Thanks for your fast reply. I've spent today working on this, but unfortunately my progress/success has been poor.

1. I cannot find a simple way to Export a single transaction from KMM in a `.kmy` file format.

2. I made a copy of my "real" `.kmy` file, then from within KMM i deleted that target/suspicious transaction, then tried to Import this modified `.kmy` file into Skrooge. It still failed, but this time with a large number of new error messages targeting other transactions.


3. I abandoned that copy file, returned to my "real" `.kmy` file, then "Saved As" the Anonymised KMM file format [ie, replacing all my personal data with randomised values & text, but preserving the structures of accounts, payees, categories, transactions etc]. Hoping that Skrooge would also fail again with this new file upon Import, i was disappointed to find that instead Skrooge was successful with this Import... hence there is no analytical value to be gained by sending you this file.

4. In desperation i created a new KMM file, in which i manually created that suspicious 4/5/2017 transaction [copying & pasting all text entries from my "real" file's transaction]. Then i attempted to Import this extremely small new `.kmy` file into Skrooge, & once more was disappointed to find that Skrooge was successful with this Import... hence there is no analytical value to be gained by sending you this file either.

I feel that my many hours troubleshooting this have been pointless, as there is no viable data i can send you for your own analysis -- i mean, obviously i am not going to send you my "real" `.kmy` file ... & neither my Anonymised file, nor my small manual file, could be diagnostically helpful given they did Import correctly to Skrooge.

Therefore unless you have any new ideas on viable options, i suspect that my short-lived evaluation of Skrooge as a potential replacement of KMM for me is now exhausted. Pity.

steffie wrote:Hello

Thanks for your fast reply. I've spent today working on this, but unfortunately my progress/success has been poor.

1. I cannot find a simple way to Export a single transaction from KMM in a `.kmy` file format.

2. I made a copy of my "real" `.kmy` file, then from within KMM i deleted that target/suspicious transaction, then tried to Import this modified `.kmy` file into Skrooge. It still failed, but this time with a large number of new error messages targeting other transactions.

I think this is the same kind of error. Something not expected in "Comment" ("Memo" in KMM).
Could you send me this file? (You can edit the XML file with a text editor and replace personal data by XXXX to be sure that no sensible data are transferred).

steffie wrote:3. I abandoned that copy file, returned to my "real" `.kmy` file, then "Saved As" the Anonymised KMM file format [ie, replacing all my personal data with randomised values & text, but preserving the structures of accounts, payees, categories, transactions etc]. Hoping that Skrooge would also fail again with this new file upon Import, i was disappointed to find that instead Skrooge was successful with this Import... hence there is no analytical value to be gained by sending you this file.

Normal, I think the error is due to the value of "Memo".

steffie wrote:4. In desperation i created a new KMM file, in which i manually created that suspicious 4/5/2017 transaction [copying & pasting all text entries from my "real" file's transaction]. Then i attempted to Import this extremely small new `.kmy` file into Skrooge, & once more was disappointed to find that Skrooge was successful with this Import... hence there is no analytical value to be gained by sending you this file either.

The copy/paste can replace strange characters.

steffie wrote:I feel that my many hours troubleshooting this have been pointless, as there is no viable data i can send you for your own analysis -- i mean, obviously i am not going to send you my "real" `.kmy` file ... & neither my Anonymised file, nor my small manual file, could be diagnostically helpful given they did Import correctly to Skrooge.

Therefore unless you have any new ideas on viable options, i suspect that my short-lived evaluation of Skrooge as a potential replacement of KMM for me is now exhausted. Pity.

Hi again & thanks once more.

Today after more hours of investigation & testing i made a breakthrough -- i have succeeded in importing my `.kmy` file into Skrooge. It was your recent information that gave me the clue about some bad characters/symbols in the Memo fields, PLUS you mentioned looking in the XML file version of the KMM file/database. When i thus saved my KMM file in `.xml` format, closed KMM, & opened that `.xml` file in text editor Kate, it warned me...

Steffie_2003+.xml was opened with UTF-8 encoding but contained invalid characters.

...but it did not show me what the bad characters actually were. Conversely when i opened the file in Gedit, it also warned me but additionally i saw multiple groups of three characters highlighted in red [which i hoped implied these were the bad characters]. They were:

Code: Select all

\00

...and there were 28 instances of them [all only in various Memo fields]. I manually searched for each occurrence & deleted them, saved the file, closed it, loaded it into KMM, saved it as a new `.kmy` file, closed KMM, launched Skrooge, repeated the Import... & finally it succeeded. Phew.

Thanks for your help.

So now that i can see my KMM data in your Skrooge GUI, i have a lot of new learning to do. I am very familiar with using KMM, but at the moment very unfamiliar with your program. I need to learn how to do all the operations in S that i know well in K... & especially i need to learn if S reports are as powerful & customisable as the K ones [on which i heavily rely, especially those that allow me to project my indicative account balances deep into the future based on my detailed Schedules... Skrooge MUST be able to do this or else it will be of no use to me]. If my study of S does not teach me the answers to those questions, i might need to return to this forum to ask for further help please.

Anyway i'd like to offer congratulations for creating Skrooge. KMM is excellent, but it can only be a very good outcome if KMM now has a quality "competitor" finance package for home users.

Hi,

I'm happy to see that your are not blocked. I'm sure that Skrooge will bring you all the functionalities you need. But, if some are missing don't hesitate to send me a mail to request me to develop them.

Could you provide me a sample file with the strange character, I would like to do a correction to treat correctly the data (if possible) or return a more understandable error message?

Thank you for using Skrooge.
Regards.

I'm not going to share my entire file with you / the public, for rather obvious reasons, but i'm happy for you to have this excerpt. It contains three [3] of the total 28 instances of the UTF8-invalid characters that were mysteriously buried in my original `.kmy` file:

Code: Select all

\00

I hope this will be of some help for your analysis:

Code: Select all

   </SPLITS>
  </TRANSACTION>
  <TRANSACTION memo="" postdate="2017-05-04" commodity="AUD" entrydate="2017-05-04" id="T000000000000007910">
   <SPLITS>
    <SPLIT memo="Classic Roll Neck Skivvy, SIZE 14, COLOUR CHARCOAL, x3, $12.00 ea.&#xa;-$12.00 discount&#xa;\00Classic Roll Neck Skivvy, SIZE 14, COLOUR BLACK, x1, $12.00 ea.&#xa;\00Item Savings = - $12.00&#xa;Sub Total = $36.00&#xa;Home Delivery Fee = $9.00&#xa;Total = $45.00 incl GST&#xa;Total Savings = $12.00" bankid="" reconcileflag="2" value="-45/1" action="" payee="P000036" id="S0001" reconciledate="2017-05-12" shares="-45/1" price="1/1" account="A000433" number="">
     <TAG id="G000008"/>
    </SPLIT>
    <SPLIT memo="\00Classic Roll Neck Skivvy, SIZE 14, COLOUR BLACK, x1, $12.00 ea." bankid="" reconcileflag="0" value="12/1" action="" payee="P000036" id="S0002" reconciledate="" shares="12/1" price="1/1" account="A000601" number=""/>
    <SPLIT memo="Classic Roll Neck Skivvy, SIZE 14, COLOUR CHARCOAL, x3, $12.00 ea." bankid="" reconcileflag="0" value="36/1" action="" payee="P000036" id="S0003" reconciledate="" shares="36/1" price="1/1" account="A000601" number=""/>
    <SPLIT memo="" bankid="" reconcileflag="0" value="-12/1" action="" payee="P000036" id="S0004" reconciledate="" shares="-12/1" price="1/1" account="A000367" number=""/>
    <SPLIT memo="" bankid="" reconcileflag="0" value="9/1" action="" payee="P000036" id="S0005" reconciledate="" shares="9/1" price="1/1" account="A000388" number=""/>
   </SPLITS>
  </TRANSACTION>

Hi,

Thank you, I reproduced the issue and did a fix: https://commits.kde.org/skrooge/d8fdc34 ... a9b376df5f

Regards.

Hi Stephane

Very nice [+fast] work, thanks.

I have decided to abandon my investigation into Skrooge as a potential KMM replacement for my own personal use. I made [personal] summary notes of my reasons, for my records. If you're interested i am willing to share them here FYI, but i definitely understand if you'd rather not... given obviously my decision must have been based on negative findings. Either way, best wishes to you.

Hi,

Yes, I'm very interested by your notes. This will help me to improve Skrooge.
Regards.

OK. Now please don't be offended, as i wrote these purely for my own eyes [initially i had no intention to let other eyes read them], simply as a short sharp summary... i did not & do not intend any insult to you whatsoever.

31/12/18: Did all my initial testing in my "Manjaro KDE" VM. Since today my subsequent testing is in my "real" Skrooge, after importing my KMM data [using yesterday's file version].

Initial Thoughts

1. Dashboard is visually chaotic [albeit customisable, so maybe i could improve it?]. However:
....a. Some account balances are correct, but...
....b. most are wrong, with several being spectacularly wrong; ***SHOW-STOPPER***
....c. The huge errors might be caused by #8.
2. Skrooge file size is simply huge... >50 MB, vs <1 MB for KMM.
3. Most operations in Skrooge are very laggy [maybe because of #2 ?]
4. Reports status is currently undetermined ... Config process is incomprehensibly unintuitive: ***SHOW-STOPPER***
....a. Consequently i don't yet know if KMM-like Account Balance Forecast charts based on Schedules, are available [for nominated individual accounts].
....b. Even if they are, currently they'd be unusable because of #8.
5. The transaction Memo/Comments field is far too small for visible text; is only a single scrollable row [cf; KMM's multi-row box, which furthermore has expandable length by dragging the window].
6. Does not seem to have a way to AutoSave each xx minutes.
7. Seems to have more overall UI configurability than KMM [?]
8. The imported Scheduled Operations are an unmitigated disaster; ***SHOW-STOPPER***
....a. Almost all of the Future individual scheduled items are simply missing.
....b. Dozens of Closed schedules from up to 7 years ago are listed as Ongoing, but additionally still don't match KMM.
....c. Interpreting all these errors is hard because all my later Comments/Memos are overwritten by obsolete ones.
....d. Lots of the genuinely Closed Schedules are missing; i suspect they're in #b, but analysis is tortuous because of #c.
....e. Skrooge Schedules do not have a "Type/Name" field like KMM, so rapid visual identification of items is hard.

Final Thoughts

Skrooge is NOT a viable KMM-replacement for me atm. It has UI "challenges" & fatally severe apparent data errors. I've expended several days on this investigation, but am not willing to persist now. Maybe i'll recheck in 6 - 12 mths?

Stephane i realise that you might be inclined to ask me to send you my KMM data file so you can investigate the problems yourself. I hope you realise that i certainly cannot / will not do that. However if you feel it might help, i'd be ok with sharing the Anonymised version of it with you.

Hi,

Thank you for your feedback. I will try to give you some answers.

steffie wrote:Initial Thoughts

1. Dashboard is visually chaotic [albeit customisable, so maybe i could improve it?]. However:

By default, the Dashboard contains all widgets. But, you can build all the dashboards you want with only the widgets you want for a better display.

steffie wrote:....a. Some account balances are correct, but...
....b. most are wrong, with several being spectacularly wrong; ***SHOW-STOPPER***
....c. The huge errors might be caused by #8.
2. Skrooge file size is simply huge... >50 MB, vs <1 MB for KMM.

You must know that with Skrooge you can "undo" all modifications, even after a save.
It means that the history is saved into the document. This is why the file can be "huge", especially after an import.
To remove the history, you must select the dock "History" and click on "Clear history".
After the save, you should have a normal file size.

steffie wrote:3. Most operations in Skrooge are very laggy [maybe because of #2 ?]

I don't think that this is due to #2.
Could be interesting to have more information about these performance issues: a detailed scenario on anonymized data for example.

steffie wrote:4. Reports status is currently undetermined ... Config process is incomprehensibly unintuitive: ***SHOW-STOPPER***

Reports could seem to be complex but they are very powerfull.

steffie wrote:....a. Consequently i don't yet know if KMM-like Account Balance Forecast charts based on Schedules, are available [for nominated individual accounts].

Yes, this is available.
You just have to click on page "Report" and select "Schedule" in "Forecast" combo box.

steffie wrote:....b. Even if they are, currently they'd be unusable because of #8.
5. The transaction Memo/Comments field is far too small for visible text; is only a single scrollable row [cf; KMM's multi-row box, which furthermore has expandable length by dragging the window].
6. Does not seem to have a way to AutoSave each xx minutes.

This is not available because not needed.
Indeed, even in case of crash, Skrooge will recover the last state when relaunched.
To test it, launch Skrooge from a console, open a document, do some modification then CTRL+C in the console.
Relaunch Skrooge, you will see a "recovery" message and you will see your last modification.

steffie wrote:7. Seems to have more overall UI configurability than KMM [?]
8. The imported Scheduled Operations are an unmitigated disaster; ***SHOW-STOPPER***
....a. Almost all of the Future individual scheduled items are simply missing.
....b. Dozens of Closed schedules from up to 7 years ago are listed as Ongoing, but additionally still don't match KMM.
....c. Interpreting all these errors is hard because all my later Comments/Memos are overwritten by obsolete ones.
....d. Lots of the genuinely Closed Schedules are missing; i suspect they're in #b, but analysis is tortuous because of #c.
....e. Skrooge Schedules do not have a "Type/Name" field like KMM, so rapid visual identification of items is hard.

OK. I will do more tests about "Scheduled operations" import and try to fix discovered issues.

steffie wrote:Final Thoughts

Skrooge is NOT a viable KMM-replacement for me atm. It has UI "challenges" & fatally severe apparent data errors. I've expended several days on this investigation, but am not willing to persist now. Maybe i'll recheck in 6 - 12 mths?

KMM is a very good tool too.
The most important is that you find the one that suits you best.
Happy new year.
Stéphane

Hi Stéphane

Thanks for your detailed reply. Interesting. New comments & info fyi:

1. I played about a little bit today with the Dashboard config, & find it quite an exciting thing that i can refine it a lot to my preferences.

2. I removed the history as you advised, & truly it did dramatically shrink the filesize. It [8.6 MB] remains however still much bigger than KMM [<1 MB]. Please note -- this is NOT anything i regard as being a problem, merely an interesting observation of the comparative filesizes [if i did not have any other concerns with Skrooge then this would most certainly NOT stop me wanting to use it].

3. Regarding my need for KMM-like Account Balance Forecast charts based on Schedules, for nominated individual accounts, & your "Yes, this is available. You just have to click on page "Report" and select "Schedule" in "Forecast" combo box" ... yes i had already worked that out myself the other day, but when i tried again today i still cannot achieve a report [with future scheduled balance line-chart] for my nominated individual accounts that gives me anything like as useful a chart as KMM. On another day i would like to discuss this in more detail with you, but not now, because Reports become irrelevant until & unless the two big problems are fixed [ie, the huge account balance discrepancies of my KMM data after importing it into Skrooge, AND the major differences in the Schedules].

4. The comparatively tiny Memo/Comments field remains a concern for me, but again, this is merely a minor detail until the two big problems are resolved.

5. I've not yet tested it but am relieved to learn of your "even in case of crash, Skrooge will recover the last state when relaunched" -- that really does sound very comforting.

6. Upon launching Skrooge today [ie, my real data file imported the other day from KMM], to my shock after an initially puzzling lag opening the file, this popup Notification arose. I've manually anonymised the entries via text-editor:
https://pastebin.com/Xh46dNd1
It shocked me for several reasons, including that:
(a) Line 1 is wrong [in KMM this Scheduled Transaction is configured to alert me but NOT to be automatically written into the ledger, ie, my manual intervention is mandatory for it]
(b) Why the huge number of ancient transactions?
(c) "Information: Operation '04/01/2019 jjjjj -125.00 AUD' will be inserted in 2 days" has a simply massive amount of duplication; if i understand correctly how Skrooge proposes to process these, it will result in another huge data error.

7. Finally today i performed a fresh experiment, & i propose to share all the underlying data with you for your analysis [but only if you would like it, of course].
(a) In KMM i Saved As my normal file in the Anonymised XML format
(b) Still in KMM i then Opened that Anonymised XML file, then Saved As back into standard `.kmy' format... this file is `Steffie_2003+.anon.kmy`
(c) In Skrooge i made a New Document & Imported this `Steffie_2003+.anon.kmy`
(d) In this new `Steffie_2003+.anon.skg` file in Skrooge i deleted History and Unused Categories, & made a minor layout change on Dashboard... but other that these trivial actions --In Theory-- ALL entries in `Steffie_2003+.anon.kmy` and `Steffie_2003+.anon.skg` logically SHOULD be numerically identical. As these pictures prove, that is not what actually appears. Not only are balance numerical disparities obvious for some of the Open accounts, but Skrooge also has reopened some of my old Closed accounts.





Once you reply, if you want them, i shall attempt to email both anonymised files to you.

Hi Steffie,

The point (6) is very strange and of course can explain the differences of balances.
This is clearly not normal that an old schedule is still treated as "active".

I did a test with this kmy file (https://my.pcloud.com/publink/show?code ... esy7dk8VLk) containing only 2 scheduled operations.
- Both old.
- One with automatic creation, the other without.
But, as you can see in following screenshot, the import in Skrooge is the expected one:
- The scheduled operations are disabled (date in red)
- The first scheduled operation is not automatically created.
- The second scheduled operation is automatically created.


So, I don't understand why you have some old schedule still active.
As proposed, could you send me the anonymized kmy file by mail (you can find my address in the "About" of Skrooge).

Just for information, you can have a difference of global balance if you have some operations in a unit not the principal one ($ in your case).
For example, if you have some shares and you didn't download the quotes of these units from the "Unit" page.

Regards.

As proposed, could you send me the anonymized kmy file by mail (you can find my address in the "About" of Skrooge).

Within the past couple of hours i emailed you the files, & so far i've not received any system bounce-back error message etc, so hopefully my email arrived ok at your Inbox.