Calc will not save file after sheet deleted.

I agree but there are no good front ends to the myriad of relational database backends that you can utilise on Linux.

I could list tons of SQL engines but as the copious posts on this list about Base attest, there are few decent alternatives for Rapid Application Development, Data Mining or even simple application development available; let alone something that can be integrated into LO.

When working on Windows I use Microsoft Visual Foxpro. Fantastic package but now discontinued (and I have moved my primary platform to Ubuntu now!). There is absolutely nothing comparable to it on Linux. You either have to write a complete application every time you want to do something (e.g. Python+wxPython; Dabo) or need to poke at a SQL file from the command line.

If I could recreate my spreadsheet in a database format that allowed me to quickly develop and easily maintain an application -- I would be developing it now.

Simon

> Hmm, well maybe not the absolute worst. A sieve or broken
> floppy disc or an ancient format that no program can read
> might be worse but yes, databases with an audit-trail are
> much more secure and plain text such as Csv ensure that there
> will always be some program somewhere that can at least
> access the data.

I agree but there are no good front ends to the myriad of relational
database backends that you can utilise on Linux.

I could list tons of SQL engines but as the copious posts on this list
about Base attest, there are few decent alternatives for Rapid
Application Development, Data Mining or even simple application
development available; let alone something that can be integrated into LO.

When working on Windows I use Microsoft Visual Foxpro. Fantastic package
but now discontinued (and I have moved my primary platform to Ubuntu
now!). There is absolutely nothing comparable to it on Linux. You either
have to write a complete application every time you want to do something
(e.g. Python+wxPython; Dabo) or need to poke at a SQL file from the
command line.

If I could recreate my spreadsheet in a database format that allowed me
to quickly develop and easily maintain an application -- I would be
developing it now.

--
Cheers Simon

    Simon Cropper
    Principal Consultant
    Botanicus Australia Pty Ltd
    PO Box 160, Sunshine, VIC
    W: www.botanicusaustralia.com.au

Both MySQL/MariaFB and Postgresql have GUI interfaces available for
Linux , MySQL Worbench from the MySQL site and pgadmin for Postgresql in
the Ubuntu repository. Both allow a users to most of the db work in a
desktop environment not CLI.

All these packages are administration utilities not RAD or Data Mining environments.

"MySQL Workbench is a visual database design tool that is developed by MySQL. It is the successor application of the DBDesigner4 project."
http://wb.mysql.com/?page_id=28

"pgAdmin is designed to answer the needs of all users, from writing simple SQL queries to developing complex databases. The graphical interface supports all PostgreSQL features and makes administration easy. The application also includes a syntax highlighting SQL editor, a server-side code editor,...." http://www.pgadmin.org/

Data mining allows the 'researcher' to import, export, convert, merge and manipulate data for a particular project, and maintain the original data and any derivatives.

RAD allows, at minimum, input forms, menues and reports. Ideally a programming language should be available to manipulate the data and user experience.

Granted I should have been more careful with my choice of words. All these packages do provide simple GUI interfaces that allow you to design and query a database created with the respective tool. Some projects do provide some import and export facilities for use in a once-off situation. BUT they are not really designed as RAD tools or provide you with the *easy* ability to collate disparate data sources, manipulate this information and export it again.

...

If I could recreate my spreadsheet in a database format that allowed me
to quickly develop and easily maintain an application -- I would be
developing it now.

Pick OS
We used to sell it when I was based in Australia (McDonnell
Douglas/Microdata).

Too bad Dick Pick isn't still around :slight_smile:

Tom,

I have checked all these issues. Permissions, folder permissions, owner/grp attributes have all been checked.

As stated I can turn the save / no-save behaviour on and off with deletion or reinsertion of comments into the sheet. If the file was read only or write protected then this could not be done.

No harm asking though :slight_smile:

Simon

> Simon
>
>
>> > Hmm, well maybe not the absolute worst. A sieve or broken
>> > floppy disc or an ancient format that no program can read
>> > might be worse but yes, databases with an audit-trail are
>> > much more secure and plain text such as Csv ensure that there
>> > will always be some program somewhere that can at least
>> > access the data.
>>
>> I agree but there are no good front ends to the myriad of relational
>> database backends that you can utilise on Linux.
>>
>> I could list tons of SQL engines but as the copious posts on this list
>> about Base attest, there are few decent alternatives for Rapid
>> Application Development, Data Mining or even simple application
>> development available; let alone something that can be integrated into LO.
>>
>> When working on Windows I use Microsoft Visual Foxpro. Fantastic package
>> but now discontinued (and I have moved my primary platform to Ubuntu
>> now!). There is absolutely nothing comparable to it on Linux. You either
>> have to write a complete application every time you want to do something
>> (e.g. Python+wxPython; Dabo) or need to poke at a SQL file from the
>> command line.
>>
>> If I could recreate my spreadsheet in a database format that allowed me
>> to quickly develop and easily maintain an application -- I would be
>> developing it now.
>>
>> --
>> Cheers Simon
>>
> Both MySQL/MariaFB and Postgresql have GUI interfaces available for
> Linux , MySQL Worbench from the MySQL site and pgadmin for Postgresql in
> the Ubuntu repository. Both allow a users to most of the db work in a
> desktop environment not CLI.
>

All these packages are administration utilities not RAD or Data Mining
environments.

"MySQL Workbench is a visual database design tool that is developed by
MySQL. It is the successor application of the DBDesigner4 project."
http://wb.mysql.com/?page_id=28

"pgAdmin is designed to answer the needs of all users, from writing
simple SQL queries to developing complex databases. The graphical
interface supports all PostgreSQL features and makes administration
easy. The application also includes a syntax highlighting SQL editor, a
server-side code editor,...." http://www.pgadmin.org/

Data mining allows the 'researcher' to import, export, convert, merge
and manipulate data for a particular project, and maintain the original
data and any derivatives.

RAD allows, at minimum, input forms, menues and reports. Ideally a
programming language should be available to manipulate the data and user
experience.

Granted I should have been more careful with my choice of words. All
these packages do provide simple GUI interfaces that allow you to design
and query a database created with the respective tool. Some projects do
provide some import and export facilities for use in a once-off
situation. BUT they are not really designed as RAD tools or provide you
with the *easy* ability to collate disparate data sources, manipulate
this information and export it again.

--
Cheers Simon

    Simon Cropper
    Principal Consultant
    Botanicus Australia Pty Ltd
    PO Box 160, Sunshine, VIC
    W: www.botanicusaustralia.com.au

I should read more closely sometimes.

I would think the problem with data mining is determine what bits might
be yield nuggets out of the mass.

I have not used Pick. I have used Revelation before, which is a derivative of this system. Not bad but still Windows based and files in proprietary format.

The beauty of dbase files was for a long time they were the default 'desktop' (rather than server style) database format. Information stored in this format was accessible by a wide variety of packages (foxbase, foxpro, dBaseIII/IV, Clipper, Paradox, ...).

Dbase files are old school now. The closest format is sqlite. This relational database system is serverless making it portable and able to be maintained in a similar fashion to the dissociated project based tables that formed part of the dbase database. But alais no packages are around that provide the functionality I desire.

Before anyone tells me I am aware of 'sqlite data browser' and sqliteman and the firefox addon. All useful to query and maintain files but not good for data mining or RAD.

*My attempts at recreating a clean file*

ATTEMPT 1

Created a new file with one less tab. All tabs were named the same as the original. I then cut-and-paste all the data from the original to this newly created file. This took about 10 minutes.

The problem is that all tab / sheet references after the omitted sheet moved across one. Despite the formula referencing the sheet name, the name is actually just a human readable cue for the underlying sheet number. So sheet Accounts (sheet #17 on the original file), which pointed to 'accounts lookup list' now points to the Tax sheet, which was #18 but is now #17 in the new file.

ATTEMPT 2

Created a new file with exactly the same number of tabs. All tabs were named the same as the original. I then cut-and-paste all the data from the original to this newly created file. This took about 10 minutes.

Then with baited breath I deleted the unrequired sheet that triggered this journey in the first place and clicked save. *It worked* - that is, it saved without triggering a write error.

I have checked - (a) all data is intact, and (b) all the formulas are working. :slight_smile:

Simon,

> *The upshot*
>
> Unless someone recognised these symptoms and can put forward some ideas,
> I am treating this as a corrupt file. I have drawn this conclusion
> because (a) I can only reproduce the error on the effected file, (b) the
> comment insert/delete trick is suggestive but not consistent (or at
> least from what I can tell) - random sheets cause the file save error if
> comments are inserted; inserting comments into new sheets also trigger
> the error.
>
> I just hope that recreating this file does not corrupt the new file as I
> will need to copy blocks of text, numbers and formulas.
>

             *My attempts at recreating a clean file*

ATTEMPT 1

Created a new file with one less tab. All tabs were named the same as
the original. I then cut-and-paste all the data from the original to
this newly created file. This took about 10 minutes.

The problem is that all tab / sheet references after the omitted sheet
moved across one. Despite the formula referencing the sheet name, the
name is actually just a human readable cue for the underlying sheet
number. So sheet Accounts (sheet #17 on the original file), which
pointed to 'accounts lookup list' now points to the Tax sheet, which was
#18 but is now #17 in the new file.

ATTEMPT 2

Created a new file with exactly the same number of tabs. All tabs were
named the same as the original. I then cut-and-paste all the data from
the original to this newly created file. This took about 10 minutes.

Then with baited breath I deleted the unrequired sheet that triggered
this journey in the first place and clicked save. *It worked* - that is,
it saved without triggering a write error.

I have checked - (a) all data is intact, and (b) all the formulas are
working. :slight_smile:

--
Cheers Simon

    Simon Cropper
    Principal Consultant
    Botanicus Australia Pty Ltd
    PO Box 160, Sunshine, VIC
    W: www.botanicusaustralia.com.au

+1
Now if configure out what caused the file to corrupt, but we may never
know.

Congratulations,... it was an interesting investigation.

Regards,

Jorge Rodríguez

Thanks.

It was interesting. I am upset though I was unable to pin down the exact cause of the corruption.

+/- the comment was obviously a symptom of a underlying structural problem solved through ATTEMPT 2 below.

The file was originally created in June 2008 to replace my accounting package due to my frustration at the package not being able to do simple transactions and not being able to get at the underlying data or lookup tables (e.g. tax schedules).

The ODS spreadsheet has been accessed at over 152 times from both Windows XP/Ubuntu/XP+VirtualBox, had data imported from external Internet sites 45 times as CSV files, purged then repopulated 3 times and converted from XLS to ODS once.

I think all things considered the file has been relatively stable. What some people seem to forget who 'bag' the format, is that I have been able to access my data the entire time *even when this issue occurred*. In the absence of a RAD for a database system on Linux I will be sticking to using LibreOffice Calc to complete my accounting needs.

It is obvious that you do not like OO or LO Calc, and that you don't
like the fact that your data is continually corrupted.

LO is a tool. I neither like or dislike it. As a tool, it has a major
defect in that it periodically destroys the product which was built
using it. You're right in that I don't like the tool I'm using to
destroy my work.

You also say you continuously have to rebuild your spreadsheet and that
the process can take several days.

Just spent 2 long days recreating the spreadsheet using 2 machines. Not fun.

If you are so unhappy with the format, corruption and dirty code, my
question is "why do you persist?"

I've got a lot of investment in my macro code (9000 lines of it) that
can only function within LO. I can't migrate it to something else as is.

Had I known about the corruption when I started the project 6 years ago,
I would have chosen a different platform. I would have never imagined
that over the course of several years that corruption issues would be
ignored. That's just inconceivable to me as a professional software
developer, but that's apparently the reality.

Change packages.
Can't, as was mentioned above, unless I rewrite everything. Sad to say,
LO is the only real competitor to MS Office and given its corruption
history it will turn a lot of people off that might have moved away from
MS Windows to Linux had the product been reliable.

My real interest is in moving people away from a lousy operating system
like Windows to something near bullet proof like Linux. We all need
applications tools like word processing, spreadsheet, etc, so a tool
like LO is vitally important to Linux adoption.

If I could get MS Office compiled to run on Linux as a native
application, I'd forget about LO in a heartbeat. MS Office is the best
thing Microsoft has. It works well in its native environment. I wish I
could say the same for LO, and that's why I'm critical of it.

With the time and effort you have already put into recreating corrupted
spreadsheets you

could of written your own application or mastered alternative packages.

I guess you don't write code for a living. I estimate a rewrite of my
POS system would take well over a year of my time. While that's going
on, I need something reliable to help run a business. I have sketched
out a web based replacement much much more sophisticated than what I'm
using now, and plan on developing it when I have the time.

Wasting time reporting OO / LO bugs that never get taken seriously and
recreating a spreadsheet periodically helps keeps me from such a new
development effort. My wife (also a professional software developer)
keeps telling me to stop wasting time reporting issues - that I'm
tilting at windmills. She may be right.

I am not trying to be rude here it just if I had a package that caused
me so much wasted time I would just move on.

I will move on in time, but in the interim, I'd like LO to thrive in the
market, and that won't happen unless it becomes reliable. My problem is
really with LO management. There isn't any.

Despite what you may think, I'm more interested in LO's success than
most people are, and I know it won't be successful long term unless its
reliable. It hasn't been reliable for years and that's a problem.

- --
Bill Gradwohl
Roatan, Honduras

Am 11.08.2011 01:38, Simon Cropper wrote:

> Hmm, well maybe not the absolute worst. A sieve or broken
> floppy disc or an ancient format that no program can read
> might be worse but yes, databases with an audit-trail are
> much more secure and plain text such as Csv ensure that there
> will always be some program somewhere that can at least
> access the data.

I agree but there are no good front ends to the myriad of relational
database backends that you can utilise on Linux.

A tiny collection of dBase files provides a much better data container than the best spreadsheet.
If you used to use a spreadsheet which goes down the drain now, then a set of most simple unrelated dBase files with simple forms, import ranges and some data pilots can do anything you used to do in the spreadsheet alone. Forms and pilots can live in the same spreadsheet document while the raw data are separated in a dBase directory to be used seamlessly in OOo Calc or something else.

Hi Simon,

If I work back through the historical copies of the file, I can only
reproduce the error in this years version. It is as if the file has
become partially corrupted. The question is how and can it be easily fixed.

Use xmlpp and xmldiff (perl scripts) to compare the content.xml of each
ODS between a good (older) file and one of the recent ones where the
problem started occurring. You may find that you have some odd
difference in the two files that causes the behaviour you mention.

Alex

Hi Everyone,

I have been doing some follow-up investigations that I thought you might like to be made aware.

First, why? Well after testing the file that I recreated by copying the text, formula and styles I could not cause the error to occur again. This is what I reported yesterday. I then proceeded with importing the few images and 'artwork' objects from the old file to the new. After several saves and sheet manipulations I noticed the file became unstable again.

The outcome of my investigation was that one of the image files used on one of the sheets had become corrupted* and adding this file in caused the file to become unstable - eventually exhibiting the problem I mentioned about not being able to save after a sheet was deleted. These 'corrupted' files are relatively benign with the error only becoming apparent *once* you attempt to delete the sheet. So somehow the error caused problems with the broader 'workbook' structure not the sheet structure. Although I compared files between different versions of the file the XML were too varied (usually style names and definitions; content was identical).

    * note I say corrupted but it rendered OK and only resulted
    in the observed behaviour one a sheet is deleted. I 'deem' it
    corrupted as once replaced with a clean version render and
    saved as a new file by GIMP, the problem disappeared.

The error was with the particular corrupt objects. On recreating new images and inserting them into a file I have not been able to trigger any problems. If I cut-and-paste from the original 'corrupt' file the file becomes unstable after a few saves and sheet manipulations.

So the steps for salvage is...
1. recreate a new file with the exact number of sheets as the original. Ensure each sheet names are the same.
2. Cut and Paste each sheet. Make sure you 'Paste special' limiting the content being placed in the new file to text, numbers, date & time, formulas and formats.
3. If you have images in the file. Recreate / Save using another package. As mentioned I used GIMP.
4. Insert new copies of the images into file. *Don't* cut-and-paste objects from the old file.
5. If you have any lines, text boxes and artwork; recreate them from new.

During this process...
- Save as a new version (+tabs, +data, +images, +other objects) following each step.
- Test each version thoroughly before proceeding.
- Only add one object at a time, so if something goes wrong you can isolate the problem component.
- To check it is not a bug try and recreate with a fresh file.

A couple of quick notes that may be valuable to others...
- ODS files are archives. Use an archive facility to extract the data inside. Inspect the contents in the folders to see what is different.
- On every 'Save as' the file size changes. This is not due to changes in the file contents, but rather in changes in how the components of the file is compressed/archived. If you open a file, add objects then save, the file size with be so big. Open that file and "Save as" a new name and the file will be a different size. If you extract the files the contents of all the files and directories are identical. It is just the internal archive facility in LO will decide the best compaction routine based on what it encounters.
- The content.xml file can be quite large and has no internal end-of-line characters. This make it difficult to open and be parse by various text editors, xml viewers and comparison facilities. To insert a EOL character after the end of each tag (i.e. >), I used the following command in the terminal (requires Linux).

    cat content.xml | sed -e 's/>/>\n/g' > content_with_linebreaks.xml

    cat just spews the content of the text file to the standard output.
    I then pipe it to sed, where I used regular expressions to find '>'
    and replace every instance of it with '>\n'. I then compared the
    contents with Diffuse.

Hi

  I'm some confused. Was the comment or the image the problem ? or Both ?

Regards,

Jorge Rodríguez

Hi Everyone,

I have been doing some follow-up investigations that I thought you might
like to be made aware.

First, why? Well after testing the file that I recreated by copying the
text, formula and styles I could not cause the error to occur again.
This is what I reported yesterday. I then proceeded with importing the
few images and 'artwork' objects from the old file to the new. After
several saves and sheet manipulations I noticed the file became unstable
again.

The outcome of my investigation was that one of the image files used on
one of the sheets had become corrupted* and adding this file in caused
the file to become unstable - eventually exhibiting the problem I
mentioned about not being able to save after a sheet was deleted. These
'corrupted' files are relatively benign with the error only becoming
apparent *once* you attempt to delete the sheet. So somehow the error
caused problems with the broader 'workbook' structure not the sheet
structure. Although I compared files between different versions of the
file the XML were too varied (usually style names and definitions;
content was identical).

    * note I say corrupted but it rendered OK and only resulted
    in the observed behaviour one a sheet is deleted. I 'deem' it
    corrupted as once replaced with a clean version render and
    saved as a new file by GIMP, the problem disappeared.

The error was with the particular corrupt objects. On recreating new
images and inserting them into a file I have not been able to trigger
any problems. If I cut-and-paste from the original 'corrupt' file the
file becomes unstable after a few saves and sheet manipulations.

So the steps for salvage is...
1. recreate a new file with the exact number of sheets as the original.
Ensure each sheet names are the same.
2. Cut and Paste each sheet. Make sure you 'Paste special' limiting the
content being placed in the new file to text, numbers, date & time,
formulas and formats.
3. If you have images in the file. Recreate / Save using another
package. As mentioned I used GIMP.
4. Insert new copies of the images into file. *Don't* cut-and-paste
objects from the old file.
5. If you have any lines, text boxes and artwork; recreate them from new.

During this process...
- Save as a new version (+tabs, +data, +images, +other objects)
following each step.
- Test each version thoroughly before proceeding.
- Only add one object at a time, so if something goes wrong you can
isolate the problem component.
- To check it is not a bug try and recreate with a fresh file.

A couple of quick notes that may be valuable to others...
- ODS files are archives. Use an archive facility to extract the data
inside. Inspect the contents in the folders to see what is different.
- On every 'Save as' the file size changes. This is not due to changes
in the file contents, but rather in changes in how the components of the
file is compressed/archived. If you open a file, add objects then save,
the file size with be so big. Open that file and "Save as" a new name
and the file will be a different size. If you extract the files the
contents of all the files and directories are identical. It is just the
internal archive facility in LO will decide the best compaction routine
based on what it encounters.
- The content.xml file can be quite large and has no internal
end-of-line characters. This make it difficult to open and be parse by
various text editors, xml viewers and comparison facilities. To insert a
EOL character after the end of each tag (i.e. >), I used the following
command in the terminal (requires Linux).

    cat content.xml | sed -e 's/>/>\n/g' > content_with_linebreaks.xml

    cat just spews the content of the text file to the standard output.
    I then pipe it to sed, where I used regular expressions to find '>'
    and replace every instance of it with '>\n'. I then compared the
    contents with Diffuse.

Snip

+1

I wonder how the images got corrupted, interesting.

*** Jorge, as I respond to your email I find out a range of things of note, as I try and support or refute my ideas. It is enlightening to read the entire post but you can just as easily jump to >>HERE<< ***

Sorry Jorge,

I suppose it is confusing.

The problem is the corruption due to the insertion of the image is not immediately obvious. Then as the file becomes more progressively more corrupt.

The sequence was like this
1. Cut-and-paste data
2. Reinsert images and objects
3. Other stuff...
4. Issue with comments noted

What surprised me was after recreating a clean file for use I then ended up with an error popping up. No comments were found anywhere.

So I stripped the file back to the individual components -- data, format, images, etc. Adding each one separately then testing extensively.

On adding the images I noted some funny stuff happening to the file size. as I saved, re-saved, deleted sheets, re-saved, etc... I noticed that the file showed more problems rendering images (images disappeared as well as references to files disappeared). Soon afterwards the file had problems saving after deletion of the sheet.

So I presume that the images caused the corruption of the structure, the problem progressively got worse, then the comments field tipped the scale and caused the error in saving after the deletion of the sheet. anyway that seems as plausible as any explanation I have seen already.

As stated I am unable to reproduce these errors on new files. despite trying repeatedly for hours.

As stated previously I was able to shake the problem by cutting-and-pasting the data into the spreadsheet. This included everything images and all. I could not trigger the error despite hours of testing. Consequently I posted the 'solution' to this thread.

After several hours of adding and deleting data, the problem reappeared.

As stated I then recreated the file including data-formula-format, minus other objects then progressively built up the file until it mimicked the original. Each step I tested, retested and checked yet again. In the end I noted things got 'wonky' after insertion of the images-artwork. this did not particularly make sense to me but I persisted in splitting the objects being inserted in the document until I could identify the culprit. This eventually identified a single image (originally a GIF that was converted to a PNG). Add this image to the sheet and progressively problems appear. If I recreate a PNG file using GIMP and insert this I can't get the problems to appear. This is using the same file, data and files - side by side. Very confusing.

There is only two conclusions
(a) that the file is corrupt, something corrupted it and at present the first thing I notice a problem occurring after it is inserted is this image.
(b) that LO accesses and manipulated the file incorrectly. In other words their is a bug in the code.

As I am not inclined to post a file with all my financial details on the Internet, and I have no other versions of LO around to test the second theory I am in a quantry.

>>HERE<<<

Well... As it turns out I have an old version of OpenOffice 3.1.0 OOO310m11 (Build: 9399) lying around on an old Windows Vista laptop I was unaware about.

Exact same file (except for one computer is accessing the local drive and the other a SAMBA network drive)...
- LO baulks with the error "blah blah unable to write"
- OO does not! Whaaaa?

Alright I am on Ununtu. I opened a copy of the file via nautilus from a local drive mounted using fstab - the error occurs. If I use nautilus to locate and access the same samba share the windows machine used - the error occurs.

Somehow I am relieved.

OK, I test using another machine ubuntu-ubuntu using LO 3.3.2, both via a NFS mount and Samba share. Both sheet deletions and saved worked on the exact same file that my machine baulks with - that is the error did not occur.

I am bloody confused! The problem is "computer specific and file specific". As part of the entire exercise I completely uninstalled every LO related package and reinstalled them with all the dependencies.

I have check and rechecked the file permissions, ownerships and groups. Nothing untoward.

Anyone got any ideas? if the disk was damaged then then all computer would complain. This leaves be with some conflict or background process that is interfering with the operation I am trying to do... you know the operation I can reproduce on any other file, except when I cut and paste data from the original file. Sound a lot like a corrupt disk, but how does the NFS and Samba Shares NOT trigger this error.

planas,

See my reply to Jorge. In particular the discovery that suggests that the file may be physically damaged, or at lease the sector that it resides. I am going to check for damage now.

Thats potentially dangerous, as you could have embedded greater than
signs in your data.

Use XML Copy Editor (http://xml-copy-editor.sourceforge.net/) to pretty
print and work on the pretty print copy.

Or

Use xmllint to pretty print a version via :
xmllint --format content.xml >content.xml.pp

Better still canonicalize the xml and then pretty print it if you want
to use any diff routines to highlight changes between versions.

See http://en.wikipedia.org/wiki/Canonical_XML

xmllint --c14n content.xml|xmllint --format - >
content.xml.cannonicalized.pp

- --
Bill Gradwohl
Roatan, Honduras