Base record access unacceptably slow

Hi :slight_smile:
I think it's more a case of working on several fronts within TDF.
1. joining the Steering-Discuss mailing list and promoting the cause there.
2. working out a few large organisations that could put resources towards it.
What would they gain? Canonical (of Ubuntu fame) and RedHat might reduce a
"blocker" that stops people from leaving Windows. Google migth gain an
OpenSource database that they could adapt to add to their Cloud initiative,
Google-Docs. All three would increase the level of support they could sell to
companies and perhaps even to individuals. Support contracts are one of the
main money spinners in OpenSource.

3. find places where devs are. Preferably ones that are familiar with working
on database issues and see if we can encourage some to work on Base. But they
would have to be happy about releasing the code under copy-left rather than
copyright agreements to keep things OpenSource. There might be other OpenSource
programs that rely on Base in some way that might be willing to encourage some
of their devs to take the opportunity to steer things in a useful direction for
their needs.

Errr, some of these ideas might be daft and i am very likely missing some other
positive directions. As for the press it's better if outsiders see TDF in a
positive light so that people are attracted to the quagmire that is Base and
pleased to join in and fix it.
Regards from
Tom :slight_smile:

Once again I find myself agreeing with you, on all fronts. I'm also keenly aware that I'm maxed out right now and must first get my db running (via java regression) then get some work done. Hours I don't have have been lost this morning trying to decide what to do about all this, and I must get something moving forward, immediately.

Will return.

Tom C

Hi Tom,

Where the &^%$ is the management - The Document Foundation - in all
this, right now, today? Do they even watch this list? In short, do they
give a damn that the only theoretically viable alternative to Access
(for ordinary users) is in real trouble? Why aren't they showing up here
with some clarifying position statement?

Been there, seen it, done that. The clarifying statements of opinion I
have had from the developers - they are after the all the ones that
write the code, either as volunteers and/or paid employees, are those
that Michael Meeks has expressed (albeit in other words). If you, as a
user, want Base to be actively supported, corrected, developed...then
sign up to a commercial support contract where your concerns will be
taken as a business case user by that support company, be it Novell,
Suse, Redhat, Lanedo, or whoever else might be in that loop. I would
actually probably pay for support if I knew that it would go towards
fixing the things I want fixed in Base. However, the support contracts
being offered are, to my knowledge, general support for the whole of the
suite, not specifically oriented to Base as such, in fact some of them
are even more general than that, i.e OS-support based, and some of them
are rather pricy. Again, price would probably not be an issue if I was
guaranteed that the money would go to paying someone to develop on Base.

History calls :
If one looks back to how the current HSQLDB came to be in Base, this was
only due to the fact that Sun helped pay Fred Toussi (the guy who
developed HSQLDB) to help them integrate it into OOo - otherwise it was
a non-starter. Fred also had donations from the community to help get
the work done.

So there you go, even back then, Base would not have become what it is
today without independent funding of development. Perhaps, a new set of
funding is indeed what is required to bring about changes in Base as it
is now, and develop for the future.

I'm desperate for time, a fix, and vision of a long-term solution to
this mess. I have work to do today, a lot of it, and I can't do it. I
can't solve the problem, and other than by implementing the
regress-your-java solution idea (which I have yet to be successful
with). No one else is solving it, either. For some, migrating to another
backend is not a challenge. For the rest of us, it's unknown territory.
I researched this a bit, and while there certainly IS stuff out there
about how to do it, there's not a lot, and there are multiple levels of
challenge with this solution anyway.

Well, I jumped onto the mysql bandwagon very early on, before Base 2
even came into being. As Heinz has said ODBC worked fairly well even
back then and it was lightning fast. Unfortunately, things are not going
so smoothly with that solution for me now on OSX, where I can't get it
to work, even with a commercial paid-up ODBC driver.

Sun also came along and developed the mysql connector extension. This
actually still works fairly well for me, but none of the developers are
building it and making it available on the extensions site, which is
what Sun used to do each time a new version of OOo was released. There
has not been one single release of the connector extension by the
LibreOffice project since its inception. After enquiring over at the
Apache OOo project, the mysql is not part of the software grant from
Oracle, and so because the libmysql library is GPL code, it can't be
included in the Apache repositories and therefore the connector
extension will not be built and hosted by the Apache project. In other
words, unless someone else independently and repeatedly builds andhosts
the connector for each new version of OOo/LibO that comes out, and for
each platform, even the native mysql connector is doomed.

The other alternative to MyODBC or the native connector : JDBC
Connector, but again, this is not without several major problems,
including performance from within LibO, and date string management, and
blob and object management and, and...all of which are problems that
mysql (now Oracle) were aware of and never bothered to fix.

Anyway, other avenues are out there, I am exploring them as I speak,
because I'm not going to keep flogging a dead horse for much longer. If
it can be shown that its not actually dead, then I'm all for helping out
testing, documenting, etc, but if TDF itself is not prepared to clearly
show that this module is one of its centres and remain true to its
declaration of "protecting the investments of the past 10 years", I'm at
a loss to see what difference my willingness will make.

Alex

Hi :slight_smile:

Apache are far behind TDF in this. They are about where TDF was about 11 months
ago; with no infrastructure and bloated code filled with nonsense entangling the
useful stuff.

TDF is not traditional hierarchical organisation where "they" run things and
"we" wait for "them" to do stuff. There is no "them" and "us" there is only
"us" and "some more of us". We need some of us that have experience and
knowledge about databases to join in with management levels by joining
Steering-Discuss and finding ways to manage a drive forwards for Base. We need
to manage this not sit&wait.

Regards from
Tom :slight_smile:

Hi :slight_smile:
It might be a good idea to post a bug-report
http://wiki.documentfoundation.org/BugReport
I think the guide helps you look-up to see if there is already a bug-report
about it but if you don't have time for that just post the reoprt and worry
about it later. Triagers can mark bugs as duplicates quite quickly when they
are on a roll.

Regards from
Tom :slight_smile:

Heinrich,

Hi Tom,
Just my - maybe naive - 2 pennies worth of comments:
As I mentioned before, I have been using OO/LO for years together with
BASE with MySQL (which - also "sigh") now "belongs" to Oracle as well.
As a connector to the DB I have tried out the Java version (always had
trouble with it one way or the other!) as well as ODBC/UNIXODBC (which
is what I am now using under Linux and which works sort of!). With
MySQL there is also the native connector which I would actually prefer
to use - but, yes you guess right - does NOT work on either LO 3.3.3 or
3.4.1 under Linux. A bit confusing, isn't it!
I am not giving up hope yet for LO 3.4.2 - especially since it targets
business...
Isn't there a OpenSource fork of MySQL called MariaDB?!?
Regards
H

MariaDB is an OSS fork of MySQL, it is available from the Ubuntu
repositories, I assume Debian and other Linux distros.

It apparently is only available for Linux and Windows, no Mac version
listed on their downloads. Their homepage is > mariadb.org

Hi Tom,

Tom Cloyd wrote (28-07-11 18:06)

I completely agree. LO without a function Base won't fly. Either TDF
recognizes this - by DOING something - or we have our answer.

Well, the challenges with base are known. This does not mean that there is an instant solution. However the fast growing support for TDF and from interested people with developing skills, make that there are reasons for some optimism :slight_smile:
Maybe the LibreOffice conference or another hackers event is a good opportunity too to make so first steps, share experience ...

Regards,
Cor

Am 28.07.2011 19:05, Jean-Francois Nifenecker wrote:

Hi,

In my view LO/OO do NOT require Java, neither on Linux nor on Windows.

AFAIK, LO/OOo currently *DO* require Java for Base to simply work. TDF
have announced they would get rid of the java-isms in the code. This
implies a major rewrite of Base, though.

No, this is not true. You can use any non-Java database in OOo. You can build queries and forms manually in design view, and you can use external document templates for reporting. Both built-in report generators use Java. IMHO Calc outperforms both report generators anyway.

Hit F4 in Writer or Calc, right-click>Open the dBase "Bibliography".
All the functionality of a flat (unrelational) dBase connection is there. You can connect, query, edit data through forms and dump any row set into office documents.

http://www.libreoffice.org/get-help/system-requirements/
<quote>
For certain features of the software - but not most - Java is required.
Java is notably required for Base.
</quote>

________________________________
From: Cor Nouws <oolst@nouenoff.nl>
To: users@global.libreoffice.org
Sent: Thu, 28 July, 2011 20:47:59
Subject: Re: [libreoffice-users] Re: Base record access unacceptably slow

Hi Tom,

Tom Cloyd wrote (28-07-11 18:06)

I completely agree. LO without a function Base won't fly. Either TDF
recognizes this - by DOING something - or we have our answer.

Well, the challenges with base are known. This does not mean that there is an
instant solution. However the fast growing support for TDF and from interested
people with developing skills, make that there are reasons for some optimism :slight_smile:
Maybe the LibreOffice conference or another hackers event is a good opportunity
too to make so first steps, share experience ...

Regards,
Cor

Hi :slight_smile:
If that was going to be enough then we would already have seen some efforts made
in Base in the last 11 months just as we have seen efforts and even good results
in all the other apps. Waiting and hoping is getting nowhere. It is also not a
good management strategy.

Regards from
Tom :slight_smile:

Am 28.07.2011 08:30, Tom Cloyd wrote:

As an aside, have you thoughts to share about HSQLDB vs H2? Any good
reason to migrate to H2 (a question entirely separate from the db speed
question). I'd be interested to hear your thoughts if you have time to
share them.

The tiny user community of the Base component gathers on http://user.services.openoffice.org/en/forum/index.php and http://www.oooforum.org/forum/viewforum.phtml?f=10.

Have a walk through http://user.services.openoffice.org/en/forum/viewforum.php?f=83 and http://user.services.openoffice.org/en/forum/viewforum.php?f=100.

Read contributions by most valued member "DACM". He knows "everything" about embedded HSQLDB, why not to use it, how to transform it to something useful.
http://www.oooforum.org/forum/viewtopic.phtml?t=94068 [How to: Migrate Base Projects to Multi-User]
http://www.oooforum.org/forum/viewtopic.phtml?t=97522 [Replace HSQLDB with H2 embedded multi-user]
http://user.services.openoffice.org/en/forum/viewtopic.php?f=83&t=17567&p=162653#p162653 [[Tutorial] Avoid data loss by avoiding "Embedded databases"]

Both database engines are just great, even for people who do not intend to write their own Java application around these animals.
For my last tiny project I prefered HSQLDB v2 simply because I already had working drafts in embedded HSQLDB v1.8.

I tried H2 when HSQLDB v2 was not released and I had to do some analysis work on half a million interrelated records from 2 databases. The single-user local DB simply worked out of the box, just like HSQLDB 1.8 did with less features. I copied dBase and csv data into the prepared database structure, added queries, some macros and dumped the final aggregations in Calc's pivot tables.

Replacing one excellent database backend with another excellent database backend makes no sense. The "database in a single zip archive" (the so called "Base document") is the major trouble maker which makes up a slow, inflexible, unsafe, insecure caricature of a database while the advantage is close to zero.

Hi :slight_smile:
Could that community be prevailed upon to join TDF and manage the Base part of
the project? Presumably they have knowledge of key players and have a good idea
of what needs to be done to improve Base?
Regards from
Tom :slight_smile:

Am 29.07.2011 13:42, Tom Davies wrote:

Hi :slight_smile:
Could that community be prevailed upon to join TDF and manage the Base part of
the project? Presumably they have knowledge of key players and have a good idea
of what needs to be done to improve Base?
Regards from
Tom :slight_smile:

Be assured that those guys did their very best to assist the OOo developers at Sun/Oracle with user support, bug hunting, documentation and moderate (doable) feature requests. Now those days are over.

As a heavy Base user I would be very pleased to see a functional, standard compliant, actively maintained, one-way data import facility in LibreOffice. This would require a tiny fraction of the current code without any Java dependency at all.

No more database development in this office suite. No more office users overstrained by abstract data models. Simply connect, query and link already existing, well formed databases to document fields as it always used to work since the first version of OOo. This is how the majority of office users uses Base when they create serial letters without even noticing the .odb file on their disk (well, until they delete it ...).

Like in OOo 1.x, the "Base document" should be a simple registration entry with connection info and some SELECT strings because the "embedded database document" failed completely, drowning gigabytes of user data in binary swamps.

Andreas Säger

Am 29.07.2011 03:22, NoOp wrote:

Hit F4 in Writer or Calc, right-click>Open the dBase "Bibliography".
All the functionality of a flat (unrelational) dBase connection is
there. You can connect, query, edit data through forms and dump any row
set into office documents.

http://www.libreoffice.org/get-help/system-requirements/
<quote>
For certain features of the software - but not most - Java is required.
Java is notably required for Base.
</quote>

As a matter of fact you can disable/remove Java, hit F4 ... connect, query, edit and dump.
If I would convert our Java databases and remove Java from our systems nobody would notice the change.

Interesting. Thanks for the info/confirmation.

Hi All,

________________________________
From: Cor Nouws <oolst@nouenoff.nl>
To: users@global.libreoffice.org
Sent: Thu, 28 July, 2011 20:47:59
Subject: Re: [libreoffice-users] Re: Base record access unacceptably slow

Hi Tom,

Tom Cloyd wrote (28-07-11 18:06)

> I completely agree. LO without a function Base won't fly. Either TDF
> recognizes this - by DOING something - or we have our answer.

Well, the challenges with base are known. This does not mean that there is an
instant solution. However the fast growing support for TDF and from interested
people with developing skills, make that there are reasons for some optimism :slight_smile:
Maybe the LibreOffice conference or another hackers event is a good opportunity
too to make so first steps, share experience ...

Regards,
Cor

Hi :slight_smile:
If that was going to be enough then we would already have seen some efforts made
in Base in the last 11 months just as we have seen efforts and even good results
in all the other apps. Waiting and hoping is getting nowhere. It is also not a
good management strategy.

Regards from
Tom :slight_smile:

On the subject of databases, is there a good back end that could be used
and our code is only a front end for MySQL, SQLlite, etc. My idea is
most office users want a friendlier interface that hides many of the
details of database operation (it develops the actual query) and allows
relatively easy design, report generation, and data entry.

I have been playing around with MariaDB/MySQL with MySQL-Workbench as
the GUI. This is probably more advanced than most office users would
need but is has some good ideas for graphical database design.

.
tomcloyd wrote:

, On 07/28/2011 12:53 AM, Tom Cloyd wrote:

Ug. This is getting ugly really fast. I'm really not on home ground
here at all.

FWIW, here is a relatively painless way to install an older JRE along side
the default Ubuntu one.
( I'm using Ubuntu 10.10 64bit )

It definitely sped up my Base table browsing, but then I don't have any
binaries embedded in my tables. It has the bonus of not messing with the
existing JRE

Overview:

1. Download the JRE archive (approx. 20 mb)
2. Extract in /tmp
3. as root, copy the extracted directory to /usr/lib/jvm
4. set this JRE as the JRE of choice in LO (Tools>Options>Java)
5. Exit LO & restart

Instructions:

1.Download jdk-6u21-linux-i586.bin (for i386) or jdk-6u21-linux-x64.bin for
x86_64 from
http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html#jdk-6u21-b07-oth-JPR
JRE archive at ORACLE . Save to /tmp
2.run it using: "sh jdk-6u21-linux-i586.bin" . The JRE is now extracted to
/tmp/jre1.6.0_21/
3. copy to /usr/lib/jvm "sudo cp -a jre1.6.0_21/ /usr/lib/jvm"
4. Exit & restart office. In Tools>Options>Java, choose 1.6.0_21. Exit &
restart office
5. Load you Base file & compare the speed.

If you want to remove it simply set the JRE back to the old one in

Hi

.
tomcloyd wrote:
>
> , On 07/28/2011 12:53 AM, Tom Cloyd wrote:
>
> Ug. This is getting ugly really fast. I'm really not on home ground
> here at all.
>

FWIW, here is a relatively painless way to install an older JRE along side
the default Ubuntu one.
( I'm using Ubuntu 10.10 64bit )

It definitely sped up my Base table browsing, but then I don't have any
binaries embedded in my tables. It has the bonus of not messing with the
existing JRE

Overview:

1. Download the JRE archive (approx. 20 mb)
2. Extract in /tmp
3. as root, copy the extracted directory to /usr/lib/jvm
4. set this JRE as the JRE of choice in LO (Tools>Options>Java)
5. Exit LO & restart

Instructions:

1.Download jdk-6u21-linux-i586.bin (for i386) or jdk-6u21-linux-x64.bin for
x86_64 from
http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html#jdk-6u21-b07-oth-JPR
JRE archive at ORACLE . Save to /tmp
2.run it using: "sh jdk-6u21-linux-i586.bin" . The JRE is now extracted to
/tmp/jre1.6.0_21/
3. copy to /usr/lib/jvm "sudo cp -a jre1.6.0_21/ /usr/lib/jvm"
4. Exit & restart office. In Tools>Options>Java, choose 1.6.0_21. Exit &
restart office
5. Load you Base file & compare the speed.

If you want to remove it simply set the JRE back to the old one in
>>Java & "sudo rm -rf /usr/lib/jvm/jre1.6.0_21/

--
View this message in context: http://nabble.documentfoundation.org/Base-record-access-unacceptably-slow-tp3202820p3211034.html
Sent from the Users mailing list archive at Nabble.com.

Thank you for the step by step instructions, I am sure many will find
them useful

Hi

Thanks you for this tips and to all for Base and data bases tips !

Regards,

Jorge Rodríguez

I'll chime in - thanks indeed. I just removed my current sun-java, and installed

sun-java6-bin_6.22-0ubuntu1~10.04_i386.deb and sun-java6-jre_6.22-0ubuntu1~10.04_all.deb

Everything seems to run fine. However, the increase in Base/HYPERSQL speed is only about 25%. Disappointing. This still isn't a good solution.

I'll restore fully updated sun-java, then follow your instructions, as a temporary fix.

Meanwhile, I working on a solution is not dependent upon a graphical db at all, using a more primitive approach which I recently used for an in-memory entity-relationship db/DSL I wrote about a year ago: I write db output not to an interactive programmed GUI, but to a set of files which lie open in the jEdit programmers editor. As soon as an output file is updated (rewritten), jEdit reloads it, so it's available for me to peruse. I can have far more files open for ready viewing than I can screens.

If I need to write into something, I use the DSL to say what, and I get an open file labeled fields, into which I write, or edit, if I'm making a correction. Returning to the command line, I indicate the file is read to be read (since I just saved and closed it) and the program receives the input.

I'll grant you that this is primitive, but it works right well, is far more flexible, allows me to use my Ruby skills (such as they are), and frees me from dependance upon someone else's GUI. Why don't I write my own? I don't have that skill, nor the time to acquire it. I like "simple", not "shiny", and I just need information, not colors on the screen, which this surely gives me. And, it can be used with any db engine.

So...that's where I'm going with my db needs. I just cannot afford to get further invested in the sink-hole that Base is looking like, much as I really do like the interface, etc. It's too slow, I might be able to figure out how to have two subforms, but why not TWELVE? With my approach, no problem.

Something from my library of quotes:

"The most important rule in our development is always to do the simplest thing that could possibly work. ...Simplicity is the most important contributor to the ability to make rapid progress." < http:// www.xprogramming.com/Practices/PracSimplest.html >

t.