Adding fields to existing Base table

I'm setting up a database that is small (three tables, may grow to 3GB
over next year). I need a strategy to deal with the unknown, which is
how to add fields to an existing table. I read in the docs that doing
this can be painful and it is required to put something in each field
for each record. Do I have this correct? If so, how do I handle the
inevitable "...would you add..." that is sure to come within the next
few months? I would add them now, but I really can't anticipate how
many fields will be added or their requirements.

In general, do other databases have similar restrictions? At some time,
when I have time, I will consider MySQL, MariaDB, and others.

Thanks,
Dave

Dave,
I use the MySQL server and have added table fields before. It is simple in the LO Base table design window.
Tables -> right click on table -> select Edit
MySQL accepts these changes and adds the field. Of course, then you must add the new field to your forms, queries, relationships and reports, as needed.

Deleting fields is another matter. I have had problems with MySQL field deletions before - especially messing with the primary key. Then it may be a matter of directly using the MySQL interface program, with some SQL code, bypassing LO Base. But it is possible. So be careful what you do. Think it out first.

I have no experience with the internal HSQLDB server. From what I read on this forum, using the internal HSQLDB server with a large 3GB database is not recommended. Best to use MySQL, MariaDB, or Postgresql for such large databases. Also, remember that there may be no way to easily port your data from the internal HSQLDB database to an external database if you so decide to make the change in the future. You may end up re-entering it all. Best to make the commitment now, before you run into a wall. Note that I consider my databases "large" at 2.7_M_B for the current largest. IMHO, 3GB is out there.

HTH.
Girvin Herr

Hi :slight_smile:
1st thing is that Base is best when used 'just' as a front-end
and with all the data in some back-end such as MySql/MariaDb,
Postgresql, HsqlDb or whatever.

Generally i think it should be possible to fill an existing
field, in all rows, by using an action Query. However, i am
really surprised to hear that a new field can't have
zero-length or be completely empty.
Which docs have you been reading?

From Getting Started, 4.0:

Once tables have been created using the wizard, and data has
been entered, editing them should be very limited. Fields can
be added or deleted, but adding a field requires you to enter
the data for that one field for every existing record with an
entry for that field. Deleting a field deletes all the data
once contained in that field. Changing the field type of a
field can lead to data being lost either partially or
completely.

When creating a new table, it pays to create the fields with
the correct names, length, and format before you add any data.
Deleting a table removes all of the data contained in every
field of the table. Unless
you are sure, do not delete a table

Chances are that they know tons more than me but jic it is FUD
it'd be nice to know. The Base Guides are at
[1]https://wiki.documentfoundation.org/Documentation/Publicatio
ns#LibreOffice_Base_Handbook

I hope someone with more experience of Base lets us know soon!
:wink:
Regards from
Tom :slight_smile:

Hi Dave,

Having worked as a business systems designer and coder, I've dealt with
databases and database changes a lot. And the users always end up
changing stuff later. Some of what I am going to say may be obvious or
stuff you already know; if so I apologise, but your question makes it
seem like you are unfamiliar with databases, so I will try to cover the
basics.

The short answer is that this sort of thing is done all the time,
applies to all databases, and is pretty easy stuff, if somewhat
timeconsuming for the client, unless there are complicating factors.

At least, pretty easy just from the database side of things, I'm not
sure if LO makes it harder than it should be, as I haven't used LO Base
much. I'm mainly talking about the underlying principles here, which
your post suggested to me that you don't fully understand yet (again,
sorry if I got that wrong).

The long answer...

First off, if your database may grow to 3 Gb, it suggests that this may
be a serious application. A rough guesstimate suggests maybe a million
rows, and that sort of growth in a year means about 8 rows a minute
added (very rough guesses, but just to get an idea of the scale we
are talking about). I would think carefully now, rather than investigate
at some later date. As often happens, you could end up with the client
deciding that it's not worth the cost of making a change, and whatever
poor choices you made up front "just for now" end up sticking with you
and being a major headache when things get beyond their capabilities.

Straight away I question using LO at all. A custom front-end often has
many advantages in this sort of situation, making them worth the cost.
But that's the sort of thing I used to do for a living, so I guess
that's my instinctive response, not to say LO isn't a good fit for
this; I don't know the requirements, so I can't say.

But at the very least, I would *strongly* suggest using an external
database, not the built-in HSQLDB. ***With a good backup solution***.
And consider where it will be hosted. If the database is that big, you
need to consider questions like how many people will be using it at the
same time, and how many transactions will be occuring per second. The
hardware used for the database server makes a big difference to the
responsiveness of these sorts of systems.

As to adding fields, you just got to live with that one. Happens all
the time. No amount of getting the client to think carefully about his
design is going to prevent them coming later and wanting stuff changed.
So says my experience.

Luckily it's not hard to do, per se. The problem is of course that if
you add fields, you are adding them to the structure of the table,
which determines what will be in each row, so it affects not only rows
that will be added but also all already existing rows. And therefore
each existing row needs to have something (even if the something is
nothing) in that new field. That will always be a problem. Luckily,
mostly a problem for the client, who now has to go back and add all that
missing info for the existing rows, to make sure that the data in the
database makes sense.

So when you say that the docs said it might be painful, that's not so
much from the practical point of actually adding the fields, but from
the point of making the existing rows make sense with the new
information. So yes, it applies for all databases. They all make
actually adding the new field pretty much just as easy, but the
reconciling the existing rows has nothing to do with the database and
everything to do with the data itself.

When adding new fields, as with adding fields at the beginning when
designing the table, you can choose both a default value for a field,
and to allow the field to be null. Nulls are misused and abused a lot,
and different people have different ideas about how to use them. It's a
whole other topic to explain how to use nulls properly, but suffice to
say that when you add a field to a table, you can allow the field to be
null, and then all existing rows in the table will simply have null for
that field. Or, you could choose a default value for the field, and
then all existing rows in the table will have the default value for
that new field. That value could be a blank value, like an empty
string, which is different to a null value.

So for simple additions of information, the developer's job is pretty
straightforward. Yes, you have to add that field to the table, and then
to all forms and reports, but it is the client who has to make sure all
the new and existing entries have valid information in that new field.
The problem for you as a developer comes in when they want additional
functionality tied in to that field. Given that you're developing this
in LO, I'm guessing that there won't be too much of this, it will
mostly be simple informational fields. But some of the stuff I've done
in the past has required things like "when they fill in this field,
they must get this custom screen to choose a value from, and then, if
they choose "A", they must get this custom icon on the report, and if
they choose "B", then this mustn't show up on the report at all, and
instead must make one of the other rows show up differently". Ok, not
actually anything like that, but you get the picture. That's some
serious developer headache there.

As to deleting fields, it's also pretty easy. You just drop the field,
and the field is gone from all existing rows. Which means loss of some
data. Which presumably is what you want. Provided, of course, there are
no other considerations, like other tables linking to this table via
the field. Then it gets a little trickier, and really depends on what
you want to do.

Sorry if this post was rather long, but I hope it showed you the basic
factors you need to consider with the database design.

Paul

All,

I want to thank all of you for your very helpful responses. Yes, it has
bee a long time since I have used databases. Last was about 10 years
ago when I used Access with SQL for a real time data acquisition system.
That was a challenge!

What I'm doing now is creating a database of data that changes about
twice per week, but the number of areas (of the US) covered will grow
from 12 now, to over 100 if the project gets the go-ahead. I likely
will be stuck adding the data as I am developing a program to acquire
the raw data automatically. I will also create reports that get sent to
a few individuals (for now), and more later if the project is go. There
has been some talk about adding a web site that is interactive like some
stock web sites or the really nice Weather Underground site, but that is
a bridge to cross in 2015.

My take-away for the discussion is that LO Base will not take me to
where I need to be, and I need to make some decisions ASAP. I did a lot
of Googling this afternoon, and some reading. The plan based on your
input is:
1. Use LO Base to do the quick demo since the data is currently in a
spreadsheet I don't need to worry about loosing it altogether. The
reason for the spreadshet is that eraly on I have no idea what the data
is going to look like, and spreadsheets are easier to creat and modify.
2. Move to an industrial strength database engine. After some fast
research, I'm leaning toward MariaDB. It doesn't appear to have a GUI
client for input forms or a report generator, so there will be a lot of
work here. JasperRreports and Next Reports seem to be good for the
output.
3. Plan on spending a lot more time than orriginally budgeted for this
porject - but that seems to always be the case with developemnt.

Thoughts or comments??

Thanks,
Dave

Hi :slight_smile:
Errr, actually, using your spreadsheets as the back-end might be a good way
of doing it. At least while you are still exploring and planning. Once
proper data starts coming in it's going to need a much heftier back-end.

One of the advantages of Base is that it can be swapped to reading a
different back-end without having to redesign everything all over again.
So fleshing things out in a spreadsheet and then flipping over to using a
proper database back-end sounds like it should be fairly easy.

Another of the advantages of using Base is that it's then fairly easy
(allegedly) to use Writer (or Calc) to generate the Reports and maybe the
Forms too. I've heard that it's not easy to use them to put data into the
database nor for editing corrections.

It sounds like those 2 advantages might not be enough. Also what "should
be" and what "is" often seem to be poles apart in life.
Regards from
Tom :slight_smile:

Hi :slight_smile:
1st thing is that Base is best when used 'just' as a front-end and with all
the data in some back-end such as MySql/MariaDb, Postgresql, HsqlDb or
whatever.

Generally i think it should be possible to fill an existing field, in all
rows, by using an action Query. However, i am really surprised to hear
that a new field can't have zero-length or be completely empty.

Which docs have you been reading? Chances are that they know tons more
than me but jic it is FUD it'd be nice to know. The Base Guides are at
https://wiki.documentfoundation.org/Documentation/Publications#LibreOffice_Base_Handbook

I hope someone with more experience of Base lets us know soon! :wink:
Regards from
Tom :slight_smile:

Hi Dave,

You might also like to have a look at the H2 Database Engine, see -
http://www.h2database.com/html/main.html

I have found that it is a good stable replacement for HSQLDB and have used
it in 6 different databases with good success. There is an active users
group, and some of them report success with very large databases.

Noel

Thank you Noel. I will check this out. My head is spinning right now
with all of the choices. I need to get something together for next
Fri., so I'm going with LO Base for now. Seems like a decent choice
because I will be the only user for most of the rest of the year. Later
this year there may be two other users. So the convenience of LO Base
makes sense, and the data size for the rest of the year is not likely to
excede 200 MB. The biggest limitation with LO Base is the report
spftware - doesn't seem to support embedding charts in the report.
Hopefully this problem gets fixed in 4.3. Alternatively, I'll use a
stand-alone report software product. Hey, did I mention that this was
SUPPOSED to be a quick simple project? Aren't they all??

My hope is that I can use LO Base as the front end for a more robust
RDMS later this year. I have been looking at MariaDB because it seems
to have a lot of buz, but may not be the best chioce. One of the things
that really needs to happen is that I have to get my arms around this
project in terms of potential size, number of users, on the web or not,
ect. My guess is that I won't have that done until very late this year.

Dave,

Hi Dave

My penny's worth - I battled with plain Base for about 3 years. Horrendous crashes and freezes and the amount of data I "lost" and had to re-enter was unbelievable!!! (I'm a single user, single machine running with Linux as my OpSys). Every time Base goes down it takes the entire Database with it!!!!! So the only way round it is to make copies every time you add data so that when it crashes you can rename the copy and get going again. You WILL lose data I promise you!!

I did everything I could but no luck. It then became obvious to me that the problem was the embedded DB so I took the plunge and moved to MySQL last year. I've had NO crashes or lost data since then. It's as solid as a rock!!

My recommendation would be to START right now with MySQL (or other backend). Base as a front end is great as long as the backend is solid!!

Just my opinion for what it's worth but I speak from hard experience!!!

Best wishes

IanW
Pretoria RSA.

Hi :slight_smile:
Actually HsqlDb from their own site is usually a huge improvement over the
internal back-end version and is apparently really good too. However i
think HsqlDb aims to be for small, light-weight but extremely fast
databases so it probably wouldn't suit this particular use-case. I get
the impression that H2 is also a small, fast one but could easily be
wrong.

We often mention the heavy-duty back-ends such as Postgresql and
MySql/MariaDb but rarely hear about the faster, lighter-weight ones and how
they compare against each other.

In general i think it is good to be reminded that there are other choices
out there and what some of them are.
Many thanks and regards from
Tom :slight_smile:

IanW,

I'm concerned with Base, but I have some things to put together for next
Fri. which will be the data in some report. No big deal - just a
starting point. Since time is tight, do you think I would be smarter to
continue with Base/HSQL, or go to Base as the front tend and MySQL or
MariaDB as the back end now? For this week, my data requirements are
only 150KB, two tables, four queries, 1 report (my estimates as I type).

MySQL has some graphic front-ends for creation and admin., but I don't
think MariaDB has this (yet), though there are some third party software
that may do the same thing. So while MariaDB has the buzz, I may be
better off with MySQL and linve with Oracle - at least for now.

For desktop use, there is also Kexi. Ever use it and any comments? I
would consider this as an interim step that will tide me over for at
least a few months.

My big concern is time. Learning MySQL/MariaDB is not something I'll do
in a day, so this may have to be a longer term project.

Dave,

Hi Tom,

In relation to the H2 database you said -

I get the impression that H2 is also a small, fast one but could easily

be wrong.

All of my H2 databases would be classified as small, but a couple of years
ago an H2 user posted on the Users Group that inserts to his data base
slowed down significantly after it got to the 1 billion record mark! See -
https://groups.google.com/forum/?hl=en#!searchin/h2-database/billion/h2-database/tB6QP8MeDj8/6tyTZ4s5kf0J

​You will notice from the thread that he got several responses on how to
overcome his problem​ and these enabled him to improve the response he was
getting.

So I think H2 should be included in the list of heavy duty databases. But
I must admit that it will probably be quite a while before I need a table
with 1 billion rows in it!

Noel

Hi :slight_smile:
Thanks! :slight_smile: I think it's definitely good to learn more about what back-ends
are available and what they are each best for. Also good when FUD or just
misunderstandings can be cleared up. Thanks for that :slight_smile:

Regards from
Tom :slight_smile: