Hi all there!

I was reading this discussion with attention since we update from 9.72 to 10.3.5 an then 10.3.6 we start to experimenting this issue, almost every single day an in the worst cases, we lost o zero'd ALL the .dat files.

Actually I'm to dissagree with Justin, Martin and all people says that it's not a E-Blah problem. But before start a polemic and unuseful discussion let me say that personally I have a lot to thank to E-Blah software and of course to Justin and all colaborators of this team.

I'm going to explain what is my idea after moths to have to "fight" with this problem. I need to reconstruct the user list every shor time, some days 3 o 4 times.

All info showed bellow was taken at the moment I write this:

Site: http://foros.pesca.org.mx/


Server specs:
==========
CPU info:
[root@124780-web1 Members]# cat /proc/cpuinfo
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 67
model name      : Dual-Core AMD Opteron(tm) Processor 1212
stepping        : 3
cpu MHz         : 2010.947
cache size      : 1024 KB

Hard disk stats:
[root@124780-web1 Members]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             227G   61G  155G  29% /
/dev/sda1              99M   12M   82M  13% /boot
none                 1014M     0 1014M   0% /dev/shm
/dev/sda2             2.0G  161M  1.8G   9% /tmp

This server host to others small sites, but its is almost 90% dedicated to our forum. And as you can see, the server has plenty disk space not logically divided or shared. So the argument of few space do not explain the problem, at least not in our case.

OS:
[root@124780-web1 Members]# uname -a
Linux 124780-web1.bir.com.mx 2.6.9-55.ELsmp #1 SMP Fri Apr 20 17:03:35 EDT 2007 i686 athlon i386 GNU/Linux
(Red Hat Entrerprise Ed.)

Until two weeks ago, we install an aditional GB of RAM:

[root@124780-web1 Members]# cat /proc/meminfo
MemTotal:      2074780 kB
MemFree:         81712 kB
Buffers:        121752 kB
Cached:        1632724 kB
SwapCached:          4 kB
Active:         855152 kB
Inactive:       967108 kB
HighTotal:     1179456 kB
HighFree:         2944 kB
LowTotal:       895324 kB
LowFree:         78768 kB
SwapTotal:     1052248 kB
SwapFree:      1052044 kB
Dirty:            3064 kB
Writeback:           0 kB
Mapped:          87780 kB
Slab:           154072 kB
CommitLimit:   2089636 kB
Committed_AS:   634916 kB
PageTables:       4468 kB
VmallocTotal:   106488 kB
VmallocUsed:      1884 kB
VmallocChunk:   104552 kB

Well, all starts when we (actually Justin do it) update the forum. Since the differences between the user databases structures of the diferents version, we "lost" about a 30% of the users. I say "lost" because there was no erased o zoroe'd but their info could not be changed or updated, so now we see a lots of "guest" signed messages. (even with my own original user)

Now, at the beginnig of this month start to experimented the problem becuase the amount of traffic generated by the users and in particular when we RAN WITH LOW MEMORY, so we limit the access only for 250 users, with out no success, then limit again to 200, again with no success. Since the server is monitored by the provider, RackSpace for your information, we received lot of reports and the recommended us a RAM expansion:



So we expand the RAM from 1 GB to 2 GB, the problem was removed but only for 2 or 3 day when the user can login again because we retired the limitation. Look at this top comand output:

top - 15:06:14 up 7 days, 20:25,  1 user,  load average: 0.19, 0.28, 0.29
Tasks: 123 total,   2 running, 120 sleeping,   0 stopped,   1 zombie
Cpu(s):  1.7% us,  0.3% sy,  0.0% ni, 98.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   2074780k total,  2011732k used,    63048k free,   125452k buffers
Swap:  1052248k total,      204k used,  1052044k free,  1649304k cached

Considere this output was taken today, saturday when we have low activity because all members went to fish!

If you visit our portal at any day of the in-week, you can see the speed with the users post his messages: it shows 100 replays. In a very short time, the portal is renewed, at least the replays section.

So, instead a problem with the disk space, now belive that is more RAM related and the RAM related problems many times are cuased by the software. Also I think the lock management is a little deficient.

In particular, I understad Perl in a little so revising the code, I noted that its approach to read and write to the files in some places are:

open A, $file;
  @CONTAINER = <A>;
close A;

foreach (@CONTAINER) {
  ...
  ...
}

Althoug in the newer versions the last aproach was changed to:

open A, $file;
while (<A>) {
...
...
}
close A;

This change was applied only in some places but not in others so, as natural, since the first method is a very "hungry" RAM consumer, very qickly, when you have many users reading and writing at the same time, is very easy that the List2.txt file becomes "corrupted" (In some cases, it becomes binary instead ascii).

A similar ocurrs with .DAT files, even in rare cases, the other files, like PM and other. Only once we experiment the loss of one large discusion): To explain an empty .DAT file: If in the moment when the file is open and the server becomes low or not memory available or goes with no resourses, the file was closed with no data or even the pointer or the variable that holds the handler is missing then the .DAT file is also lost.

The final "mistery" that I can not imagine out a reason, is that we have three particular user whom very often lost is .DAT files and some times, the lost ALL his files. One in particular, named "JOSERRA" has lost his user data for a least 7 times in two months, of this, 2 times lost all files.

As I told you all, is not my intention to create a debate, I belive E-Blah is a very good software and its worth to revise it and make it even better.

What do you think?

h yamasaki

Unfortuantely I cannot find the answer on this thread. My hosting company migrated servers and does not have a back up for me. The only .dat file erased is the admin (figures)... so I am hoping someone can tell me how to recreate the text for the .dat file so I can simply edit the blank file and allow myself to log in again (if it is that simple?).. other wise.. please let me know what to do?

Heart?
Heartinternet?
dang!
I'm With Them, But I Recommend Having 1/2 Forum(s)
Cause The Free Starter Pro Is Way 2 Small!

Thnks,
TMFK

I've seen this on 10.5 and 10.6 E-Blah forums.  I just back up regularly and restore the affected .dat for the user account.  Sometimes, I make a spot check to be proactive.  It's kind of a hassle, but it's one that can be managed.

As always, add 98 cents to this, and you may (or may not) have a buck.

All the best,

Jimbo  

Quoted from hachiman

What do you think?

h yamasaki

Before I would say anything, I would say you need to figure out what's eating up all your ram.  If it's Perl, then that would probably mean it is E-Blah but that's because there's just too much activity on the forum to be using flat text files.  You should move to a forum that uses a SQL backend as it would greatly improve things.

If you server is hard rebooted or if processes on your server are terminated either by Apache or by an outside source (with kill) then that can also explain data loss.

While E-Blah is great for small to medium forums -- and even those that are huge, but aren't overly loaded -- there's no way it can handle a massive load like you are explaining for a long time because of it's use of flat text files.  In those cases, you need to actually spend a little bit of money and get a forum that uses a SQL backend.

Hi all there, Justin,

Thanks for your comments.

After I posted in this thread, (and after sevaral months with more RAM) our forum grows a lot, and still working with out the problem, just a very ocassional issues caused by Apache intentional restarts).



This reinforce my idea that the problem is not caused. mainly, for a low hardisk free space , but by a high memory ammount use. Any way nothing is perfect, but Blah is close to be.

Now our goal, before clean up the forum is reach a 1 milion posts.

Receive my warm congratulations for this magnificent software!!

hector

I had the problem of members dispear in my forum when i was with heartinternet . but after moving to my current hosting ,i have no problem . so i am sure its nothing with eblah . it's problem of hosting companies . what justin  saying is 100 right ))

As much as Nat says it's not heart internet's problem, when ever a user mentions any files becoming zero byte, corrupt or dissappearing users, heart internet is always mentioned... Just an observation...  

I am sorry about qoting heart hosting , its only caz i was with them during problems,

Quoted from iCONICA

As much as Nat says it's not heart internet's problem, when ever a user mentions any files becoming zero byte, corrupt or dissappearing users, heart internet is always mentioned... Just an observation...  

Quoted from Ronymon

I am sorry about qoting heart hosting , its only caz i was with them during problems,

No need for apologies, I had endless problems with them too.

Quoted from hachiman

Hi all there, Justin,

Thanks for your comments.

After I posted in this thread, (and after sevaral months with more RAM) our forum grows a lot, and still working with out the problem, just a very ocassional issues caused by Apache intentional restarts).



This reinforce my idea that the problem is not caused. mainly, for a low hardisk free space , but by a high memory ammount use. Any way nothing is perfect, but Blah is close to be.

Now our goal, before clean up the forum is reach a 1 milion posts.

Receive my warm congratulations for this magnificent software!!

hector

Make sure click logging is either turned off or changed to only log the last 10 minutes or so.  Having a large number can significantly slow the forum.  Also, on large boards, you may want to split them into two boards: an archive board and a main board.  That will also significantly lessen the burden on the server.  Again, this is just a limitation of flat-text databases.  Database servers are dedicated to handing lots of data and serving it up quickly.  Unfortunately, flat text isn't if it's very large.