Do You Believe in a Flat-File Driven Content Management System?

As the developer of a flat-file CMS -- Big Medium at http://beta.bigmedium.com/ -- I'm admittedly biased, but I've found that file-based storage works quite well for sites of up to several thousand pages. This shouldn't be a huge surprise: OS file systems are, after all, mighty efficient for fetching and organizing data.

While it's true that flat-file systems make for mildly easier installation for tech neophytes and budget hosting companies, I think that a better reason for them is that they're more easily backed up than database contents. (Not that it's particularly *hard* to backup a database; it's just that too many hosting companies do this poorly.)

Just as it's possible to do a terrible data-model implementation with a database, it's also possible to do a sublime implementation with flat files. In any case, I'd argue that at least at the scale of small and medium sites, where you store data is less important than how well you manage and organize it.

Me, I'm not particularly dogmatic about any of this. Although Big Medium uses flat files, I'm certainly not *against* databases. For my market of small- to medium-sized sites, I simply found that Big Medium didn't need the overhead of a database and the performance implications/complexity of a SQL front end. My customers are satisfied.

Flat-file storage apparently works well for much larger data sets, too. Bloglines stores the majority of its data in flat files:
http://radar.oreilly.com/archives/2006/04/database_war_stories_2_bloglin.html

(Interesting comments on that page, too, which underscores the rather emotional relationship that many developers have with their favorite storage option.) I've read that outfits like the Internet Archive and Alexa also use flat files for data analysis and permanent storage.

As always, it's about using the right tool for the right job, and flat files often fit the bill.

Best,
Josh

Thanks for filling us in Josh - v. informative. I am somewhat 'db-phobic' tbh - probably like a lot of people out there, I don't really want to think too much about how my content is stored.
I'd rather just go with the established consensus, throw some guy a few extra bucks, and let him/her look after the whole thing. Which is not really a good attitude to have, I suppose.

Hi John,

And thank you for a very interesting article.

I am not sure which forums posts or emails of your were ignored. Could you please point where they are. In www.web-app.net there is usually no need to wait for more than 30 min to get an answer to any support question. However we did have some problems with our mail server because we moved to a new server recently.

When it comes to using of relational vs. flat db file systems. There are advantages and disadvantages in both. Its not for no reason that the Genome Project, (the largest human project since the first man on the moon) has started on the basis of flat db file systems (if my memory doe not betray me!), this is because flat db files are compatible with all other db file systems and that is a great advantage for converting and sharing information.

When it comes to speed or db processing, webapp has been actually shown in several tests to be faster than its php competitors, this is because it has been recently been optimized with cache and memorization "tricks".

Apart from that, show me one free CMS out there that has an automatic installshield... and I would eat both my socks and my hat.

Thanks for your attention!
Kind regards
On Elpeleg
WebAPP
http://www.web-app.net

ah yes... that whole 'ignored on forums' thing was just poetic license, to add to the 'Outlaw' aura of WebApp...

But I did send off a couple of emails that were never responded to -- but as you say you've had an email server problem.

Ah hell, now I've just gone and ruined the WebApp aura. Darn it. That was not the plan friends.

It was a very good aura, because if it wasn't we would never have added a link to this great site here: http://www.cmswire.com

And my last 2 cents, I really hope you could try installing it, I guarantee you the fastest install experience you ever had in your life with our installshield, have you heard of any CMS that provides an auto install shield? I doubt it :)

Thank you

Kind regards
On Elpeleg
WebAPP
http://www.web-app.ne

I think I will, because I'm interested in buying a few domains and just 'parking' some content on them for a while, letting them mature in the face of Google (I believe Google likes mature domains) and then going to work on them at some unspecified future date (or just selling them).
Flat-file would allow me to manage the content and build the site in a convenient fashion, and would be a cheap way of doing things, I think. And would give me a great opportunity to check out the technology.

I do suggest taking a long hard look at the forums before trying this set of scripts. It's pretty obvious they have a huge axe to grind with a former developer, and it leads them to do some pretty weird things:

1 - Lots of links back to the main site for parts of the script like the help text files. (why?? Are they tracking the installs?)

2 - Some questionable banner ads which are displayed by default - including one with a gun pointed at what I can only guess is a former developer. Yeikes!

3 - Lots of spam for their hosting provider you have to find out how to turn off.

4 - Two default banner ads for their site that already had over 100 "views" on a newly installed site, no way to edit the number of views, and no explanations as to why they are like that.

5 - Google Adsense spammed all over the site, defaulted to "on", and preloaded with someones adsense account - again - no explanation. When you try to remove the adsense code - it comes back.

6 - The site comes with an IP address already banned, but with no explanation as to why or who it belonged to (looked like it resolved to a comcast customer).

7 - A spam e-mail already configured in the default install for their hosting company.

And this is just scratching the surface. If they had put their war with the old developer behind them and put their resources into the code, this would be a nice CMS system. But there is just too much bad blood in here to get past it. It's like trying to find the good in a certain 1940's German dictators regime. Very unprofessional, but you do get warned by their motto of "Our world, Our way".

No thanks - don't like that world. If I could have the app without the attitude, I would. You should probably do what the other developer did and bail on these guys.

...ouch!!

Hi again,

I see that Ms. Jos Brown, a former and an expelled “developer” of the WebAPP team bothered also to stop by here (as she done in dozens other sites that write about WebAPP, spaming them with lies and links to her sites).

1.) Firstly and most important to mention, the default banned IP on the banned txt file has been reserved for your "reasonable" poster above, whom happened to be a well known criminal from Washington, with heavy drugs and sex addictions as a cause. I would save you from the tragic details but at the same time would summarize that spaming, manipulation and code theft are only few of the virtues this self claimed “developer” has been contributing with. And this is why we have expelled and banned that person. A fast search on her name would provide you some idea of what you are dealing with.

2.) Our google module has been developed in cooperation with Google adsense and has been approved by Google, I would be more than happy to provide a copy of the emails exchanged with google. Many of our users earns money from those ads, while WebAPP does not get even a flat cent. The google ads can easily be disabled from the site admin control panel. Most of our users have chosen WebAPP because of the google module, as they tend to cover their hosting with google ads. To use the module they simply need to enter the control panel and insert their account number, information concerning that is provided upon install completion and in the control panel under ads.

3.) "Many links"?, I assume that the drugs consumption must have had its effect on the poor poster above, as this is a complete nonsense, since when providing a help link in an intern site admin control panel is considered as spam? I assume that all software’s developers should hide their help links? We at WebAPP are dealing with open source and free script support ad development and not heroine selling, and thereby see no reason to hide our help link

4.) Banners can easily be edited, enabled and disabled from the easily accessed site admin, I haven’t got a clue what count editing the poster from above is concern about.

5.) Link to our webhost, dreamhost? Is that a problem? The WebAPP is a free and an open source script, many of the users that install the script chooses to use the same serer as were the project is hosted, when doing so, 5 usd is deuced from our hosting expenses. This horrible crime of telling the users about it (yes also telling them how much we deuce from our hosting expenses) is actually a direct contribution to the WebAPP project and keeps it alive.
The users who install the WebAPP script can and do replace this ad example easily from the admin site (in 99.99%) of the WebAPP sites installing the script. So I guess we can forgive them and you can forgive us for doing this horrible crime.

I am very sorry for that your fine article attracted this soap opera.

Kind regards
On Elpeleg
WebAPP
http://www.web-app.net

Thats a funny comment a couple up...

Its obviously actually posted by this specific now defunct ex-developer we booted out of the team for theft and spamming trying to start some fight with the package...

And the funny thing... the change of the motto to "Our World, Our Way" was the idea of this ex-developer to begin with...
The old "Your World Your Way" may be the right way to go... but its funny how some so-called "tester" can find all this attitude within a script they obviously never took the time to ask questions about any issue they found... I am sure with all open source scripts there are preset things in there... if I was worried about something in there then I would go to the scripts main site and forums and enquire... something this ex-developer didnt do when testing our product... well its nice to see these comments nonetheless... its obvious that no matter how idiot proof a script is... there will always be a new idiot developed for us to dumb the world down to...

In the end, WebAPP Is a Great Script developed by great people, oh and I am one of them if you didnt notice by now...

Come on by, grab the script... install the script... got questions? Come to http://www.web-app.net and ASK your questions there so we can answer them....

Its funny how every negative comment I have seen on the internet so far is so full of crap when it comes to the issues of the script because people are too lazy to either read whats right in front of their eyes, or come on by and ask us about something they dont understand.

I agree with Jack.

I see no complaints whatever in our support forums related to any of the above trashing points from Jos Brown (hiding under the nick: Joshua), on the contrary we have many positive feedback concerning V. 1.0, and the best evidence for that is the fact that WebAPP has never had so many of the original developers making a come back, including the original developer of WebAPP Carter, (whom resigned several years ago), we have never had so many new sites and new codes and ideas contributors as now!

My conclusion: Jos Brown is simply upset because we have banned her and is having a real tough time dealing with this rejection. She finds it difficult to realize and accept that after she was expelled, WebAPP has reached a new top popularity level. I am sure that it will take her some time to realize things. But I do not think that going around writing this sort of trash would do any good for her and her personal issues, for this she can address professional help.

On

As a webapp user, I can easily confirm that all the stuff 'joshua' was ranting about is easily solved in the admin controls. I installed v.1 last week, and the very first thing I did was turn off adsense, banners and advertising emails. If I wanted, I could insert my own banners to replace the stock ones.

I have found webapp to be an excellent CMS, with easy controls and very good support from the development team. In fact, from my point of view, the accessability of the dev team is a BIG selling point - almost instant support.
As an example, I asked if a feature was possible, and within a day and a half I have a patch to install. Great support.

Mr Conroy, download it, give it a go, and don't exaggerate the email problems.

Hi, about the mentioned banner ad with the gun pointing at the cat. It states, "Ted must be stopped". Well I'm Ted and it's all in good clean fun. The WebAPP team are good guys and passionite about their hobby and interests. It's what makes a great project such as WebAPP.

On another note, the trials and tribulations of WebAPP over the years are the makings of a great story. There are many like it in the open source and other markets. They go untold and mostly hidden from the public eye but contain the suspense and intrigue of a captivating novel. I'm glad to be part of it.

Great article, BUT, I installed 1.0 from web-app.net and in a matter of one week someone gained root access bringing my site down and causing havoc on my server. The hacker was able to put up a bunch of porn on my front page.

About those Google ads, I couldn't get those off my site. I have no "war" with any of these people, I'm just saying what happened with my site and server. It took a week to get the damage corrected because of this CMS.

Carrie, please go trollnig somewhere else.

Security: WebAPP has over 6200 users with not even one report of such vulnerability. They are having securityfocuse and others testing the code. There is NOT even one security issue reported since WebAPP v.1.0 SE has been released.

Gogle ads: if you do not know how to click a form box under enable/disable ads then you are either a troll, dumb or both.

Master of trolls

Now we all see the power on the world wide web where any idiot can post anything for people to read and get away with it. Hence, the "war" continues. Carrie is a liar and, if you know the truth, a bad one at that. WebAPP v1.0 has been tested and re-tested by experts and known associations the deal with security vulnerabilities and is verified to be very secure. Why would Carrie post such an ugly lie is just another example about how intense this war is. WebAPP V1.0 is far above and beyond the scope of the more popular CMSes that it drives people crazy and it's free.

I like WebAPP and I think that those idiots/trolls are just jealous.

I see that one of them opened a site now where the backbone of the entire script was created by Andy (5 years ago), the whole front page was made by Ted frot page mod (3 years ago), the CSS by another person, the map locator is a plugin from Steve (SafePit), security patched for issues discovered by On and Monty53, though it does not prevent this half cent/ half lunatic pirate to claim a new license and copyrights for it... There is no limit for what those mad lunatics may arrive at, ignore them and keep up the good work! Lies have no feet and lunatics get what they deserve sooner or later.

The new license for WebAPP 1.0 at our site at web-app.net is no longer Open Source. It's commercial license now so all you spammers and trolls with the org site need to keep up with the times.

WebAPP 1.0 SE is Free, Open Source GPL, GNU. If you need glasses go see an optician. If you want to become a troll try harder.

The real Yaza

Hi,

I appreciate your view on flat file systems, and to a degree agree that they are not for full blown all singing all dancing websites.

However they do exist and are completely stable. Anyone touting instability as a reason not to use one is talking rubbish. I have had just as much trouble working with database types as with flatfiles.

I myself have wrote a flat file system called razorcms, you can find it at www.razorcms.co.uk.

It is open source under gplv3 and is for small to medium sites wanting a front for the project. It boasts a small footprint and is pretty darn quick too.

all you have to remember is that everything has it's place, and flat file systems have a place in the small to medium size market.

smiffy6969

Hi Smiffy,

I like razorcms, its clean, easy and gives a good overview of all functions. Is it also free, open source licensed?

Otherwise I think that the author of the article we referring to the old generation of flat cms portals, which is often did have problems with db files instability. Also non flat systems have had similar problems. It is a matter of fact that any CMS out there regardless of its language is still vulnerable to such issues under extraordinary conditions, eg. denial of services and diverse injections (SQL is more vulnerable to this one).

WebAPP deals with these issues with automated cache, backup and restore. It even has a system restore just like windows xp/vista! In addition it has automated routine and blocking of d.o.s and all known remote code injections.

Good luck with your CMS project!

On

Hi,

Thanks for your comments, yes it's free and offered under GPL3.

I know what your saying about cms solutions in general and any website security, it's like this, a house has a door, to help you get access, it doesn't matter how good the door, it's still a door and can be circumvented. Just comes down to how desperately they want to get through it.

Yes I agree webapp may have a better door than most, but it's still a door.

The only way to have your website free from all attacks is to have it running on a free standing disconnected local server, just like the only true protection for a desktop is to remove the network cable.

So you have to compromise, and part of that is designing a better door, to the point where the time and effort taken to get in, far outweighs the contents in the house. If they can't be bothered to attack you for this reason then your secure, until you buy that nice new plasma, then the stakes have gone up.

I am willing to bet webapp is vulnerable to some degree, for the reasons I said above, they just have a really good door, most php apps only have a barn door unfortunately.

I have the locksmiths in at the mo fitting new locks on razorCMS, lets hope it does the job.

Again thanks for your comments, i'm trying, lets hope I am on to something. RC2 should be out in a week or so, lots of improvements, bug fixes and add-ons all coming.

smiffy6969

Much of the above misses one main point. All CMS systems rely on manipulating small fragments of HTML. Those fragments get stuffed into display areas, surrounded by boilerplate page layout. The one oft-recurring and annoying problem with HTML fragments are links buried inside the fragments. Those links often become obsolete and need to be updated. Perhaps an images directory has been moved, so suddenly hundreds of links buried inside fragment files are broken.

Finding those links and updating them in a database driven system is difficult.
Finding those broken links and fixing them in a files-based system is easy.

I looked at razor cms. It seems to be storing the content files all in a single directory, named datastore, with numberical file names. That sounds a lot like trying to turn files in a relational system.

I use hierarchical directories for storage. Directories that represent leaf level pages have a special prefix.

datastructures/arrays/_-hashes would be a directory that stores the files-based content bits for a page named "hashes" the _-hashes directory would contain one or more files, whose file names match the div id they get stuffed into by the layout engine.

So the layout is css positined divs, where each div gets mapped to file of the same name, or (depending on the current layout scheme) a named function.

...each page assumes a default layout scheme, which can be changed by an incoming GET parameter. The layout name maps to an XML file that maps each div block either to a fragment file or a function call at runtime. Storing fragments in a hierarchical file system that relies on meaningful names (something like _-tarpon ......rather than 32.data) makes maintenance easier.

Complex sites with shopping carts and customer management do need an rdbms in the background. But not for page generation. Dynamic but files-based systems are oh so much easier to hack quickly together, and to maintain (perl/sed/awk instead of complex sql)