The Problem For Google
I think Google and the other search engines have a problem. I think the programs are dynamically generating too many web pages for Google to crawl and index.
Has anyone noticed all those news announcements about the new Google data centers. They are going to consume all the power of Niagra, or whatever the local dam is near where they are being built. They do not want power consumption figures to be in the public record. Does that bother anyone? Arn't computers supposed to be getting more efficient? Have you seen those graphs about growth of size on the internet. I think that all of our little tiny servers are going to overwhelm Google.
I do not think that there should be that much data on the internet. Not that many servers are needed. Just look at my site. Aside from the resume about 20Kb, I can define a person using about 100 bytes of data. Say there are 6 billion people registered. I can save all that data in 6*10**11 bytes of data. Excluding Resumes. 600 Gigabytes. I can do that on 1 server! Maybe the resumes are more. A quick scan of my resumes shows they average 20Kbytes, so the resume data is another 120*10**12 or 120 Terrabytes. Maybe each person has a few things to sell, and some photos. and some videos, so the size of the web could be several times that. Really not that big.
So why do they need all the servers? Because dynamically generated web pages are increasingly numerous, and are much much larger than the core data. Google has an exploding problem to index just my site, let alone all the others ones. . My 100 bytes of data becomes a web page of 24.55 KB.
It will soon get worse than that. Say I have each candidate draw a polygon where they are willing to work, then I can have 50 more job markets for the 50 states. Google would have to index 50 times more stuff. A little less than that because not every candidate will work anywhere. But worse yet, say I start putting up job boards for each city in California, think 1000 cities, 1000 time more web pages that Google would have to index.
Eventually there will just be too many pages for google to index. They could call me a web spammer, and disbar me, but really I am not, I am putting up legitimate pages. I am putting up many different views of the same data. I am just overwhelming their servers.
I think that is what is happening. Google may be selling themselves as having these huge assets call server farms. But historically I have always treated my computers as depreciating assets, with a short half-life. It sounded like a weird sales pitch to me. Invest in us, because we own lots of depreciating computers. Weird.
I think google has a problem.
