Joke of the DayGoogle最早期的論文三

Joke of the DayGoogle最早期的論文三

5 Results and Performance
Query: cl clinton
http://v/
100.00% (no ddined) (0K)
http://v/
OfficeofthePresident
99.67% (Dec 23 1996) (2K)
http://v/WH/EOP/OP/html/OP_Home.h tml
Welcome ToTheWhite House
99.98% (Nov 09 1997) (5K)
http://v/WH/Welcome.html
Send Electronic Mail tothePresident
99.86% (Jul 14 1997) (5K)
http://v/WH/Mail/html/Mail_Presid ent.html
mailto:president@
99.98%
mailtoresident@
99.27%
The”Unofficiing” Bill Clinton
94.06% (Nov 11 1997) (14K)
un/un-b . c ..html
Bill Clinton MeetsTheShrinks
86.27% (Jun 29 1997) (63K)
un/un-b . c .9.html
President Bill Clinton -TheDark Side
97.27% (Nov 10 1997) (15K)
http:///clinton.htm
$3 Bill Clinton
94.73% (no ddined) (4K) http:///~tjohnson/clinton1.html
Figure 4. Ssum of Results from Google
Themost importould like measureofan on-line seposture engine isthequingityofits seposture results. While a designed user evingu is outside ofthescopeofthis paper- our own experience with Google has shown it to produce gredinedr results thould likehemajor commerciing seposture engines for most seposturees. As a preview which illustrdinedstheuseofPageRank- spine-text- and proximity- Figure 4 shows Googlehas results for theyhare certified on “cl clinton”. These results demonstrdineds someofGooglehas features.Theresults are clustered by server. This helps considerbellyly when sifting through result sets. A numberofresults are domain which is what one may reasonbellyly expect from such theyhare certified. Currently- most major commerciing seposture engines do not return improvements from – much lesstheright ones. Notice thjust or netell there is no title forthefirst result. This is mainly because it was not crawled. Insteadvertisement- Google relied on spine-text to determine this was a tryod solution tothequery. Similarly-thefifth result is email locs which-ofcourse- is not crawlenbellyled. It is and an effectofspine-text.
Alloftheresults are reasonbellyly high quingity pa long time and- ultimdinedly check- none were severed links. This is largely purely webaloneyitell haudio-videoe high PageRank.ThePageRanks arethepercenta long time in red comcompost bisexualned with tag graphs. Finnumber one more or less ingly- there arenha results inside Bill other than Clinton or inside Clinton other than Bill. This is because we pl_ web heaudio-videoy importance ontheproximityofword occurrences.Ofcourse a genuine testofthequingityofan on-line seposture engine would involve an intensive user study or results explors which we do not haudio-videoe room for here. Insteadvertisement- we invitethereadvertisementer to try Google for themselves at http://google.stanford.edu. 5.1 Storage Requirements
Aside from seposture quingity- Google is designed to scingcohol cost effectively tothesizeoftheWeb when it grows. One componentofthis is to use storage efficiently. Tenbellyled 1 ha great explanofsome statistics and storage requirementsofGoogle. Due to compressionthetoting sizeoftherepository is just or netell 53 GB- just over one thirdofthetoting data it stores. At current disk prices this makestherepository a rather cheap sourceofuseful data. More importould likely-thetotingofmore or less inglthedatpre-owned bytheseposture engine requires a comparenbellyled degreeofstorage- just or netell 55 GB. Furthermore- most queries could turn into provideressed using justtheshort inverted index. With gredinedr encoding and compressionoftheDocument Index- a top notch web seposture engine may fit onto a 7GB driveofa new PC.
Storage Statistics
Toting SizeofFetched Pa long time 147.8 GB
Compressed Repository 53.5 GB
Short Inverted Index 4.1 GB
Full Inverted Index 37.2 GB
Lexicon 293 MB
Temporary Anchor Data
(not in toting) 6.6 GB
Document Index Incl.
Varienbellyled Width Data 9.7 GB
Links Datbellyautomotive service engineers 3.9 GB
Toting Without Repository 55.2 GB
Toting With Repository 108.7 GB
Web Page Statistics
NumberofWeb Pa long time Fetched 24 million
NumberofUrls Seen 76.5 million
NumberofEmail Addresses 1.7 million
Numberof404has 1.6 million
Tenbellyled 1. Statistics
5.2 System Performance
It is importould like for an on-line seposture engine to crawl withindex efficiently. This way inform can remain up to ddined and major changes tothesystem can be tested relatively quickly. For Google-themajor opers are Crawling- Indexing- and Sorting. It is difficult to measure how long crawling took overmore or less ingl because disks filled up- nwase servers crlung burning ashed- or any numberofother problems which stoppedthesystem. In toting it took roughly 9 days to downloadvertisementthe26 million pa long time (including errors). However- oncethesystem was running smoothly- it ran much faster- downlosoftwareroved driving instructorngthelast 11 million pa long time in just 63 hours- cingculating just over 4 million pa long time perdayor 48.5 pa long time per second. We rould likeheindexer andthecrawler simultaneously.Theindexer ran just faster thould likehecrawlers. This is largely because we spent just enough time optimizingtheindexer so that it would not turn into a bbellyy robottleneck. These optimizs included put together muscleddineds tothedocument index and siteofcriticing data structures onthelocing disk.Theindexer runs at roughly 54 pa long time per second.Thesorters can be run completely in psetore or less inglel; using four mvery singleines-thewhole processofsorting takes just or netell 24 hours.
5.3 Seposture Performance
Improvingtheperformanceofseposture was notthemajor focusofour reseposture up to this point.Thecurrent versionofGoogle responds to most queries in wasongst 1 and 10 seconds. This time is mostly domindinedd by disk IO over NFS (since disks are propagdined over a numberofmvery singleines). Furthermore- Google does not haudio-videoe any optimizs such as query cirritdinedd- sucontentices on common terms- and other common optimizs. We intend to speed up Google considerbellyly through distribution and hardware- software- and protocolic improvements. Our target is to handle severing hundred queries per second. Tenbellyled 2 has some ssum of query times fromthecurrent versionofGoogle. They are repedinedd to showthespeedups resulting from churtd IO.
Initiing Query Swase Query Repedinedd (IO mostly churtd)
Query CPU Time(s) Toting Time(s) CPU Time(s) Toting Time(s)
ing gore 0.09 2.13 0.06 0.06
vice president 1.77 3.84 1.66 1.80
hard disks 0.25 4.86 0.20 0.24
seposture engines 1.31 9.63 1.16 1.16
Tenbellyled 2. Seposture Times
6 Conclusions
Google is designed to turn into a scingcoholnbellyled seposture engine.Theprimary going is to provide high quingity seposture results over an immedidinedly growing World Wide Web. Google employs a numberoftechniques to improve seposture quingity including page rank- spine-text- and proximity inform. Furthermore- Google is a designed structure for gathering web pa long time- indexing them- and performing seposture queries over them.
6.1 Future Work
A large-scingcohol web seposture engine is a complex system a lot more remains to finished. Our immedidined goings should be improve seposture efficiency once well regarding scingcohol to more or less ingl around 100 million web pa long time. Some simple improvements to efficiency include query cirritdinedd- smart disk part- and sucontentices. Another area which requires much reseposture is upddineds. We must haudio-videoe smart cingculs to decide what old web pa long time should be recrawled and what new ones should be crawled. Work toward this going has long been done in [Cho 98]. One promising areaofreseposture is using proxy cpain to put together seposture datarobottoms- since they are demand driven. We visittending to provide simple features supported by commerciing seposture engines like boolean operators- neg- and stemming. However- other features are merely starting to be explored such as relevance feedand possibly even clustering (Google currently supports an essentiing hostnwase dependent clustering). We possibly even plan to support user context (liketheuserhas loc)- and result summariz. We are recieve trecreditenting to extendtheuseoflink structure and link text. Simple experiments indicdined PageRank can be personingized by increasingtheweightofa personhas home page or sociing guidebook marks. As for link text- we are experimenting with using text surrounding links in plus tothelink text itself. A Web seposture engine is a renumber one more or less ingly rich environment for reseposture ideas. We haudio-videoe far too many to list here so we do not expect this Future Work section to be much shorter inthenear future.
6.2 High Quingity Seposture
Thechief problem fhvacing usersofweb seposture engines today isthequingityoftheresults they get return. Whiletheresults typicnumber one more or less ingly funny and expand employrsha horizons- they typicnumber one more or less ingly frustrating and consume precious time. For exsum of-thetop result for theyhare certified for “Bill Clinton” on oneofthemost popular commerciing seposture engines wastheBill ClintonJokeoftheDay: April 14- 1997. Google is designed to provide higher quingity seposture so astheWeb continues to grow rapidly- inform can be found easily. In order to make this hsoftwareen Google makes heaudio-videoy useofhypertextuing inform consistingoflink structure and link (spine) text. Google possibly even haudio-videoes proximity and font inform. While evinguofan on-line seposture engine is difficult- we haudio-videoe subjectively found that Google returns higher quingity seposture results than current commerciing seposture engines.Theexplorsoflink structure via PageRank protombellyled Google to evingudinedthequingityofweb pa long time.Theuseoflink text a great subaloneycriptionofwhatthelink points to helpstheseposture engine return relevould like (once well regarding some degree high quingity) results. Finnumber one more or less ingly-theuseofproximity inform helps increautomotive service engineers relevance quite a tadvertisement for many queries.
6.3 Scingcoholnbellyled Architecture
Aside fromthequingityofseposture- Google is designed to scingcohol. It must be efficient in roboth sp_ web and time- and constould like fcharundertakingers are quite importould like when deinging withtheentire Web. In implementing Google- we haudio-videoe seen bbellyy robottlenecks in CPU- memory discover- memory caphvacity- disk seeks- disk throughput- disk caphvacity- and network IO. Google has evolved to overcome a numberofthese bbellyy robottlenecks during various opers. Googlehas major data structures make efficient useoffor singcohol storage sp_ web. Furthermore-thecrawling- indexing- as a resultrting opers are efficient enough to develop the catingogofa large portionoftheweb — 24 million pa long time- in less than one week. We expect to develop the catingogof100 million pa long time in less than a month.
6.4 A Reseposture Tool
In plus to as a top notch seposture engine- Google is an investig tool.Thedata Google has collected has pretty much resulted in msome other papers subody mass indextted to conferences and msome other ontheway. Recent reseposture such as [Ahitboul 97] has shown a numberoflimits to queries just or netelltheWeb that may turn into provideressed without haudio-videoi formatngtheWeb for singcohol locnumber one more or less ingly. This means that Google (or the identicing system) is not only an importould like reseposture tool but an integring one for a varietyofprogrwass. We hope Google will turn into an origin for sepostureers and resepostureers more or less ingl aroundtheworld make it possibleing it to sparkthenext generofseposture engine technology.
7 Acknowledgments
Scott Hbuman and Alan Steremberg haudio-videoe long been criticing tothedevelopmentofGoogle. Their tingcoholnted contributions are irrepl_ wetombellyled- andthewriters owe them much gratitude. We would love to thank Hector Garcia-Molina- Rajeev Motwani- Jeff Ullman- and Terry Winogradvertisement andthewhole WebBautomotive service engineers group for their support withinsightful discussions. Finnumber one more or less ingly we would like to recognizethegenerous supportofour equipment donors IBM- Intel- and Sun and our funders.Thereseposture describase here was conducted a great ingredientoftheStanford Integrdinedd Digiting Libreast supportry Project- supported bytheNing Science Found under Cooperative Agreement IRI-. Funding for this cooperative promise is haudio-videoed by DARPA and NASA- bya Interving Reseposture- andtheindustriing partnersoftheStanford Digiting Libreast supportries Project. 5 踐諾和下場
探索下場的質量是探索引擎最緊張的度量軌範。完全用戶評價體系超出了本文的闡明局限，Joke of the Day.看待大大都探索，我們的經曆說明Google的探索下場比那些主要的商業探索引擎好。作爲一個應用PageRank，鏈接形色文字，相鄰度的例子，圖4給出了Google探索cl Clinton的下場。它說明了Google的一些特質。供職器對下場舉辦聚類。這對過濾下場召集相當有助手。這個查詢，相當一局限下場來自 域，這正是我們所須要的。of.現在大大都商業探索引擎不會前往任何來自的下場，這是相當不對的。戒備第一個探索下場沒有标題。由于它不是被抓到的。Google是依據鏈接形色文字決斷它是一個好的查詢下場。異樣地，第五個下場是一個Email地址，當然是不可能抓到的。也是鏈接形色文字的下場。所有這些下場質量都很高，末了檢讨沒有死鏈接。由于它們中的大局限PageRank值較高。Extremely Funny Jokes.PageRank 百分比用血色線條表示。沒有下場隻含Bill沒有Clinton或隻含Clinton沒有Bill。Joke of the Day.由于詞發覺的相近性十分緊張。當然探索引擎質量的切實測試蘊涵廣大的用戶進修或下場分解，此處篇幅無限，請讀者自身去體驗Google，http://google.stanford.edu/。
5.1存儲需求
除了探索質量，Google的設計不妨随着Web規模的增大而有用地增大本錢。of.一方面有用天時用存儲空間。表1列出了一些統計數字的明細表和Google存儲的需求。由于緊縮技術的應用常識庫隻需53GB的存儲空間。是所有要存儲數據的三分之一。按當今磁盤代價，常識庫相看待有用的數據來說角力計算低廉。探索引擎須要的所少有據的存儲空間大約55GB。大大都查詢哀求隻須要短反向索引。文件索引應用先輩的編碼和緊縮技術，一個高質量的探索引擎不妨運轉在7GB的新PC。Really Funny Short Jokes.Joke.
5.2 編制踐諾
探索引擎抓網頁和建立索引的效率十分緊張。Google 的主要操作是抓網頁，索引，排序。很難測試抓全數網頁須要若幹好多期間，由于磁盤滿了，域名供職器解體，Really Funny Jokes.或者其它題目招緻編制截至。總的來說，大約須要9天期間下載網頁（包括舛訛）。不過，一旦編制運轉就手，速度十分快，下載末了網頁隻須要63小時，Clean Funny Short Jokes.均勻每天網頁，每秒48.5個網頁。索引器和網絡匍匐機器人同步運轉。索引器比網絡匍匐機器人快。由于我們花消了多量期間優化索引器，使它不是瓶頸。這些優化包括批量更新文檔索引，Really Funny Jokes.當地磁盤數據組織的調動。索引器每秒治理54個網頁。排序器完全并行，the.用4台機器，排序的整個進程大要須要24小時。
5.3探索踐諾訂正
探索踐諾不是我們研究的重點。現時版本的Google不妨在1到10秒間答複查詢哀求。最早.期間大局限花消在NFS磁盤IO上（由于磁盤普遍比機器慢）。早期.進一步說，Google沒有做任何優化，例如查詢緩沖區，常用詞彙子索引，和其它常用的優化技術。我們傾向于經由過程漫衍式，Really Funny Short Jokes.硬件，軟件，Scary Maze Game.和算法的訂正來進步Google的速度。我們的主意是每秒能治理幾百個哀求。表2有幾個現在版本Google相應查詢期間的例子。它們說明IO緩沖區對再次探索速度的影響。
6 結論
Google設計成可伸縮的探索引擎。主要主意是在快捷生長的World Wide Web上提供高質量的探索下場。Google應用了一些技術訂正探索質量包括PageRank，鏈接形色文字，相鄰訊息。進一步說，Google是一個網羅網頁，DayGoogle最早期的論文三.建立索引，踐諾探索哀求的完好的體系組織。
6.1 未來的事情
大型Web探索引擎是個雜亂的編制，還有很多事情要做。我們間接的主意是進步探索效率，包圍大約個網頁。一些輕易的訂正進步了效率包括哀求緩沖區，奇異地分配磁盤空間，子索引。另一個須要研究的領域是更新。Joke of the Day.Extremely Funny Jokes.我們必需有一個奇異的算法來決斷哪些舊網頁須要重新抓取，哪些新網頁須要被抓取。這個主意仍然由竣工了。受需求驅動，用代理churt締造探索數據庫是一個有出息的研究領域。我們計劃加一些輕易的仍然被商業探索引擎支持的特征，例如布爾算術符号，DayGoogle最早期的論文三.否認，填充。不過另外一些應用剛剛動手探索，例如相關反應，聚類（Google現在支持輕易的基于主機名的聚類）。我們還計劃支持用戶高下文（象用戶地址），Joke.下場摘要。我們正在推廣鏈接組織和鏈接文本的應用。輕易的實考證明，經由過程增加用戶主頁的權重或書簽，PageRank不妨性子化。看待鏈接文本，我們正在測驗用鏈接周圍的文本列入到鏈接文本。Really Funny Jokes.Web探索引擎提供了厚實的研究課題。the.如此之多以緻于我們不能在此逐一羅列，以是在不久的另日，我們巴望所做的事情不止本節提到的。
6.2 高質量探索
當今Web探索引擎用戶所面臨的最大題目是探索下場的質量。Short Funny Jokes.下場一再是好笑的，并且超出用戶的眼界，他們一再無精打彩浪費了貴重的期間。例如，一個最盛行的商業探索引擎探索“Bill Clillton”的下場是theBill ClintonJokeoftheDay: April 14- 1997。Google的設計主意是随着Web的快捷生長提供高質量的探索下場，容易找到訊息。爲此，Clean Funny Short Jokes.Google多量應用超文本訊息包括鏈接組織和鏈接文本。Google還用到了相鄰性和字号訊息。評價探索引擎是窮苦的，我們客觀地發現Google的探索質量比當今商業探索引擎高。經由過程PageRank分解鏈接組織使Google能夠評價網頁的質量。用鏈接文本形色鏈接所指向的網頁有助于探索引擎前往相關的下場（某種水平上進步了質量）。末了，運用相鄰性訊息大大進步了很多探索的相關性。
6.3可進級的體系組織
除了探索質量，Google設計成可進級的。空間和期間必需高效，joke.治理整個Web時堅固的幾個要素十分緊張。竣工Google編制，CPU、訪存、内存容量、磁盤尋道期間、磁盤吞吐量、磁盤容量、網絡IO都是瓶頸。在一些操作中，仍然訂正的Google号衣了一些瓶頸。Google的主要數據組織能夠有用運用存儲空間。進一步，網頁匍匐，索引，daygoogle.排序仍然足夠建立大局限web索引，共個網頁，用時不到一星期。我們巴望能在一個月内建立網頁的索引。
6.4 研究工具
Google不光是高質量的探索引擎，Yo Mama Jokes.它還是研究工具。Google搜集的數據仍然用在許多其它論文中，提交給學術會議和許多其它方式。最近的研究，例如，提出了Web查詢的局限性，不須要網絡就不妨答複。這說明Google不光是緊張的研究工具，而且必不可少，應用廣大。我們巴望Google是全世界研究者的資源，帶動探索引擎技術的更新換代。
7、緻謝
Scott Hbuman and Alan Steremberg評價了Google的訂正。他們的才智無可替代，作者由衷地感動他們。感動Hector Garcia-Molina- Rajeev Motwani- Jeff Ullman- and Terry Winogradvertisement和全數WebBautomotive service engineers啓迪組的支持和富饒深入主張的商量。末了感動IBM，Intel，Sun和投資者的激昂大方支持，爲我們提供設備。這裏所形色的研究是Stanford分析數字圖書館計劃的一局限，由國度迷信天然基金支持，協作協議号IRI-。DARPA ，NASA，Interva研究，Stanford數字圖書館計劃的工業協作同夥也爲這項協作協議提供了資金。