{"id":1407,"date":"2015-01-18T13:49:13","date_gmt":"2015-01-18T10:19:13","guid":{"rendered":"http:\/\/vua.nadiran.com\/?p=1407"},"modified":"2015-01-18T17:15:39","modified_gmt":"2015-01-18T13:45:39","slug":"%d9%be%db%8c%d8%a7%d8%af%d9%87-%d8%b3%d8%a7%d8%b2%db%8c-%d9%85%d9%88%d8%aa%d9%88%d8%b1-%d8%ac%d8%b3%d8%aa%d8%ac%d9%88-%d8%a8%d8%a7-hbase-nutch-elasticsearch","status":"publish","type":"post","link":"https:\/\/vua.nadiran.com\/?p=1407","title":{"rendered":"\u067e\u06cc\u0627\u062f\u0647 \u0633\u0627\u0632\u06cc \u0645\u0648\u062a\u0648\u0631 \u062c\u0633\u062a\u062c\u0648 \u0628\u0627 HBase Nutch Elasticsearch"},"content":{"rendered":"
\n
\n
\n

Nutch 2.2 with ElasticSearch 1.x and HBase<\/h1>\n<\/header>\n
\n

This document describes how to install and run Nutch 2.2.1 with HBase 0.90.4 and ElasticSearch 1.1.1 on Ubuntu 14.04<\/p>\n

Prerequisites<\/h4>\n

Make sure you installed the Java-SDK 7.<\/p>\n

\r\n$ sudo apt-get install openjdk-7-jdk\r\n<\/code><\/pre>\n
And you set JAVA_HOME in your .bashrc:
\nAdd the following line at the bottom of HOME\/.bashrc:<\/p>\n
\r\nexport JAVA_HOME=\/usr\/lib\/jvm\/java-7-openjdk\r\n<\/code><\/pre>\n
(the jdk might differ)<\/p>\n
Now you need to either reconnect with your terminal or type:<\/p>\n
\r\n$ source ~\/.bashrc\r\n<\/code><\/pre>\n
To load the changes in that file.<\/p>\n
Download Nutch 2.2.x<\/h4>\n
Download the latest release or 2.2.1 from:
\nhttps:\/\/nutch.apache.org\/downloads.html<\/a><\/p>\n
Unpack it and follow the steps described in the tutorial:
\nhttp:\/\/wiki.apache.org\/nutch\/Nutch2Tutorial<\/a><\/p>\n
Download HBase<\/h4>\n
It\u2019s proven to work with version 0.90.4. This version is quite old (2011) so you might try with newer versions but nutch doesn\u2019t support them. Hopefully there will be an upgrade soon.<\/p>\n
http:\/\/archive.apache.org\/dist\/hbase\/hbase-0.90.4\/<\/a><\/p>\n
Download ElasticSearch<\/h4>\n
Download and unpack ElasticSearch 1.x from:<\/p>\n
http:\/\/www.elasticsearch.org\/overview\/elkdownloads\/<\/a><\/p>\n
To run ElasticSearch with the default configuration just go to ES_HOME and type:<\/p>\n
\r\n$ bin\/elasticsearch\r\n<\/code><\/pre>\n
Install HBase<\/h4>\n
Install HBase according to:
\nhttp:\/\/hbase.apache.org\/book\/quickstart.html<\/a><\/p>\n
If you\u2019re running on Ubuntu you need to change the file \/etc\/hosts
\nDue to some internal problems with old versions of HBase and the loopback of IP-addresses you need to specify localhost as 127.0.0.1
\nJust change all localhost-ips to the format above. Sometimes (on Ubuntu) localhost is 127.0.1.1.
\nApparently this is fixed in newer versions of HBase, but you cannot use them yet.<\/p>\n
Now you have to change the configuration of HBASE_HOME\/conf\/hbase-site.xml.
\nHbase and Zookeper need directories where to save data to. Default is \/temp which would be gone after restarting the computer.
\nSo create 2 folders one for HBase and one for Zookeeper where they can save their data.<\/p>\n
\r\n<property>\r\n<name>hbase.rootdir<\/name>\r\n<value>file:\/\/\/DIRECTORY\/hbase<\/value>\r\n<\/property>\r\n<property>\r\n<name>hbase.zookeeper.property.dataDir<\/name>\r\n<value>\/DIRECTORY\/zookeeper<\/value>\r\n<\/property>\r\n<\/code><\/pre>\n
Just replace DIRECTORY whith a folder of your choice. Don\u2019t forget file:\/\/ in front of your hbase.rootdir
\nYou need to specify a location on your local filesytem for running HBase in standalone-mode (without hdfs).<\/p>\n
Now start Hbase and run in HBASE_HOME:<\/p>\n
\r\n$ .\/bin\/start-hbase.sh\r\n<\/code><\/pre>\n
Now you can check the logs at the specified location.<\/p>\n
Now please use the shell and test your HBase installation.<\/p>\n
\r\n$ .\/bin\/hbase shell\r\n<\/code><\/pre>\n
You should be able to create a table:<\/p>\n
\r\n$ create 'test', 'ab'\r\n<\/code><\/pre>\n
Expected output:<\/p>\n
\r\n$ \u06f0 row(s) in 1.2200 seconds\r\n<\/code><\/pre>\n
With the command scan you can just list all the content of the created table:<\/p>\n
\r\n$ scan 'test'\r\n<\/code><\/pre>\n
If there are no errors, you\u2019re HBase should be set up correctly.<\/p>\n
Setting up Nutch to work with HBase and ElasticSearch 1.x<\/h4>\n
Go to your NUTCH_HOME and edit conf\/nutch-site.xml:
\nEnable HBase as backend-database:<\/p>\n
\r\n<property>\r\n<name>storage.data.store.class<\/name>\r\n<value>org.apache.gora.hbase.store.HBaseStore<\/value>\r\n<description>Default class for storing data<\/description>\r\n<\/property><\/code><\/pre>\n
 <\/p>\n
<property>
\n<name>http.agent.name<\/name>
\n<value>My Private Spider Bot<\/value>
\n<\/property><\/p>\n
<property>
\n<name>http.robots.agents<\/name>
\n<value>My Private Spider Bot<\/value>
\n<\/property>
\n<\/code><\/p>\n
Now set the versions in your dependency-manager in NUTCH_HOME\/ivy\/ivy.xml:<\/p>\n
\r\n<!-- Uncomment this to use HBase as Gora backend. -->\r\n<dependency org=\"org.apache.gora\" name=\"gora-hbase\" rev=\"0.3\" conf=\"*->default\" \/>\r\n<\/code><\/pre>\n
To make sure that the correct version of ElasticSearch is used you also need to change the default version to the one you want to use:<\/p>\n
\r\n<dependency org=\"org.elasticsearch\" name=\"elasticsearch\" rev=\"1.1.1\" conf=\"*->default\"\/>\r\n<\/code><\/pre>\n
Now you need to edit a line of Java-Source-Code.
\nNUTCH_HOME\/src\/java\/org\/apache\/nutch\/indexer\/elastic\/ElasticWriter.java
\nThe line with item.failed() needs to be changed. Since there was an API-Update from the version that was used per default.<\/p>\n
\r\nif (item.isFailed()) {...}\r\n<\/code><\/pre>\n
Now you need to edit in gora.properties:
\nEnable HBase as a default datastore:<\/p>\n
\r\ngora.datastore.default=org.apache.gora.hbase.store.HBaseStore\r\n<\/code><\/pre>\n
Compile Nutch<\/h4>\n
Just go to your NUTCH_HOME directory and run:<\/p>\n
\r\n$ ant runtime\r\n<\/code><\/pre>\n
When the build was succesful you can start working.<\/p>\n
Make sure Hbase is running!<\/h4>\n
Now you can start crawling a website<\/h4>\n
Create a folder called e.g. \u2018urls\u2019 in NUTCH_HOME\/runtime
\nCreate a file called seed.txt inside and add, line per line all the URLs that you want to crawl.<\/p>\n
Now for the standalone mode (not using hadoop) go to NUTCH_HOME\/runtime\/local:<\/p>\n
Now you need to execute a pipeline of commands all starting with bin\/nutch:
\nhttp:\/\/wiki.apache.org\/nutch\/CommandLineOptions<\/a><\/p>\n
\r\n\u06f1 $ bin\/nutch inject <seed-url-dir>\r\n\u06f2 $ bin\/nutch generate -topN <n>\r\n\u06f3 $ bin\/nutch fetch -all\r\n\u06f4 $ bin\/nutch parse -all\r\n\u06f5 $ bin\/nutch updatedb\r\n\u06f6 $ bin\/nutch elasticindex <clustername> -all\r\n<\/code><\/pre>\n
To check whether everything worked you can look at hbase (via hbase-shell):<\/p>\n
\r\n$ > scan 'webpage'\r\n<\/code><\/pre>\n
Then repete the steps 2-5 as much as you want and then write everything to the index (6).<\/p>\n
To check whether something has been written to the ElasticSearch index just execute:<\/p>\n
\r\n$ curl -XGET 'http:\/\/localhost:9200\/index\/_search?q=*&pretty=true'\r\n<\/code><\/pre>\n
There you should see the crawled and downloaded documents with the raw text and all the metadata in json-format.<\/p>\n
Nutch saves everything from HBase \u2018webpage\u2019 to an index called \u2018index\u2019 per default and exports all \u2018documents\u2019 to ElasticSearch with the type \u2018doc\u2019.<\/p>\n
Useful Links:<\/p>\n
http:\/\/www.sigpwned.com\/content\/nutch-2-and-elasticsearch<\/a>
\nhttp:\/\/etechnologytips.com\/create-web-crawler-data-miner\/<\/a>
\nhttp:\/\/wiki.apache.org\/nutch\/CommandLineOptions<\/a>
\nhttp:\/\/de.slideshare.net\/digitalpebble\/j-nioche-lucenerevoeu2013<\/a>
\nhttps:\/\/www.inkling.com\/read\/hadoop-definitive-guide-tom-white-3rd\/chapter-16\/nutch-search-engine<\/a><\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
\u0646\u0635\u0628 ElasticSearch \u062f\u0631 \u06f5 \u062f\u0642\u06cc\u0642\u0647 :<\/p>\n
\n
\n
Install Elasticsearch in 5 Minutes<\/h1>\n<\/header>\n
\n
This is a short tutorial to install Elasticsearch in 5 minutes on Ubuntu in a Digital Ocean droplet.<\/p>\n
I\u2019ve been working with\u00a0WordPress for a long time and what really got me hooked in the early days was the\u00a0\u201cFamous 5-Minute Install\u201d<\/a>.\u00a0I\u2019m extending that same concept to one of my new favorite tools\u00a0\u2013\u00a0Elasticsearch.<\/a>\u00a0It\u2019s a super fast search service built on Lucene that has\u00a0an embedded RESTful JSON API.<\/p>\n
Since it\u2019s native\u00a0JSON, any object you have in your code \u2013 whether it be a Javascript object or a C# object \u2013 can be serialized and inserted into an Elasticsearch Index. So technically you can use it as a NoSQL database. It clusters and does a lot of other fancy stuff but that\u2019s not the point of this article. Anyway you probably already know what it is if you stumbled on this post so\u00a0lets get your very own Elastic Search sandbox up and running\u2026<\/p>\n
Step 1: Get A Server<\/strong><\/h2>\n
\"ScreenIn order to get this done in 5 minutes we\u2019re going to useDigital Ocean<\/a>\u00a0to spin up a cloud server. Why? Because it\u2019s awesome and your server will be ready in 55 seconds\u2026 It\u2019s cheap to run and free to get started if you use one of\u00a0their many promo codes<\/a>. If this doesn\u2019t sound awesome to you, feel free to spend an hour or so setting up a Linux virtual machine. Either way, this tutorial assumes you are going to run ElasticSearch on Linux,\u00a0specifically Ubuntu.<\/p>\n
So after you\u00a0sign up for Digital Ocean<\/a>, setup a free\u00a0\u00a0Ubuntu Droplet (more info than you need is here<\/a>). They\u2019ll email you the root password and you should be good to go to access the Linux console from their website.<\/p>\n
\"Screen<\/a><\/p>\n
Note: there are a bunch of other things you\u2019ll want to do if you run this server in production \u2013 like setting up SSH, disabling root login, and other things. Follow this tutorial for \u2018Initial Server Setup With Ubuntu<\/a>\u2018 for more details.<\/em><\/p>\n
Step 2: Install Elasticsearch<\/strong><\/h2>\n
Now you are ready to install Elasticsearch. Fortunately that\u2019s the easy part. Run the shell script in\u00a0this gist<\/a>\u00a0to get up and running.<\/p>\n
 <\/p>\n
\n
\n
\n
\n\n\n\n
\u06f1\u06f2\u06f3\u06f4\u06f5\u06f6\u06f7\u06f8\u06f9<\/td>\n\n
cd ~<\/div>\n
sudo apt-get update<\/div>\n
sudo apt-get install openjdk-7-jre-headless -y<\/div>\n
<\/div>\n
wget https:\/\/download.elasticsearch.org\/elasticsearch\/elasticsearch\/elasticsearch-1.2.2.deb<\/div>\n
sudo dpkg -i elasticsearch-1.2.2.deb<\/div>\n
sudo service elasticsearch start<\/div>\n
<\/div>\n
#curl http:\/\/localhost:9200<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
view raw<\/a>es.sh<\/a>\u00a0hosted with \u2764 by\u00a0GitHub<\/a><\/div>\n<\/div>\n<\/div>\n
 <\/p>\n
Aaaand you\u2019re done.<\/p>\n
Want to make sure it\u2019s running? Run a curl in your console, hitting port 9200.<\/p>\n
curl http:\/\/localhost:9200<\/pre>\n
You should see something like this giving you some meta data about your Elasticsearch instance.<\/p>\n
 <\/p>\n
\"Install<\/a><\/p>\n
Now, if I had DNS setup for this hostname, you will now be able to hit Elasticsearch externally with http:\/\/elastic.brudtkuhl.com:9200 but for now you can just go at the public IP address that Digital Ocean provides.<\/p>\n
This is the first in a series of posts on my experiences working with Elasticsearch. Do you have any questions on how to install Elasticsearch?<\/p>\n
Now onto your next step:\u00a0Securing Elasticsearch<\/a>.<\/p>\n
\u00a0<\/em><\/small><\/p>\n<\/div>\n
 <\/p>\n<\/div>\n