блог студента программиста

разбираемся с этой проблемой http://fkn.ktu10.com/?q=node/5790

вот ссылка http://yadi.sk/d/2CMtA2ieHCTow

заметка: http://fkn.ktu10.com/?q=node/5781

на линукс точно есть)

Разбираемся с числом запускаемых редукторов: http://fkn.ktu10.com/?q=node/5778

added manifest
adding: hadoop/(in = 0) (out= 0)(stored 0%)
adding: hadoop/mytest/(in = 0) (out= 0)(stored 0%)
adding: hadoop/mytest/WordCount$TokenizerMapper.class(in = 1855) (out= 806)(deflated 56%)
adding: hadoop/mytest/WordCount$IntSumCombiner.class(in = 2680) (out= 1135)(deflated 57%)
adding: hadoop/mytest/WordCount.class(in = 2010) (out= 1040)(deflated 48%)
adding: hadoop/mytest/WordCount$ExpPartitioner.class(in = 836) (out= 424)(deflated 49%)
adding: hadoop/mytest/WordCount$IntSumReducer.class(in = 2655) (out= 1110)(deflated 58%)

Алексей Иванов выпустил мощнейшую "Неразбериху". Слушайте. Думайте.

-client-core-2.2.0.jar:$HADOOP_HOME/share/hadoop/common/lib/commons-cli-1.2.jar -d wordcount_classes WordCount.java
/usr/local/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar(org/apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
Note: WordCount.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 warning

Результат относится к этой заметке: http://fkn.ktu10.com/?q=node/5751

reduser start 	0
----combiner start-------	0
!	2
!!	2
%%	1
&	1
&&	1
)	1
+	1
++	2
.	1
888	1
=)	1
Freedom	3
Life	1
Linux	1
are.	1
good	1
is	4
life.	1
of	2
part	1
----maximum of combiner=====  	0
!	2
-------------- 	0
part	1
we	1
where	1
----maximum of combiner=====  	0
is	2
-------------- 	0
----maximum=====  	0
is	4
-------------- 	0

public static class IntSumCombiner
extends Reducer<Text,IntWritable,Text,IntWritable> {
int max_sum = 0;
Text max_occured_key = new Text();

узнаем когда запускаются подзадачи - комбинаторов и финального редуктора:

reduser start 	0
reduser start 	0
Freedom	1
Life	1
are.	1
is	2
life.	1
of	1
part	1
reduser start 	0
!	2
!!	2
%%	1
&	1
&&	1
)	1
+	1
++	2
.	1
888	1
=)	1
Freedom	2
Linux	1
good	1
is	2
of	1
part	1
----maximum=====  	0
!	2
-------------- 	0
we	1
where	1
----maximum=====  	0
is	2
-------------- 	0
----maximum=====  	0
is	2
-------------- 	0