Воронеж

Разбираемся с числом запускаемых редукторов: http://fkn.ktu10.com/?q=node/5778

added manifest
adding: hadoop/(in = 0) (out= 0)(stored 0%)
adding: hadoop/mytest/(in = 0) (out= 0)(stored 0%)
adding: hadoop/mytest/WordCount$TokenizerMapper.class(in = 1855) (out= 806)(deflated 56%)
adding: hadoop/mytest/WordCount$IntSumCombiner.class(in = 2680) (out= 1135)(deflated 57%)
adding: hadoop/mytest/WordCount.class(in = 2010) (out= 1040)(deflated 48%)
adding: hadoop/mytest/WordCount$ExpPartitioner.class(in = 836) (out= 424)(deflated 49%)
adding: hadoop/mytest/WordCount$IntSumReducer.class(in = 2655) (out= 1110)(deflated 58%)

Алексей Иванов выпустил мощнейшую "Неразбериху". Слушайте. Думайте.

-client-core-2.2.0.jar:$HADOOP_HOME/share/hadoop/common/lib/commons-cli-1.2.jar -d wordcount_classes WordCount.java
/usr/local/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar(org/apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
Note: WordCount.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 warning

Результат относится к этой заметке: http://fkn.ktu10.com/?q=node/5751

reduser start 	0
----combiner start-------	0
!	2
!!	2
%%	1
&	1
&&	1
)	1
+	1
++	2
.	1
888	1
=)	1
Freedom	3
Life	1
Linux	1
are.	1
good	1
is	4
life.	1
of	2
part	1
----maximum of combiner=====  	0
!	2
-------------- 	0
part	1
we	1
where	1
----maximum of combiner=====  	0
is	2
-------------- 	0
----maximum=====  	0
is	4
-------------- 	0

public static class IntSumCombiner
extends Reducer<Text,IntWritable,Text,IntWritable> {
int max_sum = 0;
Text max_occured_key = new Text();

узнаем когда запускаются подзадачи - комбинаторов и финального редуктора:

reduser start 	0
reduser start 	0
Freedom	1
Life	1
are.	1
is	2
life.	1
of	1
part	1
reduser start 	0
!	2
!!	2
%%	1
&	1
&&	1
)	1
+	1
++	2
.	1
888	1
=)	1
Freedom	2
Linux	1
good	1
is	2
of	1
part	1
----maximum=====  	0
!	2
-------------- 	0
we	1
where	1
----maximum=====  	0
is	2
-------------- 	0
----maximum=====  	0
is	2
-------------- 	0

1. Тасовка - это процесс пересылки данных от разных мэпперов разным редукторам?
2. Один мэппре работает с одним входным разбиением (InputSplit) -но может ли быть так что InputSplit настолько большой,что не умещается на одном узле (или их память столь мала) -сколько мэпперов запустится тогда?

.addInputPath http://fkn.ktu10.com/?q=node/5739

?????

/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
*
*
* Unless required by applicable law or agreed to in writing, software