Map Reduce: Understanding the reduce function and its limitations. What happens with the processing of first element?

When we have a dataset its often necessary to reduce this dataset to single values that correspond to indicators, for instance, normally related to statistics indicators. Some indicators examples are the average, the variance and the standard deviation.

The goal of the reduce function is to assist us to calculate such indicators.  To use the reduce function we have to import that from functools module:

from functools import reduce

 Now to use the reduce function we need to understand its interface, it has two parameters:

a)    a function which responsible for reducing. It needs to have two parameters – the first one a counter (variable shared among the processing of all iterated elements) and the second current element. The function needs to return the updated value of counter, which will be de input of the next iterating element.

b)    a list of elements to be reduced.

Here a example of how to use the reduce function, which will sum all elements of a certain list.

items = [1, 2, 3, 4, 5]

sum = reduce((lambda x, y: x + y), items)

If you want to understand deeper how it works, we could implement an example which will print each step of the process, as the example below:

def product(counter, element):
    print ("element", element, "counter", counter)
    return counter*element
product = reduce(product, items)

 

The output of this processing will print the following:

·        element 2 counter 1

·        element 3 counter 2

·        element 4 counter 6

·        element 5 counter 24

What we can identify here? The first element is not processed… It is just added as input for the processing of the second element. So, for situations in which all elements has to be processed before aggregated to counter, including the first one, the reduce function is not suitable for this purpose. It is the example for the variance calculation. To suppress this limitation we can implement that not using the reduce, but using the “foreach” resource of python programming language.

#variance
print("####VARIANCE####")
counter=0
for element in items:
    counter+=((element-average)**2)/(length-1)
print("VARIANCE",counter)
print("STANDARD DEVIATION",math.sqrt(counter))

 

The complete code is available to be downloaded here:

https://github.com/rafaelqg/code/blob/main/map_reduce_reduce_example.py

 You may see a video class about this video here: 



Comments

Post a Comment

Popular posts from this blog

HASHLIB: Using HASH functions MD5 and SHA256

Image
  The usage of hash functions in python is quite simple. Basically we can use the native library hashlib, which give us the possibly to create message digest based on several algorithms, such as MD5 or SHA256. Using this library we may create a hash of some content and then retrieve its binary output or transform that in hexadecimal strings to easy the data transfer in different medias. The library can be imported with the following instruction: import hashlib   If you want to encode strings, it is necessary to encode that with UTF-8, and it can be done using the encode method of a string object: myPassword= "python3.x" myPasswordEncoded=myPassword.encode( 'utf-8' )       The n, the hashlib module has a md5 function which receives as parameter the the we would like to generate the hash. And then return the hash object. hashObject=hashlib.md5(myPasswordEncoded)   The hashObject already contains the ...
Read more

Spread Operator

Image
  The spread operator can be utilized both on arrays and objects. For arrays it will transform the array in a plain list of objects/variables, which can be utilized as input to functions that accept a list of arguments (example: array.push(el1, el2, el3). In This example if we have one array named array01 with options [1,2,3] and we want to add these values to another array, named array02, then we can do array02.push(…array01). Performing that, using the spread operator (…) actually the array02 will have a copy of all elements that are inside array01. Without using the spread operator , for example, array02.push(array01), the array02 will contains one single object that is the array01 itself, and only inside array01 that the values will be found. Another applicability for spread operator is for objects itself. When we want to have a new object with contains a copy of all attributes an existing object already have, in addition to new attributes, we can use the spread operator. L...
Read more

Dart: implementing Object Oriented Programming (OOP)

Image
The object oriented programming provides several benefits for system design. It makes the code easier to maintain, easier to extends, and also implements several well-know design patterns. Using the Dart programing language classes can be simple defined utilizing the class keyword, followed by the class name, and then delimitated the class body with the key brackets {}, like in the example below, presenting one class with two methods. class ProgrammingLanguage {   void compile (){     print( "Compiling..." );   }   void execute (){     print ( "Executing..." );   } } When we need to add new features to this class, it is possible to use the inherence concept, defining a new class that will receive all implementation this ProgrammingLanguage class already has done. And this inherence is done using the extends keywork, like in the example below, which the Dart class has the compile() and the execute() methos from its superclass...
Read more