Introduction to Python Multiprocessing
Multiprocessing is somewhat of a computerized version of multitasking. Multitasking is the process of handling several tasks at the same time efficiently. Similarly, Multiprocessing in Python is the ability to handle more than one process at the same time. In real life, a multitasker person is very successful at his work; similarly, a python program using multiprocessing is far better than one which is not using it.
Python Multiprocessing uses the idea of parallel processing, in which code from a single program can be executed on the different cores of a computer using parallel code.
Multiprocessing in Python
Python has a module named multiprocessing which helps us write parallel code, thus resulting in parallel computing. The following classes in Python multiprocessing help us create a parallel program:
In parallel programming, a code is run on different cores. We can know the number of cores in our system in the following way:
import multiprocessing print("The number of cores in the system is",multiprocessing.cpu_count())
How can we create a parallel program using the different classes?
Below are various points to build parallel programming:
import numpy as np from multiprocessing import Process numbers = [2.1,7.5,5.9,4.5,3.5] def print_func(element=5): print('Square of the number : ', np.square(element)) if __name__ == "__main__": # confirmation that the code is under main function procs =  proc = Process(target=print_func) # instantiating without any argument procs.append(proc) proc.start() for number in numbers: proc = Process(target=print_func, args=(number,)) procs.append(proc) proc.start() for proc in procs: proc.join()
The following are the two important functions of the Process class:-
- start(): The start() function initiates the process object’s processing it has been created.
- join(): The join() function instructs the process to complete it’s execution. Without the join() function, the process would remain idle and would not terminate.
It is very important to call the join() function after the termination of the process to free the resources for future processes; otherwise, it has to be done manually. The args keyword is used to send an argument through the process.
from multiprocessing import Queue objects = ["John",34,6578.9,True] counter = 1 # We would instantiate a queue object queue = Queue() print('Pushing items to queue:') for o in objects: print('Object No: ', counter, ' ', o) queue.put(o) counter = counter + 1 print('\nPopping items from queue:') counter = 1 while not queue.empty(): print('Object No: ', counter, ' ', queue.get()) counter = counter + 1
Python Multiprocessing has a Queue class that helps to retrieve and fetch data for processing following FIFO(First In First Out) data structure. They are very useful for storing Python pickle objects and eases sharing objects among different processes, thus helping parallel programming.
They are passed as a parameter to the Process target function to enable the Process to consume that data during execution. There are mainly two functions that help us store and fetch data to and from the Queue:-
- put: put() function helps to insert data into the Queue.
- get: get() function helps to retrieve data from the Queue.
In the FIFO data structure, the element which is stored first is to be retrieved first.
FIFO data structure is analogous to Customer Service calls, whereby the customers being the elements, the ones who call the earliest have to wait for the least for getting the call connected to the customer service personal.
from multiprocessing import Pool import time work = (["1", 5], ["2", 2], ["3", 1], ["4", 3]) def work_log(work_data): print(" Process %s waiting %s seconds" % (work_data, work_data)) time.sleep(int(work_data)) print(" Process %s Finished." % work_data) def pool_handler(): p = Pool(2) p.map(work_log, work) if __name__ == '__main__': pool_handler()
Python Multiprocessing Pool class helps in the parallel execution of a function across multiple input values. The variable work when declared it is mentioned that Process 1, Process 2, Process 3, and Process 4 shall wait for 5,2,1,3 seconds respectively. During execution, the above-mentioned processes wait for the aforementioned interval of time as it is evident from the print statements’ orders.
Python Multiprocessing Lock class allows code to claim lock so that no other process can work on a similar code. There are two important functions of Lock as follows: –
- acquire: acquire() function claims the lock
- release: release() function releases the lock
Let us consolidate all the things that we have learned into a single example:-
from multiprocessing import Lock, Process, Queue, current_process import time import queue def do_job(tasks_to_do, tasks_finished): while True: try: ''' get_nowait() function raises queue empty exception if the queue is empty. queue(False) function also does the same task. ''' task = tasks_to_do.get_nowait() except queue.Empty: break else: ''' if no exception is raised, add the task completion message to tasks_finished queue ''' print(task) tasks_finished.put(task + ' is finished by ' + current_process().name) time.sleep(0.25) return True def main(): number_of_task = 6 number_of_processes = 2 tasks_to_do = Queue() tasks_finished = Queue() processes =  for i in range(number_of_task): tasks_to_do.put("Task no " + str(i)) # creating processes for w in range(number_of_processes): p = Process(target=do_job, args=(tasks_to_do, tasks_finished)) processes.append(p) p.start() # completing process for p in processes: p.join() # print the output while not tasks_finished.empty(): print(tasks_finished.get()) return True if __name__ == '__main__': main()
It is time to draw closure to this article as we have discussed the basic concepts of Multiprocessing in Python. Next time you write a complex big code, do remember to apply the multiprocessing concepts in it to fully appreciate its powering.
This is a guide to Python Multiprocessing. Here we discuss the introduction, multiprocessing, and how can we create a parallel program using different classes. You can also go through our other related articles to learn more –