Introduction to Python Multiprocessing
Multiprocessing is somewhat of a computerized version of multitasking. Multitasking is the process of handling several tasks at the same time efficiently. Similarly, Multiprocessing in Python is the ability to handle more than one process at the same time. In real life a person who is a multitasker is very successful at his work, similarly, a python program using multiprocessing is far better than one which is not using it.
Python Multiprocessing uses the idea of parallel processing, in which code from a single program can be executed on the different cores of a computer using parallel code.
Multiprocessing in Python
Python has a module named multiprocessing which helps us write parallel code thus resulting in parallel computing. The following classes in Python multiprocessing help us create a parallel program:
In parallel programming, a code is run on different cores. We can know the number of cores in our system in the following way:
print("The number of cores in the system is",multiprocessing.cpu_count())
How can we create a parallel program using the different classes?
Below are various points to build parallel programming:
import numpy as np
from multiprocessing import Process
numbers = [2.1,7.5,5.9,4.5,3.5] def print_func(element=5):
print('Square of the number : ', np.square(element))
if __name__ == "__main__": # confirmation that the code is under main function
procs =  proc = Process(target=print_func) # instantiating without any argument
# instantiating process with arguments
for number in numbers:
proc = Process(target=print_func, args=(number,))
# complete the processes
for proc in procs:
The following are the two important functions of the Process class:-
- start(): The start() function initiates the processing of the process object after it has been created.
- join(): The join() function instructs the process to complete it’s execution. Without the join() function, the process would remain idle and would not terminate.
It is very important to call the join() function after the termination of the process to free the resources for future processes, otherwise, it has to be done manually. The args keyword is used to send an argument through the process.
from multiprocessing import Queue
objects = ["John",34,6578.9,True] counter = 1
# We would instantiate a queue object
queue = Queue()
print('Pushing items to queue:')
for o in objects:
print('Object No: ', counter, ' ', o)
counter = counter + 1
print('\nPopping items from queue:')
counter = 1
while not queue.empty():
print('Object No: ', counter, ' ', queue.get())
counter = counter + 1
Python Multiprocessing has a Queue class that helps to retrieve and fetch data for processing following FIFO(First In First Out) data structure. They are very useful for storing Python pickle objects and eases sharing objects among different process thus helping parallel programming.
They are passed as a parameter to Process target function to enable the Process to consume that data during execution. There are mainly two functions which help us store and fetch data to and from the Queue:-
- put: put() function helps to insert data to the Queue.
- get: get() function helps to retrieve data from the Queue.
In the FIFO data structure, the element which is stored first is to be retrieved first.
FIFO data structure is analogous to Customer Service calls, whereby the customers being the elements, the ones who call the earliest have to wait for the least for getting the call connected to the customer service personal.
from multiprocessing import Pool
work = (["1", 5], ["2", 2], ["3", 1], ["4", 3])
print(" Process %s waiting %s seconds" % (work_data, work_data))
print(" Process %s Finished." % work_data)
p = Pool(2)
if __name__ == '__main__':
Python Multiprocessing Pool class helps in parallel execution of a function across multiple input values. The variable work when declared it is mentioned that Process 1, Process 2, Process 3 and Process 4 shall wait for 5,2,1,3 seconds respectively. During execution, the above-mentioned processes wait for the aforementioned interval of time as it is evident from the order of the print statements.
Python Multiprocessing Lock class allows code to claim lock so that no other process can work on a similar code. There are two important functions of Lock as follows: –
- acquire: acquire() function claims the lock
- release: release() function releases the lock
Let us consolidate all the things that we have learned into a single example:-
from multiprocessing import Lock, Process, Queue, current_process
def do_job(tasks_to_do, tasks_finished):
get_nowait() function raises queue empty exception if the queue is empty.
queue(False) function also does the same task.
task = tasks_to_do.get_nowait()
if no exception is raised, add the task completion
message to tasks_finished queue
tasks_finished.put(task + ' is finished by ' + current_process().name)
number_of_task = 6
number_of_processes = 2
tasks_to_do = Queue()
tasks_finished = Queue()
processes =  for i in range(number_of_task):
tasks_to_do.put("Task no " + str(i))
# creating processes
for w in range(number_of_processes):
p = Process(target=do_job, args=(tasks_to_do, tasks_finished))
# completing process
for p in processes:
# print the output
while not tasks_finished.empty():
if __name__ == '__main__':
It is time to draw closure to this article as we have discussed the basic concepts of Multiprocessing in Python. So next time you write a complex big code do remember to apply the concepts of Multiprocessing in it to fully appreciate it’s powering.
This is a guide to Python Multiprocessing. Here we discuss the introduction, multiprocessing and how can we create a parallel program using different classes. You can also go through our other related articles to learn more–