Why Does PyTorch Have Three Different Tanh() Functions?

When I was first learning how to use the PyTorch neural network library, I remember being confused by some of the example programs I found on the Internet. There seemed to be three different tanh() functions. After some investigation I discovered that yes, there are (at least) three different versions of tanh() in PyTorch.

Suppose you have a simple 4-7-3 neural network (for example, for the Iris dataset):

import torch as T

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(4, 7)
    self.oupt = T.nn.Linear(7, 3)

  def forward(self, x):
    z = T.tanh(self.hid1(x))
    z = T.log_softmax(self.oupt(z), dim=1)
    return z

The code uses the tanh() function that is part of the base torch library. Simple, effective, good. There’s also a different tanh() function in the torch.nn.functional module:

import torch as T

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(4, 7)
    self.oupt = T.nn.Linear(7, 3)

  def forward(self, x):
    z = T.nn.functional.tanh(self.hid1(x))
    z = T.log_softmax(self.oupt(z), dim=1)
    return z

Note: The torch.nn.functional.tanh() function has been deprecated but it still exists.

There’s also a Tanh() class:

import torch as T

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(4, 7)
    self.oupt = T.nn.Linear(7, 3)
    self.my_tanh = T.nn.Tanh()

  def forward(self, x):
    z = self.my_tanh(self.hid1(x))
    z = T.log_softmax(self.oupt(z), dim=1)
    return z

The class version is a bit confusing. The Tanh() object has a built-in forward() method that calls a derived built-in __call__() method. The class can also be called directly:

  . . .
  z = T.nn.Tanh()(self.hid1(x))
  . . .

PyTorch has three different tanh() functions because, well, that’s the way Open Source software is. Open Source software evolves somewhat organically which leads to some inconsistencies, but overall, that’s a small price to pay.



Three different book covers for “Murder on the Orient Express” (1934) by Agatha Christie. One of my favorite mystery novels. The film version (1974) is one of my favorite mystery movies.


This entry was posted in PyTorch. Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s