when your multiple gpu code is not workingking.
from https://github.com/tensorflow/tensorflow/issues/36510
TF_FORCE_GPU_ALLOW_GROWTH=true
of course, you can also set this option on in your source code.
gpu_devices = tf.config.experimental.list_physical_devices('GPU')
for device in gpu_devices:
tf.config.experimental.set_memory_growth(device, True)
if you want to check the status of nvlink.
nvidia-smi nvlink --status