Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unintended session initialization on TF backend #8377

Merged
merged 3 commits into from
Nov 5, 2017
Merged

Remove unintended session initialization on TF backend #8377

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3908b38
Removing the list_local_devices() call and check if candidate_vars empty
datumbox Nov 3, 2017
9605ed7
Use list_devices() instead of list_local_devices()
datumbox Nov 3, 2017
8201921
Use a cache variable to store local devices.
datumbox Nov 4, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 18 additions & 14 deletions keras/backend/tensorflow_backend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import tensorflow as tf
from tensorflow.python.client import device_lib
from tensorflow.python.training import moving_averages
from tensorflow.python.ops import tensor_array_ops
from tensorflow.python.ops import control_flow_ops
Expand Down Expand Up @@ -45,9 +44,10 @@
# Change its value via `manual_variable_initialization(value)`.
_MANUAL_VAR_INIT = False

# This list queries the devices.
# This list holds the available devices.
# It is populated when `_get_available_gpus()` is called for the first time.
# We assume our devices don't change during our lifetime.
_LOCAL_DEVICES = device_lib.list_local_devices()
_LOCAL_DEVICES = None


def get_uid(prefix=''):
Expand Down Expand Up @@ -174,17 +174,18 @@ def get_session():
for v in variables:
if not getattr(v, '_keras_initialized', False):
candidate_vars.append(v)
# This step is expensive, so we only run it on variables
# not already marked as initialized.
is_initialized = session.run(
[tf.is_variable_initialized(v) for v in candidate_vars])
uninitialized_vars = []
for flag, v in zip(is_initialized, candidate_vars):
if not flag:
uninitialized_vars.append(v)
v._keras_initialized = True
if uninitialized_vars:
session.run(tf.variables_initializer(uninitialized_vars))
if candidate_vars:
# This step is expensive, so we only run it on variables
# not already marked as initialized.
is_initialized = session.run(
[tf.is_variable_initialized(v) for v in candidate_vars])
uninitialized_vars = []
for flag, v in zip(is_initialized, candidate_vars):
if not flag:
uninitialized_vars.append(v)
v._keras_initialized = True
if uninitialized_vars:
session.run(tf.variables_initializer(uninitialized_vars))
return session


Expand Down Expand Up @@ -250,6 +251,9 @@ def _get_available_gpus():
# Returns
A list of available GPU devices.
"""
global _LOCAL_DEVICES
if _LOCAL_DEVICES is None:
_LOCAL_DEVICES = get_session().list_devices()
return [x.name for x in _LOCAL_DEVICES if x.device_type == 'GPU']


Expand Down
6 changes: 2 additions & 4 deletions keras/utils/training_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,11 @@


def _get_available_devices():
from tensorflow.python.client import device_lib
local_device_protos = device_lib.list_local_devices()
return [x.name for x in local_device_protos]
return [x.name for x in K.get_session().list_devices()]


def _normalize_device_name(name):
name = name.lower().replace('device:', '')
name = '/' + name.lower().split('device:')[1]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would this line change at all?

Note that tests for this util are not run on Travis, so please run them manually on a multi-GPU machine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output of the two functions for some reason is different so I need to treat it differently to get the same names out.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it different, and what does it look like in both cases?

Copy link
Contributor Author

@datumbox datumbox Nov 4, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea why it looks diffferent, this is on the side of Tensorflow. Here is how they look in each case:

>>> from tensorflow.python.client import device_lib
>>> l1=[d.name for d in device_lib.list_local_devices()]
>>> l1
[u'/cpu:0']

>>> from keras import backend as K
>>> l2=[d.name for d in K.get_session().list_devices()]
>>> l2
['/job:localhost/replica:0/task:0/device:CPU:0']

That's why I'm splitting on "device:", lowercase and prepend slash to make them look the same which is required in the GPU utils code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW this device_lib.list_local_devices method is extremely problematic. In earlier versions of TF it was actually registering devices that were not whitelisted in CUDA_VISIBLE_DEVICES. The K.get_session().list_devices() method does the trick just fine; I've been using this in large GPU clusters (Keras+Spark) for distributed prediction and never had an issue.

return name


Expand Down