session.run Requesting multiple values from graph at same time

From https://stackoverflow.com/questions/35512046/requesting-multiple-values-from-graph-at-same-time/35513102#35513102

In the code below l2 surprisingly returns the same value as l1, but since the optimizer is being requested in the list before l2, I expected the loss to be the new loss after training. Can I not request multiple values at the same time from the graph and expect consistent output?

import tensorflow as tf
import numpy as np

x = tf.placeholder(tf.float32, shape=[None, 10])
y = tf.placeholder(tf.float32, shape=[None, 2])

weight = tf.Variable(tf.random_uniform((10, 2), dtype=tf.float32))

loss = tf.nn.sigmoid_cross_entropy_with_logits(tf.matmul(x, weight), y)

optimizer = tf.train.AdamOptimizer(0.1).minimize(loss)

with tf.Session() as sess:
    tf.initialize_all_variables().run()

    X = np.random.rand(1, 10)
    Y = np.array([[0, 1]])

    # Evaluate loss before running training step
    l1 = sess.run([loss], feed_dict={x: X, y: Y})[0][0][0]
    print(l1) # 3.32393

    # Running the training step
    _, l2 = sess.run([optimizer, loss], feed_dict={x: X, y: Y})
    print(l2[0][0]) # 3.32393 -- didn't change?

    # Evaluate loss again after training step as sanity check
    l3 = sess.run([loss], feed_dict={x: X, y: Y})[0][0][0]
    print(l3) # 2.71041

No - the order in which you request them in the list has no effect on the evaluation order. For side-effect-having operations such as the optimizer, if you want to guarantee a specific ordering, you need to enforce it using with_dependencies or similar control-flow constructs. In general, ignoring side-effects, TensorFlow will return results to you by grabbing the node from the graph as soon as it's computed - and, obviously, the loss is computed before the optimizer, since the optimizer requires the loss as one of its input. (Remember that 'loss' is not a variable; it's a tensor; so it's not actually affected by the optimizer step.)

sess.run([loss, optimizer], ...)

and

sess.run([optimizer, loss], ...)

are equivalent.