Attention and Memory in Deep Learning Systems

1. Which interpretation best connects attention and memory in the context of cognitive-inspired neural models?

2. What does analyzing a Jacobian (output sensitivity with respect to input) help reveal about implicit attention in neural networks?

3. In a two-head deep reinforcement learning setup that predicts state value and advantage, what is a key difference in how attention-like sensitivities may appear?

4. How does a recurrent neural network (RNN) provide a form of memory compared to a feed-forward network?

5. What distinguishes hard attention from soft attention in training and behavior?

6. In location-based attention, what determines where the model looks in an input sequence?

7. In associative (content-based) attention, how are attention weights typically computed?

8. Why can associative attention help with long-range dependencies compared to relying only on time-based memory in an RNN?

9. What is the general “read” operation in a soft differentiable attention/memory system?

10. What is a core goal of memory-augmented neural models like neural Turing machine-style architectures?

1. Attention is the ability to ______ on one thing while ignoring others.

2. A ______ shows how sensitive network outputs are to changes in inputs, helping locate implicit attention.

3. In many attention mechanisms, soft attention differs from hard attention by using an ______ rather than sampling a discrete glimpse.

4. Recurrent neural networks encode memory in their internal ______ across time steps.

5. Associative (content-based) attention computes weights using similarity between a query ______ and embeddings of candidate items.