Some thoughts on this:

(1) The language-game of emotion starts not with a private entity which is then described, but with primitive reactions to emotion in others (e.g. being frightened by someone’s facial expression) (CE 409-410; RPP II 689). 

(2) Our language is an auxillary to, and further extension of, primitive reactions towards emotion in other human beings. The language-game of emotion is an extension of primitve behavior (cf. Z 545; RPP I 151). 

(3) In primitive reactions, emotion is not cognitively disencoded from another’s face or posture, but is personified explicitly on the face (RPP II 570). In primitive contexts, perception of emotion is embodied (Z 225). 

(4) Embodied expressions of emotion do not simply communicate but also function to share, to influence the perceiver to feel the same (e.g. motor mimicry and ’emotional contagion’). To perceive a smile is to be inclined to feel certain things (RPP II 614; PR 89). 

(5) But: ‘It would be an incorrect use of language to say: ‘I see fear in this face’. We would be taught: a fearful face can be ‘seen’; but the fear in a face, or the similarity or dissimilarity between two faces, is ‘noticed’ (RPP II 552).  

‘I notice the fear in his face’. Or: we look at two faces and I say to someone: ‘I see a similarity in them’. Does this adduce evidence or specify a perceptual capacity? I propose the latter. Wittgenstein writes: ‘I begin seeing the similarity when it ‘strikes’ me…If the similarity strikes me, I perceive something…’ (RPP II 555).  

Moreover: the perceived dissimilarity in two faces is not given by reference to measurement (the difference may be no more than a thousandth of an inch). We see the difference – this is a primitive reaction (PI 285). Motor mimicry is also relevant here: we can reproduce different facial expressions in the mirror without checking the exact orientation of our features. 

Finally, I propose: insofar as it is true that I cannot see or otherwise perceive his emotion (RPP II 552), neither can he. I see the manifestations of emotion which he exhibits; but when he exhibits emotion, it is manifest. It can be said that I cannot see his emotion only in the sense in which I cannot see sounds or hear colors (LW I 885).

In contrast to primitive reactions, there can be no proof of subtle third-person ascriptions of emotion, and we may be unable to decide whether someone is (e.g.) anxious. Those who are closely acquainted with a person can make the most subtle emotional ascriptions with certainty, without being able to specify conclusive criteria, since their evidence is ‘imponderable’, that is, consists of a syndrome of behavior, context and antecedent events (PI II 227-228; LW II 70, 87, 90-95) (this does not represent a ‘retreat into the mental’. The point is that the circumstances for the use of some mental terms constitute a highly complex syndrome the evidence of which is not always connected in a rigid way).