openface_2.1.0_zeromq generated .csv data incompatibility with input_facsfromcsv

[Hunanbean]

The .csv data generated by openface_2.1.0_zeromg is not compatible with the input_facsfromcsv interpreter. The demo.csv data works perfectly, and captures keyframes. I can also locally stream from OpenFaceOffline to blender directly from a video file, but this method does not capture keyframes

EDIT: this method is only capturing keyframes if you start the OpenFace program after you set the Blender plugin to connected and tick insert keyframes. If the Openface program was open before you set the plugin, it seems as though the signal to capture frames does not get through. So, in order to get key frame capture when streaming a video from OpenFace, it seems you must have blender open and set to connected, and insert key frames checked before you open OpenFace at all.

When trying to send a .csv i generated with the same OpenFace through input_facsfromcsv/main.py, errors are as follows

frame   face_id   timestamp   confidence   success   gaze_0_x  ...   AU20_c   AU23_c   AU25_c   AU26_c   AU28_c   AU45_c
0      1         0       0.259         0.98         1   0.072391  ...      0.0      1.0      0.0      0.0      0.0      0.0
1      2         0       1.411         0.98         1   0.081622  ...      0.0      1.0      0.0      0.0      0.0      0.0
2      3         0       1.446         0.98         1   0.174707  ...      0.0      1.0      0.0      0.0      0.0      0.0
3      4         0       1.494         0.98         1   0.179774  ...      0.0      1.0      0.0      0.0      0.0      0.0
4      5         0       1.535         0.98         1   0.197015  ...      0.0      1.0      0.0      0.0      0.0      0.0

[5 rows x 714 columns]
Data rows in data frame: 2579
C:\Users\MyAccountName\anaconda3\lib\site-packages\pandas\core\frame.py:4133: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,
FRAME TRACKER: 0
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<FACSvatarMessages.facs_pub() done, defined at main.py:382> exception=KeyError('timestamp')>
Traceback (most recent call last):
  File "C:\Users\MyAccountName\anaconda3\lib\site-packages\pandas\core\indexes\base.py", line 4411, in get_value
    return libindex.get_value_at(s, key)
  File "pandas\_libs\index.pyx", line 44, in pandas._libs.index.get_value_at
  File "pandas\_libs\index.pyx", line 45, in pandas._libs.index.get_value_at
  File "pandas\_libs\util.pxd", line 98, in pandas._libs.util.get_value_at
  File "pandas\_libs\util.pxd", line 83, in pandas._libs.util.validate_indexer
TypeError: 'str' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 388, in facs_pub
    async for msg in self.openface_msg.msg_gen():
  File "main.py", line 283, in msg_gen
    async for i, msg in self.msg_from_csv(csv_group):
  File "main.py", line 343, in msg_from_csv
    ofmsg.set_msg(frame_tracker)
  File "main.py", line 200, in set_msg
    self.msg['timestamp'] = row['timestamp']
  File "C:\Users\MyAccountName\anaconda3\lib\site-packages\pandas\core\series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Users\MyAccountName\anaconda3\lib\site-packages\pandas\core\indexes\base.py", line 4419, in get_value
    raise e1
  File "C:\Users\MyAccountName\anaconda3\lib\site-packages\pandas\core\indexes\base.py", line 4405, in get_value
    return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
  File "pandas\_libs\index.pyx", line 80, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 90, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1618, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1626, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'timestamp'

Thank you, and all who are involved in this project and OpenFace itself, very much for your talent and hard work. This really is an amazing project with top tier performance.

[NumesSanguis]

Thank you for your detailed report @Hunanbean ! and sorry for my late reply.

It seems that the output of OpenFace has changed in newer versions, and I didn't test this.
I'll get back to you once I've figured out (and hopefully fixed) the issue.

[Hunanbean]

Sounds great. thanks a bunch!

[NumesSanguis]

If I understand correctly, the issue before the edit:

The .csv data generated by openface_2.1.0_zeromg is not compatible with the input_facsfromcsv interpreter. The demo.csv data works perfectly, and captures keyframes. I can also locally stream from OpenFaceOffline to blender directly from a video file, but this method does not capture keyframes

is unrelated to the issue found in the "EDIT" right? In case of the pre-EDIT issue, please check this issue: #25

I still have to analyse your "EDIT" issue. Can you tell me your Pandas version?

[Hunanbean]

Correct, it seems to be an unrelated issue.
pandas 1.0.5

Thank you

[NumesSanguis]

Got my new Windows 10 PC setup

this method is only capturing keyframes if you start the OpenFace program after you set the Blender plugin to connected and tick insert keyframes. If the Openface program was open before you set the plugin, it seems as though the signal to capture frames does not get through. So, in order to get key frame capture when streaming a video from OpenFace, it seems you must have blender open and set to connected, and insert key frames checked before you open OpenFace at all.

I think it actually does capture the keyframes. It uses the message number to determine the insert location (if I remember correctly). The confusing part is that if you activate Insert key frames after OpenFace has send 260 messages, you don't see it on the timeline unless you zoom out.

Could you try zooming out and see if the keyframes are indeed inserted at later frames?

Hmm, I guess I can add a toggle to the menu that does either "Insert at current frame" or "Match message number to frame number".

---

When trying to send a .csv i generated with the same OpenFace through input_facsfromcsv/main.py, errors are as follows

From the Pandas dataframe output, it seems that it is using the uncleaned .csv.
input_facsfromcsv assumes that everything in a folder that ends with _clean, has already been processed. So if you put your new .csv in input_facsfromcsv/openface/default_clean/, it won't work. You need to put it e.g. in a folder input_facsfromcsv/openface/some_folder/myrec.csv.

• I think this is the same issue as: How to read csv output by OpenFaceOffline ? #25 (comment)
• Please check the new (in progress) documentation: https://facsvatar.readthedocs.io/en/v0.4.0/trackers/openface.html#use-your-own-videos

Could you try if putting your new .csv in a different named folder solved your issue?