Certain diseases, and the success of their treatment, are reflected in the structure of mitochondria within human cells. Automating the characterization ofmitochondrial states may accelerate trials to find life-saving drugs. This study explores the use of deep learning tools in preliminary characterization of cell states. We attempt to model morphological changes over the time, using a CNN to embed frames, then comparing several established methods for aggregating time information across frames. We train this deep model with a classification task, to emphasize the presence of mitochondrial fission and fusion in the representation. After obtaining this representation, the video embeddings are reduced through UMAP to show a cohesive progression through the course of a video. Clustering like DBSCAN reveals groups of frames that link to mitochondrial events, showing promise for detecting behavior shifts.