Daksh Thapar

selected publications (see Google Scholar for full list)

1. CVPR

   Merry Go Round: Rotate a Frame and Fool a DNN

   Daksh Thapar, Aditya Nigam, and Chetan Arora

   In CVPR 2022

   Abs

   A large proportion of videos captured today are first per-son videos shot from wearable cameras. Similar to other computer vision tasks, Deep Neural Networks (DNNs) are the workhorse for most state-of-the-art (SOTA) egocentric vision techniques. On the other hand DNNs are known to be susceptible to Adversarial Attacks (AAs) which add im-perceptible noise to the input. Both black-box, as well as white-box attacks on image as well as video analysis tasks have been shown. We observe that most AA techniques basically add intensity perturbation to an image. Even for videos, the same process is essentially repeated for each frame independently. We note that the definition of imperceptibility used for images may not be applicable for videos, where a small intensity change happening randomly in two consecutive frames may still be perceptible. In this paper we make a key novel suggestion to use perturbation in optical flow to carry out AAs on a video analysis system. Such perturbation is especially useful for egocentric videos, because there is a lot of shake in the egocentric videos anyways, and adding a little more, keeps it highly imperceptible. In general, our idea can be seen as adding structured, para-metric noise as the adversarial perturbation. Our implementation of the idea by adding 3D rotations to the frames reveal that using our technique, one can mount a black-box AA on an egocentric activity detection system in one-third of the queries compared to the SOTA AA technique.

2. ICCV

   Anonymizing Egocentric Videos

   Daksh Thapar, Aditya Nigam, and Chetan Arora

   In ICCV 2021

   Abs PDF

   In egocentric videos, the face of a wearer capturing the video is never captured. This gives a false sense of security that the wearer's privacy is preserved while sharing such videos. However, egocentric cameras are typically harnessed to wearer's head, and hence, also capture wearer's gait. Recent works have shown that wearer gait signatures can be extracted from egocentric videos, which can be used to determine if two egocentric videos have the same wearer. In a more damaging scenario, one can even recognize a wearer using hand gestures from egocentric videos, or identify a wearer in third person videos such as from a surveillance camera. We believe, this could be a death knell in sharing of egocentric videos, and fatal for egocentric vision research. In this work, we suggest a novel technique to anonymize egocentric videos, which create carefully crafted, but small, and imperceptible optical flow perturbations in an egocentric video's frames. Importantly, these perturbations do not affect object detection or action/activity recognition from egocentric videos but are strong enough to dis-balance the gait recovery process. In our experiments on benchmark Epic-Kitchens dataset, the proposed perturbation degrades the wearer recognition performance, from 66.3% to 13.4%, while preserving the activity recognition performance from 89.6% to 87.4%. To test our anonymization with more wearer recognition techniques, we also developed a stronger, and more generalizable wearer recognition method based on camera egomotion cues. The approach achieves state-of-the-art (SOTA) performance of 59.67% on Epic-Kitchens, compared to 55.06%. However, the accuracy of our recognition technique also drops to 12% using the proposed anonymizing perturbations.

3. IEEE CEM

   LakshmanRekha: AI-biometric driven Smartphone App for strict Post-COVID Home Quarantine Management

   Gaurav Jaiswal, Rohit J. Bharadwaj, Kamlesh Tiwari, Daksh Thapar, Piyush Goyal, and Aditya Nigam

   In IEEE Consumer Electronics Magazine

   Abs PDF

   COVID-19 has been announced as a Global Communal Health Extremity by WHO on January 2020. Meaningful preventive solutions have been taken with smartphone selfie/geofencing apps toward managing mandatory home quarantine and physical distancing. In the post-COVID world, fast screening and strict quarantine can play a crucial role in bringing back normality. Quarantine being offered at home can be a comfortable solution for both government and patients. On the other hand, it can be hazardous if not strictly followed and adequately realized. However, the existing geofencing/face selfie apps take static photographs and location data at certain time intervals that can allow patients to violate the rules between those periods, thus failing to ensure active user identity. To realize unbreached home quarantine policies, this article introduces a CUBA-HQM smartphone app that performs continuous user biometric authentication (CUBA) augmented with geofencing using AI technology. The purpose of continuous tracking is to strictly control the spread of infectious diseases in society by monitoring the individual move in/out in the quarantine zone.

4. ECCV

   Is sharing of egocentric video giving away your biometric signature?

   Daksh Thapar, Aditya Nigam, and Chetan Arora.

   In ECCV 2020

   Abs PDF Project Page

   Easy availability of wearable egocentric cameras, and the sense of privacy propagated by the fact that the wearer is never seen in the captured videos, has led to a tremendous rise in public sharing of such videos. Unlike hand-held cameras, egocentric cameras are harnessed on the wearer’s head, which makes it possible to track the wearer’s head motion by observing optical flow in the egocentric videos. In this work, we create a novel kind of privacy attack by extracting the wearer’s gait profile, a well known biometric signature, from such optical flow in the egocentric videos. We demonstrate strong wearer recognition capabilities based on extracted gait features, an unprecedented and critical weakness completely absent in hand-held videos. We demonstrate the following attack scenarios: (1) In a closed-set scenario, we show that it is possible to recognize the wearer of an egocentric video with an accuracy of more than 92.5% on the benchmark video dataset. (2) In an open-set setting, when the system has not seen the camera wearer even once during the training, we show that it is still possible to identify that the two egocentric videos have been captured by the same wearer with an Equal Error Rate (EER) of less than 14.35%. (3) We show that it is possible to extract gait signature even if only sparse optical flow and no other scene information from egocentric video is available. We demonstrate the accuracy of more than 84% for wearer recognition with only global optical flow. (4) While the first person to first person matching does not give us access to the wearer’s face, we show that it is possible to match the extracted gait features against the one obtained from a third person view such as a surveillance camera looking at the wearer in a completely different background at a different time. In essence, our work indicates that sharing one’s egocentric video should be treated as giving away one’s biometric identity and recommend much more oversight before sharing of egocentric videos.

5. ACM MM

   Recognizing Camera Wearer from Hand Gestures in Egocentric Videos

   Daksh Thapar, Aditya Nigam, and Chetan Arora

   In ACM MM 2020

   Abs PDF Project Page

   Wearable egocentric cameras are typically harnessed to a wearer's head, giving them the unique advantage of capturing their points of view. Hoshen and Peleg have shown that egocentric cameras indirectly capture the wearer's gait, which can be used to identify a wearer based on their egocentric videos. The authors have shown a wearer recognition accuracy of up to 77% over 32 subjects. However, an important limitation of their work is that such gait features can be extracted only from walking sequences of a wearer. In this work, we take the privacy threat a notch higher and show that even the wearer's hand gestures, as seen through an egocentric video, leak wearer's identity. We have designed a model to extract and match hand gesture signatures from egocentric videos. We demonstrate the threat on the EPIC kitchen dataset containing 55 hours of the egocentric videos acquired from 32 subjects doing various activities. We show that: (1) Our model can recognize a wearer with an accuracy of up to 73% based on the same activity, i.e., the model has seen 'cut' activity by a wearer in the train set, and recognizes the wearer based on another 'cut' activity by him/her while testing. (2) The hand gesture signatures transfer across activities, i.e., even if our model does not see 'cut' activity of a wearer at the train time, but sees other activities such as 'wash', 'mix' etc., the model can still recognize a wearer with an accuracy of up to 60%, by matching hand gesture signatures of 'cut' at test time with train time signatures of 'wash' or 'mix'. (3) The hand gesture features even transfer across subjects, i.e., even if the model has not seen any activity by some subject, one can still verify a wearer (open-set), and predict that the same wearer has performed both activities with an Equal Error Rate of 15.21%.

6. JASA

   Deep metric learning for bioacoustic classification: Overcoming training data scarcity using dynamic triplet loss

   Anshul Thakur, Daksh Thapar, Padmanabhan Rajan, and Aditya Nigam

   In The Journal of the Acoustical Society of America 2019

   Abs PDF

   Bioacoustic classification often suffers from the lack of labeled data. This hinders the effective utilization of state-of-the-art deep learning models in bioacoustics. To overcome this problem, the authors propose a deep metric learning-based framework that provides effective classification, even when only a small number of per-class training examples are available. The proposed framework utilizes a multiscale convolutional neural network and the proposed dynamic variant of the triplet loss to learn a transformation space where intra-class separation is minimized and inter-class separation is maximized by a dynamically increasing margin. The process of learning this transformation is known as deep metric learning. The triplet loss analyzes three examples (referred to as a triplet) at a time to perform deep metric learning. The number of possible triplets increases cubically with the dataset size, making triplet loss more suitable than the cross-entropy loss in data-scarce conditions. Experiments on three different publicly available datasets show that the proposed framework performs better than existing bioacoustic classification methods. Experimental results also demonstrate the superiority of dynamic triplet loss over cross-entropy loss in data-scarce conditions. Furthermore, unlike existing bioacoustic classification methods, the proposed framework has been extended to provide open-set classification.

7. EMBC

   Effectiveness of GAN-based Synthetic Samples Generation of Minority Patterns in HEp-2 Cell Images

   Krati Gupta, Daksh Thapar, Arnav Bhavsar, and Anil K. Sao

   In EMBC 2020

   Abs PDF

   In this paper, we present a framework to address the augmentation of images for the rare and minor appearance of mitotic type staining patterns, for Human Epithelium Type2 (HEp-2) cell images. The identification of mitotic patterns among non-mitotic/interphase patterns is important in the process of diagnosis of various autoimmune disorders. This task leads to a pattern classification problem between mitotic v/s interphase. However, among the two classes, typically, the number of mitotic cells are relatively very less. Thus, in this work, we propose to generate synthetic mitotic samples, which can be used to augment the number of mitotic samples and balance the samples of mitotic and interphase patterns in classification paradigm. An effective feature representation is used, to validate the usefulness of the synthetic samples in classification task, along with a subjective validation done by a medical expert. The results demonstrate that the approach of generating and mingling synthetic samples with existing training data works well and yields good performance, with 0.98 balanced class accuracy (BcA) in one case, over a public dataset, i.e., UQ-SNP I3A Task-3 mitotic cell identification dataset.

8. ACCV

   Hierarchical X-Ray Report Generation via Pathology tags and Multi Head Attention

   Preethi Srinivasan, Daksh Thapar, Arnav Bhavsar, and Aditya Nigam

   In ACCV 2020

   Abs PDF Project Page

   Examining radiology images, such as X-Ray images as accurately as possible, forms a crucial step in providing the best healthcare facilities. However, this requires high expertise and clinical experience. Even for experienced radiologists, this is a time-consuming task. Hence, the automated generation of accurate radiology reports from chest X-Ray images is gaining popularity. Compared to other image captioning tasks where coherence is the key criterion, medical image captioning requires high accuracy in detecting anomalies and extracting information along with coherence. That is, the report must be easy to read and convey medical facts accurately. We propose a deep neural network to achieve this. Given a set of Chest X-Ray images of the patient, the proposed network predicts the medical tags and generates a readable radiology report. For generating the report and tags, the proposed network learns to extract salient features of the image from a deep CNN and generates tag embeddings for each patient's X-Ray images. We use transformers for learning self and cross attention. We encode the image and tag features with self-attention to get a finer representation. Use both the above features in cross attention with the input sequence to generate the report's Findings. Then, cross attention is applied between the generated Findings and the input sequence to generate the report's Impressions. We use a publicly available dataset to evaluate the proposed network. The performance indicates that we can generate a readable radiology report, with a relatively higher BLEU score over SOTA.