Markus Marks

We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short noun phrases (e.g., "yellow school bus"), image exemplars, or a combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching object instances. To advance PCS, we build a scalable data engine that produces a high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of an image-level detector and a memory-based video tracker that share a single backbone. Recognition and localization are decoupled with a presence head, which boosts detection accuracy. SAM 3 doubles the accuracy of existing systems in both image and video PCS, and improves previous SAM capabilities on visual segmentation tasks.

Cells are a fundamental unit of biological organization, and identifying them in imaging data is a critical task for various cellular imaging experiments. We present CellSAM, a universal model for cell segmentation that generalizes across diverse cellular imaging data. CellSAM builds on top of the Segment Anything Model by developing a prompt engineering approach for mask generation. We show that this approach allows a single model to achieve human-level performance for segmenting images of mammalian cells, yeast and bacteria collected across various imaging modalities.

Diffusion models are generative models with impressive text-to-image synthesis capabilities. We find that automatically generated captions can improve text-image alignment and significantly enhance a model’s cross-attention maps, leading to better perceptual performance. Our approach improves upon the current state-of-the-art in diffusion-based semantic segmentation on ADE20K and the current overall SOTA for depth estimation on NYUv2. Furthermore, our method generalizes to the cross-domain setting, achieving SOTA results on Watercolor2K, Dark Zurich-val, and Nighttime Driving.

We introduce MABe22, a multi-species multi-task benchmark for learned representations of behavior. The benchmark encompasses behavioral data from flies, mice, and rats across multiple experimental paradigms, providing a standardized evaluation framework for self-supervised and unsupervised methods in computational ethology.

Quantification of behaviours of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. We present SIPEC, a novel deep learning pipeline that performs segmentation, identification, pose estimation and classification of complex behaviour for freely moving individual mice as well as socially interacting non-human primates in three dimensions, using data only from simple mono-vision cameras in home-cage set-ups.

We propose rPU-VAE, a method for robust disentanglement of a few factors at a time. Our approach addresses the challenge of learning disentangled representations in an unsupervised fashion by progressively uncovering independent factors of variation. We demonstrate improved disentanglement performance and robustness compared to existing variational autoencoder approaches on standard benchmarks.