Cancer cells degrade the extracellular matrix through actin-rich protrusions termed invadopodia. The formation of functional invadopodia requires polarized membrane trafficking driven by Rho GTPase-mediated cytoskeletal remodeling. We identify the Rho GTPase-activating protein Deleted in Liver Cancer 3 (DLC3) as an integral component of the endosomal transport and sorting machinery. We provide evidence for the direct regulation of RhoB by DLC3 at endosomal membranes to which DLC3 is recruited by interacting with the sorting nexin SNX27. In TGF-$\beta$-treated MCF10A breast epithelial cells, DLC3 knockdown enhanced metalloproteinase-dependent matrix degradation, which was partially rescued by RhoB co-depletion. This was recapitulated in MDA-MB-231 breast cancer cells in which early endosomes demonstrated aberrantly enriched F-actin and accumulated the metalloproteinase MT1-MMP upon DLC3 knockdown. Remarkably, Rab4 downregulation fully rescued the enhanced matrix degradation of TGF-$\beta$-treated MCF10A and MDA-MB-231 cells. In summary, our findings establish a novel role for DLC3 in the suppression of MT1-MMP-dependent matrix degradation by inactivating RhoB signaling at endosomal membranes. We propose that DLC3 function is required to limit endosomal actin polymerization, Rab4-dependent recycling of MT1-MMP and, consequently, matrix degradation by invadopodial activity.