K-means clustering of lines for big data

Yair Marom, Dan Feldman

Research output: Contribution to journalConference articlepeer-review

Abstract

The input to the k-mean for lines problem is a set L of n lines in Rd, and the goal is to compute a set of k centers (points) in Rd that minimizes the sum of squared distances over every line in L and its nearest center. This is a straightforward generalization of the k-mean problem where the input is a set of n points instead of lines. We suggest the first PTAS that computes a (1 + e)-approximation to this problem in time O(n log n) for any constant approximation error e ? (0, 1), and constant integers k, d = 1. This is by proving that there is always a weighted subset (called coreset) of dkO(k) log(n)/e2 lines in L that approximates the sum of squared distances from L to any given set of k points. Using traditional merge-and-reduce technique, this coreset implies results for a streaming set (possibly infinite) of lines to M machines in one pass (e.g. cloud) using memory, update time and communication that is near-logarithmic in n, as well as deletion of any line but using linear space. These results generalized for other distance functions such as k-median (sum of distances) or ignoring farthest m lines from the given centers to handle outliers. Experimental results on 10 machines on Amazon EC2 cloud show that the algorithm performs well in practice. Open source code for all the algorithms and experiments is also provided.

Original languageAmerican English
JournalAdvances in Neural Information Processing Systems
Volume32
StatePublished - 2019
Event33rd Annual Conference on Neural Information Processing Systems, NeurIPS 2019 - Vancouver, Canada
Duration: 8 Dec 201914 Dec 2019

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Fingerprint

Dive into the research topics of 'K-means clustering of lines for big data'. Together they form a unique fingerprint.

Cite this