spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Jain <>
Subject Using custom class as a key for groupByKey() or reduceByKey()
Date Sun, 15 Jun 2014 15:45:51 GMT
I have a simple Java class as follows, that I want to use as a key while
applying groupByKey or reduceByKey functions:

private static class FlowId {
		public String dcxId;
		public String trxId;
		public String msgType;
		public FlowId(String dcxId, String trxId, String msgType) {
			this.dcxId = dcxId;
			this.trxId = trxId;
			this.msgType = msgType;
		public boolean equals(Object other) {
			if (other == this) return true;
			if (other == null) return false;
			if (getClass() != other.getClass()) return false;
			FlowId fid = (FlowId) other;
			if (this.dcxId.equals(fid.dcxId) && this.trxId.equals(fid.trxId) &&
					this.msgType.equals(fid.msgType)) {
				return true;
			return false;

I figured that an equals() method would need to be overridden to ensure
comparison of keys, but still entries with the same key are listed
separately after applying a groupByKey(), for example. What further
modifications are necessary to enable usage of above class as a key. Right
now, I have fallen back to using Tuple3<String, String, String> instead of
the FlowId class, but it makes the code unnecessarily verbose.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message