Stubbisms – Tony’s Weblog

February 18, 2009

Fighting Scala – Scala to Java List Conversion

Filed under: Scala — Tags: , , — Antony Stubbs @ 2:42 pm

Apparently this is going to be addressed in Scala 2.8, but until then there’s an annoying little detail when dealing with Java libraries (in my case, Wicket), from Scala. That being that methods requiring java.util.List types cannot be called with scala.List types.

The offending code:

val issues = List( 1, 2, 3 )
new org.apache.wicket.markup.html.list.ListView(wicketId, issues) {...

3

Causes this compilation error:
error: overloaded method constructor ListView with alternatives (java.lang.String,java.util.List[T])org.apache.wicket.markup.html.list.ListView[T] (java.lang.String,org.apache.wicket.model.IModel[java.util.List[T]])org.apache.wicket.markup.html.list.ListView[T] cannot be applied to (String,List[String])

There is a collection of implicit conversion functions for going from java.util.List to scala.List in the scala.collection.jcl.Conversions object, but not go go the other way around. These functions look like:

object Conversions {

implicit def convertSet[T](set : java.util.Set[T]) = Set(set)
implicit def convertList[T](set : java.util.List[T]) = Buffer(set)
implicit def convertSortedSet[T](set : java.util.SortedSet[T]) = SortedSet(set)
implicit def convertMap[T,E](set : java.util.Map[T,E]) = Map(set)
implicit def convertSortedMap[T,E](set : java.util.SortedMap[T,E]) = SortedMap(set)</code>

implicit def unconvertSet[T](set : SetWrapper[T]) = set.underlying
implicit def unconvertCollection[T](set : CollectionWrapper[T]) = set.underlying
implicit def unconvertList[T](set : BufferWrapper[T]) = set.underlying
implicit def unconvertSortedSet[T](set : SortedSetWrapper[T]) = set.underlying
implicit def unconvertMap[T,E](set : MapWrapper[T,E]) = set.underlying
implicit def unconvertSortedMap[T,E](set : SortedMapWrapper[T,E]) = set.underlying

}

22

With some friendly advice, I created the following conversion functions to do the conversion, including 2-dimensional (lists of lists) lists.

implicit def convertScalaListToJavaList(aList:List[String]) = java.util.Arrays.asList(aList.toArray: _*)
implicit def convertScalaListListToJavaList(aList:List[List[String]]) = java.util.Arrays.asList(aList.toArray: _*)

11

The obscure notation (aList.toArray: _*) is required as described in section 8.8 Repeated Parameters page 188 of the Programming in Scala book:

“This notation tells the compiler to pass each element of arr as its own argument to echo, rather than all of it as a single argument.”

and in 4.6.2 Repeated Parameters in The Scala Language Specification Version 2.7:

Furthermore, assume the definition:
def sum(args: Int*)
val xs = List(1, 2, 3)

The following applications method sum is ill-formed:
sum(xs) // ***** error: expected: Int, found: List[Int]
By contrast, the following application is well formed and yields again the result 6:
sum(xs: _*)

To cut the story short, it is required because the asList method accepts variable number of object arguments, but we are passing in a single list object. The : _* tells the compiler to instead pass in each element of the list individually, so that it looks like varargs. And as suggested by someone on the channel, it is consistent, if obscure, notation. When you read it, keep in mind that variableName: ObjectType is normal notation in Scala for specifying the type of a declared parameter. So instead of a concrete type, we are specifying “any” type ( ‘_’ ), “any” number of times ( ‘*’ ).

Not so bad eh? ;P

14 Comments »

  1. We stumbled on the same problem as you, while developing a Wicket application with Scala. If you make your implicit function typed, it can also be used with other generic Java libraries, and ready for generics in Wicket 1.4! Our function actually is just about the same:

    implicit def listToJavaList[T](l: List[T]) = java.util.Arrays.asList(l.toArray: _*)

    We had a small discussion in our group of what the most efficient implementation of the conversion would be. Do you have any idea if for instance creating the list with foldleft would be more efficient?

    Comment by stoyle — February 18, 2009 @ 7:21 pm

  2. Hi,

    The most efficient implementation will depend on how you intend to use the converted list. For example, if you don’t need mutation, the initial cost will be lowest if you have an implementation of java.util.List that delegates to scala.List. However, you then have a linked list and that has different performance characteristics than ArrayList or Arrays.asList.

    If you want an array-backed list, then the suggested approach is decent but it means that methods like add throw an UnsupportedOperationException. If you want the ability to add to the converted list, you should just manually add each item into a java.util.ArrayList.

    Ismael

    Comment by Ismael Juma — February 18, 2009 @ 11:00 pm

  3. Yes, this has screaming efficiency problems. The dude on #scala didn’t elaborate as to how 2.8 will address it, but I hope it’s not what I’ve described!

    I don’t see how foldLeft could be more effective, or even applied to this problem – doesn’t fold left reduce lists to single elements?

    It’s a shame isn’t it? Groovy wins out here for interoperability because I think it simply extends java.util.List doesn’t it?

    Comment by Antony Stubbs — February 19, 2009 @ 2:10 am

  4. I would not say it has “screaming efficiency problems”. Copying lists/arrays is very common in Java as a defensive programming mechanism. In any case, it’s nice to avoid unnecessary copies and in some cases (e.g. large lists), it may even be very important. In such cases, a java.util.List implementation that simply delegates to the backing scala.List is probably the way to go. Note, however, that for very large lists, you may want to skip scala.List altogether and use an array-backed structure to avoid the memory overhead of the list nodes.

    They haven’t really said what approach will be taken for 2.8.0 and the new collection library is still in flux so we’ll have to wait and see. Martin listens to user feedback though, so if the proposed solution has issues we can always suggest changes.

    Comment by Ismael Juma — February 19, 2009 @ 2:21 am

  5. You can use foldLeft like this:

    implicit def listToJavaList[T](l: Seq[T]) = l.foldLeft(new java.util.ArrayList[T](l.size)){(al, e) => al.add(e); al}

    This function even returns an ArrayList. I’ve also changed the input param to Seq, so that you may use the function to convert Lists, Arrays, ListBuffers, ArrayBuffers, Stacks and even Strings (if that makes sense).

    Since foldLeft is defined on Iterable, you could even use that as input param. But converting Sets and Maps to an ArrayList may not make sense.

    As with regards to performance, this is probably not optimal. However in applications I usually develop, IO is normally is a bigger performance sinner than array conversions.

    Comment by Alf — February 19, 2009 @ 4:42 am

  6. If efficiency is a concern and it’s okay to maintain the immutable nature of the original list, it’s still a tiny bit of code to “fight” Scala. By the way, why is this fighting? There are two different data structures, both called List, and you want to convert from one to the other. Is it “fighting” Java when you convert from a set to a list? Anyway, here’s the code:

    import java.util.{AbstractList, Iterator}

    case class ListWrapper[T](list : List[T]) extends AbstractList[T] {
    def get(i : Int) = list(i)
    def size : Int = list.size

    // the default AbstractList version of iterator gets the size and then calls
    // get(0), get(1), etc. But that doesn’t work so well with a linked list. The
    // better way to do an iterator on a linked list is to keep track of the
    // portion of the list that hasn’t been iterated over
    override def iterator : Iterator[T] =
    new Iterator[T] {
    private var remainder = list

    def remove = throw new UnsupportedOperationException(“can’t remove from an immutable list”)
    def next : T = remainder match {
    case head :: tail => {
    remainder = tail
    head
    }
    case _ => throw new IndexOutOfBoundsException(“tail of an empty list”)
    }
    def hasNext = !remainder.isEmpty
    }

    // case classes get a hashCode and equals, but we need to follow the java.util.List standard –
    // the case class generated versions would be too strict. We’ll just use the versions defined
    // in AbstractList
    override def hashCode = super.hashCode
    override def equals(other : Any) = super.equals(other)
    }

    Comment by jamesiry — February 20, 2009 @ 3:34 am

  7. I called it fighting because it was a struggle to get it to work, as I’m still learning Scala, and hadn’t encountered this problem. Also, I had expected Scala’s List to be based on Java’s and so interoperable, as are String, Integer etc etc… So was surprised that id didn’t work.

    Also, the compiler error didn’t give the fully qualified name when referring to Scala’s List (e.g. “cannot be applied to (String,List[String])” instead of scala.List), so that wasn’t absolutely clear that that was the problem to begin with either.

    Add to that the existing implicit conversions from Java to Scala lists didn’t include one to do the other way around.

    Thanks for the code! Could you give a summary of the main difference/advantage over:
    implicit def convertScalaListToJavaList(aList:List[String]) = java.util.Arrays.asList(aList.toArray: _*)
    ?

    Thanks for your input! – Cheers!

    Comment by Antony Stubbs — February 20, 2009 @ 3:48 am

  8. Antony,

    The code that James Iry provided is what I meant by “if you don’t need mutation, the initial cost will be lowest if you have an implementation of java.util.List that delegates to scala.List. However, you then have a linked list and that has different performance characteristics than ArrayList or Arrays.asList”. The main advantage is that no iteration or copying is required.

    Ismael

    Comment by Ismael Juma — February 20, 2009 @ 3:53 am

  9. Hmm, interesting. Is there much you miss out on by not using the scala.List ? (when using it in Scala that is)

    Comment by Antony Stubbs — February 20, 2009 @ 4:50 am

  10. Unfortunately, Scala’s List type *cannot* implement java.util.List. This is because the Java List is inextricably tied to the concept of mutability. Think about it, how would scala.List implement methods like `void add(T)`? Really, the best that can be done is a conversion.

    On that note, I would be willing to bet that foldLeft is going to give you better performance than `toList compose isArray`. There is no need to actually convert the list into an array prior to converting it into a Java List. On the flip side, ArrayList might just use the passed array directly, in which case there will be no performance gain whatsoever from foldLeft, and it may even hurt performance on larger lists.

    Comment by Daniel Spiewak — February 22, 2009 @ 5:39 am

  11. Note that Arrays.asList does not return a java.util.ArrayList. It uses a special implementation that uses the passed in array without copying. It would be possible to implement java.util.List since UnsupportedOperationException is allowed for mutations, but that is ugly. Note that Martin is considering to make scala’s Iterable/Iterator subtypes of Java’s[1] and the Iterator case has a similar issue although for a single method.

    [1] http://www.nabble.com/Re%3A-How-to-set-the-scale-for-scala.BigDecimal%27s—method–p20732644.html

    Comment by Ismael Juma — February 22, 2009 @ 5:48 am

  12. There are several helpers in javautils just append asJava:

    import org.scala_tools.javautils.Implicits._

    new org.apache.wicket.markup.html.list.ListView(wicketId, issues asJava) {…

    org.scala-tools
    javautils
    2.7.4-0.1

    Comment by kurtharriger — February 8, 2010 @ 1:38 pm

  13. […] a chance to apply closures and FP much * There are some gotchas you need to be aware of like this: Fighting Scala – Scala to Java List Conversion – for the time being I managed using Java collections for domain object […]

    Pingback by The Basement Coders » Scala + Wicket — August 24, 2010 @ 4:28 am

  14. btw, this was added in 2.8: Conversions between Java and Scala collections http://www.scala-lang.org/docu/files/collections-api/collections_46.html

    resolves the issue IMHO

    Comment by joseph — August 8, 2011 @ 11:13 am


RSS feed for comments on this post. TrackBack URI

Leave a comment

Blog at WordPress.com.