[Ikvm-developers] IKVM classloader mechanism + loading native DLLs with JNA

[voiceinsight]

Hi,

First let me say that I find IKVM extremely useful. Also, I was extremely
impressed to see it run an application of mine (several jar's, loads native
DLLs through JNA) on the first try ! So: Jeroen, thanks a million!

Now to the issues: I am new to .NET, I know Java (not confident with class
loaders). I am unclear about IKVM's class loading mechanisms. I have read
(though perhaps not fully understood)
http://weblog.ikvm.net/PermaLink.aspx?guid=8a457a80-1e5f-4182-8f78-b2cd67845553
. I'd like to find a fix for the issues I'm encountering, described below.
I'm using the latest stable release 0.36.0.11

I use several methods to execute my application. To start with, I looked at
bin/starter.cs and took inspiration from it to run my application. In all
cases I ikvmc-compile everything, referencing some Dll's I need, and
specifying "-classloader:ikvm.runtime.ClassPathAssemblyClassLoader" (not
that I exactly know why I'm doing that...).

- method 1: (works fine up to a point, see below) set java classpath with
Startup.setProps(); use reflection methods to construct and manipulate Java
objects (to do this I use java.lang.Class.forName("com.mynamespace.MyClass",
true, ClassLoader.getSystemClassLoader() - which I got from starter.cs, I'm
not sure I'm using it correctly)
- method 2: set java classpath with Startup.setProps(); reference
ikvmc-generated Dlls in project, and directly use Java objects on .Net code
- method 3: set java classpath with java.lang.System.setProperty() from
within .Net; reference ikvmc-generated Dlls in project, and directly use
Java objects on .Net code

To work further with method 1, my issue is that I can't seem to cast an
object obtained through Constructor.newInstance() into its actual Java type.
If I try
----
object c = constructor.newInstance(null); // creates a MyClass instance
MyClass c2 = (MyClass) c;
----
.NET will complain on the second line with
----
System.InvalidCastException: Unable to cast object of type
'com.mynamespace.MyClass' to type 'com.mynamespace.MyClass'.
----

With method 2 and 3, my issue is the following. I use JNA (www.jna.org) to
access a native library in my Java project. The native library also performs
callbacks to my Java code. All of this works fine using method 1, provided I
reference ikvm-native.dll in my Visual Studio project. However, it won't
work any more if I do one of the following
- use method 1': set java classpath through java.lang.System.setProperty
from within .NET (instead of using Startup.setProps())
- use my methods 2 or 3: i.e. replace reflection by direct Java object
manipulation in .NET
In either case, it will fail on the first callback the native Dll performs,
with the following error message:
----
*** exception in native code ***
System.AccessViolationException: Attempted to read or write protected
memory. Th
is is often an indication that other memory is corrupt.
at com.sun.jna.Pointer._getInt(Int64 l)
java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Method.java:625)
at
com.sun.jna.CallbackReference$DefaultCallbackProxy.callback_inner(Cal
lbackReference.java:272)
at
com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackR
eference.java:294)
at IKVM.Runtime.JNIEnv.InvokeHelper(Unknown Source)
at IKVM.Runtime.JNIEnv.CallObjectMethodA(Unknown Source)
Caused by: cli.System.AccessViolationException: Attempted to read or write
prote
cted memory. This is often an indication that other memory is corrupt.
at com.sun.jna.Pointer._getInt(Native Method)
at com.sun.jna.Pointer.getInt(Pointer.java:425)
at com.sun.jna.Structure.readField(Structure.java:382)
at com.sun.jna.Structure.read(Structure.java:278)
at com.sun.jna.Structure.readField(Structure.java:365)
at com.sun.jna.Structure.read(Structure.java:278)
at com.sun.jna.Structure.toArray(Structure.java:998)
at com.mynamespace.MyClass.extractArray(Dll.java:338)
----

The reason why I tried to replace Startup.setProps() by
java.lang.System.setProperty() for the java classpath is that it seemed to
solve an earlier issue I had, namely when using log4j, I would get
----
log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to
a "
org.apache.log4j.Appender" variable.
log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
log4j:ERROR [log4j-1.2.14, Version=0.0.0.0, Culture=neutral,
PublicKeyToken=null
] whereas object of type
log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
[sun.misc.Launcher$
***@55c1be].
log4j:ERROR Could not instantiate appender named "A1".
----
My log4j.properties file is
----
# Set root logger appender to A1.
log4j.rootLogger=ALL, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender

# Print the date in ISO 8601 format
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d %-5p %c.%M:%L - %m%n
----
Why are there several loaders for these classes ? I'm not sure I understand
the mechanisms...

What is the difference between setting the java classpath one way or the
other ?
What is the difference (in terms of classloaders etc perhaps) between
manipulating java objects using reflection or directly ? Why is there an
impact on JNI/JNA ?
Does it make a difference if I -reference:ikvm-native.dll when generating
the Dll using ikvmc ? I do it, but I'm not sure whether I should.
What is the effect of Startup.enterMainThread() ?


Please ask back for details or a worked-out test case if I'm unclear. I'll
need to work myself through this at any rate, and using IKVM seems the most
promising tool to use. Especially, seeing that it can actually run my app
(JNA/JNI code included) under some circumstances, I'm quite confident there
is a way to stabilize it when using another paradigm.

Thanks.

Sebastien

[Jeroen Frijters]

Post by voiceinsight
I use several methods to execute my application. To start with, I
looked at bin/starter.cs and took inspiration from it to run my
application. In all cases I ikvmc-compile everything, referencing
some Dll's I need, and specifying
"-classloader:ikvm.runtime.ClassPathAssemblyClassLoader" (not
that I exactly know why I'm doing that...).

ClassPathAssemblyClassLoader is only needed rarely. Its primary purpose it to allow code that loads configuration files from the classpath.

Post by voiceinsight
To work further with method 1, my issue is that I can't seem to cast an
object obtained through Constructor.newInstance() into its actual Java type.
If I try
----
object c = constructor.newInstance(null); // creates a MyClass instance
MyClass c2 = (MyClass) c;
----
.NET will complain on the second line with
----
System.InvalidCastException: Unable to cast object of type
'com.mynamespace.MyClass' to type 'com.mynamespace.MyClass'.
----

This means that you have two version of the class. It can be either in two different assemblies, or one can be in an ikvmc generated assembly and another loaded dynamically at runtime from a .class file (or from a jar).

Post by voiceinsight
*** exception in native code ***
System.AccessViolationException: Attempted to read or write protected
memory.

This is either a bug in JNA or in IKVM's JNI implementation. If you have a repro scenario that you can send me, I'd like to have a look.

Post by voiceinsight
Why are there several loaders for these classes ? I'm not sure I
understand the mechanisms...

My guess would be that you are using both statically compiled and dynamically compiled versions of the same code. In virtually all scenarios, you should be able to run your ikvmc compiled code without having any of the original .class or jar files around (and also without setting the classpath in any way).

Post by voiceinsight
What is the difference between setting the java classpath one way or
the other ?

In principle nothing, but in practice if you set it too late it (i.e. if the system class loader has already initialized) it will be ignored.

Post by voiceinsight
What is the difference (in terms of classloaders etc perhaps) between
manipulating java objects using reflection or directly ?

There is no difference, other than the fact that directly (i.e. from C#) you can only access statically compiled classes and with reflection you can also load them dynamically from their Java from (as .class or jar files).

Post by voiceinsight
Does it make a difference if I -reference:ikvm-native.dll when
generating the Dll using ikvmc ?

No. ikvm-native.dll is loaded by the .NET runtime. It should be in the PATH or in the directory where the main application executable lives.

I hope this helps. Please let me know if you have more questions, or if some of the above wasn't clear.

Regards,
Jeroen

[voiceinsight]

Post by Jeroen Frijters

Post by voiceinsight
To work further with method 1, my issue is that I can't seem to cast an
object obtained through Constructor.newInstance() into its actual Java type.
If I try
----
object c = constructor.newInstance(null); // creates a MyClass instance
MyClass c2 = (MyClass) c;
----
.NET will complain on the second line with
----
System.InvalidCastException: Unable to cast object of type
'com.mynamespace.MyClass' to type 'com.mynamespace.MyClass'.
----

This means that you have two version of the class. It can be either in two
different assemblies, or one can be in an ikvmc generated assembly and
another loaded dynamically at runtime from a .class file (or from a jar).

Post by voiceinsight
Why are there several loaders for these classes ? I'm not sure I
understand the mechanisms...

My guess would be that you are using both statically compiled and
dynamically compiled versions of the same code. In virtually all
scenarios, you should be able to run your ikvmc compiled code without
having any of the original .class or jar files around (and also without
setting the classpath in any way).

Thanks for your answer. You were indeed right, and I found out that I was
dynamically reading from JAR files which were on the JAva class path.

Now, I am still pursuing several avenues.
- compiling all my Jars to DLLs, and using Java coding in C#: this avenue is
blocked by the System.AccessViolationException error I mentioned; I will try
isolate a test case for you to look into
- dynamically loading from JARs and using reflection all the way. This way
is blocked by issue 1 stated below.
- would it make sense to work directly use Java classes in C# (after having
ikvmcompiled the Jars), and then remove the JAva DLLs at runtime to rely on
dynamically loading the classes from Jars ?
- if I want to work with ikvmcompiled Jars and reflection: what classloader
mechanism should I specify/how in order to have my classes loaded from the
DLLs ?


ISSUE 1:
I would like to call .NET from Java, compile this and run it from .NET
again. Here is what I do, and it doesn't work.

----- Ant script to generate DLL from Java jar (${dll} refers to DLL
generated from MyDotNetClass, called IKVMTestCase2DLL)
<target name="IKVMTestCaseJava2.dll" depends="jar">
<exec executable="${ikvm}/bin/ikvmc.exe" dir=".">
<arg value="-out:${execproject}/IKVMTestCaseJava2.dll" />
<arg value="-debug" />
<arg value="-classloader:ikvm.runtime.ClassPathAssemblyClassLoader" />
<arg value="-reference:${dll}" />
<arg file="${ant.project.name}.jar" />
</exec>
</target>
----- MyDotNetClass
using System;
using System.Collections.Generic;
using System.Text;

namespace IKVMTestCase2DLL
{
public class MyDotNetClass
{
public MyDotNetClass()
{
Console.WriteLine("in MyDotNetClass constr");
}
}
}
---- MyJavaClass
package com.mynamespace;

public class MyJavaClass {
private cli.IKVMTestCase2DLL.MyDotNetClass in;
public MyJavaClass(cli.IKVMTestCase2DLL.MyDotNetClass in) {
System.out.println("in MyJavaClass constr");
this.in = in;
new cli.IKVMTestCase2DLL.MyDotNetClass();
}
}
----- DotNetStarter, my launcher
using System;
using System.Collections.Generic;
using System.Text;
using IKVM.Internal;
using ikvm.runtime;
using System.Collections;
using java.lang;
using java.lang.reflect;

namespace IKVMTestCase2
{
class DotNetStarter
{
static int Main(string[] args)
{
Tracer.EnableTraceForDebug();

Hashtable props = new Hashtable();
props["java.class.path"] = "C:/Documents and
Settings/Sebastien/My
Documents/workspace/IKVMTestCase2/IKVMTestCaseJava2.jar";
Startup.setProperties(props);

Class clazz =
java.lang.Class.forName("com.mynamespace.MyJavaClass", true,
ClassLoader.getSystemClassLoader());
Console.WriteLine("have class" + clazz);
Constructor constructor = clazz.getConstructor(new Class[] {
java.lang.Class.forName("cli.IKVMTestCase2DLL.MyDotNetClass", false,
ClassLoader.getSystemClassLoader()) });
Console.WriteLine("found constructor ");

object javaDotNetXMLlet = constructor.newInstance(new object[] {
new IKVMTestCase2DLL.MyDotNetClass() });

return 0;
}
}
}
----- console output on launching DotNetStarter (I have IKVMTestCase2DLL
containing MyDotNetClass, and IKVMTestCaseJava2.dll, containing the
CLI-compiled version of MyJavaClass, referenced in my Visual Studio Project;
in addition, note that IKVMTestCase2DLL is referenced from
IKVMTestCaseJava2.dll)
[15:32:18.91973 ] Core assembly:
C:\WINDOWS\assembly\GAC\IKVM.OpenJDK.ClassLibra
ry\0.36.0.11__13235d27fcbfff58\IKVM.OpenJDK.ClassLibrary.dll
[15:32:18.92975 ] Remapping type System.Exception to java.lang.Throwable
[15:32:18.92975 ] Remapping type System.String to java.lang.String
[15:32:18.92975 ] Remapping type System.IComparable to java.lang.Comparable
[15:32:18.92975 ] Remapping type System.Object to java.lang.Object
[15:32:19.77096 ] constructing JavaTypeImpl for com.mynamespace.MyJavaClass
[15:32:19.81101 ] Class not found: cli.IKVMTestCase2DLL.MyDotNetClass
[15:32:19.82103 ] Class not found: cli.IKVMTestCase2DLL.MyDotNetClass
[15:32:19.85107 ] Finishing: com.mynamespace.MyJavaClass
have classclass com.mynamespace.MyJavaClass

Unhandled Exception: java.lang.ClassNotFoundException:
cli.IKVMTestCase2DLL.MyDo
tNetClass
-----

I can't understand why this class cannot be found while it is referenced in
the Visual Studio Project, and while it is referenced while creating the
IKVMTestCaseJava2.dll.

ISSUE 2:
Situation: Jars compiled to DLLs; using direct Java calls in C#

I am seeing that
- if early on in my .NET code I print out
ClassLoader.getSystemClassLoader(), I get a sun.misc.Launcher; this seems to
be used in java.lang.Class.forName("com.mynamespace.MyClass",
ClassLoader.getSystemClassLoader())
- however, if I enable tracing with
IKVM.Internal.Tracer.EnableTraceForDebug(), I see that all the other
resources are loaded by ikvm.runtime.ClassPathAssemblyClassLoader, in
particular when I construct Java objects directly in C# code.

Could this be the reason for some of my class mismatch problems ? How can I
force the classloaders to be the same, or to not vary with time ? Is this
something transient that is happening during the system bootstrap ? Could I
somehow make sure reflection uses the constructed
ikvm.runtime.ClassPathAssemblyClassLoader that is also used to load the
classes directly called from within C# code ?

Thank you for your help.

Sebastien

[Jeroen Frijters]

Post by voiceinsight
I would like to call .NET from Java, compile this and run it from .NET
again. Here is what I do, and it doesn't work.

This will only work if the Java class loader has the assembly class loader as one of its parents.

In you example you could do that by explicitly creating your own class loader and parenting that to the assembly class loader:

Class c = typeof(DotNetStarter);
URLClassLoader loader = new URLClassLoader(new URL[] { new URL("file:///c:/.../IKVMTestCaseJava2.jar") }, c.getClassLoader());
Class clazz =
java.lang.Class.forName("com.mynamespace.MyJavaClass", true, loader);

Now MyJavaClass will see the classes in the DotNetStarter assemblies (and the assemblies it references).

I want to stress that I don't really recommend this approach, because static compilation is *much* more efficient.

Regards,
Jeroen