Category: Programing

build release version for Storm

Mingmin1 Comment

Compared with the branches of Storm source code in GitHub, Storm doesn’t have any release package for version 1.1.* and 2.*. So I decide to build it from source code, to test the new features on Storm 2.*.

Step 1. clone source code from github

git clone https://github.com/apache/storm.git master

Step 2. build the project with Maven

cd  storm
mvn clean install -DskipTests=true # ignore test to save time

Step 3. create the binary distribution

cd storm-dist/binary
mvn package

The .zip and .tar.gz binary archives will be created in the storm-dist/binary/final-package/target/ sub-directory.

$ ls -lth final-package/target/
total 1071952
-rw-r--r--  1 mingmxu  110050932   801B Jan 26 12:23 apache-storm-2.0.0-SNAPSHOT.zip.asc
-rw-r--r--  1 mingmxu  110050932   801B Jan 26 12:23 apache-storm-2.0.0-SNAPSHOT.tar.gz.asc
-rw-r--r--  1 mingmxu  110050932   3.3K Jan 26 12:23 apache-storm-2.0.0-SNAPSHOT.pom
-rw-r--r--  1 mingmxu  110050932   801B Jan 26 12:23 apache-storm-2.0.0-SNAPSHOT.pom.asc
-rw-r--r--  1 mingmxu  110050932   262M Jan 26 12:23 apache-storm-2.0.0-SNAPSHOT.zip
drwxr-xr-x  8 mingmxu  110050932   272B Jan 26 12:23 archive-tmp
-rw-r--r--  1 mingmxu  110050932   262M Jan 26 12:23 apache-storm-2.0.0-SNAPSHOT.tar.gz
drwxr-xr-x  3 mingmxu  110050932   102B Jan 26 12:11 maven-shared-archive-resources

Issue I meet and solutions:

1. Could not find artifact org.apache.storm:storm-maven-plugins:jar:1.1.0-SNAPSHOT

solution: I change the version used in storm-core/pom.xml directly to an available version.

@@ -875,7 +875,7 @@
             <plugin>
                <groupId>org.apache.storm</groupId>
                <artifactId>storm-maven-plugins</artifactId>
-               <version>${project.version}</version>
+               <version>1.0.2</version>
                <executions>
                  <execution>
                    <id>version-info</id>
@@ -970,6 +970,41 @@

2. Failed to execute goal org.apache.maven.plugins:maven-gpg-plugin:1.6:sign (default) on project final-package: Unable to execute gpg command: Error while executing process. Cannot run program “gpg”: error=2, No such file or directory

solution: You need to install GPG. In Mac, I install GPG suit, and generate a new key pair in GPG Keychain.

heap and non-heap memory in JVM

Mingmin1 Comment

The JVM memory consists of the following segments:

  • Heap Memory, which is the storage for Java objects
  • Non-Heap Memory, which is used by Java to store loaded classes and other meta-data
  • other, JVM code itself, JVM internal structures, loaded profiler agent code and data, etc.

Heap

The JVM has a heap that is the runtime data area from which memory for all class instances and arrays are allocated. It is created at the JVM start-up.

The heap size may be configured with the following VM options:

    -Xmx<size> - to set the maximum Java heap size
    -Xms<size> - to set the initial Java heap size

By default, the maximum heap size is 64 Mb.

Heap memory for objects is reclaimed by an automatic memory management system which is known as a garbage collector. The heap may be of a fixed size or may be expanded and shrunk, depending on the garbage collector’s strategy.

Non-Heap

Also, the JVM has memory other than the heap, referred to as non-heap memory. It is created at the JVM startup and stores per-class structures such as runtime constant pool, field and method data, and the code for methods and constructors, as well as interned Strings.

Unfortunately, the only information JVM provides on non-heap memory is its overall size. No detailed information on non-heap memory content is available.

The abnormal growth of non-heap memory size may indicate a potential problem, in this case you may check up the following:

  • If there are class loading issues such as leaked loaders.
  • If there are strings being massively interned. For detection of such problem, Object allocation recording may be used.

If the application indeed needs that much of non-heap memory and the default maximum size of 64 Mb is not enough, you may enlarge the maximum size with the help of -XX:MaxPermSize VM option. For example, -XX:MaxPermSize=128m sets the size of 128 Mb.