Monthly Archives: August 2015

Relational to JSON with Node.js(

Node.js is a platform for building JavaScript based applications that has become very popular over the last few years. To best support this rapidly growing community, Oracle developed a new driver for Node.js which was released in January of this year as a preview release. Written in C and utilizing the Oracle Instant Client libraries, the Node.js driver is both performant and feature rich.

When it comes to generating JSON, Node.js seems like a natural choice as JSON is based on JavaScript objects. However, it’s not exactly fair to compare a solution crafted in Node.js to the PL/SQL solutions as Node.js has a clear disadvantage: it runs on a separate server. That means we have to bring the data across the network before we can transform it to the desired output. Whatever, let’s compare them anyway! 😀

Please Note: This post is part of a series on generating JSON from relational data in Oracle Database. See that post for details on the solutions implemented below as well as other options that can be used to achieve that goal.

Solution 1 – Emulating the PL/SQL solutions

This first solution follows the same flow as the PL/SQL based solutions. We start off by making a connection to the database and then begin creating the object we need by fetching data from the departments table. Next we go to the locations table, then the regions table, and on and on until finally we have the object we want. We then use JSON.stringify to serialize the object into the JSON result we are after.


When called with the department id set to 10, the function returns the serialized JSON that matches the goal 100%.

So we got the output we were after, but what about performance? Let’s explore the test results…

Test results

When I finished the solutions for PL/JSON and APEX_JSON, I wanted to see if one was faster than the other. I ran a very simple test: I generated the JSON for all 27 departments in the HR schema 100 times in a loop (2,700 invocations of the solution). APEX_JSON finished in around 3.5 seconds while PL/JSON took 17 seconds. How long did it take the Node.js solution from above to do this? 136 seconds! What??? Node.js must be slow, right? Well, not exactly…

You may have noticed that, on line 5, I used the base driver class to get a connection to the database. If I was just doing this once the code would be fine as is, but 2,700 times? That’s not good. In fact, that’s really bad! But what’s a Node.js developer to do?

Using a connection pool

The Node.js driver has supported connection pools from the beginning. The idea is that, rather than incur the cost of making a connection to the database each time we need one, we’ll just grab a connection from a pool of connections that have already been established and release it back to the pool when we’re done with it. Best of all, this is really simple!

Here’s a new module that I’ll use to create, store, and fetch the connection pool:

With that in place, I can update the test code to open the connection pool prior to starting the test. Then I just need to update the solution to use the connection pool:

That’s it! I just swapped out the driver module for the pool module and used it to get a connection instead. How much did using the connection pool help? With that one simple change the test completed in 21.5 seconds! Wow, that’s just a little longer than it took PL/JSON. So Node.js is still slow, right? Well, not exactly… 😉

Solution 2 – Optimizing for Node.js

Remember when I said that the solution mimicked the PL/SQL code? There are two major problems with doing this in Node.js:

  1. Excessive round trips: Each query we’re executing is a round trip to the database. In PL/SQL, this is just a context switch from the PL/SQL to the SQL engine and back. I’m not saying we shouldn’t minimize context switches in PL/SQL, we should, but in Node.js this is a round trip across the network! By using the database to do some joins we can reduce the number of queries from 7 to 3.
  2. Sequential execution: The way the PL/SQL solutions were written, a query was executed, the results were processed, and then the next query was executed. This sequence was repeated until all work was complete. It would have been nice to be able to do some of this work in parallel. While Oracle does have some options for doing work in parallel, such as Parallel Execution and DBMS_PARALLEL_EXECUTE, PL/SQL is for the most part a single threaded environment. I could probably have worked some magic to execute the queries and process the results in parallel, perhaps via scheduled jobs or Advanced Queues, but it would have been difficult using PL/SQL.

    However, this is an area where Node.js shines! Doing work in parallel is quite easy with the the async module. In the following solution, I use async’s parallel method to fire off three functions at the same time: the first builds the basic department object, the second builds the employees array, and the last builds the jobHistory array. The final function, which is fired when all the others have completed, puts the results of the first three functions together into a single object before returning the JSON.

Here’s the solution optimized for Node.js:

How fast did that solution finish the test? 7.8 seconds! Not too shabby! That’s faster than the PL/JSON solution but not quite as fast as APEX_JSON. But could we optimize even further?

Client Result Caching

Because I was running my tests on Oracle XE I wasn’t able to do a final optimization that would have been possible with the Enterprise Edition of the database: Client Result Caching. With Client Result Caching, Oracle can automatically maintain a cache of the data on the server where Node.js is running. This could have eliminated some round trips and data having to move across the network. I’ll revisit this feature in a future post were we can explore it in detail.


Relational to JSON in Oracle Database(

More and more often these days, front-end developers want their data in JSON format. And why not? JSON is a simple data-interchange format that’s lightweight and easy to use. Plus, many languages now provide a means of parsing and converting JSON data into native object types. However, not all data is best persisted in JSON format. For many applications, the relational model will be the best way to store data. But can’t we have the best of both worlds? Of course!

There’s no shortage of options when it comes generating JSON from relational data in Oracle Database. In this series I’ll introduce you to several options, from lower level PL/SQL based solutions to higher level solutions that do more than just generate and parse JSON. Here’s a list of the options I’ll be covering:

Which solution is right for you? That depends on your specific use case… Are you simply converting data and using it to populate another table? If so, one of the PL/SQL based options would probably be your best choice. Are you creating an API for others to consume? If so, you’ll probably want to utilize one of the higher level options and only use the PL/SQL options when needed. Are you trying to stream JSON via AJAX in APEX? Then the APEX_JSON package is probably the way to go.

Hopefully you see where I’m going with this. It’s not a one size fits all kind of thing. I’ll provide an overview of how these options can be used to accomplish a goal (defined next). Hopefully that will help you decide which option would work best for you for a given project.

The Goal

You’re probably familiar with, or at least aware of, the HR schema in Oracle Database, one of several sample schemas that’s included for learning purposes. Well we’re going to convert it to JSON! This is what the HR schema looks like:

HR Schema

You could convert the HR schema into JSON several different ways – it just depends on where you start. For example, if we started with employees table we could traverse down like this:

HR schema from emps down

On the other hand, we could start with the departments table and traverse down like this:

HR schema from depts down

That’s the path we’ll be traversing, from the departments table on down. Given a department id, our goal will be to create the JSON representation of that department. We’ll follow some basic rules to keep the JSON from growing unruly for this demo:

  • No more than 4 or 5 attributes per object – This one is pretty self explanatory. We don’t need to include every column from every table to get the point.
  • No more than 3 levels deep – Following one of the attribute chains in the image above, you’ll see “departments > locations > countries > regions”, which is 4 levels deep. Again, we don’t need to traverse down as far as we can to show how traversal works – three levels should do nicely.

In the end, we’ll be creating JSON that looks like this:

There are a couple things I want to note about this JSON. First of all, it includes all of the possible types of values you’ll find in JSON: objects, arrays, strings, numbers, booleans, and null. To get this output I had to do a little finagling. For null values I included the commission_pct column of the employees table, as not all employees get a commission. Additionally, I made it a rule that each solution should be written in such a way that if a department doesn’t have a manager or employees, then those properties should be displayed with a null value.

Boolean (true/false) values are not valid data types in Oracle’s SQL engine but I wanted to include them as they are common in JSON. Oracle developers typically use another datatype to represent Boolean values, e.g., 0/1, ‘T’/’F’, or ‘Y’/’N’. But I couldn’t find any of these types of flags in the HR schema so I decided to use another business rule instead: if an employee was hired before January 1st, 2005, then they should have an “isSenior” attribute set to true, otherwise it should be false.

I’ll do a separate blog post on dates in JSON, as they can be tricky. The reason is that dates are not valid datatypes in JSON. So as with Booleans in Oracle’s SQL engine, developers must make use of other datatypes (number and string) to represent dates. Issues arise around selecting a date format and handling timezone conversions. To keep things simple in this series, I’ll use a non-standard, string-based date format (DD-MON-YYYY) that is meant for clients to simply display as a string. Later, when I do the post on dates I’ll include a link here.

Okay, let’s generate some JSON! Here are the links from above for convenience:

LDOM live migration basics

One of the best features included in the Oracle VM server for SPARC 2.1 version was live migration. Live migration enables migrating active LDOMs from one physical system to other without any downtime. This may help you in doing any maintenance activity like patching or hardware changes on the physical server also helps in load balancing between 2 servers.

Live Migration phases

Phase 1 : Pre-checks
In this phase the source machine does pre-checks on the target system to ensure that the migration will succeed.
Phase 2 : Target LDOM creation
Here a LDOM is created on the target machine, which will be in bound state until the migration is complete.
Phase 3 : Run time state transfer
The run time state of the source LDOM is transferred to the target machine. Any changes to the source LDOM is also transferred to the target machine during the migration process by the LDOM manager. The information is retrieved from Hypervisor on source machine and transferred to the Hypervisor on the target machine.
Phase 4 : Source domain suspension
In this phase the source domain is suspended for a fraction of time and the remaining state information of the source LDOM is transferred to the target machine.
Phase 5 : Hand-off
In this last step a hand-off occurs between the ldom manager from source machine to ldom manager on the target machine. It occurs when the ldom is migrated and resumes execution on target machine and the source ldom is destroyed completely.

Hardware Requirements

1. CPU
a. same CPU migration
– Sufficient number of CPUs on target machine to accommodate migrating LDOM
– For system running Solris 10 , both target and source systems must have same processor type.
To check CPU type and frequency

# psrinfo -pv
The physical processor has 8 virtual processors (0-7)
  SPARC-T4 (chipid 0, clock 2548 MHz)

– Also the stick frequency of both source and target system CPU must match. Solaris 11 doesn’t have this condition.

# prtconf -pv | grep stick-frequency
    stick-frequency:  05f4bc08

b. Cross CPU migration
– UltraSPARC T2+ CPU or higher
– Both target and source systems should have Solaris 11 running.
– Set “cpu-arch” property on source machine to “generic”. generic cpu-arch uses common CPU hardware feaatures enabling LDOMs to perform a CPU independant migration. Default value is “native” which uses CPU specific hardware features, thus can be used if you don’t want to do a cross CPU migration. Generic attribute enables migration to an old or newer CPU type. To enable generic attribute

primary# ldm set-domain cpu-arch=generic ldom01

2. Memory
– Sufficient memory on target machine to accommodate migrating LDOM.
– Same number of identical sized memory blocks with same real addresses.

3. I/O related
– Domain with direct I/O access can not be migrated (I/O domains)
– Each virtual disk back end on target system must have same volume name but the path to actual back end device may be same.
for example when you add back end device to guest ldom in source machine as :

primary# ldm add-vdsdev /dev/dsk/c2t6d0s2 vol01@primary-vds0

Here vol01 should be there on target machine as vol01 only , but the disk slice c2t6d0s2 can have different name on the target machine , but referring the same device off course.

4. Virtual Services
– all 3 service vsw, vds and vcc (in case of console group only) must be present on target machine.
– arleast one free port in vcc

Software requirements

– Oracle VM version 2.1 or later for same CPU migration, version 2.2 or later for cross CPU migration

Migrating a domain (examples)

primary# ldm migrate ldom01 root@target-system
Target Password:
primary# ldm migrate -p password_file ldom01 root@target-system     (unattended migration)
primary# ldm migrate ldom01 root@target-system:ldom_other    (Migrating and renaming the ldom)

Monitoring migration status

You can check status of the migration from both source and target machines as below:

primary# ldm list ldom-source
ldom-source  suspended  -n---s         1     1G     0.0%  2h 7m
primary# ldm list ldom-target
ldom-target  bound      -----t  5000   1     1G

The sixth column in the FLAGS field indicates whether the system is the source or target of the migration

primary# ldm list -o status ldom-target 
    migration    34%         source-system

Canceling a Migration in progress

Either a KILL signal to ldm migration command or ldm cancel-operation can terminate a migration. Both commands must be executed from the source machine. Target machine can not control the migration process. As effect of cancel operation the ldom created on target machine is destroyed.

How to clone LDOMs using ZFS

The ZFS snapshot and cloning feature can be used to clones LDOMs. This comes very handy when you need to create multiple ldoms with some softwares already installed. The steps involved are :

1. Setup the primary domain
2. Create a guest LDOM (base LDOM)
3. Unconfigure, stop and unbind the base LDOM
4. Take zfs snapshot of base LDOM (called as golden image)
5. Clone the golden image to create new LDOMs

Setup the primary domain

Setup the primary domain with necessary resources and services and reboot the machine, for configuration changes to take effect.
Create default services

primary# ldm add-vcc port-range=5000-5100 primary-vcc0 primary
primary# ldm add-vds primary-vds0 primary
primary# ldm add-vsw net-dev=nxge0 primary-vsw0 primary
primary# ldm list-services primary 
    NAME             VOLUME         OPTIONS          DEVICE
    NAME             PORT-RANGE
    primary-vcc0     5000-5100
    NAME             MAC               NET-DEV   DEVICE     MODE
    primary-vsw0     02:04:4f:fb:9f:0d nxge0     switch@0   prog,promisc

Ensure ldmd deamon is online and set CPU, memory resources fro primary domain.

primary# svcs -a | grep ldmd
online 14:23:34 svc:/ldoms/ldmd:default
primary# ldm set-mau 1 primary
primary# ldm set-vcpu 8 primary
primary# ldm start-reconf primary          (delayed reconfiguration)
primary# ldm set-memory 4G primary
primary# ldm add-config new_config
primary# ldm list-config 
new_config [current]

Reboot primary domain for new configuration (new_config) to become active.

primary# shutdown -y -g0 -i6

Enable networking between primary and guest domains

primary# ifconfig nxge0 down unplumb
primary# ifconfig vsw0 plumb
primary# ifconfig vsw0 netmask + broadcast + up
primary# mv /etc/hostname.nxge0 /etc/hostname.vsw0

Enable virtual network terminal server daemon if not already enabled.

primary# svcadm enable vntsd
primary# svcs vntsd
    STATE          STIME    FMRI
    online         Oct_12   svc:/ldoms/vntsd:default

Setting up the base LDOM

Setup the base LDOM (base_ldom) with 8 VCPU, 2GB Memory, virtual network device vnet1 and zfs volume (base_ldom) as a virtual disk (vdisk1).

primary# ldm add-domain base_ldom
primary# ldm add-vcpu 8 base_ldom
primary# ldm add-memory 2G base_ldom
primary# ldm add-vnet vnet1 primary-vsw0 base_ldom
primary# zfs create -V 5gb ldompool/base_ldomvol
primary# ldm add-vdsdev /dev/zvol/dsk/ldompool/base_ldomvol vol01@primary-vds0
primary# ldm add-vdisk vdisk1 vol01@primary-vds0 base_ldom

Set the boot environment variables

primary# ldm set-var auto-boot?=true base_ldom
primary# ldm set-var boot-device=vdisk1 base_ldom

Install Solaris 10 on base ldom using solaris 10 iso image. We will add solaris 10 iso image as a virtual disk and then boot the base LDOM from this disk to install solaris 10.

primary# ldm add-vdsdev options=ro /data/sol_10.iso iso@primary-vds0
primary# ldm add-vdisk sol10_iso iso@primary-vds0 base_ldom

Bind and start the base LDOM. The solaris 10 iso should reflect in the devalias output at OK prompt as sol10_iso. Boot from this image to start the installation.

primary# ldm bind base_ldom
primary# ldm start base_ldom
LDom base_ldom started
ok> devalias
sol10_iso                /virtual-devices@100/channel-devices@200/disk@1
vdisk0                   /virtual-devices@100/channel-devices@200/disk@0
vnet1                    /virtual-devices@100/channel-devices@200/network@0
net                      /virtual-devices@100/channel-devices@200/network@0
disk                     /virtual-devices@100/channel-devices@200/disk@0
virtual-console          /virtual-devices/console@1
name                     aliases
ok> boot sol10_iso

Unconfigure, stop and unbind base LDOM

Unconfigure the base LDOM which automatically halts it. We would then stop the LDOM and unbind it so that we can take a snapshot of the base LDOM boot disk volume (base_ldomvol).

base_ldom# sys-unconfigure  (the ldom halts after this)
primary-domain# ldm stop base_ldom
primary-domain# ldm unbind base_ldom

Create the golden image

To create the golden image take a snapshot of the base_ldomvol from the base ldom.

primary-domain# zfs snapshot ldompool/bas_ldomvol@golden

Clone the golden image to create new LDOM

Clone the base_ldomvol snapshot (golden image) and use it to create the new LDOM, ldom01 with 4 VCPU, 4G, 1 MAU

primary-domain# zfs clone ldompool/bas_ldomvol@golden ldompool/ldom01_bootvol
primary-domain# ldm create ldom01
primary-domain# ldm set-mau 1 ldom01
primary-domain# ldm set-vcpu 4 ldom01
primary-domain# ldm set-mem 4G ldom01
primary-domain# ldm add-vnet vnet1 primary-vsw0 ldom01
primary-domain# ldm add-vdsdev ldompool/ldom01_bootvol vol01@primary-vds0
primary-domain# ldm add-vdisk vdisk1 vol01@primary-vds0
primary-domain# ldm set-variable auto-boot?=false ldom01
primary-domain# ldm bind ldom01
primary-domain# ldm start ldom01

When you boot the new LDOM, you will have to configure it with hostname, IP, timezone etc settings as it is an unconfigured LDOM.

How to install and configure LDOMs

Virtualization has been a need of time over several past years as we have machines now even with 16 cores and memory in TBs. A single machine is now capable of accommodating even more than 100 VMs at a time. Oracle VM for SPARC formerly known as LDOMs has played a key role in oracles virtualization strategies and is improving with every version. Before start configuring our first oracle VM for SPARC let us understand types of ldoms, ldom services and virtual devices.

Types of logical domains

Name Purpose
Guest No direct access to underlying hardware and does not provide virtual device or services to other ldoms. Uses virtual device.
I/O has direct access to underlying hardware in the server. It can be used in cases like oracle DB which wants direct/raw access to the storage devices.
Service provides virtualized devices and services to guest domains.
Control Service domain that also runs the ldoms manager software to control the configuration of hypervisor. This ldom manager is responsible for mapping between physical and virtual devices.

Virtual Services and Devices

Abbreviation Name Purpose
VLDC virtual logical domain channel communication channel between logical domain and hypervisor
VCC Virtual console concentrator Acts as a virtual console for each logical domain
VSW Virtual switch service provides network access for guest ldoms to the physical network ports
VDS virtual disk service provides virtual storage service for guest ldoms
VCPU virtual CPU Each thread of a T series CPU acts as a virtual CPU
MAU Mathematical arithmetic unit Each core of T series CPU will have a MAU for accelerated RAS/DSA encryption
Memory Physical memory is mapped into virtual memory and assigned to ldoms
VCONS Virtual console a port in guest ldom that connects to the VCC service in control domain
VNET Virtual network network port in guest ldom which is connected to the VSW service in the control domain
VSDEV Virtual disk service device physical storage device that is virtualized by VDS service in control domain
VDISK Virtual disk VDISK in guest domain is connected to the VDS service in control domain/service domain

Installing the OVM software

To install the LDOM software simply unzip the software zip and run the install-ldm script with -s option in case you don’t want to use the configuration assistant to configure the primary and guest ldoms.

primary # unzip
primary # ./install-ldm -s

Creating the default services

Create the essential services like vsw, vcc and vds required to serve the guest LDOMs.

primary# ldm add-vcc port-range=5000-5100 primary-vcc0 primary
primary# ldm add-vds primary-vds0 primary
primary# ldm add-vsw net-dev=nxge0 primary-vsw0 primary
primary# ldm list-services primary 
    NAME             VOLUME         OPTIONS          DEVICE
    NAME             PORT-RANGE
    primary-vcc0     5000-5100
    NAME             MAC               NET-DEV   DEVICE     MODE
    primary-vsw0     02:04:4f:fb:9f:0d nxge0     switch@0   prog,promisc

Initial configuration of the control domain

By default all the VCPUs, Memory and MAUs are assigned to the primary domain which is the default domain created after installing the OVM for SPARC software. Primary or control domain is used to configure all the guest ldoms and provide necessary virtual services to them like vcc, vsw and vds. Logical domain manager is responsible to create, delete, modify and control ldoms, thus make sure the ldmd service is running before configuring the primary and guest domains. Use delayed reconfiguration in order to configure the primary ldom without rebooting for previous changes to take effect.

primary# svcs -a | grep ldmd
online 14:23:34 svc:/ldoms/ldmd:default
primary# ldm set-mau 1 primary
primary# ldm set-vcpu 8 primary
primary# ldm start-reconf primary          (delayed reconfiguration)
primary# ldm set-memory 4G primary
primary# ldm add-config new_config
primary# ldm list-config 
new_config [current]

Reboot the primary domain for configuration settings to take effect

primary# shutdown -y -g0 -i6

Enable networking between primary and guest domains

By default communication between control domain and all the guest domains is disabled. To enable it, virtual switch has to be configured as the network device instead of nxge0.

primary# ifconfig nxge0 down unplumb
primary# ifconfig vsw0 plumb
primary# ifconfig vsw0 netmask + broadcast + up
primary# mv /etc/hostname.nxge0 /etc/hostname.vsw0

Enable virtual network terminal server daemon

The vntsd daemon is responsible to provide the virtual network terminal services to the guest ldoms. If this service is not running enable it with svcadm command.

primary# svcadm enable vntsd
primary# svcs vntsd
    STATE          STIME    FMRI
    online         Oct_12   svc:/ldoms/vntsd:default

Setting up the Guest Domain

We would assign 8 VCPUs, 2 GB of memory and 1 MAU to our first guest ldom. Also a virtual network vnet1 will be created and associated with the virtual switch vsw0.

primary# ldm add-domain ldom01
primary# ldm add-vcpu 8 ldom01
primary# ldm add-memory 2G ldom01
primary-domain# ldm set-mau 1 ldom01
primary# ldm add-vnet vnet1 primary-vsw0 ldom01

Adding storage to the guest domain

Here we first need to specify the physical device that needs to be exported by vdsdev to the guest domain and then we actually add the virtual disk thus created to the guest domain. Now use any one of the 3 methods mentioned below.
1. Adding physical disks

primary# ldm add-vdsdev /dev/dsk/c2t1d0s2 vol1@primary-vds0
primary# ldm add-vdisk vdisk1 vol1@primary-vds0 ldom01

2. Adding file

primary# mkfile 10g /ldoms/ldom01_boot
primary# ldm add-vdsdev /ldoms/ldom01_boot vol1@primary-vds0
primary# ldm add-vdisk vdisk1 vol1@primary-vds0 ldom01

3. Adding a volume

primary# zfs create -V 5gb pool/vol01
primary# ldm add-vdsdev /dev/zvol/dsk/pool/vol01 vol1@primary-vds0
primary# ldm add-vdisk vdisk1 vol1@primary-vds0 ldom01

Setting variables

Setup the boot environment variable for the guest ldom.

primary# ldm set-var auto-boot?=true ldom01
primary# ldm set-var boot-device=vdisk1 ldom01

Setting up the solaris ISO image for installing guest ldom

Now we can also do a jumpstart installation of the guest domain. But one of the easiest and most widely used method is add iso image as virtual disk to the guest ldom and install it from it. Here you can access the vdisk sol10_iso in the ok prompt and boot from it.

primary# ldm add-vdsdev options=ro /data/sol_10.iso iso@primary-vds0
primary# ldm add-vdisk sol10_iso iso@primary-vds0 ldom01

Bind and start installing the ldom

primary# ldm bind ldom01
primary# ldm start ldom01
LDom ldom01 started
ok> devalias
ok> boot sol10_iso

Connect the guest domain

Now check the port which is bound with the guest domain and connect the virtual console of the guest domain.

primary:~ # ldm list
primary active -n-cv  SP   8    4G    0.3% 8h 46m 
ldom01  active -n--- 5000  8    2G     48% 1h 52m
primary# telnet localhost 5000 Trying
    Connected to localhost.
    Escape character is ’^]’.
Connecting to console "ldom01" in group "ldom01" .... Press ~? for control options ..

Flag definitions

Now you can see various flags in the “ldm list” command output. The falgs represent the current state of the ldom.

column 1 column 2 column 3
s starting or stopping
– placeholder
n normal
t transition
d delayed reconfiguration
– placeholder
column 4 column 5 column 6
c control domain
– placeholder
v virtual I/O service domain
– placeholder
s source domain in migration
t target domain in migration
e error occurred in migration
– placeholder

Other useful Commands

View current version of Oracle VM server for SPARC software

primary# ldm -V

Long listing of domains

primary# ldm list -l

List the resource for all LDOMs and per LDOM

# ldm list -o cpu primary
# ldm list -o network,memory ldom01

List the boot variables

# ldm list-variable boot-device ldg1 

List the bindings of all the LDOMs

# ldm list-bindings ldom

List all server resources, bound and unbound.

# ldm list-devices -a
# ldm list-devices mem