One of the most time consuming and tedious development tasks is data migration or ETL (Extract Transfer Load). In many cases, the task of moving a client’s existing data into the EVA data structure for Magento can take longer than any functional requirement. Since every client and application contains its own data format, its difficult to find a Magento module that will satisfy your needs. Sure, Magento does offer some data loading capabilities, but in many cases, a client’s data relationships are far too complex and specific to their domain to get the import/export offered through Magento to load the data correctly.

It’s for the reasons listed above that Brim is now offering custom Magento ETL services. While working on various Magento projects in the past, Brim has developed a core set of migration modules that are easily customized to support most data formats. A great example would be the work we delivered to MotionSavers. For this project, our modules were capable of migrating Sql Server data from TheOnlineCatalog, Excel data from Strong-Hold, CSV data from Rousseau Metal, TSV data from PFI Industrial Equipment, and many others.

We have streamlined the process of data migration, and we’re confident that we can provide the same services for other clients in a very time efficient manner and at a very low service cost. If you’re ready to let Brim help you out on your project, drop us a line at our contact page.

Per the Hibernate documentation, Hibernate can perform dirty checks only when the objects are loaded and changed in the scope of a single Session. This means we cannot use detached objects, but we must keep our session open for one conversation (multiple http requests) using Managed Sessions, just to get hibernate’s dirty check. This introduces new challenges of storing the Hibernate Session object in HttpSession, which fails if we deploy in a clustered environment, since most of the fields in Hibernate org.hibernate.impl.SessionImpl class are transient. So, how do we get the benefit of using Detached Objects and also get Hibernate’s automatic dirty checking?

The following is a brief background of how Hibernate does its dirty checking:

Hibernate Session contains a PersistenceContext object, which maintains a cache of all the objects read from the database as a Map. In the same Session, when I read an object from the db and make changes to it, Hibernate compares the objects and triggers the updates when the session is flushed. Every object in the PersistenceContext is called Persistent object. Once the session closes, the PersistenceContext is lost and so is the cached copy. A detached object, when saved, opens a new session containing an empty PersistenceContext, so there is nothing to compare against for the dirty check.

If we have an original cached copy of the Detached Object (a clone saved in HttpSession) and we are able to place that copy in the PersistenceContext somehow, we can get the dirty check to work. Here is how to do it.

Couple of points here, before we start reading the code -

1)  All your domain models have a generic way to expose their primary key. Here I have defined PrimaryKeyAwareDomainModelObject interface, which exposes a method to access the primary key property.

public interface PrimaryKeyAwareDomainModelObject {
public Integer getPrimaryKey();
}

2) You already have the original cached copy of the Detached Object, and have made changes to service layers to pass it to the DAO layer.

3) Also, your Detached Objects are POJO’s.

4) You are using Spring HibnerateDaoSupport, else you can get the SessionFactory to access the Hibernate Session.

/*** This class handles adding the cached Detached Object to PersistenceContext of a new Session.

** 1) Get a new session, or the session associated with this transaction from the
* sessionFactory.

*2) Inside the PersistentContext of the session, add the old
* Model object, so hibernate thinks that it was loaded as part of read
* operation in the same session

*3) Copy the changed values from the new Model
* object to the old Model object, since hibernate does identity check on the
* object in the persistenceContext, passing a new Model directly to hibernate
* will not trigger dirty checking, but create issues.

*4) Call usual saveOrUpdate, which will check the object if its dirty before update.
***/

import org.hibernate.EntityMode;
import org.hibernate.LockMode;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.engine.EntityKey;
import org.hibernate.engine.PersistenceContext;
import org.hibernate.engine.SessionFactoryImplementor;
import org.hibernate.engine.Status;
import org.hibernate.impl.SessionImpl;
import org.springframework.orm.hibernate3.support.HibernateDaoSupport;

public class HibBeforeUpdateListener<T extends PrimaryKeyAwareDomainModelObject, Q extends HibernateDaoSupport> {

/***
* Before doing a call on saveOrUpdate on hibernate session - fetch the
* session - get the PersistenceContext, and add the oldModelObj - copy the
* modified fields from new modelObj to oldModelObj.
* You can pass SessionFactory directly instead of HibernateDaoSupport
*/
@SuppressWarnings("unchecked")

public T beforeUpdate(T modelObj, T oldModelObj, Q dataAccessor) {

SessionFactory factory = dataAccessor.getSessionFactory();
Session session = factory.getCurrentSession();
// get the PersistenceContext of this session
PersistenceContext persistenceContext = session instanceof SessionImpl ? ((SessionImpl) session).getPersistenceContext(): null;
if (null != persistenceContext) {
addEntityToPersistenceContext(persistenceContext, oldModelObj,(SessionFactoryImplementor) factory);
}
// copy modified values from newModel to oldModel

// I am using Dozer

return (T) DozerUtility.copy(modelObj, oldModelObj);
}

private EntityKey generateKey(T modelObj, SessionFactoryImplementor sessionFactory) {
// EntityKey is the key used in the map of objects stored in
// PersistenceContext  it requires the primary key, defined by the @Id annotation in the model class
// , persister object which is SingleTableEntityPersister class in my case, and our models type, which is POJO
return new EntityKey(modelObj.getPrimaryKey(), sessionFactory.getEntityPersister(modelObj.getClass().getCanonicalName()),EntityMode.POJO);
}

/***
 * Hibernate stores the entities as an EntityEntry inside a IdentityMap, ie.
 * 2 objects are equal, if they reference the same object, rather than
 * Object.equals and hashcode is equal. EntityEntry contains the original
 * object as loaded from the db, plus also its original state as an Object
 * array, called loadedState. This method mimics the same behavior of
 * hibernate internals, and stores the object and its loadedState.
 *
 * @param context
 * @param oldModelObj
 * @param sessionFactory
 */
 private void addEntityToPersistenceContext(PersistenceContext context, T oldModelObj, SessionFactoryImplementor sessionFactory) {
       Map<String, T> allEntitiesMap = new HashMap<String, T>();
       for(T modelObj : getAllEntityObjects(oldModelObj, allEntitiesMap).values()){
                 context.addEntity(modelObj, Status.MANAGED, sessionFactory.getEntityPersister(modelObj.getClass().getCanonicalName())
                                 .getPropertyValues(modelObj, EntityMode.POJO), generateKey(
                                  modelObj, sessionFactory), null, LockMode.READ, true,
                                  sessionFactory.getEntityPersister(modelObj.getClass().getCanonicalName()), false, true);
     }
 }

 /***
 * Make a list of all sub Entity objects instances inside the object,
 * and add them separately
 * @return
 */
 private Map<String, T> getAllEntityObjects(T object, Map<String, T> allEntityObjects){
   // definitely this object will be added to persistence context
   if(!allEntityObjects.containsKey(object.getClass().getCanonicalName())){
            allEntityObjects.put(object.getClass().getCanonicalName(), (T)object);
    }
   // get all the instance variables and check their annotations
   Field[] fields = object.getClass().getDeclaredFields();
   // if no fields (impossible scenario), then just add the object and send it back
    if(fields.length == 0){
     return allEntityObjects;
   }
   for(Field f : fields){
    Class fieldClass = f.getType();
    if(fieldClass.getAnnotation(javax.persistence.Entity.class) != null){
     try{
         f.setAccessible(true);
         Object fieldValue = f.get(object);
        // only add the domain model objects
         if(fieldValue instanceof PrimaryKeyAwareDomainModelObject && !allEntityObjects.containsKey(fieldClass.getCanonicalName())){
            allEntityObjects.put(fieldClass.getCanonicalName(), (T)fieldValue);
            getAllEntityObjects((T)fieldValue, allEntityObjects);
         }
      }catch(IllegalAccessException ex){
        // do nothing and move to next field
      }
    }
  }
    return allEntityObjects;
 }
}

And here is how to use it in a DAO -

public class MyDataAccessorImpl extends
HibernateDaoSupport{

public void update(T modelObj, T oldModelObj) {
PrimaryKeyAwareDomainModelObject modelObjToUpdate = new HibBeforeUpdateListener().beforeUpdate(modelObj, oldModelObj, this);
getHibernateTemplate().saveOrUpdate(modelObjToUpdate);
}

}

This approach was the most efficient that used Hibernate’s own public APIs. If anyone else has a similar approach to the same problem, I would love to hear…

Even though our latest iPhone app, Scrambled Apps!, wasn’t approved for the app store, it still might be beneficial for me to document some of the work that made it possible. If you’re not familiar with the app, visit http://www.brimllc.com/mobile-applications/scrambledapps/.

The app used Chipmunk physics engine with the cocos2d graphics library to mimic a user’s home screen, which would “fall” apart when clicked. The “falling” was made possible using Chipmunk; however, I had to make some modifications to a 3rd party utility class to enable the dragging and dropping.

On the Chipmunk forums, I found a class called cpMouse. This class managed constraints between objects to allow dragging. The only issue I found was that the version of cpMouse that was available didn’t work with the latest version of Chipmunk, what at the time of this writing is v5.0.0 I believe.

The updated cpMouse files are located here: http://www.brimllc.com/wp-content/uploads/2010/08/cpMouse.h and here: http://www.brimllc.com/wp-content/uploads/2010/08/cpMouse.c.

To use this class in an iPhone game, you simply reference this class in your ccTouchesBegan, ccTouchesMoved, and ccTouchesEnded method handlers. Here are examples of each:

ccTouchesBegan

- (void) ccTouchesBegan:(NSSet *)touches withEvent:(UIEvent *)event{
UITouch *myTouch = [touches anyObject];
CGPoint location = [myTouch locationInView: [myTouch view]];
location = [[CCDirector sharedDirector] convertToGL: location];
mouse = cpMouseNew(space);
cpMouseMove(mouse, cpv(location.x, location.y));
cpMouseGrab(mouse, cpv(location.x, location.y));
}


ccTouchesMoved

- (void) ccTouchesMoved:(NSSet *)touches withEvent:(UIEvent *)event{
UITouch *myTouch = [touches anyObject];
CGPoint location = [myTouch locationInView: [myTouch view]];
location = [[CCDirector sharedDirector] convertToGL: location];
//move the nouse to the click
cpMouseMove(mouse, cpv(location.x, location.y));
if(mouse->grabbedBody == nil){
//if there's no associated grabbed object
// try get one
cpMouseGrab(mouse, cpv(location.x, location.y));
}
}


ccTouchesEnded

- (void)ccTouchesEnded:(NSSet *)touches withEvent:(UIEvent *)event
{
[self ccTouchesCancelled:touches withEvent:event];
}

- (void)ccTouchesCancelled:(NSSet *)touches withEvent:(UIEvent *)event {
cpMouseDestroy(mouse);
}


I would think that the above methods are fairly self explanatory, but I can answer questions if anyone needs more clarification.

So, after a few weeks of waiting for the latest app to be reviewed, it has been rejected. Scrambled Apps! is Brim’s latest iPhone app, that was slated to be one of its kind. Allowing a user to load an image of their own iPhone home screen (or any image for that matter), the app contains image recognition and a physics engine, which would cause the screen to “fall” once a user clicked on the screen. This would give the illusion that the app icons on the screen had “broken” apart, thus giving the user instant gratification when he’s able to prank his friends/family.

The iPhone app review team quoted the terms of service by saying “Simulating failures of graphics, actions, or images is a violation of the iPhone Developer Program License Agreement.”. Is this really what Apple has become? Not even a silly practical joke app can pass its review process?

Here’s the email in all of its glory:

The idea for the app came to my last year when I saw the Wario marketing site that caused YouTube to fall apart using an intuitive Flash movie. Its documented here: http://www.crunchgear.com/2008/09/24/wario-land-shake-it-advertisement-breaks-youtube/

I’ll load some screens of Scrambled Apps! in a few here: http://www.brimllc.com/mobile-applications/scrambledapps/

Hopefully Apple will re-review the application and later review it, else I can use ad-hoc distribution and allow people to have the app – first come, first served.

So I just implemented order printing functionality, so I did what any other developer would do, and I searched the Magento modules (actually my client found a good module for the job). We found: http://www.magentocommerce.com/magento-connect/piotrn/extension/1054/admin-order-printing-extension.

The problem we found was that this module is for Magento version 1.2, but my client is running on 1.3. After installing the module, there were a few errors, which I assumed would happened, but it turns out there is only one file that needs updating. This is the Grouped.php that is included in the module. Below is the code that you’ll want to change to in order to get the module working correctly:

class Nastnet_OrderPrint_Model_Order_Pdf_Items_Order_Grouped extends Nastnet_OrderPrint_Model_Order_Pdf_Items_Order_Default
{
    public function draw()
    {
        $type = $this->getItem()->getOrder()->getRealProductType();
        $renderer = $this->getRenderedModel()->getRenderer($type);
        $renderer->setOrder($this->getOrder());
        $renderer->setItem($this->getItem());
        $renderer->setPdf($this->getPdf());
        $renderer->setPage($this->getPage());

        $renderer->draw();
    }
}

The changes required were modifying the name of the class along with a couple API changes (getOrder rather than getOrderItem).

I hope the creator of the module updates his code to be compatible with 1.3 soon.

NOTE: The version of this module that we used was 0.1.3 with API key magento-community/Nastnet_OrderPrint-0.1.3.

A client of mine has a lot of simple products with required options that define the minimum price of a product. Instead of the pricing showing up as $0.00 before a user selects a value for a required option, I wanted to default the input to the minimum value. This doesn’t appear as a possibility through the core Magento code; however, with a core override and a small piece of Javascript, it can be quite simple.

First, extend Mage_Catalog_Block_Product_View_Options_Type_Select, which is the select input, and override the getValuesHtml method. The code will look as such:

public function getValuesHtml()
	{
		$_option = $this->getOption();
		$_minimumPriceValue = $this->getProduct()->getMinimumPriceOptionValue();

		if ($_option->getType() == Mage_Catalog_Model_Product_Option::OPTION_TYPE_DROP_DOWN
		|| $_option->getType() == Mage_Catalog_Model_Product_Option::OPTION_TYPE_MULTIPLE) {
			$require = ($_option->getIsRequire()) ? ' required-entry' : '';
			$extraParams = '';
			$select = $this->getLayout()->createBlock('core/html_select')
			->setData(array(
                    'id' => 'select_'.$_option->getId(),
                    'class' => $require.' product-custom-option'
                    ));
                    if ($_option->getType() == Mage_Catalog_Model_Product_Option::OPTION_TYPE_DROP_DOWN) {
                    	$select->setName('options['.$_option->getid().']')
                    	->addOption('', $this->__('-- Please Select --'));
                    } else {
                    	$select->setName('options['.$_option->getid().'][]');
                    	$select->setClass('multiselect'.$require.' product-custom-option');
                    }
                    foreach ($_option->getValues() as $_value) {
                    	$priceStr = $this->_formatPrice(array(
                    'is_percent' => ($_value->getPriceType() == 'percent') ? true : false,
                    'pricing_value' => $_value->getPrice(true)
                    	), false);
                    	$select->addOption(
                    	$_value->getOptionTypeId(),
                    	$_value->getTitle() . ' ' . $priceStr . ''
                    	);

                    	// if this value determines the product's minimal price, set it as selected
                    	if($_minimumPriceValue && $_value->getOptionTypeId() == $_minimumPriceValue->getOptionTypeId()){
                    		$select->setValue(array($_minimumPriceValue->getOptionTypeId()));
                    	}
                    }
                    if ($_option->getType() == Mage_Catalog_Model_Product_Option::OPTION_TYPE_MULTIPLE) {
                    	$extraParams = ' multiple="multiple"';
                    }
                    $select->setExtraParams('onchange="opConfig.reloadPrice()"'.$extraParams);

                    return $select->getHtml();
		}

		if ($_option->getType() == Mage_Catalog_Model_Product_Option::OPTION_TYPE_RADIO
		|| $_option->getType() == Mage_Catalog_Model_Product_Option::OPTION_TYPE_CHECKBOX
		) {
			$selectHtml = '
';
			$require = ($_option->getIsRequire()) ? ' validate-one-required-by-name' : '';
			$arraySign = '';
			switch ($_option->getType()) {
				case Mage_Catalog_Model_Product_Option::OPTION_TYPE_RADIO:
					$type = 'radio';
					$class = 'radio';
					if (!$_option->getIsRequire()) {
						$selectHtml .= '

' . $this->__('None') . '

';
					}
					break;
				case Mage_Catalog_Model_Product_Option::OPTION_TYPE_CHECKBOX:
					$type = 'checkbox';
					$class = 'checkbox';
					$arraySign = '[]';
					break;
			}
			$count = 1;
			foreach ($_option->getValues() as $_value) {
				$count++;
				$priceStr = $this->_formatPrice(array(
                    'is_percent' => ($_value->getPriceType() == 'percent') ? true : false,
                    'pricing_value' => $_value->getPrice(true)
				));
				$selectHtml .= '
' .
                               '
' .
                               ''.$_value->getTitle().' '.$priceStr.'';
				if ($_option->getIsRequire()) {
					$selectHtml .= '';
				}
				$selectHtml .= '

';
			}
			$selectHtml .= '

';
			return $selectHtml;
		}
	}

This code will pull the required option from the product and set the input to the minimum allowed price. Now, in the template file catalog/product/view/options/wrapper.phtml, modify the Javascript on the page to look as such:

decorateGeneric($$('#product-options-wrapper dl'), ['last']);
	document.observe("dom:loaded", function(){
		// reload the price in case the option is pre-set
		opConfig.reloadPrice();
	});

This will cause the pricing to be reloaded when the page is displayed so that the price block will display appropriately.

It would make sense that this functionality would be provided by core Magento in the future; however, until then, I would be interested if anyone else has solved this problem in a more elagent way.

Today I was tasked with converting some queries to HQL; however, in one instance, a mapping was not defined. My entities looked something like:

AuditObject
– id
– value
– modifiedBy
– updatedBy

UserObject
– id
– name

In my scenario, the AuditObject’s modifiedBy value should be the name of the user who’s ID is equal to the updatedBy field in the Audit table (I know this is a very strange situation, but I’ve inherited a legacy system).

The approach I used to solve this was to use a Cartesion/Cross join in HQL. So my HQL looked something similar to:

getHibernateTemplate.find("select new AuditObject({...}, user.name) from AuditObject as audit, UserObject as user where audit.id = ? and audit.updatedBy = user.id")

This returns me a list of AuditObjects with the modifiedBy property set as the user’s name. Also notice that I’m using the data-transfer approach by defining a constructor that takes in all the properties that I want set. This is very convenient, and hibernate creates the correct join syntax for me.

I’d be interested if any others out there have found other solutions to this problem.

So I’ve had this problem off and on for the past couple of days while working on my latest iPhone application, but it seemed to come and go, so I didn’t pay much attention to it until today.

The problem got so bad, that I couldn’t do anything within xcode without it freezing up and displaying “unrecognized selector sent to instance” in an error popup dialog. I searched Google, and it seems that many people have experienced the same issue, but not to the extend that I was having. I finally stumbled upon a link that seemed to solve my issue: http://djuncas.blogspot.com/2009/03/unrecognized-selector-sent-to-instance.html.

It turns out that I had modified the method signatures of some files that were referenced from IB (Interface Builder); however, I didn’t upload those references, and so xcode didn’t know how to link properly. Although this was a mistake on my part, I feel that xcode should at least handle this exception more elegantly than freezing up. Props go to Mads Godvin Jensen for documenting the problem in his blog, and I’m echoing the solution as a way to remember this myself.

I ran into this problem today while working at a client, and I thought I could help others out with a bit more information than I was able to find online.

The problem is that my client has a standard to have “WITH UR” appended to all SELECT queries. This wasn’t a problem with Spring JDBC, but now with the move to Hibernate, I had to find a way to provide this functionality. Keep in mind that I don’t have a choice in this standard, so I’m not arguing if appending “WITH UR” is a good approach or not.

Anyway, I came across this link,  http://opensource.atlassian.com/projects/hibernate/browse/HHH-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=32665#action_32665, which provides a workaround for this very problem. It involves creating a Hibernate interceptor that checks the query before executing, and appending “WITH UR” as appropriate. For my case, since I can assume that every SELECT query requires this suffix, the interceptor code looks as follows:

public class DB2Interceptor extends EmptyInterceptor {

	private static final long serialVersionUID = 1L;

	private static final Logger logger = Logger.getLogger(DB2Interceptor.class
			.getName());

	@Override
	public String onPrepareStatement(String str) {

		// Transform the original query into lower case
		String compstr = str.toLowerCase();

		// Check if we're dealing with a simple select statement
		if (compstr.matches("^select.*") && !compstr.matches(".*for update.*")) {
			if (!compstr.matches(".*with ur.*")) {
				str += " with ur ";

				if (logger.isDebugEnabled()) {
					logger.debug("Appending \"WITH UR\" to query.");
				}
			}
		}
		return str;
	}
}

Now you can wire this up in your Spring application context as such:

<bean id="sessionFactory"
		class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
		<property name="dataSource" ref="dataSource" />
		<property name="entityInterceptor">
    		<bean class="com.company.dao.hibernate.DB2Interceptor"/>
  		</property>
...

Since this satisfies my requirement, I’m leaving the code as-is for my client; however, if you have a need of setting a specific transaction isolation value (as in the case of the link I mentioned previously), I have a better solution than his proposed Hashtable/thread approach.

Instead, create the ThreadUtils class as he has done, but make the TransactionManager class extend ThreadLocal to ensure this is specific to the calling thread. Be sure to remove the thread local after use, and you can even generate the getter/setter code in an aspect.

Since I wasn’t required to code the ThreadLocal side of things, I left that out; however, if anyway would benefit from seeing some extra code, let me know.

Sometimes it is nice to be able to allow customers to specify custom shipping comments during the checkout process; however, since the order isn’t actually created until after this process, you have to hook into various events to get this data to the final order entry. This process can be done in the following steps:

1. Add an input on the checkout process to capture the shipping comments. For my case, I added the following input to available.phtml in checkout/onepage/shipping_method directory:

<textarea id=”shippingComment” name=”shippingComment” rows=”3″ cols=”50″></textarea>

2. Create a couple listener methods that can handle two events: checkout_controller_onepage_save_shipping_method and checkout_type_onepage_save_order.

3. Implement these methods as such:

public function setShippingCommentToQuote($observer)
	{
		$event = $observer->getEvent();
		$quote = $event->getQuote();
		$request = $event->getRequest();

		$orderComment = $request->getPost('shippingComment', false);
		$session = Mage::getSingleton('checkout/session');
		$session->setData('shipping_comment', $orderComment);
	}

	public function setShippingCommentToOrder($observer)
	{
		$session = Mage::getSingleton('checkout/session');
		$shippingComment = $session->getData('shipping_comment');

		if($shippingComment){
			$observer->getEvent()->getOrder()->setShippingComment($shippingComment);
		}
	}

The first method handles the checkout_controller_onepage_save_shipping_method event, and basically just takes the input from the request and stores it in the session. The setShippingCommentToOrder() method handles the checkout_type_onepage_save_order event and sets the comment from the session to the order.

Now, you can view these comments from the order information page in the admin section.