Co-written with  fellow Future Shaper, Sabine Simon.

Curt Richardson’s perspective that “Failure is a part of innovation – perhaps the most important part “provides an important finding. Without failure, there will be no innovation as both are closely linked.

For organisations, this insight is fundamental in the dynamic and complex environment of today’s markets. Fundamental, because to make innovation work, there needs to be a learning process from failing to avoid making the same mistakes twice. As more innovation projects fail than succeed, there is plenty of information on the organisational culprits that jeopardise them.

In our first article, “Killers of Innovation: The Failure,“ we focused on the reasons, difficulties, and consequences to admitting mistakes and failures. Regarding the ratio between the number of successful innovations and failures, while seeing the rising number of innovations in total, it is a given that we must learn to improve. This only works by admitting failure, relentlessly assessing the causes, and sharing them within the corporation or even better with the larger community. It is not enough to settle with a proximate cause. It is crucial to uncover the root causes, requiring more energy and resilience by all involved in the process.

As mentioned in the first article, there is a twofold benefit to this. We gain insights, fundamentally understand and learn for the next time around. And others can benefit from these insights as well and avoid pitfalls on their respective innovative journeys.

The specific requirements of tomorrow’s organisations

 “The organisation of tomorrow will demand mistakes and failures. It is only by trying lots of initiatives that we can improve our chances that one of them will be a star, “Deepak Seethi (CEO of AT&T) said. The successful corporate of tomorrow will need to adapt to dynamically and continuously changing environments successfully. The current crisis will certainly increase the speed of change. The structures, hierarchies, and procedures to control that made enterprises successful over the last centuries will not suffice in the knowledge economy’s upcoming era. Too many factors, like silos, strict rules, or the pressure to achieve quarter results or other KPIs inhibit innovation – root causes for rigid structures will be examined more closely in factor 3 below. In the knowledge economy, enterprises will benefit from the creativity and knowledge of their employees even more. This requires forms of organisations, leadership styles, and empowering environments that, as Deepak Seethi put it, have both innovation and learning culture at its heart. Without it, it will be increasingly difficult in growing dynamic markets to stay competitive.

In his very insightful and lauded book Loonshots, Safi Bahcall provides a valuable lesson about the importance of the relation between two groups in the organisations. He distinguishes the artists (innovative-driven and creative minds) and the soldiers (task- and rule-oriented employees) following Steve Jobs. He sets up principles that can lead to more sustainable innovation processes – or, if not followed, can doom innovative projects to failure. 

From our point of view, three aspects are fundamental for tomorrow’s organisations: 

  • Ideas and innovation projects in organisations require shelter in their early stages as they are still very fragile. It is easy for established team members to dismiss them or shoot them down.
  • Rigid task allocation does not work well with ideation as it inhibits creativity. Bahcall cites Art Fry, the inventor of the Post It notes in this case.
  • Achieve a dynamic equilibrium, where the exchange of progressive ideas between the artists and the soldiers is possible. To reach this balance requires a lot of attention from leaders, guidance, and continuous monitoring to maintain it. Bahcall regards this interface as the weakest link, considering the different mindsets of the two groups. A successful exchange between them is crucial in two ways: Soldiers require innovation and product enhancements to grow or maintain their business to be future safe. The artists need the feedback to improve their solutions and discover what does not work.

Organisational factors that drive failure of innovation

Various organisational factors can spell doom for innovation in enterprises. An entire library exists about corporate innovation deficits, and it provides multiple insights on the potential killers. If an organisation does not provide the proper foundation, successful innovation is like trying to grow a flower in a barren land of concrete. It does not work! But let’s start with the soil.

Image by Anest from Getty Images, licensed from Canva Pro, modified by Sabine Simon

Factor 1: Culture

Organisational culture is cited as the culprit for failed innovation efforts very often. However, this assessment is too general to provide insight. A closer look at specific details concerning innovation is necessary.

Organisational culture specifies the character of an organisation, defined by its shared values and beliefs. The latter are collective beliefs that guide the actions of employees. They do not have to be openly shared, nor do they have to be in the organisation’s members’ active consciousness. They do, however, strongly affect both the existence and success of the organisation. They form the core of the corporate culture and serve to sustain the system. To get a better picture of what organisational beliefs can be, here are a few examples:

  • “Only those who work hard work well.”
  • “Employees abuse freedom when given it.” 
  • “The right plan metrics lead to success.“

These or similar sentences may sound familiar. They are unconscious and implicit behavioural guidelines that ensure that an existing system works the way it does. They act like a kind of gravitational source that organises its individual parts. The “system” provides psychological security and structural stability2. Both are essential. Structural stability conveys a purpose and, to some extent, a treasured predictability, thereby giving employees a sense of security. This, however, regarding innovation is a two-edged sword. The gravitational balance is challenged by any kind of change, as it causes uncertainty. Innovation is change, unpredictable, and often involves failure – states we like to avoid. It can trigger an immune reaction of the “system, “especially if it is adverse to innovation.

After this initial analysis, we now want to highlight some characteristics of organisational cultures that make the conditions for successful innovation particularly difficult: 

  • Culture, dominated by Theory X. The concept of Theory X and Theory Y, focusing on human work motivation, was developed by Douglas McGregor in the 1950s and 60s. Innovation is all but impossible in a culture shaped by the internal beliefs that only those who work a lot work well, that employees will only exploit free spaces, and worry about the personnel’s efficiency. Theory X causes a dysfunctional imbalance and makes it almost impossible to achieve dynamic equilibrium essential for innovation that Safi Bahcall calls for in his book mentioned above. Theory X is still prevalent in a lot of organisational cultures today. It should not go unmentioned, though, that a dominance of Theory Y in the culture can also lead to innovations’ failure. In this case, many promising new ideas may emerge, but the innovation process often comes to a halt at the implementation level.
  • A lack of Psychological Safety. Coined by Professor Amy Edmundson from Harvard Business School, this term describes a desired mindset where employees are willing to bring their ideas forward due to an open and trustful environment. Companies that establish a „no-failure-allowed culture“ by punishing failure, mistakes, or errors force employees to hold their suggestions back, thereby strangling innovation. 
  • Insufficient or inadequate feedback. Feedback is at the heart of the innovation process and an essential foundation for creating psychological safety. It helps to reflect on ideas or to identify blind spots and pitfalls. A lack of personal or inadequate or not well-intended feedback (only top-down, too seldom due to lack of time, strong hierarchies that inhibit exchange, lack of feedback skills, without sufficient respect) can drive up the failure rate of innovation projects. Feedback is vital for another reason, too. As mentioned above, innovation can result in altering the portfolio or establishing new processes. Avoiding feedback from the employees about their specific concerns and fears will cause resistance towards innovation. Also, regard the domino effect! Without regular feedback, there will be no psychological safety, and without psychological safety, an essential foundation of trust cannot develop. A lack of mutual trust – or if trust is reduced to messages on poster boards in entrance halls but not a part of the daily business – massively restricts creativity conditions.  
  • Insufficient curiosity and lack of risk-taking-mentality. To some extent, these two factors go hand in hand. Learning cultures, born out of curiosity and commitment to improve, involve experimentation, which also means failure. If the systemic believers prefer to play the safe game, or are reluctant to take risks, they will suffocate both curiosity and innovation. 

Factor 2: Leadership responsibility

We continue from culture to leadership. Establishing an innovative culture and shaping the structure is a leadership responsibility. It is essential to add that culture is the sum of all the behaviours and the habits of employees and their leaders. The unconditional support of the top-level management is the central factor for innovation to succeed. As so-called system parents, managers exercise a role model function for their employees. Employees smell a lack of management buy-in for innovation ten miles into the wind. If management praises innovation but acts differently, for example, by expecting innovative results but not making the required time available for it or insufficiently appreciating additional efforts, employees always get the message. As a result, their activities are affected. Their motivation and sense of urgency will drop, and they will automatically reduce their commitment to innovation projects and bring them to a halt.

Leading employees is challenging. Leading innovation teams is even more so. Innovation on prescription is doomed to fail. Quite a few organisations and their managers start from and use Theory X. Theory X promotes the use of strict control and supervision. Transactional leadership inhibits innovation as it does neither allow free space nor growth of employees beyond the imagination of the leader. This alone obstructs the urgency of something new because only ideas that find a place in the manager’s imagination will survive. On the other hand, there is a need for appreciation and (transactional) reward for progress and milestones3. The literature speaks of ambidextrous leadership – transactional and transformational4. The smallest deviations in the leader’s role model function and other individual factors will increase the probability of failure. We will discuss this in more detail in Article 3. The assumption that leaders were born to be leaders and that they can lead innovation teams is in itself an innovation killer.

Factor 3: Rigid Structures

The organisational structure defines how specific activities are arranged and carried out to achieve the determined enterprise’s objectives. A variety of different types of organisational structures exist, e. g. functional, divisional, or matrix. They all have their advantages and disadvantages, especially when considering specific market conditions.

Most of today’s organisations are designed to be efficient with a clear chain of command in an established hierarchy. These structures that the management has optimised to coordinate work, foster specialisation, and drive performance and results over the last decades pose a natural threat to innovation for several reasons. Departmental silos, rules, and procedures tend to limit the opportunity for exchange and collaboration across an enterprise. 

Chains of command can slow down the communication and the decision-making process when ideas are communicated and evaluated up and down the structure between different groups or upper and lower management. Ideas from lower-level employees have to go through many stages on the way to implementation. They face an uphill climb and veto rights from supervisors, who have a strong focus on efficiency and thus different priorities. That is particularly serious, as employees at the lower levels of the hierarchy have the most customer contact. They understand the specific and changing needs best and therefore often develop good ideas. 

Dongil D. Keum (Graduate School of Business, Columbia University) and Kelly E. See (School of Business, University of Colorado Denver) published their research results on „The Influence of Hierarchy on Idea Generation and Selection in the Innovation Process“5 in July 2017. They elaborated on how the hierarchy of authority influences employees’ behaviour in the idea generation and idea selection phase of the innovation process. This particular distinction is beneficial as it provides a more nuanced perspective, including aspects of behavioural economics on specific aspects of innovation within enterprises. They discovered that while the hierarchy of authority inhibits idea generation to some extent, it is beneficial in the idea selection process. This positive effect might be lessened as the ideas still have to make progress through different hierarchies of an organisation. Jeff Bezos has often pointed out that supervisors can kill proposals at many levels and that they need to pass many gates. Furthermore, it has to be regarded that idea generation and idea selection are part of a process and thus interlinked. If fewer ideas are generated, a smaller number can be processed further and selected.

The NIH (Not Invented Here) Syndrome can impede innovation as well. This theory is based on the in-group bias, also regarded as in-group-out-group bias. It implies that a team (in-group) might be defensive about or reject ideas from another team (out-group). A strong competition can evolve that hampers both exchange and collaboration. Especially in strong siloed structures, the NIH can cause severe damage to innovation as ideas have to overcome even more obstacles.

Strong and established vertical hierarchies have difficulties in anticipating and adapting to a dynamic environment. For the reasons mentioned above, they reduce the sense of urgency and the drive of innovation projects significantly, increasing the probability of failure.

Factor 4: Short-sightedness

Image by Graeme MacKay, MacKayCartoons

With the pandemic, individuals, societies, and organisations alike currently experience a crisis that has caused many limitations so far. It has changed the way we work, the way we do business, or how we meet and interact. It has pushed few companies to new economic heights but led many others to the brink of extinction. The overall consequences are still difficult to assess. Especially within a crisis, the focus usually shifts from medium- or long-term to short-term.  

In March 2020, a wonderful cartoon by Graeme MacKay went viral – a great work of art, by the way.

While the cartoon has a different intention, it impressively visualises our genetic predisposition. In the face of danger, as humans, we automatically focus on what lies immediately ahead and make our decisions much more concerning our imminent future. When we encounter a grizzly bear or mountain lion while hiking on our current vacation, we will immediately quit mulling over our next trip to Hawaii. 

So, in the face of danger, we instantly turn to deal with the potential threat. Furthermore, research in the fields of behavioural economics and psychology on cognitive biases shows that we also tend to overvalue an immediate or short-term reward over a medium- or long-term one. This is called the Present Bias – $10 today is better than $20 in a week. 

The Oxford Dictionary defines Short-Termism as „Concentration on short-term projects or objectives for immediate profit at the expense of long-term security.“ The preferred valuation of immediate rewards and – additionally – the reaction patterns to crises point to a clear prevalent short-term perspective in today’s economy. A predominant focus on monthly, quarterly, or annual financial objectives, the imperative to perform well in dynamic markets and not lose ground to competitors has always been a challenge for innovation. In terms of crises, additional pressure forces enterprises to be alert and completely shift the attention to the more immediate risks. 

The fact that short-term objectives dominate long-term perspectives is a prominent innovation killer. Focusing on the long-term is an essential principle for Jeff Bezos. He points out that it enables innovation. Innovative projects require funding, patience, tend to fail more often, and ultimately take longer to pay off. When there is danger ahead, innovation projects are usually the first to be either stopped or cut. Thus, prioritising short-term objectives, for whatever reason, will always jeopardise successful innovation.

Conclusion Part 2: The Organisation

Many enterprises struggle with the volatile, uncertain, complex, ambiguous and dynamic world of today. Yesterday’s success factors are ineffective today and might even be  harmful tomorrow. Long-established organisational structures and out-of-date cultures impede creativity and thus discovery and implementation of innovative approaches to tackle the arising challenges. In enterprises four factors threaten innovation: 

  • Organisational culture,
  • lack of leadership,
  • rigid structures and
  • short-sightedness. 

Generally, it is not a singular of these causes that leads to the failure of innovation, but a combination. It is however, just the tip of the iceberg.

Organisations consist of teams and individuals. Their significance in the failure of innovation also needs to be examined more closely – in Part 3 of the Innovation Killers: The Individual and the Team we will discuss this in more detail. Please stay tuned…

Many thanks to Graeme MacKay sharing his masterfully crafted cartoon with us for use in this article.

Sources:

1 Vogel, R., Schlossberger, E. (2018): Welle der Wirksamkeit – Das Modell für nachhaltig erfolgreiche Organisationen im digitalen Zeitalter, Springer Gabler, Wiesbaden.

2 Schein, E.H. (2004): Organizational Culture and Leadership, 3rd Edition, John Wiley & Sons, San Francisco

3 Almirall, E, Casadesus-Masanell, R (2010) Open versus closed innovation: a model of discovery and divergence. Academy of Management Review 35(1): 27–47

4 O’Reilly, Charles A. and Tushman, Michael L. (2008). Ambidexterity as a dynamic capability: Resolving the innovator’s dilemma. Research in Organizational Behavior, 28: 185-206.

5 Study „The Influence of Hierarchy on Idea Generation and Selection in the Innovation Process“ by Dongil D.Keum (Graduate School of Business, Columbia University, New York), Kelly E.Seeb (School of Business, University of Colorado Denver, Denver), published 27th July 2017