Maestro is a workflow module for Drupal which is in development by Nextide. We have created a design spec for this application, which we would like to share with the Drupal community for feedback. Maestro will be based on an existing workflow application Nextide has developed on another OS platform. Rather than a simply porting over the application, we would like to take the right steps to ensure it is properly integrated into Drupal.
For a quick 2 minute introduction of what Maestro is, check out our video
1. Maestro Overview
Workflow for CMS can be defined as the publishing of site content but there are also many business processes that can be automated and better managed via workflow like expense submissions to complex business processes such as new client provisioning, change management systems and work order approvals. While there is a clear need for solutions to provide hooks or triggers to existing Drupal content management data types and processes, we feel there is also a need for a solution that can define and manage complex business workflows. These can involve the routing of forms and attachments among task owners for different levels of approval, and include route changes depending on form data or other real-time routing control decisions.
Over the past 4 years, Nextide has developed a series of tools and applications for another PHP, Open Source framework which is used to manage and automate complex business workflows. We want to adopt and migrate these applications to Drupal and in doing so have been investigating what's available now for Drupal, and how best to integrate our solution into the Drupal framework as a native, best-in-class workflow module. Our other contributed module, filedepot, is an example of a tool that was developed in a similar manner over several years on another framework and which we have since ported to Drupal.
The existing workflow modules on Drupal provide a number of great solutions and show how flexible Drupal is and easy to extend with it's triggers and actions. These are features that we definitely want to include, however our solution goes beyond what is currently available. What if you could use a visual graphics workflow application to drag and drop a series of different task types with 'OR' & 'AND' type connectors to model your business workflow? These tasks could be: fill out a form, review and approve followed by conditional test task type that would route the workflow back to the originator if rejected with comments or move the workflow forward to another series of tasks.
Users will have a task console to let them manage their assigned tasks but will also receive notifications for new tasks and reminders. Supervisors can automatically be notified if tasks are past due and can review where in the process the workflow is. What tasks are outstanding, re-assign tasks, review forms, attachments and comments associated with the workflow instance.
There are a number of administrative functions that allow workflow admins to get detailed information on running and completed workflow instances, including process tracing. Workflow admins can re-assign tasks, send reminders and look at other users task console. If needed, workflow admins can execute tasks on their behalf and users can also setup their 'away' status and automatically re-assign new tasks to their delegate.
The workflow engine maintains statistics on all the tasks (assigned time, started time and completed time) so that it's possible to report on elapsed time for tasks, complete workflow instances. Managers really like to know what the average and worst case turn around times are for dept ABC to complete certain tasks.
The workflow admin maps out the workflow using the workflow editor creating a template. It will be possible for a workflow template to link or trigger other workflows. The single template or combination in case of more complex flows are defined as a project type. Each instance of a project type is tracked as complete project and all related tasks, forms and comments are viewable.
An example of a project is a 'New Employee Hire' or 'Office Supplies Order'. Users initiate a new workflow of type 'Office Supplies Order' and it would have tasks assigned to 'Supervisor' and 'Office Manager' roles. If the amount is greater the dollar amount then it requires the Department VP approval. The order is received and the initiator is notified. Complete workflow history and data is tracked and available for all that need to view and report on.
Additionally, it is worth mentioning that the tasks executed in a workflow are not advanced by user clicks, they are advanced by a task orchestrator which runs in your system behind the scenes. One of the main reasons for this is to ensure scalability in a site that runs thousands of projects and tens of thousands of tasks. The orchestrator handles all behind the scenes logic, such as sending a task notification or processing system tasks.
1.1 How Can the Drupal Community Help
The purpose of this document is to engage the Drupal community before we fully define the specification or start the module development. We need to be careful about scope creep, but we have much already available to migrate. Our experience migrating the filedepot module is that this will be at minimum a few man months of effort. We want to ensure that we include any key core Drupal integration features in the first release and plan for design items that need to be in the post 1.0 release. Identifying and prioritizing these requirements is where we would really appreciate feedback from the Drupal community so we can can have the best possible end result.
Our existing workflow solution is called Nexflow and will be known as Maestro when we port it over to Drupal as a module. We do not want to just port Nexflow over as it exists, it's important for the new module to be fully integrated and take advantage of the Drupal functionality. We have looked at a number of current Drupal workflow modules which all lack the full capabilities of Maestro and ultimately a true workflow automation solution. The key areas we think would be good integration points are actions/rules modules, CCK and the webform module.
In addition to porting and integration with Drupal, Maestro gives us the chance to take a step back and look at the Nexflow application improve its underlying framework. We have had Nexflow in production for several years with 200,000 tasks executed with 1000s of forms. After watching Nexflow in production environments we still see areas for improvement that are now possible when porting it to the new environment.
In summary, the goal with Maestro is to port Nexflow to the Drupal environment, integrate it fully, and improve upon the architecture to create a flexible and fully robust work automation system.
1.2 Drupal Integration
There are core areas that Maestro will integrate with Drupal. These areas include actions/rules, Drupal API hooks, as well as nodes, content types and taxonomy. The design of how Maestro's design uses Nodes, Content Types and Taxonomy is still in flux and an area we are looking for more Drupal development insights.
A workflow starts off life initially as a template created by the Template Editor. A workflow template will consist of many different types of tasks as well as process variables. Once defined, a workflow automates a specific process, anything from a capital request to adding a new user to the system. A project on the other hand, consists of many different workflows. A project also houses project-central data such as project resources, a project numbering scheme, and distribution emailing. With a project you can define that certain workflows must be completed before a new workflow can start, such as a project kick-off workflow, followed by the project closure workflow. Using Drupal's taxonomy, the related workflows will be linked to as projects. Its also worth noting that one workflow can be used in two different projects, sometimes can be executed more than once within a project, example: a request for information workflow.
- Can we manage this relationship using taxonomy or will we need to maintain these relationships in our own module tables?
The current thoughts are that workflow instances will be nodes and related workflow instances are grouped into projects using taxonomy. Each workflow template is unique and describes the workflow tasks and task routing with connectors and conditional logic.
- Should a workflow template be a content type or not? We think they need to be and that each instance of the workflow instance is a node. We can then link other related workflow content like forms, comments, attachments and even other linked workflow instances (as in a multi-workflow process) together using taxonomy.
What other Drupal data model related strategies should be considered in the design of this module? Most sites may only have a few workflow templates but our largest client would have about 30 workflow templates.
- Would that be difficult to manage if these were individual content types?
- Should workflow templates be nodes and not content types?
- If workflow template are nodes then how are we relating workflow instances once a user starts a new process - do we use taxonomy for that?
- If workflow instances are not nodes, how do we link in all related content like forms, attachments - do we just use module specific tables. With workflow instances being unique notes of a content_type, we have a nice high level Drupal data structure.
- What are the pros and cons that we should be aware of an considering when developing this high level data model / Drupal module architecture?
Drilling down to the task level, we do not see individual tasks being nodes but would be related data maintained in our module tables.
- What would the benefit be to site admin's for tasks to be nodes?
As part of the task definition, there would be a number of task properties that can be defined such as:
- Task Owner - assigned by role, username or process variable
- Task Notifications on assignment, completion and escalation rules
- Task type specific settings
We also see an opportunity to hook in actions and/or rules modules at the task level. When defining a task, it will be possible to set an action to launch on task creation or task completion. In addition, there will be a new task type added which simply wait for a trigger to occur. This will provide the ability for the workflow to automatically realize system events have occurred, such as a new page being added.
1.3 Existing Drupal Workflow Modules
We are not experienced enough with the existing workflow modules to know all of their strengths and recognize that there are work arounds to achieve almost anything. Perhaps other community members can add their comments and ideas based on use cases that they have deployed so that we are presenting our comments in a fair light. However, at this time there appears to be some key differences between these modules and what we are proposing with Maestro.
One consistent difference is that the idea of assigning a user a task does not exist, instead site users do something which triggers and action to occur. With Maestro, site users will initiate a new process (example request office supplies), and then other site users will be assigned tasks to complete in the order required. Users assigned tasks will be able to access them from their task console and may have only 1 task or possibly many for various other and different workflow instances. The types of the tasks in the other modules are essentially to complete a content type. With Maestro other task types exist such as to review submissions, approve or reject submissions, or answer other interactive questions such as assigning subsequent workflow step owners and changing the workflow routing. Workflow admin's have the ability to cancel workflow instances (projects) or re-assign task owners.
While all three of the modules compared in this section are quite powerful, there is a common drawback with all of them that really separates Maestro. The main difference between Maestro and the other modules can be summed up in one line: rather than waiting for site actions to occur and acting on them, Maestro drives the site actions to occur.
1.3.1 Workflow
The Workflow module, when combined with the Actions module, has the ability to create basic workflows. It can create "pseudo-tasks", which are represented by the state of the workflow. For example, if the task was to create a form, the state of the workflow would be "Create a Form", rather an a generic "Draft" or "Initiated" state.
The Workflow module integrates well with the Actions module, which is almost required to create a working process. Since workflows are built around content types in the system, the workflow module is well suited to facilitate site content creation.
Unfortunately, as we will see with the other modules, the Workflow module is based more around site actions and site content types rather than defining standard processes that get executed. Instead of a specific user being assigned complete a task as part of their job, it is up to the user to create a content type in the system, at which point the system will direct the lifecycle of that particular content type.
Maestro aims to build business project information, where multiple content pieces come together to form a project. Arbitrary documents can be uploaded and site users are actively assigned tasks which they can easily see in their task console. While Maestro will integrate with site content types, its purpose is not to build site content, but rather facilitate a company's procedure to complete work.
The Workflow module also lacks in conditional task abilities, it does not have any metric reporting, and has a clunky interface to create a workflow, which are all areas that Maestro will improve upon as well.
1.3.2 Workflow NG
Workflow NG is a more developed workflow solution which can handle certain conditional logic, and integrates well with other modules. However, it is no longer supported for future releases of Drupal as it has migrated to the Rules module. All of its strengths and weaknesses are similar to the Rules module, so lets leave it at that and jump straight into the Rules module comparison.
1.3.3 Rules
The Rules module is a great module to track system events and act on them. You can build up a workflow based on system events, so, for example, a form is created, that event is trapped and signals the next task to be created. You can define a ruleset in the Rules module which allows you to group related rules and execute them in order, to simulate a workflow. Rules is well suited to define a workflow to built site content and to act on site events.
However, similar to the workflow module, it does not support the creation of a standard "task", such as requiring a user to complete an action. Like the Workflow module, it waits for an action to happen, such as a new site content to be created, then performs actions on that, where as with Maestro, a process is defined to assign site users to create the content, and metrics are stored to track how long certain tasks have taken to execute.
With rules, there is no way to know what tasks are assigned to you, unless you set up an action to email the task owner on task assignment. The lack of a task console and a project dashboard page is another area that Rules lacks that Maestro will improve upon. Again, like with the Workflow module, Rules relies on system content types as the means of executing a workflow, where Maestro helps facilitate the building of business project information.
1.4 Planned Integration with Drupal Modules
1.4.1 Actions
Integration with the actions module is key. There are a number of different opportunities to integrate with the Actions module. Number one, we plan to include an action that launches a new workflow when trigger is fired. That will allow, for example, when a site user is created, launch the workflow that defines all the things that need to happen when adding a new employee to your company, like setting up their email address, etc.
Another key opportunity for integration is inside each different task type. On both task creation and task completion we will have the opportunity to execute an action, such as sending an email, or publishing a content type. The main purpose of this would be to send task notifications to users, however by integrating it into an action it makes that extensible.
The final area of integration into actions will is that we will have a task type that waits for a trigger to occur. So, if the workflow requires a user to be created, then the "new user created" trigger is fired, the workflow will continue on.
1.4.2 Tokens
For email notifications, tokens will be used to change out values like username, etc. It's important to be able to send out email notifications for task assignment but also send out task reminders, task alerts as to an identified contact for escalation if task not completed in x days and also task completion notifications. Our present workflow solution allows these different notifications and notification rules like sending the initial task reminder after 5 days and then every day, send an escalation to the initiator of the workflow after 10 days if task not completed. Being able to customize these notification email messages using tokens will be supported.
1.4.3 Rules
Maestro will be initially lightly integrated into Rules, where there will be an opportunity within Rules to create a rule which launches a new workflow. That again will allow a workflow to be launched after a specific site action has occurred.
In Closing
We would like to include the Drupal community in the development of this application, and include your ideas and feature requests as well. While initially we want to get this up and running in Drupal all feature ideas and feedback on how to properly integrate into Drupal will be considered in the initial port, and subsequently implemented in future releases.
This post contains the initial section of the Maestro spec and our plan is to publish the additional sections over the coming weeks which will get into more technical detail. That detail though is dependent on our initial high level spec and ideas/questions proposed in this document and why we are anxious for community input.
The Nextide Team
Focusing on Business Applications but heck we do CMS and Social Networking projects too
www.nextide.ca
Comments
Looks cool!
Looks cool!
Discuss with fago
Hi - I encourage you to discuss your project thoroughly with fago, the maintainer of Rules. You may find a way to build an add-on to rules or otherwise collaborate on that. Plus you may benefit from his experience with this type of module.
Is this intended to be d6 or
Is this intended to be d6 or d7 stuff? I'd suggest you to do any serious development in d7 now, it will make sure it plays well with d7, but much more important you can leverage lots new great d7 stuff, mostly entities + improved fields.
In d7, I'd suggest making your workflow task a fieldable entity, therefore any entity can be easily categorized by adding a taxonomy reference field. In d6, you can't.
ad rules for workflows:
Yep, it's a rules engine, no workflow engine. You can use it to build workflows, but there is no definition that drives the workflow. But having such a workflow definition (-> entity) which allows reacting on its changes / applying changes to it makes much sense to me. Anyway, I'm working on integrating web services with rules-7.x-2.x such that this enables one to create distributed workflows.
Generally for d7 and simple workflows, I think it makes also sense to leverage the group module. Assigning the a group to an entity, let's say a node would represent a state of the node. Such we could have a group for "drafts, needing work", "drafts, needing review" and maybe also for published nodes. Then you can assign people to the groups that are allows to edit the group's content + approve it such it goes a step forward + you can define group based permissions. This is fine for usual editorial workflows, e.g. alfresco does it that way.
Assigning a task to a single user has some similarity to usual case tracking system, such as open atrium. So it might be useful to integrate with something like that, or not.
You could implement this notificiations with Rules. Questionable is just how the UI should look like. With rules 7.x it's easy to integrate existing UI components of the module somewhere else though.
Our intention is for D7
Thanks for the quick reply fago. Our intention is to make this a D7 release in order to take advantage of as much new D7 functionality as possible.
Maestro's goal is to extend workflow well beyond the content type workflow that is currently very well supported in Drupal. Our existing engine is currently in production at a number of corporate sites which handles a great deal of workflow functionality that is not CMS/rule based. We're trying to bring that functionality into the Drupal environment with Maestro.
Good points on using Rules to implement notifications. We'll take a look at Rules to see how that can help us.
Very interesting
Hi,
Maestro sounds very interesting and I will definitely follow its development.
I am no module developer myself, but have a fair understanding about it and I would like to offer some feedback, as well as questions about your project.
For me it looks like Maestro will be a very flexible module that not only will will offer traditional workflows, but also tasks, triggering and dependencies of other workflows. Since websites today are getting more and more complex, as well as integrated with other websites, mobile devices and almost anything that can be digital, Maestro, from my point of view, will greatly enhance the ability to get a better overview as well as management of both content and functionality.
I have previously looked at most of the workflow modules for Drupal and while they are good and perform their tasks most of them are either lacking features we need or making them work the way we need them adds complexity either in configuration or usability. One example of that is to be able to work on an update of a node without having to unpublish the existing live version.
I do hope that Maestro will make this much easier. Will it for example be possible to:
Tasks
The tasks integrations sounds intriguing for me. Especially now when publishing new content or adding new features often involve much more than just the content/feature itself.
By being able to add and assign tasks to workflows it would be possible to make sure that nothing important is missed due to stress or other things when managing a busy website.
Today many websites utilise Twitter to post a short message about that a new blog post or article has been published. If Maestro would offer an API, maybe using Rules, so that modules like Scheduler and Twitter easily can integrate with it. Then when the Scheduler is publishing the node, the Twitter module is triggered to post the Tweet with the content the author/publisher added when scheduling the publishing during that workflow task.
Another way of adding integration could be if the site offer newsletter subscriptions. Then a task could be to decide if the new content or feature shall be queued for inclusions in the next newsletter. Integration here could be to be able to set if the inclusions should be as simple as the teaser or custom content just for the newsletter.
Then that will be queued for the newsletter workflow and the owner of it will have everything needed in one place making organising and publishing the new newsletter a snap.
Bigger Project Workflows
If I understand your proposal correct, the above would be quite simple task and Maestro will be much more powerful that that. Especially since it will offer the possibility of dividing a project into several different workflows, where they depend on each other.
Will it then also offer the ability to change configurations or automatically create content based on custom forms built into the workflow?
Lets say I have a site where I create topic pages. These pages are focused around a certain topic and consists of a combination of modules, content types, blocks and so on.
Would it then be possible to break the creation of these pages up into smaller workflows and tasks in combination of forms that contain the unique settings, such as creating a new context, as well as text and images to be used on one such page? Then it would be very easy to let anyone with the right role be able to create these pages on the fly without having to involve developers or designers.
The module itself
As I mentioned in the beginning I am no Drupal module developer, but have a fair understanding of the module concepts.
In your text you talk about Maestro being one module, but integrated with other modules such as actions, triggers and rules. With one module do you imply it will be one monolithic module containing all the listed feature, or will it be one framework API and then lots of small sub modules that websites can pick and chose between based on their particular needs?
Personally I would prefer the latter as it would both offer much more flexibility as well as making it easier for others to either easily integrate their existing modules or create new modules that take advantage of your framework.
To end my comment, I am glad you are focusing on D7 as that will provide so many new and exiting possibilities as mentioned by Fago and yourself.
Looking forward to the progress and would also like to offer to help as much as possible with testing and other things.
--
/thomas
T: @tsvenson | S: tsvenson.com
Good points
Hi Thomas,
You have some great ideas for integration points. To answer some of your questions, the scenario you posted is do-able with Maestro, but would actually be a simpler scenario. Maestro is capable of handling many tasks in a workflow. The scenario you posted would be a basic two task workflow:
I think that we're getting hung up on the idea of building website content. Maestro's predecessor, Nexflow, has been in production with workflows that orchestrate anywhere from 5 to 50 tasks! Granted, some of the larger workflows could really be broken down into smaller ones. While Maestro will of course be used to facilitate creating website content like pages and blog articles, that is not its only goal. Where Nexflow has been used is to organize a large business process where the company has a flowchart of a process, such as bringing a new product to market. That process can include multiple different flowcharts, and when Maestro is used to orchestrate, it captures all the information that company needs to make sure all the bases are covered for deploying their new product. Those bases can include research and development grants, safety regulations, copyright issues, and even how they will pack the product into the shipping truck. That is just one example, but it gives you the idea that Maestro is more than just a website content building tool.
Getting into your "Bigger Project Workflows" questions, I think the above paragraph applies here too. An example of how Maestro will orchestrate custom forms, lets say you create a form content type with the Webform module. With Maestro, you will be able to define a task to have someone submit that form, and along the process of the workflow, continue to work on that form, editing it and adding information along the lifetime of the project. In fact that node created is not just limited to the workflow it was created in but also other workflows that belong to the overall project. To directly answer your question, Maestro will be able to break apart the creation of content, that can be completed over multiple tasks and throughout different workflows within the project.
Our current action plan for Maestro is to port Nexflow over to Drupal, taking the necessary steps to properly integrate it with Drupal. In terms of breaking it into multiple sub-modules, the core functionality will all be in one module, as well as a standard set of task types. However we are working on ways to modularize task types, to allow developers to write their own task type modules that they find helpful to process content. And as for the Maestro APIs, we are definitely planning to expose some APIs to allow developers to integrate easily with Maestro. Just one other point of clarification, in terms of our integration with Rules and Triggers, we will not be rolling it into Maestro but simply using the features they allow. For example for triggers, we would like to add a trigger which will launch a workflow, so say if a new site user is created, it will launch the add company employee workflow. Also we would like to add the opportunity to fire a trigger when a task is assigned or a task is completed.
Your integration with twitter is a good idea, when Maestro is used for building website content and we will start thinking about the best ways to develop that feature. The other one you mentioned was scheduler which was talked amongst the team here awhile back and must have been forgotten, so thanks for reminding us as that is another good enhancement we would like to add.
Regards,
Eric
__
Hi Eric,
Many thanks for your detailed response. Made things much clearer as well even more interesting.
A couple of thoughts:
If I understand you correct, it will be very easy to make the workflow more or less invisible. Personally I am not very fond of having to teach staff about the proper way of moving between workflow stages. With Maestro, I did look at the NextFlow introduction video on your site, I foresee that for the users stages wont exist. Either they will have a list of tasks to pick from, or the tasks are completely invisible but still managed by the workflow.
I also like the way you describe that forms can be filled out along the workflow. Especially since it is not always possible to have all information and filling in garbage just to pass any form verification is never a good solution.
As you say, the idea behind Maestro is not only to manage content, its much more than that. I definitely see uses of it for a lot of other things. Such as administration, user management, product launches and so on. By utilising features in Triggers and Rules it will be easy to for example create a workflow that triggers when a user applies on the site to gain access to more features. The workflow can then present the user manager with both the application information for review and a UI where they quickly can apply all settings for the user, such as changing/adding role(s), adding info on their profile and more, without having to manually go to each individual configuration screen.
The scheduler I mentioned was more about when content shall be published. However, I can see a lot of use if it was possible to schedule workflows. Take the newsletter I mentioned as an example. Newsletters are usually published on a regular schedule basis. Then if it was possible to schedule the workflow itself the person responsible for creating the new newsletter can automatically get "Create this weeks newsletter" at a specific time in their task list.
Will it also be possible to control when a task has to be completed as well as notify owners/managers about it?
I've noticed that you talk about the workflows as nodes. Is that the Drupal nodes you mean? If so, I'm not sure that is the best way to do it. The nodes for me is for content that is presented on the pages, while I see a workflow more as something that manage content.
A problem with Drupal is that there is no clear separation of what is content and what is configurations in the database. Modules such as Chaos Tools, Features and Strongarm are trying to solve that. Especially Chaos Tools and Features should be interesting as they will make it easy to export Workflows into modules to make it easy to move from for example dev site to staging to live, including managing dependencies on other modules.
With that it would also be a snap for you to host a workflow repository where users of Maestro can share workflows.
And a wild one:
We have started to use BaseCamp for our project management needs. If Maestro could act like a bridge between Drupal and BaseCamp I can see a lot of advantages, especially when it comes to tasks linking them directly to the right pages on the Drupal site among other things.
--
/thomas
T: @tsvenson | S: tsvenson.com
___
Hi Thomas,
I'm glad you're taking interest in our module. Maestro will, from your perspective, make the workflow invisible. Users are assigned tasks and they complete them. As far as stages go, within a project you'll be able to define key points in a group of workflows that sets the stage of a project from initiated to active to completed, etc. Now this is a different concept from how the Workflow module handles stages, as a stage in the workflow module is more closely related to an actual task assigned in Maestro. Users will never have to change the state themselves, that is all defined by the administrator when creating a workflow / project.
In terms of the scheduler, my initial thoughts were to be able to schedule tasks to be completed at a specific time. Scheduling content to be published at a specific time is a feature of the scheduler module that exists today, and hopefully with some crafty development of Maestro, assigning a user a task to complete a content type, will inherit the scheduler's features as well as any other inherited features. The last point about scheduling the launch of a new workflow is another good idea too that I'll add to our would-likes list. Keep in mind for version 1.0 we would like to get this working in the Drupal environment, paying attention to the spots that need to be integrated off the bat. Some of these features we're talking about will not be forgotten but moved to version 1.1, which will start as soon as we get version 1.0 complete; Just a development road map to keep things moving smoothly.
In terms of nodes, it was an early thought to have either workflows or the full project type as a node, so the use could simply go to the site menu and "create content > Maestro project". Then whenever you viewed that node it would show you the full project dashboard view. However, there has been some discussion internally which now directs us to your thinking. Nodes are meant to display content, and projects/workflows will manage it. So what we have decided for now is to not include that type of node integration at least for version 1.0, at which point we can take a step back and see if there will be value in it.
In terms of exporting a workflow, that feature already exists in Maestro's predecessor, so we should be able to export workflows without the use of the modules you mentioned. We'll definitely take a look at them to see if maybe we'll just use their features instead of building our own. Great idea about hosting pre-made workflow solutions.
Last but not least, integration with BaseCamp is definitely something to look at after the version 1.0 or even after version 1.1, but our direction with Maestro is to build a project management tool around the workflow system. Hopefully when you test out Maestro you'll see it can handle some of the project management abilities too :)
Thanks again for your interest and ideas. Your thoughts are giving us lots to talk about here!
Regards,
Eric
Interesting
Hi, I too used to work in the workflow automation arena and I've spent a lot of time thinking about it. Now I work with Drupal so the lack of automation tools has been something I'm aware of (though it has to be said that most of my clients are not!).
One thing that I noticed was that you have a flowchart -style editor in your current product. While I realise that the flowchart-based approach is fairly easy to sell to end users, my experience has been that it's not always so great when developing really complex flows unless you have some specific tools to aid re-usability. Typical scenarios will often involve business rules that can affect multiple objects across a flow, for example.
An example of this is escalations that apply right across a flow. Say, in a customer service situation if an issue hasn't been dealt with in a specific time, it might need to be escalated regardless of the current state of various subtasks. You don't want to have to put a condition like that on practically every box in a chart, you want to be able to group them all together and say "this rule applies to this entire set".
Since you've been in this game quite a while, I expect you have answers to this type of issue, but I'd be interested to know how you tackle them.
Looking forward to learning more about Maestro, anyway. Looks like a major step forward.
____
Hi Al,
The flowchart style editor was a feature that was added to Maestro's predecessor, Nexflow, as a natural progression for the tool. The much earlier version was having a "card based" layout that was just straight list of tasks with no lines pointing to the next task however a "next task id". That made it very hard to administer a workflow and the more complex a workflow got the less appealing the card based layout became.
In terms of re-usability, there are two points where Maestro will help reusing an existing workflow. You can use the same workflow in different projects, but also, in the template editor, you can copy and export workflows, and from the copy of the workflow you can make changes to make a different workflow. That helps save time when creating new workflows, but in our experiences, it takes about a day to think out and define a workflow, then that workflow gets used for a few years. Over those years of course there are small changes to the workflows but again, a few mouse clicks to edit the tasks and you're good to go for another long stretch.
So when it comes to applying escalations to multiple tasks, we do have the escalation feature which can be defined in each individual task to allow flexibility, but to "bulk edit" tasks has not been a issue that has come up. Additionally, when there were multiple tasks that needed escalation within a workflow, we found that they still had different escalation rules. Though bulk editing tasks would be a nice feature to have in future versions, the development priority tends to go towards the user's task console which gets used every day rather than the actual defining of a workflow, while important and still nicely enhanced.
Regards,
Eric
Great new demo video
Hi,
Saw your new demo video on YouTube yesterday and it blowed me away. Can't wait to be able to take it for a testdrive.
Wrote a blog post, Maestro - Supercharged Workflow for Drupal 7, hopefully it will create a little more awareness about this project.
--
/thomas
T: @tsvenson | S: tsvenson.com
New blog post on our site
We will be adding new blog articles on our site about maestro as we prepare for the first release.
Our first blog article that explains the different task types is now online.
Alpha version released!
We've now released an ALPHA of Maestro available for download from the Maestro project page: http://drupal.org/project/maestro
As we switch from heads-down-full-on development, we're also trying to follow up this first release with Blog posts that highlight functionality as well as how-to information.
We will be adding more posts over the next little while to help expand the Maestro knowledge base.
Good job and a simple request :)
Hi this module is really great and helpful. I would like to ask about using option of this module. I need a workflow like this;
1- Staff or customers will choose the items that they want from administration or store. (I can use a shopping cart module without payment or ubercart)
2- When the order is given; this list will be approved by supervisor and will send the order to director or reject the items that is not necessary (inform the staff or customer and director at the same time)
3- order will go to director
4- director will send the order to be bought or reject the items that is not necessary result will go both to supervisor and staff or customer.
What I mean is I need to use this module together with ubercart or another shopping cart (store) module.
this can be a great option :)
good question - a few ideas
Good question and this is definitely the kind of workflow or process we want to see maestro handle.
Here are a few ideas:
A maestro workflow can be launched from a trigger so in your scenario, that could be a way to launch the item(s) review/approve workflow.
In the workflow, you would use the interactiveTask as a review task where you supply via a custom function the information or HTML to display the items and options for the supervisor to review. We have provided an example in the content_publish workflow that is included - it's packaged up as a module. Since it's your own custom function, you would be able to access the ubercart data or API's and do whatever you need.
This task being interactive, would present the supervisor with options to accept all, some or no items. You can then test the results of this task and branch in the workflow accordingly. Maestro allows you to branch forward or loop back in the workflow - effectively regenerating the workflow but tracking it as such. Refer and test the content_publish workflow for an example of this.
Upon final approval, you can use a batchTask to automatically call the ubercart API's to process the order.
At each task, you can set email notifications or use the TriggerTask to fire off a email action. There is also the option of writing your own batchTask if you want to do something more complex.
Alpha Version 2 released
We have just wrapped up development on our Alpha Version 2 release which adds some new features and refinements.
The maestro module was designed to allow new task types to be developed and installed as modules and so in this release, we have added a new task type, the InlineFormAPI task. This task lets you easily create an interactive task that can use the full power of the form api. The workflow Editor let's you edit the task details online and define the form fields - assign the task owner, notifications policy and map it into your visual workflow template. One common use of the formApiTask will be to prompt a user (maybe a manager) for a department, request type, assignee etc and have a subsequent workflow branch depending on these results or set the future task owner.
Similarly the ContentType task was extended so that field results can also be use to set process variables that are used to assign future task owners or control the workflow routing. We show an example of that in our latest video, where we have used the 'user_reference' field in a custom content type 'Service Request' to automatically route the work request.
Additionally, we have added a few new blog posts about maestro over the past couple weeks.
Interested in your feedback and questions.
Restrictions & Revisions of content type instances/nodes
Your project sounds very flexible in many ways but a little restrictive in others. I'm planning to use the D6 workflow module together with views in order to create custom task consoles for different users/roles and workflows, and it doesn't look as if this can be done with maestro.
So for example, in the way I'm thinking about setting workflow up:
So with views and workflow it's possible to set up the task console for people in pretty much any way you want. If you want users to be able to track pages they have authored through the workflow process then it's very easy to make it so. It doesn't look as if Maestro would allow this. Am I right or do I not understand the module properly?
Also, does this module allow for saving revisions of pages too?
Maestro can handle that
Rukaya, you can essentially do everything you're after out of the box with Maestro. If you install the technical support request submodule of Maestro, you'll get a good example of how to accomplish what you're after. It requires the user reference submodule, so you'll need to get that from the CCK module for D7. Also it requires the features module which will install the content type automatically for you. The technical support request workflow is an example of multiple users collaborating on the same node, with the process guided by Maestro.
Maestro already has a built in task console. It does not use views for the task console, although views is used when viewing overall workflows. The task console is specific to each user based on what tasks they are assigned.
The main task type you're after is the content type task. This type of task is assigned to a user to create/update a content type and it is retained within the workflow for later use.
In terms of revisioning, Maestro does not directly support revisioning. But we do support content types which support revisioning. Again, I suggest installing the technical support request workflow, and install the Revisioning module to see if that suits your revisioning needs.
Eric
Will it play well with Storm?
I've been setting up a Drupal Storm site and this would be great if integrated. The one big problem I have now w storm is not having reminders for when a task comes due.
Thanks,
Wes
Storm Integration
Not sure about what the Storm requirements are (never used it), however we designed Maestro to have an extensible notification system. If needed, a new notification mechanism can easily be plugged into the Maestro framework without rewriting any core Maestro code.
Therefor, if Storm requires something specific to be done for it, a new Maestro Storm notification mechanism can be created.
I included a sample Twitter notification mechanism in the current alpha download for Maestro. Its just a placeholder mechanism to show how easy it is to create a new Maestro notification mechanism.
So not sure if that helps or hinders at this point -- but sounds like this is something that can easily be overcome.
Looking good...
I haven't really dug into Maestro that much yet but from what I've seen, it's looking great. A few comments....
I think you're right on with not making workflows into nodes.
I think that people are going to be itching like crazy to get more control over the task console using views. How likely is it that we'll see that any time soon?
I hope you reach out even further to leverage existing Modules and APIs. I'm sure it's tempting to want to keep a lot of this internal to this project but please let this thing become one with Drupal. I just think it would be a shame if this became yet another program within a program. I'm sorry to say it seems like there's a tendency for complex projects like this to go this way. The few discussions I've looked at about messaging, notifications, and the lack of views support, raise some general concerns about this. On the other hand, it looks like you may be striking a good balance in your level of integration with Actions, Rules, and Triggers.
Kind of along those lines, I think doing your own work implementing a module-specific mechanism for doing Twitter notifications seems misplaced (however easy it was to implement). I think this should be the domain of modules focused on implementing a universal framework for that type of thing. I know those other projects could benefit from you helping them out.
Lastly, I can imagine a lot of different business processes that could benefit from this. I do hope the development here doesn't end up too specifically supporting document and issue queue workflows at the expense of other types of workflows -- even though I'm sure that's what many of your initial implementers will be doing with it. I think this has a LOT of great applications beyond those use cases.
iterative task
Hi all, I hope this is the right place for posting a questiong about maestro workflow
I need to implement a task wich enables users belonging to a role, to attach many files to a content type, in the lapse of a week. So the task should remains available for that role at task console until a variable is set to a certain value (say close)
by now, I've accomplish this by doing a loop using an "IF" task, but the task is showed now with a preceeding [R] ("repeated"? or something) and thats not the real thing, really the task is still available to complete it (adding more files) and not being repeated
can you please give some light on how to leave a task "open", avaliable at task console untill a variable is set to a "close" value ?
thank you in advance
Need a batch task or custom interactive task
The [R] in the task name is for Repeated or Regenerated because your workflow is setup to repeat the same task. You can add a batch task before the IF task which checks the date and have that Task set the task state and then the IF task can branch back 'Repeat' the content task or else complete the workflow or proceed with the remaining workflow.
The other option is to create a new interactive task that will just not complete the 'Attach files' related task until a week has passed - leaving the task in the users task console.
thank you blainelang
thank you blainelang, I'll try second option
BPMN 2.0
Anyone ever consider shifting this module to BPMN 2.0 and using a BPMS like Activiti to handle the actual workflow and then using Drupal Services module and Rules HTTP Client for Integration between Acitiviti and Drupal.
BPMN 2.0 is a well defined standard that handles the most complex and simple scenarios with lots of documentation.
Let the BPMS handle workflow, and using Service Tasks in BPMN to communicate with Drupal using Services to tell it what to do.
I agree with you, BPMN 2.0
I agree with you, BPMN 2.0 could add a more standard approach to the business processes designed under Maestro.
Workflow and Maestro Modules
Hi! I'm new in drupal (and English too haha). I'm trying create a type of structure and I tested Maestro, Trigger, Workflow and Rules modules, but none them help me. When I finish a content, after updating a few times, I would like to close edition by the end date, and only a particular user had permission to reopen the content type to be edited. After close to update content, it would be possible to automate permission for the user to update other content (or webform)? Example: I create a content of content type A and update with others users. After a certain date, closes and can no longer be edited. After closing, you can edit content of content type B and same for content type C. With webform I can close and it's a formulary, it would better, but content type also useful.