By way of establishing a useful and effective benchmark that helps A:M Users produce a short film or movie I submit the following for consideration:
Any benchmark undertaken should help to advance an A:M User toward the creation of their image, short film, series or feature.
Any benchmark in the 1.0 series should be A:M-centric. (Comparing A:M to other programs is not the goal but comparing A:M to itself on various systems will reveal useful information that can be adjusted and optimized).
The fewer variable in a benchmark the better. With these considerations in mind we can move toward a proposal for the first 'official' A:M benchmark. Note that this would be the second series of benchmarks because the first would use only default settings found in A:M as determined by selecting "Reset Settings" in the A:M Help menu then performing and measuring a stated task.
The default benchmarks (right out of the box) would likely be termed Benchmark 0.0. Through effective benchmarking that progresses the A:M User toward their goal all variables can be known, their effects measured and altered to suit the needs of the user. Any subsequent benchmarks beyond the default should be meticulously documented with all variable identified to the maximum extent possible (an unknown variation may still be within control limits but effectively out of the user's control).
Where possible a default file should be shared with the those who will conduct the benchmark and for testing purposes it would replace the default in A:M (example: replacing the default.cho file).
Assets required for a particular benchmark can be narrowed in scope to a Project file, Choreography, Model, Action, Material, etc. to benchmark each of those areas. It is hoped the best benchmarks would then be incorporated into A:M as optimized procedures which when run automatically (at installation or other time specified by user) establish a realistic foundation from which to project the user's production schedule.
Note: Wherever possible the introduction of variables within the Benchmark should be used as an opportunity to learn. (Example: Telling A:M what file format to render to, where on the user's harddrive or network to render to and then padding the filename with zeros in order to optimize the sequential order of images produced) These represent three user specified variables that will have an effect on the user's production schedule.
A proposal for the first 'useful benchmark' beyond default settings tests would be to render a two minute movie (from a project file provided). Assets rendered from this benchmark would be used to assist in the creation of the user's own two minute movie. (This setup can then be easily altered to target still image (single frame), 30 minute short, 90 minute movie, etc.)