Design for the Sixth Circuit
Pre-Argument Conference Program
The evaluation of the Sixth Circuit’s pre-argument conference program will utilize a controlled experiment design. Cases will be randomly assigned to either of two groups (a treatment group, i.e., the group to be conferenced, or a control group—the group that will not be conferenced).
The appeals assigned to treatment will be randomized from the universe of conference-eligible cases ﬁled during the case selection phase of the study. The case selection phase of the evaluation will last for approximately one year.
Cases will be drawn from all of the divisional units in the Sixth Circuit.
The only civil appeals to be excluded from the study are those permitted by permission under 28 U.S.C. 1292(b) and appeals in habeas cases under sections 2254 and 2255 of Title 28. No other civil appeals will be excluded from the evaluation.
A small number of cases will be automatically assigned to the treatment group when a speciﬁc request to do so is received from either the Clerk’s Ofﬁce or the Conference Attorney. It is expected that the number of requests for automatic assignment to the treatment group will be relatively small (perhaps, less than a dozen) and will be accompanied by ample justiﬁcation. For example, automatic assignments might be utilized for cases in which counsel have already participated in a number of pre-argument conferences and have expressed a strong preference for continuing to have access to the pre-argument conference program.
In addition to collecting and analyzing case ﬁle data on appeals included in the study, questionnaires will be administered to panel judges in a sample of study cases. Attorneys involved in cases assigned to the treatment group will be surveyed as well.
The primary objective of the pre-argument conference program evaluation is to assist the Sixth Circuit to determine whether the program has an impact on: (1) the number of appeals settled; (2) the number of motions ﬁled; (3) the quality and length of briefs; (4) the quality of oral argument in cases submitted to the court following pre-argument conferences; and (5) instructing members of the bar concerning appellate practice in the circuit. In addition to assessing the above potential case management-related effects of the program, an effort will be made to evaluate the program’s impact on the number of appeals settled, voluntarily dismissed, or dismissed for want of prosecution. Increasing the settlement rate of civil appeals is viewed by the Sixth Circuit as the major objective of its pre-argument conference program.
The study will use the following criteria to assist the Sixth Circuit in determining whether the program is meeting its objectives:
Objective One: Increasing the Number of Cases Which are Settled, Voluntarily Dismissed, or Dismissed for Want of Prosecution
Case ﬁle data on the number of cases in the treatment and control groups which were settled, voluntarily dismissed, or dismissed for want of prosecution will be collected and analyzed. It will be an indication that the program has an effect on settlements and dismissals if the treatment group has a signiﬁcantly greater percentage of cases in the above three categories of termination when compared with the control group.
Objective Two: Terminates Cases Earlier in the Appellate Process
An effort will be made to determine whether the pre-argument conferencing program facilitates the termination of cases at earlier stages in the appellate process. In order to examine this issue, the following approach will be taken:
A. Calculation of median days treatment and control cases are active. For the purposes of the study, a case is deﬁned as active from the time of the ﬁling of the notice of appeal to time at which it is either terminated or submitted to the court. It will be an indication that the program is meeting the objective if the treatment group has a signiﬁcantly lower measure of median days active when compared with the control group.
B. Calculation of the percentage of cases terminating during each stage of the appellate process along with the median time cases remain in each stage. For purposes of this study, the following stages of the appellate process will be examined for each case in the study:
1) Time from date case is docketed in the Sixth Circuit until the date of the ﬁling of the certiﬁcate of record in the Sixth Circuit.
2) Time from ﬁling of record in the Sixth Circuit to ﬁling of appellant’s brief.
3) Period from ﬁling of appellant’s brief to ﬁling of appellee’s brief.
4) Period from ﬁling of appellee’s brief to time of argument.
5) Time from ﬁling of joint appendix to time of argument
It will be an indication that the program has the anticipated effect if the treatment group has a signiﬁcantly greater percentage of cases that terminate in the earlier stages when compared with the control group.
Objective Three: Reducing the Number of Motions Filed
A major objective of the pre-argument conference program is that of permitting informal resolution of procedural matters, such as joinder of briefs in appeals having multiple parties, thereby reducing the number of formal motions on which the court must act.
It will be an indication that the program is meeting this objective if the treatment group is found to have a signiﬁcantly lower number of formal motions ﬁled than the control group. It should be noted, however, that the process of counting the motions in each appeal included in the study may not be as straightforward a task as it might at ﬁrst appear. For cases assigned to the treatment group, for example, the brieﬁng schedule permitted by the conference attorney may operate as the equivalent of the ﬁrst extension granted on motion in the control group cases. Therefore, any observed differences between the treatment and control groups in the number of actual motions ﬁled will have to be interpreted with that possibility in mind.
Case ﬁle data will be collected and analyzed on both the frequency and type of motions ﬁled in all of the appeals included in the study.
Objective Four: Impact on Quality and Length of Briefs Filed
As part of the survey of attorneys in treatment cases proceeding to brieﬁng, each attorney will be asked whether the pre-argument conference program assisted in the preparation of the briefs.
It will be an indication that the program is achieving this objective if a signiﬁcant number of attorneys indicate that the program assisted in the preparation of briefs.
In addition, responses from panel judges at, or near, the time of argument or submission will be gathered on a sample of cases to assess brief quality. Responses will be collected by questionnaires, with each judge being asked to rate the quality of selected briefs in appeals submitted to a panel on which he or she sat.
It will be an indication that the program is meeting this objective if the treatment group is found to have a signiﬁcantly larger number of positive ratings than the control group.
As part of the pre-argument conference program procedures, the conference attorneys set the brieﬁng schedule along with limits on the maximum brief length that each party may submit. As part of the analysis of brief lengths, an effort will be made to compare actual brief length with the limits set by the conference attorney.
The process of determining the overall impact of the program on brief length is somewhat complicated by the probability that the pre-argument conference program is changing the universe of briefed cases in the circuit. That is, it is very likely that the program results in a number of the less complicated appeals settling or being withdrawn prior to brieﬁng. The program is also likely to change the timing at which less complex cases settle or withdraw such that there are more appeals settling or withdrawing at the pre-argument rather than post-brieﬁng stage. Assuming that this in fact occurs, we are likely to observe an increase in the average brief length in those cases that are briefed. Such a ﬁnding would mask the impact of the program on the less complex cases that settle or withdraw at the pre-argument stage. As a solution to this, the evaluation will utilize a case weighting system to separate out possible settlement effects of the program from any brief length-reduction effects.
For purposes of this evaluation, an effort will be made to utilize the case weight system in effect at the start-up of the evaluation. Assuming that the case weight assigned to each appeal is independent of brief length, the measure of the program’s impact on brief length is whether brief lengths within a particular case weight are shorter in the treatment group than in the control group. This approach would enable a determination to be made as to whether brief lengths are increasing because cases are becoming more complex.
The aggregate length of all briefs in an appeal will be taken as the measure of brief length, excluding amicus briefs. Some question remains as to whether differences in printing styles might impact the accuracy of any effort aimed at determining brief length. Preliminary review of case ﬁles indicate a high level of standardization in typing style and formatting of the briefs.
Since brief length notations are routinely entered as docket entries for each case, the process of capturing these data will be relatively straightforward. Preliminary reliability checks of the accuracy of the docket entries on brief length indicate that they are, indeed, highly accurate. Nonetheless, additional reliability checks on this item will be made in a sample of cases throughout the study.
Objective Five: Improving the Quality of Oral Argument in Cases Submitted After Brieﬁng
The pre-argument conference program is expected to improve the quality of oral argument in cases submitted after brieﬁng. The anticipated improvement in the quality of oral argument is expected to result from substantive contact with the conference attorney, as well as from efforts by the conference attorney to narrow the issues involved in the appeal.
Panel judges will be asked to assess the quality of oral argument in a sample of cases in the treatment and control groups. A rating instrument focusing on several different aspects of the presentation of the oral argument will be developed and pretested for use by the judges. Given the very subjective nature of this approach, it is not unreasonable to expect that there will be considerable variability in the ratings. Ideally, the ratings will be done by the judges without knowledge of whether the case was conferenced. In order to do this, the docket sheets sent by the Clerk’s Ofﬁce to the judges should be modiﬁed so as to contain no conference attorneys’ initials or any other notations which would enable the judges to determine the source of the brieﬁng schedule.
It will be an indication that the program is achieving this objective if the treatment group is found to have signiﬁcantly higher ratings from the judges than the control group.
Objective Six: Instructing Members of the Bar Regarding Appellate Practices and Procedures in the Circuit
Another objective of the pre-argument conference program is that of fostering good relations with the bar. The program is expected to facilitate the dissemination of information about practices and procedures in the Sixth Circuit. In a sense, the conference attorneys not only assist the bar, but serve as liaison between the bar and the court.
To the extent that the pre-argument conference program operates to assist attorneys in meeting various ﬁling deadlines, an effort will be made to determine whether there are any signiﬁcant differences, in terms of the timeliness of the ﬁling of the briefs, between the treatments and controls reaching submission.
Analysis will be made of the total number of cases in which the appellee’s brief was not ﬁled in a timely manner as required by Rule 25 of the Federal Rules of Appellate Procedure. It will be an indication that the program is meeting the objective if the treatment group has a signiﬁcantly lower ﬁgure than the control group.
Examination will be made of the total number of cases in which the joint appendix was not ﬁled in timely manner as set out in the local rule for ﬁling appendixes. It will be an indication that the program is meeting this objective if the treatment group has a signiﬁcantly lower number of late ﬁlings than the control group.
Similarly, an effort will be made to identify the total number of times joint appendixes are returned to counsel for procedural defects. It will be an indication that the program is meeting this objective if the ﬁgure is signiﬁcantly lower for the treatment group than for the control group.
In order to further assess whether the program meets this objective a number of items concerning the instructional role of the program will be included in the survey questionnaires to attorneys in the cases assigned to treatment. In addition, several questions on this issue may be included in the judges’ questionnaires for cases reaching submission and/or oral argument. Responses from both counsel just after brieﬁng, or, in situations where the case terminates before that time, at the time of termination, as to whether any difﬁculties were experienced in following procedures involved in the handling of their cases through the appeals process.
It is estimated that during the period of study there be approximately 1,500 conferenceable appeals ﬁled in the Sixth Circuit. Based on pre-implementation projections, it is estimated that approximately 75% (of appeals) of the conference-eligible cases will in fact be conferenced. Two-thirds of the cases will be randomly assigned to the treatment group, with a third assigned to the control. This will give the conference attorneys a pool of approximately 1,000 cases to work with during the one year case selection period.
The random assignment will be accomplished by accessing the special purpose program on the FJC’s DEC computer. The program is relatively simple to run and requires the entry of a case name, docket number, and the date of the entry. The program then randomly assigns a “0” (designating a control group assignment) or a “1” (for a treatment group assignment). The random assignment program will automatically create a case assignment data set from which routine summary updates may be generated. The pre-argument statements for all cases assigned to treatment will then be returned to the conference attorneys.
In instances where the pre-argument statement indicates that the case is a cross appeal, is to be consolidated, or is otherwise related to another case, the trailing case will be assigned to the same group as the lead case. That is, all related cases are to be handled as a unit. This is the only exception to the random assignment procedures.
All cross appeals, consolidated, or related cases will be so identiﬁed in the case assignment data set.
In instances where it is learned, sometime after random assignment, that a particular case is a cross, consolidated, or a related appeal, the original group assignment will be maintained for the case. This will be done even though it results in the case receiving a treatment which is inconsistent with its original assignment.
For purposes of this study, cross and consolidated appeals are deﬁned as those cases so designated on the court of appeals’ docket sheets. Related cases are deﬁned as cases which are not cross or consolidated appeals but which have the same case number in the lower court.
Once assigned to a group, cases will not be removed from the study. For this reason care must be exercised by the pre-argument conference program staff, the Clerk’s Ofﬁce, and the judges to avoid having cases in the control group conferenced.
It is anticipated that, either by direction from the court or by request of a party, a case in the control group may have to be conferenced. When this occurs, or if a case is in some other way treated inconsistently with its assigned group, this will not be grounds for excluding the case from the study. However, the fact that a case is treated inconsistently will be noted. During data analysis, cases so noted will be subjected to further analysis to determine if the overall ﬁndings are affected.
Cases in the treatment group will be identiﬁed as being either conferenced or not conferenced. The term conferenced will mean that some staff contact (either verbal or written, including telephone contact) was made with any party in the case.
Duration of the Study:
As noted earlier, cases will be randomly assigned for a period of year. The data collection phase of the study will be completed when all cases in both groups have been either terminated or submitted.
A trial run of the case assignment procedures will be conducted to enable the Clerk’s Ofﬁce and the Conference Attorney, and the FJC to resolve any case processing issues that may arise.
The Clerk’s Ofﬁce, with coordination from the Conference Attorney, will be responsible for collecting the case ﬁle data. Most, if not all, of the case ﬁle data needed for this evaluation are likely to be routinely maintained on New AIMS. It will be necessary to collect the following case ﬁle data:
1. case number
2. case type
3. date case is docketed
4. date pre-argument statement ﬁled
5. date of disposition
6. type of disposition (includes: submitted/argued dismissals for lack of prosecution; voluntary dismissals; settlement; other) dismissals are for treatment cases, voluntary, those appear withdrawn without any conference attorney contact. Appeals withdrawn after a conference will be handled as settlements.
7. date on which each stage of the appellate process is completed for the case
8. stage, if case at termination
9. motions ﬁled by method of disposition
10. number of motions ﬁled relating to the ﬁling of the joint appendix
11. number of times briefs were returned for procedural defects
12. number of times joint appendices were returned for procedural defect
13. timeliness of ﬁling of appellant’s brief
14. timeliness of ﬁling of appellee’s brief
15. timeliness of ﬁling of joint appendix
16. number and type of briefs ﬁled (types include appellant, appellee, reply, amicus)
17. number of “dun” notices sent to counsel by Clerk’s Ofﬁce or Conference Attorney
18. total pages in appellant’s brief
19. total pages in appellee’s brief
20. identiﬁcation of consolidated, cross, and/or related appeals
21. the group assignment of the consolidated, cross, and/or related appeals
22. date at which it was determined that the case was to be consolidated, cross, and/or related (i.e., before or
after group assignment)
23. case weight
The Conference Attorneys will maintain a listing of the names and addresses of the attorneys who were actually involved in each pre-argument conference. At least twice a month, a copy of the listing will be given to the individual in the Clerk’s Ofﬁce who has been assigned responsibility for sending counsel the questionnaire accompanied by a cover letter from the Chief Judge of the Sixth Circuit. Counsel will be asked to send their completed questionnaires to the Research Division of the FJC. A list of each attorney to whom a questionnaire is mailed will be kept by the Clerk’s Ofﬁce. The list will indicate the date on which the mailing occurred.
Only one questionnaire will be sent to an attorney during the course of the study, irrespective of the number of cases he or she may have had conferenced during the period of study.
The Clerk’s Ofﬁce will provide the panel judges with the questionnaires and will be responsible for collecting them. Judges will not know whether a case is in the treatment or control group.
In consultation with the Clerk’s Ofﬁce, the Conference Attorneys will develop the procedures and questionnaires to be submitted to the judges.