Appendix B

Evaluation Design

Judge Questionnaire

Attorney Questionnaire

 

Evaluation Design for the Sixth Circuit
Pre-Argument Conference Program

 

           The evaluation of the Sixth Circuit’s pre-argument conference program will utilize a controlled experiment design. Cases will be randomly assigned to either of two groups (a treatment group, i.e., the group to be conferenced, or a control group—the group that will not be conferenced).

           The appeals assigned to treatment will be randomized from the universe of conference-eligible cases filed during the case selection phase of the study. The case selection phase of the evaluation will last for approximately one year.

           Cases will be drawn from all of the divisional units in the Sixth Circuit.

           The only civil appeals to be excluded from the study are those permitted by permission under 28 U.S.C. 1292(b) and appeals in habeas cases under sections 2254 and 2255 of Title 28. No other civil appeals will be excluded from the evaluation.

           A small number of cases will be automatically assigned to the treatment group when a specific request to do so is received from either the Clerk’s Office or the Conference Attorney. It is expected that the number of requests for automatic assignment to the treatment group will be relatively small (perhaps, less than a dozen) and will be accompanied by ample justification. For example, automatic assignments might be utilized for cases in which counsel have already participated in a number of pre-argument conferences and have expressed a strong preference for continuing to have access to the pre-argument conference program.

           In addition to collecting and analyzing case file data on appeals included in the study, questionnaires will be administered to panel judges in a sample of study cases. Attorneys involved in cases assigned to the treatment group will be surveyed as well.

Evaluation Objectives:

           The primary objective of the pre-argument conference program evaluation is to assist the Sixth Circuit to determine whether the program has an impact on: (1) the number of appeals settled; (2) the number of motions filed; (3) the quality and length of briefs; (4) the quality of oral argument in cases submitted to the court following pre-argument conferences; and (5) instructing members of the bar concerning appellate practice in the circuit. In addition to assessing the above potential case management-related effects of the program, an effort will be made to evaluate the program’s impact on the number of appeals settled, voluntarily dismissed, or dismissed for want of prosecution. Increasing the settlement rate of civil appeals is viewed by the Sixth Circuit as the major objective of its pre-argument conference program.

Evaluation Criteria:

           The study will use the following criteria to assist the Sixth Circuit in determining whether the program is meeting its objectives:

Objective One: Increasing the Number of Cases Which are Settled, Voluntarily Dismissed, or Dismissed for Want of Prosecution

           Case file data on the number of cases in the treatment and control groups which were settled, voluntarily dismissed, or dismissed for want of prosecution will be collected and analyzed. It will be an indication that the program has an effect on settlements and dismissals if the treatment group has a significantly greater percentage of cases in the above three categories of termination when compared with the control group.

Objective Two: Terminates Cases Earlier in the Appellate Process

           An effort will be made to determine whether the pre-argument conferencing program facilitates the termination of cases at earlier stages in the appellate process. In order to examine this issue, the following approach will be taken:

           A.    Calculation of median days treatment and control cases are active. For the purposes of the study, a case is defined as active from the time of the filing of the notice of appeal to time at which it is either terminated or submitted to the court. It will be an indication that the program is meeting the objective if the treatment group has a significantly lower measure of median days active when compared with the control group.

           B.    Calculation of the percentage of cases terminating during each stage of the appellate process along with the median time cases remain in each stage. For purposes of this study, the following stages of the appellate process will be examined for each case in the study:

1)       Time from date case is docketed in the Sixth Circuit until the date of the filing of the certificate of record in the Sixth Circuit.

2)       Time from filing of record in the Sixth Circuit to filing of appellant’s brief.

3)       Period from filing of appellant’s brief to filing of appellee’s brief.

4)       Period from filing of appellee’s brief to time of argument.

5)       Time from filing of joint appendix to time of argument

It will be an indication that the program has the anticipated effect if the treatment group has a significantly greater percentage of cases that terminate in the earlier stages when compared with the control group.

Objective Three: Reducing the Number of Motions Filed

           A major objective of the pre-argument conference program is that of permitting informal resolution of procedural matters, such as joinder of briefs in appeals having multiple parties, thereby reducing the number of formal motions on which the court must act.

           It will be an indication that the program is meeting this objective if the treatment group is found to have a significantly lower number of formal motions filed than the control group. It should be noted, however, that the process of counting the motions in each appeal included in the study may not be as straightforward a task as it might at first appear. For cases assigned to the treatment group, for example, the briefing schedule permitted by the conference attorney may operate as the equivalent of the first extension granted on motion in the control group cases. Therefore, any observed differences between the treatment and control groups in the number of actual motions filed will have to be interpreted with that possibility in mind.

           Case file data will be collected and analyzed on both the frequency and type of motions filed in all of the appeals included in the study.

Objective Four: Impact on Quality and Length of Briefs Filed              

           As part of the survey of attorneys in treatment cases proceeding to briefing, each attorney will be asked whether the pre-argument conference program assisted in the preparation of the briefs.

           It will be an indication that the program is achieving this objective if a significant number of attorneys indicate that the program assisted in the preparation of briefs.

           In addition, responses from panel judges at, or near, the time of argument or submission will be gathered on a sample of cases to assess brief quality. Responses will be collected by questionnaires, with each judge being asked to rate the quality of selected briefs in appeals submitted to a panel on which he or she sat.

           It will be an indication that the program is meeting this objective if the treatment group is found to have a significantly larger number of positive ratings than the control group.

           As part of the pre-argument conference program procedures, the conference attorneys set the briefing schedule along with limits on the maximum brief length that each party may submit. As part of the analysis of brief lengths, an effort will be made to compare actual brief length with the limits set by the conference attorney.

           The process of determining the overall impact of the program on brief length is somewhat complicated by the probability that the pre-argument conference program is changing the universe of briefed cases in the circuit. That is, it is very likely that the program results in a number of the less complicated appeals settling or being withdrawn prior to briefing. The program is also likely to change the timing at which less complex cases settle or withdraw such that there are more appeals settling or withdrawing at the pre-argument rather than post-briefing stage. Assuming that this in fact occurs, we are likely to observe an increase in the average brief length in those cases that are briefed. Such a finding would mask the impact of the program on the less complex cases that settle or withdraw at the pre-argument stage. As a solution to this, the evaluation will utilize a case weighting system to separate out possible settlement effects of the program from any brief length-reduction effects.

            For purposes of this evaluation, an effort will be made to utilize the case weight system in effect at the start-up of the evaluation. Assuming that the case weight assigned to each appeal is independent of brief length, the measure of the program’s impact on brief length is whether brief lengths within a particular case weight are shorter in the treatment group than in the control group. This approach would enable a determination to be made as to whether brief lengths are increasing because cases are becoming more complex.

           The aggregate length of all briefs in an appeal will be taken as the measure of brief length, excluding amicus briefs. Some question remains as to whether differences in printing styles might impact the accuracy of any effort aimed at determining brief length. Preliminary review of case files indicate a high level of standardization in typing style and formatting of the briefs.

           Since brief length notations are routinely entered as docket entries for each case, the process of capturing these data will be relatively straightforward. Preliminary reliability checks of the accuracy of the docket entries on brief length indicate that they are, indeed, highly accurate. Nonetheless, additional reliability checks on this item will be made in a sample of cases throughout the study.

Objective Five: Improving the Quality of Oral Argument in Cases Submitted After Briefing

           The pre-argument conference program is expected to improve the quality of oral argument in cases submitted after briefing. The anticipated improvement in the quality of oral argument is expected to result from substantive contact with the conference attorney, as well as from efforts by the conference attorney to narrow the issues involved in the appeal.

           Panel judges will be asked to assess the quality of oral argument in a sample of cases in the treatment and control groups. A rating instrument focusing on several different aspects of the presentation of the oral argument will be developed and pretested for use by the judges. Given the very subjective nature of this approach, it is not unreasonable to expect that there will be considerable variability in the ratings. Ideally, the ratings will be done by the judges without knowledge of whether the case was conferenced. In order to do this, the docket sheets sent by the Clerk’s Office to the judges should be modified so as to contain no conference attorneys’ initials or any other notations which would enable the judges to determine the source of the briefing schedule.

           It will be an indication that the program is achieving this objective if the treatment group is found to have significantly higher ratings from the judges than the control group.

Objective Six: Instructing Members of the Bar Regarding Appellate Practices and Procedures in the Circuit

           Another objective of the pre-argument conference program is that of fostering good relations with the bar. The program is expected to facilitate the dissemination of information about practices and procedures in the Sixth Circuit. In a sense, the conference attorneys not only assist the bar, but serve as liaison between the bar and the court.

           To the extent that the pre-argument conference program operates to assist attorneys in meeting various  filing deadlines, an effort will be made to determine whether there are any significant differences, in terms of the timeliness of the filing of the briefs, between the treatments and controls reaching submission.

           Analysis will be made of the total number of cases in which the appellee’s brief was not filed in a timely manner as required by Rule 25 of the Federal Rules of Appellate Procedure. It will be an indication that the program is meeting the objective if the treatment group has a significantly lower figure than the control group.

           Examination will be made of the total number of cases in which the joint appendix was not filed in timely manner as set out in the local rule for filing appendixes. It will be an indication that the program is meeting this objective if the treatment group has a significantly lower number of late filings than the control group.

           Similarly, an effort will be made to identify the total number of times joint appendixes are returned to counsel for procedural defects. It will be an indication that the program is meeting this objective if the figure is significantly lower for the treatment group than for the control group.

           In order to further assess whether the program meets this objective a number of items concerning the instructional role of the program will be included in the survey questionnaires to attorneys in the cases assigned to treatment. In addition, several questions on this issue may be included in the judges’ questionnaires for cases reaching submission and/or oral argument. Responses from both counsel just after briefing, or, in situations where the case terminates before that time, at the time of termination, as to whether any difficulties were experienced in following procedures involved in the handling of their cases through the appeals process.

Group Assignment:

           It is estimated that during the period of study there be approximately 1,500 conferenceable appeals filed in the Sixth Circuit. Based on pre-implementation projections, it is estimated that approximately 75% (of appeals) of the conference-eligible cases will in fact be conferenced. Two-thirds of the cases will be randomly assigned to the treatment group, with a third assigned to the control. This will give the conference attorneys a pool of approximately 1,000 cases to work with during the one year case selection period.

           The random assignment will be accomplished by accessing the special purpose program on the FJC’s DEC computer. The program is relatively simple to run and requires the entry of a case name, docket number, and the date of the entry. The program then randomly assigns a “0” (designating a control group assignment) or a “1” (for a treatment group assignment). The random assignment program will automatically create a case assignment data set from which routine summary updates may be generated. The pre-argument statements for all cases assigned to treatment will then be returned to the conference attorneys.

           In instances where the pre-argument statement indicates that the case is a cross appeal, is to be consolidated, or is otherwise related to another case, the trailing case will be assigned to the same group as the lead case. That is, all related cases are to be handled as a unit. This is the only exception to the random assignment procedures.

           All cross appeals, consolidated, or related cases will be so identified in the case assignment data set.

           In instances where it is learned, sometime after random assignment, that a particular case is a cross, consolidated, or a related appeal, the original group assignment will be maintained for the case. This will be done even though it results in the case receiving a treatment which is inconsistent with its original assignment.

           For purposes of this study, cross and consolidated appeals are defined as those cases so designated on the court of appeals’ docket sheets. Related cases are defined as cases which are not cross or consolidated appeals but which have the same case number in the lower court.

           Once assigned to a group, cases will not be removed from the study. For this reason care must be exercised by the pre-argument conference program staff, the Clerk’s Office, and the judges to avoid having cases in the control group conferenced.

           It is anticipated that, either by direction from the court or by request of a party, a case in the control group may have to be conferenced. When this occurs, or if a case is in some other way treated inconsistently with its assigned group, this will not be grounds for excluding the case from the study. However, the fact that a case is treated inconsistently will be noted. During data analysis, cases so noted will be subjected to further analysis to determine if the overall findings are affected.

           Cases in the treatment group will be identified as being either conferenced or not conferenced. The term conferenced will mean that some staff contact (either verbal or written, including telephone contact) was made with any party in the case.

Duration of the Study:

           As noted earlier, cases will be randomly assigned for a period of year. The data collection phase of the study will be completed when all cases in both groups have been either terminated or submitted.

           A trial run of the case assignment procedures will be conducted to enable the Clerk’s Office and the Conference Attorney, and the FJC to resolve any case processing issues that may arise.

Data Collection:

           The Clerk’s Office, with coordination from the Conference Attorney, will be responsible for collecting the case file data. Most, if not all, of the case file data needed for this evaluation are likely to be routinely maintained on New AIMS. It will be necessary to collect the following case file data:

1.     case number

2.     case type

3.     date case is docketed

4.     date pre-argument statement filed

5.     date of disposition

6.     type of disposition (includes: submitted/argued dismissals for lack of prosecution; voluntary dismissals; settlement; other) dismissals are for treatment cases, voluntary, those appear withdrawn without any conference attorney contact. Appeals withdrawn after a conference will be handled as settlements.

7.     date on which each stage of the appellate process is completed for the case

8.     stage, if case at termination

9.     motions filed by method of disposition

10.   number of motions filed relating to the filing of the joint appendix

11.   number of times briefs were returned for procedural defects

12.   number of times joint appendices were returned for procedural defect

13.   timeliness of filing of appellant’s brief

14.   timeliness of filing of appellee’s brief

15.   timeliness of filing of joint appendix

16.   number and type of briefs filed (types include appellant, appellee, reply, amicus)

17.   number of “dun” notices sent to counsel by Clerk’s Office or Conference Attorney

18.   total pages in appellant’s brief

19.   total pages in appellee’s brief

20.   identification of consolidated, cross, and/or related appeals

21.   the group assignment of the consolidated, cross, and/or related appeals

22.   date at which it was determined that the case was to be consolidated, cross, and/or related (i.e., before or

        after group assignment)

23.   case weight

Survey Questionnaires:

           The Conference Attorneys will maintain a listing of the names and addresses of the attorneys who were actually involved in each pre-argument conference. At least twice a month, a copy of the listing will be given to the individual in the Clerk’s Office who has been assigned responsibility for sending counsel the questionnaire accompanied by a cover letter from the Chief Judge of the Sixth Circuit. Counsel will be asked to send their completed questionnaires to the Research Division of the FJC. A list of each attorney to whom a questionnaire is mailed will be kept by the Clerk’s Office. The list will indicate the date on which the mailing occurred.

           Only one questionnaire will be sent to an attorney during the course of the study, irrespective of the number of cases he or she may have had conferenced during the period of study.

           The Clerk’s Office will provide the panel judges with the questionnaires and will be responsible for collecting them. Judges will not know whether a case is in the treatment or control group.

     In consultation with the Clerk’s Office, the Conference Attorneys will develop the procedures and questionnaires to be submitted to the judges.