Behavioural effects of methylphenidate in the spontaneously hypertensive rat model of attention-deficit/hyperactivity disorder: a systematic review and meta-analysis protocol

Introduction Attention-deficit/hyperactivity disorder (ADHD) is a prevalent condition related to several negative outcomes, and its pathophysiology is still poorly understood. The spontaneously hypertensive rats (SHRs) are the most commonly used animal model of ADHD. How ever, its validity, and especially its predictive validity, has been questioned. Therefore, the current protocol discloses the background, aims and methods of a systematic review and meta-analysis of studies reporting the behavioural effects of methylphenidate (MPH), the most commonly prescribed treatment for ADHD, in the SHR. Search strategy Studies will be identified through a literature search using three different electronic databases: Medline, Embase and Web of Science. There will be no language restrictions. All s tudies that administered MPH to SHR and evaluated locomotion, attention, impulsivity or memory will be included. Screening and annotation Studies will be prescreened based on title and abstract, and a full-text review will be performed if necessary. Screening will be performed by two authors, and any disagreement will be discussed with a third author. Data management and reporting Data extraction will be performed by two independent authors according to a standardised form. Studies will be grouped according to the behavioural outcomes reported, and a meta-analysis will be performed for each group. The influence of predefined covariates on the effects of MPH will be evaluated using meta-regression and sensitivity analyses. Data will be reported following PRISMA guidelines.

There are some minor concerns and comments: 1. there is no final search mentioned with all search terms. Please add (in supplementary file) the complete final search that has been inserted in PubMed / Embase etc. Through this, you are absolutely sure that the search used in the primary paper is the same as the one in the protocol. 2. (2.2 p7) in '2.2 study selection' the authors state that they will do a full text screening if necessary. I would advice to always do a full text screening to be absolutely sure that your included paper meets al your inclusion criteria (sometimes abstracts can be misleading). 3. (2.2) Please state who the authors will be (with initials) who will conduct the search and who will act as third screener/referee. 4. (2.2) You exclude SHR substrains (stroke-prone SHR). Is there a reason to exclude these? Are they different from one another? In literature I can find authors saying that both are used as ADHD models. If there is a specific reason to exclude these, please elaborate on this and provide literature on this choice. 5. (2.6.2) Please state how many studies you think you will approximately include, as this officially limits the amount of covariates you can examine. A general rule of thumb is 1 covariate per 10 studies included. If you choose to include more covariates, please state this in the primary paper, as this increases the chance of finding false-positive results. 6. (2.6.2, p10) you will perform a multivariate meta-regression model, in which studies with missing values will be excluded. please be aware that doing a 'complete case analysis' might bias your data and reduce the number of included studies. 7. (2.6.3) you mention sensitivity analysis, based on excluding one result at a time. If you want to do this, to get rid of any outliers I would suggest starting with a 'regular' outlier analysis on your outcome data-points (deviance residuals, Schoenfeldt residuals) and try to exclude those. (NB. This is not a request that certainly needs to be implemented, but just a suggestion.)

VERSION 1 -AUTHOR RESPONSE
Editor in Chief Comments to Author: The reviewer has been some excellent suggestions. In addition, although you have clearly stated the aim of the systematic review in the introduction it would be beneficial to include your specific objectives to achieve this aim. R: Thank you for your comment. Specific objectives were included in the introduction.
I suggest for section 2.5 you make it clear how you will ensure consistency in your relevance ranking of behavioural tests extracted (I assume this means that for each outcome only one behavioural test data point will be included?).
R: Yes, only one data point will be included for each behavioral test. We will rank the variables from each behavioral test subjectively according to their importance, and the relevance rank will be organized by one reviewer. Even though the ranking is organized in a subjectively way, the extraction of all papers will be based on the same rank, thus providing consistency to the method. A similar strategy has been previously used (1, 2). I also suggest a short description of the pros and cons of this approach should be included in the discussion -i.e. why you have chosen this method rather than to include all data from a behavioural test to nest into a single data point for your outcome.
R: Thank you for your comment. The pros and cons of this approach have been incorporated in the discussion.
Associate Editor Comments to Author: I agree with the suggestions for minor revisions as made by reviewer 1, and suggest that, if still possible, you try to incorporate these suggestions into your protocol.
Reviewer 1: In 'Behavioural effects of methylphenidate in an animal model of ADHD, the spontaneously hypertensive rats: a systematic review and meta-analysis protocol' Texeira Leffa et al. describe their protocol in detail for their systematic review and meta-analysis. This is important to guarantee that the methods used were declared upfront and should be applauded. The protocol is excellently written and includes most important aspects. There are some minor concerns and comments: 1. there is no final search mentioned with all search terms. Please add (in supplementary file) the complete final search that has been inserted in PubMed / Embase etc. Through this, you are absolutely sure that the search used in the primary paper is the same as the one in the protocol. R: Thank you for your comment. The complete final search has now been added as supplementary material.
2. (2.2 p7) in '2.2 study selection' the authors state that they will do a full text screening if necessary. I would advice to always do a full text screening to be absolutely sure that your included paper meets al your inclusion criteria (sometimes abstracts can be misleading).
R: Thank you for your suggestion. We do agree that abstracts can be misleading. In order to avoid the exclusion of a potentially eligible study and at the same time increase the efficiency of our search strategy, we decided to exclude studies without a full-text review only in those situation that an exclusion criteria is clearly defined (e.g. reviews or editorials, studies in human subjects). The information is now mentioned in "2.2 Study Selection". We would like to highlight that this strategy is advised for both clinical and preclinical meta-analyses (3, 4).

(2.2)
Please state who the authors will be (with initials) who will conduct the search and who will act as third screener/referee. R: The authors responsible for the search are now described.

(2.2) You exclude SHR substrains (stroke-prone SHR)
. Is there a reason to exclude these? Are they different from one another? In literature I can find authors saying that both are used as ADHD models. If there is a specific reason to exclude these, please elaborate on this and provide literature on this choice. R: The stroke-prone SHR are a substrain of SHR created by breeding animals presenting rapid increase in blood pressure at a younger age and severe hypertension, leading to a high incidence of stroke (5). Although few studies have been conducted using the stroke-prone SHR as a model of ADHD, biochemical (6) and genetic (7,8) differences between the two strains have been reported. Those differences, together with the increase incidence of stroke in stroke-prone SHR, potentially leading to brain damage, prevented us from including the substrain in the analysis. 5. (2.6.2) Please state how many studies you think you will approximately include, as this officially limits the amount of covariates you can examine. A general rule of thumb is 1 covariate per 10 studies included. If you choose to include more covariates, please state this in the primary paper, as this increases the chance of finding false-positive results. R: Thank you for your comment. Based on our previous experience with basic studies, we believe that at least 30 studies will be included in the final analysis. However, if this is not the case, a statement will be included in the primary paper.
6. (2.6.2, p10) you will perform a multivariate meta-regression model, in which studies with missing values will be excluded. please be aware that doing a 'complete case analysis' might bias your data and reduce the number of included studies. R: Thank you for your comment. We do agree that this method may bias the final result, and have added a sentence in the discussion section.
7. (2.6.3) you mention sensitivity analysis, based on excluding one result at a time. If you want to do this, to get rid of any outliers I would suggest starting with a 'regular' outlier analysis on your outcome data-points (deviance residuals, Schoenfeldt residuals) and try to exclude those. (NB. This is not a request that certainly needs to be implemented, but just a suggestion.) R: Thank you for your suggestion. Excluding one result at a time in order to observe how individual studies may affect the final effect size is also called the jackknife method (9), and it is commonly used in meta-analysis. We have decided to maintain the method, and included more information about it in the text.

Introduction
In general the intro could be improved by information why face and construct validity are apparently not a problem in SHR rats. This could be nicely tied in with a bit describing the proposed mechanism of action of MPH.

2.2.
It would be better to use the 'conventional' steps in the PRISMA diagram instead of "Step 1-3". I.e. abstract screening , full-text etc.
Step 3 should be reversed by specifying inclusion criteria criteria -> criterion 2.3.
gender -> sex 2.5 The authors have decided to that they will include only one behavioural experiment from the same category and will choose the most relevant or the first. This seems very prone to bias and unclear. The first means the first reported in the article? This may be a negative bias as authors usually build up towards the more dramatic results in the article. On the other hand relevance decided by the authors of this article may also lead to bias. My suggestion would be to extract all of them and include the one with the largest effect size in the main analysis (effect of MPH). The results of different test variants (e.g. diff locomotion tests) can be presented in separate subgroup analyses (or if need be under sensitivity). Which test 'works' best (and thus has highest predictive validity) is an important question and can refine animal research.
The last two sentences seem vague to me. "If the same animals were evaluated more than once in the same test, the last one will be selected for data extraction. If the manuscript divides the results by time, the first timepoint will be selected..." To my mind these say the exact opposite, please clarify. line 5: two attempt -> two attempts 2.6 Page 9, last line. Actually the four hypotheses are likely far from independent. Assuming a true underlying construct for ADHD and an effect of MPH thereupon, one expect that relative scores on the four outcome measures are highly correlated and therefore not independent. The Bonferroni correction is conservative and I don't oppose to it, but it may be overkill.
Page 10. Any reason why the authors transform the values for locomotion? Common practice would be to show the negative values and put and additional 'favors MPH"/"favors control" on the axis.

2.6.2
The meta-regression test are independent hypotheses, and should be corrected for.