In our last blog Discovery Early Career Researcher Awards: Benchmarking this year’s results we looked at the track record characteristics of this year’s DECRA recipients. While that is useful for getting a sense of what the average recipient looks like, it is less useful for those who are interested in applying but whose key metric – say their h-index – is on the extreme lower range for a successful candidate. What is more relevant for people in that position is whether an h-index is predictive of success. Should they even bother to apply if it is low?
To work out what characteristics of candidates seem to be related to their success, we took a journey back. We went through our database of past DECRA reviews and ran a backwards logistic regression (1) to see if we could determine which features of a candidate’s track record predicted their success.
We collected data on the track record of 238 DECRA applications (DE16-DE20).
The variables used were (2):
Full time equivalent since graduation
Field of research – divided up by the ARC panel the project was most likely to go to
Whether the applicant had interruptions (yes/no)
Whether the applicant had supervised PhD students (yes/no)
Whether the applicant had supervised masters or honours students (yes/no)
The number of invited talks
The number of research grants
The total funding received
The number of patents
Whether they had commercial partners (yes/no)
Whether they had undertaken commercialisation activities (yes/no)
How many different journals they had reviewed for
How many journals they had edited
Engagement with mainstream media, which was coded on a 4 point scale with 0 = no mention, 1 = mentioned but don’t give the number of media engagements, 2 = mention less than 10 (3) engagements and 3 = mention 11 or more engagements
Whether they mentioned government or NGO engagement (yes/no)
Whether they mentioned public outreach (yes/no)
Whether they mentioned being a member of a professional society (yes/no)
Their total number of publications
Their h-index
Their total number of citations
The percentage of journal articles which were in e.g. Q1 journals
The percentage of academic outputs they were first/lead author on
We screened all of the variables individually to determine which ones to include in the model (using either a t-test of independence, or chi squared test of independence, depending on the nature of the data). Those with a p<0.1 were included: (4)
PhD supervision
Honours/masters supervision
Patents
Total publications
Total citations
H-index
What is interesting about these variables is that they are all fairly traditional academia metrics. This is obvious for number of publications, citations and h-index. We suspect that higher degree research student supervision is relevant because it is a proxy for a candidate’s acceptance into their department – senior academics are unlikely to suggest that the weaker early career researchers in their department supervise students. We suspect patents were significant due to the high number of applications we reviewed which fell under the Engineering, Information and Computing Science Panel as the next most frequent panel (Social, Behavioural and Economic Sciences (5)).
While this is interesting it is not deeply informative; you would expect there to be significant correlation between the variables. By their nature, as the number of citations increases the value of your h-index will increase. Similarly, if you have supervised PhD students you have probably also supervised honours or masters students. So, what does the multiple regression analysis show?
In the first model which included all of the variables identified, none were significant. What this means is that none of them did independent work predicting the odds of success:
We then dropped h-index and it turned out that (log) citations did significant predictive work, but none of the other variables did. As we dropped each of the other variables out, only citations did predictive work. What this means is that while all the variables can predict success, the number of citations is the one which does independent predictive work. In predicting success, once you know the total number of citations, the others are redundant, there is no value in collecting them. Out of interest, given how correlated h-index was with (log) citations, we decided to drop citations out first and include h-index instead. It behaved the same as citations. This means it was significant in the model without citations, and as the other variables were omitted it was the only one which remained significant.
Our data suggests the best predictors of success are either the number of citations or h-index. There is no point using both – you can take your pick. Regarding how much predictive work they do, the odds ratio for log citations by itself in the model was 1.60 (95% CI 1.15-2.30, p=0.008). The odds ratio is a measure of how much the odds of success increase for each unit increase in the independent variable. Because we used the log of total number of citations this shows that each 2.72 additional citations increases the odds of being funded by 60%. For h-index by itself the odds ratio was 1.12 (95%CI 1.03-1.21, p=0.009). In other words, each increase in h-index increases the odds of being funded by 12%.
What does this mean for those thinking of applying for a DECRA? According to our data the best predictors of success are your total citations or h-index. Recall our previous blog though, describing the characteristics of this year’s successful candidates. While having a “large” h-index increases your chances of success, there was a successful candidate who had an h-index of 1. Similarly, while having a lot of citations increases your chances of success, there was a successful candidate with 3 citations. Obviously there is a lot more to predicting DECRA success than what we could find in our data (6). Does this mean that you shouldn’t apply if your h-index and number of citations are low? Ultimately that is a decision for you to make based on a range of factors including the amount of time it can take to prepare an application (see for example von Hippel & von Hippel, 2015) (7).
What we do know is a GrantEd review of your application increases the odds of your DECRA being funded by 53%. So book your review today! Email us at hello@thegrantedgroup.com.au and ask for our services brochure.