Mark J Butler, Thevaa Chandereng, Heejoon Ahn, Stefani Slotnick, Danielle Miller, Alexandra Perrin, Jordyn Rodillas, Ciaran P Friel, Ashley M Goodwin, Ying Kuen Cheung, Karina W Davidson
8 min
Personalized N-of-1 trials are designed to identify individual-level treatment effects for interventions with high variability. Researchers aimed to determine if such trials could effectively evaluate the impact of different melatonin doses (3.0 mg and 0.5 mg) versus placebo on self-reported poor sleep, while simultaneously assessing the feasibility and acceptability of a remote, digital trial platform.
This study conducted 60 personalized N-of-1 crossover trials. Each participant underwent a 2-week baseline period followed by a 12-week intervention period. During the intervention, participants were randomized into two treatment orders (ABCCBA or CBAABC) to test two melatonin doses against a placebo. Adherence and outcomes were monitored using Nomi smart pill bottles, Fitbit Charge 5 activity trackers, and daily ecological momentary assessment (EMA) text-message surveys tracking sleep quality, fatigue, and stress. Feasibility and satisfaction were evaluated through the System Usability Scale (SUS) and post-trial surveys.
The trial demonstrated high feasibility, with participants reporting an average SUS score of 76.3 and 96% stating they would recommend the experience. Despite the successful implementation, melatonin did not provide statistically significant improvements in sleep duration or quality compared to placebo. Furthermore, formal analysis of the heterogeneity of treatment effects (HTE) revealed minimal variation in how participants responded to the melatonin, indicating that there is no individual-level benefit that would necessitate the continued use of personalized N-of-1 trials for melatonin in this population. These results suggest that while the digital trial framework itself is a valid research tool, melatonin may not be an appropriate intervention for this personalized approach.
Background: Poor sleep (defined by short sleep duration or poor quality) is a common condition with potential serious health consequences. Exogenous melatonin supplements have been found to effectively improve poor sleep but have also been shown to have heterogeneity of treatment effects (HTEs) between individuals. Personalized N-of-1 trials, in which each participant is the unit of analysis, are ideal for identifying whether a treatment with high HTE is beneficial for each individual patient. Objective: This study aimed to identify the feasibility, acceptability, and effectiveness of a series of personalized N-of-1 trials of melatonin for poor sleep. Methods: This study consisted of 60 digital, personalized N-of-1 crossover trials comparing the effects of 3.0 mg and 0.5 mg of melatonin versus placebo for poor sleep with randomization to 1 of 2 orders. The trial comprised a 2-week baseline period and a 12-week intervention period. The primary outcomes were usability of the personalized trial system (measured using the System Usability Scale [SUS]) and participant satisfaction with the trial. Effectiveness outcomes included sleep duration (measured using a Fitbit activity tracker [Google]) and sleep quality (measured using the consensus sleep diary). Results: Participants rated the usability of the personalized trial as acceptable (average SUS score 76.3, SD 17.1), and 96% (55/57) of those who completed satisfaction surveys stated that they would recommend the trial to others. Importantly, indices of HTE were low for 3.0 mg and 0.5 mg doses of melatonin, indicating that the effect of these treatments on sleep duration and sleep quality did not substantially vary between participants and that averaged treatment responses are appropriate. Averaged participant sleep duration did not significantly differ between the 3.0 mg (P=.70) and 0.5 mg (P=.90) melatonin intervention periods and the baseline period. In addition, regression models did not show differences between different levels of melatonin and placebo periods for sleep duration or quality. Conclusions: Participant ratings of the usability of and satisfaction with this series of personalized N-of-1 trials of melatonin for sleep suggest these trials are both feasible and acceptable. However, our results show that melatonin supplements did not significantly improve sleep duration or sleep quality. Furthermore, the treatment effects' lack of heterogeneity among participants suggests that future use of N-of-1 trials of melatonin for poor sleep is not needed.
Sam: That sounds workable remotely, but it also asks for sustained effort. What did participants have to record?
Alex: The Fitbit recorded sleep duration. A morning text asked about the previous night’s sleep quality, using a modified consensus sleep diary—a standardized sleep questionnaire. Additional texts asked about symptoms, including fatigue and stress. Participants also received a personalized report after completing the trial.
Sam: Before we get to whether they liked it, who counted as having poor sleep? That could shape the treatment result.
Alex: Eligibility required recurring difficulty falling asleep, staying asleep, or feeling refreshed, persisting for at least a month. Symptoms also had to bother participants substantially. But the enrolled sample’s average sleep duration was only moderately short, and average reported sleep quality was near the middle of the scale.
Sam: So these were people troubled by sleep, but not necessarily people with the most severe measured sleep problems.
Alex: That’s an important boundary. Another is that participants had to meet adherence requirements during baseline before progressing. The usability findings therefore describe a group already able to engage with the monitoring and surveys, rather than everyone who initially expressed interest.
Sam: Within that group, did the remote system actually feel manageable?
Alex: On the System Usability Scale, a validated usability questionnaire, the average score was about 76 out of a hundred. The authors classified that as acceptable. Ratings varied considerably, though: some participants found the system less than acceptable, while others rated it excellent.
Sam: An average usability score can conceal specific friction. Which parts worked, and which parts wore people down?
Alex: Participants especially liked the Fitbit and the presentation of their results. Text-message prompts and survey questions received lower satisfaction ratings. Still, among satisfaction-survey respondents, 96 percent said they would recommend the trial. That supports acceptability, without suggesting every component was equally welcome.
Sam: Now the treatment comparison. What doses were tested, and how much extra sleep did they produce?
Alex: The lower dose was half a milligram. The higher dose was three milligrams. Neither significantly increased sleep duration relative to placebo. For the higher dose, the pooled model estimated roughly five extra minutes per night compared with placebo.
Sam: Five minutes sounds small, but an estimate is not the same as a precise answer. How uncertain was it?
Alex: Its confidence interval included both less sleep and more sleep than placebo. So the result does not establish an exactly zero effect. My reading is narrower: this study found no clear sleep-duration advantage at either tested dose, using these measurements in this sample.
Sam: What about feeling that you slept better? Device-recorded duration might miss something people care about.
Alex: Reported sleep quality improved from baseline during all intervention conditions, including placebo. But neither melatonin dose beat placebo on that outcome. There was a small statistically significant difference favouring the lower dose over the higher dose. That did not translate into superiority over placebo.
Sam: So a before-and-after improvement would tell a more encouraging story than the actual treatment comparison.
Alex: It would, and that’s why the placebo comparison matters here. Fatigue also improved slightly from baseline in some conditions, including placebo. Neither melatonin dose significantly outperformed placebo for fatigue, and stress showed no significant treatment differences.
Sam: That still leaves the original personalization argument. Could some people benefit even when the pooled result is unconvincing?
Alex: The researchers explicitly tested that possibility. Their models accounted for autocorrelation, meaning that measurements close together in time can be related. They then compared models allowing treatment effects to vary between people with models that did not allow that variation.
Sam: Did allowing variation improve the models? That seems like the obvious evidence for individual differences.
Alex: It improved model fit for sleep duration, fatigue, and stress. But the authors’ separate indices of treatment-effect heterogeneity—how much effects differ between people—were low. They interpreted those indices as indicating little usefulness for future personal trials of these treatments and outcomes in this population.
Sam: So better model fit did not automatically mean differences large enough to make personalized testing worthwhile. Was sleep quality assessed the same way?
Alex: No. The formal heterogeneity analysis for sleep quality could not be completed because responses varied too little. The authors thought that also suggested limited differences in treatment effects. But we should distinguish that suggestion from a completed statistical test.
Sam: How far should we take the negative result? It would be easy to turn this into “melatonin never helps.”
Alex: That would go beyond the evidence. The authors discuss earlier research suggesting melatonin works best with sleep restriction or altered sleep schedules. They connect that to circadian rhythm, the body’s daily timing cycle. This trial broadly recruited people reporting poor sleep, whose difficulties could have varied causes.
Sam: And which limitation most affects your confidence in what they measured?
Alex: The sleep outcomes came from self-report and a consumer wearable. The authors acknowledge that more precise methods might detect nuances these tools missed. The sample also was not representative, and only the selected doses were tested. Sample size was chosen primarily for a preliminary feasibility assessment.
Sam: Who should read the full document, and where should they start?
Alex: Researchers building remote personalized trials should start with study design, adherence rules, and usability results. Sleep researchers should examine the placebo comparisons and the treatment-heterogeneity analysis, then the limitations. For everyone else: this study shows that people can value a personal trial even when the treatment does not beat placebo.
Sam: Keep those two successes separate.