Content Validity and Test-Retest Reliability of a Questionnaire to Measure Virtual Social Network Addiction Among University Students

AUTHORS

Samaneh Torkian ORCID 1 , Armita Shahesmaeili 2 , * , Neda Malekmohammadi 1 , Vahid Khosravi ORCID 3

1 Student Research Committee, Kerman University of Medical Sciences, Kerman, Iran

2 HIV/STI Surveillance Research Center, and WHO Collaborating Center for HIV Surveillance, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman , Iran

3 Department of Public Health, School of Health, Hamadan University of Medical Sciences, Hamadan, Iran

How to Cite: Torkian S, Shahesmaeili A, Malekmohammadi N, Khosravi V. Content Validity and Test-Retest Reliability of a Questionnaire to Measure Virtual Social Network Addiction Among University Students, Int J High Risk Behav Addict. Online ahead of Print ; 9(1):e92353. doi: 10.5812/ijhrba.92353.

ARTICLE INFORMATION

International Journal of High Risk Behaviors and Addiction: 9 (1); e92353
Published Online: March 4, 2020
Article Type: Research Article
Received: April 16, 2019
Revised: January 1, 2020
Accepted: February 2, 2020
Crossmark
Crossmark
CHECKING
READ FULL TEXT

Abstract

Background: Addiction to virtual social networks (VSNs), especially among students, has become a crisis during the recent years.

Objectives: This study aimed to assess the validity and reliability of a questionnaire on VSN addiction among Iranian university students.

Methods: The initial questionnaire was designed based on extensive literature review and consulting with experts. To measure the item and content validity indexes (I-CVI and S-CVI) and to measure the content validity ratio (CVR), a panel of 24 experts reviewed the questionnaire. To measure the test-retest reliability, the questionnaire was administered on 30 students within the interval of 14 - 21 days and the intra-class correlation coefficient (ICC) was calculated. Cronbach’s alpha and the corrected item-total correlation were calculated to measure internal consistency. All analysis was done using SPSS 20 software.

Results: All items had satisfactory CVR and I-CVI. The S-CVI was 0.98. The value of Cronbach’s alpha was 0.88. The corrected item-total correlation for all items, except one, was in acceptable range. This item was removed from the final questionnaire. The test-retest reliability of the questionnaire was almost perfect (ICC = 0.9).

Conclusions: The current study provides a valid and reliable questionnaire to measure VSN addiction among university students. The designed instrument could be used in addiction evaluation studies.

Copyright © 2020, Author(s). This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/) which permits copy and redistribute the material just in noncommercial usages, provided the original work is properly cited.

1. Background

Virtual social networks (VSNs) have gained popularity over the past decades and millions of users are currently using them (1, 2). These networks are so popular that life seems to be difficult without them. These VSNs have turned into a new way of communication, building relationships, finding new friends, entertainment and communication among people all over the world (3, 4), and have changed the way and quality of interpersonal communications significantly (5).

The attractiveness of virtual social networks has made people spend a lot of time online (6). There are a variety of terms for describing the overuse of the Internet, among them “Internet addiction” is more commonly used. This term is used for excessive and harmful use of the internet (7). Addiction to VSNs is a subset of Internet addiction which has become especially problematic within the recent years (6). It is shown that Internet addiction fulfils all the main components of the addiction model proposed by Griffiths in 2005 (8). These components are salience, tolerance, mood modifications, withdrawal, conflict and relapse. In fact, people who are addicted to VSNs spend a lot of time thinking of them and try to find opportunities to use these networks (salience). They often spend more time on these networks than they have planned, and try to increase the duration of their use to achieve enjoyment (tolerance). In addition, they use these networks to escape from the loneliness and depression (mood modifications) and if they are banned from using these networks, they get stressed, restless and irritable (withdrawal). They do not pay attention to the advice of others to diminish the time that they spend on these networks, and even when they decide to reduce the use of VSNs, they would not succeed (relapse). Other activities including their study, sports, and being with their families will be of less importance to these people, and it affects their quality of sleep, health, and communication (conflict) (8).

University students are a group at higher risk of Internet addiction. Around 85% of students use VSNs as the main root of communication (9). Establishing the communication regardless of distance, inserting and reading the university announcements, saving time and costs are the benefits of using these networks among students. However, spending too much time on VSNs may have negative effects on the academic performance of students (10, 11). In a study conducted in Birjand, Iran, the prevalence of mild and severe Internet addiction among high school students was reported to be 20.5% and 5.8% respectively (12). Another study in Iran showed that around 83.61% of university students suffer from mild to moderate Internet addiction (13).

Most studies conducted on Internet addiction among Iranian students were conducted before the emergence and widespread use of new VSNs such as WhatsApp and Telegram. Furthermore, they only focused on the Internet addiction while the problem of the recent years in not the Internet addiction but addiction to VSNs. In addition, most studies used researcher made questionnaires that their validity and reliability haven’t been evaluated properly.

2. Objectives

As a result, we conducted the present study to design and validate a questionnaire to measure VSN addiction among university students.

3. Methods

3.1. Instrument Development

After extensive literature review, we found variety of instrument used for identifying Internet addiction. These instruments were the Young Internet Addiction questionnaire (14) and the questionnaires and scales used in studies conducted by Mahmood and Farooq (15), Matar Boumosleh and Jaalouk (16), Savari et al. (17), Obaidi et al. (18), Liuokai et al. (19) and Bodroza and Jovanovic (20). After precise reading of these instruments and discussion with experts, we designed a questionnaire which could capture most dimensions of VSN addiction. The Questionnaire Consisted of Three Parts: The forepart included 3 demographic questions on gender (female, male), level of education (undergraduate, postgraduate, doctoral, PhD) and place of residence (dormitory, non-dormitory). The second part contained 3 questions on (1) the most common purpose of using VSNs; (2) the types of VSNs commonly used; and (3) the average time that students spend daily on using them. In the final part, we asked students 15 questions on various aspects of VSN addiction. The answering scheme contained two types of five point Likert scale including “never, slightly, moderate, high, and very high” for questions 1 - 5 and “never, rarely, sometimes, most often, and always” for questions 6 - 15.

3.2. Evaluating the Face and Content Validity

As most researchers believe that the face validity is a component of content validity and it is not an active measure of validity (21), we measured both face and content validity in the same manner using qualitative and quantitative approaches. In the qualitative phase, a team of 7 experts including two psychiatrists, two psychologists and three epidemiologists assessed the first version of the designed questionnaire through a face-to-face interview. Their suggestions for revision were addressed and the questions were modified according to their comments. Then the content validity of the modified version was evaluated quantitatively by calculating the content validity index (CVI) and content validity ratio (CVR).

3.3. Content Validity Ratio

The CVR was calculated for each question based on the formula developed by Lawsche. A panel of 24 experts including psychologists, psychiatrists, epidemiologists and health education and promotion specialists, were asked to review and rate the appropriateness of each question using a three point scale: “essential,” “useful but not essential,” or “not necessary.” Calculated CVRs were compared to the value of 0.41, the minimum required value for 24 raters based on Lawsche’s table (22, 23).

3.4. Content Validity Index

The content validity of the questionnaire was assessed using item and scale content validity index (I-CVI and S-CVI). The experts panel were asked to score the clarity, simplicity and relevance of each question using a four point Likert scale (ranging from not relevant/unclear to completely relevant/clear). To calculate the I-CVI for each item, the proportion of experts who gave a rating of either 3 or 4 to the total number of experts was computed (24, 25). To measure the S-CVI, the average of all I-CVIs was calculated (21, 26). Content validity indexes were considered to be acceptable when I-CVI and S-CVI were at least 0.78 and 0.90 respectively.

3.5. Evaluating the Reliability

To confirm the reliability of the questionnaire, two methods including the internal reliability and test-retest reliability were used. In order to measure the internal reliability and consistency of the questionnaire, Cronbach’s alpha and corrected item-total correlation were calculated. The minimum acceptable coefficient for Cronbach’s alpha was set at 0.7. To calculate the corrected item-total correlation, the correlation of each item with the total score of the questionnaire was assessed by calculation of the Pearson correlation coefficient. The items which had a coefficient of more than 0.3 were accepted (27). To measure the test-retest reliability, 20 students underwent two administrations of questionnaires with the interval of 14 - 21 days. Then, the intra-class correlation coefficients (ICC) were calculated. The coefficient of 0 - 0.2 was considered as slight, 0.21 - 04 as fair, 0.41 - 0.6 as moderate, 0.61-0.8 as substantial and > 0.8 as almost perfect reliability. All analysis were done using SPSS 20 software.

3.6. Ethics

The study protocol was reviewed and approved by the Ethics Committee of Kerman University of Medical Sciences (ethic code: IR.KMU.REC.1396.1806).

4. Results

The demographic characteristics of students who participated in the reliability study is presented in Table 1.

Table 1. Demographic Characteristics of Students Recruited in Test-Retest Reliability Study of Questionnaire on Virtual Network Addiction Among University Students
VariableNumber
Gender
Female10
Male10
Education
Undergraduate8
Postgraduate8
Doctoral2
PhD2
Place of residence
Dormitory10
Non-dormitory10

The quantitative measures of validity and reliability are summarized in Table 2. All items had satisfactory CVR and I-CVI (Table 2). The S-CVI calculated for the whole questionnaire was 0.98. The value of Cronbach’ alpha was also satisfactory (0.88). The corrected item-total correlation for all items except the item number 3 was in acceptable range (Table 2). This item was removed from the final questionnaire. The test-retest reliability of the questionnaire was almost perfect (ICC = 0.9). The final questionnaire consisted of 3 demographic questions (gender, education and place of residence), 1 question on the most common purpose of using VSNs, 1 question on the type of VSNs used, 1 question on average time that someone spent daily on using VSNs and 14 questions about VSNs addiction.

Table 2. Test-Retest and Content Validity Indicators of Questionnaire on virtual Network Addiction Among University Students, Kerman, Iran
RowPhrasesCVRI-CVI SimplicityI-CVI ClarityI-CVI RelateCorrected Item-Total Correlation
To what extend
1Spending time on VSANs has harmed your academic education?0.911110.66
2Do you prefer to go online than going out with your friends?0.911110.5
3Do you prefer to go online than going out with your family?0.831110.45
4 You can predict the time that you go back to use VSNs again?0.750.870.9110.11
5Spending time on VSNs has adversely affected your work?0.581110.74
6If someone disturbs you while you are online on VSNs make you upset, angry or screaming? 0.581110.55
7You tell this sentence to others while you are using VSNs: “Just a few minutes; I’m coming now”0.831110.67
8Have you tried to shorten the time you spend on VSNs, but you couldn’t do that?0.911110.61
9Do you try to hide from others that you are online?0.751110.44
10Have you ignored your family members because of being online?0.9110.950.950.64
11Other people complained to you because of being online? 0.751110.65
12Do you lose your sleep because of being late at night on VSNs?0.830.950.9510.64
13How much longer have you spent on VSNs than you planned?0.8310.950.870.54
14Have you checked VSNs before doing your priority tasks?11110.59
15Reading a message on VSNs occupies your mind when you get out of it?0.911110.49

5. Discussion

The purpose of present study was to develop a standard questionnaire for measurement of addiction to virtual social networks among students. The review of the previous studies showed a gap in measurement of addiction to VSNs. Therefore, the need for a specific instrument for measurement of this disorder made us develop and validated a questionnaire for this purpose. One of the strengths of the present study is that we asked a team of experts, including 24 different specialists in the field of addiction and psychology to review and validate the questionnaire. Furthermore, we recruited a representative sample of students from different fields of study and degrees for reliability of the study. Therefore, the instrument would be applicable in a wide range of settings. In the design stage, we tried to address most components of addiction suggested by Griffiths in 2005. These components are salience, mood modification, tolerance, withdrawal, conflict and relapse (8). The conflict is defined by Griffiths as the “conflict between the addict and those around them (interpersonal conflict) or from within the individual themselves (intrapsychic conflict) which are concerned with the particular activity” (8). As the conflict itself has a variety of aspects, 8 of our questions were assigned to this component. The questions number 2, 3, and 10 which ask about compromised relationship of individuals with their family and friends measure the personal conflicts while questions 9 and 11 measure the social conflicts that affect social life of individuals. In addition, questions 1 and 5 measure the educational and work conflict respectively.

Another component of addiction based on the Griffiths model is relapse and refers to “the tendency for repeated reversions to earlier patterns of the particular activity to recur and for even the most extreme patterns typical of the height of the addiction to be quickly restored after many years of abstinence or control” (8). We addressed this component by asking the question number 6: “have you tried to shorten the time you spend on VSNs, but you couldn’t do that?”. To measure the salience, which is defined as the situation “when the particular activity becomes the most important activity in the person’s life and dominates their thinking (preoccupations and cognitive distortions), feelings (cravings) and behavior (deterioration of socialized behavior” (8) we asked the questions number 12: “Do you lose your sleep because of being late at night on VSNs?”, number 14: “Have you checked VSNs before doing your priority tasks?” and number 15: “Reading a message on VSNs occupies your mind when you get out of it?”. The question number 6 asks about the withdrawal component of the addiction model which is defined as “unpleasant feeling states and/or physical effects which occur when the particular activity is discontinued or suddenly reduced” (8). Finally, we measured the tolerance, “the process whereby increasing amounts of the particular activity are required to achieve the former effects” (8), by asking the question number 13: “How much longer have you spent on VSNs than you planned?”.

In addition to the comprehensiveness of the designed questionnaire in measuring various components of addiction, the high value of I-CVI and S-CVI confirms the high validity of the instrument. Furthermore, the high value of Cronbach’s alpha coefficient and ICC and acceptable values of corrected item-total correlation indicate the satisfactory reliability of the questionnaire.

We would like to acknowledge the limitation of our study. We just measured the face and content validity of the questionnaire which are the most important aspects of questionnaire validation. Other aspects of validity including criterion, convergence and predictive validity were not addressed in this study.

5.1. Conclusions

The designed questionnaire provides a standardized, valid, and reliable instrument to measure the extent of virtual social network addiction among Iranian university students. The instrument could be applicable in routine evaluation of students and designing educational interventions.

Acknowledgements

Footnotes

References

  • 1.

    Pelling EL, White KM. The theory of planned behavior applied to young people's use of social networking Web sites. Cyberpsychol Behav. 2009;12(6):755-9. doi: 10.1089/cpb.2009.0109. [PubMed: 19788377].

  • 2.

    Wilson K, Fornasier S, White KM. Psychological predictors of young adults' use of social networking sites. Cyberpsychol Behav Soc Netw. 2010;13(2):173-7. doi: 10.1089/cyber.2009.0094. [PubMed: 20528274].

  • 3.

    Raacke J, Bonds-Raacke J. MySpace and Facebook: applying the uses and gratifications theory to exploring friend-networking sites. Cyberpsychol Behav. 2008;11(2):169-74. doi: 10.1089/cpb.2007.0056. [PubMed: 18422409].

  • 4.

    Hanany Abd Rahman S. Can’t Live without my FB, LoL: The influence of Social Networking Sites on the communication skills of TESL Students. Proced Soc Behav Sci. 2014;134:213-9. doi: 10.1016/j.sbspro.2014.04.241.

  • 5.

    Stieger S, Burger C, Bohn M, Voracek M. Who commits virtual identity suicide? Differences in privacy concerns, Internet addiction, and personality between Facebook users and quitters. Cyberpsychol Behav Soc Netw. 2013;16(9):629-34. doi: 10.1089/cyber.2012.0323. [PubMed: 23374170].

  • 6.

    Kuss DJ, Griffiths MD. Online social networking and addiction--a review of the psychological literature. Int J Environ Res Public Health. 2011;8(9):3528-52. doi: 10.3390/ijerph8093528. [PubMed: 22016701]. [PubMed Central: PMC3194102].

  • 7.

    Spada MM. An overview of problematic internet use. Addict Behav. 2014;39(1):3-6. doi: 10.1016/j.addbeh.2013.09.007. [PubMed: 24126206].

  • 8.

    Griffiths MD. A ‘components’ model of addiction within a biopsychosocial framework. J Subst Use. 2009;10(4):191-7. doi: 10.1080/14659890500114359.

  • 9.

    Heiberger G, Harper R. Have you facebooked Astin lately? Using technology to increase student involvement. New Dir Stud Serv. 2008;2008(124):19-35. doi: 10.1002/ss.293.

  • 10.

    Seder JP, Oishi S. Ethnic/racial homogeneity in college students’ Facebook friendship networks and subjective well-being. J Res Pers. 2009;43(3):438-43. doi: 10.1016/j.jrp.2009.01.009.

  • 11.

    Thompson LA, Dawson K, Ferdig R, Black EW, Boyer J, Coutts J, et al. The intersection of online social networking with medical professionalism. J Gen Intern Med. 2008;23(7):954-7. doi: 10.1007/s11606-008-0538-8. [PubMed: 18612723]. [PubMed Central: PMC2517936].

  • 12.

    Turi A, Miri M, Beheshti D, Yari E, Khodabakhshi H, Anani Sarab G. Prevalence of Internet addiction and its relationship with anxiety, stress, and depression in intermediate students in Birjand City in 2014. J Birjand Univ Med Sci. 2015;22(1):67-75.

  • 13.

    Doosti Irani A, Bagheri Amiri F, Khajehkazemi R, Mostafavi E. Prevalence of internet addiction among students and graduates of epidemiology, clinical sciences, and basic sciences in Iran: A cross-sectional study. Iran J Epidemiol. 2017;13(1):14-21.

  • 14.

    Faraci P, Craparo G, Messina R, Severino S. Internet Addiction Test (IAT): Which is the best factorial solution? J Med Internet Res. 2013;15(10). e225. doi: 10.2196/jmir.2935. [PubMed: 24184961]. [PubMed Central: PMC3806548].

  • 15.

    Mahmood S, Farooq U. Facebook addiction: A study of big-five factors and academic performance amongst students of IUB. Global J Manage Bus Res. 2014.

  • 16.

    Matar Boumosleh J, Jaalouk D. Depression, anxiety, and smartphone addiction in university students- A cross sectional study. PLoS One. 2017;12(8). e0182239. doi: 10.1371/journal.pone.0182239. [PubMed: 28777828]. [PubMed Central: PMC5544206].

  • 17.

    Savari K. Build and accelerate mobile addiction inventory. Educ Measur. 2014;4(15):126-42.

  • 18.

    Obaidi N, Kiani MR/, RostamiNejad MA. [Validation of questionnaire of psycho-social aspects of phone based social networks' use]. Hum Inf Interact. 2016;3. Persian.

  • 19.

    Shi Y, Luo YLL, Yang Z, Liu Y, Cai H. The development and validation of the Social Network Sites (SNSs) usage questionnaire. 8531. Germany: Springer; 2014. p. 113-24. doi: 10.1007/978-3-319-07632-4_11.

  • 20.

    Bodroza B, Jovanovic T. Validation of the new scale for measuring behaviors of Facebook users: Psycho-Social Aspects of Facebook Use (PSAFU). Computers Hum Behav. 2016;54:425-35. doi: 10.1016/j.chb.2015.07.032.

  • 21.

    Bolarinwa OA. Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Niger Postgrad Med J. 2015;22(4):195-201. doi: 10.4103/1117-1936.173959. [PubMed: 26776330].

  • 22.

    Jafari MJ, Yazdi FDF, Mehrabi Y, Rakhshanderou S, Saremi M. Development and validation of a new scale for prediction of low back pain occurrence among nurses. EXCLI J. 2019;18:277.

  • 23.

    Wilson FR, Pan W, Schumsky DA. Recalculation of the critical values for Lawshe’s content validity ratio. Measur Eval Couns Dev. 2017;45(3):197-210. doi: 10.1177/0748175612440286.

  • 24.

    Joshi A, Kale S, Chandel S, Pal D. Likert scale: Explored and explained. Br J Appl Sci Tech. 2015;7(4):396-403. doi: 10.9734/bjast/2015/14975.

  • 25.

    Asun RA, Rdz-Navarro K, Alvarado JM. Developing multidimensional Likert scales using item factor analysis. Soc Methods Res. 2015;45(1):109-33. doi: 10.1177/0049124114566716.

  • 26.

    Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007;30(4):459-67. doi: 10.1002/nur.20199. [PubMed: 17654487].

  • 27.

    Mukaka MM. A guide to appropriate use of correlation coefficient in medical research. Malawi Med J. 2012;69(71):69-71.

  • COMMENTS

    LEAVE A COMMENT HERE: