Analysis of Website Usability of Provincial Governments in Indonesia with The Heuristic Evaluation Method

– The Indonesian government initiated the implementation of E-government as a public service in 2003 with Presidential Instruction No.3 of 2003 issuance of National Policy and Strategy for E-Government Development. This rule was strengthened in 2007 by Regulation of the Minister of Communication and Information Technology No.41/PER/M.KOMINFO/2007 concerning General Guidelines for National Information and Communication Technology Governance. However, many provincial government sites still have not implemented a usability test. This problem is reinforced by the website interface not meeting the standards set by the government. So, usability testing is needed to find the problems found on the provincial government website. The method used in this usability test is the heuristic evaluation method. The evaluation was carried out by involving three evaluators to evaluate 28 provincial government sites. The evaluator detects problems based on ten heuristic principles proposed by Jakob Nielsen and is assessed with a severity rating. analysis results show the province of Aceh and the Special Region of Yogyakarta website is ranked first with the lowest severity rating value with a value of 0.43. In contrast, North Kalimantan is ranked last with the highest severity rating value with a value of 1.08. Around 71.4% of provincial government websites have no problems with user control & freedom, and consistency & standards. Then half of the provincial government's websites have problems with recognition rather than recall. Last, more than 50% of websites have problems with other heuristic principles.


INTRODUCTION
In 2003, the rapid development of information technology (IT) made Indonesia issue a policy to utilize the IT by providing access to information to the public. According to [1], E-government is applying information technology in the administration level to realize better public services and support the industry. This matter is manifested in Presidential Instruction No. 3 of 2003 concerning National Policy and Strategy for E-Government Development as a form of seriousness of the Indonesian government to implement e-government [2] The Presidential Instruction explains that all institutions or institutions must implement good governance through e-government for accountability in government administration and to improve efficient, effective, and transparent public services. In 2007, The e-government policy is also regulated in the Regulation of the Minister of Communication and Information Technology No. 41/PER/M.KOMINFO/2007 concerning General Guidelines for National Information and Communication Technology Governance [3]. The decision holds the guidelines or standarts for E-Government, which must be a reference for e-government development in every government institution.
With this regulation, government agencies began to develop public service facilities for E-government, such as websites. However, many agencies, especially the provincial government, still create websites without conducting usability tests. This issue can be seen in many provincial government websites that do not pay attention to website guidelines that the Indonesian government has determined. Without evaluation, it will not know how the condition of the website has been in its implementation since it was developed [4]. In addition, problems in the website design interface will affect the impression and experience when users use it [5].
In other studies, usability testing is useful for meeting the community's needs, which will experience changes over time [6]. Therefore, maintenance is needed by conducting a usability test. According [7], definition of usability is the degree to which specific users can use a system, product, or service to achieve specific goals with effectiveness, efficiency, and satisfaction in a particular usage context. Another definition according [8], is the experience experienced by users in using specific applications, websites, or technologies.
Usability evaluation is needed to find out the extent of the ease of using the website. The method used in this study is heuristic evaluation. This method aims to identify usability problems related to interface design. The benefit of heuristic evaluation is that the process can be conducted easily and quickly [9].
Previous research [10] with the same object of this research, titled "EVALUATION OF PROVINCE GOVERNMENT'S WEB PERFORMANCE IN INDONESIA," inspire for conducting this research. Unlike research [10] evaluating website performance through pagespeed applications, this research evaluates in terms website design interface through usability tests. Compared to other methods, such as the cognitive walkthrough method, where the evaluation process involves new users as respondents, heuristic evaluation involves respondents from UX experts as evaluators [11]. This method is carried out by evaluators who evaluate the website's interface by considering Jakob Nielsen's ten heuristics principles for interaction design. By using this method, it can be seen which provincial government websites that have been appropriately developed meet the Jakob Nielsen's ten heuristics principles and as comparison for further research.

Procedure
The heuristic evaluation method is a technique to measure the extent of the usability problem (usability) of software in the design interface [12]. This research is quantitative. The number of evaluators in this study was three people. The more evaluators, the more usability problems found [13].  Testing was carried out from June 4, 2022, to June 10, 2022, using the zoom application. The evaluator evaluates the website by analyzing several pages from each provincial website. Next Every time the evaluator finishes analyzing a website, the evaluator fills out the assessment form in the google form that has been provided. The results of the three evaluators will be analyzed.

Ten heuristics principles
In using heuristic evaluation, there are ten design principles put forward by Jakob Nielsen as a reference in evaluating website interfaces [6]. Table 1. Jakob Nielsen's ten heuristics principles

Kode
Principle Description Z1 visibility of system status The system provides status information to users. Z2 Match between system and the real world Presents language and icons that are easy to understand.

Z3
User control and freedom The system gives users the freedom to operate the system Z4 Consystency and standards elements in the system consistently and do not change. Z5 Error prevention The system design prevents users from making mistakes. Z6 Recognition rather than recall The system's design is easy to remember to use, minimizing to remember. Z7 Flexibility and efficiency of use The system provides options for operating the system. Z8 Aesthetic and minimalist design The appearance of the design is simple and not too crowded. Z9 Help user recognize , diagnose, and recover from errors The system alerts if there is an error and how to fix it.

Z10 Help and documentation
The system assists in finding information quickly.

Sub-aspects of 10 principles of design
The following sub-aspects are used as assessment questions during testing.  immediately to be fixed before it was released.
According to [15], The evaluation value is obtained by calculating the heuristic principle in Table 1. Jakob Nielsen's ten heuristics principles. Calculation of the heuristic evaluation using the equation Error! Reference source not found..

∑
= * + * + * + * + * Descripstion: ∑ : Heuristic principles (Z1, Z2, Z3, . . . , Z10), the total value of the sub-aspects of each aspect. : The number of evaluators who gave a rating Then use the equation Error! Reference source not found. to generate severity rating values for each aspect of the heuristic principle.

= ∑
(2) Description: : Severity rating results in every aspect of heuristic principles : Total of sub-aspects in each aspect of the heuristic principle.

Provincial government website
The following is a list of provincial government from [10], whose websites are the research object.
0 0 3 Description : Z1 -Z10: Jakob Nielsen's ten heuristics principles. A1 -J3 : Sub-aspects of heuristic principles PR20 : West Kalimantan Province PR21 : Central Kalimantan Province PR22 : South Kalimantan Province During the evaluation process, some websites were not accessible due to server problems. The websites of the governments of the Provinces of South Sumatra, the Special Region of Yogyakarta, and West Papua could not be accessed by the two evaluators. Meanwhile, the websites of the Provincial Governments of West Nusa Tenggara, South Kalimantan and North Sulawesi could not be accessed by one evaluator.

Calculation Process
The results of the evaluation from the evaluator are then searched for the severity rating of the aspect from each province. The following is an example of calculating the severity rating aspect of the visibility of system status in the province of South Kalimantan and the Aesthetic and minimalist design aspect of the province of Central Kalimantan.  : Total : the number of severity ratings for each sub-aspect H1 -H5 : subaspect of aesthetic & minimalist design A1 -A5 : subaspect of visibility of system status Value : the number of severity ratings of each sub-aspect divided by five In the two tables above, the severity rating of the aspect is calculated using Error! Reference source not found. and Error! Reference source not found.. For example, the calculation of Table 5 sub-aspects of A3. Using Error! Reference source not found., acquired 0*1 + 1*0 + 2*1 + 3*0 + 4*0 (x is the number of evaluators who gave the score) the result is 2. Then divided by five (severity rating value, 0 to 4), the result is 0.4. Then add up the severity rating of the sub-aspects A1 -A4; the result is 1.2. Then using Error! Reference source not found., the sum of the severity rating values of the sub-aspects divided by four (the number of sub-aspects for each aspect) results in the severity rating of the usability aspect.
With the explanation of the previous calculations, the following is the severity rating value for each province's government website. : Jakob Nielsen's ten heuristics principles. See Table 1.

CONCLUSION
The analysis results show that from 28 websites, most provincial government websites have no problems with aspects of "user control & freedom" and "consistency & standards". The usability problems found in aspects of "visibility of system status", "match between system & the real world", "error prevention", "recognition rather than recall", "flexibility & efficiency of use", "aesthetic & minimalist design", "help user recognize", "diagnose, & recover from errors", and "help & documentation". Based on the averages in Table 8, Aceh and Special Region Of Yogyakarta provinces ranked first for websites with the lowest usability problems with a score of 0.43 (closer to 0). Meanwhile, the North Kalimantan website was in the last position as the website with the most usability problems, with a score of 1.08.